You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 00:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Apr 28, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31239285510,832sleep131275-21ntpq10:21:321
25125281616,784sleep018405-21kworker/u8:108:20:420
8919264543,562sleep1191-21jbd2/mmcblk2p2-09:11:061
88123498,303sleep1927-21ntpq08:45:581
2223229310,268sleep2191-21jbd2/mmcblk2p2-12:15:472
11422248103,31sleep10-21swapper/107:05:461
26269224339,20sleep10-21swapper/110:05:571
16864223918,207sleep316936-21ls09:36:033
16864223918,207sleep316936-21ls09:36:033
21692220429,161sleep121768-21latency_hist11:35:331
8135220311,172sleep18286-21apt-key09:10:331
8135220311,172sleep18286-21apt-key09:10:331
4329220110,175sleep14471-21latency_hist10:40:331
32428219922,163sleep032469-21ntpq07:00:570
12341219115,155sleep112465-21latency_hist11:05:331
2495421800,13sleep10-21swapper/111:45:331
32163217915,149sleep132275-21cat12:06:011
19490217512,148sleep119519-21fschecks_count09:45:501
19490217512,148sleep119519-21fschecks_count09:45:501
21548217430,129sleep121686-21latency_hist08:10:331
30449217216,22sleep10-21swapper/112:01:001
28465217014,133sleep10-21swapper/108:30:521
25981216816,127sleep10-21swapper/111:46:591
787121679,144sleep17386-21apt-get10:50:431
11637216716,38sleep00-21swapper/009:20:460
163372165122,25sleep116465-21timerwakeupswit11:16:021
2911216414,21sleep10-21swapper/112:16:021
3012321608,92sleep10-21swapper/108:35:531
1331215916,100sleep11480-21grep12:11:021
20326215816,126sleep320386-21fschecks_time08:05:513
18536215315,20sleep10-21swapper/109:41:021
13271214917,76sleep10-21swapper/109:25:521
5100214414,80sleep10-21swapper/110:41:021
29039213912,26sleep00-21swapper/008:31:030
17062213962,63sleep117210-21uptime09:36:051
17062213962,63sleep117210-21uptime09:36:051
6331213630,37sleep10-21swapper/107:20:511
16943213518,95sleep017059-21cat09:36:040
16943213518,95sleep017059-21cat09:36:040
535721349,113sleep013441-21snmpd12:25:500
9339213217,22sleep00-21swapper/007:30:480
22436213215,76sleep10-21swapper/111:36:011
10092213113,75sleep10-21swapper/109:15:461
925213089,19sleep10-21swapper/107:05:331
29656213015,100sleep00-21swapper/012:00:310
836212628,29sleep3889-21systemd-run07:05:313
30799212620,85sleep330784-21apt-get10:20:493
23333212514,72sleep10-21swapper/109:56:571
7498212316,91sleep00-21swapper/007:25:310
1222121229,98sleep112357-21date07:40:341
987821179,71sleep19925-21ntpq10:55:561
28059211511,89sleep028061-21sshd11:55:220
17341211419,78sleep317363-21fschecks_count11:20:493
25457211225,66sleep225540-21tune2fs11:46:482
13773211114,82sleep113929-21date07:45:341
9948211014,80sleep310073-21timerandwakeup07:31:033
8167211012,82sleep28206-21/usr/sbin/munin07:25:572
24037211017,77sleep024044-21modprobe10:00:310
1205211011,82sleep21318-21cat10:30:332
4319210712,81sleep04328-21/usr/sbin/munin10:40:320
10795210716,75sleep210827-21latency_hist11:00:332
815721069,83sleep18238-21date12:35:321
487621058,84sleep14912-21systemd-run09:00:321
487621058,84sleep14912-21systemd-run09:00:321
26418210569,23sleep126419-21cron08:25:311
348021049,78sleep23495-21/usr/sbin/munin10:35:582
2673210313,74sleep12777-21cat08:51:031
15267210313,78sleep115302-21awk07:50:311
15267210313,78sleep115302-21awk07:50:311
4816210217,67sleep20-21swapper/208:59:522
4816210217,67sleep20-21swapper/208:59:522
27953210116,63sleep328024-21/usr/sbin/munin10:11:003
1289621019,21sleep112934-21modprobe09:25:311
16715210016,64sleep216289-21/usr/sbin/munin09:35:582
16715210016,64sleep216289-21/usr/sbin/munin09:35:582
15642210020,63sleep015732-21fschecks_time07:50:500
15642210020,63sleep015732-21fschecks_time07:50:500
1344821008,74sleep00-21swapper/009:25:550
1383729920,61sleep213847-21modprobe11:10:302
2953429812,16sleep10-21swapper/110:15:581
278029814,65sleep22006-21/usr/sbin/munin08:51:042
1603929812,17sleep10-21swapper/107:50:591
1603929812,17sleep10-21swapper/107:50:591
524729711,14sleep10-21swapper/112:25:431
2543229715,65sleep325448-21/usr/sbin/munin08:21:013
1931129713,17sleep119383-21mailstats11:26:001
1844229766,20sleep11-21systemd08:00:321
1844229766,20sleep11-21systemd08:00:321
178729715,27sleep30-21swapper/308:50:333
737129631,45sleep27421-21apt-config10:50:322
737129631,45sleep27421-21apt-config10:50:322
359129512,19sleep13654-21mailstats07:10:581
359129512,19sleep13654-21mailstats07:10:581
323902957,15sleep10-21swapper/110:25:481
2744229512,17sleep10-21swapper/111:51:031
2398129515,37sleep10-21swapper/108:16:021
2398129515,37sleep10-21swapper/108:16:021
52922948,15sleep10-21swapper/109:00:521
52922948,15sleep10-21swapper/109:00:521
1999829460,19sleep120107-21sed08:05:331
1999829460,19sleep120107-21sed08:05:331
128472949,16sleep10-21swapper/111:05:531
807729322,48sleep38104-21munin-run12:35:303
1599329326,47sleep316009-21modprobe09:35:313
2410429223,44sleep224252-21grep08:16:032
2410429223,44sleep224252-21grep08:16:032
2175829213,46sleep10-21swapper/108:10:421
2253329112,61sleep222610-21grep11:36:022
14852916,13sleep10-21swapper/110:30:481
971429010,67sleep19755-21ntpq07:30:571
49782909,13sleep10-21swapper/107:15:551
49782909,13sleep10-21swapper/107:15:551
371928957,20sleep13718-21apt-get12:20:431
1512928910,14sleep10-21swapper/109:30:561
3184328818,47sleep30-21swapper/308:41:003
2228328812,59sleep322363-21mailstats08:11:003
1440128816,49sleep314105-21/usr/sbin/munin11:10:563
2836728734,32sleep10-21swapper/111:55:431
244302879,13sleep10-21swapper/110:00:521
84792868,12sleep10-21swapper/107:26:041
63852856,39sleep10-21swapper/110:45:541
63852856,39sleep10-21swapper/110:45:541
2576728511,55sleep00-21swapper/011:46:540
2797028420,52sleep127996-21munin-run08:30:321
1510028414,48sleep014707-21/usr/sbin/munin09:30:550
136828419,42sleep21143-21/usr/sbin/munin07:05:552
839528317,44sleep08476-21cat07:26:030
211802838,38sleep10-21swapper/109:50:511
316022826,38sleep10-21swapper/108:40:541
286852826,11sleep10-21swapper/110:15:261
184042825,11sleep10-21swapper/111:24:061
1626228220,27sleep00-21swapper/011:16:010
84528122,25sleep00-21swapper/008:45:570
3210228117,47sleep232135-21latency_hist10:25:322
693127911,16sleep00-21swapper/009:05:520
693127911,16sleep00-21swapper/009:05:520
45727915,20sleep30-21swapper/310:26:023
2960927918,38sleep229693-21timerandwakeup10:15:592
2122027916,48sleep221303-21latency09:50:522
2909127830,31sleep30-21swapper/310:15:483
1994727811,44sleep319980-21awk08:05:313
1994727811,44sleep319980-21awk08:05:313
147322788,51sleep20-21swapper/209:30:452
793427720,31sleep27890-21apt-get10:51:122
3094127711,51sleep031040-21grep10:21:260
2937527712,45sleep028981-21/usr/sbin/munin10:15:530
143132777,39sleep10-21swapper/111:10:541
119072778,55sleep011908-21sshd07:37:090
320227613,41sleep03296-21fschecks_time07:10:490
320227613,41sleep03296-21fschecks_time07:10:490
863227518,36sleep38764-21tail10:51:423
516127517,40sleep25225-21ps07:16:002
516127517,40sleep25225-21ps07:16:002
437727523,32sleep24438-21sendmail-msp07:15:312
437727523,32sleep24438-21sendmail-msp07:15:312
3062327513,47sleep330737-21sed12:01:023
2221427525,33sleep022218-21cron09:55:300
207162759,50sleep020835-21latency_hist09:50:340
207162759,50sleep020835-21latency_hist09:50:340
244627414,46sleep02534-21latency12:15:530
2161127425,31sleep321627-21modprobe11:35:303
1689427417,36sleep216963-21latency_hist07:55:322
1689427417,36sleep216963-21latency_hist07:55:322
1282727417,24sleep30-21swapper/307:40:563
587627324,25sleep00-21swapper/010:45:330
3114827322,24sleep20-21swapper/208:40:342
2486527314,34sleep024951-21latency_hist11:45:330
2458727326,24sleep30-21swapper/310:00:543
198827315,41sleep22056-21/usr/sbin/munin10:31:012
1895127320,33sleep20-21swapper/208:00:532
1895127320,33sleep20-21swapper/208:00:532
1171927322,27sleep30-21swapper/309:20:493
1751727225,22sleep20-21swapper/211:20:522
1139927214,43sleep011429-21nfs4_client11:00:550
65127119,23sleep30-21swapper/312:10:473
190427112,37sleep01420-21/usr/sbin/munin10:30:590
1845327123,26sleep20-21swapper/208:00:332
1845327123,26sleep20-21swapper/208:00:332
507527018,28sleep30-21swapper/312:25:333
3010127018,27sleep00-21swapper/008:35:520
3141126912,40sleep031455-21/usr/sbin/munin08:40:480
247126914,30sleep02541-21munin-node07:10:330
2344426915,25sleep00-21swapper/008:15:490
2344426915,25sleep00-21swapper/008:15:490
130032697,14sleep00-21swapper/007:41:010
496126813,23sleep00-21swapper/007:15:540
496126813,23sleep00-21swapper/007:15:540
292626810,40sleep32939-21/usr/sbin/munin08:51:053
193926812,42sleep02029-21ls12:15:330
659826715,37sleep36669-21tr10:46:003
659826715,37sleep36669-21tr10:46:003
2365926720,31sleep223722-21/usr/sbin/munin11:40:502
828826618,30sleep28003-21apt-get09:10:342
22712669,14sleep00-21swapper/008:50:540
2142626610,47sleep00-21swapper/008:09:400
1571726624,22sleep20-21swapper/211:15:472
752026521,25sleep00-21swapper/010:50:340
752026521,25sleep00-21swapper/010:50:340
372726520,23sleep20-21swapper/212:20:482
2756726517,30sleep227630-21/usr/sbin/munin10:10:512
271642656,13sleep30-21swapper/308:25:583
2713926511,39sleep227163-21ntp_states08:25:572
2666626521,26sleep00-21swapper/010:06:040
2657326513,39sleep326692-21cat11:50:323
1469026513,31sleep314828-21grep07:46:053
1469026513,31sleep314828-21grep07:46:053
728026415,26sleep00-21swapper/012:30:590
582426416,28sleep25849-21awk10:45:312
502626421,24sleep30-21swapper/310:41:003
487026416,34sleep04875-21modprobe09:00:310
487026416,34sleep04875-21modprobe09:00:310
476526422,26sleep34774-21df_inode07:15:493
476526422,26sleep34774-21df_inode07:15:493
2326726420,18sleep20-21swapper/209:56:542
1428026418,26sleep20-21swapper/207:45:562
1428026418,26sleep20-21swapper/207:45:562
1364826415,30sleep213689-21proc_pri09:26:012
1347326412,37sleep313482-21/usr/sbin/munin11:06:033
1245926420,22sleep20-21swapper/207:40:472
314922635,13sleep20-21swapper/212:05:462
2482026322,24sleep20-21swapper/210:01:012
2441326318,29sleep00-21swapper/010:00:510
205726317,28sleep32183-21/usr/sbin/munin10:31:023
2004426316,33sleep220171-21cat09:46:032
2004426316,33sleep220171-21cat09:46:032
1869026312,35sleep218756-21users09:41:032
1481126317,21sleep00-21swapper/011:11:050
1415826318,25sleep00-21swapper/007:45:540
1415826318,25sleep00-21swapper/007:45:540
1369026313,20sleep30-21swapper/309:26:023
380026213,19sleep00-21swapper/012:20:500
3230626220,17sleep20-21swapper/207:00:552
212562626,12sleep20-21swapper/211:31:052
1128226220,24sleep30-21swapper/307:35:553
965926114,34sleep09701-21idleruntime10:55:510
3109326122,25sleep331102-21modprobe08:40:313
276726121,24sleep30-21swapper/307:10:363
276726121,24sleep30-21swapper/307:10:363
180522610,6sleep00-21swapper/011:21:010
1619826112,18sleep30-21swapper/311:16:003
658026016,20sleep20-21swapper/212:30:332
52126020,17sleep00-21swapper/010:26:030
2951026011,15sleep30-21swapper/308:35:263
2346526014,31sleep323584-21sed09:56:593
1986726019,25sleep319891-21/usr/sbin/munin09:46:003
1986726019,25sleep319891-21/usr/sbin/munin09:46:003
1721926013,25sleep00-21swapper/007:55:510
1721926013,25sleep00-21swapper/007:55:510
740025918,19sleep20-21swapper/209:06:022
740025918,19sleep20-21swapper/209:06:022
3263925914,26sleep232668-21munin-run08:45:312
294422597,10sleep20-21swapper/211:57:462
2261125922,20sleep30-21swapper/311:36:033
1888625917,26sleep218890-21diskstats11:25:482
1163525920,18sleep20-21swapper/207:36:022
919625813,29sleep39315-21sed09:11:123
312192589,36sleep031245-21sh12:05:310
2722625817,18sleep20-21swapper/210:10:342
185925818,25sleep21887-21munin-run12:15:312
174025815,26sleep21765-21/usr/sbin/munin08:50:322
119432587,12sleep20-21swapper/211:01:062
1059525822,23sleep310673-21mailstats09:16:003
2959725717,20sleep20-21swapper/208:35:342
2595925714,26sleep325971-21/usr/sbin/munin10:05:513
2166725722,18sleep30-21swapper/309:51:013
1388925720,18sleep00-21swapper/011:10:320
833025621,19sleep00-21swapper/010:51:350
524625612,29sleep05291-21fschecks_count09:00:510
524625612,29sleep05291-21fschecks_count09:00:510
32625615,23sleep00-21swapper/012:10:320
1582525613,28sleep315913-21sed07:50:533
1582525613,28sleep315913-21sed07:50:533
555925516,18sleep20-21swapper/209:00:572
555925516,18sleep20-21swapper/209:00:572
2178725520,18sleep20-21swapper/208:10:472
1946825510,29sleep319590-21timerwakeupswit11:26:023
1293525512,27sleep212946-21munin-run09:25:322
1103225514,24sleep311027-21apt-get11:00:463
1082825515,20sleep30-21swapper/311:00:343
1046725512,25sleep20-21swapper/209:15:552
1036225514,21sleep00-21swapper/009:15:530
985525416,19sleep00-21swapper/009:15:330
574325416,18sleep30-21swapper/309:01:023
574325416,18sleep30-21swapper/309:01:023
494325418,16sleep30-21swapper/309:00:343
494325418,16sleep30-21swapper/309:00:343
797825313,17sleep30-21swapper/309:10:313
797825313,17sleep30-21swapper/309:10:313
2815225312,24sleep20-21swapper/211:55:322
2802625316,17sleep00-21swapper/010:11:010
2567525311,25sleep225767-21mailstats08:21:082
706025213,16sleep20-21swapper/212:30:532
2689125212,24sleep326897-21diskstats11:50:493
159525213,23sleep31618-21ntp_states07:06:003
1070825212,15sleep20-21swapper/207:35:342
2683025111,16sleep00-21swapper/008:25:510
237382516,11sleep30-21swapper/308:15:563
237382516,11sleep30-21swapper/308:15:563
2323225117,19sleep00-21swapper/011:40:320
217525112,22sleep32171-21apt-get12:15:443
786425012,23sleep37873-21diskstats07:25:503
709025012,22sleep27142-21users07:21:052
665625012,23sleep36693-21ntp_states07:20:573
570125015,20sleep35724-21ntp_states12:25:573
2861825012,23sleep328659-21irqstats11:55:543
2102425015,18sleep00-21swapper/011:31:030
195962509,16sleep00-21swapper/011:26:030
2477524914,22sleep324863-21gpgv11:45:323
2312224910,25sleep023077-21latency_hist08:15:330
145592498,16sleep20-21swapper/211:10:592
1738124811,22sleep317404-21irqstats07:55:543
1738124811,22sleep317404-21irqstats07:55:543
1785424710,20sleep317851-21apt-get09:40:433
303862459,20sleep230397-21ntp_states12:00:572
202382458,19sleep20-21swapper/208:05:462
1916724511,15sleep00-21swapper/009:45:330
188132456,16sleep00-21swapper/008:00:510
188132456,16sleep00-21swapper/008:00:510
386224311,16sleep30-21swapper/310:36:033
314882398,16sleep30-21swapper/312:05:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional