You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 07:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sat Dec 20, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11264254688,5447sleep218394-21kworker/2:119:40:462
11264254688,5447sleep218394-21kworker/2:119:40:462
3120823078,284sleep231275-21systemctl20:45:272
27636229613,254sleep227692-21fschecks_time23:59:572
14962227996,125sleep215106-21unixbench_singl21:35:092
1296022766,261sleep2369-21snmpd19:47:452
4084227016,234sleep24146-21fschecks_time22:44:552
1553722676,196sleep215611-21find19:55:072
27271223851,113sleep20-21swapper/223:59:372
13477223757,164sleep213558-21fschecks_time19:49:552
329223236,148sleep20-21swapper/219:09:492
3708222145,115sleep23718-21modprobe22:44:342
3708222145,115sleep23718-21modprobe22:44:342
24041221254,116sleep20-21swapper/222:04:492
23856219341,100sleep223894-21http23:49:352
23856219341,100sleep223894-21http23:49:352
20522219113,161sleep020413-21latency_hist21:54:370
27587218617,152sleep327611-21/usr/sbin/munin22:15:043
15329217512,110sleep20-21swapper/223:20:042
13990216526,84sleep214000-21munin-run21:34:372
22594216415,133sleep022686-21fschecks_time21:59:550
1761216247,93sleep20-21swapper/220:55:202
4423216115,102sleep20-21swapper/200:29:532
4423216115,102sleep20-21swapper/200:29:532
17244215862,34sleep217386-21unixbench_singl20:00:082
17244215862,34sleep217386-21unixbench_singl20:00:082
28999215714,102sleep229033-21fschecks_count20:39:532
6494215616,125sleep06559-21cpuspeed21:09:520
29966215612,107sleep21-21systemd22:24:362
31995215514,124sleep132146-21latency_hist00:14:381
30940215510,47sleep20-21swapper/200:09:582
10061215512,95sleep210097-21latency_hist19:39:372
5619215238,98sleep25640-21apt-get19:24:472
2442521518,49sleep20-21swapper/223:50:202
996321509,126sleep210022-21date23:04:372
25886214815,94sleep225966-21fschecks_time20:29:562
18009214556,73sleep218053-21ntp_kernel_err21:45:352
3244214212,88sleep20-21swapper/200:25:012
30708213910,25sleep20-21swapper/222:25:042
3964213715,106sleep14063-21fschecks_time19:19:531
5344213322,93sleep20-21swapper/219:24:392
18428213312,106sleep218488-21find23:30:052
3356213215,89sleep00-21swapper/020:59:460
3356213215,89sleep00-21swapper/020:59:460
186121310,7sleep00-21swapper/019:14:380
31947212817,92sleep331973-21grep20:49:353
24376212811,16sleep20-21swapper/220:24:552
24376212811,16sleep20-21swapper/220:24:552
23965212715,95sleep223996-21awk20:24:352
17761212419,85sleep117782-21modprobe20:04:351
17761212419,85sleep117782-21modprobe20:04:351
596121229,18sleep20-21swapper/200:34:502
3554212016,60sleep20-21swapper/221:00:332
3554212016,60sleep20-21swapper/221:00:332
2826211919,78sleep12858-21sh20:59:351
2595521199,65sleep20-21swapper/222:10:022
5905211613,85sleep35955-21/usr/sbin/munin19:25:333
11158211611,92sleep211178-21cpu21:24:522
5639211316,80sleep15732-21fschecks_time22:49:541
2155021139,93sleep221617-21systemctl23:40:062
32290211241,44sleep232350-21munin-run19:09:372
21467211213,83sleep20-21swapper/221:55:532
21467211213,83sleep20-21swapper/221:55:532
24736211113,84sleep124811-21cat23:51:001
30416211014,59sleep230500-21wc00:09:382
16557211029,22sleep20-21swapper/221:40:062
8794210613,54sleep28922-21unixbench_singl21:15:232
8794210613,54sleep28922-21unixbench_singl21:15:232
2039021068,61sleep220411-21apt-get21:54:352
9983210419,67sleep09588-21/usr/sbin/munin21:20:010
25802210413,78sleep325872-21/usr/sbin/munin20:29:533
23249210315,69sleep122692-21/usr/sbin/munin20:20:061
19762210312,78sleep219817-21cat23:34:572
31493210112,72sleep131572-21unixbench_singl20:45:301
2662210012,49sleep22703-21ntp_kernel_err19:15:022
12494210017,60sleep012610-21timerwakeupswit19:45:080
3179029911,49sleep20-21swapper/222:29:532
409929821,53sleep14216-21cat21:00:441
409929821,53sleep14216-21cat21:00:441
1194929813,69sleep111993-21cat23:09:571
180282978,52sleep20-21swapper/220:04:482
180282978,52sleep20-21swapper/220:04:482
2995629619,61sleep1228-21systemd-journal22:24:351
1250229610,70sleep012658-21date21:29:380
1065529610,47sleep20-21swapper/223:05:012
3107129524,49sleep031184-21fschecks_time19:04:550
273462957,47sleep20-21swapper/220:34:532
273462957,47sleep20-21swapper/220:34:532
2658329518,57sleep326709-21wc23:55:093
2318829517,59sleep322581-21/usr/sbin/munin23:45:083
2318829517,59sleep322581-21/usr/sbin/munin23:45:083
2074329514,63sleep320765-21modprobe23:39:353
195962957,48sleep20-21swapper/220:09:492
2859529418,59sleep028614-21latency_hist20:39:350
2859529418,59sleep028614-21latency_hist20:39:350
2712729434,17sleep20-21swapper/222:14:502
137229439,39sleep20-21swapper/222:35:052
302272939,66sleep130339-21gpgv20:44:371
1641029317,18sleep20-21swapper/223:24:522
1400029317,49sleep114147-21grep23:15:101
2162329211,43sleep20-21swapper/220:15:062
263529133,44sleep22707-21date00:24:362
1299729124,46sleep313029-21munin-plugin-st23:14:353
221882906,76sleep00-21swapper/023:42:550
101429010,68sleep21079-21sendmail-msp00:19:352
2425728910,17sleep10-21swapper/120:24:501
2425728910,17sleep10-21swapper/120:24:501
1217628910,41sleep20-21swapper/223:10:022
774328840,24sleep20-21swapper/222:55:082
1592128811,63sleep00-21swapper/019:56:060
1091028817,53sleep311055-21/usr/sbin/munin23:05:073
1184228711,60sleep011947-21fschecks_time23:09:560
285628613,60sleep12898-21tr22:40:011
285628613,60sleep12898-21tr22:40:011
395228517,55sleep03963-21awk19:19:520
317862859,23sleep30-21swapper/322:29:503
297628513,54sleep13128-21awk19:15:091
717928427,27sleep30-21swapper/322:54:573
1124128412,53sleep111246-21/usr/sbin/munin21:24:551
801228317,43sleep08031-21apt-get21:14:480
801228317,43sleep08031-21apt-get21:14:480
2718328322,37sleep01-21systemd23:59:350
2534328318,43sleep125308-21latency_hist22:09:371
2091728325,35sleep020911-21apt-get21:54:500
2091728325,35sleep020911-21apt-get21:54:500
1432728323,36sleep10-21swapper/121:34:561
605228218,39sleep36141-21fschecks_time00:34:553
2947928211,15sleep00-21swapper/000:05:000
187528211,57sleep01936-21latency20:55:220
54332819,16sleep20-21swapper/221:05:132
3223128132,32sleep132266-21munin-run19:09:361
2292728132,35sleep20-21swapper/222:00:012
911628026,32sleep10-21swapper/123:00:041
2931528012,53sleep329436-21timerandwakeup22:20:043
507927915,40sleep14915-21/usr/sbin/munin21:05:001
1844927913,50sleep318488-21ntpq20:05:023
1844927913,50sleep318488-21ntpq20:05:023
1706027825,35sleep317083-21apt-get21:44:353
375927717,30sleep00-21swapper/022:44:360
375927717,30sleep00-21swapper/022:44:360
1827927715,47sleep118406-21sed21:45:421
56072767,14sleep20-21swapper/222:49:512
2621627613,17sleep00-21swapper/020:30:020
2394727616,44sleep024101-21latency_hist23:49:370
2394727616,44sleep024101-21latency_hist23:49:370
205362764,61sleep00-21swapper/023:36:490
306602755,14sleep00-21swapper/022:25:020
173827521,34sleep31857-21munin-node19:14:383
72792748,15sleep20-21swapper/219:29:562
2082427424,27sleep10-21swapper/123:39:371
2033127418,30sleep00-21swapper/020:10:080
815927323,24sleep30-21swapper/321:14:593
815927323,24sleep30-21swapper/321:14:593
37892739,48sleep00-21swapper/000:25:110
3112227321,34sleep331186-21ps00:10:043
1889927316,20sleep00-21swapper/021:49:370
1185327313,27sleep10-21swapper/119:44:531
1039127321,33sleep30-21swapper/319:39:543
1039127321,33sleep30-21swapper/319:39:543
874027119,24sleep30-21swapper/319:34:533
323202716,13sleep20-21swapper/220:49:552
323202716,13sleep20-21swapper/220:49:552
2943827119,31sleep029576-21sed22:20:050
1709427120,22sleep00-21swapper/021:44:360
1365927123,34sleep20-21swapper/223:15:032
766927031,28sleep20-21swapper/221:13:372
665327019,26sleep10-21swapper/121:10:091
393527013,41sleep34014-21sendmail_mailtr21:00:423
393527013,41sleep34014-21sendmail_mailtr21:00:423
2688627015,21sleep10-21swapper/122:14:371
2462527011,41sleep324685-21/usr/sbin/munin23:50:573
2218127010,44sleep022182-21sshd20:16:460
146422707,14sleep30-21swapper/319:54:363
250142697,50sleep00-21swapper/022:05:560
1822126918,33sleep00-21swapper/023:29:590
1067826914,35sleep010743-21grep23:05:030
63126816,27sleep10-21swapper/122:34:361
3157826811,27sleep30-21swapper/322:29:373
1083626829,28sleep20-21swapper/221:24:112
1803826717,33sleep118126-21tune2fs23:29:561
1468326723,25sleep00-21swapper/019:54:370
1329726719,26sleep00-21swapper/021:30:070
180026622,25sleep00-21swapper/000:20:060
1389726612,21sleep00-21swapper/023:15:090
2867526519,26sleep328614-21latency_hist20:39:383
2867526519,26sleep328614-21latency_hist20:39:383
2795226521,19sleep10-21swapper/120:35:081
2795226521,19sleep10-21swapper/120:35:081
2271726521,27sleep322712-21apt-get20:19:523
1780226512,38sleep317812-21pluginstate20:04:373
1780226512,38sleep317812-21pluginstate20:04:373
1315426519,26sleep10-21swapper/121:30:031
625226422,22sleep00-21swapper/019:25:420
55732646,49sleep00-21swapper/000:33:180
55732646,49sleep00-21swapper/000:33:180
28282646,12sleep30-21swapper/319:15:073
2769926414,33sleep127769-21/usr/sbin/munin22:15:061
2611826413,28sleep10-21swapper/123:54:591
2091526418,28sleep320411-21apt-get21:54:493
2091526418,28sleep320411-21apt-get21:54:493
1153326420,22sleep30-21swapper/321:25:013
273342636,13sleep00-21swapper/022:14:570
2180526317,30sleep10-21swapper/123:40:091
2114026315,33sleep121160-21cpu20:14:531
1954026321,20sleep10-21swapper/121:50:021
174182636,12sleep00-21swapper/023:26:270
1643126313,33sleep116554-21timerwakeupswit21:40:051
152632638,12sleep30-21swapper/323:20:003
1513426318,26sleep015138-21/usr/sbin/munin23:19:570
132042638,40sleep313292-21grep21:30:053
3195226216,30sleep00-21swapper/000:14:370
150102628,39sleep115017-21diskstats19:54:541
104526217,28sleep31-21systemd20:54:363
104526217,28sleep31-21systemd20:54:363
10032625,13sleep10-21swapper/100:19:261
96812615,12sleep00-21swapper/019:35:350
733026118,18sleep10-21swapper/100:39:371
728826121,21sleep00-21swapper/000:39:360
2751326114,29sleep327508-21apt-get23:59:503
2171526121,18sleep10-21swapper/121:55:561
2171526121,18sleep10-21swapper/121:55:561
1925326114,22sleep30-21swapper/323:34:383
1495126124,20sleep10-21swapper/123:19:541
110226114,25sleep00-21swapper/020:54:370
110226114,25sleep00-21swapper/020:54:370
107912619,14sleep10-21swapper/119:40:041
107912619,14sleep10-21swapper/119:40:041
81402607,12sleep10-21swapper/119:30:351
728426019,23sleep30-21swapper/319:29:573
444126011,24sleep30-21swapper/319:20:043
1470226012,33sleep114846-21cat19:54:381
1240526022,20sleep30-21swapper/319:45:073
892525919,20sleep00-21swapper/022:59:590
774225917,26sleep37769-21munin-run21:14:353
1863625911,22sleep00-21swapper/020:05:070
1863625911,22sleep00-21swapper/020:05:070
1090425912,33sleep310915-21munin-run21:24:363
493925813,27sleep04933-21apt-get21:04:510
2450125811,26sleep324541-21ps22:05:023
186125815,24sleep00-21swapper/019:14:390
173325815,26sleep11774-21grep00:20:031
734025714,18sleep00-21swapper/019:29:580
551425713,25sleep00-21swapper/022:49:450
530325712,19sleep00-21swapper/022:49:370
3230625711,19sleep00-21swapper/000:14:570
1424125714,25sleep014235-21apt-get21:34:510
683925613,23sleep30-21swapper/322:54:373
479025613,25sleep30-21swapper/300:30:023
479025613,25sleep30-21swapper/300:30:023
3239125617,21sleep30-21swapper/319:09:383
3099625613,23sleep00-21swapper/020:45:200
2306825620,20sleep10-21swapper/123:45:061
2306825620,20sleep10-21swapper/123:45:061
96722555,37sleep10-21swapper/119:35:281
961525514,24sleep19609-21apt-get21:19:501
3273425512,17sleep10-21swapper/120:50:061
3273425512,17sleep10-21swapper/120:50:061
296825517,21sleep10-21swapper/100:24:551
2919625518,17sleep30-21swapper/300:04:553
2570825511,23sleep00-21swapper/023:54:380
142225517,16sleep30-21swapper/300:19:573
106732557,13sleep00-21swapper/019:39:590
106732557,13sleep00-21swapper/019:39:590
460025418,21sleep14675-21find22:45:061
42725418,16sleep30-21swapper/300:15:083
2683425415,22sleep326857-21munin-run22:14:353
2619425416,18sleep00-21swapper/022:10:080
2089625417,20sleep10-21swapper/120:14:371
1959725412,24sleep319592-21apt-get20:09:503
1431425413,25sleep314320-21df_inode21:34:543
88525312,24sleep0879-21apt-get22:34:510
733925313,24sleep17355-21irqstats22:54:591
633325314,15sleep10-21swapper/119:25:431
446425313,19sleep30-21swapper/322:45:043
442225312,24sleep14416-21apt-get00:29:521
442225312,24sleep14416-21apt-get00:29:521
2580025312,24sleep125794-21apt-get20:29:511
2248925313,24sleep122484-21apt-get21:59:471
221122536,11sleep30-21swapper/320:16:003
1886625315,23sleep318897-21munin-run21:49:363
94082529,20sleep10-21swapper/121:19:381
94082529,20sleep10-21swapper/121:19:381
2748725210,18sleep00-21swapper/023:59:470
246025211,24sleep32455-21apt-get22:39:513
246025211,24sleep32455-21apt-get22:39:513
2425425212,24sleep324249-21apt-get20:24:483
2425425212,24sleep324249-21apt-get20:24:483
2270425214,21sleep022712-21apt-get20:19:490
2257525215,15sleep30-21swapper/321:59:543
1959925218,18sleep30-21swapper/321:50:043
1805325213,23sleep118047-21apt-get20:04:521
1805325213,23sleep118047-21apt-get20:04:521
1771025215,15sleep30-21swapper/323:29:373
1640925213,24sleep10-21swapper/123:24:511
1561425213,17sleep30-21swapper/319:55:083
154925213,23sleep31567-21/usr/sbin/munin20:54:483
132425214,23sleep31340-21ntp_states22:35:023
632625114,18sleep30-21swapper/321:09:383
298482516,31sleep30-21swapper/320:40:273
1691825114,18sleep00-21swapper/020:00:020
1691825114,18sleep00-21swapper/020:00:020
295325013,22sleep32959-21diskstats00:24:533
247032509,16sleep00-21swapper/020:25:010
247032509,16sleep00-21swapper/020:25:010
214942509,13sleep30-21swapper/323:40:033
20912496,32sleep10-21swapper/122:38:561
162824912,22sleep11649-21munin-run19:14:361
728024713,15sleep00-21swapper/022:54:580
2868324711,22sleep128700-21apt-get00:03:231
1061824711,21sleep110654-21ntp_states23:05:001
314102465,32sleep10-21swapper/122:28:261
907124511,20sleep39107-21ntp_states23:00:023
3186924410,17sleep00-21swapper/022:29:550
2911024411,16sleep10-21swapper/100:04:491
1336724310,17sleep30-21swapper/323:14:563
291222419,13sleep00-21swapper/020:39:550
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional