You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 13:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Nov 23, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266742178412,20sleep00-21swapper/020:58:580
266742178412,20sleep00-21swapper/020:58:580
13196265181,517sleep013218-21idleruntime-cro23:43:350
5199250111,470sleep25192-21apt-get21:33:532
13402245825,414sleep213455-21/usr/sbin/munin21:58:562
13402245825,414sleep213455-21/usr/sbin/munin21:58:562
20708241518,377sleep220852-21/usr/sbin/munin20:39:092
27639241014,384sleep124342-21kworker/1:021:02:181
27639241014,384sleep124342-21kworker/1:021:02:181
1665234217,306sleep21688-21/usr/sbin/munin19:39:002
18631229710,229sleep018691-21cat20:33:580
9567227610,246sleep09632-21sendmail_mailst20:04:060
14742223661,128sleep014752-21find22:03:470
14742223661,128sleep014752-21find22:03:470
3329223521,199sleep23349-21idleruntime-cro21:28:362
17251222739,177sleep0362-21snmpd22:09:280
24086222316,90sleep00-21swapper/019:08:420
24086222316,90sleep00-21swapper/019:08:420
22649222042,164sleep022739-21cut22:28:580
24430220847,111sleep00-21swapper/020:53:370
24430220847,111sleep00-21swapper/020:53:370
284122075,32sleep00-21swapper/023:08:590
23748220770,116sleep023809-21latency_hist19:08:380
25426220258,28sleep00-21swapper/020:54:300
2963421959,142sleep029680-21latency_hist21:08:390
2963421959,142sleep029680-21latency_hist21:08:390
1228921930,6sleep00-21swapper/020:13:540
1228921930,6sleep00-21swapper/020:13:540
11073219316,159sleep111210-21unixbench_multi23:34:111
11073219316,159sleep111210-21unixbench_multi23:34:111
23129219119,112sleep00-21swapper/000:13:510
1603721868,164sleep016039-21sshd23:50:170
1603721868,164sleep016039-21sshd23:50:170
2231621817,161sleep022361-21date22:28:380
29258217810,107sleep00-21swapper/022:49:090
2019321787,114sleep00-21swapper/000:03:580
16055217731,47sleep016204-21unixbench_singl20:24:090
31996217010,93sleep00-21swapper/022:58:530
18208216928,127sleep018303-21latency_hist20:33:380
4570216813,100sleep00-21swapper/019:49:000
27324216840,75sleep00-21swapper/019:18:390
24773216619,47sleep024863-21fschecks_time00:18:560
22186216510,112sleep022319-21ls00:09:070
7545216224,123sleep07625-21latency23:24:130
7545216224,123sleep07625-21latency23:24:130
16924214511,24sleep00-21swapper/023:54:130
22816214422,14sleep00-21swapper/020:48:360
10307214419,74sleep00-21swapper/021:49:050
4160214332,90sleep00-21swapper/019:48:390
15163214310,120sleep215199-21ntpq22:04:012
15163214310,120sleep215199-21ntpq22:04:012
1413213211,20sleep30-21swapper/323:04:013
26457213041,43sleep30-21swapper/320:58:493
26457213041,43sleep30-21swapper/320:58:493
32507212985,22sleep00-21swapper/021:16:260
28293212522,81sleep128272-21apt-get21:04:001
28293212522,81sleep128272-21apt-get21:04:001
19120212548,65sleep019239-21timerwakeupswit23:59:460
19120212548,65sleep019239-21timerwakeupswit23:59:460
2067212422,17sleep00-21swapper/021:23:490
663521226,29sleep00-21swapper/023:19:210
663521226,29sleep00-21swapper/023:19:210
21247212225,79sleep321285-21ntpq18:59:043
24536212143,62sleep324579-21/usr/sbin/munin22:34:013
22362212014,90sleep122512-21latency_hist22:28:391
8806211822,81sleep00-21swapper/023:28:490
2795821178,89sleep00-21swapper/000:28:580
26910211521,72sleep127043-21unixbench_multi00:24:101
13193211510,44sleep00-21swapper/021:58:460
13193211510,44sleep00-21swapper/021:58:460
8662211424,46sleep00-21swapper/021:44:000
8662211424,46sleep00-21swapper/021:44:000
2860321136,11sleep00-21swapper/019:21:180
995211213,77sleep31055-21/usr/sbin/munin21:19:123
25951211216,81sleep325967-21kernelversion22:38:583
10732211223,65sleep00-21swapper/020:08:590
20179211040,56sleep020313-21head22:19:090
18613211024,59sleep118761-21grep22:14:091
14118211010,16sleep00-21swapper/023:44:200
4078210910,84sleep00-21swapper/021:29:060
6048210745,20sleep00-21swapper/019:53:560
2610210713,75sleep00-21swapper/019:43:390
10654210744,51sleep010668-21grep23:34:020
10654210744,51sleep010668-21grep23:34:020
23081210516,70sleep10-21swapper/120:48:491
23081210516,70sleep10-21swapper/120:48:491
20850210521,42sleep20-21swapper/218:58:512
14143210430,63sleep014179-21ntpq20:19:040
14143210430,63sleep014179-21ntpq20:19:040
2178210023,37sleep00-21swapper/023:07:170
461929920,13sleep00-21swapper/023:14:040
120929822,57sleep21366-21tail21:19:152
116282988,14sleep00-21swapper/021:53:460
116282988,14sleep00-21swapper/021:53:460
2723329719,44sleep00-21swapper/000:25:220
75352969,71sleep00-21swapper/019:58:530
571229611,71sleep15714-21grep23:18:531
571229611,71sleep15714-21grep23:18:531
276562968,42sleep00-21swapper/022:44:040
1437829613,67sleep214454-21timerandwakeup20:19:102
1437829613,67sleep214454-21timerandwakeup20:19:102
1236329622,52sleep212489-21wc21:54:412
1236329622,52sleep212489-21wc21:54:412
69552957,13sleep30-21swapper/321:38:593
69552957,13sleep30-21swapper/321:38:593
322792959,31sleep00-21swapper/019:33:580
2163329514,60sleep121720-21fschecks_time20:43:571
1013929517,63sleep310155-21grep23:33:373
57042947,15sleep01-21systemd21:34:070
3033229440,16sleep030364-21munin-run19:28:360
3033229440,16sleep030364-21munin-run19:28:360
2481929421,50sleep224968-21tail22:34:092
1284829416,60sleep212970-21timerwakeupswit20:14:102
1284829416,60sleep212970-21timerwakeupswit20:14:102
719029310,33sleep00-21swapper/021:39:040
719029310,33sleep00-21swapper/021:39:040
3010629315,53sleep10-21swapper/121:09:001
2442829330,36sleep10-21swapper/100:18:371
6112927,72sleep0691-21tune2fs21:19:010
3077829030,17sleep00-21swapper/019:29:050
2244229015,61sleep122491-21/usr/sbin/munin19:03:541
325628910,50sleep30-21swapper/319:44:023
308362896,47sleep00-21swapper/021:10:210
2777528913,33sleep20-21swapper/200:28:522
2797128816,20sleep20-21swapper/222:44:102
2459428710,64sleep124600-21uname22:34:041
2436728726,41sleep324390-21munin-run20:53:363
2436728726,41sleep324390-21munin-run20:53:363
304332855,29sleep00-21swapper/022:53:530
304332855,29sleep00-21swapper/022:53:530
2217528423,43sleep122231-21cyclictest19:03:381
1902128418,52sleep119118-21switchtime20:34:071
3188028310,59sleep331949-21/usr/sbin/munin19:33:363
209012835,30sleep00-21swapper/020:39:110
72628215,50sleep1728-21cron23:03:351
2749828215,51sleep227515-21/usr/sbin/munin19:18:492
268722827,29sleep00-21swapper/022:42:390
112792820,6sleep30-21swapper/320:09:093
626728118,50sleep36339-21sort23:19:113
626728118,50sleep36339-21sort23:19:113
286028110,14sleep10-21swapper/121:24:071
2606028115,29sleep20-21swapper/219:13:572
250982815,28sleep00-21swapper/022:34:470
745828019,44sleep27494-21/usr/sbin/munin23:24:112
745828019,44sleep27494-21/usr/sbin/munin23:24:112
2947128019,38sleep329618-21grep22:49:123
188552805,28sleep00-21swapper/022:14:410
2983227918,25sleep20-21swapper/221:08:532
2605427918,26sleep10-21swapper/122:39:001
2006927918,35sleep220126-21fschecks_count00:03:562
1767427922,43sleep217742-21sed22:13:382
1629727925,34sleep31-21systemd23:53:353
1629727925,34sleep31-21systemd23:53:353
1079127813,49sleep310834-21ntp_offset23:34:053
1079127813,49sleep310834-21ntp_offset23:34:053
488027721,33sleep14910-21munin-run21:33:361
2524027711,47sleep30-21swapper/319:09:093
2524027711,47sleep30-21swapper/319:09:093
1827327713,48sleep218313-21grep22:14:022
2624427618,32sleep10-21swapper/119:14:001
183127621,21sleep10-21swapper/121:23:381
1805827629,24sleep10-21swapper/123:58:381
179032768,48sleep30-21swapper/322:13:513
166327619,38sleep11796-21wc23:04:071
1212327614,26sleep20-21swapper/223:39:092
1193527614,46sleep311931-21apt-get23:38:503
891727323,26sleep10-21swapper/123:29:071
2839827315,40sleep328517-21timerwakeupswit00:29:093
2166827317,26sleep30-21swapper/322:24:083
485427217,41sleep3225-21systemd-journal21:33:173
3231627214,41sleep332359-21/usr/sbin/munin22:59:013
1551927213,38sleep115561-21wc23:49:171
1551927213,38sleep115561-21wc23:49:171
2036727114,44sleep320410-21tr20:39:023
1842127119,32sleep318417-21apt-get23:58:483
1842127119,32sleep318417-21apt-get23:58:483
1666927114,25sleep10-21swapper/120:28:381
2541627014,29sleep30-21swapper/320:54:293
2381927015,32sleep223844-21munin-run22:33:362
805726930,26sleep18121-21cat21:43:371
805726930,26sleep18121-21cat21:43:371
47032699,12sleep20-21swapper/223:14:062
321526921,27sleep20-21swapper/219:44:012
3117726915,40sleep231219-21latency_hist21:13:382
2472926918,28sleep10-21swapper/100:18:531
196626923,29sleep12096-21timerwakeupswit19:39:071
573726817,25sleep20-21swapper/219:53:382
2730426814,39sleep327315-21df_inode22:43:553
265626825,17sleep10-21swapper/123:08:561
111626823,20sleep20-21swapper/223:03:562
2944126712,12sleep20-21swapper/219:24:022
2944126712,12sleep20-21swapper/219:24:022
2025126716,24sleep10-21swapper/100:03:591
1839826715,33sleep117930-21apt-get23:58:461
1839826715,33sleep117930-21apt-get23:58:461
1707726717,33sleep117127-21if_eth020:28:591
1120926712,35sleep210612-21/usr/sbin/munin20:09:092
886326619,18sleep20-21swapper/220:03:402
1502826613,40sleep315052-21idleruntime-cro20:23:353
1502826613,40sleep315052-21idleruntime-cro20:23:353
1006826621,23sleep20-21swapper/223:33:362
634526533,17sleep30-21swapper/319:54:043
399326526,22sleep20-21swapper/221:29:042
2785126517,22sleep10-21swapper/122:44:091
95342645,10sleep30-21swapper/321:48:193
95342645,10sleep30-21swapper/321:48:193
2253626414,28sleep30-21swapper/322:28:493
2076326426,19sleep10-21swapper/122:23:381
1593126418,27sleep30-21swapper/320:24:083
1228626419,25sleep10-21swapper/121:54:401
1228626419,25sleep10-21swapper/121:54:401
869126312,32sleep1369-21sshd20:02:171
2805926319,24sleep10-21swapper/119:19:091
2301526317,22sleep30-21swapper/319:04:083
2012826318,19sleep30-21swapper/300:03:573
1748126315,24sleep30-21swapper/320:29:093
3082926213,22sleep30-21swapper/322:54:043
3082926213,22sleep30-21swapper/322:54:043
2952826219,27sleep129534-21ntp_states19:24:051
2952826219,27sleep129534-21ntp_states19:24:051
2357326215,22sleep20-21swapper/220:49:072
2357326215,22sleep20-21swapper/220:49:072
2201726212,32sleep222103-21mailstats00:09:052
2060526215,21sleep10-21swapper/120:39:081
1722626217,25sleep30-21swapper/323:54:203
147272626,11sleep30-21swapper/323:48:183
675026117,25sleep26817-21df21:38:542
675026117,25sleep26817-21df21:38:542
410826119,25sleep14141-21awk19:48:361
247622618,12sleep30-21swapper/300:18:553
2349726117,26sleep323508-21netstat00:14:033
1765426113,27sleep10-21swapper/122:13:371
1526326117,22sleep20-21swapper/223:49:102
1526326117,22sleep20-21swapper/223:49:102
1508126117,23sleep10-21swapper/120:23:371
1508126117,23sleep10-21swapper/120:23:371
546826016,21sleep10-21swapper/121:34:001
3273226013,22sleep20-21swapper/221:18:392
3220126016,26sleep132194-21apt-get19:33:541
2604126018,22sleep30-21swapper/319:13:543
2249226017,18sleep20-21swapper/219:03:552
2071126016,27sleep320740-21munin-run22:23:363
971925913,23sleep20-21swapper/221:48:392
971925913,23sleep20-21swapper/221:48:392
908525917,23sleep30-21swapper/320:03:533
3270625916,21sleep30-21swapper/319:34:093
2710425915,25sleep227236-21timerwakeupswit20:59:082
2710425915,25sleep227236-21timerwakeupswit20:59:082
1854425917,22sleep30-21swapper/320:33:573
1082625910,34sleep110886-21if_eth020:09:001
3120625813,31sleep231299-21cat19:29:152
2286725823,21sleep20-21swapper/220:48:382
1398025819,23sleep20-21swapper/223:44:182
2525325719,17sleep20-21swapper/200:19:062
2516825715,23sleep125225-21irqstats20:54:231
249525714,25sleep32518-21ntp_states21:24:013
1713925714,17sleep20-21swapper/223:54:192
1488425719,16sleep10-21swapper/123:48:411
1297525716,18sleep30-21swapper/321:58:373
1297525716,18sleep30-21swapper/321:58:373
1245925715,19sleep10-21swapper/120:14:001
1245925715,19sleep10-21swapper/120:14:001
73025618,19sleep20-21swapper/223:03:362
566525621,20sleep30-21swapper/321:34:063
3168225618,15sleep10-21swapper/122:58:381
3168225618,15sleep10-21swapper/122:58:381
2565525617,23sleep225683-21munin-run19:13:362
2565525617,23sleep225683-21munin-run19:13:362
2522625615,21sleep20-21swapper/220:54:242
2465525617,21sleep10-21swapper/119:08:591
2465525617,21sleep10-21swapper/119:08:591
984525512,25sleep29841-21apt-get21:48:512
725725513,25sleep17250-21apt-get23:23:501
725725513,25sleep17250-21apt-get23:23:501
571525514,15sleep20-21swapper/223:18:542
571525514,15sleep20-21swapper/223:18:542
413025513,26sleep34148-21apt23:13:463
3036325514,25sleep230356-21apt-get22:53:512
3036325514,25sleep230356-21apt-get22:53:512
2801225512,27sleep128045-21irqstats00:28:591
1967325512,23sleep20-21swapper/222:18:592
1824825518,16sleep20-21swapper/223:58:392
1824825518,16sleep20-21swapper/223:58:392
168925514,25sleep31726-21ntp_kernel_pll_19:39:013
1038925513,23sleep210457-21df23:33:522
1038925513,23sleep210457-21df23:33:522
1031125522,18sleep10-21swapper/121:49:061
3151025412,23sleep131602-21perl21:13:571
2752525415,18sleep30-21swapper/319:18:543
1584725410,35sleep30-21swapper/322:06:203
1584725410,35sleep30-21swapper/322:06:203
1116325412,22sleep311181-21munin-run21:53:363
75322536,33sleep30-21swapper/319:58:523
597725312,24sleep15971-21apt-get19:53:531
312225312,24sleep3362-21snmpd23:09:063
2886425311,16sleep10-21swapper/122:48:571
2313125312,24sleep223125-21apt-get00:13:532
2157625318,17sleep20-21swapper/222:24:072
1911925312,17sleep20-21swapper/220:34:082
1453125319,18sleep30-21swapper/322:03:383
1453125319,18sleep30-21swapper/322:03:383
3072425213,24sleep330737-21/usr/sbin/munin21:09:113
2288025215,16sleep30-21swapper/320:48:393
1447825213,23sleep114507-21munin-run22:03:361
1447825213,23sleep114507-21munin-run22:03:361
1322225217,20sleep10-21swapper/123:43:361
971025119,16sleep20-21swapper/220:04:082
908325111,22sleep19078-21apt-get20:03:511
725125112,16sleep30-21swapper/323:23:483
725125112,16sleep30-21swapper/323:23:483
606325115,16sleep20-21swapper/219:53:592
3196925119,16sleep10-21swapper/119:33:381
2630625112,24sleep226311-21diskstats00:23:552
1488825113,19sleep10-21swapper/122:03:561
1488825113,19sleep10-21swapper/122:03:561
879625013,21sleep1284-21dbus-daemon21:44:061
879625013,21sleep1284-21dbus-daemon21:44:061
448625016,19sleep30-21swapper/319:48:593
439925012,20sleep10-21swapper/119:48:531
415425011,22sleep14150-21apt-get23:13:481
318482509,16sleep30-21swapper/321:14:033
2824925012,23sleep227761-21apt-get21:03:442
268952506,32sleep20-21swapper/222:42:572
2211125018,16sleep00-21swapper/019:03:370
1794325014,22sleep217946-21sshd20:31:182
2881824912,19sleep30-21swapper/321:05:033
2881824912,19sleep30-21swapper/321:05:033
2197324915,17sleep30-21swapper/300:09:023
1232024914,21sleep30-21swapper/320:13:583
1232024914,21sleep30-21swapper/320:13:583
2907124812,18sleep30-21swapper/319:23:513
2907124812,18sleep30-21swapper/319:23:513
158622487,16sleep20-21swapper/220:24:072
25732479,14sleep20-21swapper/223:08:532
2887224610,19sleep228876-21diskstats22:48:582
2663124613,19sleep30-21swapper/300:24:043
2163324511,13sleep10-21swapper/100:08:561
323232448,15sleep20-21swapper/219:34:002
2274024210,12sleep10-21swapper/122:28:591
163952419,16sleep30-21swapper/322:08:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional