You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-23 - 06:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Thu Oct 23, 2025 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1179899260,24cyclictest313-21systemd-journal00:29:592
1179499240,24cyclictest0-21swapper/023:14:310
11799992320,2cyclictest612-21gdbus20:10:003
1179999230,22cyclictest23101-21cron20:40:003
1179999230,0cyclictest0-21swapper/319:39:313
1179899230,22cyclictest6904-21apt-get00:34:352
11796992321,1cyclictest0-21swapper/120:20:011
11796992320,2cyclictest23791-21idleruntime-cro00:00:001
11796992320,2cyclictest12009-21turbostat22:30:001
1179699230,2cyclictest24864-21apt-get00:04:331
1179699230,22cyclictest11114-21munin-run00:40:001
1179499230,2cyclictest313-21systemd-journal00:40:000
11799992220,1cyclictest23831-21perf19:35:003
11799992220,1cyclictest11283-21perf23:35:003
1179999220,21cyclictest32023-21perf22:04:593
11798992222,0cyclictest0-21swapper/221:29:322
11798992221,1cyclictest609-21polkitd22:45:002
11798992220,1cyclictest10215-21apt-get22:29:302
1179899220,22cyclictest29251-21apt-get19:49:292
1179899220,21cyclictest14161-21cat19:14:592
1179899220,21cyclictest0-21swapper/223:59:312
1179899220,1cyclictest766-21snmpd00:04:332
1179899220,1cyclictest15349-21apt-get21:34:302
1179899220,1cyclictest12351-21perf21:25:002
1179899220,1cyclictest0-21swapper/220:20:012
11796992222,0cyclictest5004-21apt-get22:19:311
11796992222,0cyclictest0-21swapper/120:29:311
11796992220,1cyclictest9921-21idleruntime-cro21:20:001
11796992220,1cyclictest1022-21turbostat19:54:591
11796992220,1cyclictest0-21swapper/122:40:001
1179699220,21cyclictest31050-21turbostat19:49:591
1179699220,21cyclictest28531-21apt-get20:54:331
1179699220,21cyclictest22286-21apt-get22:54:311
1179699220,21cyclictest0-21swapper/120:34:311
1179699220,1cyclictest30928-21apt-get20:59:331
1179699220,1cyclictest29705-21apt-get00:14:321
1179699220,1cyclictest29058-21perf00:10:001
1179699220,1cyclictest19621-21sendmail-msp21:39:591
1179699220,1cyclictest12371-21apt-get19:14:291
1179499220,21cyclictest26472-21apt-get23:00:000
1179499220,21cyclictest19564-21apt-get23:54:320
1179499220,21cyclictest19296-21munin-run22:45:010
1179499220,21cyclictest11270-21apt-get23:35:010
1179499220,21cyclictest0-21swapper/021:29:300
1179499220,21cyclictest0-21swapper/021:14:310
1179499220,21cyclictest0-21swapper/020:39:310
1179499220,21cyclictest0-21swapper/000:34:350
11799992120,1cyclictest0-21swapper/322:20:023
11799992120,1cyclictest0-21swapper/319:14:303
11799992120,0cyclictest0-21swapper/322:54:303
11799992120,0cyclictest0-21swapper/321:09:303
11799992119,1cyclictest28621-21cron19:45:003
11799992119,1cyclictest12023-21munin-run22:30:013
11799992119,1cyclictest0-21swapper/323:30:013
1179999210,21cyclictest0-21swapper/323:00:013
1179999210,21cyclictest0-21swapper/320:49:323
1179999210,20cyclictest0-21swapper/321:39:323
1179999210,1cyclictest29603-21perf21:59:593
1179999210,1cyclictest0-21swapper/300:15:013
1179999210,0cyclictest0-21swapper/321:54:273
11798992121,0cyclictest7095-21apt-get23:29:302
11798992120,0cyclictest19564-21apt-get23:54:322
11798992120,0cyclictest13671-21apt-get20:24:302
1179899210,21cyclictest26472-21apt-get23:00:022
1179899210,20cyclictest0-21swapper/223:34:312
1179899210,20cyclictest0-21swapper/221:54:272
1179899210,20cyclictest0-21swapper/220:10:002
1179899210,0cyclictest0-21swapper/221:39:322
1179899210,0cyclictest0-21swapper/219:54:292
11796992121,0cyclictest0-21swapper/122:49:321
11796992120,1cyclictest20614-21apt-get21:44:321
11796992119,1cyclictest1683-21munin-run23:15:001
1179699210,20cyclictest0-21swapper/122:34:301
1179699210,20cyclictest0-21swapper/121:09:311
1179699210,20cyclictest0-21swapper/100:19:311
1179699210,0cyclictest0-21swapper/123:49:311
11794992120,1cyclictest0-21swapper/023:29:310
1179499210,21cyclictest0-21swapper/020:14:330
1179499210,20cyclictest4699-21apt-get23:24:320
1179499210,20cyclictest0-21swapper/021:19:300
1179499210,0cyclictest0-21swapper/023:04:320
1179499210,0cyclictest0-21swapper/019:49:300
1179499210,0cyclictest0-21swapper/019:34:300
1179499210,0cyclictest0-21swapper/000:29:330
1179499210,0cyclictest0-21swapper/000:00:020
11799992020,0cyclictest0-21swapper/323:59:313
11799992020,0cyclictest0-21swapper/323:24:323
11799992020,0cyclictest0-21swapper/322:59:343
11799992020,0cyclictest0-21swapper/300:19:313
11799992019,0cyclictest23016-21apt-get21:49:343
1179999200,19cyclictest0-21swapper/320:59:323
1179999200,0cyclictest0-21swapper/322:09:333
1179999200,0cyclictest0-21swapper/320:54:333
1179999200,0cyclictest0-21swapper/319:19:293
1179999200,0cyclictest0-21swapper/300:29:323
11798992020,0cyclictest27259-21apt-get00:09:322
11798992020,0cyclictest24436-21apt-get19:39:312
11798992020,0cyclictest14771-21apt-get19:19:292
11798992020,0cyclictest0-21swapper/222:14:282
11798992020,0cyclictest0-21swapper/220:54:322
1179899200,20cyclictest0-21swapper/222:24:342
1179899200,20cyclictest0-21swapper/221:09:312
1179899200,0cyclictest0-21swapper/223:35:002
11796992020,0cyclictest26856-21apt-get19:44:311
11796992020,0cyclictest0-21swapper/123:54:321
11796992020,0cyclictest0-21swapper/123:39:361
11796992020,0cyclictest0-21swapper/122:24:341
11796992020,0cyclictest0-21swapper/120:39:321
11796992020,0cyclictest0-21swapper/119:59:291
11796992020,0cyclictest0-21swapper/119:19:291
11796992019,1cyclictest19639-21apt-get19:29:311
11796992019,0cyclictest30225-21apt-get22:04:321
11796992017,2cyclictest10630-21cron20:15:011
1179699200,0cyclictest0-21swapper/123:29:301
1179699200,0cyclictest0-21swapper/121:54:281
1179699200,0cyclictest0-21swapper/120:24:301
11794992020,0cyclictest1630-21apt-get19:59:300
11794992020,0cyclictest0-21swapper/023:49:310
11794992020,0cyclictest0-21swapper/022:24:330
11794992020,0cyclictest0-21swapper/020:09:320
11794992020,0cyclictest0-21swapper/019:39:300
11794992020,0cyclictest0-21swapper/000:14:330
11794992018,1cyclictest14423-21munin-run22:35:010
11794992017,2cyclictest1-21systemd21:55:010
1179499200,0cyclictest0-21swapper/021:04:310
1179499200,0cyclictest0-21swapper/020:19:330
11799991919,0cyclictest17164-21apt-get23:49:303
11799991919,0cyclictest0-21swapper/323:44:303
11799991919,0cyclictest0-21swapper/323:19:323
11799991919,0cyclictest0-21swapper/322:49:323
11799991919,0cyclictest0-21swapper/322:44:333
11799991919,0cyclictest0-21swapper/322:19:303
11799991919,0cyclictest0-21swapper/321:04:303
11799991919,0cyclictest0-21swapper/300:24:323
11799991917,1cyclictest0-21swapper/321:40:023
1179999190,18cyclictest16520-21apt-get20:29:323
11798991919,0cyclictest1630-21apt-get19:59:292
11798991919,0cyclictest0-21swapper/223:24:322
11798991919,0cyclictest0-21swapper/222:19:312
11798991919,0cyclictest0-21swapper/221:49:332
11798991919,0cyclictest0-21swapper/221:44:302
11798991919,0cyclictest0-21swapper/221:04:312
11798991919,0cyclictest0-21swapper/220:29:312
11798991919,0cyclictest0-21swapper/219:44:312
11798991919,0cyclictest0-21swapper/200:19:302
11798991916,2cyclictest18939-21awk23:50:002
1179899190,17cyclictest313-21systemd-journal19:30:012
11796991919,0cyclictest0-21swapper/123:19:321
11796991919,0cyclictest0-21swapper/122:09:331
11796991918,1cyclictest7520-21perf21:15:001
11796991918,1cyclictest0-21swapper/100:34:351
11796991917,1cyclictest0-21swapper/120:44:301
11794991919,0cyclictest0-21swapper/022:14:270
11794991919,0cyclictest0-21swapper/021:44:320
11794991919,0cyclictest0-21swapper/020:44:310
11794991919,0cyclictest0-21swapper/020:29:310
11794991919,0cyclictest0-21swapper/019:24:320
11794991918,1cyclictest0-21swapper/019:44:300
11794991918,1cyclictest0-21swapper/000:24:330
11794991918,0cyclictest0-21swapper/021:49:340
1179499190,18cyclictest15316-21/usr/sbin/munin21:34:320
11799991818,0cyclictest0-21swapper/323:54:313
11799991818,0cyclictest0-21swapper/322:14:283
11799991818,0cyclictest0-21swapper/319:24:313
11799991818,0cyclictest0-21swapper/300:34:353
11798991818,0cyclictest8127-21apt-get21:19:312
11798991814,3cyclictest31495-21apt00:15:002
11796991818,0cyclictest9497-21apt-get23:34:321
11796991818,0cyclictest2113-21apt-get00:24:341
11794991818,0cyclictest0-21swapper/023:34:320
11794991818,0cyclictest0-21swapper/022:39:310
11794991818,0cyclictest0-21swapper/021:39:330
11794991816,1cyclictest30314-21latency_hist20:55:010
11794991816,1cyclictest0-21swapper/022:49:320
1179499180,11cyclictest0-21swapper/022:09:340
1179499180,0cyclictest0-21swapper/022:54:310
11799991717,0cyclictest23734-21apt-get20:44:313
11799991717,0cyclictest0-21swapper/323:39:363
11799991717,0cyclictest0-21swapper/321:29:323
11799991717,0cyclictest0-21swapper/319:49:293
11799991716,0cyclictest0-21swapper/323:09:313
11798991717,0cyclictest29895-21apt-get23:09:322
11798991717,0cyclictest17245-21apt-get19:24:322
11798991717,0cyclictest0-21swapper/222:39:302
1179899170,16cyclictest0-21swapper/220:59:332
11796991717,0cyclictest2603-21apt-get22:14:281
11796991717,0cyclictest0-21swapper/123:00:021
11796991717,0cyclictest0-21swapper/122:44:331
11796991717,0cyclictest0-21swapper/121:29:311
11796991717,0cyclictest0-21swapper/121:24:311
11796991715,1cyclictest6494-21perf23:25:001
11794991717,0cyclictest0-21swapper/019:14:300
11794991716,0cyclictest0-21swapper/023:44:310
11794991715,1cyclictest0-21swapper/022:00:000
11799991616,0cyclictest0-21swapper/321:34:303
11799991616,0cyclictest0-21swapper/320:24:293
11799991615,1cyclictest8850-21apt-get20:14:323
11799991615,0cyclictest0-21swapper/321:24:323
11799991614,1cyclictest24645-21chrt00:02:073
1179999160,15cyclictest7820-21taskset21:15:593
1179999160,0cyclictest0-21swapper/321:14:303
11798991616,0cyclictest0-21swapper/221:59:372
11798991616,0cyclictest0-21swapper/220:44:292
11798991616,0cyclictest0-21swapper/220:34:312
11798991616,0cyclictest0-21swapper/220:04:322
11798991616,0cyclictest0-21swapper/200:24:322
11798991614,1cyclictest27759-21grep20:49:402
11798991614,1cyclictest10651-21perf20:15:002
11798991613,2cyclictest6765-21ntpq21:14:362
11796991616,0cyclictest0-21swapper/121:04:311
11796991616,0cyclictest0-21swapper/100:29:331
11796991615,1cyclictest0-21swapper/123:44:301
11796991614,1cyclictest17179-21cron21:34:591
1179499167,8cyclictest766-21snmpd19:54:340
11794991616,0cyclictest0-21swapper/023:19:310
11794991616,0cyclictest0-21swapper/022:04:320
11794991616,0cyclictest0-21swapper/020:59:330
11794991614,1cyclictest951-21grep00:19:380
11794991614,1cyclictest26133-21apt-get20:49:320
1179499160,2cyclictest30527-21tail23:09:350
1179499160,15cyclictest20761-21phc2sys-jitter19:29:370
1179499160,15cyclictest12316-21cron21:24:590
1179499160,15cyclictest0-21swapper/020:24:340
11799991515,0cyclictest32291-21apt-get23:14:323
11799991515,0cyclictest0-21swapper/322:34:323
11799991515,0cyclictest0-21swapper/300:39:303
11799991513,1cyclictest5900-21perf20:05:013
11799991513,1cyclictest20724-21munin-run20:35:003
1179999150,14cyclictest27683-21diskstats00:09:343
11798991515,0cyclictest32623-21apt-get22:09:342
11798991515,0cyclictest0-21swapper/222:54:302
11798991514,1cyclictest2282-21apt-get23:19:322
11798991513,1cyclictest25304-21gltestperf22:59:372
1179899150,14cyclictest0-21swapper/223:14:402
11796991515,0cyclictest0-21swapper/119:39:311
11796991513,1cyclictest26472-21apt23:00:001
11794991515,0cyclictest0-21swapper/021:09:310
11794991513,1cyclictest28600-21mailstats00:09:390
1179499150,1cyclictest0-21swapper/022:29:390
1179499150,15cyclictest0-21swapper/020:04:330
1179499150,14cyclictest14961-21cut19:19:300
11799991414,0cyclictest0-21swapper/320:17:023
11799991414,0cyclictest0-21swapper/319:50:263
1179999140,14cyclictest0-21swapper/319:55:493
1179999140,14cyclictest0-21swapper/319:26:093
1179999140,0cyclictest0-21swapper/322:36:533
1179899140,14cyclictest0-21swapper/223:40:152
1179899140,14cyclictest0-21swapper/222:01:242
1179899140,14cyclictest0-21swapper/200:36:182
1179899140,0cyclictest0-21swapper/222:48:562
1179899140,0cyclictest0-21swapper/222:31:552
11796991414,0cyclictest0-21swapper/123:06:281
11796991414,0cyclictest0-21swapper/120:45:091
1179699140,14cyclictest0-21swapper/121:58:461
1179699140,0cyclictest0-21swapper/121:46:451
1179699140,0cyclictest0-21swapper/120:00:111
1179699140,0cyclictest0-21swapper/119:30:081
1179699140,0cyclictest0-21swapper/119:21:241
11794991414,0cyclictest0-21swapper/022:16:500
1179499140,14cyclictest0-21swapper/020:30:070
1179499140,0cyclictest0-21swapper/023:55:050
11798991313,0cyclictest0-21swapper/219:34:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional