You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 05:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Jul 01, 2025 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1588299251,23cyclictest27896-21tr20:40:000
1588799240,23cyclictest12852-21basename00:40:003
15884992422,1cyclictest593-21dbus-daemon22:19:591
1588499240,23cyclictest24749-21tr21:40:001
1588299240,1cyclictest15759-21apt-get21:24:460
1588299240,1cyclictest13-21ksoftirqd/022:44:470
1588799230,22cyclictest0-21swapper/319:30:013
15886992321,1cyclictest12802-21sh00:40:002
15886992321,1cyclictest0-21swapper/221:40:002
15886992320,3cyclictest0-21swapper/219:34:412
15886992320,2cyclictest313-21systemd-journal22:19:592
1588699230,22cyclictest25479-21basename20:35:002
15884992320,2cyclictest15867-21idleruntime-cro20:15:001
1588499230,22cyclictest593-21dbus-daemon23:20:001
1588499230,22cyclictest24048-21apt-get19:29:441
1588299230,2cyclictest21012-21perf19:20:000
1588299230,22cyclictest9916-21awk21:10:000
15887992222,0cyclictest0-21swapper/323:09:453
15887992221,1cyclictest27725-21basename00:00:003
15887992221,1cyclictest0-21swapper/322:44:473
1588799220,21cyclictest0-21swapper/322:59:473
1588799220,21cyclictest0-21swapper/321:54:473
1588799220,0cyclictest0-21swapper/300:14:453
15886992220,1cyclictest14036-21perf22:25:002
15886992220,1cyclictest0-21swapper/220:24:422
1588699220,21cyclictest2566-21apt-get22:04:502
1588699220,21cyclictest19463-21apt-get22:39:472
1588699220,21cyclictest0-21swapper/220:04:432
1588699220,1cyclictest7540-21munin-plugin-st21:05:002
1588699220,1cyclictest0-21swapper/223:14:462
15884992221,1cyclictest25367-21apt-get21:44:441
15884992220,1cyclictest16441-21apt-get19:10:011
15884992220,1cyclictest14729-21apt-get21:20:021
15884992219,2cyclictest25471-21basename20:35:001
1588499220,21cyclictest31653-21apt-get19:44:441
1588499220,21cyclictest30211-21munin-run00:05:001
1588499220,21cyclictest21129-21apt-get23:49:441
15882992220,1cyclictest8044-21munin-run00:30:000
1588299220,21cyclictest5751-21apt-get21:04:440
1588299220,21cyclictest27686-21dump-pmu-power00:00:000
1588299220,21cyclictest27069-21apt-get22:54:470
1588299220,21cyclictest0-21swapper/022:10:000
1588299220,1cyclictest8672-21munin-plugin-st20:00:000
1588299220,1cyclictest30288-21munin-run20:45:000
1588299220,1cyclictest13302-21awk23:30:000
1588299220,1cyclictest10879-21idleruntime-cro23:25:000
15887992120,1cyclictest0-21swapper/300:09:463
15887992120,0cyclictest21300-21apt-get20:29:453
15887992119,1cyclictest0-21swapper/322:05:003
15887992119,1cyclictest0-21swapper/319:25:003
15887992119,1cyclictest0-21swapper/300:19:443
1588799210,20cyclictest0-21swapper/323:39:473
1588799210,20cyclictest0-21swapper/321:34:443
1588799210,20cyclictest0-21swapper/319:19:473
1588799210,0cyclictest0-21swapper/322:09:453
1588799210,0cyclictest0-21swapper/321:29:443
1588799210,0cyclictest0-21swapper/300:04:463
15886992120,1cyclictest30210-21perf00:05:002
15886992119,1cyclictest0-21swapper/219:40:002
1588699210,20cyclictest19242-21apt-get19:19:472
1588699210,20cyclictest14713-21perf21:20:002
1588699210,20cyclictest0-21swapper/220:59:432
1588699210,20cyclictest0-21swapper/220:49:452
1588699210,1cyclictest7405-21apt-get22:14:472
1588699210,0cyclictest0-21swapper/222:59:462
15884992121,0cyclictest0-21swapper/122:25:001
15884992120,1cyclictest29244-21apt-get19:39:481
15884992120,1cyclictest21880-21apt-get22:44:471
15884992120,0cyclictest30162-21apt-get21:54:481
15884992120,0cyclictest27766-21apt-get21:49:451
15884992120,0cyclictest0-21swapper/121:09:441
15884992119,1cyclictest18815-21cron22:35:001
1588499210,20cyclictest0-21swapper/122:59:461
1588499210,1cyclictest23441-21munin-plugin-st19:25:001
1588499210,0cyclictest0-21swapper/123:44:451
15882992121,0cyclictest0-21swapper/020:24:420
15882992120,1cyclictest26840-21apt-get19:34:410
15882992120,0cyclictest17065-21apt-get22:34:470
15882992120,0cyclictest0-21swapper/022:00:000
15882992119,1cyclictest13-21ksoftirqd/022:25:000
15882992119,1cyclictest0-21swapper/022:30:000
15882992118,2cyclictest25294-21idleruntime-cro23:54:590
1588299210,21cyclictest0-21swapper/023:39:460
1588299210,1cyclictest593-21dbus-daemon23:00:000
1588299210,0cyclictest0-21swapper/020:59:440
15887992020,0cyclictest4255-21apt-get23:14:463
15887992020,0cyclictest2036-21apt-get19:49:453
15887992020,0cyclictest0-21swapper/322:54:473
15887992020,0cyclictest0-21swapper/321:59:443
15887992020,0cyclictest0-21swapper/321:49:453
15887992020,0cyclictest0-21swapper/321:04:433
15887992020,0cyclictest0-21swapper/320:44:463
15887992020,0cyclictest0-21swapper/320:19:423
15887992020,0cyclictest0-21swapper/300:34:433
15887992018,1cyclictest9170-21perf22:15:003
15887992018,1cyclictest25812-21apt19:30:003
15887992018,1cyclictest22954-21apt-get21:39:463
15887992018,1cyclictest0-21swapper/321:20:023
1588799200,0cyclictest0-21swapper/323:34:443
1588799200,0cyclictest0-21swapper/323:24:473
1588799200,0cyclictest0-21swapper/319:39:483
1588799200,0cyclictest0-21swapper/319:14:463
15886992020,0cyclictest0-21swapper/223:44:452
15886992020,0cyclictest0-21swapper/220:54:442
15886992020,0cyclictest0-21swapper/219:44:442
15886992020,0cyclictest0-21swapper/219:29:432
15886992019,1cyclictest6883-21apt-get19:59:472
15886992019,0cyclictest0-21swapper/223:24:472
15886992018,1cyclictest8145-21apt-get21:09:432
15886992018,1cyclictest1829-21/usr/sbin/munin23:09:452
15886992018,1cyclictest0-21swapper/220:15:002
15886992017,2cyclictest593-21dbus-daemon22:35:002
1588699200,0cyclictest0-21swapper/220:19:422
1588699200,0cyclictest0-21swapper/220:09:432
15884992020,0cyclictest875-21apt-get20:54:441
15884992020,0cyclictest11046-21apt-get00:39:461
15884992020,0cyclictest0-21swapper/123:59:441
15884992020,0cyclictest0-21swapper/123:24:481
15884992020,0cyclictest0-21swapper/123:09:451
15884992020,0cyclictest0-21swapper/121:14:431
15884992020,0cyclictest0-21swapper/120:49:461
15884992020,0cyclictest0-21swapper/120:39:471
15884992020,0cyclictest0-21swapper/100:14:441
15884992019,0cyclictest0-21swapper/120:24:421
1588499200,20cyclictest0-21swapper/119:59:471
1588499200,0cyclictest0-21swapper/119:34:411
15882992020,0cyclictest1861-21apt-get23:09:440
15882992020,0cyclictest0-21swapper/022:39:480
15882992020,0cyclictest0-21swapper/021:19:450
15882992020,0cyclictest0-21swapper/019:54:490
15882992020,0cyclictest0-21swapper/000:09:460
1588299200,1cyclictest0-21swapper/022:19:590
1588299200,19cyclictest0-21swapper/023:14:470
1588299200,0cyclictest30905-21apt-get20:49:450
1588299200,0cyclictest0-21swapper/021:44:440
1588299200,0cyclictest0-21swapper/020:09:430
15887991919,0cyclictest0-21swapper/322:39:473
15887991919,0cyclictest0-21swapper/322:24:463
15887991919,0cyclictest0-21swapper/320:49:463
15887991919,0cyclictest0-21swapper/319:44:443
15887991916,2cyclictest25451-21cron20:35:003
15887991915,3cyclictest14713-21perf21:20:003
15886991919,0cyclictest0-21swapper/221:44:442
15886991919,0cyclictest0-21swapper/221:20:022
15886991919,0cyclictest0-21swapper/221:14:432
15886991919,0cyclictest0-21swapper/220:39:472
15886991919,0cyclictest0-21swapper/219:10:012
15886991919,0cyclictest0-21swapper/200:14:452
15886991919,0cyclictest0-21swapper/200:09:462
15886991917,1cyclictest15700-21munin-run23:35:002
15884991919,0cyclictest11687-21apt-get20:09:431
15884991919,0cyclictest0-21swapper/123:35:001
15884991919,0cyclictest0-21swapper/122:04:491
15884991919,0cyclictest0-21swapper/120:59:431
15884991919,0cyclictest0-21swapper/120:44:441
15884991919,0cyclictest0-21swapper/120:29:451
15884991919,0cyclictest0-21swapper/119:49:441
15884991919,0cyclictest0-21swapper/100:19:441
15884991918,0cyclictest9292-21apt-get20:04:431
15884991918,0cyclictest0-21swapper/122:09:451
1588499190,1cyclictest1237-21munin-run23:05:001
15882991919,0cyclictest16486-21apt-get20:19:420
15882991919,0cyclictest0-21swapper/020:14:440
15882991919,0cyclictest0-21swapper/019:44:440
15882991919,0cyclictest0-21swapper/000:35:010
15882991917,1cyclictest9182-21latency_hist22:15:000
15882991917,1cyclictest2650-21cron20:55:000
15882991917,1cyclictest1008-21apt-get19:45:020
15882991916,2cyclictest24745-21grep21:40:000
15887991818,0cyclictest3282-21apt-get20:59:433
15887991818,0cyclictest0-21swapper/322:49:463
15887991818,0cyclictest0-21swapper/320:39:473
15886991818,0cyclictest21652-21apt-get19:24:432
15886991818,0cyclictest0-21swapper/223:59:442
15886991818,0cyclictest0-21swapper/223:19:452
15886991818,0cyclictest0-21swapper/222:44:472
15886991818,0cyclictest0-21swapper/221:54:472
15886991818,0cyclictest0-21swapper/221:49:452
15886991818,0cyclictest0-21swapper/220:29:462
15886991817,1cyclictest32554-21apt-get21:59:442
15886991817,0cyclictest0-21swapper/220:44:442
15886991817,0cyclictest0-21swapper/200:19:442
15884991818,0cyclictest0-21swapper/123:39:471
15884991818,0cyclictest0-21swapper/121:59:441
15884991818,0cyclictest0-21swapper/121:29:451
15884991816,1cyclictest26453-21basename22:50:001
15882991818,0cyclictest0-21swapper/019:29:440
15882991816,1cyclictest22325-21perf21:35:000
1588299180,17cyclictest0-21swapper/019:39:550
15887991717,0cyclictest0-21swapper/323:19:453
15887991717,0cyclictest0-21swapper/320:15:003
15887991716,1cyclictest0-21swapper/322:29:483
15887991716,0cyclictest0-21swapper/321:44:443
15887991710,2cyclictest0-21swapper/320:09:433
15886991717,0cyclictest0-21swapper/222:29:482
15886991717,0cyclictest0-21swapper/222:09:452
15886991717,0cyclictest0-21swapper/219:49:442
15886991715,1cyclictest4437-21apt-get19:54:492
15884991717,0cyclictest0-21swapper/123:54:441
15884991717,0cyclictest0-21swapper/121:04:441
15884991717,0cyclictest0-21swapper/119:54:491
15884991715,1cyclictest32605-21awk00:10:001
15884991715,1cyclictest0-21swapper/122:40:011
1588499170,0cyclictest0-21swapper/122:29:481
15882991717,0cyclictest0-21swapper/023:49:450
15882991717,0cyclictest0-21swapper/000:04:450
15882991716,1cyclictest30161-21apt21:54:470
15882991715,1cyclictest2583-21latency_hist00:15:000
15887991616,0cyclictest0-21swapper/323:29:453
15887991616,0cyclictest0-21swapper/323:04:463
15887991616,0cyclictest0-21swapper/321:09:423
15887991614,1cyclictest0-21swapper/320:00:003
1588799160,15cyclictest0-21swapper/323:54:593
15886991616,0cyclictest0-21swapper/223:39:472
15886991615,1cyclictest0-21swapper/222:50:012
15886991615,0cyclictest0-21swapper/223:54:442
1588699160,14cyclictest0-21swapper/221:25:442
1588499163,12cyclictest15984-21munin-node00:26:141
15884991616,0cyclictest4255-21apt-get23:14:461
15884991616,0cyclictest0-21swapper/120:19:421
15884991614,1cyclictest9697-21ntp_states00:34:491
1588299166,9cyclictest781-21lldpd00:24:370
15882991616,0cyclictest6705-21apt-get23:19:450
15882991616,0cyclictest0-21swapper/020:29:450
15882991616,0cyclictest0-21swapper/020:04:430
15882991616,0cyclictest0-21swapper/000:39:460
15882991614,1cyclictest593-21dbus-daemon21:15:000
15882991614,1cyclictest19991-21munin-run21:30:000
15882991614,1cyclictest0-21swapper/023:34:460
1588299160,0cyclictest24669-21apt-get22:49:460
15887991515,0cyclictest0-21swapper/322:34:473
15887991513,1cyclictest5648-21munin-run00:25:003
15887991513,1cyclictest11620-21latency_hist22:20:003
15887991513,1cyclictest0-21swapper/300:29:453
15886991515,0cyclictest0-21swapper/221:34:442
15884991513,1cyclictest28848-21munin-run22:55:001
1588499150,14cyclictest0-21swapper/119:15:511
15882991515,0cyclictest18733-21apt-get23:44:450
1588299150,1cyclictest4995-21perf00:20:000
1588299150,1cyclictest0-21swapper/023:04:480
15887991414,0cyclictest0-21swapper/321:10:433
1588799140,14cyclictest0-21swapper/320:50:423
1588799140,14cyclictest0-21swapper/320:21:023
1588799140,14cyclictest0-21swapper/320:00:063
1588799140,14cyclictest0-21swapper/319:51:413
1588799140,0cyclictest0-21swapper/323:47:023
1588799140,0cyclictest0-21swapper/323:40:083
15886991414,0cyclictest0-21swapper/223:25:172
15886991414,0cyclictest0-21swapper/223:01:112
1588699140,14cyclictest0-21swapper/223:45:132
1588699140,14cyclictest0-21swapper/222:50:102
1588699140,14cyclictest0-21swapper/200:30:322
1588699140,14cyclictest0-21swapper/200:26:042
1588699140,14cyclictest0-21swapper/200:20:012
15884991414,0cyclictest0-21swapper/123:25:021
1588499140,0cyclictest0-21swapper/122:10:011
1588499140,0cyclictest0-21swapper/100:21:011
15882991414,0cyclictest0-21swapper/022:00:300
1588299140,14cyclictest0-21swapper/021:46:110
1588299140,14cyclictest0-21swapper/019:20:300
1588299140,0cyclictest0-21swapper/020:32:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional