You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-30 - 19:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sat Aug 30, 2025 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
329899260,25cyclictest7653-21plymouth10:40:002
329999250,24cyclictest7745-21date10:40:003
329899240,1cyclictest23938-21apt-get07:54:532
3297992420,3cyclictest496-21idleruntime-cro10:25:001
329999230,22cyclictest21690-21cron12:15:003
3298992320,2cyclictest32413-21/usr/sbin/munin11:29:592
3298992320,2cyclictest32317-21awk08:10:002
329899230,22cyclictest28994-21munin-run12:29:592
329899230,22cyclictest21714-21perf12:15:002
329899230,0cyclictest10140-21apt-get09:39:542
3297992321,1cyclictest30125-21basename10:20:001
329799230,2cyclictest22860-21latency_hist07:49:591
3296992321,1cyclictest612-21gdbus11:40:000
329699230,22cyclictest17715-21sendmail-msp10:59:590
329699230,22cyclictest14915-21dump-pmu-power10:54:590
329699230,0cyclictest0-21swapper/011:49:550
3299992220,1cyclictest7347-21wc11:44:593
3299992220,1cyclictest7015-21apt-get07:19:543
3299992220,1cyclictest7015-21apt-get07:19:543
3299992220,1cyclictest30090-21sh08:05:003
3299992220,1cyclictest0-21swapper/310:59:593
3299992219,2cyclictest26635-21perf12:25:003
3299992219,2cyclictest25918-21awk10:09:593
329999220,21cyclictest32346-21munin-run08:10:003
329999220,21cyclictest27337-21apt-get11:20:043
329999220,21cyclictest0-21swapper/310:04:593
329999220,21cyclictest0-21swapper/309:49:533
329999220,1cyclictest19145-21apt-get07:44:543
3298992222,0cyclictest0-21swapper/207:49:522
3298992220,1cyclictest12520-21perf10:49:592
3298992219,2cyclictest14355-21perf08:40:002
329899220,2cyclictest7340-21turbostat11:44:592
329899220,21cyclictest612-21gdbus08:45:012
329899220,21cyclictest6065-21apt-get08:24:532
329899220,21cyclictest593-21dbus-daemon12:40:002
329899220,21cyclictest26602-21cron12:25:002
329899220,21cyclictest0-21swapper/211:49:552
329899220,21cyclictest0-21swapper/211:20:032
329899220,21cyclictest0-21swapper/209:14:542
329899220,21cyclictest0-21swapper/207:45:002
329899220,1cyclictest471-21apt-get09:19:562
329899220,1cyclictest3898-21apt-get11:39:542
3297992222,0cyclictest0-21swapper/107:14:531
3297992221,1cyclictest0-21swapper/111:04:541
3297992220,1cyclictest12025-21grep08:34:591
3297992220,1cyclictest0-21swapper/112:40:001
329799220,22cyclictest0-21swapper/109:09:561
329799220,21cyclictest8713-21apt-get11:49:561
329799220,21cyclictest29801-21turbostat.cron11:25:001
329799220,21cyclictest27989-21apt-get12:29:551
329799220,21cyclictest0-21swapper/112:04:551
329799220,21cyclictest0-21swapper/110:29:551
329799220,21cyclictest0-21swapper/110:09:531
329799220,21cyclictest0-21swapper/109:44:531
329799220,21cyclictest0-21swapper/109:40:001
329799220,21cyclictest0-21swapper/109:29:531
329799220,1cyclictest313-21systemd-journal10:00:001
329799220,1cyclictest0-21swapper/111:09:541
3296992220,1cyclictest9191-21turbostat08:29:590
3296992220,1cyclictest593-21dbus-daemon11:10:000
3296992220,1cyclictest26857-21perf09:05:010
3296992220,1cyclictest24372-21idleruntime-cro09:00:010
3296992220,1cyclictest1555-21turbostat.cron09:19:590
3296992219,2cyclictest12924-21turbostat07:30:000
329699220,21cyclictest5807-21cut07:15:000
329699220,21cyclictest20736-21munin-plugin-st10:00:000
329699220,21cyclictest0-21swapper/012:14:550
329699220,21cyclictest0-21swapper/010:04:550
329699220,21cyclictest0-21swapper/008:54:550
329699220,21cyclictest0-21swapper/008:54:550
329699220,21cyclictest0-21swapper/008:49:540
329699220,1cyclictest25281-21idleruntime-cro07:54:590
329699220,1cyclictest15348-21munin-run07:35:000
329699220,1cyclictest0-21swapper/012:40:010
329699220,1cyclictest0-21swapper/009:39:540
329699220,1cyclictest0-21swapper/008:45:000
3299992120,0cyclictest0-21swapper/308:19:543
3299992119,1cyclictest0-21swapper/312:30:003
329999210,21cyclictest0-21swapper/307:54:543
329999210,20cyclictest0-21swapper/312:34:583
329999210,20cyclictest0-21swapper/311:29:563
329999210,20cyclictest0-21swapper/310:15:003
329999210,20cyclictest0-21swapper/309:34:543
329999210,1cyclictest17350-21apt-get09:54:523
3298992121,0cyclictest18321-21apt-get12:09:542
3298992120,1cyclictest22167-21apt-get10:04:542
3298992120,0cyclictest0-21swapper/210:09:542
3298992120,0cyclictest0-21swapper/207:24:532
3298992119,1cyclictest27015-21apt-get10:14:552
3298992118,2cyclictest14567-21perf12:00:012
329899210,21cyclictest0-21swapper/210:19:582
329899210,20cyclictest26388-21apt-get07:59:522
329899210,0cyclictest23843-21apt-get11:14:582
329899210,0cyclictest0-21swapper/210:29:552
329899210,0cyclictest0-21swapper/208:34:542
3297992121,0cyclictest0-21swapper/107:54:531
3297992119,1cyclictest32425-21latency_hist11:29:591
3297992119,1cyclictest12954-21idleruntime-cro07:30:011
329799210,20cyclictest0-21swapper/111:39:541
329799210,20cyclictest0-21swapper/110:34:551
329799210,20cyclictest0-21swapper/108:59:541
329799210,20cyclictest0-21swapper/107:24:541
329799210,0cyclictest0-21swapper/109:49:541
329799210,0cyclictest0-21swapper/108:14:561
329799210,0cyclictest0-21swapper/107:39:551
3296992121,0cyclictest0-21swapper/011:29:560
3296992121,0cyclictest0-21swapper/011:24:570
3296992121,0cyclictest0-21swapper/011:14:570
3296992120,1cyclictest6065-21apt-get08:24:520
3296992120,1cyclictest31255-21apt-get08:09:560
3296992119,1cyclictest30089-21awk08:05:010
3296992119,1cyclictest17003-21cron12:05:000
329699210,2cyclictest8075-21cron07:20:010
329699210,2cyclictest8075-21cron07:20:010
329699210,20cyclictest1819-21apt-get10:29:550
329699210,20cyclictest0-21swapper/010:19:570
329699210,20cyclictest0-21swapper/010:09:530
329699210,20cyclictest0-21swapper/009:50:010
329699210,20cyclictest0-21swapper/007:49:520
329699210,20cyclictest0-21swapper/007:39:550
329699210,0cyclictest18321-21apt-get12:09:530
3299992020,0cyclictest0-21swapper/312:04:543
3299992020,0cyclictest0-21swapper/311:39:553
3299992020,0cyclictest0-21swapper/307:34:543
3299992017,2cyclictest16723-21sh08:45:013
3299992017,2cyclictest1400-21sh12:40:003
329999200,20cyclictest0-21swapper/312:19:563
329999200,20cyclictest0-21swapper/307:59:533
329999200,0cyclictest0-21swapper/309:39:553
3298992020,0cyclictest0-21swapper/208:55:002
3298992020,0cyclictest0-21swapper/208:55:002
3298992020,0cyclictest0-21swapper/208:04:532
3298992020,0cyclictest0-21swapper/207:19:542
3298992020,0cyclictest0-21swapper/207:19:542
329899200,20cyclictest0-21swapper/210:59:552
329899200,0cyclictest0-21swapper/212:34:572
329899200,0cyclictest0-21swapper/211:34:572
3297992020,0cyclictest14227-21apt-get07:34:541
3297992020,0cyclictest0-21swapper/112:19:541
3297992020,0cyclictest0-21swapper/111:34:571
3297992020,0cyclictest0-21swapper/110:04:551
3297992019,1cyclictest7015-21apt-get07:19:551
3297992019,1cyclictest7015-21apt-get07:19:541
3297992019,1cyclictest0-21swapper/111:15:001
3297992019,1cyclictest0-21swapper/109:24:541
3297992019,1cyclictest0-21swapper/109:14:541
3297992018,1cyclictest26649-21basename12:24:591
3296992020,0cyclictest4226-21apt-get10:34:560
3296992020,0cyclictest0-21swapper/012:34:580
3296992020,0cyclictest0-21swapper/011:59:560
3296992020,0cyclictest0-21swapper/011:34:580
3296992020,0cyclictest0-21swapper/010:49:540
3296992020,0cyclictest0-21swapper/010:44:560
329699200,20cyclictest0-21swapper/011:44:560
329699200,19cyclictest24193-21latency_hist12:19:590
329699200,0cyclictest0-21swapper/009:14:530
329699200,0cyclictest0-21swapper/007:24:530
3299991919,0cyclictest9432-21apt-get07:24:533
3299991919,0cyclictest1486-21apt-get11:34:583
3299991919,0cyclictest0-21swapper/311:54:553
3299991919,0cyclictest0-21swapper/310:44:553
3299991919,0cyclictest0-21swapper/310:34:563
3299991919,0cyclictest0-21swapper/310:29:563
3299991919,0cyclictest0-21swapper/309:09:553
3299991918,1cyclictest21541-21apt-get07:49:533
3299991917,1cyclictest0-21swapper/311:49:553
329999190,18cyclictest16730-21apt-get07:39:563
3298991919,0cyclictest0-21swapper/211:09:552
3298991919,0cyclictest0-21swapper/210:34:552
3298991919,0cyclictest0-21swapper/209:44:532
3298991919,0cyclictest0-21swapper/207:14:532
3298991917,1cyclictest8918-21munin-run09:35:002
329799191,7cyclictest0-21swapper/111:19:551
3297991917,1cyclictest0-21swapper/109:54:521
3296991919,0cyclictest0-21swapper/012:24:550
3296991919,0cyclictest0-21swapper/011:19:550
3296991919,0cyclictest0-21swapper/009:29:540
3296991919,0cyclictest0-21swapper/008:19:550
3296991919,0cyclictest0-21swapper/008:14:560
3296991919,0cyclictest0-21swapper/007:59:530
3296991919,0cyclictest0-21swapper/007:44:530
329699190,18cyclictest612-21gdbus09:45:010
3299991818,0cyclictest30530-21apt-get09:14:533
3299991818,0cyclictest26290-21apt-get11:19:563
3299991818,0cyclictest25735-21apt-get09:04:543
3299991818,0cyclictest1231-21apt-get08:14:553
3299991818,0cyclictest0-21swapper/311:14:583
3299991818,0cyclictest0-21swapper/309:59:553
3299991818,0cyclictest0-21swapper/309:19:553
3299991818,0cyclictest0-21swapper/308:39:553
3299991818,0cyclictest0-21swapper/308:34:543
3299991817,1cyclictest11437-21apt-get10:49:543
3299991816,1cyclictest0-21swapper/312:09:533
3299991816,1cyclictest0-21swapper/311:05:013
3298991818,0cyclictest0-21swapper/207:29:552
3298991816,1cyclictest2332-21munin-run08:15:002
3298991816,1cyclictest20707-21dump-pmu-power09:59:592
3297991818,0cyclictest0-21swapper/112:09:531
3297991818,0cyclictest0-21swapper/110:54:551
3297991817,1cyclictest19206-21perf08:50:001
3297991817,1cyclictest13510-21apt-get11:59:561
3297991817,0cyclictest0-21swapper/108:24:531
329799180,17cyclictest0-21swapper/108:09:551
3296991818,0cyclictest13260-21apt-get08:39:540
3296991818,0cyclictest0-21swapper/010:39:560
3296991818,0cyclictest0-21swapper/009:54:530
3299991717,0cyclictest2938-21apt-get09:24:533
3299991717,0cyclictest0-21swapper/310:19:583
3299991717,0cyclictest0-21swapper/308:24:523
3299991715,1cyclictest4615-21apt-get07:14:533
3299991715,1cyclictest0-21swapper/308:55:013
3299991715,1cyclictest0-21swapper/308:55:003
3298991717,0cyclictest0-21swapper/212:19:552
3298991717,0cyclictest0-21swapper/209:49:542
3298991717,0cyclictest0-21swapper/209:29:542
3298991717,0cyclictest0-21swapper/209:04:542
3298991716,0cyclictest11111-21apt-get11:54:542
329899170,14cyclictest0-21swapper/211:15:042
3297991717,0cyclictest0-21swapper/108:54:541
3297991717,0cyclictest0-21swapper/108:54:541
3297991717,0cyclictest0-21swapper/108:04:531
3297991717,0cyclictest0-21swapper/107:44:531
3296991717,0cyclictest0-21swapper/011:54:550
3296991717,0cyclictest0-21swapper/008:34:530
3299991616,0cyclictest0-21swapper/311:59:553
3299991616,0cyclictest0-21swapper/311:09:553
3299991614,1cyclictest18121-21/usr/sbin/munin08:49:553
3299991614,1cyclictest12555-21apt-get09:44:543
329999160,1cyclictest0-21swapper/309:00:003
3298991616,0cyclictest29232-21perf09:10:002
3298991616,0cyclictest15926-21apt-get12:04:552
3298991616,0cyclictest0-21swapper/211:04:552
3298991616,0cyclictest0-21swapper/208:59:542
3298991616,0cyclictest0-21swapper/207:39:562
329899160,15cyclictest4081-21cron09:25:002
329899160,15cyclictest17406-21timerandwakeup08:45:042
3297991616,0cyclictest20719-21apt-get12:14:551
3297991616,0cyclictest0-21swapper/110:14:561
3297991616,0cyclictest0-21swapper/109:19:561
3297991616,0cyclictest0-21swapper/108:19:541
329799160,15cyclictest25890-21unixbench_multi07:55:011
329799160,15cyclictest12547-21ntp_states10:50:001
3296991616,0cyclictest0-21swapper/009:24:530
3296991616,0cyclictest0-21swapper/009:09:560
3298991514,1cyclictest3650-21apt-get08:19:532
3297991513,1cyclictest9412-21latency_hist08:30:001
329799150,6cyclictest0-21swapper/110:39:551
3296991515,0cyclictest0-21swapper/009:34:530
3299991414,0cyclictest0-21swapper/310:20:583
3299991414,0cyclictest0-21swapper/307:29:183
329999140,14cyclictest0-21swapper/310:50:373
329999140,14cyclictest0-21swapper/309:25:273
329999140,14cyclictest0-21swapper/308:25:483
3298991414,0cyclictest0-21swapper/210:44:542
329899140,14cyclictest0-21swapper/210:20:332
329899140,14cyclictest0-21swapper/207:32:342
329899140,0cyclictest0-21swapper/209:54:202
329899140,0cyclictest0-21swapper/208:25:032
3297991414,0cyclictest0-21swapper/109:00:011
329799140,14cyclictest0-21swapper/108:35:181
329799140,0cyclictest0-21swapper/112:30:031
329799140,0cyclictest0-21swapper/111:50:041
329799140,0cyclictest0-21swapper/108:40:331
329699140,14cyclictest0-21swapper/011:03:070
329699140,14cyclictest0-21swapper/010:20:390
329699140,14cyclictest0-21swapper/010:10:020
329699140,0cyclictest0-21swapper/012:25:500
329899130,0cyclictest0-21swapper/210:50:362
3297991313,0cyclictest0-21swapper/109:31:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional