You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 11:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Mon Dec 01, 2025 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21880992523,1cyclictest0-21swapper/100:15:001
2188199240,23cyclictest18324-21sh00:35:002
2188099240,0cyclictest0-21swapper/121:57:561
2187999240,23cyclictest14724-21apt-get23:22:580
2188299231,1cyclictest23552-21latency_hist20:20:003
21880992321,1cyclictest1641-21cron23:15:001
21880992320,2cyclictest5964-21idleruntime-cro20:49:591
21880992320,2cyclictest4184-21sendmail-msp22:59:591
2188099230,2cyclictest11502-21idleruntime-cro19:55:001
2187999230,22cyclictest30761-21perf20:35:000
2187999230,22cyclictest28995-21perf22:45:010
21882992222,0cyclictest0-21swapper/323:52:563
21882992220,1cyclictest1446-21munin-run00:00:003
21882992220,1cyclictest14255-21lockfile-remove23:20:003
2188299220,21cyclictest0-21swapper/323:37:583
2188299220,21cyclictest0-21swapper/322:45:023
2188299220,21cyclictest0-21swapper/321:47:563
2188299220,21cyclictest0-21swapper/319:32:553
2188299220,21cyclictest0-21swapper/300:32:573
2188299220,1cyclictest8363-21munin-run20:55:003
2188299220,1cyclictest31884-21idleruntime-cro19:30:003
2188299220,1cyclictest0-21swapper/320:47:563
21881992222,0cyclictest0-21swapper/221:17:562
2188199220,21cyclictest2340-21apt19:37:562
2188199220,21cyclictest0-21swapper/222:50:012
2188199220,21cyclictest0-21swapper/222:17:552
2188199220,21cyclictest0-21swapper/220:40:012
2188199220,1cyclictest0-21swapper/223:45:002
21880992220,1cyclictest0-21swapper/122:35:001
2188099220,21cyclictest11304-21apt-get21:02:561
2188099220,21cyclictest0-21swapper/120:02:541
2188099220,21cyclictest0-21swapper/119:15:001
2188099220,21cyclictest0-21swapper/100:27:561
2188099220,21cyclictest0-21swapper/100:07:561
2188099220,1cyclictest9216-21apt-get00:17:561
2188099220,1cyclictest7735-21apt-get22:02:551
2188099220,1cyclictest6434-21apt-get20:52:571
21879992221,0cyclictest0-21swapper/021:27:570
2187999220,21cyclictest31233-21apt-get20:37:530
2187999220,21cyclictest27556-21apt-get19:22:540
2187999220,21cyclictest26737-21apt-get23:47:550
2187999220,21cyclictest21433-21cron23:35:010
2187999220,21cyclictest0-21swapper/019:42:570
2187999220,21cyclictest0-21swapper/000:17:560
2187999220,1cyclictest720-21apt-get20:40:010
2187999220,1cyclictest20448-21sendmail-msp21:20:000
21882992120,0cyclictest19846-21apt-get22:27:573
21882992119,1cyclictest743-21awk20:40:003
21882992119,1cyclictest16950-21sh22:20:003
21882992118,2cyclictest8724-21basename00:15:003
2188299210,20cyclictest0-21swapper/323:07:583
2188299210,20cyclictest0-21swapper/320:27:543
2188299210,1cyclictest15949-21munin-run00:30:003
2188299210,0cyclictest0-21swapper/321:07:573
2188299210,0cyclictest0-21swapper/320:40:013
21881992119,1cyclictest30535-21apt-get21:42:552
2188199210,20cyclictest16821-21apt-get20:07:552
2188199210,20cyclictest0-21swapper/223:07:582
2188199210,20cyclictest0-21swapper/221:07:562
2188199210,20cyclictest0-21swapper/220:52:572
2188199210,20cyclictest0-21swapper/219:52:582
2188199210,20cyclictest0-21swapper/219:22:542
2188199210,20cyclictest0-21swapper/200:17:552
2188199210,0cyclictest0-21swapper/223:22:582
21880992121,0cyclictest0-21swapper/123:52:561
21880992121,0cyclictest0-21swapper/122:17:561
21880992121,0cyclictest0-21swapper/121:27:571
21880992120,0cyclictest0-21swapper/123:42:571
21880992120,0cyclictest0-21swapper/123:27:571
2188099210,21cyclictest0-21swapper/119:17:581
2188099210,20cyclictest1633-21apt-get20:42:571
2188099210,20cyclictest0-21swapper/122:47:581
2188099210,20cyclictest0-21swapper/121:32:551
2188099210,0cyclictest13704-21apt-get21:07:571
2188099210,0cyclictest0-21swapper/120:17:541
2188099210,0cyclictest0-21swapper/100:02:551
21879992121,0cyclictest16099-21apt-get21:12:560
21879992121,0cyclictest0-21swapper/022:37:560
21879992121,0cyclictest0-21swapper/020:52:570
21879992121,0cyclictest0-21swapper/000:22:570
21879992120,1cyclictest598-21rs:main0
21879992120,1cyclictest22832-21turbostat21:25:000
21879992120,1cyclictest14032-21apt-get00:27:560
21879992120,1cyclictest12305-21apt-get23:17:560
21879992119,1cyclictest31778-21perf22:50:000
21879992119,1cyclictest27628-21latency_hist21:35:000
2187999210,20cyclictest0-21swapper/020:12:560
2187999210,20cyclictest0-21swapper/000:02:550
2187999210,0cyclictest9555-21apt-get19:52:580
2187999210,0cyclictest1756-21perf22:54:590
2187999210,0cyclictest17118-21apt-get23:27:560
2187999210,0cyclictest0-21swapper/000:07:570
21882992020,0cyclictest0-21swapper/322:57:563
21882992020,0cyclictest0-21swapper/322:32:553
21882992020,0cyclictest0-21swapper/321:02:563
21882992020,0cyclictest0-21swapper/319:52:583
21882992019,1cyclictest26551-21cron22:40:003
21882992018,1cyclictest4423-21apt-get00:07:573
2188299200,20cyclictest0-21swapper/323:29:593
2188299200,1cyclictest0-21swapper/323:02:563
2188299200,0cyclictest0-21swapper/321:52:543
2188299200,0cyclictest0-21swapper/321:17:563
2188299200,0cyclictest0-21swapper/319:17:573
21881992020,0cyclictest21922-21apt-get23:37:572
21881992020,0cyclictest0-21swapper/223:27:562
21881992020,0cyclictest0-21swapper/220:57:562
21881992020,0cyclictest0-21swapper/220:22:582
21881992020,0cyclictest0-21swapper/220:17:542
21881992019,0cyclictest18348-21apt-get00:35:012
21881992019,0cyclictest0-21swapper/200:07:572
21881992018,1cyclictest2244-21apt-get22:57:572
21881992018,1cyclictest0-21swapper/220:14:592
2188199200,20cyclictest0-21swapper/221:47:562
2188199200,19cyclictest593-21dbus-daemon22:40:002
2188199200,0cyclictest0-21swapper/219:12:542
21880992020,0cyclictest8832-21apt-get20:57:561
21880992020,0cyclictest470-21apt-get21:47:561
21880992020,0cyclictest0-21swapper/123:47:561
21880992020,0cyclictest0-21swapper/122:22:561
21880992020,0cyclictest0-21swapper/119:42:571
21880992020,0cyclictest0-21swapper/100:22:571
21880992019,1cyclictest19216-21apt-get20:12:561
21880992019,0cyclictest0-21swapper/123:22:581
21880992018,1cyclictest19356-21sendmail00:40:001
2188099200,20cyclictest0-21swapper/122:55:001
2188099200,20cyclictest0-21swapper/121:17:561
2188099200,20cyclictest0-21swapper/100:32:571
2188099200,0cyclictest0-21swapper/123:37:581
21879992020,0cyclictest24339-21apt-get23:42:570
21879992020,0cyclictest0-21swapper/023:55:010
21879992020,0cyclictest0-21swapper/022:12:560
21879992020,0cyclictest0-21swapper/021:47:560
21879992019,1cyclictest6817-21apt-get00:12:560
21879992019,1cyclictest0-21swapper/000:35:010
21879992018,1cyclictest593-21dbus-daemon21:00:000
2187999200,20cyclictest16821-21apt-get20:07:550
2187999200,0cyclictest0-21swapper/022:17:560
21882991919,0cyclictest0-21swapper/323:47:563
21882991919,0cyclictest0-21swapper/323:10:023
21882991919,0cyclictest0-21swapper/322:02:563
21882991919,0cyclictest0-21swapper/321:57:563
21882991919,0cyclictest0-21swapper/320:32:553
21882991919,0cyclictest0-21swapper/319:47:563
21882991919,0cyclictest0-21swapper/300:02:553
21882991918,1cyclictest0-21swapper/300:22:573
21881991919,0cyclictest0-21swapper/223:57:562
21881991919,0cyclictest0-21swapper/223:10:012
21881991919,0cyclictest0-21swapper/222:22:552
21881991919,0cyclictest0-21swapper/221:52:542
21881991919,0cyclictest0-21swapper/221:02:552
21881991919,0cyclictest0-21swapper/220:32:552
21881991919,0cyclictest0-21swapper/200:12:562
21881991917,1cyclictest15933-21basename00:29:592
21880991919,0cyclictest0-21swapper/122:27:581
21880991919,0cyclictest0-21swapper/121:12:571
21880991919,0cyclictest0-21swapper/120:37:541
21880991917,1cyclictest2341-21apt-get19:37:551
2188099190,18cyclictest0-21swapper/123:32:561
21879991918,1cyclictest21614-21apt-get20:17:540
21879991918,0cyclictest0-21swapper/022:07:560
21879991917,1cyclictest7225-21cron22:00:010
21879991917,1cyclictest18348-21apt-get00:35:000
21882991818,0cyclictest25717-21apt-get21:32:553
21882991818,0cyclictest0-21swapper/300:17:563
21882991816,1cyclictest612-21gdbus00:39:593
21882991816,1cyclictest0-21swapper/323:25:003
2188299180,2cyclictest25946-21dump-pmu-power20:24:593
2188299180,17cyclictest0-21swapper/319:37:553
21881991818,0cyclictest4761-21apt-get19:42:562
21881991818,0cyclictest0-21swapper/222:27:582
21881991817,1cyclictest32354-21apt-get19:32:552
21881991816,1cyclictest4661-21apt-get23:02:572
21881991816,1cyclictest0-21swapper/200:05:002
21880991818,0cyclictest24044-21apt-get20:22:571
21880991818,0cyclictest12630-21apt-get22:12:561
21880991818,0cyclictest0-21swapper/123:07:581
21880991816,1cyclictest24647-21apt-get22:37:561
21880991816,1cyclictest13958-21awk19:59:591
2187999182,15cyclictest593-21dbus-daemon19:15:000
21879991818,0cyclictest0-21swapper/022:57:570
21879991818,0cyclictest0-21swapper/020:27:540
21879991816,2cyclictest6084-21sed23:03:040
21882991717,0cyclictest0-21swapper/322:22:563
21882991717,0cyclictest0-21swapper/322:12:563
21882991715,1cyclictest21137-21apt20:15:003
21881991717,0cyclictest0-21swapper/222:12:552
21881991717,0cyclictest0-21swapper/221:22:572
21881991717,0cyclictest0-21swapper/219:27:562
21881991715,1cyclictest19517-21apt-get23:32:572
21881991715,1cyclictest12156-21munin-run22:10:002
21881991714,2cyclictest313-21systemd-journal20:00:002
21881991714,2cyclictest1769-21awk22:55:002
2187999176,10cyclictest0-21swapper/021:07:590
21879991717,0cyclictest7060-21apt-get23:07:580
21879991717,0cyclictest17447-21apt-get22:22:550
21879991717,0cyclictest0-21swapper/021:52:530
21879991717,0cyclictest0-21swapper/021:02:560
21879991717,0cyclictest0-21swapper/019:40:000
21879991715,1cyclictest598-21rs:main0
21879991715,1cyclictest23862-21grep23:40:010
2187999170,0cyclictest0-21swapper/022:27:580
21882991616,0cyclictest0-21swapper/321:42:563
21882991616,0cyclictest0-21swapper/319:22:533
21882991614,1cyclictest30038-21perf21:40:003
21882991614,1cyclictest18735-21wc20:10:003
21881991616,0cyclictest0-21swapper/220:27:542
21881991614,1cyclictest21878-21cyclictest19:47:562
21881991614,1cyclictest0-21swapper/221:38:072
21880991616,0cyclictest0-21swapper/119:27:561
21880991614,1cyclictest14190-21cron23:20:001
21880991614,1cyclictest10657-21diskstats22:07:571
2188099160,8cyclictest0-21swapper/121:37:581
2188099160,2cyclictest898-21swap23:58:041
2188099160,0cyclictest0-21swapper/121:42:551
21879991616,0cyclictest0-21swapper/020:02:550
21879991615,0cyclictest0-21swapper/021:37:560
21879991613,2cyclictest31036-21awk23:55:000
2187999160,15cyclictest766-21snmpd19:56:190
21882991515,0cyclictest0-21swapper/321:12:573
21882991515,0cyclictest0-21swapper/319:57:533
21882991513,1cyclictest24331-21munin-run19:15:003
21881991515,0cyclictest27063-21apt-get22:42:552
21881991515,0cyclictest0-21swapper/220:47:562
21881991515,0cyclictest0-21swapper/220:37:542
21881991514,1cyclictest0-21swapper/219:15:012
21881991513,1cyclictest31131-21latency_hist23:55:002
2188199150,14cyclictest0-21swapper/223:15:462
21880991515,0cyclictest0-21swapper/123:02:551
21880991515,0cyclictest0-21swapper/122:42:551
21880991515,0cyclictest0-21swapper/119:22:541
2188099150,14cyclictest21565-21smartctl21:22:591
21879991515,0cyclictest0-21swapper/022:32:560
21879991514,1cyclictest9638-21idleruntime-cro22:05:010
21879991513,1cyclictest9589-21apt-get23:10:020
2187999151,0cyclictest0-21swapper/019:17:580
21882991414,0cyclictest0-21swapper/322:06:433
21882991414,0cyclictest0-21swapper/320:55:013
2188299140,14cyclictest0-21swapper/323:40:193
2188299140,14cyclictest0-21swapper/323:33:053
2188299140,14cyclictest0-21swapper/322:40:013
2188299140,14cyclictest0-21swapper/321:25:173
2188299140,14cyclictest0-21swapper/321:20:323
2188299140,0cyclictest0-21swapper/322:50:313
2188299140,0cyclictest0-21swapper/320:00:383
2188299140,0cyclictest0-21swapper/319:40:113
21881991414,0cyclictest11634-21apt-get00:22:562
21881991414,0cyclictest0-21swapper/221:27:132
2188199140,14cyclictest0-21swapper/221:55:182
2188199140,14cyclictest0-21swapper/220:04:072
2188199140,0cyclictest0-21swapper/223:45:532
2188199140,0cyclictest0-21swapper/222:31:192
2188199140,0cyclictest0-21swapper/222:02:182
21880991414,0cyclictest0-21swapper/120:08:161
21880991414,0cyclictest0-21swapper/119:47:431
21880991414,0cyclictest0-21swapper/119:33:251
2188099140,14cyclictest0-21swapper/120:31:011
2188099140,0cyclictest0-21swapper/120:26:071
21879991414,0cyclictest0-21swapper/020:45:430
2187999141,13cyclictest1778-21sendmail-mta19:46:080
2187999140,14cyclictest0-21swapper/019:25:150
21881991313,0cyclictest0-21swapper/221:11:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional