You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 15:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Tue Jul 15, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2674399250,24cyclictest0-21swapper/300:00:553
2674399230,22cyclictest6689-21apt-get20:40:563
26738992321,1cyclictest0-21swapper/220:00:002
26738992320,2cyclictest23705-21apt23:25:002
2673899230,22cyclictest16289-21apt19:54:592
26734992321,1cyclictest4375-21apt-get22:45:591
2673499230,22cyclictest2827-21apt-get21:40:011
2673499230,22cyclictest16438-21ls23:10:001
2673499230,21cyclictest313-21systemd-journal20:55:001
2673499230,1cyclictest0-21swapper/121:05:571
26729992323,0cyclictest0-21swapper/023:55:560
2672999230,22cyclictest22853-21apt-get22:20:010
26743992220,1cyclictest6078-21dump-pmu-power23:55:003
26743992220,1cyclictest11242-21dump-pmu-power23:00:003
26743992220,1cyclictest10404-21perf21:55:003
2674399220,21cyclictest29098-21apt-get20:20:573
2674399220,21cyclictest16763-21apt-get23:10:563
2674399220,21cyclictest0-21swapper/323:25:573
2674399220,21cyclictest0-21swapper/322:45:593
26738992222,0cyclictest9164-21apt-get22:55:562
26738992220,2cyclictest10993-21basename00:05:002
26738992220,1cyclictest0-21swapper/219:49:592
2673899220,21cyclictest593-21dbus-daemon19:15:002
2673899220,21cyclictest20779-21apt-get22:15:562
2673899220,1cyclictest0-21swapper/200:35:012
26734992221,1cyclictest0-21swapper/122:20:561
26734992220,1cyclictest0-21swapper/121:25:541
2673499220,21cyclictest25617-21sendmail-msp21:20:011
2673499220,21cyclictest16620-21apt-get19:55:551
2673499220,21cyclictest0-21swapper/122:35:591
2673499220,21cyclictest0-21swapper/120:40:551
2673499220,21cyclictest0-21swapper/119:35:011
2673499220,21cyclictest0-21swapper/100:00:551
2673499220,1cyclictest28025-21dump-pmu-power21:25:011
2673499220,1cyclictest27029-21apt-get19:10:531
26729992222,0cyclictest0-21swapper/019:45:550
26729992220,1cyclictest0-21swapper/022:00:550
2672999220,21cyclictest2823-21perf21:40:000
2672999220,21cyclictest0-21swapper/023:50:540
2672999220,21cyclictest0-21swapper/022:55:570
2672999220,21cyclictest0-21swapper/022:05:570
2672999220,21cyclictest0-21swapper/019:50:570
2672999220,21cyclictest0-21swapper/019:35:570
2672999220,21cyclictest0-21swapper/019:10:540
26743992121,0cyclictest0-21swapper/300:25:553
26743992120,0cyclictest0-21swapper/320:25:543
26743992119,1cyclictest16307-21munin-run19:54:593
26743992119,1cyclictest0-21swapper/319:25:003
2674399210,21cyclictest0-21swapper/321:30:583
2674399210,21cyclictest0-21swapper/300:16:003
2674399210,20cyclictest28782-21apt-get22:30:543
2674399210,20cyclictest0-21swapper/320:30:573
2674399210,20cyclictest0-21swapper/320:00:573
26738992120,1cyclictest1466-21apt-get20:30:582
26738992120,0cyclictest31488-21apt-get20:25:552
26738992120,0cyclictest0-21swapper/222:50:552
26738992119,1cyclictest4306-21munin-run19:30:002
26738992119,1cyclictest15606-21dump-pmu-power22:05:002
2673899210,20cyclictest26670-21apt-get20:15:542
2673899210,1cyclictest8767-21perf20:45:002
2673899210,1cyclictest6416-21apt-get23:55:562
26734992121,0cyclictest0-21swapper/120:35:561
26734992121,0cyclictest0-21swapper/100:16:021
26734992119,1cyclictest11248-21grep23:00:001
26734992119,1cyclictest10410-21apt21:55:001
2673499210,21cyclictest0-21swapper/120:30:571
2673499210,20cyclictest0-21swapper/119:25:561
2673499210,0cyclictest0-21swapper/123:45:551
2673499210,0cyclictest0-21swapper/122:15:571
2673499210,0cyclictest0-21swapper/119:35:571
2673499210,0cyclictest0-21swapper/100:35:561
26729992121,0cyclictest0-21swapper/021:25:540
26729992120,1cyclictest4270-21apt-get20:35:560
26729992120,1cyclictest23702-21apt-get00:30:550
26729992120,1cyclictest0-21swapper/020:00:000
26729992120,0cyclictest25665-21apt-get22:25:010
26729992120,0cyclictest0-21swapper/022:45:590
26729992118,2cyclictest21172-21perf20:05:000
2672999210,20cyclictest0-21swapper/021:15:550
2672999210,0cyclictest30746-21apt-get21:30:580
2672999210,0cyclictest0-21swapper/023:45:550
2672999210,0cyclictest0-21swapper/023:05:010
2672999210,0cyclictest0-21swapper/022:30:550
2672999210,0cyclictest0-21swapper/020:55:580
26743992020,0cyclictest7024-21apt-get19:35:573
26743992020,0cyclictest28348-21apt-get21:25:543
26743992020,0cyclictest0-21swapper/323:35:573
26743992019,0cyclictest0-21swapper/321:05:573
26743992018,1cyclictest6355-21sendmail20:39:593
2674399200,0cyclictest27029-21apt-get19:10:543
2674399200,0cyclictest0-21swapper/323:15:573
2674399200,0cyclictest0-21swapper/322:40:563
2674399200,0cyclictest0-21swapper/322:25:013
2674399200,0cyclictest0-21swapper/322:00:023
2674399200,0cyclictest0-21swapper/320:10:573
26738992020,0cyclictest0-21swapper/222:30:542
26738992020,0cyclictest0-21swapper/222:20:012
26738992020,0cyclictest0-21swapper/221:50:002
26738992020,0cyclictest0-21swapper/221:10:582
26738992020,0cyclictest0-21swapper/200:20:552
26738992019,0cyclictest1573-21apt-get22:40:562
26738992018,1cyclictest20817-21munin-run21:10:002
2673899200,20cyclictest0-21swapper/221:20:572
2673899200,20cyclictest0-21swapper/219:20:542
2673899200,0cyclictest20913-21apt-get00:25:562
26734992020,0cyclictest24277-21apt-get20:10:561
26734992020,0cyclictest0-21swapper/123:35:581
26734992020,0cyclictest0-21swapper/121:35:531
26734992020,0cyclictest0-21swapper/121:30:571
26734992020,0cyclictest0-21swapper/100:30:021
26734992020,0cyclictest0-21swapper/100:05:561
26734992019,1cyclictest0-21swapper/123:30:011
26729992020,0cyclictest18737-21apt-get21:05:580
26729992020,0cyclictest0-21swapper/022:35:590
26729992020,0cyclictest0-21swapper/021:00:580
26729992020,0cyclictest0-21swapper/019:30:550
26729992019,1cyclictest0-21swapper/000:16:010
26729992019,0cyclictest0-21swapper/020:15:540
26729992018,2cyclictest10397-21cron21:55:000
2672999200,20cyclictest0-21swapper/021:20:570
2672999200,20cyclictest0-21swapper/020:10:570
2672999200,19cyclictest28899-21perf23:35:000
2672999200,19cyclictest23549-21awk20:10:000
2672999200,19cyclictest13376-21dump-pmu-power00:10:010
2672999200,0cyclictest0-21swapper/023:10:560
2672999200,0cyclictest0-21swapper/022:10:580
2672999200,0cyclictest0-21swapper/020:50:550
2672999200,0cyclictest0-21swapper/019:15:570
2672999200,0cyclictest0-21swapper/000:35:560
26743991919,0cyclictest0-21swapper/321:00:573
26743991919,0cyclictest0-21swapper/320:45:573
26743991919,0cyclictest0-21swapper/319:55:563
26743991918,1cyclictest5947-21apt-get21:46:013
26743991917,1cyclictest593-21dbus-daemon20:55:003
26738991919,0cyclictest1619-21apt-get23:45:552
26738991919,0cyclictest0-21swapper/223:25:582
26738991919,0cyclictest0-21swapper/223:15:572
26738991919,0cyclictest0-21swapper/223:04:592
26738991919,0cyclictest0-21swapper/221:40:572
26738991919,0cyclictest0-21swapper/220:15:002
26738991919,0cyclictest0-21swapper/200:35:562
26738991919,0cyclictest0-21swapper/200:10:562
26738991918,1cyclictest15944-21apt21:00:002
26734991919,0cyclictest0-21swapper/122:30:541
26734991919,0cyclictest0-21swapper/121:55:561
26734991919,0cyclictest0-21swapper/100:25:551
26734991918,1cyclictest19091-21apt-get20:00:561
26729991919,0cyclictest11162-21perf20:50:000
26729991919,0cyclictest0-21swapper/000:00:570
26729991918,1cyclictest0-21swapper/020:30:570
26729991918,0cyclictest0-21swapper/022:15:560
26743991818,0cyclictest23555-21apt-get20:10:003
26743991818,0cyclictest0-21swapper/323:30:013
26743991818,0cyclictest0-21swapper/322:20:563
26743991818,0cyclictest0-21swapper/319:40:563
26743991816,1cyclictest593-21dbus-daemon21:45:003
26738991818,0cyclictest4017-21apt-get23:50:542
26738991818,0cyclictest2827-21apt-get21:40:012
26738991818,0cyclictest26717-21apt-get23:30:022
26738991818,0cyclictest0-21swapper/220:50:552
26738991818,0cyclictest0-21swapper/219:30:552
26738991817,0cyclictest16450-21munin-run23:10:002
26734991818,0cyclictest31640-21apt-get23:40:581
26734991818,0cyclictest0-21swapper/123:55:561
26734991818,0cyclictest0-21swapper/120:25:011
26734991818,0cyclictest0-21swapper/120:05:551
26734991818,0cyclictest0-21swapper/119:40:561
26734991816,1cyclictest596-21in:imuxsock22:10:001
26734991816,1cyclictest11579-21apt-get23:00:551
2673499180,0cyclictest0-21swapper/121:10:581
26729991818,0cyclictest0-21swapper/023:40:580
26743991717,0cyclictest23526-21apt-get21:15:553
26743991717,0cyclictest0-21swapper/323:20:543
26743991717,0cyclictest0-21swapper/322:10:573
26743991717,0cyclictest0-21swapper/321:20:573
26743991717,0cyclictest0-21swapper/300:35:573
26743991715,1cyclictest16442-21awk23:10:003
26738991717,0cyclictest31577-21apt-get22:35:592
26738991717,0cyclictest0-21swapper/222:45:592
26738991717,0cyclictest0-21swapper/221:15:552
26738991717,0cyclictest0-21swapper/220:35:552
26738991717,0cyclictest0-21swapper/220:20:572
26738991716,1cyclictest16073-21/usr/sbin/munin00:16:012
2673899170,0cyclictest29225-21apt-get23:35:582
26734991717,0cyclictest0-21swapper/120:55:581
26734991717,0cyclictest0-21swapper/119:15:571
26734991716,0cyclictest18517-21apt-get00:20:561
2673499170,1cyclictest26102-21perf23:30:001
2673499170,16cyclictest0-21swapper/120:25:541
26729991717,0cyclictest29225-21apt-get23:35:570
26729991717,0cyclictest0-21swapper/000:25:550
26729991717,0cyclictest0-21swapper/000:10:560
26729991716,0cyclictest0-21swapper/023:00:560
26743991616,0cyclictest13706-21apt-get00:10:563
26743991616,0cyclictest0-21swapper/321:35:533
26743991616,0cyclictest0-21swapper/320:55:583
26743991615,1cyclictest0-21swapper/319:15:573
26743991614,1cyclictest22853-21apt-get22:19:593
26743991614,1cyclictest0-21swapper/321:55:563
2674399160,9cyclictest30850-21apt-get22:35:023
2674399160,1cyclictest23216-21/usr/sbin/munin21:15:003
26738991616,0cyclictest16289-21apt-get21:00:572
26738991616,0cyclictest0-21swapper/221:50:572
26738991616,0cyclictest0-21swapper/200:05:562
2673899160,15cyclictest31619-21acpi23:40:572
26734991616,0cyclictest0-21swapper/123:50:541
26734991616,0cyclictest0-21swapper/121:45:591
26734991614,1cyclictest15604-21apt22:05:001
26729991616,0cyclictest0-21swapper/021:55:560
26729991616,0cyclictest0-21swapper/020:20:570
26729991616,0cyclictest0-21swapper/019:20:540
26743991515,0cyclictest0-21swapper/300:20:553
26743991513,1cyclictest0-21swapper/319:46:013
2674399150,14cyclictest5677-21ntp_states19:31:003
26738991515,0cyclictest26378-21apt-get22:25:572
26738991514,1cyclictest126712sleep221:58:462
26738991513,1cyclictest23552-21latency_hist20:10:002
2673899150,1cyclictest9082-21apt-get20:45:562
26734991515,0cyclictest0-21swapper/122:25:561
26734991515,0cyclictest0-21swapper/121:00:571
2673499150,15cyclictest18601-21cstates22:10:581
2672999152,0cyclictest151rcu_preempt20:25:580
26729991515,0cyclictest0-21swapper/021:45:590
26729991515,0cyclictest0-21swapper/021:40:560
26729991515,0cyclictest0-21swapper/021:10:580
2672999150,0cyclictest0-21swapper/020:44:100
26743991414,0cyclictest0-21swapper/323:40:283
26743991414,0cyclictest0-21swapper/323:00:133
2674399140,14cyclictest0-21swapper/323:45:363
2674399140,14cyclictest0-21swapper/322:51:133
2674399140,14cyclictest0-21swapper/320:15:103
2674399140,0cyclictest0-21swapper/323:55:083
2674399140,0cyclictest0-21swapper/322:06:013
2674399140,0cyclictest0-21swapper/300:05:073
26738991414,0cyclictest0-21swapper/223:10:152
2673899140,14cyclictest0-21swapper/222:10:152
2673899140,14cyclictest0-21swapper/222:05:252
2673899140,14cyclictest0-21swapper/221:30:332
2673899140,14cyclictest0-21swapper/221:25:512
2673899140,14cyclictest0-21swapper/220:00:122
2673899140,14cyclictest0-21swapper/219:15:132
2673899140,0cyclictest0-21swapper/219:40:272
2673899140,0cyclictest0-21swapper/219:35:462
26734991414,0cyclictest0-21swapper/123:20:161
26734991414,0cyclictest0-21swapper/119:20:241
2673499140,14cyclictest0-21swapper/123:10:421
2673499140,14cyclictest0-21swapper/122:40:191
2673499140,14cyclictest0-21swapper/120:45:241
2673499140,14cyclictest0-21swapper/120:15:401
2673499140,14cyclictest0-21swapper/119:45:011
2673499140,0cyclictest0-21swapper/123:15:041
2673499140,0cyclictest0-21swapper/119:50:091
26729991414,0cyclictest0-21swapper/022:50:530
26729991414,0cyclictest0-21swapper/019:40:120
2672999140,14cyclictest0-21swapper/023:25:130
2672999140,14cyclictest0-21swapper/019:25:030
2672999140,14cyclictest0-21swapper/000:20:070
2672999140,0cyclictest0-21swapper/023:20:060
2672999140,0cyclictest0-21swapper/023:15:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional