You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 13:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Fri Nov 21, 2025 00:44:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10213992624,1cyclictest1820-21sh20:00:013
1021399250,24cyclictest22481-21apt22:55:003
10202992521,4cyclictest0-21swapper/019:42:030
1020699240,23cyclictest0-21swapper/123:02:031
1021399230,23cyclictest0-21swapper/300:12:043
1021399230,22cyclictest7030-21awk20:10:003
1020699230,22cyclictest6270-21munin-run00:35:001
1020699230,22cyclictest313-21systemd-journal22:25:011
1020699230,22cyclictest14681-21apt-get20:27:031
1020699230,1cyclictest10019-21dump-pmu-power22:30:011
1020699230,0cyclictest0-21swapper/121:12:051
1020299230,22cyclictest14970-21turbostat19:20:000
10213992220,1cyclictest24648-21apt19:40:003
1021399220,21cyclictest7308-21apt-get23:32:023
1021399220,21cyclictest23864-21turbostat.cron00:05:013
1021399220,21cyclictest20400-21sh21:45:003
1021399220,21cyclictest19381-21apt-get23:57:033
1021399220,21cyclictest14133-21munin-run23:44:593
1021399220,21cyclictest0-21swapper/319:42:033
1020899220,21cyclictest585-21apt-get22:10:022
1020899220,21cyclictest4228-21cron20:05:012
1020899220,21cyclictest0-21swapper/221:32:032
1020699220,21cyclictest20250-21apt-get19:32:031
1020699220,21cyclictest1127-21awk21:05:001
1020699220,21cyclictest10582-21apt-get19:12:011
1020699220,21cyclictest0-21swapper/123:12:031
1020699220,21cyclictest0-21swapper/122:07:031
1020699220,21cyclictest0-21swapper/100:07:071
1020699220,1cyclictest8643-21ls00:40:001
1020699220,1cyclictest20100-21sh22:50:001
10202992221,1cyclictest3874-21awk00:30:000
10202992220,1cyclictest16728-21perf20:30:000
1020299220,2cyclictest28676-21grep00:15:000
1020299220,21cyclictest8687-21sendmail-msp00:40:000
1020299220,21cyclictest31846-21turbostat19:54:590
1020299220,21cyclictest17852-21apt-get19:27:030
1020299220,21cyclictest11726-21munin-run23:40:000
1020299220,21cyclictest0-21swapper/022:12:030
1020299220,21cyclictest0-21swapper/020:17:020
10213992121,0cyclictest0-21swapper/322:16:463
10213992120,1cyclictest13565-21apt-get21:32:023
10213992119,2cyclictest17452-21idleruntime-cro19:25:003
10213992118,2cyclictest11876-21munin-run20:20:003
1021399210,21cyclictest0-21swapper/320:22:023
1021399210,20cyclictest0-21swapper/319:27:023
1021399210,0cyclictest0-21swapper/322:42:083
1021399210,0cyclictest0-21swapper/319:32:023
10208992121,0cyclictest0-21swapper/219:57:032
10208992119,1cyclictest8340-21dump-pmu-power21:20:002
10208992119,1cyclictest593-21dbus-daemon23:40:002
10208992118,2cyclictest5241-21idleruntime-cro22:20:012
1020899210,21cyclictest16016-21apt-get21:37:032
1020899210,21cyclictest0-21swapper/222:02:052
1020899210,20cyclictest29078-21apt-get00:17:022
1020899210,1cyclictest22217-21idleruntime-cro19:34:592
1020899210,1cyclictest19847-21munin-run19:30:002
1020899210,1cyclictest13149-21latency_hist21:30:002
1020899210,1cyclictest0-21swapper/220:55:002
10206992120,1cyclictest20502-21apt-get22:52:051
10206992120,1cyclictest12840-21apt-get22:37:031
10206992120,0cyclictest0-21swapper/121:17:041
10206992119,1cyclictest28675-21munin-run00:14:591
1020699210,21cyclictest0-21swapper/123:27:041
1020699210,20cyclictest23227-21apt-get21:52:051
1020699210,20cyclictest0-21swapper/121:42:021
1020699210,20cyclictest0-21swapper/120:05:011
1020699210,20cyclictest0-21swapper/119:52:011
1020699210,1cyclictest25622-21apt-get21:57:021
1020699210,0cyclictest29138-21apt-get20:57:041
10202992121,0cyclictest10441-21apt-get22:32:030
10202992120,1cyclictest0-21swapper/020:05:020
10202992120,0cyclictest0-21swapper/023:57:030
10202992119,1cyclictest31059-21latency_hist00:20:000
10202992119,1cyclictest1118-21wc21:04:590
10202992119,1cyclictest0-21swapper/023:45:000
1020299210,2cyclictest30009-21cron22:05:000
1020299210,21cyclictest0-21swapper/020:22:020
1020299210,21cyclictest0-21swapper/019:12:000
1020299210,20cyclictest0-21swapper/022:40:010
1020299210,0cyclictest0-21swapper/023:02:030
1020299210,0cyclictest0-21swapper/019:37:030
10213992020,0cyclictest27464-21apt-get19:47:023
10213992020,0cyclictest27464-21apt-get19:47:013
10213992020,0cyclictest2268-21apt-get20:02:033
10213992020,0cyclictest0-21swapper/323:07:043
10213992020,0cyclictest0-21swapper/322:22:043
10213992020,0cyclictest0-21swapper/321:57:023
10213992020,0cyclictest0-21swapper/321:52:053
10213992020,0cyclictest0-21swapper/321:47:033
10213992019,1cyclictest28043-21apt-get22:02:053
10213992019,0cyclictest0-21swapper/300:22:043
10213992018,1cyclictest2086-21ls23:20:003
1021399200,20cyclictest0-21swapper/320:52:033
1021399200,0cyclictest0-21swapper/323:27:053
1021399200,0cyclictest0-21swapper/322:57:053
1021399200,0cyclictest0-21swapper/322:37:023
1021399200,0cyclictest0-21swapper/322:12:033
1021399200,0cyclictest0-21swapper/320:27:033
10208992020,0cyclictest27062-21perf19:45:002
10208992020,0cyclictest0-21swapper/223:57:032
10208992020,0cyclictest0-21swapper/221:57:022
10208992020,0cyclictest0-21swapper/221:12:052
10208992020,0cyclictest0-21swapper/220:40:002
10208992020,0cyclictest0-21swapper/220:17:012
10208992020,0cyclictest0-21swapper/219:52:012
10208992020,0cyclictest0-21swapper/219:47:022
10208992020,0cyclictest0-21swapper/219:47:022
10208992020,0cyclictest0-21swapper/219:17:042
10208992019,1cyclictest7638-21munin-plugin-st22:25:002
10208992019,1cyclictest2516-21apt-get23:22:022
10208992018,1cyclictest0-21swapper/220:12:002
1020899200,20cyclictest6669-21apt-get00:37:022
1020899200,0cyclictest0-21swapper/222:42:092
1020899200,0cyclictest0-21swapper/221:52:052
10206992020,0cyclictest27713-21apt-get23:07:031
10206992020,0cyclictest20829-21apt-get21:47:031
10206992020,0cyclictest12133-21apt-get23:42:031
10206992020,0cyclictest0-21swapper/121:27:091
10206992020,0cyclictest0-21swapper/119:57:031
10206992020,0cyclictest0-21swapper/119:42:041
10206992020,0cyclictest0-21swapper/119:17:041
1020699200,0cyclictest32474-21/usr/sbin/munin23:17:041
10202992020,0cyclictest0-21swapper/023:27:040
10202992020,0cyclictest0-21swapper/022:47:030
10202992020,0cyclictest0-21swapper/022:27:040
10202992020,0cyclictest0-21swapper/022:22:040
10202992020,0cyclictest0-21swapper/019:47:030
10202992020,0cyclictest0-21swapper/019:47:030
10202992019,1cyclictest6343-21apt-get21:17:050
10202992019,0cyclictest0-21swapper/022:17:040
10202992018,1cyclictest15540-21cron21:35:010
1020299200,0cyclictest0-21swapper/023:07:040
1020299200,0cyclictest0-21swapper/020:57:040
1020299200,0cyclictest0-21swapper/020:32:030
10213991919,0cyclictest10582-21apt-get19:12:013
10213991919,0cyclictest0-21swapper/323:22:043
10213991919,0cyclictest0-21swapper/323:12:043
10213991919,0cyclictest0-21swapper/322:07:043
10213991919,0cyclictest0-21swapper/319:52:013
10213991918,0cyclictest14532-21apt-get23:47:013
10213991916,2cyclictest3855-21idleruntime-cro00:30:003
1021399190,0cyclictest0-21swapper/320:42:053
10208991919,0cyclictest0-21swapper/222:57:052
10208991919,0cyclictest0-21swapper/222:37:022
10208991919,0cyclictest0-21swapper/220:27:022
10208991919,0cyclictest0-21swapper/219:22:042
10208991917,1cyclictest32091-21dump-pmu-power23:15:012
10206991919,0cyclictest0-21swapper/122:40:011
10206991919,0cyclictest0-21swapper/122:12:021
10206991919,0cyclictest0-21swapper/120:55:001
10206991919,0cyclictest0-21swapper/120:47:041
10206991919,0cyclictest0-21swapper/120:02:021
10206991919,0cyclictest0-21swapper/100:27:051
10206991919,0cyclictest0-21swapper/100:22:041
10206991918,1cyclictest1532-21apt-get21:07:031
10206991917,1cyclictest15554-21apt21:35:011
10202991919,0cyclictest15405-21apt-get19:22:040
10202991919,0cyclictest0-21swapper/023:32:030
10202991919,0cyclictest0-21swapper/021:42:020
10202991919,0cyclictest0-21swapper/000:02:020
10202991918,1cyclictest612-21gdbus20:00:010
10202991918,1cyclictest0-21swapper/000:25:000
10202991917,1cyclictest26271-21perf00:09:590
10202991917,1cyclictest22227-21cron19:35:000
10213991818,0cyclictest0-21swapper/323:02:043
10213991818,0cyclictest0-21swapper/322:47:043
10213991818,0cyclictest0-21swapper/322:32:023
10213991818,0cyclictest0-21swapper/321:22:043
10213991818,0cyclictest0-21swapper/321:22:043
10213991818,0cyclictest0-21swapper/320:47:033
10213991816,1cyclictest14983-21idleruntime-cro19:20:003
10208991818,0cyclictest25316-21apt-get23:02:032
10208991818,0cyclictest0-21swapper/220:32:042
10208991818,0cyclictest0-21swapper/219:12:012
10208991818,0cyclictest0-21swapper/200:02:032
10208991817,1cyclictest28672-21perf00:15:002
10208991817,1cyclictest24344-21apt-get20:47:042
10206991818,0cyclictest0-21swapper/123:52:061
10206991818,0cyclictest0-21swapper/122:17:031
10206991818,0cyclictest0-21swapper/121:37:031
10206991818,0cyclictest0-21swapper/100:17:011
10206991818,0cyclictest0-21swapper/100:02:031
10206991816,1cyclictest29461-21awk19:50:001
10206991816,1cyclictest29461-21awk19:50:001
10206991816,1cyclictest24661-21perf19:40:001
1020699180,0cyclictest0-21swapper/123:37:061
10202991818,0cyclictest0-21swapper/023:47:020
10202991818,0cyclictest0-21swapper/022:52:050
10202991818,0cyclictest0-21swapper/021:47:030
10202991818,0cyclictest0-21swapper/000:35:000
10213991717,0cyclictest0-21swapper/300:37:033
10213991716,0cyclictest0-21swapper/321:27:093
1021399170,4cyclictest0-21swapper/300:07:063
10208991717,0cyclictest4871-21apt-get20:05:022
10208991717,0cyclictest0-21swapper/223:32:032
10208991717,0cyclictest0-21swapper/222:52:052
10208991717,0cyclictest0-21swapper/221:07:032
10206991717,0cyclictest0-21swapper/120:32:031
10202991717,0cyclictest0-21swapper/023:17:030
10202991717,0cyclictest0-21swapper/022:37:020
10202991717,0cyclictest0-21swapper/022:07:030
10202991717,0cyclictest0-21swapper/020:02:030
10213991616,0cyclictest0-21swapper/323:37:053
10213991616,0cyclictest0-21swapper/321:17:053
10213991614,1cyclictest4112-21grep21:12:063
10208991616,0cyclictest31498-21apt-get00:22:042
10208991616,0cyclictest0-21swapper/223:47:032
10208991616,0cyclictest0-21swapper/219:37:022
1020899160,0cyclictest0-21swapper/223:17:032
10206991614,1cyclictest21530-21munin-run20:40:001
10206991614,1cyclictest0-21swapper/122:35:001
1020699160,7cyclictest0-21swapper/123:22:041
10202991616,0cyclictest0-21swapper/020:37:020
10202991614,1cyclictest27641-21idleruntime-cro22:00:000
10202991614,1cyclictest26332-21basename20:50:010
10202991614,1cyclictest24911-21munin-run23:00:000
10202991614,1cyclictest0-21swapper/023:22:110
1020299160,15cyclictest0-21swapper/023:52:070
10213991515,0cyclictest0-21swapper/320:37:013
10213991514,0cyclictest0-21swapper/320:32:033
10213991513,1cyclictest3530-21perf21:10:003
10208991515,0cyclictest0-21swapper/220:57:042
10208991515,0cyclictest0-21swapper/220:22:012
10208991514,1cyclictest4913-21apt-get23:27:032
10208991514,1cyclictest0-21swapper/222:32:032
1020899150,1cyclictest0-21swapper/200:34:592
1020899150,14cyclictest0-21swapper/223:52:102
10206991515,0cyclictest0-21swapper/120:12:001
10206991515,0cyclictest0-21swapper/119:22:031
10202991515,0cyclictest0-21swapper/021:07:010
1020299150,1cyclictest17195-21awk21:37:100
1020299150,1cyclictest0-21swapper/021:27:090
1020299150,14cyclictest0-21swapper/020:12:070
10213991414,0cyclictest0-21swapper/322:25:383
10213991414,0cyclictest0-21swapper/321:02:123
10213991414,0cyclictest0-21swapper/320:55:003
1021399140,14cyclictest0-21swapper/323:51:373
1021399140,14cyclictest0-21swapper/320:10:343
1021399140,14cyclictest0-21swapper/300:34:133
1021399140,14cyclictest0-21swapper/300:16:153
1021399140,0cyclictest0-21swapper/321:36:163
1020899142,7cyclictest0-21swapper/222:47:042
10208991414,0cyclictest0-21swapper/222:25:182
10208991414,0cyclictest0-21swapper/221:45:032
10208991414,0cyclictest0-21swapper/221:40:022
10208991414,0cyclictest0-21swapper/200:27:042
10208991412,1cyclictest1117-21grep21:05:002
1020899140,14cyclictest0-21swapper/223:40:432
1020899140,14cyclictest0-21swapper/221:24:202
1020899140,14cyclictest0-21swapper/221:24:202
1020899140,14cyclictest0-21swapper/220:41:302
1020899140,14cyclictest0-21swapper/200:06:012
1020899140,0cyclictest0-21swapper/223:06:222
1020899140,0cyclictest0-21swapper/222:07:312
10206991414,0cyclictest0-21swapper/120:22:381
1020699140,14cyclictest0-21swapper/123:56:131
1020699140,0cyclictest0-21swapper/122:55:531
1020699140,0cyclictest0-21swapper/120:41:131
10202991414,0cyclictest0-21swapper/023:10:220
10202991414,0cyclictest0-21swapper/021:52:040
10202991414,0cyclictest0-21swapper/021:22:040
10202991414,0cyclictest0-21swapper/021:22:040
10202991414,0cyclictest0-21swapper/021:11:280
10202991414,0cyclictest0-21swapper/020:40:010
1020299140,0cyclictest0-21swapper/020:52:090
10206991313,0cyclictest0-21swapper/121:22:041
10206991313,0cyclictest0-21swapper/121:22:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional