You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 23:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Thu Dec 04, 2025 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11147129910080,13cyclictest1114715-21grep07:10:020
11147129910080,13cyclictest1114715-21grep07:10:020
111471299752,68cyclictest1116241-21sshd07:38:220
1114717997020,8cyclictest1121192-21cron09:14:591
1114712997031,35cyclictest306-21in:imuxsock08:35:010
1114712997023,42cyclictest1122509-21run-parts09:40:000
1114726996936,29cyclictest235-21systemd-journal12:10:003
111471799695,20cyclictest1119385-21munin-run08:40:011
1114717996925,4cyclictest1130474-21latency_hist12:10:021
111472699661,12cyclictest1125858-21sshd10:40:253
1114712996625,36cyclictest272-21dbus-daemon11:00:010
1114726996557,6cyclictest1119431-21kworker/3:208:50:023
111472699651,60cyclictest1127407-21sshd11:11:283
1114717996529,31cyclictest1115449-21sh07:25:011
111471299651,59cyclictest0-21swapper/007:27:230
111472699642,58cyclictest1123863-21cat10:05:023
1114712996425,34cyclictest1129814-21latency_hist12:00:010
111471299642,13cyclictest1131835-21cron12:40:000
111472699633,55cyclictest363-21snmpd09:53:203
111472699633,55cyclictest363-21snmpd09:53:203
1114726996328,31cyclictest1123630-21latency_hist10:00:013
111472699632,57cyclictest1117040-21munin-run07:55:003
111472699632,56cyclictest1131891-21munin-run12:40:013
111472699631,58cyclictest1130536-21sshd12:14:243
111472699631,58cyclictest1122221-21sshd09:31:253
111472699631,58cyclictest1122221-21sshd09:31:243
111472699631,57cyclictest1124537-21sshd10:15:253
111472699631,57cyclictest1124537-21sshd10:15:253
111472699630,58cyclictest1118317-21sshd08:18:213
1114717996323,6cyclictest1118616-21latency_hist08:25:021
1114717996323,6cyclictest1118616-21latency_hist08:25:021
111471299633,19cyclictest1130301-21munin-run12:10:000
1114712996324,36cyclictest1127792-21latency_hist11:20:020
111472699622,56cyclictest1126908-21sshd11:03:253
111472699622,55cyclictest1128240-21munin-run11:30:013
111472699622,55cyclictest1125112-21munin-run10:29:593
111472699622,54cyclictest1127183-21munin-run11:10:003
111472699621,56cyclictest1117539-21sshd08:00:463
111472699621,55cyclictest1124847-21cron10:25:003
1114717996225,7cyclictest23-21ksoftirqd/111:30:021
1114717996223,6cyclictest306-21in:imuxsock09:10:011
1114712996241,16cyclictest1-21systemd08:45:000
111472699613,54cyclictest1122473-21sshd09:36:253
111472699612,56cyclictest1126161-21munin-run10:50:013
111472699612,56cyclictest1126161-21munin-run10:50:013
111472699612,54cyclictest363-21snmpd11:54:193
111472699612,54cyclictest1128702-21sshd11:36:313
111472699612,54cyclictest1123035-21munin-run09:50:013
111472699612,54cyclictest1120703-21munin-run09:05:013
111472699612,54cyclictest1118883-21munin-run08:30:013
111472699611,55cyclictest1130855-21munin-run12:20:013
111472699611,55cyclictest1115454-21munin-run07:25:013
111471799613,19cyclictest1125039-21latency_hist10:25:021
1114717996121,14cyclictest1125602-21cron10:40:001
111471299613,15cyclictest1123545-21sendmail10:00:000
111471299612,15cyclictest1120403-21modprobe09:00:000
1114712996121,3cyclictest0-21swapper/008:27:230
1114712996113,12cyclictest12-21ksoftirqd/009:15:000
111472699602,54cyclictest1127673-21sshd11:17:253
111472699602,53cyclictest1125598-21sshd10:39:253
111472699602,53cyclictest1124319-21munin-run10:15:003
111472699601,55cyclictest363-21snmpd12:28:513
111472699601,55cyclictest363-21snmpd12:28:513
111472699601,54cyclictest363-21snmpd11:58:163
111472699601,54cyclictest1125332-21sshd10:31:223
111472699601,54cyclictest1124070-21munin-run10:10:013
111472699601,54cyclictest1120159-21munin-run08:55:013
111472699601,54cyclictest1119620-21sshd08:40:223
111471799607,11cyclictest235-21systemd-journal07:30:001
111471799604,16cyclictest1129776-21grep12:00:001
111471799603,17cyclictest1126931-21cut11:05:011
111471299603,12cyclictest1119386-21grep08:40:010
1114712996025,31cyclictest12-21ksoftirqd/007:20:000
1114712996021,5cyclictest1128023-21latency_hist11:25:010
1114712996013,9cyclictest1117734-21cut08:05:020
111472699592,53cyclictest363-21snmpd08:58:093
111472699591,54cyclictest1127945-21sshd11:21:253
111472699591,53cyclictest363-21snmpd11:33:403
111472699591,53cyclictest363-21snmpd07:55:223
111472699591,53cyclictest1131077-21sshd12:24:243
111472699591,53cyclictest1121173-21sshd09:11:143
111472699591,53cyclictest1119418-21munin-run08:40:013
1114717995922,8cyclictest23-21ksoftirqd/109:53:201
1114717995922,8cyclictest23-21ksoftirqd/109:53:201
1114712995932,23cyclictest1-21systemd07:35:010
1114712995925,30cyclictest1123084-21latency_hist09:50:020
1114712995922,9cyclictest12-21ksoftirqd/010:33:250
111471299592,17cyclictest1124306-21cron10:15:000
111471299591,18cyclictest1129402-21latency_hist11:50:020
1114712995911,7cyclictest1116551-21systemd-run07:45:010
1114712995911,7cyclictest1116551-21systemd-run07:45:000
111472699582,52cyclictest1114876-21sshd07:10:373
111472699581,52cyclictest363-21snmpd09:15:313
111472699581,52cyclictest363-21snmpd09:15:303
111472699581,52cyclictest363-21snmpd08:13:003
111472699581,52cyclictest1115668-21sshd07:25:223
111471799583,13cyclictest1124111-21latency_hist10:10:021
1114712995813,12cyclictest272-21dbus-daemon11:45:010
111472699572,51cyclictest363-21snmpd11:40:033
111472699572,51cyclictest1122781-21munin-run09:45:003
111472699572,50cyclictest363-21snmpd12:00:133
111472699572,50cyclictest1118584-21modprobe08:20:223
111472699572,50cyclictest1118584-21modprobe08:20:223
111472699572,49cyclictest1126660-21munin-run11:00:013
1114726995721,32cyclictest1122175-21latency_hist09:30:023
111472699571,5cyclictest1115921-21sshd07:30:233
111472699571,52cyclictest1131566-21sshd12:31:243
111472699571,51cyclictest1115311-21cat07:20:013
111471799573,20cyclictest1123956-21grep10:05:011
111471299573,15cyclictest1121744-21modprobe09:25:000
111471299572,20cyclictest1131349-21munin-plugin-st12:30:000
111471299572,20cyclictest1131349-21munin-plugin-st12:30:000
111471299572,17cyclictest1128404-21grep11:30:020
1114712995713,9cyclictest0-21swapper/011:15:000
111472699563,49cyclictest363-21snmpd08:09:473
111472699561,51cyclictest1129238-21sshd11:47:243
111472699561,51cyclictest1121716-21sshd09:21:253
111472699561,51cyclictest1116762-21sshd07:46:223
111472199561,35cyclictest150-21pr/ttyS207:55:022
111471799565,11cyclictest1121747-21cut09:25:001
111471799564,17cyclictest1120752-21wc09:05:021
111471799564,12cyclictest1131347-21munin-plugin-st12:30:001
111471799564,12cyclictest1131347-21munin-plugin-st12:30:001
1114717995629,23cyclictest23-21ksoftirqd/112:15:021
1114717995622,13cyclictest1-21systemd07:15:001
111471799562,19cyclictest1129404-21latency_hist11:50:021
1114717995620,9cyclictest23-21ksoftirqd/111:20:011
111471299563,13cyclictest1121461-21modprobe09:20:000
111471299563,13cyclictest1121461-21modprobe09:19:590
111471299563,12cyclictest1117220-21wc07:55:010
111471299562,5cyclictest1119994-21latency_hist08:50:020
111472699551,50cyclictest363-21snmpd09:05:073
1114721995511,5cyclictest131rcu_preempt09:05:022
111471799556,11cyclictest1131837-21cron12:40:001
111471799553,15cyclictest1120404-21modprobe09:00:001
111471799553,11cyclictest272-21dbus-daemon09:40:001
1114717995525,7cyclictest23-21ksoftirqd/112:35:011
1114712995520,10cyclictest1116811-21wc07:50:010
1114712995512,7cyclictest306-21in:imuxsock09:05:030
111472699541,49cyclictest1114854-21cat07:10:033
111472699541,49cyclictest1114854-21cat07:10:023
1114721995425,10cyclictest30-21ksoftirqd/209:20:012
1114721995425,10cyclictest30-21ksoftirqd/209:20:012
111471799546,14cyclictest1117218-21cat07:55:021
111471799543,20cyclictest1-21systemd07:45:011
111471799543,20cyclictest1-21systemd07:45:011
1114717995423,9cyclictest23-21ksoftirqd/108:15:001
1114717995415,12cyclictest1-21systemd08:00:011
1114717995411,10cyclictest1123532-21modprobe10:00:001
111471299546,43cyclictest0-21swapper/011:40:000
1114712995434,14cyclictest1-21systemd10:05:010
111471299542,19cyclictest1130800-21cron12:20:000
111471299542,15cyclictest1124110-21latency_hist10:10:020
111472699532,46cyclictest363-21snmpd07:38:073
1114717995324,4cyclictest412-21sshd09:33:251
1114717995324,4cyclictest412-21sshd09:33:251
111471299533,11cyclictest1126929-21munin-run11:05:010
1114712995326,23cyclictest1122907-21cat09:45:020
111472699521,47cyclictest363-21snmpd10:50:133
1114721995234,13cyclictest235-21systemd-journal11:55:002
1114721995216,6cyclictest131rcu_preempt10:03:252
111471799524,11cyclictest235-21systemd-journal09:20:001
111471799524,11cyclictest235-21systemd-journal09:20:001
1114717995237,12cyclictest1-21systemd10:35:011
111471299522,20cyclictest1123288-21munin-plugin-st09:55:000
111471299522,20cyclictest1123288-21munin-plugin-st09:55:000
1114712995212,10cyclictest1124710-21wc10:20:020
1114712995212,10cyclictest1124710-21wc10:20:020
111472199514,11cyclictest30-21ksoftirqd/212:25:012
1114721995138,8cyclictest306-21in:imuxsock09:15:002
1114721995118,8cyclictest30-21ksoftirqd/211:44:252
111472199511,7cyclictest1116267-21modprobe07:40:002
111471799512,16cyclictest1115302-21latency_hist07:20:021
111471299512,19cyclictest0-21swapper/010:25:010
111471299511,16cyclictest1118641-21grep08:25:020
111471299511,16cyclictest1118641-21grep08:25:020
111472699501,44cyclictest363-21snmpd08:32:063
111472199506,9cyclictest30-21ksoftirqd/211:40:022
1114717995032,13cyclictest272-21dbus-daemon11:35:011
111471799503,13cyclictest1115961-21grep07:35:001
111471799503,12cyclictest1129516-21munin-plugin-st11:55:001
111471299503,17cyclictest1118348-21munin-run08:19:590
1114712995018,5cyclictest1118074-21latency_hist08:15:030
1114712995015,11cyclictest1129555-21cat11:55:010
111472699491,3cyclictest363-21snmpd07:42:543
111472699491,3cyclictest363-21snmpd07:42:533
111471799497,23cyclictest0-21swapper/108:50:011
1114717994933,12cyclictest235-21systemd-journal09:50:001
111472199484,37cyclictest1115183-21cut07:20:002
1114721994829,15cyclictest1-21systemd12:20:002
111472199481,18cyclictest0-21swapper/211:35:012
1114712994830,14cyclictest272-21dbus-daemon09:05:010
111472199476,8cyclictest30-21ksoftirqd/211:00:012
1114712994739,5cyclictest16950irq/224-800000012:25:000
111471299471,27cyclictest0-21swapper/007:10:540
111472199464,9cyclictest30-21ksoftirqd/212:40:002
111472199464,9cyclictest30-21ksoftirqd/207:25:002
111472199462,39cyclictest235-21systemd-journal10:25:002
111471299461,4cyclictest0-21swapper/010:46:010
111471299461,4cyclictest0-21swapper/010:46:000
111472199456,10cyclictest30-21ksoftirqd/209:25:002
1114721994535,8cyclictest30-21ksoftirqd/210:54:592
1114721994533,7cyclictest306-21in:imuxsock11:25:012
111471799455,35cyclictest1131153-21cat12:25:011
111471299451,28cyclictest0-21swapper/008:07:100
111472199443,4cyclictest131rcu_preempt11:10:252
1114721994434,7cyclictest30-21ksoftirqd/209:10:012
1114721994431,9cyclictest291rcuc/208:00:012
1114721994426,13cyclictest272-21dbus-daemon07:35:022
1114721994424,16cyclictest30-21ksoftirqd/208:35:012
111471799448,6cyclictest131rcu_preempt10:45:021
1114717994434,7cyclictest1120158-21modprobe08:55:011
111471299442,18cyclictest1117303-21systemctl08:00:010
1114712994415,20cyclictest16850irq/134-800000009:25:250
111472199438,6cyclictest131rcu_preempt10:50:022
111472199438,6cyclictest131rcu_preempt10:50:022
111472199434,8cyclictest30-21ksoftirqd/209:30:012
111472199432,7cyclictest30-21ksoftirqd/208:15:032
1114721994326,12cyclictest235-21systemd-journal10:40:002
1114721994313,5cyclictest131rcu_preempt08:55:012
111471799437,6cyclictest131rcu_preempt09:29:251
1114712994316,19cyclictest16850irq/134-800000012:33:520
111472199427,9cyclictest30-21ksoftirqd/211:20:012
111472199425,8cyclictest30-21ksoftirqd/208:38:062
1114721994232,8cyclictest30-21ksoftirqd/208:50:012
1114721994224,13cyclictest272-21dbus-daemon09:39:592
1114712994222,3cyclictest0-21swapper/011:33:190
111471299421,22cyclictest0-21swapper/009:33:400
111471299421,22cyclictest0-21swapper/009:33:400
111472199418,5cyclictest131rcu_preempt07:40:022
111472199418,5cyclictest131rcu_preempt07:40:012
1114721994132,6cyclictest30-21ksoftirqd/207:30:012
1114721994125,11cyclictest235-21systemd-journal08:05:012
1114717994125,12cyclictest1-21systemd07:35:041
1114712994133,5cyclictest16950irq/224-800000010:54:500
1114712994129,9cyclictest235-21systemd-journal10:40:000
1114712994121,4cyclictest0-21swapper/010:44:520
111471299411,26cyclictest0-21swapper/007:24:550
111472199402,7cyclictest30-21ksoftirqd/209:51:192
111472199402,7cyclictest30-21ksoftirqd/209:51:192
1114721994026,11cyclictest1130521-21sshd12:11:242
1114721994025,12cyclictest30-21ksoftirqd/207:15:012
1114721994011,6cyclictest131rcu_preempt09:59:592
111471799408,3cyclictest131rcu_preempt12:20:011
1114717994028,8cyclictest1-21systemd10:55:001
1114717994025,12cyclictest272-21dbus-daemon11:00:011
111472199399,5cyclictest131rcu_preempt10:15:012
111472199399,5cyclictest131rcu_preempt07:10:022
111472199399,5cyclictest131rcu_preempt07:10:012
111472199395,7cyclictest131rcu_preempt12:35:012
1114721993929,7cyclictest1120386-21sshd08:57:252
1114721993927,9cyclictest30-21ksoftirqd/208:30:012
1114721993927,8cyclictest30-21ksoftirqd/210:10:012
111472199391,7cyclictest1-21systemd12:10:012
1114717993930,6cyclictest1127421-21cron11:15:001
1114712993923,4cyclictest0-21swapper/010:28:090
111472199387,5cyclictest131rcu_preempt12:00:012
1114721993825,8cyclictest306-21in:imuxsock11:28:252
1114721993821,13cyclictest272-21dbus-daemon08:20:002
111472199381,6cyclictest0-21swapper/208:25:022
111472199381,6cyclictest0-21swapper/208:25:012
1114717993829,6cyclictest23-21ksoftirqd/108:05:021
111471299381,25cyclictest0-21swapper/012:03:400
111472199374,5cyclictest131rcu_preempt10:30:012
1114721993728,6cyclictest30-21ksoftirqd/212:30:022
1114721993728,6cyclictest30-21ksoftirqd/212:30:022
111472199371,8cyclictest0-21swapper/211:04:142
111472199371,5cyclictest0-21swapper/208:06:192
111471299373,29cyclictest235-21systemd-journal11:06:420
111471299371,3cyclictest131rcu_preempt12:15:010
111472199368,5cyclictest131rcu_preempt10:20:022
111472199368,5cyclictest131rcu_preempt10:20:022
1114721993626,8cyclictest291rcuc/211:10:022
1114721993625,9cyclictest30-21ksoftirqd/209:50:012
1114721993620,12cyclictest235-21systemd-journal10:35:002
111471799362,5cyclictest131rcu_preempt10:50:021
111471799362,5cyclictest131rcu_preempt10:50:021
1114717993624,9cyclictest272-21dbus-daemon10:20:001
1114717993624,9cyclictest272-21dbus-daemon10:20:001
1114717993624,8cyclictest23-21ksoftirqd/111:40:001
1114717993621,11cyclictest23-21ksoftirqd/108:20:001
1114717993620,4cyclictest1116805-21latency_hist07:50:001
111472199359,5cyclictest131rcu_preempt07:46:222
111472199352,8cyclictest30-21ksoftirqd/212:05:012
1114721993524,8cyclictest30-21ksoftirqd/209:43:562
1114721993520,11cyclictest235-21systemd-journal08:45:002
1114717993529,4cyclictest1-21systemd08:45:011
111471799352,6cyclictest0-21swapper/108:10:021
1114717993523,10cyclictest23-21ksoftirqd/110:15:011
1114712993529,3cyclictest0-21swapper/008:51:120
1114721993422,9cyclictest30-21ksoftirqd/211:50:022
1114721993422,9cyclictest30-21ksoftirqd/210:45:012
111471799342,5cyclictest131rcu_preempt11:43:251
111471799342,5cyclictest131rcu_preempt07:10:031
111471799342,3cyclictest131rcu_preempt11:07:251
1114717993423,8cyclictest23-21ksoftirqd/109:45:021
1114721993316,13cyclictest30-21ksoftirqd/209:35:002
1114721993316,13cyclictest30-21ksoftirqd/209:35:002
111471799333,4cyclictest131rcu_preempt12:05:001
1114717993326,4cyclictest1127972-21latency_hist11:25:021
1114717993326,4cyclictest1125108-21latency_hist10:30:021
111471799301,5cyclictest131rcu_preempt07:10:021
111471799301,3cyclictest131rcu_preempt08:32:221
1114717992917,9cyclictest1-21systemd08:25:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional