You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 15:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Thu Nov 20, 2025 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19931992523,1cyclictest19902-21perf21:55:010
19941992320,2cyclictest20582-21idleruntime-cro00:10:003
1994199230,22cyclictest24699-21dump-pmu-power19:20:003
19939992321,1cyclictest31217-21perf23:25:002
1993199230,22cyclictest4517-21sleep20:00:000
19941992219,2cyclictest27105-21turbostat19:25:013
1994199220,21cyclictest3238-21perf19:50:003
1994199220,21cyclictest0-21swapper/323:36:573
1994199220,21cyclictest0-21swapper/321:46:563
1994199220,21cyclictest0-21swapper/300:10:023
1994199220,21cyclictest0-21swapper/300:01:593
19939992220,1cyclictest15675-21turbostat00:00:002
19939992220,1cyclictest1-21systemd23:00:002
19939992220,1cyclictest0-21swapper/221:19:592
1993999220,2cyclictest27845-21perf21:05:012
1993999220,21cyclictest25519-21taskset22:06:562
1993999220,21cyclictest0-21swapper/223:51:572
1993999220,21cyclictest0-21swapper/221:22:002
1993999220,21cyclictest0-21swapper/220:31:592
1993999220,21cyclictest0-21swapper/219:19:592
1993999220,20cyclictest313-21systemd-journal22:55:002
1993999220,1cyclictest313-21systemd-journal19:40:002
1993999220,1cyclictest26400-21munin-run23:15:002
1993999220,1cyclictest2590-21cron19:45:002
1993999220,1cyclictest22287-21perf19:15:002
1993999220,1cyclictest20561-21cron20:50:002
19934992221,1cyclictest20566-21apt-get00:10:021
19934992220,1cyclictest2380-21latency_hist22:25:001
1993499220,21cyclictest583-21avahi-daemon00:00:001
1993499220,21cyclictest5483-21dump-pmu-power21:25:001
1993499220,21cyclictest0-21swapper/123:41:571
1993499220,21cyclictest0-21swapper/123:21:571
1993499220,1cyclictest211rcuc/119:14:591
1993499220,1cyclictest0-21swapper/123:27:001
1993499220,1cyclictest0-21swapper/121:00:001
19931992220,1cyclictest28547-21sh00:25:000
1993199220,22cyclictest0-21swapper/021:02:010
1993199220,21cyclictest3504-21apt-get21:22:000
1993199220,21cyclictest18575-21apt-get20:46:590
1993199220,21cyclictest0-21swapper/022:51:560
1993199220,21cyclictest0-21swapper/020:54:590
1993199220,0cyclictest0-21swapper/021:11:580
19941992121,0cyclictest0-21swapper/323:46:573
19941992120,0cyclictest0-21swapper/321:06:583
19941992116,4cyclictest0-21swapper/300:22:013
1994199210,20cyclictest29213-21apt-get23:21:563
1994199210,20cyclictest26793-21apt-get23:16:583
1994199210,20cyclictest0-21swapper/323:51:563
1994199210,20cyclictest0-21swapper/322:51:573
1994199210,20cyclictest0-21swapper/322:32:003
1994199210,20cyclictest0-21swapper/300:36:593
1994199210,1cyclictest313-21systemd-journal00:35:003
1994199210,1cyclictest11979-21latency_hist22:45:013
1994199210,0cyclictest7569-21apt-get22:36:573
1994199210,0cyclictest0-21swapper/321:51:563
1994199210,0cyclictest0-21swapper/321:26:543
1994199210,0cyclictest0-21swapper/320:26:523
19939992120,1cyclictest23999-21perf23:10:002
19939992120,0cyclictest0-21swapper/220:51:592
1993999210,20cyclictest5890-21apt-get21:26:552
1993999210,20cyclictest30309-21apt-get22:16:562
1993999210,20cyclictest0-21swapper/223:36:572
1993999210,20cyclictest0-21swapper/223:01:582
1993999210,20cyclictest0-21swapper/222:36:572
1993999210,20cyclictest0-21swapper/222:26:572
1993999210,0cyclictest0-21swapper/222:31:592
19934992121,0cyclictest0-21swapper/122:05:011
19934992120,1cyclictest13760-21apt-get20:36:571
19934992120,1cyclictest11363-21apt-get20:32:001
19934992120,0cyclictest29923-21apt-get19:32:011
19934992119,1cyclictest14390-21awk22:50:001
1993499210,21cyclictest0-21swapper/122:36:571
1993499210,20cyclictest4049-21apt-get23:36:571
1993499210,20cyclictest0-21swapper/100:17:021
1993499210,1cyclictest28969-21apt-get00:26:581
1993499210,0cyclictest11261-21apt-get23:51:571
1993499210,0cyclictest0-21swapper/123:16:571
19931992121,0cyclictest1065-21apt-get21:16:570
19931992119,1cyclictest26399-21perf23:15:000
19931992119,1cyclictest0-21swapper/020:05:000
1993199210,20cyclictest0-21swapper/021:57:000
1993199210,0cyclictest0-21swapper/000:01:580
19941992020,0cyclictest20309-21apt-get21:57:003
19941992020,0cyclictest0-21swapper/323:10:003
19941992020,0cyclictest0-21swapper/321:21:593
19941992020,0cyclictest0-21swapper/320:56:583
19941992018,1cyclictest8469-21munin-run23:44:593
1994199200,0cyclictest0-21swapper/320:22:063
19939992020,0cyclictest0-21swapper/223:46:582
19939992020,0cyclictest0-21swapper/223:41:572
19939992020,0cyclictest0-21swapper/219:32:012
19939992019,1cyclictest17469-21cron21:50:012
19939992019,1cyclictest13757-21hddtemp_smartct21:42:012
1993999200,0cyclictest31364-21apt-get00:31:572
1993999200,0cyclictest0-21swapper/222:11:552
19934992020,0cyclictest16172-21apt-get00:01:581
19934992020,0cyclictest13108-21apt-get21:41:591
19934992020,0cyclictest0-21swapper/122:41:551
19934992020,0cyclictest0-21swapper/122:01:561
19934992020,0cyclictest0-21swapper/121:36:581
19934992020,0cyclictest0-21swapper/121:31:571
19934992020,0cyclictest0-21swapper/119:26:581
19934992019,1cyclictest20566-21sh20:50:001
19934992018,1cyclictest313-21systemd-journal22:30:001
19934992017,2cyclictest22286-21latency_hist22:00:001
1993499200,20cyclictest0-21swapper/122:11:561
1993499200,0cyclictest0-21swapper/121:05:011
1993499200,0cyclictest0-21swapper/119:36:591
1993499200,0cyclictest0-21swapper/119:16:571
19931992020,0cyclictest23428-21apt-get20:56:560
19931992020,0cyclictest0-21swapper/023:06:580
19931992020,0cyclictest0-21swapper/022:11:560
19931992020,0cyclictest0-21swapper/020:36:570
19931992020,0cyclictest0-21swapper/019:37:000
19931992020,0cyclictest0-21swapper/019:16:570
19931992019,1cyclictest5805-21awk20:10:000
19931992018,1cyclictest13247-21cron23:55:000
1993199200,20cyclictest0-21swapper/023:36:570
1993199200,20cyclictest0-21swapper/000:26:580
1993199200,20cyclictest0-21swapper/000:06:560
1993199200,0cyclictest0-21swapper/023:27:000
1993199200,0cyclictest0-21swapper/021:41:580
1993199200,0cyclictest0-21swapper/020:26:520
1993199200,0cyclictest0-21swapper/019:22:000
19941991919,0cyclictest20303-21apt-get19:11:573
19941991919,0cyclictest0-21swapper/323:01:583
19941991919,0cyclictest0-21swapper/322:56:563
19941991919,0cyclictest0-21swapper/321:41:593
19941991919,0cyclictest0-21swapper/321:16:573
19941991919,0cyclictest0-21swapper/321:02:013
19941991919,0cyclictest0-21swapper/320:46:583
19941991917,1cyclictest7073-21perf20:20:013
19941991917,1cyclictest5782-21dump-pmu-power20:10:003
19939991919,0cyclictest0-21swapper/222:01:552
19939991918,1cyclictest1648-21apt-get23:31:562
19939991917,1cyclictest27833-21apt-get21:05:022
19939991917,1cyclictest0-21swapper/200:06:572
1993999190,0cyclictest0-21swapper/221:57:002
19934991919,0cyclictest0-21swapper/123:01:581
19934991917,1cyclictest3209-21cron19:50:001
19931991919,0cyclictest6469-21apt-get23:41:570
19931991919,0cyclictest0-21swapper/021:05:020
19931991919,0cyclictest0-21swapper/020:22:060
19931991919,0cyclictest0-21swapper/019:32:010
19931991917,1cyclictest593-21dbus-daemon20:20:010
19941991818,0cyclictest29923-21apt-get19:31:593
19941991818,0cyclictest0-21swapper/323:27:003
19941991818,0cyclictest0-21swapper/321:11:583
19941991818,0cyclictest0-21swapper/320:42:003
19941991818,0cyclictest0-21swapper/300:17:033
19941991816,1cyclictest0-21swapper/319:55:003
1994199180,4cyclictest5332-21cut20:05:013
1994199180,2cyclictest0-21swapper/321:37:003
19939991818,0cyclictest16181-21apt-get20:42:002
19939991818,0cyclictest0-21swapper/223:16:582
19939991818,0cyclictest0-21swapper/221:11:592
19939991817,1cyclictest7502-21apt-get20:22:052
19939991817,1cyclictest5163-21perf20:05:002
19934991818,0cyclictest0-21swapper/123:46:581
19934991818,0cyclictest0-21swapper/121:02:001
19934991818,0cyclictest0-21swapper/120:41:591
19934991817,0cyclictest25125-21apt-get19:22:001
19934991817,0cyclictest0-21swapper/123:12:001
19934991816,1cyclictest7875-21latency_hist21:30:001
1993499180,17cyclictest0-21swapper/120:26:521
19931991818,0cyclictest9983-21apt-get22:41:540
19931991818,0cyclictest0-21swapper/021:26:550
19931991818,0cyclictest0-21swapper/000:17:030
19931991817,1cyclictest27520-21apt-get19:26:580
19931991816,1cyclictest2771-21apt22:26:580
1993199180,10cyclictest313-21systemd-journal23:00:000
19941991717,0cyclictest0-21swapper/323:31:583
19941991717,0cyclictest0-21swapper/321:31:583
19941991717,0cyclictest0-21swapper/320:32:003
19941991715,1cyclictest0-21swapper/322:15:003
19941991713,3cyclictest0-21swapper/322:16:583
1994199170,16cyclictest22819-21ls22:01:573
19939991717,0cyclictest0-21swapper/200:22:012
19939991716,1cyclictest15778-21munin-run20:40:002
19939991716,0cyclictest0-21swapper/223:26:592
19939991714,3cyclictest12677-21awk21:40:002
19934991717,0cyclictest1340-21apt-get00:36:591
19934991717,0cyclictest0-21swapper/122:56:561
19934991717,0cyclictest0-21swapper/122:31:591
19934991717,0cyclictest0-21swapper/121:16:561
19934991717,0cyclictest0-21swapper/100:31:581
19934991717,0cyclictest0-21swapper/100:06:571
1993499170,16cyclictest31237-21cut21:11:591
19931991717,0cyclictest32731-21apt-get22:21:590
19931991717,0cyclictest19596-21apt-get23:01:580
19931991717,0cyclictest0-21swapper/023:31:580
19931991715,1cyclictest22275-21idleruntime-cro19:15:000
19941991616,0cyclictest32319-21apt-get19:37:003
19941991616,0cyclictest22845-21apt-config20:53:323
19941991615,0cyclictest25333-21apt-get22:05:013
19941991614,1cyclictest0-21swapper/322:27:043
19941991611,4cyclictest0-21swapper/323:57:003
1994199160,6cyclictest0-21swapper/320:36:563
1994199160,15cyclictest12568-21cpuspeed_turbos22:46:593
19939991616,0cyclictest0-21swapper/200:01:582
1993999160,1cyclictest0-21swapper/220:15:012
1993999160,15cyclictest1535-21cut00:36:592
1993999160,15cyclictest0-21swapper/219:27:012
19934991614,1cyclictest0-21swapper/122:51:581
1993499160,2cyclictest16804-21rt-features21:47:021
1993499160,15cyclictest26765-21tr00:22:011
1993499160,15cyclictest0-21swapper/122:17:041
1993499160,15cyclictest0-21swapper/121:51:561
19931991616,0cyclictest15503-21apt-get21:46:560
1993199160,15cyclictest61550irq/126-eno100:11:530
1994199150,14cyclictest0-21swapper/322:21:583
19939991515,0cyclictest24162-21apt-get00:17:032
19939991515,0cyclictest17897-21apt-get21:51:552
19939991515,0cyclictest0-21swapper/200:10:022
19939991513,1cyclictest4533-21sendmail-msp20:00:002
19939991513,1cyclictest0-21swapper/222:42:012
19939991512,2cyclictest313-21systemd-journal19:55:002
1993999150,1cyclictest0-21swapper/219:22:082
1993999150,14cyclictest23428-21apt-get20:56:582
1993999150,14cyclictest0-21swapper/220:19:162
1993499150,1cyclictest8410-21if_enp1s0.820:24:201
19931991515,0cyclictest0-21swapper/020:32:000
19931991515,0cyclictest0-21swapper/000:36:590
19931991512,2cyclictest28434-21tail23:17:070
1993199150,1cyclictest8369-21irqstats22:37:010
19941991414,0cyclictest0-21swapper/320:14:403
1994199140,14cyclictest0-21swapper/323:14:223
1994199140,14cyclictest0-21swapper/319:57:023
1994199140,14cyclictest0-21swapper/319:40:353
1994199140,0cyclictest0-21swapper/300:26:233
19939991414,0cyclictest0-21swapper/222:21:582
19939991414,0cyclictest0-21swapper/220:09:472
1993999140,14cyclictest0-21swapper/200:26:052
1993999140,0cyclictest0-21swapper/222:45:492
1993999140,0cyclictest0-21swapper/221:31:212
1993999140,0cyclictest0-21swapper/219:45:172
19934991414,0cyclictest0-21swapper/119:51:081
1993499140,14cyclictest0-21swapper/123:30:151
1993499140,14cyclictest0-21swapper/123:07:341
1993499140,14cyclictest0-21swapper/120:10:501
1993499140,0cyclictest0-21swapper/120:08:121
1993499140,0cyclictest0-21swapper/119:56:351
1993499140,0cyclictest0-21swapper/119:40:341
19931991414,0cyclictest0-21swapper/022:05:070
19931991414,0cyclictest0-21swapper/021:32:040
19931991414,0cyclictest0-21swapper/019:54:410
19931991414,0cyclictest0-21swapper/000:30:260
1993199140,14cyclictest0-21swapper/023:21:310
1993199140,14cyclictest0-21swapper/022:17:340
1993199140,14cyclictest0-21swapper/022:00:450
1993199140,14cyclictest0-21swapper/020:12:400
1993199140,14cyclictest0-21swapper/019:40:340
1993199140,0cyclictest0-21swapper/023:55:140
1993199140,0cyclictest0-21swapper/023:45:450
1993199140,0cyclictest0-21swapper/022:45:440
1993199140,0cyclictest0-21swapper/019:45:400
19931991313,0cyclictest0-21swapper/020:41:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional