You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-27 - 13:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Mon Oct 27, 2025 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3270999280,1cyclictest0-21swapper/319:25:003
32709992620,6cyclictest2792-21perf20:19:593
32709992519,5cyclictest0-21swapper/320:29:533
32708992522,2cyclictest32642-21ptp4l21:20:012
3270399250,24cyclictest9609-21cron21:40:000
32709992422,1cyclictest0-21swapper/300:15:003
3270899242,2cyclictest14053-21ptp4l-jitter22:54:592
32707992423,1cyclictest30920-21apt-get21:19:541
3270799240,23cyclictest474-21cron20:15:011
32703992423,1cyclictest0-21swapper/023:14:520
32709992322,1cyclictest0-21swapper/320:44:523
32709992321,1cyclictest0-21swapper/322:55:003
3270999230,1cyclictest8465-21basename23:50:013
32707992323,0cyclictest1558-21apt-get19:14:511
3270799230,22cyclictest17551-21munin-plugin-st20:50:001
32703992322,1cyclictest141rcuc/019:44:590
32703992321,1cyclictest0-21swapper/022:30:000
32709992222,0cyclictest0-21swapper/319:54:543
32709992221,0cyclictest28194-21apt-get22:19:513
32709992218,3cyclictest0-21swapper/322:59:543
3270999220,21cyclictest4155-21cron22:35:003
3270999220,21cyclictest306172sleep322:24:553
3270999220,21cyclictest24847-21perf21:05:003
32708992220,1cyclictest0-21swapper/221:59:512
32708992219,2cyclictest0-21swapper/220:44:512
3270899221,21cyclictest0-21swapper/220:49:522
3270899220,21cyclictest8777-21apt-get19:29:512
3270899220,21cyclictest27760-21sh20:05:002
3270899220,21cyclictest0-21swapper/221:34:532
3270899220,21cyclictest0-21swapper/220:34:542
3270899220,21cyclictest0-21swapper/200:29:512
3270899220,21cyclictest0-21swapper/200:19:532
32707992221,1cyclictest3432-21idleruntime-cro23:39:591
32707992221,0cyclictest0-21swapper/121:14:541
32707992220,1cyclictest22-21ksoftirqd/121:40:001
3270799220,22cyclictest10023-21perf20:35:001
3270799220,21cyclictest6383-21apt-get19:24:521
3270799220,21cyclictest21602-21apt-get00:19:521
3270799220,21cyclictest17954-21cron00:10:001
3270799220,21cyclictest0-21swapper/123:04:531
3270799220,1cyclictest15071-21munin-run19:40:001
32703992221,1cyclictest25787-21awk23:20:000
32703992220,1cyclictest7790-21ptp4l-jitter20:29:590
3270399220,22cyclictest0-21swapper/023:24:520
3270399220,21cyclictest593-21dbus-daemon21:00:010
3270399220,21cyclictest0-21swapper/020:19:540
3270399220,21cyclictest0-21swapper/019:29:510
3270399220,1cyclictest0-21swapper/019:59:550
3270399220,1cyclictest0-21swapper/019:39:590
32709992121,0cyclictest0-21swapper/322:09:533
32709992121,0cyclictest0-21swapper/321:09:503
32709992120,0cyclictest0-21swapper/323:04:533
32709992119,1cyclictest31533-21swap23:30:023
32709992119,1cyclictest27094-21apt-get23:24:533
32709992119,1cyclictest21205-21munin-run23:10:003
32709992119,1cyclictest18485-21users20:50:023
32709992119,1cyclictest0-21swapper/319:45:023
32709992118,2cyclictest19634-21turbostat22:00:003
32709992117,4cyclictest0-21swapper/320:04:523
3270999210,20cyclictest7475-21idleruntime-cro21:35:013
3270999210,20cyclictest13633-21apt-get19:39:523
3270999210,20cyclictest0-21swapper/323:39:543
3270999210,20cyclictest0-21swapper/323:29:533
3270999210,20cyclictest0-21swapper/322:39:553
3270999210,20cyclictest0-21swapper/320:49:523
3270999210,20cyclictest0-21swapper/320:24:523
3270999210,1cyclictest11447-21cpuspeed_turbos20:39:543
3270999210,0cyclictest0-21swapper/321:14:533
3270999210,0cyclictest0-21swapper/319:29:503
3270999210,0cyclictest0-21swapper/319:14:523
3270999210,0cyclictest0-21swapper/300:19:523
32708992120,1cyclictest0-21swapper/223:30:022
32708992120,0cyclictest24038-21apt-get19:59:552
3270899210,21cyclictest0-21swapper/220:10:002
3270899210,20cyclictest0-21swapper/223:44:522
3270899210,0cyclictest0-21swapper/200:04:512
3270799215,15cyclictest28782-21/usr/sbin/munin00:34:541
32707992121,0cyclictest0-21swapper/121:20:021
32707992120,1cyclictest1632-21apt-get20:19:541
32707992120,1cyclictest0-21swapper/100:04:511
32707992119,1cyclictest0-21swapper/120:50:011
3270799210,21cyclictest0-21swapper/121:54:541
3270799210,20cyclictest18841-21apt-get19:49:511
3270799210,1cyclictest593-21dbus-daemon23:34:591
3270799210,1cyclictest211rcuc/100:40:001
3270799210,0cyclictest19203-21apt-get00:14:531
32703992121,0cyclictest0-21swapper/020:44:520
32703992120,1cyclictest32140-21apt-get21:20:010
32703992120,1cyclictest0-21swapper/022:09:530
3270399210,21cyclictest0-21swapper/022:59:530
3270399210,20cyclictest0-21swapper/023:09:540
3270399210,0cyclictest0-21swapper/023:39:540
3270399210,0cyclictest0-21swapper/021:44:530
3270399210,0cyclictest0-21swapper/020:39:520
3270399210,0cyclictest0-21swapper/020:34:530
3270399210,0cyclictest0-21swapper/020:24:530
32709992020,0cyclictest0-21swapper/322:29:533
32709992020,0cyclictest0-21swapper/321:24:543
32709992020,0cyclictest0-21swapper/300:39:563
3270999200,19cyclictest0-21swapper/322:04:523
3270999200,19cyclictest0-21swapper/321:44:533
3270999200,0cyclictest0-21swapper/323:14:513
3270999200,0cyclictest0-21swapper/321:29:543
3270999200,0cyclictest0-21swapper/321:29:543
3270999200,0cyclictest0-21swapper/319:44:523
32708992020,0cyclictest0-21swapper/223:09:542
32708992020,0cyclictest0-21swapper/223:04:522
32708992020,0cyclictest0-21swapper/222:44:542
32708992020,0cyclictest0-21swapper/219:14:522
32708992019,1cyclictest18914-21apt-get20:54:552
32708992019,0cyclictest0-21swapper/223:19:542
32708992018,1cyclictest7903-21perf19:25:012
32708992018,1cyclictest28500-21awk23:25:002
32708992018,1cyclictest0-21swapper/222:25:012
3270899200,20cyclictest0-21swapper/220:24:542
3270899200,20cyclictest0-21swapper/200:09:532
3270899200,0cyclictest0-21swapper/223:54:532
3270899200,0cyclictest0-21swapper/222:19:522
3270899200,0cyclictest0-21swapper/200:34:542
32707992020,0cyclictest3733-21apt-get21:29:551
32707992020,0cyclictest3733-21apt-get21:29:551
32707992020,0cyclictest0-21swapper/122:19:521
32707992020,0cyclictest0-21swapper/119:34:521
32707992019,1cyclictest7824-21apt-get22:44:531
32707992018,1cyclictest24818-21cron21:05:001
32707992018,1cyclictest22-21ksoftirqd/120:30:001
32707992017,2cyclictest12548-21cron20:39:591
3270799200,20cyclictest0-21swapper/122:00:031
3270799200,0cyclictest558-21apt-get22:29:531
3270799200,0cyclictest21240-21apt-get19:54:531
3270799200,0cyclictest10947-21apt-get21:44:521
32703992020,0cyclictest17487-21apt-get23:04:530
32703992020,0cyclictest0-21swapper/020:49:530
32703992020,0cyclictest0-21swapper/019:19:540
32703992020,0cyclictest0-21swapper/000:19:520
32703992019,1cyclictest5407-21apt-get22:39:540
32703992018,1cyclictest12620-21apt-get22:54:530
32703992018,1cyclictest0-21swapper/022:25:000
32703992018,1cyclictest0-21swapper/000:05:000
3270399200,20cyclictest0-21swapper/022:34:520
3270399200,20cyclictest0-21swapper/021:34:520
3270399200,1cyclictest28814-21apt-get00:34:540
3270399200,0cyclictest23728-21apt-get21:04:530
3270399200,0cyclictest0-21swapper/023:34:560
3270399200,0cyclictest0-21swapper/021:19:530
3270399200,0cyclictest0-21swapper/021:09:510
3270399200,0cyclictest0-21swapper/020:54:550
32709991919,0cyclictest8852-21apt-get20:34:543
32709991919,0cyclictest0-21swapper/321:39:543
32709991919,0cyclictest0-21swapper/319:59:553
32709991918,1cyclictest0-21swapper/323:19:533
32709991917,1cyclictest0-21swapper/319:20:013
3270999190,1cyclictest0-21swapper/321:49:593
3270999190,18cyclictest0-21swapper/300:29:503
3270999190,0cyclictest0-21swapper/321:19:533
32708991919,0cyclictest0-21swapper/223:29:542
32708991919,0cyclictest0-21swapper/222:04:512
32708991919,0cyclictest0-21swapper/221:29:542
32708991919,0cyclictest0-21swapper/221:29:542
32708991919,0cyclictest0-21swapper/220:19:542
32708991919,0cyclictest0-21swapper/219:49:522
32708991916,2cyclictest23545-21tail23:14:592
32708991916,2cyclictest20794-21unixbench_multi20:55:042
32708991916,2cyclictest19399-21cut00:14:532
32708991916,2cyclictest11446-21tr20:39:542
3270899190,18cyclictest28522-21apt-get21:14:542
32707991919,0cyclictest0-21swapper/121:49:511
32707991916,2cyclictest31724-21dump-pmu-power22:25:011
32703991919,0cyclictest28522-21apt-get21:14:540
32703991919,0cyclictest0-21swapper/023:54:540
32703991919,0cyclictest0-21swapper/023:44:540
32703991919,0cyclictest0-21swapper/022:49:530
32703991919,0cyclictest0-21swapper/022:44:540
32703991919,0cyclictest0-21swapper/022:19:520
32703991919,0cyclictest0-21swapper/021:54:530
32703991919,0cyclictest0-21swapper/019:14:520
32703991919,0cyclictest0-21swapper/000:14:510
32703991918,1cyclictest0-21swapper/023:59:540
3270399190,1cyclictest32690-21munin-run00:40:010
32709991818,0cyclictest0-21swapper/319:34:513
32709991815,2cyclictest30194-21ps00:35:013
32709991813,5cyclictest0-21swapper/320:09:593
3270999180,1cyclictest0-21swapper/322:49:543
3270999180,17cyclictest0-21swapper/322:44:543
32708991818,0cyclictest13341-21apt-get21:49:522
32708991818,0cyclictest0-21swapper/222:59:522
32708991818,0cyclictest0-21swapper/222:39:552
32708991816,1cyclictest3008-21apt-get22:34:532
32708991814,3cyclictest25580-21unixbench-2d21:05:032
32707991818,0cyclictest29490-21apt-get23:29:541
32707991818,0cyclictest24678-21apt-get23:19:541
32707991818,0cyclictest23004-21apt-get22:09:531
32707991818,0cyclictest15069-21apt-get22:59:541
32707991818,0cyclictest0-21swapper/122:49:521
32707991818,0cyclictest0-21swapper/122:39:541
32707991818,0cyclictest0-21swapper/119:44:521
32707991817,1cyclictest10475-21perf19:30:011
3270799180,17cyclictest9499-21apt-get23:54:541
32703991818,0cyclictest7099-21apt-get23:49:520
32703991818,0cyclictest13341-21apt-get21:49:520
32703991818,0cyclictest0-21swapper/023:29:520
32703991818,0cyclictest0-21swapper/021:29:540
32703991818,0cyclictest0-21swapper/021:29:540
32703991818,0cyclictest0-21swapper/020:09:520
32703991818,0cyclictest0-21swapper/019:24:520
32703991818,0cyclictest0-21swapper/000:24:540
32709991717,0cyclictest0-21swapper/323:45:003
32709991717,0cyclictest0-21swapper/322:14:543
3270999170,2cyclictest0-21swapper/300:04:553
3270999170,13cyclictest0-21swapper/320:57:553
32708991717,0cyclictest0-21swapper/200:24:542
32708991716,1cyclictest0-21swapper/221:04:552
32708991716,1cyclictest0-21swapper/219:34:562
32708991715,2cyclictest0-21swapper/221:54:522
32708991715,1cyclictest0-21swapper/219:54:542
32708991714,2cyclictest7781-21idleruntime-cro20:30:002
3270899170,16cyclictest0-21swapper/222:09:522
32707991717,0cyclictest25400-21apt-get22:14:541
32707991717,0cyclictest0-21swapper/123:49:531
32707991717,0cyclictest0-21swapper/121:34:531
32707991717,0cyclictest0-21swapper/120:44:501
32707991717,0cyclictest0-21swapper/119:19:541
32707991716,1cyclictest21174-21cron23:10:011
32707991715,2cyclictest22-21ksoftirqd/121:00:011
32703991717,0cyclictest20605-21apt-get22:04:510
32703991717,0cyclictest16804-21apt-get00:09:530
32703991717,0cyclictest0-21swapper/019:54:540
32703991717,0cyclictest0-21swapper/019:49:510
32703991717,0cyclictest0-21swapper/000:29:500
32709991616,0cyclictest0-21swapper/323:54:543
3270999160,15cyclictest0-21swapper/321:54:543
3270999160,0cyclictest0-21swapper/300:24:293
32708991616,0cyclictest0-21swapper/221:20:022
32708991614,1cyclictest8453-21latency_hist23:50:002
32708991613,2cyclictest9570-21cat22:45:022
32708991613,2cyclictest12910-21ntp_kernel_err23:59:592
32708991611,4cyclictest0-21swapper/221:44:592
32707991616,0cyclictest0-21swapper/119:59:541
32707991616,0cyclictest0-21swapper/100:24:531
3270999151,2cyclictest31293-21taskset20:10:503
32708991515,0cyclictest0-21swapper/222:29:532
32708991515,0cyclictest0-21swapper/219:44:522
32708991513,2cyclictest0-21swapper/219:39:542
3270899150,14cyclictest0-21swapper/223:35:032
3270899150,13cyclictest0-21swapper/220:14:582
32707991515,0cyclictest0-21swapper/122:54:531
32707991515,0cyclictest0-21swapper/121:59:521
32707991515,0cyclictest0-21swapper/120:10:011
3270799150,1cyclictest0-21swapper/122:34:561
3270799150,1cyclictest0-21swapper/120:02:381
32703991515,0cyclictest11181-21apt-get19:34:520
32703991515,0cyclictest0-21swapper/022:14:540
3270399150,1cyclictest0-21swapper/021:55:590
32709991413,1cyclictest0-21swapper/323:55:113
3270999140,14cyclictest0-21swapper/300:06:033
32708991414,0cyclictest0-21swapper/200:35:012
32708991412,1cyclictest0-21swapper/222:10:032
3270899140,14cyclictest0-21swapper/221:35:212
3270899140,0cyclictest0-21swapper/219:16:162
32707991414,0cyclictest0-21swapper/123:55:521
3270799140,14cyclictest0-21swapper/120:20:091
3270799140,0cyclictest0-21swapper/123:41:501
3270799140,0cyclictest0-21swapper/121:05:531
3270799140,0cyclictest0-21swapper/100:26:331
3270399140,14cyclictest0-21swapper/020:00:500
3270399140,0cyclictest0-21swapper/020:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional