You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 02:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Mon Jul 14, 2025 12:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28390992422,1cyclictest14130-21munin-run08:55:003
2839099240,23cyclictest4751-21latency_hist10:50:003
2839099240,1cyclictest682-21cron07:20:003
28383992422,1cyclictest22825-21apt-get08:05:551
28390992323,0cyclictest0-21swapper/307:40:573
28390992321,1cyclictest3595-21grep11:54:593
2838799230,22cyclictest313-21systemd-journal08:10:002
28383992321,1cyclictest8425-21cron12:05:001
28383992321,1cyclictest3586-21idleruntime-cro11:54:591
28383992321,1cyclictest0-21swapper/111:35:001
28383992321,1cyclictest0-21swapper/109:29:591
28383992320,2cyclictest2126-21perf08:30:001
2838399230,2cyclictest6938-21munin-run08:40:001
2838399230,22cyclictest26219-21turbostat09:20:001
2838399230,22cyclictest0-21swapper/109:50:001
2837999232,2cyclictest24895-21cat08:10:010
28379992320,2cyclictest8422-21cron12:05:000
2837999230,22cyclictest26217-21awk09:20:000
2837999230,22cyclictest19194-21awk11:20:000
2837999230,1cyclictest4500-21cron08:34:590
28390992222,0cyclictest0-21swapper/307:45:553
28390992221,1cyclictest0-21swapper/309:10:533
2839099221,1cyclictest34-21ksoftirqd/309:45:553
2839099220,21cyclictest17149-21apt12:20:013
2839099220,21cyclictest0-21swapper/307:25:553
28387992222,0cyclictest0-21swapper/211:40:562
28387992221,1cyclictest4501-21apt08:35:002
28387992220,2cyclictest8585-21apt-get09:50:562
28387992220,2cyclictest6925-21perf08:40:002
28387992220,1cyclictest10780-21apt12:08:522
2838799221,1cyclictest14990-21apt-get12:15:532
2838799220,21cyclictest598-21rs:main2
2838799220,21cyclictest25159-21munin-plugin-st10:25:002
2838799220,21cyclictest0-21swapper/211:15:542
2838799220,1cyclictest26771-21dump-pmu-power12:39:592
28383992220,1cyclictest7940-21munin-plugin-st07:35:001
28383992220,1cyclictest28652-21munin-plugin-st09:24:591
28383992219,2cyclictest21594-21cron11:25:001
28383992215,5cyclictest2342-21idleruntime-cro10:45:011
28383992215,5cyclictest2342-21idleruntime-cro10:45:011
2838399220,2cyclictest3151-21perf07:25:001
2838399220,21cyclictest31234-21munin-run11:45:011
2838399220,21cyclictest2448-21apt-get08:30:551
2838399220,21cyclictest16519-21apt09:00:001
2838399220,21cyclictest0-21swapper/110:10:531
2838399220,21cyclictest0-21swapper/108:15:521
2838399220,1cyclictest8761-21apt-get12:05:541
28379992221,1cyclictest3575-21cron11:54:590
28379992221,0cyclictest0-21swapper/010:10:540
28379992220,1cyclictest26398-21dump-pmu-power11:35:000
28379992220,1cyclictest0-21swapper/012:25:530
2837999220,2cyclictest29948-21ls10:35:010
2837999220,21cyclictest0-21swapper/011:10:530
2837999220,21cyclictest0-21swapper/010:20:530
2837999220,21cyclictest0-21swapper/009:25:540
2837999220,21cyclictest0-21swapper/008:20:520
2837999220,21cyclictest0-21swapper/007:20:530
28390992121,0cyclictest0-21swapper/309:50:553
28390992121,0cyclictest0-21swapper/308:55:563
28390992120,1cyclictest0-21swapper/312:08:513
28390992120,0cyclictest0-21swapper/310:55:553
2839099210,21cyclictest0-21swapper/310:40:513
2839099210,21cyclictest0-21swapper/310:40:513
2839099210,20cyclictest29150-21apt-get11:40:563
2839099210,20cyclictest0-21swapper/312:25:003
2839099210,20cyclictest0-21swapper/307:35:563
2839099210,1cyclictest34-21ksoftirqd/308:30:553
2839099210,1cyclictest0-21swapper/311:40:003
2839099210,1cyclictest0-21swapper/310:30:533
2839099210,0cyclictest0-21swapper/309:20:543
2839099210,0cyclictest0-21swapper/308:20:523
28387992121,0cyclictest0-21swapper/210:10:532
28387992120,1cyclictest26725-21apt-get11:35:572
28387992120,1cyclictest26564-21apt-get09:20:542
28387992120,1cyclictest21753-21apt-get09:10:532
28387992119,1cyclictest13170-21apt-get07:45:552
2838799210,21cyclictest0-21swapper/212:11:112
2838799210,20cyclictest0-21swapper/210:35:532
2838799210,1cyclictest2127-21munin-plugin-st08:30:002
28383992120,1cyclictest19881-21apt-get12:25:531
28383992119,1cyclictest28816-21munin-run11:40:011
28383992119,1cyclictest18015-21apt-get07:55:551
28383992119,1cyclictest17117-21apt-get11:15:541
28383992119,1cyclictest0-21swapper/112:35:001
2838399210,0cyclictest23074-21apt-get10:20:531
2838399210,0cyclictest0-21swapper/108:20:531
28379992121,0cyclictest27610-21apt-get08:15:520
28379992120,0cyclictest8254-21apt-get07:35:560
28379992119,1cyclictest7934-21awk07:35:000
28379992119,1cyclictest31362-21apt-get09:30:520
28379992119,1cyclictest2128-21munin-run08:30:000
28379992119,1cyclictest12197-21apt-get12:10:540
28379992118,1cyclictest21577-21cron11:25:000
28379992113,3cyclictest2337-21sh10:45:000
28379992113,3cyclictest2337-21sh10:45:000
2837999210,21cyclictest0-21swapper/011:45:000
2837999210,0cyclictest2676-21apt-get10:45:550
2837999210,0cyclictest17490-21apt-get12:20:540
2837999210,0cyclictest0-21swapper/012:05:530
2837999210,0cyclictest0-21swapper/009:20:530
2837999210,0cyclictest0-21swapper/009:05:550
28390992020,0cyclictest30269-21apt-get10:35:533
28390992020,0cyclictest0-21swapper/312:30:563
28390992020,0cyclictest0-21swapper/311:55:563
28390992020,0cyclictest0-21swapper/311:00:543
28390992020,0cyclictest0-21swapper/310:25:543
28390992020,0cyclictest0-21swapper/309:40:533
28390992020,0cyclictest0-21swapper/309:35:533
28390992020,0cyclictest0-21swapper/309:30:533
28390992020,0cyclictest0-21swapper/309:15:533
28390992018,1cyclictest20339-21awk10:15:003
28390992018,1cyclictest0-21swapper/309:55:013
2839099200,20cyclictest0-21swapper/308:40:523
2839099200,1cyclictest14644-21cron12:15:013
2839099200,19cyclictest20428-21apt-get08:00:523
28387992020,0cyclictest6007-21apt-get12:00:022
28387992020,0cyclictest3770-21apt-get09:40:532
28387992020,0cyclictest3463-21apt-get07:25:542
28387992020,0cyclictest0-21swapper/208:15:522
28387992020,0cyclictest0-21swapper/208:00:522
28387992020,0cyclictest0-21swapper/207:50:522
28387992020,0cyclictest0-21swapper/207:35:572
28387992020,0cyclictest0-21swapper/207:15:532
28387992019,1cyclictest29950-21turbostat10:35:002
28387992019,1cyclictest0-21swapper/207:25:002
28387992019,0cyclictest0-21swapper/211:20:542
28387992018,1cyclictest0-21swapper/209:50:002
2838799200,20cyclictest0-21swapper/210:25:532
2838799200,0cyclictest7253-21apt-get08:40:522
2838799200,0cyclictest19309-21apt-get09:05:552
2838799200,0cyclictest0-21swapper/209:55:522
28383992020,0cyclictest0-21swapper/109:50:551
28383992020,0cyclictest0-21swapper/109:30:521
28383992020,0cyclictest0-21swapper/107:35:561
28383992019,1cyclictest1339-21apt-get09:35:531
28383992019,0cyclictest13938-21apt12:11:111
28383992019,0cyclictest0-21swapper/111:55:561
28383992019,0cyclictest0-21swapper/110:35:531
28383992018,1cyclictest28662-21apt-get07:10:531
28383992018,1cyclictest15559-21apt-get07:50:521
28383992018,1cyclictest0-21swapper/109:40:531
2838399200,20cyclictest0-21swapper/112:15:531
2838399200,0cyclictest0-21swapper/109:10:541
2838399200,0cyclictest0-21swapper/109:00:561
28379992020,0cyclictest0-21swapper/010:50:580
28379992020,0cyclictest0-21swapper/009:40:530
28379992020,0cyclictest0-21swapper/008:55:570
28379992020,0cyclictest0-21swapper/008:10:520
28379992019,0cyclictest0-21swapper/011:55:560
2837999200,20cyclictest0-21swapper/009:10:540
2837999200,0cyclictest0-21swapper/011:25:550
2837999200,0cyclictest0-21swapper/008:40:520
2837999200,0cyclictest0-21swapper/008:35:520
28390991919,0cyclictest0-21swapper/312:35:573
28390991919,0cyclictest0-21swapper/310:20:523
28390991919,0cyclictest0-21swapper/310:15:533
28390991919,0cyclictest0-21swapper/308:15:523
28390991918,1cyclictest0-21swapper/307:55:563
28390991916,2cyclictest7837-21timerandwakeup12:01:023
28387991919,0cyclictest30036-21apt-get08:20:522
28387991919,0cyclictest0-21swapper/212:20:542
28387991919,0cyclictest0-21swapper/210:45:552
28387991919,0cyclictest0-21swapper/210:00:542
28387991919,0cyclictest0-21swapper/207:55:552
28387991918,1cyclictest14669-21apt-get11:10:532
28387991918,1cyclictest0-21swapper/209:20:012
28387991917,1cyclictest16521-21turbostat.cron09:00:002
28383991919,0cyclictest0-21swapper/112:35:571
28383991919,0cyclictest0-21swapper/112:20:541
28383991919,0cyclictest0-21swapper/110:55:561
28383991919,0cyclictest0-21swapper/109:05:551
28383991919,0cyclictest0-21swapper/108:40:521
28383991919,0cyclictest0-21swapper/107:25:561
28383991918,1cyclictest10777-21apt-get07:40:571
28383991917,1cyclictest17954-21munin-run10:10:001
28379991919,0cyclictest0-21swapper/012:15:530
28379991919,0cyclictest0-21swapper/009:50:560
28379991919,0cyclictest0-21swapper/009:45:020
28379991918,1cyclictest24383-21perf12:35:010
28379991918,0cyclictest0-21swapper/007:55:550
28390991818,0cyclictest0-21swapper/311:30:543
28390991817,0cyclictest0-21swapper/307:10:533
28387991818,0cyclictest3927-21apt-get11:55:562
28387991818,0cyclictest28967-21apt-get09:25:542
28387991818,0cyclictest0-21swapper/212:25:532
28387991818,0cyclictest0-21swapper/207:30:502
28387991816,1cyclictest3576-21cron11:54:592
28383991818,0cyclictest31545-21apt-get11:45:561
28383991818,0cyclictest0-21swapper/108:00:521
28379991818,0cyclictest0-21swapper/012:35:570
28379991818,0cyclictest0-21swapper/011:40:000
28379991818,0cyclictest0-21swapper/010:55:560
28379991818,0cyclictest0-21swapper/010:35:540
28379991818,0cyclictest0-21swapper/009:35:530
28379991818,0cyclictest0-21swapper/007:25:550
28379991817,0cyclictest0-21swapper/008:00:520
2839099172,0cyclictest151rcu_preempt11:20:453
28390991717,0cyclictest0-21swapper/310:05:523
28390991717,0cyclictest0-21swapper/309:05:553
28390991717,0cyclictest0-21swapper/309:00:553
28390991717,0cyclictest0-21swapper/308:25:543
28390991716,1cyclictest34-21ksoftirqd/311:28:503
28390991715,1cyclictest5072-21apt-get10:50:573
28390991714,2cyclictest0-21swapper/308:35:583
28387991717,0cyclictest0-21swapper/211:25:542
28387991716,0cyclictest32685-21apt-get10:40:512
28387991716,0cyclictest32685-21apt-get10:40:512
28387991715,1cyclictest26397-21apt11:35:002
28383991715,1cyclictest4738-21dump-pmu-power10:50:011
2838399170,15cyclictest21929-21apt-get11:25:531
28379991717,0cyclictest0-21swapper/011:45:550
28379991717,0cyclictest0-21swapper/007:50:520
28379991717,0cyclictest0-21swapper/007:40:570
28379991717,0cyclictest0-21swapper/007:10:530
28379991715,1cyclictest0-21swapper/010:25:530
28390991616,0cyclictest0-21swapper/311:45:553
28390991616,0cyclictest0-21swapper/311:10:543
28390991616,0cyclictest0-21swapper/310:00:533
28390991616,0cyclictest0-21swapper/309:25:553
28390991616,0cyclictest0-21swapper/307:20:533
28390991614,1cyclictest0-21swapper/312:26:053
28387991616,0cyclictest25214-21apt-get08:10:512
28383991616,0cyclictest25469-21apt-get10:25:531
28383991616,0cyclictest12273-21apt-get11:05:521
28383991616,0cyclictest0-21swapper/108:10:511
28383991615,0cyclictest10652-21apt-get09:55:021
28383991614,1cyclictest0-21swapper/110:15:591
2838399160,15cyclictest7151-21perf10:55:011
2838399160,15cyclictest32106-21ntp_states07:15:581
28379991616,0cyclictest17952-21perf10:10:000
2837999160,15cyclictest0-21swapper/010:15:520
28390991515,0cyclictest15559-21apt-get07:50:523
28390991515,0cyclictest0-21swapper/308:45:533
28390991515,0cyclictest0-21swapper/308:10:513
28390991514,0cyclictest0-21swapper/311:05:533
28390991513,1cyclictest0-21swapper/308:05:553
28387991515,0cyclictest0-21swapper/210:50:582
28387991515,0cyclictest0-21swapper/209:30:522
28387991513,1cyclictest24395-21awk12:35:002
2838799150,7cyclictest0-21swapper/207:40:572
28383991513,1cyclictest11923-21cron11:04:591
2838399150,1cyclictest0-21swapper/110:31:021
2838399150,14cyclictest0-21swapper/108:47:311
28379991515,0cyclictest0-21swapper/011:05:520
28379991513,1cyclictest609-21polkitd11:04:590
28379991513,1cyclictest0-21swapper/007:46:000
2839099140,14cyclictest0-21swapper/311:15:063
2839099140,14cyclictest0-21swapper/307:30:453
28387991413,1cyclictest0-21swapper/207:10:532
28387991413,0cyclictest7466-21apt-get10:55:562
2838799140,14cyclictest0-21swapper/211:46:082
2838799140,14cyclictest0-21swapper/211:05:272
2838799140,14cyclictest0-21swapper/211:00:242
2838799140,14cyclictest0-21swapper/210:16:442
2838799140,14cyclictest0-21swapper/210:05:242
2838799140,14cyclictest0-21swapper/208:45:232
2838799140,0cyclictest0-21swapper/209:35:512
2838799140,0cyclictest0-21swapper/208:52:382
28383991414,0cyclictest0-21swapper/111:10:121
2838399140,14cyclictest0-21swapper/107:45:191
2838399140,0cyclictest0-21swapper/110:01:101
28379991414,0cyclictest0-21swapper/009:55:010
2837999140,14cyclictest0-21swapper/010:00:000
2837999140,14cyclictest0-21swapper/009:00:190
2837999140,14cyclictest0-21swapper/007:15:020
2837999140,0cyclictest0-21swapper/008:50:070
2837999140,0cyclictest0-21swapper/008:45:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional