You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 08:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Thu Jul 03, 2025 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1744399240,1cyclictest11161-21apt-get23:29:573
17439992422,1cyclictest31530-21apt-key20:50:002
1743599240,23cyclictest31763-21basename00:10:001
1743599240,1cyclictest3375-21apt-get20:59:531
1743299240,23cyclictest26546-21grep00:00:000
1744399230,2cyclictest31198-21perf21:55:003
1744399230,22cyclictest26533-21sendmail00:00:003
1744399230,22cyclictest1408-21sh22:00:003
1744399230,22cyclictest11519-21basename00:35:003
17435992321,1cyclictest11439-21munin-run20:05:001
17435992320,2cyclictest313-21systemd-journal20:35:001
1743599231,21cyclictest30041-21munin-run19:35:001
1743599230,22cyclictest583-21avahi-daemon00:00:001
1743599230,22cyclictest0-21swapper/100:40:001
17432992321,1cyclictest0-21swapper/000:04:570
1743299230,22cyclictest5790-21apt21:04:550
1743299230,22cyclictest4281-21munin-run00:20:000
1743299230,22cyclictest17319-21munin-plugin-st20:20:000
1743299230,1cyclictest313-21systemd-journal23:09:550
17443992221,1cyclictest0-21swapper/320:44:533
17443992221,0cyclictest12305-21apt-get22:24:553
17443992220,1cyclictest27172-21munin-run20:40:003
17443992220,1cyclictest18138-21cron22:35:003
1744399220,2cyclictest1877-21cron00:15:013
1744399220,21cyclictest6702-21awk00:24:593
1744399220,21cyclictest661-21apt-get19:44:533
1744399220,21cyclictest593-21dbus-daemon21:35:003
1744399220,21cyclictest4266-21sendmail00:20:013
1744399220,21cyclictest1489-21apt-get23:09:553
1744399220,21cyclictest10640-21apt-get21:14:563
1744399220,1cyclictest19468-21sh23:45:003
17439992220,1cyclictest21178-21latency_hist21:35:002
17439992220,1cyclictest0-21swapper/223:35:002
17439992220,1cyclictest0-21swapper/222:00:002
1743999220,2cyclictest4703-21awk23:15:002
1743999220,21cyclictest4293-21needreboot00:20:002
1743999220,21cyclictest313-21systemd-journal00:40:002
1743999220,21cyclictest2722-21munin-run23:10:002
1743999220,21cyclictest25595-21dump-pmu-power22:50:002
1743999220,21cyclictest20435-21sed23:45:022
1743999220,21cyclictest0-21swapper/219:35:002
1743999220,1cyclictest313-21systemd-journal22:54:592
1743999220,1cyclictest28202-21apt-get20:44:522
1743999220,1cyclictest0-21swapper/219:45:002
17435992221,1cyclictest23496-21apt-get19:24:521
17435992221,1cyclictest0-21swapper/121:25:001
17435992221,0cyclictest0-21swapper/100:24:551
17435992220,1cyclictest29109-21apt-get22:59:591
17435992220,1cyclictest10852-21perf22:20:011
17435992219,2cyclictest3957-21grep22:05:001
1743599220,21cyclictest8878-21apt00:30:001
1743599220,21cyclictest5480-21apt-get19:54:541
1743599220,21cyclictest30251-21apt-get21:54:561
1743599220,21cyclictest21194-21basename21:35:001
1743599220,21cyclictest13035-21apt-get21:19:541
1743599220,21cyclictest10441-21apt-get00:34:541
1743599220,21cyclictest0-21swapper/123:39:541
1743599220,21cyclictest0-21swapper/119:19:531
1743599220,1cyclictest612-21gdbus20:50:001
1743599220,1cyclictest593-21dbus-daemon19:15:001
1743599220,1cyclictest27851-21apt-get21:49:541
1743599220,1cyclictest27142-21apt20:40:001
17432992221,0cyclictest0-21swapper/019:59:540
17432992220,1cyclictest20254-21apt-get21:34:550
17432992220,1cyclictest1930-21awk19:45:010
17432992220,1cyclictest0-21swapper/022:55:010
1743299220,21cyclictest8760-21apt-get23:24:540
1743299220,21cyclictest8721-21latency_hist22:15:000
1743299220,21cyclictest26503-21ntp_states21:45:000
1743299220,21cyclictest0-21swapper/023:49:530
1743299220,21cyclictest0-21swapper/021:29:560
1743299220,21cyclictest0-21swapper/021:09:530
1743299220,1cyclictest1410-21basename22:00:000
17443992121,0cyclictest0-21swapper/320:10:013
17443992120,1cyclictest26715-21apt-get22:55:003
17443992120,1cyclictest0-21swapper/322:59:593
17443992120,0cyclictest0-21swapper/320:04:543
1744399210,20cyclictest26539-21apt21:45:003
1744399210,20cyclictest0-21swapper/320:34:533
1744399210,20cyclictest0-21swapper/319:49:543
1744399210,20cyclictest0-21swapper/300:09:573
1744399210,1cyclictest8727-21munin-plugin-st22:15:003
1744399210,1cyclictest27320-21awk19:30:003
1744399210,1cyclictest0-21swapper/322:50:003
1744399210,0cyclictest15949-21apt-get23:39:543
1744399210,0cyclictest0-21swapper/322:29:543
17439992121,0cyclictest0-21swapper/221:24:552
17439992121,0cyclictest0-21swapper/220:19:512
17439992120,0cyclictest23392-21apt-get20:34:532
17439992120,0cyclictest0-21swapper/221:49:542
17439992119,1cyclictest23972-21awk21:40:012
17439992118,2cyclictest9997-21latency_hist23:25:002
1743999210,20cyclictest22980-21smartctl22:45:002
1743999210,20cyclictest12759-21apt-get20:10:012
1743999210,20cyclictest0-21swapper/221:14:562
1743999210,1cyclictest1-21systemd20:00:002
1743999210,1cyclictest0-21swapper/222:09:552
17435992121,0cyclictest5791-21apt-get21:04:551
17435992121,0cyclictest0-21swapper/121:14:561
17435992120,0cyclictest0-21swapper/122:39:571
17435992119,1cyclictest26715-21apt-get22:55:001
17435992118,2cyclictest2205-21dump-pmu-power20:55:001
1743599210,0cyclictest22642-21apt-get21:39:531
1743599210,0cyclictest17854-21apt-get21:29:571
1743599210,0cyclictest0-21swapper/123:14:561
1743599210,0cyclictest0-21swapper/123:09:551
1743599210,0cyclictest0-21swapper/100:14:561
17432992121,0cyclictest0-21swapper/022:39:570
17432992120,1cyclictest0-21swapper/023:19:560
17432992120,1cyclictest0-21swapper/020:29:530
17432992119,1cyclictest31196-21perf20:50:000
1743299210,21cyclictest0-21swapper/019:34:500
1743299210,20cyclictest15452-21apt-get21:24:570
1743299210,20cyclictest0-21swapper/023:29:570
1743299210,20cyclictest0-21swapper/020:44:530
1743299210,20cyclictest0-21swapper/020:39:530
1743299210,20cyclictest0-21swapper/020:14:480
1743299210,0cyclictest0-21swapper/000:34:540
1743299210,0cyclictest0-21swapper/000:14:560
17443992020,0cyclictest0-21swapper/321:09:533
17443992020,0cyclictest0-21swapper/320:59:533
17443992018,1cyclictest20996-21apt-get20:29:533
1744399200,20cyclictest0-21swapper/322:44:553
1744399200,20cyclictest0-21swapper/321:49:543
17439992020,0cyclictest28470-21apt-get00:04:582
17439992020,0cyclictest0-21swapper/222:14:532
17439992020,0cyclictest0-21swapper/221:44:542
17439992019,0cyclictest0-21swapper/220:39:532
1743999200,20cyclictest0-21swapper/222:34:552
1743999200,20cyclictest0-21swapper/220:59:532
1743999200,0cyclictest0-21swapper/223:44:542
1743999200,0cyclictest0-21swapper/220:29:532
17435992020,0cyclictest0-21swapper/122:09:551
17435992018,1cyclictest26537-21perf21:45:001
17435992018,1cyclictest14699-21apt-get22:29:541
1743599200,1cyclictest22994-21awk22:45:001
1743599200,0cyclictest0-21swapper/100:04:571
17432992020,0cyclictest5044-21apt-get22:09:550
17432992020,0cyclictest0-21swapper/023:54:560
17432992020,0cyclictest0-21swapper/023:34:550
17432992020,0cyclictest0-21swapper/022:59:590
17432992020,0cyclictest0-21swapper/020:24:550
17432992018,1cyclictest19806-21grep19:15:000
17432992017,2cyclictest4776-21munin-run23:15:000
1743299200,20cyclictest0-21swapper/021:39:530
1743299200,0cyclictest0-21swapper/022:19:550
1743299200,0cyclictest0-21swapper/021:19:560
1743299200,0cyclictest0-21swapper/020:54:530
17443991919,0cyclictest0-21swapper/322:39:583
17443991919,0cyclictest0-21swapper/322:19:543
17443991919,0cyclictest0-21swapper/322:04:543
17443991919,0cyclictest0-21swapper/321:19:553
17443991919,0cyclictest0-21swapper/320:54:533
17443991919,0cyclictest0-21swapper/320:19:553
17443991919,0cyclictest0-21swapper/300:39:543
17443991918,1cyclictest21081-21apt-get19:19:533
17443991918,0cyclictest0-21swapper/323:34:553
17443991917,1cyclictest0-21swapper/321:24:563
17439991919,0cyclictest831-21apt-get00:14:562
17439991919,0cyclictest25889-21apt-get19:29:522
17439991919,0cyclictest0-21swapper/223:04:572
17439991919,0cyclictest0-21swapper/222:59:592
17439991919,0cyclictest0-21swapper/221:29:552
17439991919,0cyclictest0-21swapper/220:54:542
17439991919,0cyclictest0-21swapper/220:24:552
17439991919,0cyclictest0-21swapper/219:39:522
17439991918,0cyclictest0-21swapper/221:54:562
17439991917,1cyclictest19505-21apt-get22:39:572
1743999190,19cyclictest0-21swapper/219:19:532
17435991919,0cyclictest8237-21apt-get21:09:531
17435991919,0cyclictest0-21swapper/123:49:541
17435991919,0cyclictest0-21swapper/119:29:521
17435991918,0cyclictest0-21swapper/122:24:541
17432991919,0cyclictest0-21swapper/019:24:520
17432991917,1cyclictest13980-21munin-plugin-st00:40:000
1743299190,19cyclictest0-21swapper/022:34:550
1743299190,0cyclictest0-21swapper/019:29:520
17443991818,0cyclictest19451-21apt-get23:45:013
17443991818,0cyclictest0-21swapper/319:54:533
17443991816,1cyclictest18602-21apt-get20:24:553
17443991816,1cyclictest0-21swapper/321:05:013
17439991818,0cyclictest23567-21apt-get23:54:562
17439991818,0cyclictest0-21swapper/223:59:582
17439991818,0cyclictest0-21swapper/221:09:532
1743999180,18cyclictest30861-21apt-get00:09:562
17435991818,0cyclictest3086-21apt-get19:49:541
17435991818,0cyclictest0-21swapper/123:19:561
17435991818,0cyclictest0-21swapper/119:39:531
17435991817,0cyclictest0-21swapper/122:14:531
17435991816,1cyclictest25598-21dump-pmu-power22:50:001
17432991818,0cyclictest0-21swapper/022:24:540
17432991818,0cyclictest0-21swapper/021:14:560
17432991818,0cyclictest0-21swapper/019:39:520
17432991818,0cyclictest0-21swapper/019:19:530
17432991818,0cyclictest0-21swapper/000:09:560
17443991717,0cyclictest8053-21apt-get00:29:563
17443991717,0cyclictest7930-21apt-get19:59:543
17443991717,0cyclictest0-21swapper/323:19:563
17443991717,0cyclictest0-21swapper/300:04:573
17443991716,1cyclictest0-21swapper/323:24:533
17443991715,1cyclictest31202-21apt-get20:50:003
17443991714,2cyclictest12761-21perf20:10:003
1744399170,16cyclictest32048-21munin-run19:40:003
17439991717,0cyclictest6295-21apt-get23:19:572
17439991717,0cyclictest0-21swapper/223:29:562
17439991717,0cyclictest0-21swapper/222:29:552
17439991717,0cyclictest0-21swapper/220:04:562
17439991717,0cyclictest0-21swapper/200:34:542
17435991717,0cyclictest17100-21apt-get22:34:551
17435991717,0cyclictest0-21swapper/120:44:531
17435991715,1cyclictest1425-21grep22:00:001
17432991717,0cyclictest24308-21apt-get22:49:530
17432991717,0cyclictest21917-21apt-get22:44:540
17432991717,0cyclictest0-21swapper/022:04:540
17432991717,0cyclictest0-21swapper/021:54:560
17432991717,0cyclictest0-21swapper/020:59:530
17432991716,0cyclictest18373-21apt-get23:44:540
17432991714,2cyclictest15772-21ntp_states22:30:000
17432991714,2cyclictest11410-21ntp_states20:05:000
1743299170,0cyclictest0-21swapper/023:04:580
17443991616,0cyclictest0-21swapper/323:14:563
17443991616,0cyclictest0-21swapper/322:09:543
17443991616,0cyclictest0-21swapper/321:39:533
17443991614,1cyclictest0-21swapper/323:00:053
17439991616,0cyclictest0-21swapper/219:24:532
17439991616,0cyclictest0-21swapper/200:29:562
17439991615,0cyclictest0-21swapper/223:39:542
1743999160,15cyclictest1686-21apache200:22:552
1743999160,15cyclictest12948-21hddtemp_smartct22:24:572
1743999160,15cyclictest12791-21cut20:10:002
1743999160,15cyclictest0-21swapper/219:54:572
17435991616,0cyclictest0-21swapper/123:24:531
17435991616,0cyclictest0-21swapper/120:24:551
17435991616,0cyclictest0-21swapper/100:19:561
17435991613,2cyclictest19482-21munin-run23:45:001
1743599160,15cyclictest11925-21/usr/sbin/munin23:30:001
17432991615,0cyclictest5653-21apt-get00:24:540
17432991614,1cyclictest3992-21memory19:49:580
17439991513,1cyclictest7403-21/usr/sbin/munin22:15:022
17439991513,1cyclictest3942-21cron22:05:002
17435991514,1cyclictest13772-21apt-get20:14:481
1743599150,1cyclictest0-21swapper/123:30:021
1743599150,1cyclictest0-21swapper/123:00:021
17443991414,0cyclictest0-21swapper/323:51:063
1744399140,14cyclictest0-21swapper/321:25:083
1744399140,14cyclictest0-21swapper/319:30:073
1744399140,14cyclictest0-21swapper/319:20:123
1744399140,14cyclictest0-21swapper/319:10:023
1743999140,14cyclictest0-21swapper/221:15:112
1743999140,14cyclictest0-21swapper/221:00:112
1743999140,14cyclictest0-21swapper/219:45:202
17435991414,0cyclictest0-21swapper/120:15:351
1743599140,14cyclictest0-21swapper/123:51:301
1743599140,14cyclictest0-21swapper/120:27:181
1743599140,0cyclictest0-21swapper/119:55:501
1743599140,0cyclictest0-21swapper/119:40:251
17432991414,0cyclictest0-21swapper/021:45:180
17432991414,0cyclictest0-21swapper/000:25:130
1743299140,14cyclictest0-21swapper/023:35:580
1743299140,14cyclictest0-21swapper/020:30:350
1743299140,14cyclictest0-21swapper/020:05:420
1743299140,14cyclictest0-21swapper/019:50:440
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional