You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 14:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Sun Dec 21, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17697992420,3cyclictest7899-21cron23:15:012
1769799240,23cyclictest5502-21latency_hist19:50:002
17695992423,1cyclictest0-21swapper/122:29:551
1769599240,23cyclictest29423-21apt-get19:34:551
1769599240,23cyclictest26712-21perf21:40:011
17700992320,2cyclictest2629-21idleruntime-cro00:10:013
1770099230,22cyclictest6740-21munin-run22:05:013
1770099230,22cyclictest475-21awk19:40:013
1770099230,22cyclictest28223-21turbostat.cron19:30:013
1769799230,22cyclictest11436-21awk22:15:002
17695992321,1cyclictest10251-21turbostat20:00:001
17695992320,2cyclictest12662-21perf20:05:001
1769599230,2cyclictest2882-21munin-run19:45:011
1769599230,0cyclictest0-21swapper/120:39:571
17690992321,1cyclictest28314-21perf22:50:010
17690992321,1cyclictest26716-21munin-run21:40:010
17690992321,1cyclictest2293-21apt-get21:55:020
17690992320,2cyclictest62550irq/127-eno119:59:560
17690992320,2cyclictest19230-21sshd23:39:540
1769099230,22cyclictest1433-21munin-plugin-st21:55:010
1769099230,1cyclictest14186-21apt-get00:34:560
17700992220,1cyclictest23245-21sendmail-msp19:20:003
17700992220,1cyclictest12666-21munin-run20:05:003
17700992219,2cyclictest25084-21awk20:30:013
17700992219,2cyclictest22374-21apt-get22:39:563
17700992219,2cyclictest21859-21idleruntime-cro21:30:003
17700992219,2cyclictest21024-21latency_hist22:35:013
1770099220,21cyclictest25298-21perf23:50:003
1770099220,21cyclictest14834-21dump-pmu-power20:10:013
1770099220,21cyclictest14434-21apt-get23:29:573
1770099220,21cyclictest10259-21sendmail-msp20:00:003
1770099220,21cyclictest0-21swapper/323:59:543
1770099220,21cyclictest0-21swapper/320:19:553
1770099220,21cyclictest0-21swapper/319:24:543
17697992222,0cyclictest28676-21apt-get20:39:572
17697992222,0cyclictest0-21swapper/223:39:542
17697992219,2cyclictest15432-21perf23:30:012
1769799220,21cyclictest30753-21munin-run19:35:012
1769799220,21cyclictest17576-21apt-get22:29:552
1769799220,21cyclictest14219-21apt-get20:09:582
1769799220,21cyclictest12469-21basename00:30:002
1769799220,21cyclictest0-21swapper/221:20:012
1769799220,1cyclictest32031-21apt-get22:59:542
17695992222,0cyclictest0-21swapper/120:34:541
17695992221,0cyclictest22156-21apt-get19:19:551
17695992220,1cyclictest6975-21apt-get00:19:541
17695992220,1cyclictest487-21turbostat00:05:011
17695992219,2cyclictest16299-21cron22:25:001
1769599220,21cyclictest6936-21perf21:00:011
1769599220,21cyclictest1574-21cut21:55:011
1769599220,1cyclictest3475-21ntp_states00:10:021
1769599220,1cyclictest25276-21cron23:50:001
1769599220,0cyclictest0-21swapper/100:39:581
17690992222,0cyclictest0-21swapper/000:04:550
17690992221,1cyclictest0-21swapper/020:39:570
17690992220,1cyclictest3478-21apt-get00:10:020
17690992219,2cyclictest25283-21latency_hist23:50:000
1769099220,21cyclictest12769-21apt-get22:19:560
1769099220,21cyclictest11901-21turbostat.cron21:10:010
1769099220,21cyclictest11793-21apt-get00:29:580
1769099220,21cyclictest0-21swapper/023:09:550
1769099220,21cyclictest0-21swapper/022:59:550
1769099220,21cyclictest0-21swapper/021:29:530
1769099220,21cyclictest0-21swapper/021:20:010
1769099220,1cyclictest25711-21munin-run22:45:010
1769099220,1cyclictest0-21swapper/022:24:540
17700992121,0cyclictest27242-21apt-get22:49:563
17700992121,0cyclictest1823-21apt-get19:44:563
17700992121,0cyclictest0-21swapper/322:54:543
17700992120,1cyclictest0-21swapper/322:14:553
17700992120,0cyclictest0-21swapper/323:54:553
17700992119,1cyclictest13801-21latency_hist22:20:003
1770099210,20cyclictest7935-21munin-run23:15:013
1770099210,20cyclictest0-21swapper/321:44:533
1770099210,20cyclictest0-21swapper/320:49:543
1770099210,0cyclictest24794-21apt-get22:44:573
1770099210,0cyclictest0-21swapper/321:39:543
17697992121,0cyclictest0-21swapper/222:44:572
17697992119,1cyclictest6921-21turbostat21:00:012
17697992119,1cyclictest30770-21perf22:55:002
17697992119,1cyclictest21038-21awk22:35:012
17697992119,1cyclictest20844-21latency_hist19:15:012
1769799210,21cyclictest31796-21apt-get00:04:542
1769799210,21cyclictest0-21swapper/219:50:022
1769799210,20cyclictest0-21swapper/222:04:542
1769799210,20cyclictest0-21swapper/200:24:552
1769799210,1cyclictest31816-21apt-get19:39:552
1769799210,1cyclictest31089-21apt-get20:44:552
1769799210,0cyclictest2646-21apt-get00:10:022
1769799210,0cyclictest13468-21apt-get21:14:552
1769799210,0cyclictest11075-21apt-get21:09:572
1769799210,0cyclictest0-21swapper/220:29:552
17695992120,1cyclictest2041-21apt-get23:04:571
17695992120,0cyclictest0-21swapper/123:09:551
17695992119,1cyclictest6728-21perf22:05:011
17695992119,1cyclictest20060-21ntp_states20:20:011
17695992119,1cyclictest13773-21cron22:20:001
1769599210,21cyclictest0-21swapper/119:39:551
1769599210,1cyclictest7926-21apt-get22:09:541
1769599210,1cyclictest10034-21munin-run23:20:011
1769599210,0cyclictest3465-21apt-get20:54:521
1769599210,0cyclictest29634-21apt-get22:54:541
1769599210,0cyclictest26888-21apt-get23:54:551
1769599210,0cyclictest0-21swapper/122:49:561
1769599210,0cyclictest0-21swapper/122:39:561
17690992120,0cyclictest0-21swapper/019:49:540
17690992119,1cyclictest1824-21sendmail-mta00:39:580
17690992119,1cyclictest10030-21perf23:20:010
17690992119,1cyclictest0-21swapper/022:29:550
17690992118,2cyclictest23240-21grep19:20:000
1769099210,20cyclictest9394-21apt-get00:24:540
1769099210,20cyclictest0-21swapper/023:59:540
1769099210,1cyclictest7598-21smartctl21:00:020
1769099210,0cyclictest0-21swapper/021:49:580
1769099210,0cyclictest0-21swapper/020:59:560
1769099210,0cyclictest0-21swapper/020:44:550
1769099210,0cyclictest0-21swapper/020:09:580
17700992020,0cyclictest16824-21apt-get23:34:533
17700992020,0cyclictest0-21swapper/323:19:583
17700992020,0cyclictest0-21swapper/320:39:573
17700992020,0cyclictest0-21swapper/319:34:553
17700992020,0cyclictest0-21swapper/319:14:563
17700992020,0cyclictest0-21swapper/300:04:543
17700992018,1cyclictest0-21swapper/321:05:013
1770099200,20cyclictest0-21swapper/322:59:553
1770099200,20cyclictest0-21swapper/320:14:553
1770099200,19cyclictest0-21swapper/320:59:563
1770099200,0cyclictest23080-21apt-get21:34:553
17697992020,0cyclictest0-21swapper/223:04:562
17697992020,0cyclictest0-21swapper/221:49:572
17697992018,1cyclictest10277-21basename20:00:002
1769799200,20cyclictest0-21swapper/223:34:532
1769799200,20cyclictest0-21swapper/200:19:532
1769799200,0cyclictest3115-21apt-get21:59:542
1769799200,0cyclictest0-21swapper/221:34:552
1769799200,0cyclictest0-21swapper/219:44:562
1769799200,0cyclictest0-21swapper/200:09:572
17695992020,0cyclictest10369-21apt-get22:14:551
17695992020,0cyclictest0-21swapper/121:14:551
17695992019,0cyclictest0-21swapper/120:44:551
17695992018,1cyclictest0-21swapper/121:20:011
1769599200,20cyclictest0-21swapper/123:29:571
1769599200,19cyclictest6826-21apt-get23:14:551
1769599200,19cyclictest27941-21apt-get21:44:531
1769599200,19cyclictest0-21swapper/122:59:551
1769599200,0cyclictest0-21swapper/120:14:551
17690992020,0cyclictest0-21swapper/022:39:560
17690992020,0cyclictest0-21swapper/021:34:550
17690992020,0cyclictest0-21swapper/019:34:550
17690992020,0cyclictest0-21swapper/019:24:540
17690992020,0cyclictest0-21swapper/000:09:570
17690992018,1cyclictest28247-21perf19:30:010
1769099200,20cyclictest0-21swapper/023:54:550
1769099200,20cyclictest0-21swapper/022:09:550
1769099200,0cyclictest0-21swapper/022:54:540
17700991919,0cyclictest0-21swapper/321:49:583
17700991919,0cyclictest0-21swapper/319:49:543
17700991917,1cyclictest0-21swapper/321:55:013
17697991919,0cyclictest9228-21apt-get23:19:582
17697991919,0cyclictest0-21swapper/223:59:542
17697991919,0cyclictest0-21swapper/220:04:562
17697991918,1cyclictest16606-21apt-get20:14:552
17697991918,1cyclictest0-21swapper/221:44:532
17695991919,0cyclictest29279-21apt-get23:59:541
17695991919,0cyclictest0-21swapper/123:34:531
17695991919,0cyclictest0-21swapper/123:24:551
17695991919,0cyclictest0-21swapper/121:59:551
17695991919,0cyclictest0-21swapper/121:29:531
17695991919,0cyclictest0-21swapper/120:24:541
17695991918,0cyclictest1783-21apt-get00:09:571
17695991918,0cyclictest0-21swapper/121:04:551
17690991919,0cyclictest0-21swapper/023:29:570
17690991919,0cyclictest0-21swapper/020:54:520
17690991919,0cyclictest0-21swapper/020:14:550
17690991918,1cyclictest0-21swapper/019:39:550
17690991918,0cyclictest0-21swapper/020:34:540
17690991917,1cyclictest0-21swapper/023:35:010
17700991818,0cyclictest0-21swapper/322:24:543
17700991818,0cyclictest0-21swapper/300:29:583
17697991818,0cyclictest20677-21apt-get21:29:532
17697991817,1cyclictest16580-21apt-get00:39:582
17697991817,1cyclictest0-21swapper/220:49:542
17697991816,1cyclictest606-21dbus-daemon19:25:012
17697991816,1cyclictest25472-21apt-get21:39:542
17697991815,2cyclictest9010-21latency_hist22:10:002
1769799180,17cyclictest28324-21awk22:50:012
17695991818,0cyclictest7013-21apt-get19:54:561
17695991817,0cyclictest19227-21apt-get23:39:541
1769599180,17cyclictest0-21swapper/100:34:561
17690991817,1cyclictest13-21ksoftirqd/023:44:550
17690991817,0cyclictest0-21swapper/023:14:550
17690991817,0cyclictest0-21swapper/021:44:530
17690991816,1cyclictest12672-21idleruntime-cro20:05:000
1769099180,16cyclictest313-21systemd-journal20:20:010
17700991717,0cyclictest15864-21apt-get21:19:553
17700991717,0cyclictest0-21swapper/323:44:553
17700991717,0cyclictest0-21swapper/323:39:553
17700991717,0cyclictest0-21swapper/321:09:573
17700991717,0cyclictest0-21swapper/320:54:533
17700991717,0cyclictest0-21swapper/320:24:543
17700991716,1cyclictest0-21swapper/300:14:553
17700991715,1cyclictest0-21swapper/321:20:013
17697991717,0cyclictest32736-21apt-get21:54:552
17697991717,0cyclictest0-21swapper/223:49:562
17697991717,0cyclictest0-21swapper/221:24:552
17697991717,0cyclictest0-21swapper/200:34:562
17695991717,0cyclictest0-21swapper/121:09:571
17695991717,0cyclictest0-21swapper/119:14:551
17695991716,0cyclictest27020-21apt-get19:29:541
17695991715,1cyclictest0-21swapper/119:50:001
17690991717,0cyclictest0-21swapper/021:14:550
17690991715,1cyclictest6727-21pmu-power22:05:010
17690991715,1cyclictest15005-21ptp4l-jitter21:15:020
1769099170,16cyclictest7950-21munin-plugin-st19:55:000
17700991616,0cyclictest0-21swapper/323:24:553
17700991616,0cyclictest0-21swapper/321:14:553
17700991615,1cyclictest0-21swapper/322:09:553
1770099160,15cyclictest0-21swapper/300:35:013
1770099160,11cyclictest0-21swapper/319:54:563
17697991614,1cyclictest0-21swapper/219:15:002
1769799160,7cyclictest600-21systemd-logind23:50:062
1769799160,15cyclictest0-21swapper/221:00:012
17695991616,0cyclictest1059-21apt-get20:49:541
17695991616,0cyclictest0-21swapper/121:50:011
17695991616,0cyclictest0-21swapper/100:24:541
17695991615,0cyclictest0-21swapper/120:29:541
1769599160,2cyclictest23819-21pmu-power22:40:021
17690991616,0cyclictest0-21swapper/023:04:570
17690991616,0cyclictest0-21swapper/022:14:550
17690991614,1cyclictest783-21snmpd00:15:230
1770099150,0cyclictest0-21swapper/321:59:553
17697991515,0cyclictest0-21swapper/223:24:552
17697991515,0cyclictest0-21swapper/222:19:562
17697991515,0cyclictest0-21swapper/220:54:522
17697991514,1cyclictest26273-21apt-get20:34:532
17695991515,0cyclictest0-21swapper/119:24:541
1769599150,0cyclictest0-21swapper/121:34:551
17690991515,0cyclictest12035-21apt-get23:24:550
17690991515,0cyclictest0-21swapper/020:24:540
17700991414,0cyclictest0-21swapper/320:40:363
17700991414,0cyclictest0-21swapper/300:21:103
17700991414,0cyclictest0-21swapper/300:15:143
1770099140,14cyclictest0-21swapper/323:00:483
1770099140,14cyclictest0-21swapper/300:30:253
1770099140,0cyclictest0-21swapper/323:05:083
1770099140,0cyclictest0-21swapper/320:30:313
17697991414,0cyclictest0-21swapper/223:05:102
1769799140,14cyclictest0-21swapper/223:40:182
1769799140,14cyclictest0-21swapper/222:35:022
1769799140,14cyclictest0-21swapper/222:21:212
1769799140,14cyclictest0-21swapper/220:20:422
1769799140,0cyclictest0-21swapper/220:15:122
17695991414,0cyclictest0-21swapper/120:05:211
1769599140,0cyclictest0-21swapper/122:34:261
1769599140,0cyclictest0-21swapper/121:20:171
1769599140,0cyclictest0-21swapper/100:25:211
1769099140,14cyclictest0-21swapper/020:45:090
1769099140,14cyclictest0-21swapper/020:25:080
1769099140,14cyclictest0-21swapper/019:43:310
1769099140,14cyclictest0-21swapper/019:10:510
1769099140,0cyclictest0-21swapper/022:30:100
1769799130,0cyclictest0-21swapper/219:25:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional