You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 18:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Mon Nov 17, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
991599280,21cyclictest0-21swapper/309:46:463
991599260,2cyclictest313-21systemd-journal12:35:003
991599260,21cyclictest31597-21sendmail-msp12:20:003
991599260,21cyclictest21326-21perf08:40:003
991599250,1cyclictest768-21runrttasks08:51:463
991599250,0cyclictest0-21swapper/312:11:453
9903992523,1cyclictest27479-21cron11:05:011
991599240,23cyclictest26763-21idleruntime-cro12:09:593
991599240,0cyclictest30029-21apt-get07:51:443
9907992422,1cyclictest18275-21apt-get10:46:472
990799240,23cyclictest0-21swapper/210:06:462
9907992321,1cyclictest18598-21munin-run09:39:592
990799230,23cyclictest0-21swapper/211:26:472
990799230,22cyclictest8767-21idleruntime-cro12:40:012
990799230,1cyclictest313-21systemd-journal08:20:002
9903992323,0cyclictest23788-21apt-get09:51:471
9903992321,1cyclictest25099-21cut11:00:001
990399230,22cyclictest4453-21awk08:05:001
9901992323,0cyclictest0-21swapper/011:36:440
9901992320,2cyclictest12102-21apt-get08:21:450
990199230,22cyclictest3399-21grep10:15:000
990199230,22cyclictest26760-21cron12:10:000
9915992221,1cyclictest14504-21apt-get08:26:473
991599220,21cyclictest5084-21apt-get11:26:473
991599220,21cyclictest24132-21munin-run08:45:013
991599220,1cyclictest31283-21cron08:59:593
991599220,1cyclictest19931-21basename07:30:003
9907992222,0cyclictest0-21swapper/210:26:482
990799220,21cyclictest32020-21cron07:55:012
990799220,21cyclictest14742-21munin-plugin-st07:20:002
990799220,21cyclictest0-21swapper/211:46:482
990799220,21cyclictest0-21swapper/211:00:002
990799220,21cyclictest0-21swapper/208:26:472
9903992220,1cyclictest6842-21dump-pmu-power08:10:001
9903992220,1cyclictest0-21swapper/112:25:001
9903992220,1cyclictest0-21swapper/110:10:001
990399220,21cyclictest23393-21perf09:50:001
990399220,21cyclictest0-21swapper/111:21:461
990399220,21cyclictest0-21swapper/110:31:471
990399220,21cyclictest0-21swapper/107:51:451
990399220,1cyclictest21310-21turbostat.cron08:40:001
990399220,0cyclictest0-21swapper/109:01:461
9901992221,1cyclictest0-21swapper/010:46:480
9901992221,0cyclictest21561-21apt-get12:00:020
9901992220,1cyclictest9476-21latency_hist11:34:590
9901992218,3cyclictest13-21ksoftirqd/010:00:020
990199220,22cyclictest0-21swapper/012:11:460
990199220,21cyclictest593-21dbus-daemon10:09:590
990199220,21cyclictest20667-21apt-get10:51:450
990199220,21cyclictest20318-21apt-get07:31:460
990199220,21cyclictest16899-21apt-get08:31:450
990199220,21cyclictest14181-21apt-get09:31:480
990199220,21cyclictest0-21swapper/012:21:450
990199220,21cyclictest0-21swapper/008:16:460
990199220,1cyclictest0-21swapper/010:36:480
9915992120,1cyclictest0-21swapper/307:11:463
9915992118,3cyclictest2687-21apt-get11:21:463
991599210,21cyclictest0-21swapper/309:26:453
991599210,20cyclictest9880-21apt-get11:36:463
991599210,20cyclictest2107-21apt-get09:06:463
991599210,20cyclictest0-21swapper/310:36:483
991599210,20cyclictest0-21swapper/310:31:483
991599210,20cyclictest0-21swapper/307:16:463
991599210,1cyclictest29904-21grep11:10:003
9907992121,0cyclictest0-21swapper/208:41:452
9907992120,1cyclictest19268-21/usr/sbin/munin08:36:452
9907992119,1cyclictest14364-21apt11:45:002
9907992118,2cyclictest8905-21latency_hist09:20:002
990799210,20cyclictest29307-21apt-get08:56:462
990799210,20cyclictest29000-21apt-get10:01:452
990799210,20cyclictest0-21swapper/210:16:452
990799210,0cyclictest22712-21apt-get07:36:432
9903992121,0cyclictest0-21swapper/111:51:451
9903992121,0cyclictest0-21swapper/108:31:451
9903992120,1cyclictest24760-21apt-get12:06:461
9903992119,1cyclictest17867-21awk10:45:001
9903992118,2cyclictest15466-21sendmail10:39:591
990399210,21cyclictest0-21swapper/111:11:451
990399210,21cyclictest0-21swapper/109:41:461
990399210,1cyclictest0-21swapper/111:56:471
990399210,1cyclictest0-21swapper/107:56:451
9901992121,0cyclictest0-21swapper/011:26:460
9901992120,1cyclictest0-21swapper/007:51:450
9901992119,1cyclictest21392-21apt-get09:46:460
990199210,21cyclictest0-21swapper/009:16:450
990199210,21cyclictest0-21swapper/008:06:440
990199210,21cyclictest0-21swapper/007:56:460
990199210,20cyclictest4503-21apt-get09:11:430
990199210,20cyclictest0-21swapper/007:46:440
990199210,0cyclictest0-21swapper/010:56:460
9915992020,0cyclictest0-21swapper/309:11:443
9915992019,1cyclictest28834-21dpkg10:00:023
9915992019,1cyclictest12354-21apt-get11:41:443
9915992018,1cyclictest8219-21munin-run10:25:013
9915992018,1cyclictest6901-21apt-get09:16:463
9915992018,1cyclictest17891-21grep10:45:003
991599200,20cyclictest0-21swapper/312:21:463
991599200,20cyclictest0-21swapper/310:51:463
991599200,20cyclictest0-21swapper/307:35:003
9907992020,0cyclictest0-21swapper/212:16:462
9907992020,0cyclictest0-21swapper/211:56:462
9907992020,0cyclictest0-21swapper/210:11:462
9907992020,0cyclictest0-21swapper/208:31:442
9907992020,0cyclictest0-21swapper/208:21:452
9907992020,0cyclictest0-21swapper/208:01:452
9907992020,0cyclictest0-21swapper/207:26:452
990799200,20cyclictest0-21swapper/210:21:462
990799200,20cyclictest0-21swapper/209:06:462
990799200,19cyclictest0-21swapper/211:51:452
990799200,0cyclictest0-21swapper/207:11:462
9903992020,0cyclictest0-21swapper/112:01:471
9903992020,0cyclictest0-21swapper/111:41:441
9903992020,0cyclictest0-21swapper/111:40:001
9903992020,0cyclictest0-21swapper/109:31:461
9903992019,0cyclictest4360-21apt-get12:31:461
9903992019,0cyclictest0-21swapper/107:26:441
9903992018,1cyclictest0-21swapper/110:25:011
990399200,20cyclictest0-21swapper/108:51:471
990399200,0cyclictest0-21swapper/112:36:481
990399200,0cyclictest0-21swapper/111:46:471
990399200,0cyclictest0-21swapper/109:16:461
9901992020,0cyclictest32120-21apt-get09:01:460
9901992020,0cyclictest0-21swapper/011:41:450
9901992020,0cyclictest0-21swapper/009:41:460
9901992020,0cyclictest0-21swapper/008:56:450
990199200,20cyclictest0-21swapper/010:16:450
990199200,1cyclictest0-21swapper/010:26:480
990199200,0cyclictest0-21swapper/009:36:460
990199200,0cyclictest0-21swapper/008:11:460
9915991919,0cyclictest19552-21apt-get11:56:473
9915991919,0cyclictest0-21swapper/312:36:483
9915991919,0cyclictest0-21swapper/311:51:453
9915991919,0cyclictest0-21swapper/309:41:463
9915991919,0cyclictest0-21swapper/308:01:463
9915991917,1cyclictest25803-21grep09:54:593
9915991917,1cyclictest18900-21idleruntime-cro08:35:003
991599190,19cyclictest0-21swapper/308:21:453
991599190,18cyclictest0-21swapper/309:01:543
9907991919,0cyclictest0-21swapper/212:21:462
9907991919,0cyclictest0-21swapper/211:01:452
9907991919,0cyclictest0-21swapper/209:31:472
9907991917,1cyclictest3942-21cron12:30:002
9907991917,1cyclictest0-21swapper/210:40:002
9907991915,3cyclictest2056-21awk08:00:012
9903991919,0cyclictest0-21swapper/108:26:461
9903991919,0cyclictest0-21swapper/108:21:441
9903991919,0cyclictest0-21swapper/108:11:471
9903991919,0cyclictest0-21swapper/107:46:441
9903991916,2cyclictest6519-21latency_hist09:15:001
9901991919,0cyclictest0-21swapper/011:01:450
9901991919,0cyclictest0-21swapper/010:41:480
9901991919,0cyclictest0-21swapper/008:41:450
9915991818,0cyclictest1963-21apt-get12:26:463
9915991818,0cyclictest0-21swapper/312:00:023
9915991818,0cyclictest0-21swapper/307:36:443
9915991816,1cyclictest27497-21dump-pmu-power11:05:003
9915991816,1cyclictest18595-21perf09:40:003
9915991815,3cyclictest0-21swapper/309:56:443
991599180,3cyclictest17222-21unixbench_singl07:21:543
991599180,17cyclictest0-21swapper/308:11:473
9907991818,0cyclictest0-21swapper/212:11:452
9907991818,0cyclictest0-21swapper/211:06:452
9907991818,0cyclictest0-21swapper/210:31:482
9907991818,0cyclictest0-21swapper/208:11:472
9907991818,0cyclictest0-21swapper/207:31:452
9907991817,0cyclictest26909-21apt-get08:51:472
9907991816,1cyclictest29648-21munin-run07:50:002
9903991818,0cyclictest0-21swapper/111:06:451
9903991818,0cyclictest0-21swapper/108:56:451
9903991816,1cyclictest26186-21apt-get09:56:451
9903991816,1cyclictest14708-21turbostat.cron07:19:591
9903991815,2cyclictest31540-21cron12:20:001
990399180,17cyclictest609-21polkitd10:20:011
9901991818,0cyclictest26909-21apt-get08:51:460
9901991818,0cyclictest11724-21apt-get09:26:450
9901991818,0cyclictest0-21swapper/009:56:450
9901991818,0cyclictest0-21swapper/009:51:480
9915991717,0cyclictest0-21swapper/311:46:483
9915991717,0cyclictest0-21swapper/310:11:463
9915991716,1cyclictest25235-21apt-get07:41:453
9915991714,2cyclictest11306-21cron09:25:003
991599170,5cyclictest0-21swapper/308:06:453
9907991717,0cyclictest0-21swapper/212:01:462
9907991717,0cyclictest0-21swapper/211:21:462
9907991717,0cyclictest0-21swapper/210:51:452
9907991717,0cyclictest0-21swapper/208:46:492
9907991717,0cyclictest0-21swapper/207:41:442
9907991716,0cyclictest0-21swapper/209:46:462
990799170,16cyclictest30445-21cpuspeed_turbos11:11:462
9903991717,0cyclictest0-21swapper/110:26:481
9903991717,0cyclictest0-21swapper/110:05:001
9903991717,0cyclictest0-21swapper/109:21:441
9903991717,0cyclictest0-21swapper/107:41:441
9903991717,0cyclictest0-21swapper/107:11:461
9903991714,2cyclictest3614-21unixbench_multi12:26:551
9903991714,2cyclictest18615-21grep09:40:001
990399170,0cyclictest0-21swapper/111:16:461
9901991717,0cyclictest0-21swapper/008:01:470
9901991716,1cyclictest0-21swapper/012:20:000
9901991716,0cyclictest0-21swapper/011:06:450
9901991715,1cyclictest593-21dbus-daemon12:29:590
9901991714,2cyclictest3869-21tr11:21:530
990199170,6cyclictest313-21systemd-journal08:50:010
9915991616,0cyclictest24516-21apt-get08:46:483
9915991616,0cyclictest0-21swapper/311:11:453
9915991616,0cyclictest0-21swapper/310:56:453
9915991616,0cyclictest0-21swapper/310:16:453
9915991614,1cyclictest0-21swapper/308:15:503
991599160,7cyclictest0-21swapper/309:31:473
9907991616,0cyclictest0-21swapper/212:06:462
9907991616,0cyclictest0-21swapper/211:36:452
9907991616,0cyclictest0-21swapper/209:00:012
9907991614,1cyclictest0-21swapper/210:41:492
990799160,15cyclictest4927-21diskstats09:11:462
9903991616,0cyclictest27156-21apt-get12:11:461
9903991616,0cyclictest0-21swapper/110:51:451
9903991616,0cyclictest0-21swapper/108:16:471
990399160,2cyclictest3075-21head10:11:551
990399160,15cyclictest22334-21perf07:35:001
9901991616,0cyclictest22712-21apt-get07:36:430
9901991616,0cyclictest0-21swapper/010:21:450
9901991616,0cyclictest0-21swapper/007:11:450
990199160,15cyclictest21649-21apt-config12:00:010
991599150,5cyclictest0-21swapper/311:31:473
991599150,14cyclictest0-21swapper/307:56:513
9907991515,0cyclictest0-21swapper/209:51:482
9907991515,0cyclictest0-21swapper/207:21:452
9907991514,1cyclictest11724-21apt-get09:26:452
9907991513,1cyclictest27593-21ls09:56:532
9901991515,0cyclictest0-21swapper/012:36:480
9901991515,0cyclictest0-21swapper/011:11:450
9901991515,0cyclictest0-21swapper/008:36:460
9901991512,2cyclictest3111-21needreboot09:06:490
991599140,14cyclictest0-21swapper/310:26:353
991599140,14cyclictest0-21swapper/310:05:393
991599140,14cyclictest0-21swapper/307:47:213
991599140,12cyclictest0-21swapper/310:46:483
9907991414,0cyclictest32675-21apt-get11:16:462
990799141,11cyclictest781-21lldpd11:30:272
990799140,14cyclictest0-21swapper/209:42:072
990799140,14cyclictest0-21swapper/209:23:132
990799140,14cyclictest0-21swapper/208:05:582
990799140,0cyclictest0-21swapper/212:30:492
990399140,14cyclictest0-21swapper/109:25:111
990399140,14cyclictest0-21swapper/107:35:141
990399140,0cyclictest0-21swapper/110:45:171
990399140,0cyclictest0-21swapper/108:48:531
990399140,0cyclictest0-21swapper/108:41:381
9901991414,0cyclictest61550irq/126-eno109:20:010
9901991414,0cyclictest0-21swapper/010:30:370
9901991414,0cyclictest0-21swapper/007:41:350
9901991414,0cyclictest0-21swapper/007:18:050
990199140,14cyclictest0-21swapper/012:30:210
990199140,14cyclictest0-21swapper/011:53:510
990199140,14cyclictest0-21swapper/007:25:090
990199140,14cyclictest0-21swapper/007:20:260
990199140,0cyclictest0-21swapper/011:15:170
990199140,0cyclictest0-21swapper/008:26:420
9915991313,0cyclictest0-21swapper/311:15:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional