You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 00:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #4, slot #1

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1s.osadl.org (updated Tue May 19, 2026 12:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1811421080,4sleep017707-21/usr/sbin/munin09:22:420
1811421080,4sleep017707-21/usr/sbin/munin09:22:420
26952870,2sleep20-21swapper/211:32:522
26952870,2sleep20-21swapper/211:32:522
25852810,2sleep02586-21/usr/sbin/munin10:07:440
25852810,2sleep02586-21/usr/sbin/munin10:07:440
25852810,2sleep02586-21/usr/sbin/munin10:07:440
263352760,4sleep026336-21latency_hist11:12:360
263352760,4sleep026336-21latency_hist11:12:360
278222610,2chrt27018-21/usr/sbin/munin09:47:500
278222610,2chrt27018-21/usr/sbin/munin09:47:500
278222610,2chrt27018-21/usr/sbin/munin09:47:500
282012540,3sleep30-21swapper/312:42:443
282012540,3sleep30-21swapper/312:42:443
282012540,3sleep30-21swapper/312:42:443
176682540,2sleep30-21swapper/309:22:373
176682540,2sleep30-21swapper/309:22:373
300972520,2sleep00-21swapper/011:22:330
300972520,2sleep00-21swapper/011:22:330
300972520,2sleep00-21swapper/011:22:330
213512500,3sleep00-21swapper/012:22:510
213512500,3sleep00-21swapper/012:22:510
146399411,11cyclictest15186-21latency_hist10:42:360
146399411,11cyclictest15186-21latency_hist10:42:360
1463994016,22cyclictest17616-21kernelversion10:47:450
1463994016,22cyclictest17616-21kernelversion10:47:450
1463994016,22cyclictest17616-21kernelversion10:47:450
1463994015,23cyclictest0-21swapper/011:12:510
1463994015,23cyclictest0-21swapper/011:12:510
146399401,13cyclictest15253-21ntpq07:47:490
146399401,13cyclictest15253-21ntpq07:47:490
1463993915,21cyclictest0-21swapper/011:37:500
1463993915,21cyclictest0-21swapper/011:37:500
146399391,12cyclictest6998-21/usr/sbin/munin07:27:450
146399391,12cyclictest6998-21/usr/sbin/munin07:27:450
1463993814,22cyclictest0-21swapper/010:44:460
1463993814,22cyclictest0-21swapper/010:44:460
1463993814,22cyclictest0-21swapper/010:44:460
1463993813,2cyclictest0-21swapper/009:37:440
1463993813,2cyclictest0-21swapper/009:37:440
146399381,11cyclictest14024-21ls09:12:410
1463993810,3cyclictest0-21swapper/008:24:360
1463993810,3cyclictest0-21swapper/008:24:360
1464993728,7cyclictest14656-21netstat09:12:471
146399377,6cyclictest3312-21cut11:37:370
146399377,6cyclictest3312-21cut11:37:370
146399371,13cyclictest9790-21tune2fs10:27:430
146399371,13cyclictest9790-21tune2fs10:27:430
146399371,13cyclictest9790-21tune2fs10:27:430
146399371,12cyclictest749-21runrttasks08:12:180
146399371,12cyclictest749-21runrttasks08:12:180
82882360,2sleep18395-21bingrep10:22:451
82882360,2sleep18395-21bingrep10:22:451
146399362,10cyclictest749-21runrttasks11:52:520
146399362,10cyclictest749-21runrttasks11:52:520
146399362,10cyclictest16529-21/usr/sbin/munin12:12:530
146399362,10cyclictest16529-21/usr/sbin/munin12:12:530
146399361,9cyclictest5110-21/usr/sbin/munin07:22:420
146399361,12cyclictest30647-21grep09:57:360
146399361,12cyclictest30647-21grep09:57:360
146399361,11cyclictest18708-21grep12:17:440
146399361,10cyclictest14678-21systemd-journal11:47:020
146399361,10cyclictest14678-21systemd-journal11:47:020
146399360,12cyclictest19956-21latency_hist08:02:310
146399360,12cyclictest19956-21latency_hist08:02:310
1464993527,6cyclictest5246-21netstat08:47:471
1464993527,6cyclictest5246-21netstat08:47:471
1464993527,6cyclictest14391-21threads10:37:531
1464993527,6cyclictest14391-21threads10:37:531
1463993516,17cyclictest0-21swapper/012:37:470
1463993515,18cyclictest0-21swapper/009:57:500
146399351,11cyclictest749-21runrttasks10:03:500
146399351,11cyclictest749-21runrttasks10:03:500
146399351,11cyclictest749-21runrttasks10:03:500
146399351,11cyclictest6408-21/usr/sbin/munin08:52:440
146399351,11cyclictest6408-21/usr/sbin/munin08:52:440
146399351,11cyclictest32448-21idleruntime11:27:430
146399351,11cyclictest29464-21/usr/sbin/munin08:27:440
146399351,11cyclictest29464-21/usr/sbin/munin08:27:440
146399351,11cyclictest12509-21latency_hist07:42:340
146399351,11cyclictest12509-21latency_hist07:42:340
146399351,11cyclictest11268-21sed11:57:450
146399351,11cyclictest11268-21sed11:57:450
146399351,10cyclictest9380-21grep07:32:440
146399351,10cyclictest9380-21grep07:32:440
146399351,10cyclictest31324-21/usr/sbin/munin08:32:420
146399351,10cyclictest31324-21/usr/sbin/munin08:32:420
146399351,10cyclictest27674-21/usr/sbin/munin12:42:410
146399351,10cyclictest27674-21/usr/sbin/munin12:42:410
146399351,10cyclictest27674-21/usr/sbin/munin12:42:410
146399351,10cyclictest14678-21systemd-journal09:12:300
293472340,2chrt0-21swapper/309:52:443
293472340,2chrt0-21swapper/309:52:443
1464993417,3cyclictest90-1pr/ttyS011:32:511
1464993417,3cyclictest90-1pr/ttyS011:32:511
146399341,13cyclictest31293-21ls11:22:510
146399341,13cyclictest31293-21ls11:22:510
146399341,13cyclictest20796-21grep09:27:520
146399341,13cyclictest20796-21grep09:27:520
146399341,12cyclictest749-21runrttasks12:02:450
146399341,12cyclictest749-21runrttasks12:02:450
146399341,12cyclictest749-21runrttasks12:02:450
146399341,11cyclictest749-21runrttasks08:18:160
146399341,11cyclictest749-21runrttasks08:18:160
146399341,11cyclictest18985-21/usr/sbin/munin10:52:490
146399341,11cyclictest18985-21/usr/sbin/munin10:52:490
146399341,10cyclictest8933-21needreboot08:57:480
146399341,10cyclictest8933-21needreboot08:57:480
146399341,10cyclictest8933-21needreboot08:57:480
105032340,8sleep110507-21gltestperf09:02:451
105032340,8sleep110507-21gltestperf09:02:451
1464993326,5cyclictest27774-21fschecks_count08:22:421
1464993326,5cyclictest27774-21fschecks_count08:22:421
146399337,6cyclictest21088-21grep10:57:430
146399331,12cyclictest749-21runrttasks08:41:140
146399331,12cyclictest749-21runrttasks08:41:140
146399331,12cyclictest24892-21awk12:32:520
146399331,12cyclictest24892-21awk12:32:520
146399331,11cyclictest25487-21latency_hist08:17:350
146399331,11cyclictest25487-21latency_hist08:17:350
1464993226,4cyclictest13708-21cron09:12:301
1464993225,5cyclictest19243-21fschecks_time10:52:431
1464993225,5cyclictest19243-21fschecks_time10:52:431
1464993223,7cyclictest28588-21fschecks_time11:17:441
1464993223,7cyclictest28588-21fschecks_time11:17:441
1464993223,7cyclictest28588-21fschecks_time11:17:441
1464993223,7cyclictest25881-21netstat09:42:471
1464993223,7cyclictest25881-21netstat09:42:471
1464993223,7cyclictest25881-21netstat09:42:471
146399321,11cyclictest7883-21fschecks_count10:22:420
146399321,11cyclictest7883-21fschecks_count10:22:420
146399321,10cyclictest749-21runrttasks10:14:530
146399321,10cyclictest749-21runrttasks10:14:530
146399321,10cyclictest749-21runrttasks09:20:210
146399321,10cyclictest749-21runrttasks09:20:210
1464993124,5cyclictest5031-21latency_hist11:42:301
1464993124,5cyclictest5031-21latency_hist11:42:301
1463993125,4cyclictest545-21in:imuxsock12:32:320
1463993125,4cyclictest545-21in:imuxsock12:32:320
1463993125,4cyclictest545-21in:imuxsock12:32:320
146399311,13cyclictest13933-21vmstat07:42:530
146399311,13cyclictest13933-21vmstat07:42:530
146399311,11cyclictest10846-21needreboot09:02:480
146399311,11cyclictest10846-21needreboot09:02:480
50192300,2sleep00-21swapper/008:47:440
50192300,2sleep00-21swapper/008:47:440
1464993024,5cyclictest27377-21latency_hist08:22:291
1464993024,5cyclictest27377-21latency_hist08:22:291
1464993023,5cyclictest2410-21gltestperf10:07:421
1464993023,5cyclictest11114-21gltestperf07:37:431
1464993023,5cyclictest11114-21gltestperf07:37:431
1464993022,6cyclictest649-21date08:37:351
1464993022,6cyclictest649-21date08:37:351
1464993021,3cyclictest181rcu_preempt07:52:301
1464993021,3cyclictest181rcu_preempt07:52:301
1464993017,2cyclictest181rcu_preempt09:32:471
1464993017,2cyclictest181rcu_preempt09:32:471
1464993014,3cyclictest90-1pr/ttyS012:07:491
1464993014,3cyclictest90-1pr/ttyS012:07:491
1464993014,3cyclictest90-1pr/ttyS012:07:491
1464993014,3cyclictest181rcu_preempt10:32:481
1464993014,3cyclictest181rcu_preempt10:32:481
1464993012,5cyclictest271rcuc/111:09:481
1464993012,5cyclictest271rcuc/111:09:481
146399301,10cyclictest749-21runrttasks08:46:310
146399301,10cyclictest749-21runrttasks08:46:310
146399292,9cyclictest7952-21grep11:47:490
146399292,9cyclictest7952-21grep11:47:490
146399292,9cyclictest14634-21/usr/sbin/munin12:07:510
146399292,9cyclictest14634-21/usr/sbin/munin12:07:510
146399292,9cyclictest14634-21/usr/sbin/munin12:07:510
146499289,5cyclictest281ktimers/109:40:291
146499289,5cyclictest281ktimers/109:40:291
146499288,5cyclictest281ktimers/112:16:001
146499288,5cyclictest281ktimers/112:16:001
1464992823,3cyclictest559-21cut10:02:431
1464992823,3cyclictest559-21cut10:02:431
1464992823,3cyclictest0-21swapper/107:32:291
1464992823,3cyclictest0-21swapper/107:32:291
1464992822,4cyclictest3735-21latency_hist10:12:421
1464992822,4cyclictest3735-21latency_hist10:12:421
1464992822,4cyclictest3735-21latency_hist10:12:421
146499282,11cyclictest29-21ksoftirqd/111:01:141
1464992819,7cyclictest12961-21gltestperf07:42:411
1464992819,7cyclictest12961-21gltestperf07:42:411
1464992814,3cyclictest0-21swapper/110:47:411
1464992813,2cyclictest181rcu_preempt08:52:491
1464992813,2cyclictest181rcu_preempt08:52:491
146499275,4cyclictest281ktimers/110:27:431
146499275,4cyclictest281ktimers/110:27:431
146499275,4cyclictest281ktimers/110:27:431
146499275,4cyclictest281ktimers/108:12:501
146499275,4cyclictest281ktimers/108:12:501
146499274,20cyclictest281ktimers/107:57:561
146499274,20cyclictest281ktimers/107:57:561
1464992721,4cyclictest1462-21cyclictest08:57:521
1464992721,4cyclictest1462-21cyclictest08:57:521
1464992721,4cyclictest1462-21cyclictest08:57:521
1464992720,6cyclictest21176-21timerandwakeup08:02:511
1464992720,6cyclictest21176-21timerandwakeup08:02:511
1464992720,6cyclictest21176-21timerandwakeup08:02:511
1464992720,5cyclictest30882-21needreboot11:22:471
1464992720,5cyclictest30882-21needreboot11:22:471
1464992720,5cyclictest30882-21needreboot11:22:471
1464992718,4cyclictest181rcu_preempt12:17:471
146499271,6cyclictest271rcuc/109:50:381
146499271,6cyclictest271rcuc/109:50:381
146499271,6cyclictest271rcuc/109:50:381
1464992716,2cyclictest181rcu_preempt10:21:371
1464992714,3cyclictest181rcu_preempt12:07:411
1464992713,3cyclictest181rcu_preempt12:00:511
1464992713,3cyclictest181rcu_preempt12:00:511
1464992713,3cyclictest181rcu_preempt09:32:311
1464992713,3cyclictest181rcu_preempt09:32:311
1464992713,3cyclictest181rcu_preempt07:56:551
1464992713,3cyclictest181rcu_preempt07:56:551
1464992713,2cyclictest181rcu_preempt12:42:291
1464992713,2cyclictest181rcu_preempt12:42:291
1464992713,2cyclictest181rcu_preempt12:42:291
1464992713,2cyclictest181rcu_preempt12:27:481
1464992713,2cyclictest181rcu_preempt12:27:481
1464992713,2cyclictest181rcu_preempt12:27:481
1463992722,3cyclictest0-21swapper/007:52:450
1463992722,3cyclictest0-21swapper/007:52:450
146499268,8cyclictest29-21ksoftirqd/112:22:531
146499268,8cyclictest29-21ksoftirqd/112:22:531
1464992619,6cyclictest605-21switchtime11:27:511
1464992619,6cyclictest605-21switchtime11:27:511
1464992619,5cyclictest5871-21ntp_offset07:22:481
1464992619,2cyclictest181rcu_preempt11:47:441
1464992619,2cyclictest181rcu_preempt11:47:441
1464992618,6cyclictest18239-21sort09:22:431
1464992618,6cyclictest18239-21sort09:22:431
1464992618,2cyclictest181rcu_preempt12:07:321
1464992618,2cyclictest181rcu_preempt12:07:321
1464992616,2cyclictest181rcu_preempt11:12:501
1464992616,2cyclictest181rcu_preempt11:12:501
1464992613,8cyclictest90-1pr/ttyS008:41:571
1464992613,8cyclictest90-1pr/ttyS008:41:571
1464992613,3cyclictest181rcu_preempt10:42:431
1464992613,3cyclictest181rcu_preempt10:42:431
1464992613,2cyclictest181rcu_preempt11:42:411
1464992613,2cyclictest181rcu_preempt11:42:411
1464992613,2cyclictest181rcu_preempt09:58:301
1464992611,8cyclictest90-1pr/ttyS009:56:051
1464992611,8cyclictest90-1pr/ttyS009:56:051
1464992611,2cyclictest181rcu_preempt08:07:401
1464992611,2cyclictest181rcu_preempt08:07:401
1464992610,4cyclictest181rcu_preempt07:32:421
1464992610,4cyclictest181rcu_preempt07:32:421
1464992610,3cyclictest181rcu_preempt11:52:421
1464992610,3cyclictest181rcu_preempt11:52:421
1463992620,4cyclictest749-21runrttasks10:17:420
132672260,2chrt0-21swapper/309:07:533
146499254,12cyclictest29-21ksoftirqd/110:47:491
146499254,12cyclictest29-21ksoftirqd/110:47:491
146499254,12cyclictest29-21ksoftirqd/110:47:491
1464992517,2cyclictest181rcu_preempt12:37:061
1464992517,2cyclictest181rcu_preempt12:37:061
1464992517,2cyclictest181rcu_preempt12:37:061
1464992512,2cyclictest181rcu_preempt11:05:231
1464992512,2cyclictest181rcu_preempt11:05:231
1464992512,2cyclictest181rcu_preempt11:05:231
146499251,10cyclictest3132-21interrupts08:42:441
146499251,10cyclictest3132-21interrupts08:42:441
1463992518,5cyclictest1435-21latency_hist11:32:380
1463992518,5cyclictest1435-21latency_hist11:32:380
146999241,17cyclictest18404-21chrt10:47:553
146999241,17cyclictest18404-21chrt10:47:553
146999241,17cyclictest18404-21chrt10:47:553
1464992413,2cyclictest181rcu_preempt09:17:411
1464992413,2cyclictest181rcu_preempt09:17:411
1463992419,3cyclictest0-21swapper/009:32:450
1463992419,3cyclictest0-21swapper/009:32:450
1463992418,4cyclictest23004-21fschecks_time11:02:440
1463992418,4cyclictest23004-21fschecks_time11:02:440
1463992418,4cyclictest23004-21fschecks_time11:02:440
146499236,15cyclictest281ktimers/108:27:421
146499236,15cyclictest281ktimers/108:27:421
1464992310,4cyclictest29-21ksoftirqd/110:12:511
1464992310,4cyclictest29-21ksoftirqd/110:12:511
1463992318,3cyclictest0-21swapper/008:02:480
1463992318,3cyclictest0-21swapper/008:02:480
1463992318,3cyclictest0-21swapper/008:02:480
255502220,2chrt0-21swapper/311:07:533
255502220,2chrt0-21swapper/311:07:533
1469992213,6cyclictest21990-21chrt08:07:393
1469992213,6cyclictest21990-21chrt08:07:393
1463992218,2cyclictest0-21swapper/009:42:470
1463992218,2cyclictest0-21swapper/009:42:470
1463992218,2cyclictest0-21swapper/009:42:470
1463992216,3cyclictest0-21swapper/010:32:500
1463992216,3cyclictest0-21swapper/010:32:500
262122210,3sleep30-21swapper/312:37:463
262122210,3sleep30-21swapper/312:37:463
262122210,3sleep30-21swapper/312:37:463
32122200,2sleep30-21swapper/311:37:323
32122200,2sleep30-21swapper/311:37:323
1469991812,4cyclictest109362sleep309:02:523
1469991812,4cyclictest109362sleep309:02:523
146999181,13cyclictest23315-21taskset09:37:403
146999181,13cyclictest23315-21taskset09:37:403
1469991810,6cyclictest3878-21chrt08:42:533
1469991810,6cyclictest3878-21chrt08:42:533
95542170,3sleep30-21swapper/307:32:463
95542170,3sleep30-21swapper/307:32:463
146999173,9cyclictest0-21swapper/309:28:223
146999173,9cyclictest0-21swapper/309:28:223
1469991712,3cyclictest168122sleep307:52:453
1469991712,3cyclictest168122sleep307:52:453
146999171,12cyclictest26841-21chrt11:12:433
146999171,12cyclictest26841-21chrt11:12:433
146999171,12cyclictest24654-21taskset12:32:473
146999171,12cyclictest24654-21taskset12:32:473
146999171,11cyclictest27872-21taskset08:22:433
146999171,11cyclictest27872-21taskset08:22:433
146999171,11cyclictest2539-21chrt10:07:433
146999171,11cyclictest2539-21chrt10:07:433
146999171,11cyclictest2539-21chrt10:07:433
146999171,11cyclictest20228-21chrt08:02:413
146999171,11cyclictest0-21swapper/308:37:403
146999171,11cyclictest0-21swapper/308:37:403
146799171,13cyclictest23775-21chrt08:12:372
146799171,13cyclictest23775-21chrt08:12:372
214472160,3sleep20-21swapper/209:32:412
146999163,9cyclictest0-21swapper/310:39:223
146999163,9cyclictest0-21swapper/310:39:223
146999161,11cyclictest0-21swapper/312:32:443
146999161,11cyclictest0-21swapper/309:42:423
146999161,11cyclictest0-21swapper/309:42:423
146999161,11cyclictest0-21swapper/309:42:423
146799168,6cyclictest351rcuc/211:07:572
146799168,6cyclictest351rcuc/211:07:572
146799161,9cyclictest0-21swapper/210:42:402
146799161,9cyclictest0-21swapper/210:42:402
146799161,9cyclictest0-21swapper/210:42:402
146799161,12cyclictest0-21swapper/209:27:402
146799161,12cyclictest0-21swapper/209:27:402
146799161,10cyclictest9604-21chrt10:27:402
146799161,10cyclictest9604-21chrt10:27:402
146999153,9cyclictest0-21swapper/310:44:233
146999153,9cyclictest0-21swapper/310:44:233
146999153,9cyclictest0-21swapper/310:44:233
146999152,9cyclictest0-21swapper/311:39:223
146999152,9cyclictest0-21swapper/311:39:223
146999151,9cyclictest7250-21taskset11:47:413
146999151,9cyclictest18911-21chrt07:57:463
146999151,9cyclictest18911-21chrt07:57:463
146999151,12cyclictest25164-21taskset08:13:223
146999151,12cyclictest25164-21taskset08:13:223
146999151,11cyclictest12723-21chrt10:32:523
146999151,11cyclictest12723-21chrt10:32:523
146999151,11cyclictest0-21swapper/310:57:463
146999151,10cyclictest22214-21chrt12:27:413
146999151,10cyclictest22214-21chrt12:27:413
146999151,10cyclictest0-21swapper/307:27:433
146999151,10cyclictest0-21swapper/307:27:433
146799159,3cyclictest282792chrt11:17:412
146799159,3cyclictest282792chrt11:17:412
146799151,7cyclictest0-21swapper/210:52:392
146799151,7cyclictest0-21swapper/210:52:392
146799151,7cyclictest0-21swapper/210:52:392
146799151,12cyclictest5861-21taskset08:48:222
146799151,12cyclictest5861-21taskset08:48:222
146799151,12cyclictest28057-21chrt09:47:522
146799151,12cyclictest28057-21chrt09:47:522
146799151,12cyclictest28057-21chrt09:47:522
146799151,12cyclictest0-21swapper/212:07:322
146799151,12cyclictest0-21swapper/212:07:322
146799151,12cyclictest0-21swapper/212:07:322
146799151,11cyclictest7626-21chrt07:27:462
146799151,11cyclictest7626-21chrt07:27:462
146799151,11cyclictest7126-21chrt10:17:552
146799151,11cyclictest6488-21chrt08:52:412
146799151,11cyclictest6488-21chrt08:52:412
146799151,11cyclictest4115-21taskset10:12:402
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional