You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 21:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu May 21, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
270785321290,4sleep10-21swapper/112:00:151
261048421190,2sleep30-21swapper/307:25:133
260328321030,8sleep23056-21PK-Backend07:05:262
139099770,61rtkit-daemon0-21swapper/007:05:260
260371327659,10sleep30-21swapper/307:08:013
260350827639,9sleep10-21swapper/107:05:301
26155342550,2sleep30-21swapper/307:40:013
26155342550,2sleep30-21swapper/307:40:003
26550832490,5sleep22655079-21ntp_kernel_pll_09:30:182
26640302480,2sleep33056-21PK-Backend09:55:173
26838472400,3sleep234-21ksoftirqd/210:50:192
27144992330,6sleep2341ksoftirqd/212:20:012
2604000992616,8cyclictest2627500-21spatcher08:11:112
260400099256,4cyclictest34-21ksoftirqd/210:14:592
26515352230,2sleep30-21swapper/309:20:173
2604005992320,1cyclictest3056-21PK-Backend08:08:413
260400599230,21cyclictest3056-21PK-Backend11:58:033
2603997992315,5cyclictest2660489-21needreboot09:45:181
2603997992314,7cyclictest883-21dbus-daemon12:30:011
2603997992314,7cyclictest2721323-21cron12:40:001
2603995992315,6cyclictest436-20systemd-journal09:20:000
260399599230,21cyclictest2612236-21expr07:30:120
26356552220,10sleep234-21ksoftirqd/208:35:132
2604005992220,1cyclictest3056-21PK-Backend11:23:033
2604005992220,1cyclictest3056-21PK-Backend08:53:103
260400599220,20cyclictest3056-21PK-Backend12:01:563
260400599220,20cyclictest3056-21PK-Backend10:15:093
260400599220,20cyclictest3056-21PK-Backend10:05:423
260400599220,20cyclictest3056-21PK-Backend08:14:103
2604000992214,6cyclictest2630690-21python308:20:202
2604000992213,7cyclictest883-21dbus-daemon07:11:112
2604000992211,10cyclictest2716108-21cron12:25:002
260400099220,20cyclictest2606025-21python307:15:022
2603997992216,4cyclictest2654904-21wc09:30:171
2603997992215,5cyclictest2643672-21latency_hist09:00:011
2603997992214,6cyclictest2681113-21cron10:45:011
2603997992214,6cyclictest2659345-21mktemp09:45:001
2603997992213,6cyclictest2622598-21cron07:59:591
260399799220,21cyclictest2714659-21apt12:20:111
139099220,3rtkit-daemon0-21swapper/007:58:530
26699992210,4sleep342-21ksoftirqd/310:10:183
2604005992119,1cyclictest3056-21PK-Backend10:35:023
2604005992119,1cyclictest3056-21PK-Backend07:58:283
2604005992119,1cyclictest0-21swapper/309:35:083
2604005992119,1cyclictest0-21swapper/308:23:023
2604005992118,2cyclictest3056-21PK-Backend08:59:453
2604005992118,2cyclictest3056-21PK-Backend08:25:183
260400599210,2cyclictest3056-21PK-Backend10:40:103
260400599210,20cyclictest3056-21PK-Backend11:12:013
260400599210,20cyclictest0-21swapper/310:56:053
260400599210,20cyclictest0-21swapper/308:34:443
260400599210,19cyclictest3056-21PK-Backend09:54:003
260400599210,19cyclictest3056-21PK-Backend08:48:033
2604000992117,3cyclictest2619137-21cron07:49:592
2604000992113,6cyclictest2628952-21python308:15:212
2604000992113,6cyclictest2614838-21threads07:35:192
2604000992113,6cyclictest2614838-21threads07:35:192
260400099210,20cyclictest0-21swapper/211:38:502
260399799217,12cyclictest2698518-21cron11:35:001
260399799214,15cyclictest2672467-21perf10:19:591
260399799212,6cyclictest2608245-21latency_hist07:20:011
2603997992119,1cyclictest0-21swapper/111:37:121
2603997992119,1cyclictest0-21swapper/111:28:541
2603997992119,1cyclictest0-21swapper/111:13:541
2603997992119,1cyclictest0-21swapper/110:10:471
2603997992119,1cyclictest0-21swapper/109:35:181
2603997992119,1cyclictest0-21swapper/109:29:371
2603997992118,2cyclictest0-21swapper/112:05:171
2603997992114,6cyclictest1-21systemd12:35:001
2603997992114,5cyclictest2626065-21cron08:10:011
2603997992113,6cyclictest2703962-21sshd11:50:081
2603997992112,7cyclictest436-20systemd-journal11:19:591
2603997992112,7cyclictest2628450-21df08:15:141
26997382200,3sleep33056-21PK-Backend11:35:233
2604005992018,1cyclictest3056-21PK-Backend12:14:203
2604005992018,1cyclictest3056-21PK-Backend12:05:483
2604005992018,1cyclictest3056-21PK-Backend11:45:403
2604005992018,1cyclictest3056-21PK-Backend10:51:123
2604005992018,1cyclictest3056-21PK-Backend10:47:203
2604005992018,1cyclictest3056-21PK-Backend09:40:153
2604005992018,1cyclictest3056-21PK-Backend09:07:233
2604005992018,1cyclictest3056-21PK-Backend08:01:393
2604005992018,1cyclictest0-21swapper/312:37:323
2604005992018,1cyclictest0-21swapper/312:15:203
2604005992018,1cyclictest0-21swapper/311:15:053
2604005992018,1cyclictest0-21swapper/311:05:043
2604005992018,1cyclictest0-21swapper/310:26:373
260400599200,1cyclictest3056-21PK-Backend07:48:143
260400599200,19cyclictest3056-21PK-Backend12:25:553
260400599200,19cyclictest3056-21PK-Backend07:41:373
260400599200,19cyclictest0-21swapper/309:45:223
260400599200,19cyclictest0-21swapper/307:12:103
260400599200,18cyclictest3056-21PK-Backend12:22:033
260400599200,18cyclictest3056-21PK-Backend07:51:053
2604000992018,1cyclictest0-21swapper/210:18:072
2604000992018,1cyclictest0-21swapper/210:01:382
260400099200,19cyclictest0-21swapper/211:47:572
260400099200,19cyclictest0-21swapper/211:10:522
260400099200,19cyclictest0-21swapper/211:09:512
260400099200,19cyclictest0-21swapper/210:55:522
260400099200,19cyclictest0-21swapper/208:43:162
260400099200,18cyclictest0-21swapper/207:52:332
260399799207,10cyclictest2254-21gdbus08:11:121
260399799202,15cyclictest2619419-21apt07:50:101
2603997992018,1cyclictest3056-21PK-Backend12:15:591
2603997992018,1cyclictest0-21swapper/110:30:071
2603997992018,1cyclictest0-21swapper/109:20:511
2603997992018,1cyclictest0-21swapper/109:00:511
2603997992018,1cyclictest0-21swapper/108:37:391
2603997992018,1cyclictest0-21swapper/108:20:211
2603997992018,1cyclictest0-21swapper/108:00:281
2603997992018,1cyclictest0-21swapper/107:22:081
2603997992017,2cyclictest0-21swapper/111:48:061
2603997992014,4cyclictest2663411-21grep09:55:111
2603997992013,4cyclictest2638497-21idleruntime-cro08:45:001
260399799201,13cyclictest2666732-21apt-get10:05:041
260399799200,19cyclictest0-21swapper/110:37:511
260399799200,19cyclictest0-21swapper/107:40:541
2603995992018,1cyclictest0-21swapper/010:37:160
2603995992018,1cyclictest0-21swapper/009:33:080
2603995992018,1cyclictest0-21swapper/008:46:510
2603995992014,4cyclictest2156-21gnome-shell09:40:090
2603995992013,5cyclictest436-20systemd-journal09:45:080
2603995992013,5cyclictest2633633-21expr08:30:110
2603995992012,6cyclictest436-20systemd-journal09:55:000
2603995992012,6cyclictest2719865-21apt12:35:110
260399599200,1cyclictest3056-21PK-Backend08:20:210
260399599200,1cyclictest0-21swapper/012:08:550
260399599200,19cyclictest0-21swapper/012:32:210
260399599200,19cyclictest0-21swapper/011:52:250
260399599200,19cyclictest0-21swapper/011:29:470
260399599200,19cyclictest0-21swapper/011:24:400
260399599200,19cyclictest0-21swapper/011:19:340
260399599200,19cyclictest0-21swapper/010:30:530
260399599200,19cyclictest0-21swapper/008:58:330
260399599200,19cyclictest0-21swapper/008:41:440
260399599200,19cyclictest0-21swapper/007:12:070
26907582190,3sleep00-21swapper/011:10:180
2604005991917,1cyclictest3056-21PK-Backend11:40:473
2604005991917,1cyclictest3056-21PK-Backend11:26:063
2604005991917,1cyclictest3056-21PK-Backend11:03:013
2604005991916,2cyclictest3056-21PK-Backend11:52:223
260400599190,1cyclictest3056-21PK-Backend08:43:573
2604000991917,1cyclictest0-21swapper/212:04:052
2604000991917,1cyclictest0-21swapper/211:52:232
2604000991917,1cyclictest0-21swapper/211:31:342
2604000991917,1cyclictest0-21swapper/211:28:442
2604000991917,1cyclictest0-21swapper/210:09:212
2604000991917,1cyclictest0-21swapper/209:16:092
2604000991917,1cyclictest0-21swapper/209:12:022
2604000991917,1cyclictest0-21swapper/207:57:342
2604000991912,6cyclictest2717459-21awk12:25:202
2604000991911,6cyclictest2692435-21memory11:15:162
260400099190,1cyclictest0-21swapper/210:27:542
260400099190,18cyclictest0-21swapper/212:39:142
260400099190,18cyclictest0-21swapper/208:30:392
260399799198,8cyclictest436-20systemd-journal07:34:591
2603997991917,1cyclictest0-21swapper/111:58:491
2603997991917,1cyclictest0-21swapper/111:23:121
2603997991916,2cyclictest3056-21PK-Backend10:56:431
2603997991915,3cyclictest2631955-21hddtemp_smartct08:25:141
260399799190,1cyclictest0-21swapper/110:20:331
260399799190,1cyclictest0-21swapper/108:47:591
260399799190,1cyclictest0-21swapper/107:45:271
260399799190,1cyclictest0-21swapper/107:10:191
260399799190,18cyclictest3056-21PK-Backend07:37:021
260399799190,18cyclictest3056-21PK-Backend07:37:021
260399799190,18cyclictest0-21swapper/109:11:521
2603995991917,1cyclictest0-21swapper/012:13:480
2603995991917,1cyclictest0-21swapper/011:36:450
2603995991917,1cyclictest0-21swapper/010:46:420
2603995991917,1cyclictest0-21swapper/009:55:450
2603995991917,1cyclictest0-21swapper/009:00:500
2603995991917,1cyclictest0-21swapper/007:52:160
2603995991917,1cyclictest0-21swapper/007:24:110
2603995991912,5cyclictest2713654-21cat12:15:190
2603995991912,5cyclictest2701551-21cat11:40:210
260399599190,1cyclictest0-21swapper/010:51:350
260399599190,1cyclictest0-21swapper/010:20:250
260399599190,1cyclictest0-21swapper/008:25:550
260399599190,1cyclictest0-21swapper/008:07:090
260399599190,1cyclictest0-21swapper/008:01:140
260399599190,18cyclictest0-21swapper/012:27:270
260399599190,18cyclictest0-21swapper/012:02:130
260399599190,18cyclictest0-21swapper/011:04:260
260399599190,18cyclictest0-21swapper/009:11:450
260399599190,18cyclictest0-21swapper/009:09:290
260399599190,18cyclictest0-21swapper/008:52:580
2604005991816,1cyclictest3056-21PK-Backend09:13:593
2604005991816,1cyclictest3056-21PK-Backend09:04:043
2604005991816,1cyclictest3056-21PK-Backend08:17:273
2604005991816,1cyclictest0-21swapper/307:16:503
2604005991811,5cyclictest2654870-21latency09:30:163
2604000991816,1cyclictest3056-21PK-Backend12:30:212
2604000991816,1cyclictest0-21swapper/212:11:232
2604000991816,1cyclictest0-21swapper/211:21:142
2604000991816,1cyclictest0-21swapper/210:24:502
2604000991816,1cyclictest0-21swapper/208:57:572
2604000991816,1cyclictest0-21swapper/207:28:132
2604000991815,2cyclictest2664755-21munin-plugin-st10:00:012
2604000991813,3cyclictest2616054-21sh07:40:132
2604000991812,5cyclictest2677059-21python310:30:222
2604000991811,5cyclictest2659558-21cut09:45:002
260400099180,17cyclictest0-21swapper/208:27:352
260399799187,10cyclictest2688087-21unin-run11:05:011
260399799182,11cyclictest1-21systemd12:10:351
2603997991816,1cyclictest0-21swapper/110:27:161
2603997991813,3cyclictest2661726-21grep09:50:131
2603997991813,3cyclictest1095-21runrttasks08:30:111
2603997991812,4cyclictest2156-21gnome-shell11:41:081
2603995991816,1cyclictest0-21swapper/010:09:450
2603995991816,1cyclictest0-21swapper/008:18:050
2603995991816,1cyclictest0-21swapper/007:40:120
2603995991816,1cyclictest0-21swapper/007:27:150
2603995991814,3cyclictest13-21ksoftirqd/011:10:010
2603995991813,4cyclictest13-21ksoftirqd/009:20:190
2603995991811,5cyclictest2608170-21sed07:20:010
2603995991811,5cyclictest13-21ksoftirqd/010:00:010
260399599181,13cyclictest2613865-21runrttasks07:35:020
260399599181,13cyclictest2613865-21runrttasks07:35:020
2603995991810,6cyclictest436-20systemd-journal10:45:010
260399599180,6cyclictest0-21swapper/009:29:090
2604005991715,1cyclictest3056-21PK-Backend12:33:403
260400599170,15cyclictest2635762-21irqstats08:35:153
2604000991715,1cyclictest0-21swapper/210:43:212
2604000991715,1cyclictest0-21swapper/210:37:402
2604000991715,1cyclictest0-21swapper/209:21:022
2604000991715,1cyclictest0-21swapper/208:53:512
2604000991715,1cyclictest0-21swapper/208:06:472
2604000991712,4cyclictest978-21snmpd09:27:302
2604000991710,6cyclictest2640245-21systemd08:50:012
260399799173,8cyclictest2664757-21apt-get10:00:041
2603997991715,1cyclictest0-21swapper/111:09:141
2603997991712,3cyclictest2683323-21cstates10:50:121
2603997991710,5cyclictest2646596-21ls09:05:221
260399599179,6cyclictest436-20systemd-journal11:00:000
260399599179,6cyclictest2706304-21memory11:55:170
2603995991713,3cyclictest2627078-21ntp_offset08:10:180
2603995991711,5cyclictest2702510-21expr11:45:130
2603995991710,6cyclictest2697247-21grep11:30:110
2603995991710,5cyclictest2670249-21cat10:10:210
2604005991614,1cyclictest3056-21PK-Backend10:03:393
2604005991614,1cyclictest3056-21PK-Backend07:30:163
2604000991614,1cyclictest0-21swapper/212:07:432
2604000991614,1cyclictest0-21swapper/209:08:582
2604000991613,2cyclictest3056-21PK-Backend09:45:292
2604000991611,4cyclictest2706746-21cat11:55:212
2604000991611,4cyclictest2662040-21cut09:50:182
2604000991611,3cyclictest2688059-21latency_hist11:05:022
260399799162,9cyclictest2681395-21apt10:45:091
260399799161,12cyclictest2640485-21apt08:50:111
260399599162,12cyclictest2675932-21munin-plugin-st10:29:590
2603995991614,1cyclictest0-21swapper/008:38:400
2603995991612,3cyclictest2656938-21rt-features09:35:200
2603995991611,3cyclictest2715635-21cat12:20:210
2603995991610,4cyclictest2619262-21wc07:50:010
2604005991513,1cyclictest3056-21PK-Backend11:33:023
2604005991513,1cyclictest3056-21PK-Backend07:22:393
2604005991512,2cyclictest3056-21PK-Backend10:30:573
2604005991512,2cyclictest3056-21PK-Backend09:28:443
2604005991510,4cyclictest42-21ksoftirqd/310:25:003
260400099159,4cyclictest2657552-21sh09:39:592
260400099152,2cyclictest141rcu_preempt07:22:502
2604000991513,1cyclictest0-21swapper/211:41:412
2604000991510,3cyclictest2681541-21cpuspeed_turbos10:45:112
2604000991510,3cyclictest2623825-21xargs08:00:222
260399799158,4cyclictest2649639-21kernelversion09:15:161
2603997991510,4cyclictest2610591-21basename07:25:161
260399599159,4cyclictest2672577-21cat10:20:010
2604005991412,1cyclictest3056-21PK-Backend09:19:193
260400099141,11cyclictest2612770-21python307:30:212
260400099140,1cyclictest0-21swapper/209:03:512
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional