You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-05 - 21:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Jun 05, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
279324621360,3sleep00-21swapper/012:25:170
274197321250,3sleep20-21swapper/210:00:052
27337282980,4sleep20-21swapper/209:35:192
139099700,60rtkit-daemon0-21swapper/207:09:522
268084426645,14sleep30-21swapper/307:06:063
268088826446,13sleep00-21swapper/007:06:400
268108326242,14sleep10-21swapper/107:09:101
27440162530,4sleep23056-21PK-Backend10:05:132
26994072400,2sleep00-21swapper/008:00:010
26836302380,3sleep20-21swapper/207:15:042
268130599361,6cyclictest2762816-21apt11:00:111
27108202350,4sleep30-21swapper/308:30:193
268130599351,4cyclictest2748286-21/usr/sbin/munin10:15:201
268130599311,27cyclictest2759495-21cut10:50:121
268130599310,21cyclictest2793868-21taskset12:28:251
2681305993011,17cyclictest2753879-21cron10:35:001
2681305993011,17cyclictest2740797-21awk09:55:191
2681305993010,18cyclictest2711787-21date08:35:001
268131099290,7cyclictest3056-21PK-Backend07:55:002
268130599297,20cyclictest436-20systemd-journal11:20:001
268130599297,20cyclictest436-20systemd-journal11:20:001
268130599278,17cyclictest2705394-21cat08:15:161
268130599278,16cyclictest2692328-21latency_hist07:40:001
268130599277,17cyclictest2692644-21cpuspeed_turbos07:40:121
268130599272,20cyclictest2741497-21apt-get10:00:061
2681305992716,9cyclictest2725114-21timerandwakeup09:10:191
268130599271,5cyclictest0-21swapper/109:25:111
268130599271,5cyclictest0-21swapper/109:25:111
2681305992710,15cyclictest2781861-21/usr/sbin/munin11:55:201
268130499279,9cyclictest13-21ksoftirqd/007:55:010
268130599268,16cyclictest2776654-21date11:40:011
268130599268,16cyclictest2744350-21cat10:05:181
268130599262,4cyclictest0-21swapper/110:10:091
268130599261,7cyclictest2767468-21taskset11:11:321
2681305992615,10cyclictest26-21ksoftirqd/109:30:161
2681305992610,14cyclictest2786358-21ntp_states12:05:181
268130599260,21cyclictest3056-21PK-Backend07:49:481
268130499263,6cyclictest13-21ksoftirqd/009:40:000
268131099251,22cyclictest2156-21gnome-shell12:22:592
268130599259,13cyclictest2727617-21cron09:19:591
268130599258,15cyclictest2709162-21gunzip08:25:211
268130599257,5cyclictest2772505-21ls11:25:211
268130599256,5cyclictest2796244-21cat12:35:121
268130599256,5cyclictest2778604-21expr11:45:121
268130599256,5cyclictest2717368-21cut08:50:121
268130599256,5cyclictest2715716-21cat08:45:131
268130599255,6cyclictest2764722-21expr11:05:131
268130599254,4cyclictest2714994-21seq08:42:061
268130599252,6cyclictest2719451-21irqstats08:55:161
268130599252,4cyclictest0-21swapper/108:00:101
268130599252,18cyclictest2695924-21apt07:50:131
268130599252,18cyclictest2156-21gnome-shell09:00:091
268130599251,7cyclictest2781652-21unin-run11:55:001
268130599251,19cyclictest2711511-21apt-get08:35:031
268130499251,8cyclictest13-21ksoftirqd/008:30:220
139099250,3rtkit-daemon1389-21rtkit-daemon09:17:402
26904332240,4sleep20-21swapper/207:35:002
26904332240,4sleep20-21swapper/207:35:002
2681312992415,7cyclictest436-20systemd-journal11:35:003
268131099242,5cyclictest34-21ksoftirqd/210:23:492
268131099240,9cyclictest2752596-21grep10:30:122
268130599248,14cyclictest2762028-21timerandwakeup10:55:221
268130599247,15cyclictest2687830-21needreboot07:25:181
268130599243,19cyclictest2733279-21cpuspeed_turbos09:35:131
268130599242,7cyclictest2156-21gnome-shell08:14:091
268130599241,9cyclictest2758477-21python310:45:191
268130599241,7cyclictest2723476-21cat09:05:211
268130599241,6cyclictest2791396-21seq12:20:161
268130599241,4cyclictest2756280-21hddtemp_smartct10:40:151
268130599241,4cyclictest2703038-21latency_hist08:10:011
268130599241,20cyclictest82332-21update-manager07:34:561
268130599241,20cyclictest82332-21update-manager07:34:551
268130599241,10cyclictest2775821-21runrttasks11:35:221
268130599241,10cyclictest2685179-21dump-pmu-power07:20:001
2681305992410,12cyclictest0-21swapper/111:20:201
268130599240,7cyclictest2784738-21taskset12:00:441
268130599240,7cyclictest2784738-21taskset12:00:431
268130599240,5cyclictest436-20systemd-journal12:35:001
268130499246,4cyclictest13-21ksoftirqd/009:10:000
268130499245,12cyclictest2772635-21seq11:26:070
268130499242,7cyclictest13-21ksoftirqd/012:10:180
2681304992410,4cyclictest13-21ksoftirqd/012:05:000
2681304992410,4cyclictest13-21ksoftirqd/012:05:000
268131099234,3cyclictest2731096-21munin-run09:30:002
268131099234,3cyclictest2731096-21munin-run09:30:002
268131099232,5cyclictest3056-21PK-Backend11:25:002
2681310992317,4cyclictest2715244-21munin-plugin-st08:45:012
268130599238,13cyclictest26-21ksoftirqd/111:30:151
268130599237,4cyclictest2750668-21basename10:25:101
268130599231,9cyclictest2790563-21awk12:20:001
268130599231,7cyclictest2787564-21ls12:10:111
268130599231,7cyclictest2727901-21apt09:20:111
268130599231,5cyclictest2708033-21cron08:25:001
268130599231,5cyclictest2685595-21cpuspeed_turbos07:20:111
268130599230,7cyclictest2755614-21perf10:40:011
268130499233,7cyclictest13-21ksoftirqd/011:05:170
2681304992316,5cyclictest13-21ksoftirqd/009:15:010
2681312992220,1cyclictest3056-21PK-Backend08:37:213
2681312992220,1cyclictest0-21swapper/307:52:323
2681312992219,2cyclictest3056-21PK-Backend08:47:273
268131299220,20cyclictest3056-21PK-Backend12:20:273
268131299220,20cyclictest3056-21PK-Backend11:48:033
268131299220,20cyclictest3056-21PK-Backend11:05:033
268131299220,20cyclictest3056-21PK-Backend10:25:083
268131299220,20cyclictest3056-21PK-Backend09:22:153
268131299220,20cyclictest3056-21PK-Backend08:23:143
268131099225,4cyclictest2156-21gnome-shell12:14:092
268131099224,5cyclictest82332-21update-manager11:09:572
268131099224,5cyclictest2728535-21cat09:20:172
268131099220,9cyclictest2782629-21ntp_states11:55:192
268130599223,4cyclictest2734913-21cpuspeed09:40:121
2681305992216,5cyclictest0-21swapper/109:53:451
268130599221,5cyclictest2699497-21latency_hist08:00:011
2681305992213,6cyclictest2750402-21cron10:25:001
268130499226,4cyclictest13-21ksoftirqd/012:20:000
268130499224,5cyclictest13-21ksoftirqd/008:45:110
2681304992214,6cyclictest2736287-21sh09:45:000
268130499220,6cyclictest3056-21PK-Backend07:45:210
139099220,3rtkit-daemon1389-21rtkit-daemon07:11:192
139099220,14rtkit-daemon0-21swapper/011:36:130
139099220,14rtkit-daemon0-21swapper/011:22:470
2681312992119,1cyclictest3056-21PK-Backend12:33:183
2681312992119,1cyclictest3056-21PK-Backend12:26:223
2681312992119,1cyclictest3056-21PK-Backend12:19:253
2681312992119,1cyclictest3056-21PK-Backend11:22:343
2681312992119,1cyclictest3056-21PK-Backend10:51:253
2681312992119,1cyclictest3056-21PK-Backend10:12:453
2681312992119,1cyclictest3056-21PK-Backend10:00:483
2681312992119,1cyclictest3056-21PK-Backend09:37:503
2681312992119,1cyclictest3056-21PK-Backend08:13:423
2681312992119,1cyclictest0-21swapper/311:52:563
2681312992119,1cyclictest0-21swapper/309:51:433
2681312992119,1cyclictest0-21swapper/309:41:153
2681312992119,1cyclictest0-21swapper/307:12:313
2681312992118,2cyclictest3056-21PK-Backend07:33:473
2681312992118,2cyclictest3056-21PK-Backend07:33:463
268131299210,20cyclictest3056-21PK-Backend10:42:393
268131299210,20cyclictest0-21swapper/307:57:463
268131299210,19cyclictest3056-21PK-Backend11:26:263
268131299210,19cyclictest3056-21PK-Backend10:55:173
268131299210,19cyclictest3056-21PK-Backend07:21:153
268131099214,4cyclictest2711528-21idleruntime-cro08:34:592
268131099213,5cyclictest2688661-21latency_hist07:30:012
268131099213,4cyclictest2780094-21cut11:50:012
268131099212,7cyclictest2772962-21latency_hist11:30:012
2681310992119,1cyclictest0-21swapper/208:35:192
2681310992116,3cyclictest2778163-21sh11:45:002
268131099210,20cyclictest2747216-21apt10:15:092
268131099210,20cyclictest0-21swapper/212:29:072
268131099210,20cyclictest0-21swapper/209:07:232
268131099210,19cyclictest3056-21PK-Backend07:23:212
268130599210,6cyclictest2682955-21dump-pmu-power07:15:001
268130599210,20cyclictest3056-21PK-Backend09:45:541
268130499216,7cyclictest141rcu_preempt10:53:200
268130499216,5cyclictest13-21ksoftirqd/008:35:210
268130499214,6cyclictest13-21ksoftirqd/010:45:180
268130499214,10cyclictest13-21ksoftirqd/009:25:170
268130499214,10cyclictest13-21ksoftirqd/009:25:170
268130499213,6cyclictest3056-21PK-Backend10:17:440
268130499212,7cyclictest13-21ksoftirqd/011:55:100
268130499212,6cyclictest13-21ksoftirqd/008:10:150
268130499212,10cyclictest2794970-21netstat12:30:170
268130499212,10cyclictest13-21ksoftirqd/008:55:180
2681304992114,6cyclictest2792196-21sshd12:24:100
2681304992114,6cyclictest2741489-21perf10:00:000
268130499211,2cyclictest2738031-21turbostat09:50:010
268130499210,20cyclictest0-21swapper/010:36:210
268131299201,9cyclictest886-21NetworkManager11:36:273
2681312992018,1cyclictest3056-21PK-Backend12:39:133
2681312992018,1cyclictest3056-21PK-Backend12:12:303
2681312992018,1cyclictest3056-21PK-Backend11:43:093
2681312992018,1cyclictest3056-21PK-Backend10:47:323
2681312992018,1cyclictest3056-21PK-Backend10:08:053
2681312992018,1cyclictest3056-21PK-Backend09:57:103
2681312992018,1cyclictest3056-21PK-Backend09:16:003
2681312992018,1cyclictest3056-21PK-Backend09:14:113
2681312992018,1cyclictest3056-21PK-Backend08:55:523
2681312992017,2cyclictest3056-21PK-Backend07:35:023
2681312992017,2cyclictest0-21swapper/307:46:033
2681312992012,6cyclictest2738059-21unin-run09:50:003
268131299201,17cyclictest2753282-21python310:30:193
268131299200,1cyclictest3056-21PK-Backend09:05:203
268131299200,19cyclictest3056-21PK-Backend12:01:413
268131299200,19cyclictest3056-21PK-Backend12:01:413
268131299200,19cyclictest3056-21PK-Backend08:16:313
268131099208,3cyclictest34-21ksoftirqd/211:10:252
268131099203,5cyclictest2764369-21cat11:05:022
2681310992018,1cyclictest0-21swapper/212:36:172
2681310992018,1cyclictest0-21swapper/211:30:402
2681310992018,1cyclictest0-21swapper/210:51:392
2681310992018,1cyclictest0-21swapper/210:45:452
2681310992018,1cyclictest0-21swapper/208:16:072
2681310992018,1cyclictest0-21swapper/208:08:222
268131099200,1cyclictest0-21swapper/210:14:082
268131099200,19cyclictest1389-21rtkit-daemon09:03:032
268131099200,19cyclictest0-21swapper/212:32:112
268131099200,19cyclictest0-21swapper/212:08:522
268131099200,19cyclictest0-21swapper/211:53:452
268131099200,19cyclictest0-21swapper/210:57:342
268131099200,19cyclictest0-21swapper/209:45:222
268131099200,19cyclictest0-21swapper/208:56:552
268131099200,19cyclictest0-21swapper/208:20:592
268131099200,19cyclictest0-21swapper/207:57:402
268131099200,19cyclictest0-21swapper/207:36:452
268130499208,7cyclictest13-21ksoftirqd/009:15:140
268130499206,5cyclictest2690202-21chrt07:32:500
268130499206,5cyclictest2690202-21chrt07:32:490
268130499205,9cyclictest13-21ksoftirqd/012:05:030
268130499204,9cyclictest13-21ksoftirqd/010:00:160
268130499204,6cyclictest13-21ksoftirqd/007:40:010
268130499203,9cyclictest13-21ksoftirqd/011:15:190
268130499203,9cyclictest13-21ksoftirqd/011:15:180
268130499203,9cyclictest13-21ksoftirqd/008:15:110
268130499203,7cyclictest13-21ksoftirqd/010:05:140
268130499202,7cyclictest13-21ksoftirqd/008:25:120
268130499202,10cyclictest1095-21runrttasks08:40:500
2681304992018,1cyclictest0-21swapper/008:01:560
2681304992017,2cyclictest161rcuc/009:34:110
2681304992017,1cyclictest161rcuc/009:00:280
2681304992015,4cyclictest2693933-21perf07:45:000
2681304992012,6cyclictest2708044-21cron08:24:590
268130499200,8cyclictest1788620-21WebKitWebProces11:12:470
268130499200,1cyclictest0-21swapper/012:39:380
268130499200,1cyclictest0-21swapper/010:55:410
268130499200,19cyclictest0-21swapper/010:11:330
268130499200,11cyclictest13-21ksoftirqd/007:24:590
139099200,17rtkit-daemon2683202-21munin-node08:48:512
27636032190,2sleep33056-21PK-Backend11:00:193
2681312991917,1cyclictest3056-21PK-Backend11:13:483
2681312991917,1cyclictest0-21swapper/310:22:193
2681312991917,1cyclictest0-21swapper/309:28:453
2681312991917,1cyclictest0-21swapper/309:28:443
268131299190,1cyclictest3056-21PK-Backend12:06:343
268131299190,1cyclictest3056-21PK-Backend11:17:413
268131299190,1cyclictest3056-21PK-Backend11:17:403
268131299190,1cyclictest3056-21PK-Backend08:01:453
268131299190,17cyclictest2754872-21ntp_kernel_pll_10:35:193
2681310991917,1cyclictest0-21swapper/212:16:022
2681310991917,1cyclictest0-21swapper/212:00:552
2681310991917,1cyclictest0-21swapper/212:00:552
2681310991917,1cyclictest0-21swapper/209:32:042
2681310991917,1cyclictest0-21swapper/209:13:392
2681310991917,1cyclictest0-21swapper/208:01:532
268131099190,6cyclictest34-21ksoftirqd/207:43:312
268131099190,1cyclictest0-21swapper/211:39:392
268131099190,1cyclictest0-21swapper/210:27:482
268131099190,1cyclictest0-21swapper/209:40:082
268131099190,18cyclictest0-21swapper/211:17:352
268131099190,18cyclictest0-21swapper/211:17:352
268130499198,3cyclictest13-21ksoftirqd/008:06:180
268130499197,3cyclictest141rcu_preempt11:00:110
268130499194,4cyclictest13-21ksoftirqd/011:40:100
268130499194,1cyclictest0-21swapper/011:51:010
2681304991917,1cyclictest0-21swapper/010:41:140
2681304991917,1cyclictest0-21swapper/010:20:050
2681304991917,1cyclictest0-21swapper/009:52:210
2681304991917,1cyclictest0-21swapper/008:51:500
2681304991916,2cyclictest161rcuc/009:23:410
2681304991912,6cyclictest2685398-21sed07:20:010
268130499190,1cyclictest0-21swapper/011:47:260
268130499190,1cyclictest0-21swapper/007:14:450
2681312991816,1cyclictest3056-21PK-Backend11:57:493
2681312991816,1cyclictest3056-21PK-Backend08:51:403
2681312991816,1cyclictest3056-21PK-Backend08:26:183
2681312991816,1cyclictest3056-21PK-Backend07:42:183
268131299180,1cyclictest0-21swapper/310:17:253
268131299180,17cyclictest2731365-21apt09:30:093
268131099183,6cyclictest34-21ksoftirqd/208:14:492
2681310991816,1cyclictest0-21swapper/210:43:422
2681310991816,1cyclictest0-21swapper/209:58:072
2681310991816,1cyclictest0-21swapper/208:25:052
2681310991813,3cyclictest2717420-21cpuspeed_turbos08:50:112
268131099180,6cyclictest34-21ksoftirqd/209:54:172
268130499183,4cyclictest2773229-21apt11:30:130
268130499181,7cyclictest13-21ksoftirqd/007:25:110
2681304991816,1cyclictest0-21swapper/010:32:290
268130499180,7cyclictest13-21ksoftirqd/010:25:160
2681312991714,2cyclictest3056-21PK-Backend08:07:133
2681312991710,5cyclictest2683224-21which07:15:023
268131299170,15cyclictest2688052-21python307:25:213
268131099172,5cyclictest34-21ksoftirqd/210:35:182
268131099162,5cyclictest34-21ksoftirqd/207:45:192
2681312991513,1cyclictest3056-21PK-Backend09:01:553
268131299131,1cyclictest141rcu_preempt08:40:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional