You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 12:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Apr 15, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
294992321310,3sleep10-21swapper/121:40:111
291018821150,10sleep234-21ksoftirqd/219:45:212
289639828040,14sleep00-21swapper/019:09:100
289628927539,14sleep10-21swapper/119:07:461
289625126938,25sleep30-21swapper/319:07:173
289640026645,14sleep20-21swapper/219:09:122
29658132380,1sleep32965699-21cstates22:25:123
2896616992825,2cyclictest161rcuc/019:10:190
2896616992824,3cyclictest161rcuc/022:32:210
289661699278,10cyclictest13-21ksoftirqd/020:40:140
29949352260,3sleep13056-21PK-Backend23:49:551
2896620992611,13cyclictest2949552-21cron21:40:001
289662099260,25cyclictest2963083-21users22:15:231
2896626992514,8cyclictest883-21dbus-daemon00:00:013
2896616992521,3cyclictest161rcuc/023:06:220
2896616992521,3cyclictest161rcuc/019:23:070
2896616992521,3cyclictest161rcuc/000:39:030
29489592240,3sleep30-21swapper/321:35:203
29437662240,3sleep33056-21PK-Backend21:20:203
289662699244,19cyclictest2962674-21grep22:15:193
2896616992410,4cyclictest13-21ksoftirqd/020:00:010
2896626992319,2cyclictest3056-21PK-Backend21:25:393
289662699231,7cyclictest2951313-21awk21:45:003
2896624992314,7cyclictest436-20systemd-journal20:14:082
289662499230,21cyclictest2984082-21sed23:15:232
289661699235,9cyclictest13-21ksoftirqd/019:40:000
2896616992321,1cyclictest0-21swapper/023:51:230
2896616992321,1cyclictest0-21swapper/023:47:310
2896616992320,2cyclictest161rcuc/022:51:550
2896616992320,2cyclictest161rcuc/021:59:500
29990812220,4sleep13056-21PK-Backend00:00:141
289662699229,2cyclictest29095332sleep319:45:133
2896626992220,1cyclictest0-21swapper/300:33:513
2896626992219,2cyclictest3056-21PK-Backend20:54:293
2896626992211,8cyclictest2903958-21sh19:30:003
289662699220,20cyclictest3056-21PK-Backend22:21:493
2896624992219,2cyclictest0-21swapper/200:28:442
2896624992214,6cyclictest3007114-21/usr/sbin/munin00:25:002
2896624992214,6cyclictest2913591-21awk19:55:172
289662499220,20cyclictest3056-21PK-Backend19:10:142
289662499220,20cyclictest2991280-21seq23:37:272
2896616992220,1cyclictest0-21swapper/023:36:090
2896616992220,1cyclictest0-21swapper/021:09:480
2896616992220,1cyclictest0-21swapper/020:35:000
289661699220,20cyclictest3056-21PK-Backend21:35:020
29362302210,5sleep10-21swapper/121:00:131
2896626992119,1cyclictest3056-21PK-Backend21:15:393
2896626992119,1cyclictest3056-21PK-Backend19:31:493
2896626992119,1cyclictest0-21swapper/323:47:273
2896626992119,1cyclictest0-21swapper/323:36:393
2896626992119,1cyclictest0-21swapper/323:29:563
2896626992119,1cyclictest0-21swapper/320:35:033
2896626992119,1cyclictest0-21swapper/320:22:593
2896626992119,1cyclictest0-21swapper/320:15:233
2896626992119,1cyclictest0-21swapper/300:18:233
2896626992119,1cyclictest0-21swapper/300:11:403
2896626992119,1cyclictest0-21swapper/300:11:403
2896626992118,2cyclictest3056-21PK-Backend19:14:113
289662699210,20cyclictest3056-21PK-Backend23:32:473
289662699210,20cyclictest3056-21PK-Backend20:58:283
289662699210,20cyclictest3056-21PK-Backend20:40:233
289662699210,20cyclictest3056-21PK-Backend19:43:393
289662699210,20cyclictest0-21swapper/320:49:443
289662699210,19cyclictest3056-21PK-Backend23:22:123
289662699210,19cyclictest3056-21PK-Backend22:30:553
2896624992119,1cyclictest3056-21PK-Backend20:50:242
2896624992119,1cyclictest3056-21PK-Backend20:39:432
2896624992119,1cyclictest0-21swapper/223:07:192
2896624992119,1cyclictest0-21swapper/222:24:472
2896624992119,1cyclictest0-21swapper/221:55:352
2896624992119,1cyclictest0-21swapper/221:51:212
2896624992119,1cyclictest0-21swapper/221:22:082
2896624992119,1cyclictest0-21swapper/220:45:032
2896624992119,1cyclictest0-21swapper/220:21:382
2896624992119,1cyclictest0-21swapper/220:00:362
2896624992119,1cyclictest0-21swapper/200:38:172
2896624992119,1cyclictest0-21swapper/200:12:082
2896624992119,1cyclictest0-21swapper/200:12:082
2896624992119,1cyclictest0-21swapper/200:00:592
289662499210,20cyclictest0-21swapper/222:52:122
289662499210,20cyclictest0-21swapper/222:31:502
2896620992118,2cyclictest0-21swapper/119:28:071
289662099210,20cyclictest0-21swapper/119:46:521
289662099210,20cyclictest0-21swapper/119:14:071
289661699219,3cyclictest13-21ksoftirqd/020:54:220
289661699214,7cyclictest13-21ksoftirqd/023:25:130
289661699214,7cyclictest13-21ksoftirqd/020:39:100
289661699214,6cyclictest13-21ksoftirqd/019:30:220
2896616992119,1cyclictest0-21swapper/023:11:350
2896616992119,1cyclictest0-21swapper/021:51:380
2896616992119,1cyclictest0-21swapper/021:40:510
2896616992119,1cyclictest0-21swapper/000:24:150
2896616992119,1cyclictest0-21swapper/000:17:460
2896616992117,3cyclictest161rcuc/022:40:340
2896616992117,3cyclictest161rcuc/020:55:020
2896616992117,3cyclictest161rcuc/000:10:280
2896616992117,3cyclictest161rcuc/000:10:280
2896616992116,3cyclictest161rcuc/021:45:080
289661699210,7cyclictest13-21ksoftirqd/000:25:140
289661699210,20cyclictest0-21swapper/023:18:520
289661699210,20cyclictest0-21swapper/022:23:220
289661699210,20cyclictest0-21swapper/022:02:400
289661699210,20cyclictest0-21swapper/021:26:370
289661699210,1cyclictest0-21swapper/021:02:240
139099210,3rtkit-daemon1389-21rtkit-daemon19:52:031
139099210,17rtkit-daemon3056-21PK-Backend23:40:090
289662699207,9cyclictest883-21dbus-daemon22:40:003
2896626992018,1cyclictest3056-21PK-Backend19:18:373
2896626992018,1cyclictest3056-21PK-Backend00:26:073
2896626992018,1cyclictest0-21swapper/320:30:303
2896626992017,2cyclictest3056-21PK-Backend22:57:523
2896626992017,2cyclictest3056-21PK-Backend20:06:303
2896626992013,5cyclictest2992628-21python323:40:213
2896626992012,6cyclictest436-20systemd-journal22:07:113
289662699200,2cyclictest2952861-21pool-udisksd21:48:043
289662699200,1cyclictest0-21swapper/321:54:043
289662699200,19cyclictest3056-21PK-Backend23:18:213
289662699200,19cyclictest3056-21PK-Backend22:02:433
289662699200,19cyclictest0-21swapper/323:51:193
289662699200,19cyclictest0-21swapper/323:10:503
289662699200,19cyclictest0-21swapper/322:41:023
289662699200,19cyclictest0-21swapper/300:00:043
2896624992018,1cyclictest0-21swapper/223:53:292
2896624992018,1cyclictest0-21swapper/223:32:472
2896624992018,1cyclictest0-21swapper/221:14:582
2896624992018,1cyclictest0-21swapper/200:16:482
2896624992014,4cyclictest2970430-21idleruntime-cro22:40:002
2896624992014,4cyclictest2936135-21tr21:00:132
2896624992013,6cyclictest2923478-21expr20:25:142
289662499200,19cyclictest0-21swapper/223:43:082
289662499200,19cyclictest0-21swapper/222:41:302
289662499200,19cyclictest0-21swapper/222:26:372
289662499200,19cyclictest0-21swapper/221:39:112
289662499200,19cyclictest0-21swapper/200:30:472
2896620992017,2cyclictest3056-21PK-Backend21:33:061
2896620992016,3cyclictest1095-21runrttasks23:00:461
2896620992013,5cyclictest3002954-21awk00:10:211
2896620992013,5cyclictest3002954-21awk00:10:211
2896620992013,5cyclictest2985173-21ls23:20:131
2896620992012,6cyclictest2939316-21sed21:10:021
2896620992012,6cyclictest26-21ksoftirqd/100:35:191
289662099200,1cyclictest0-21swapper/100:20:351
289662099200,19cyclictest3056-21PK-Backend23:39:181
289662099200,19cyclictest3056-21PK-Backend22:11:581
289662099200,19cyclictest3056-21PK-Backend20:11:331
289662099200,19cyclictest3056-21PK-Backend20:09:431
289662099200,19cyclictest0-21swapper/123:31:341
289662099200,19cyclictest0-21swapper/123:27:421
289662099200,19cyclictest0-21swapper/123:05:591
289662099200,19cyclictest0-21swapper/122:45:311
289662099200,19cyclictest0-21swapper/122:05:561
289662099200,19cyclictest0-21swapper/121:25:271
289662099200,19cyclictest0-21swapper/119:17:451
289662099200,18cyclictest3056-21PK-Backend23:53:451
289662099200,18cyclictest0-21swapper/122:04:071
289661699203,7cyclictest13-21ksoftirqd/000:07:110
289661699202,4cyclictest13-21ksoftirqd/020:45:220
2896616992018,1cyclictest0-21swapper/023:58:530
2896616992018,1cyclictest0-21swapper/022:29:510
2896616992018,1cyclictest0-21swapper/022:18:080
2896616992018,1cyclictest0-21swapper/020:29:390
2896616992018,1cyclictest0-21swapper/020:11:560
2896616992017,2cyclictest161rcuc/023:30:270
2896616992016,3cyclictest161rcuc/021:31:410
2896616992016,3cyclictest161rcuc/019:51:090
289661699200,1cyclictest0-21swapper/023:00:260
289661699200,1cyclictest0-21swapper/022:12:080
289661699200,19cyclictest0-21swapper/023:22:440
289661699200,19cyclictest0-21swapper/022:57:360
289661699200,19cyclictest0-21swapper/019:44:240
289661699200,19cyclictest0-21swapper/019:19:570
139099200,3rtkit-daemon1389-21rtkit-daemon21:13:080
29169432190,5sleep23056-21PK-Backend20:05:172
2896626991917,1cyclictest3056-21PK-Backend22:51:233
2896626991917,1cyclictest3056-21PK-Backend00:07:483
2896626991917,1cyclictest0-21swapper/322:13:113
2896626991917,1cyclictest0-21swapper/320:26:443
2896626991911,6cyclictest2972870-21grep22:45:153
289662699190,1cyclictest3056-21PK-Backend19:35:423
2896624991917,1cyclictest0-21swapper/222:17:102
2896624991917,1cyclictest0-21swapper/222:12:582
2896624991917,1cyclictest0-21swapper/221:43:102
2896624991917,1cyclictest0-21swapper/221:31:272
2896624991917,1cyclictest0-21swapper/220:43:282
2896624991917,1cyclictest0-21swapper/219:54:002
2896624991912,5cyclictest34-21ksoftirqd/220:30:122
2896624991912,5cyclictest2993467-21sshd23:45:112
289662499190,1cyclictest0-21swapper/223:11:452
289662499190,1cyclictest0-21swapper/220:18:132
2896620991917,1cyclictest3056-21PK-Backend21:56:301
2896620991917,1cyclictest3056-21PK-Backend20:50:061
2896620991917,1cyclictest3056-21PK-Backend20:27:071
2896620991917,1cyclictest3056-21PK-Backend20:22:001
2896620991917,1cyclictest0-21swapper/122:26:041
2896620991917,1cyclictest0-21swapper/121:22:031
2896620991917,1cyclictest0-21swapper/120:55:401
2896620991917,1cyclictest0-21swapper/119:58:291
2896620991917,1cyclictest0-21swapper/119:35:021
2896620991916,2cyclictest3056-21PK-Backend20:33:501
2896620991913,5cyclictest2927182-21basename20:35:161
2896620991912,5cyclictest2997734-21which23:55:211
2896620991912,5cyclictest2914683-21cat20:00:121
2896620991910,7cyclictest1-21systemd21:15:371
289662099190,1cyclictest3056-21PK-Backend20:15:591
289662099190,1cyclictest0-21swapper/122:20:371
289662099190,18cyclictest3056-21PK-Backend23:16:061
289662099190,18cyclictest3056-21PK-Backend20:47:421
289662099190,18cyclictest0-21swapper/123:40:201
289662099190,18cyclictest0-21swapper/123:10:391
289662099190,18cyclictest0-21swapper/119:24:151
289661699197,7cyclictest13-21ksoftirqd/019:25:470
289661699192,3cyclictest13-21ksoftirqd/000:00:170
2896616991917,1cyclictest0-21swapper/022:46:270
2896616991917,1cyclictest0-21swapper/021:22:250
2896616991915,3cyclictest161rcuc/021:15:230
2896616991912,6cyclictest2922258-21awk20:20:170
289661699190,8cyclictest3010262-21taskset00:31:100
289661699190,1cyclictest3056-21PK-Backend22:05:170
289661699190,1cyclictest0-21swapper/020:18:100
289661699190,1cyclictest0-21swapper/020:03:040
2896626991813,3cyclictest2979616-21apt23:05:123
2896626991810,6cyclictest2912620-21cron19:55:003
289662699180,6cyclictest2955748-21ntp_kernel_pll_21:55:183
289662499189,7cyclictest883-21dbus-daemon21:00:022
2896624991816,1cyclictest0-21swapper/223:00:162
2896624991816,1cyclictest0-21swapper/222:55:512
2896624991816,1cyclictest0-21swapper/222:09:332
2896624991816,1cyclictest0-21swapper/221:29:042
2896624991816,1cyclictest0-21swapper/221:17:212
2896624991816,1cyclictest0-21swapper/219:19:272
2896624991816,1cyclictest0-21swapper/200:09:182
2896624991811,5cyclictest34-21ksoftirqd/223:55:182
2896624991811,5cyclictest2905093-21smartctl19:30:202
289662499180,1cyclictest0-21swapper/223:27:202
289662499180,17cyclictest0-21swapper/221:09:242
2896620991816,1cyclictest3056-21PK-Backend00:31:091
2896620991816,1cyclictest0-21swapper/122:57:411
2896620991816,1cyclictest0-21swapper/122:54:501
2896620991816,1cyclictest0-21swapper/120:40:461
2896620991816,1cyclictest0-21swapper/119:33:001
2896620991816,1cyclictest0-21swapper/100:08:111
2896620991813,3cyclictest978-21snmpd19:41:591
2896616991814,3cyclictest161rcuc/022:37:500
2896616991814,3cyclictest13-21ksoftirqd/020:10:020
2896616991811,5cyclictest2909981-21cat19:45:170
29472822170,3sleep30-21swapper/321:30:233
2896626991715,1cyclictest3056-21PK-Backend19:23:173
2896626991715,1cyclictest0-21swapper/300:20:263
2896626991712,3cyclictest2918203-21cpuspeed_turbos20:10:123
2896626991710,5cyclictest3011227-21cat00:35:133
289662499179,6cyclictest883-21dbus-daemon21:50:012
289662499176,9cyclictest2909143-21cron19:45:002
2896624991715,1cyclictest0-21swapper/222:01:222
2896624991715,1cyclictest0-21swapper/219:35:352
2896620991715,1cyclictest0-21swapper/100:27:171
2896620991712,3cyclictest1095-21runrttasks22:35:231
2896626991614,1cyclictest3056-21PK-Backend20:03:053
2896626991613,2cyclictest3056-21PK-Backend23:00:433
2896626991610,4cyclictest2939863-21awk21:10:163
289662699160,15cyclictest2913365-21chrt19:55:153
2896624991614,1cyclictest0-21swapper/222:47:452
2896620991613,2cyclictest3056-21PK-Backend21:49:411
2896620991611,4cyclictest2939602-21expr21:10:131
289662699154,10cyclictest3056-21PK-Backend21:05:213
2896624991513,1cyclictest0-21swapper/223:22:542
2896624991513,1cyclictest0-21swapper/219:29:062
289662099159,5cyclictest2967590-21df22:30:141
2896620991513,1cyclictest3056-21PK-Backend21:52:181
2896620991512,2cyclictest3056-21PK-Backend22:41:521
289662499149,3cyclictest2902222-21latency_hist19:25:002
2896620991410,3cyclictest3056-21PK-Backend00:15:461
289662699134,2cyclictest141rcu_preempt21:00:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional