You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-05 - 14:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Aug 05, 2026 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
350186321130,3sleep13056-21PK-Backend22:00:151
344119227938,13sleep20-21swapper/219:06:522
344139327450,18sleep00-21swapper/019:09:280
139099720,61rtkit-daemon0-21swapper/119:08:291
344035727139,25sleep30-21swapper/319:05:173
34601632550,6sleep2331rcuc/220:00:132
34989282350,3sleep13056-21PK-Backend21:50:151
35350082330,3sleep00-21swapper/023:35:010
34477272310,4sleep33056-21PK-Backend19:25:133
35514192300,17sleep20-21swapper/200:20:172
139099280,4rtkit-daemon1389-21rtkit-daemon19:17:441
3441573992415,5cyclictest3529771-21idleruntime-cro23:19:593
3441573992413,9cyclictest3494591-21turbostat21:40:003
344156299240,22cyclictest3461760-21chrt20:05:130
35395172230,3sleep20-21swapper/223:45:222
344157399231,19cyclictest3557534-21perf00:40:003
344157399230,20cyclictest3516365-21/usr/sbin/munin22:40:173
344157099230,3cyclictest34-21ksoftirqd/223:15:012
344157099230,21cyclictest3553086-21cut00:25:172
3441562992321,1cyclictest0-21swapper/020:59:410
3441562992320,2cyclictest0-21swapper/021:52:410
3441562992316,5cyclictest3454436-21cron19:45:000
3441573992220,1cyclictest0-21swapper/321:27:483
3441573992220,1cyclictest0-21swapper/319:42:573
3441573992213,7cyclictest1-21systemd21:05:013
3441573992213,7cyclictest1-21systemd19:40:013
3441573992210,7cyclictest3449476-21apt-get19:30:043
3441570992220,1cyclictest0-21swapper/223:07:172
3441570992220,1cyclictest0-21swapper/223:07:172
3441570992220,1cyclictest0-21swapper/221:58:092
3441570992220,1cyclictest0-21swapper/221:20:102
3441570992220,1cyclictest0-21swapper/220:17:112
3441570992220,1cyclictest0-21swapper/220:17:112
3441570992215,5cyclictest3520833-21cron22:55:012
344157099220,2cyclictest894-21irqbalance22:42:292
3441566992220,1cyclictest3056-21PK-Backend22:58:561
3441566992220,1cyclictest0-21swapper/121:30:471
3441562992220,1cyclictest3056-21PK-Backend20:30:280
3441562992214,6cyclictest3489366-21cron21:25:000
344156299220,21cyclictest0-21swapper/023:20:070
139099220,15rtkit-daemon3056-21PK-Backend00:25:551
139099220,15rtkit-daemon0-21swapper/120:50:181
139099220,15rtkit-daemon0-21swapper/100:01:131
139099220,14rtkit-daemon0-21swapper/022:17:360
3441573992119,1cyclictest3056-21PK-Backend22:03:163
3441573992119,1cyclictest3056-21PK-Backend00:19:093
3441573992119,1cyclictest0-21swapper/323:37:403
3441573992119,1cyclictest0-21swapper/323:06:513
3441573992119,1cyclictest0-21swapper/323:06:513
3441573992119,1cyclictest0-21swapper/321:49:253
3441573992119,1cyclictest0-21swapper/321:12:263
3441573992119,1cyclictest0-21swapper/320:31:043
3441573992119,1cyclictest0-21swapper/320:19:133
3441573992119,1cyclictest0-21swapper/320:19:133
3441573992119,1cyclictest0-21swapper/319:55:273
3441573992119,1cyclictest0-21swapper/300:24:453
344157399211,8cyclictest3520879-21munin-run22:55:003
3441573992114,5cyclictest1095-21runrttasks19:45:123
3441573992113,6cyclictest42-21ksoftirqd/323:10:133
3441573992113,6cyclictest3495771-21cat21:40:223
3441573992112,7cyclictest42-21ksoftirqd/320:50:143
344157399210,20cyclictest3056-21PK-Backend22:05:053
344157399210,20cyclictest3056-21PK-Backend19:16:553
344157399210,20cyclictest0-21swapper/320:27:533
3441570992119,1cyclictest0-21swapper/223:39:142
3441570992119,1cyclictest0-21swapper/223:17:462
3441570992119,1cyclictest0-21swapper/223:04:062
3441570992119,1cyclictest0-21swapper/222:25:472
3441570992119,1cyclictest0-21swapper/222:12:352
3441570992119,1cyclictest0-21swapper/222:12:352
3441570992119,1cyclictest0-21swapper/221:40:452
3441570992119,1cyclictest0-21swapper/221:03:412
3441570992119,1cyclictest0-21swapper/220:56:582
3441570992119,1cyclictest0-21swapper/220:38:402
3441570992119,1cyclictest0-21swapper/220:11:022
3441570992119,1cyclictest0-21swapper/219:51:152
3441570992119,1cyclictest0-21swapper/219:42:502
3441570992119,1cyclictest0-21swapper/219:27:422
3441570992119,1cyclictest0-21swapper/219:19:252
3441570992115,4cyclictest3533268-21systemd-run23:30:002
344157099210,20cyclictest0-21swapper/223:30:012
344157099210,20cyclictest0-21swapper/220:26:022
344157099210,20cyclictest0-21swapper/200:19:502
3441566992116,3cyclictest3547123-21sh00:09:591
3441566992116,3cyclictest3547123-21sh00:09:591
3441566992113,6cyclictest3541889-21cron23:55:011
344156699210,20cyclictest0-21swapper/122:21:311
344156699210,1cyclictest3056-21PK-Backend21:59:471
344156699210,19cyclictest3056-21PK-Backend23:35:121
344156699210,19cyclictest3056-21PK-Backend22:45:101
344156699210,19cyclictest3056-21PK-Backend21:28:581
344156699210,19cyclictest3056-21PK-Backend20:37:141
344156699210,19cyclictest3056-21PK-Backend19:41:371
344156699210,19cyclictest3056-21PK-Backend19:37:391
3441562992119,1cyclictest0-21swapper/023:35:080
3441562992119,1cyclictest0-21swapper/022:51:540
3441562992119,1cyclictest0-21swapper/022:49:190
3441562992119,1cyclictest0-21swapper/022:29:170
3441562992119,1cyclictest0-21swapper/021:58:140
3441562992119,1cyclictest0-21swapper/020:00:060
3441562992119,1cyclictest0-21swapper/019:27:550
3441562992119,1cyclictest0-21swapper/000:08:330
3441562992119,1cyclictest0-21swapper/000:08:330
3441562992114,5cyclictest3459633-21cron20:00:010
3441562992113,6cyclictest3491121-21awk21:30:000
344156299210,20cyclictest0-21swapper/023:26:290
344156299210,20cyclictest0-21swapper/022:42:090
344156299210,20cyclictest0-21swapper/021:39:490
344156299210,20cyclictest0-21swapper/021:13:190
344156299210,20cyclictest0-21swapper/019:54:580
344156299210,20cyclictest0-21swapper/019:35:390
344156299210,20cyclictest0-21swapper/019:33:160
344156299210,19cyclictest0-21swapper/020:42:300
139099210,15rtkit-daemon3056-21PK-Backend22:51:061
139099210,14rtkit-daemon3056-21PK-Backend23:26:371
34581642200,3sleep13056-21PK-Backend19:55:121
344157399203,12cyclictest3474045-21apt20:40:133
344157399202,13cyclictest3533506-21apt23:30:103
3441573992017,2cyclictest3056-21PK-Backend00:14:093
3441573992014,4cyclictest3476416-21grep20:45:153
3441573992014,4cyclictest3462311-21needreboot20:05:183
3441573992013,6cyclictest3486625-21cat21:15:173
3441573992013,6cyclictest3486625-21cat21:15:163
3441573992012,4cyclictest3513911-21latency_hist22:35:003
3441573992011,7cyclictest3488774-21/usr/sbin/munin21:20:223
344157399200,1cyclictest0-21swapper/321:59:393
344157399200,1cyclictest0-21swapper/300:32:283
344157399200,19cyclictest3056-21PK-Backend22:13:583
344157399200,19cyclictest3056-21PK-Backend22:13:573
344157399200,19cyclictest0-21swapper/323:44:433
344157399200,19cyclictest0-21swapper/322:35:273
344157399200,19cyclictest0-21swapper/320:58:143
344157399200,18cyclictest3511065-21df22:25:133
3441570992018,1cyclictest0-21swapper/223:50:092
3441570992018,1cyclictest0-21swapper/223:42:192
3441570992018,1cyclictest0-21swapper/222:33:512
3441570992018,1cyclictest0-21swapper/222:15:542
3441570992018,1cyclictest0-21swapper/222:04:112
3441570992018,1cyclictest0-21swapper/220:06:502
3441570992018,1cyclictest0-21swapper/219:47:502
3441570992018,1cyclictest0-21swapper/219:30:052
3441570992018,1cyclictest0-21swapper/200:14:222
3441570992012,6cyclictest3452114-21swap19:35:222
3441570992011,7cyclictest3466839-21awk20:20:082
344157099200,1cyclictest0-21swapper/219:10:042
344157099200,19cyclictest0-21swapper/223:55:302
344157099200,19cyclictest0-21swapper/223:55:302
344157099200,19cyclictest0-21swapper/223:20:222
344157099200,19cyclictest0-21swapper/221:10:242
344157099200,19cyclictest0-21swapper/221:09:292
344157099200,19cyclictest0-21swapper/220:44:552
344157099200,19cyclictest0-21swapper/200:01:382
3441566992018,1cyclictest3056-21PK-Backend23:20:461
3441566992018,1cyclictest3056-21PK-Backend21:37:451
3441566992018,1cyclictest3056-21PK-Backend21:11:151
3441566992018,1cyclictest3056-21PK-Backend20:57:421
3441566992017,2cyclictest3056-21PK-Backend21:19:331
3441566992017,2cyclictest3056-21PK-Backend21:19:331
344156699200,1cyclictest0-21swapper/100:22:511
344156699200,19cyclictest3548700-21sshd00:13:181
344156699200,19cyclictest3056-21PK-Backend20:48:171
344156699200,19cyclictest3056-21PK-Backend20:33:011
344156699200,19cyclictest0-21swapper/122:43:411
344156699200,19cyclictest0-21swapper/119:23:471
344156699200,19cyclictest0-21swapper/119:23:461
344156699200,18cyclictest3056-21PK-Backend21:20:271
3441562992018,1cyclictest3056-21PK-Backend21:32:320
3441562992018,1cyclictest0-21swapper/023:47:450
3441562992018,1cyclictest0-21swapper/020:51:440
3441562992018,1cyclictest0-21swapper/020:11:410
3441562992018,1cyclictest0-21swapper/000:35:030
3441562992015,3cyclictest3510161-21cron22:24:590
3441562992015,2cyclictest13-21ksoftirqd/021:20:000
3441562992015,2cyclictest13-21ksoftirqd/021:20:000
3441562992012,6cyclictest3476470-21awk20:45:170
3441562992012,6cyclictest3472698-21fw_packets20:35:150
344156299200,1cyclictest0-21swapper/023:14:530
344156299200,1cyclictest0-21swapper/022:10:390
344156299200,1cyclictest0-21swapper/022:10:390
344156299200,1cyclictest0-21swapper/019:22:330
344156299200,1cyclictest0-21swapper/019:22:330
344156299200,1cyclictest0-21swapper/000:20:170
344156299200,19cyclictest0-21swapper/023:15:270
344156299200,19cyclictest0-21swapper/023:00:530
344156299200,19cyclictest0-21swapper/022:57:420
344156299200,19cyclictest0-21swapper/022:32:350
344156299200,19cyclictest0-21swapper/000:00:090
139099200,14rtkit-daemon3056-21PK-Backend22:27:261
139099200,14rtkit-daemon3056-21PK-Backend21:45:091
139099200,14rtkit-daemon0-21swapper/100:18:461
139099200,14rtkit-daemon0-21swapper/023:53:510
139099200,12rtkit-daemon0-21swapper/122:12:081
139099200,12rtkit-daemon0-21swapper/122:12:071
344157399192,10cyclictest141rcu_preempt22:15:283
3441573991917,1cyclictest0-21swapper/323:28:403
3441573991917,1cyclictest0-21swapper/320:36:313
3441573991917,1cyclictest0-21swapper/320:21:443
3441573991916,2cyclictest3056-21PK-Backend23:04:473
3441573991911,6cyclictest42-21ksoftirqd/320:10:223
3441573991911,6cyclictest42-21ksoftirqd/319:10:223
344157399191,13cyclictest1095-21runrttasks23:45:223
344157399190,1cyclictest3441557-21cyclictest23:20:023
344157399190,1cyclictest3056-21PK-Backend23:54:563
344157399190,1cyclictest3056-21PK-Backend00:05:523
344157399190,1cyclictest3056-21PK-Backend00:05:513
344157399190,1cyclictest0-21swapper/322:46:373
344157399190,1cyclictest0-21swapper/321:51:553
344157399190,18cyclictest3056-21PK-Backend23:56:053
344157399190,18cyclictest3056-21PK-Backend23:56:043
344157399190,18cyclictest0-21swapper/300:28:373
3441570991917,1cyclictest0-21swapper/222:23:242
3441570991917,1cyclictest0-21swapper/222:08:242
3441570991917,1cyclictest0-21swapper/221:49:302
3441570991917,1cyclictest0-21swapper/221:38:362
3441570991917,1cyclictest0-21swapper/220:54:552
3441570991917,1cyclictest0-21swapper/220:45:292
3441570991917,1cyclictest0-21swapper/200:32:422
3441570991916,2cyclictest0-21swapper/222:35:202
344157099190,1cyclictest0-21swapper/200:38:372
3441566991917,1cyclictest3056-21PK-Backend23:31:341
3441566991917,1cyclictest3056-21PK-Backend23:18:081
3441566991917,1cyclictest3056-21PK-Backend23:08:501
3441566991917,1cyclictest3056-21PK-Backend23:08:491
3441566991917,1cyclictest3056-21PK-Backend22:38:551
3441566991917,1cyclictest3056-21PK-Backend22:33:141
3441566991917,1cyclictest3056-21PK-Backend19:29:081
3441566991917,1cyclictest0-21swapper/123:55:271
3441566991917,1cyclictest0-21swapper/123:55:261
3441566991917,1cyclictest0-21swapper/123:48:571
3441566991917,1cyclictest0-21swapper/122:16:231
3441566991917,1cyclictest0-21swapper/122:05:501
3441566991917,1cyclictest0-21swapper/119:13:461
3441566991916,2cyclictest3056-21PK-Backend23:14:031
344156699190,1cyclictest3056-21PK-Backend23:44:451
344156699190,1cyclictest3056-21PK-Backend21:05:411
344156699190,1cyclictest3056-21PK-Backend21:03:161
344156699190,1cyclictest3056-21PK-Backend20:17:281
344156699190,1cyclictest3056-21PK-Backend20:17:281
344156699190,1cyclictest3056-21PK-Backend19:34:291
344156699190,1cyclictest3056-21PK-Backend00:39:281
344156699190,18cyclictest3056-21PK-Backend21:42:241
3441562991917,1cyclictest3056-21PK-Backend20:15:190
3441562991917,1cyclictest3056-21PK-Backend20:15:190
3441562991917,1cyclictest0-21swapper/021:07:450
3441562991917,1cyclictest0-21swapper/020:27:030
3441562991917,1cyclictest0-21swapper/020:22:030
3441562991917,1cyclictest0-21swapper/019:49:240
3441562991917,1cyclictest0-21swapper/000:12:050
3441562991914,3cyclictest3504976-21idleruntime-cro22:09:590
3441562991913,5cyclictest3550052-21cat00:15:230
344156299190,1cyclictest0-21swapper/019:10:310
344156299190,1cyclictest0-21swapper/000:30:570
344156299190,1cyclictest0-21swapper/000:26:190
344156299190,18cyclictest0-21swapper/022:37:220
139099190,13rtkit-daemon3056-21PK-Backend20:26:031
139099190,13rtkit-daemon0-21swapper/023:40:390
139099190,13rtkit-daemon0-21swapper/021:00:410
344157399189,7cyclictest3459889-21/usr/sbin/munin20:00:123
344157399188,8cyclictest3484209-21unin-run21:09:593
3441573991811,5cyclictest3493012-21cut21:35:003
3441573991810,6cyclictest42-21ksoftirqd/300:00:233
3441570991816,1cyclictest0-21swapper/222:55:082
3441570991816,1cyclictest0-21swapper/221:52:552
3441570991816,1cyclictest0-21swapper/221:25:382
3441570991816,1cyclictest0-21swapper/221:15:162
3441570991816,1cyclictest0-21swapper/221:15:162
3441570991816,1cyclictest0-21swapper/219:55:132
3441570991812,4cyclictest3518373-21ntpq22:45:202
3441566991816,1cyclictest3056-21PK-Backend20:13:481
3441566991816,1cyclictest3056-21PK-Backend20:09:021
3441566991816,1cyclictest3056-21PK-Backend00:30:141
3441566991816,1cyclictest0-21swapper/123:04:101
3441562991816,1cyclictest0-21swapper/021:48:560
3441562991812,5cyclictest3444559-21cat19:15:190
344156299180,1cyclictest0-21swapper/022:04:360
344157399179,2cyclictest42-21ksoftirqd/322:55:033
344157399172,11cyclictest3457928-21idleruntime-cro19:55:003
3441573991715,1cyclictest3056-21PK-Backend22:23:253
3441566991715,1cyclictest3056-21PK-Backend20:42:011
3441562991712,3cyclictest3543708-21latency_hist00:00:010
3441562991712,3cyclictest3543708-21latency_hist00:00:010
3441573991610,4cyclictest42-21ksoftirqd/319:20:233
3441573991610,4cyclictest42-21ksoftirqd/319:20:223
344157099169,6cyclictest883-21dbus-daemon19:25:002
344157099169,6cyclictest883-21dbus-daemon19:25:002
344157099169,6cyclictest3470575-21apt20:30:132
344157099168,6cyclictest3492455-21users21:30:232
3441570991610,4cyclictest3545804-21tr00:05:162
3441570991610,4cyclictest3545804-21tr00:05:162
3441566991612,2cyclictest26-21ksoftirqd/119:46:091
3441562991614,1cyclictest0-21swapper/023:08:450
3441562991614,1cyclictest0-21swapper/023:08:450
3441562991614,1cyclictest0-21swapper/021:42:190
344156699157,6cyclictest26-21ksoftirqd/120:20:091
344156699145,7cyclictest26-21ksoftirqd/120:00:221
3441566991412,1cyclictest3056-21PK-Backend19:54:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional