You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 16:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Sat Mar 14, 2026 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450350,0irq/25-eth10-21swapper/306:57:473
1281623324,0sleep20-21swapper/206:59:192
1269823324,0sleep10-21swapper/106:57:581
1266423329,0sleep00-21swapper/006:57:330
4160,0ktimersoftd/00-21swapper/009:08:430
339960,0migration/331379-21munin-node12:11:473
339960,0migration/326490-21latency_hist10:51:243
339960,0migration/326066-21sh10:29:363
339960,0migration/322844-21switchtime08:41:463
339960,0migration/316309-21sh11:21:493
339960,0migration/312752-21irqcore09:21:393
29715060,0irq/26-eth0-rx-17784-21diskstats12:06:353
149960,0migration/032238-21sh11:31:480
149960,0migration/023873-21sh12:28:560
149960,0migration/02259-21sh09:14:080
149960,0migration/015776-21sh12:23:330
130709960,0cyclictest0-21swapper/212:11:392
130709960,0cyclictest0-21swapper/211:45:582
130669966,0cyclictest0-21swapper/009:41:400
339950,0migration/39750-21sh10:18:443
339950,0migration/38052-21timerandwakeup11:16:463
339950,0migration/36385-21munin-node07:16:453
339950,0migration/35368-21kthreadcore08:31:413
339950,0migration/34064-21munin-node08:51:473
339950,0migration/34064-21munin-node08:51:473
339950,0migration/34036-21kthreadcore10:56:413
339950,0migration/33485-21kthreadcore11:56:403
339950,0migration/331316-21munin-node11:51:453
339950,0migration/331179-21kthreadcore11:31:413
339950,0migration/329083-21taskset08:25:153
339950,0migration/326084-21kthreadcore07:56:433
339950,0migration/325867-21kthreadcore08:46:403
339950,0migration/32496-21munin-node07:41:383
339950,0migration/32351-21taskset08:05:053
339950,0migration/322129-21munin-node07:31:423
339950,0migration/319500-21kthreadcore11:46:393
339950,0migration/319154-21latency_hist10:06:243
339950,0migration/311473-21kthreadcore07:46:433
339950,0migration/311472-21kthreadcore11:41:393
29725050,0irq/27-eth0-tx-4510-21kthreadcore11:56:402
29725050,0irq/27-eth0-tx-27925-21munin-node09:31:362
29725050,0irq/27-eth0-tx-27538-21ssh09:51:352
29725050,0irq/27-eth0-tx-20340-21idleruntime09:06:382
29725050,0irq/27-eth0-tx-19318-21ssh10:26:372
29725050,0irq/27-eth0-tx-18150-21munin-node11:26:362
29725050,0irq/27-eth0-tx-13647-21grep11:01:402
29725050,0irq/27-eth0-tx-0-21swapper/211:22:322
29725050,0irq/27-eth0-tx-0-21swapper/210:13:042
29725050,0irq/27-eth0-tx-0-21swapper/209:56:412
29725050,0irq/27-eth0-tx-0-21swapper/209:48:102
29725050,0irq/27-eth0-tx-0-21swapper/209:14:592
29725050,0irq/27-eth0-tx-0-21swapper/008:48:450
29725050,0irq/27-eth0-tx-0-21swapper/007:26:410
29715050,0irq/26-eth0-rx-8216-21nfs4_client10:36:433
29715050,0irq/26-eth0-rx-6832-21python12:16:463
29715050,0irq/26-eth0-rx-6559-21runrttasks10:41:353
29715050,0irq/26-eth0-rx-3676-21cpu09:16:353
29715050,0irq/26-eth0-rx-31581-21cut11:11:423
29715050,0irq/26-eth0-rx-3129-21ssh09:56:313
29715050,0irq/26-eth0-rx-27537-21if_eth1.207:11:403
29715050,0irq/26-eth0-rx-27108-21latency_hist10:11:243
29715050,0irq/26-eth0-rx-22509-21grep11:06:403
29715050,0irq/26-eth0-rx-21951-21grep09:26:433
29715050,0irq/26-eth0-rx-1900-21ssh09:13:023
29715050,0irq/26-eth0-rx-17851-21tr07:01:413
29715050,0irq/26-eth0-rx-14808-21missed_timers12:01:413
29715050,0irq/26-eth0-rx-0-21swapper/312:21:403
29715050,0irq/26-eth0-rx-0-21swapper/311:39:263
29715050,0irq/26-eth0-rx-0-21swapper/311:28:253
29715050,0irq/26-eth0-rx-0-21swapper/310:35:433
29715050,0irq/26-eth0-rx-0-21swapper/310:21:423
29715050,0irq/26-eth0-rx-0-21swapper/310:04:093
29715050,0irq/26-eth0-rx-0-21swapper/309:46:383
29715050,0irq/26-eth0-rx-0-21swapper/309:41:123
29715050,0irq/26-eth0-rx-0-21swapper/309:10:393
27150,0ktimersoftd/20-21swapper/211:31:372
27150,0ktimersoftd/20-21swapper/210:06:382
259950,0migration/227427-21if_eth011:11:392
259950,0migration/227373-21sh10:31:352
259950,0migration/225272-21sh11:08:402
259950,0migration/22520-21latency_hist10:36:242
259950,0migration/222357-21latency08:41:412
259950,0migration/215747-21missed_timers08:36:422
259950,0migration/213451-21kthreadcore12:21:402
249950,0watchdog/21621-21idleruntime07:16:392
249950,0watchdog/20-21swapper/211:36:382
179950,0migration/1776-21munin-node09:11:431
179950,0migration/17084-21taskset07:18:481
179950,0migration/126064-21kthreadcore07:31:431
179950,0migration/12596-21latency_hist08:06:251
179950,0migration/123978-21taskset11:47:041
179950,0migration/118544-21sh10:25:561
179950,0migration/113196-21latency_hist07:01:251
179950,0migration/112408-21kthreadcore08:36:401
179950,0migration/111482-21aten_r1power_en09:01:351
169950,0watchdog/126699-21ssh09:28:381
169950,0watchdog/124887-21sort09:46:421
149950,0migration/07268-21kthreadcore10:36:410
149950,0migration/06453-21kthreadcore11:16:410
149950,0migration/04185-21kthreadcore11:36:380
149950,0migration/031050-21kthreadcore10:51:420
149950,0migration/028483-21if_eth1.307:36:400
149950,0migration/027973-21taskset07:58:300
149950,0migration/027303-21aten_r1power_fr10:11:340
149950,0migration/026120-21taskset09:48:080
149950,0migration/022240-21kthreadcore11:46:390
149950,0migration/021592-21kthreadcore07:06:420
149950,0migration/018793-21kthreadcore07:51:400
149950,0migration/018049-21fschecks_time12:06:350
149950,0migration/017999-21munin-node08:41:390
149950,0migration/015501-21kthreadcore07:01:410
149950,0migration/014946-21kthreadcore10:41:420
149950,0migration/013994-21kthreadcore09:01:410
149950,0migration/01317-21sh10:33:310
149950,0migration/012560-21kthreadcore07:21:410
149950,0migration/010735-21kthreadcore07:46:420
149950,0migration/010270-21fschecks_count11:41:360
130709955,0cyclictest0-21swapper/210:51:442
130709955,0cyclictest0-21swapper/209:21:332
130709950,0cyclictest0-21swapper/207:11:382
130679950,0cyclictest11192-21diskmemload09:18:521
130669955,0cyclictest0-21swapper/011:06:380
130669955,0cyclictest0-21swapper/010:58:550
130669955,0cyclictest0-21swapper/010:47:430
130669955,0cyclictest0-21swapper/009:29:310
130669950,0cyclictest0-21swapper/012:15:040
130669950,0cyclictest0-21swapper/011:53:560
339940,0migration/35060-21kthreadcore08:06:423
339940,0migration/318723-21kthreadcore07:26:433
339940,0migration/316829-21sh11:02:223
339940,0migration/315439-21cpuspeed_turbos07:51:353
339940,0migration/312085-21kthreadcore08:36:403
339940,0migration/310805-21kthreadcore08:11:413
29725040,0irq/27-eth0-tx-0-21swapper/212:07:152
29725040,0irq/27-eth0-tx-0-21swapper/212:04:102
29725040,0irq/27-eth0-tx-0-21swapper/211:51:272
29725040,0irq/27-eth0-tx-0-21swapper/210:56:302
29725040,0irq/27-eth0-tx-0-21swapper/210:41:382
29725040,0irq/27-eth0-tx-0-21swapper/210:21:312
29725040,0irq/27-eth0-tx-0-21swapper/210:17:062
29725040,0irq/27-eth0-tx-0-21swapper/209:36:452
29725040,0irq/27-eth0-tx-0-21swapper/209:16:282
29725040,0irq/27-eth0-tx-0-21swapper/209:02:502
29725040,0irq/27-eth0-tx-0-21swapper/008:21:380
29725040,0irq/27-eth0-tx-0-21swapper/008:16:360
29725040,0irq/27-eth0-tx-0-21swapper/007:31:480
29715040,0irq/26-eth0-rx-16576-21fschecks_count08:16:383
29715040,0irq/26-eth0-rx-0-21swapper/312:26:303
29715040,0irq/26-eth0-rx-0-21swapper/310:52:183
29715040,0irq/26-eth0-rx-0-21swapper/309:51:423
29715040,0irq/26-eth0-rx-0-21swapper/309:41:403
29715040,0irq/26-eth0-rx-0-21swapper/309:32:213
29715040,0irq/26-eth0-rx-0-21swapper/309:01:373
29715040,0irq/26-eth0-rx-0-21swapper/308:26:423
259940,0migration/229596-21timerwakeupswit08:46:462
259940,0migration/22307-21fschecks_time07:41:382
259940,0migration/220281-21kthreadcore11:46:392
259940,0migration/219816-21latency_hist09:26:252
259940,0migration/212978-21munin-node07:21:432
259940,0migration/210045-21fschecks_time08:11:382
249940,0watchdog/224515-21sensors10:46:462
249940,0watchdog/216855-21munin-node10:01:422
249940,0watchdog/214620-21forks07:26:392
249940,0watchdog/20-21swapper/212:26:422
249940,0watchdog/20-21swapper/207:31:392
179940,0migration/18054-21latency10:36:421
179940,0migration/17012-21kthreadcore08:31:421
179940,0migration/16286-21kthreadcore12:16:421
179940,0migration/15987-21kthreadcore10:16:411
179940,0migration/14545-21kthreadcore08:06:421
179940,0migration/130624-21cpuspeed_turbos08:51:361
179940,0migration/130624-21cpuspeed_turbos08:51:361
179940,0migration/130120-21kthreadcore11:11:411
179940,0migration/128115-21fschecks_time07:36:381
179940,0migration/127030-21irqrtprio11:51:411
179940,0migration/119827-21kthreadcore07:51:411
179940,0migration/119069-21aten_r1power_fr10:26:361
179940,0migration/118599-21latency_hist10:46:241
179940,0migration/117008-21latency_hist08:41:251
179940,0migration/116952-21if_err_eth008:16:391
179940,0migration/115431-21kthreadcore10:41:421
179940,0migration/112834-21kthreadcore07:46:431
179940,0migration/112447-21gltestperf09:21:371
179940,0migration/110530-21irqcore12:01:391
169940,0watchdog/19810-21hostname11:41:341
169940,0watchdog/130146-21kthreadcore12:11:421
169940,0watchdog/13000-21ps11:36:381
169940,0watchdog/125266-21cut07:56:431
169940,0watchdog/12307-21memory08:26:431
169940,0watchdog/118461-21irqcore12:06:381
169940,0watchdog/118360-21ssh10:04:381
169940,0watchdog/10-21swapper/110:58:141
149940,0migration/04879-21kthreadcore07:16:420
149940,0migration/028501-21aten_r1power_en08:01:350
149940,0migration/016832-21sh12:05:370
149940,0migration/016527-21nfs_client10:21:440
130709943,0cyclictest9488-21awk07:46:412
130709943,0cyclictest3556-21lldpd12:16:402
130709943,0cyclictest3556-21lldpd11:16:412
130709943,0cyclictest3556-21lldpd07:40:102
130709943,0cyclictest3514-21snmpd08:51:372
130709943,0cyclictest3514-21snmpd08:51:372
130709943,0cyclictest3514-21snmpd08:26:372
130709943,0cyclictest3514-21snmpd08:06:372
130709943,0cyclictest3514-21snmpd07:02:072
130709943,0cyclictest12432-21awk09:41:372
130679943,0cyclictest9897-21awk12:21:371
130679943,0cyclictest5590-21awk08:56:391
130679943,0cyclictest4465-21awk09:36:381
130679943,0cyclictest4045-21awk09:56:381
130679943,0cyclictest3514-21snmpd10:35:531
130679943,0cyclictest3514-21snmpd08:23:071
130679943,0cyclictest3241-21irqbalance10:07:521
130679943,0cyclictest28457-21awk09:51:391
130679943,0cyclictest27718-21awk10:51:401
130679943,0cyclictest27420-21awk07:11:391
130679943,0cyclictest2246-21awk11:56:381
130679943,0cyclictest17938-21awk12:26:391
130679943,0cyclictest12553-21awk09:41:381
130679943,0cyclictest11045-21awk11:21:371
130679943,0cyclictest10295-21awk08:11:391
130679940,0cyclictest0-21swapper/111:19:101
130669944,0cyclictest540-21idleruntime-cro07:16:250
130669944,0cyclictest0-21swapper/011:02:230
130669944,0cyclictest0-21swapper/010:04:410
130669943,0cyclictest4464-21awk09:16:390
130669943,0cyclictest3879-21awk10:16:390
130669943,0cyclictest3556-21lldpd11:11:400
130669943,0cyclictest3514-21snmpd09:55:370
130669943,0cyclictest3514-21snmpd08:32:070
130669943,0cyclictest3241-21irqbalance09:32:030
130669943,0cyclictest2321-21awk11:56:380
130669943,0cyclictest19911-21awk10:26:400
130669943,0cyclictest11060-21awk11:21:370
29715030,0irq/26-eth0-rx-0-21swapper/308:57:333
29715030,0irq/26-eth0-rx-0-21swapper/307:23:343
259930,0migration/28156-21kthreadcore08:56:422
249930,0watchdog/20-21swapper/208:31:392
249930,0watchdog/20-21swapper/208:16:432
249930,0watchdog/20-21swapper/207:51:352
179930,0migration/127194-21sh10:11:281
179930,0migration/116773-21sh11:02:131
169930,0watchdog/10-21swapper/111:31:341
169930,0watchdog/10-21swapper/111:06:501
169930,0watchdog/10-21swapper/109:06:341
169930,0watchdog/10-21swapper/107:26:391
169930,0watchdog/10-21swapper/107:21:271
130709933,0cyclictest22766-21awk07:56:412
130709932,0cyclictest3556-21lldpd08:02:392
130709932,0cyclictest3514-21snmpd08:21:372
130709932,0cyclictest20833-21awk07:06:402
130679932,0cyclictest3924-21kthreadcore07:41:411
130679932,0cyclictest25584-21sensors07:06:461
130679932,0cyclictest23913-21aten_r1power_vo08:46:361
130679932,0cyclictest19002-21awk11:26:381
130669933,0cyclictest28400-21kworker/0:012:16:250
130669933,0cyclictest28400-21kworker/0:011:26:290
130669933,0cyclictest28400-21kworker/0:010:07:090
130669933,0cyclictest28400-21kworker/0:009:36:360
130669933,0cyclictest28400-21kworker/0:009:21:380
130669933,0cyclictest28400-21kworker/0:008:56:320
130669933,0cyclictest28400-21kworker/0:008:51:280
130669933,0cyclictest28400-21kworker/0:008:51:280
130669933,0cyclictest28400-21kworker/0:008:36:320
130669933,0cyclictest28400-21kworker/0:008:26:280
130669933,0cyclictest28400-21kworker/0:008:11:280
130669933,0cyclictest28400-21kworker/0:008:06:320
130669933,0cyclictest28400-21kworker/0:007:41:280
130669930,0cyclictest3839-21perl09:56:370
29715020,0irq/26-eth0-rx-0-21swapper/307:36:423
29715020,0irq/26-eth0-rx-0-21swapper/307:06:453
130679920,0cyclictest0-21swapper/109:31:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional