You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 17:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sun Nov 23, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
743828988,0sleep20-21swapper/206:38:272
728127978,0sleep10-21swapper/106:36:381
291350380,0irq/25-eth10-21swapper/306:34:393
719923228,0sleep00-21swapper/006:35:410
76049999,0cyclictest5824-21awk07:54:480
8225070,0irq/24-0000:00:2085-21cat06:59:342
76079965,0cyclictest17022-21grep07:59:522
339960,0migration/35475-21sh10:06:373
339960,0migration/329491-21sh10:45:143
339960,0migration/328931-21pmu-power09:59:543
28415060,0irq/27-eth0-tx-7818-21munin-node06:40:020
28415060,0irq/27-eth0-tx-6372-21iceccd08:34:500
28415060,0irq/27-eth0-tx-532-21ssh08:40:050
28415060,0irq/27-eth0-tx-275992sleep211:24:572
28415060,0irq/27-eth0-tx-0-21swapper/211:48:472
28415060,0irq/27-eth0-tx-0-21swapper/211:30:582
28415060,0irq/27-eth0-tx-0-21swapper/211:21:272
28415060,0irq/27-eth0-tx-0-21swapper/210:59:192
28415060,0irq/27-eth0-tx-0-21swapper/210:46:592
28415060,0irq/27-eth0-tx-0-21swapper/210:25:212
28415060,0irq/27-eth0-tx-0-21swapper/210:25:212
28415060,0irq/27-eth0-tx-0-21swapper/210:07:032
28415060,0irq/27-eth0-tx-0-21swapper/210:02:112
28415060,0irq/27-eth0-tx-0-21swapper/209:53:292
28415060,0irq/27-eth0-tx-0-21swapper/209:27:122
28415060,0irq/27-eth0-tx-0-21swapper/209:23:072
28415060,0irq/27-eth0-tx-0-21swapper/208:58:002
28415060,0irq/27-eth0-tx-0-21swapper/208:42:452
28405060,0irq/26-eth0-rx-19644-21ssh09:14:523
169960,0watchdog/18808-21munin-node06:39:571
169960,0watchdog/124719-21cat10:19:521
149960,0migration/06232-21pmu-power09:24:550
149960,0migration/022739-21taskset09:58:340
149960,0migration/020900-21sh11:23:410
8225050,0irq/24-0000:00:9502-21munin-run07:09:342
76079954,0cyclictest0-21swapper/207:44:342
76079950,0cyclictest0-21swapper/209:45:112
76059953,0cyclictest18553-21kworker/1:109:19:551
76049955,0cyclictest26643-21diskmemload09:33:570
76049955,0cyclictest26643-21diskmemload09:13:180
76049955,0cyclictest0-21swapper/009:02:160
76049950,0cyclictest0-21swapper/011:30:460
339950,0migration/3885-21irqrtprio09:44:533
339950,0migration/38170-21kthreadcore07:04:513
339950,0migration/38080-21kthreadcore08:19:503
339950,0migration/3801-21kthreadcore11:49:523
339950,0migration/34845-21munin-node08:14:553
339950,0migration/34726-21latency_hist11:34:333
339950,0migration/34444-21munin-node11:54:433
339950,0migration/33366-21taskset11:30:573
339950,0migration/326827-21kthreadcore09:39:533
339950,0migration/326764-21unixbench-2d11:44:573
339950,0migration/326193-21kthreadcore10:19:533
339950,0migration/325953-21kthreadcore11:24:503
339950,0migration/322274-21latency_hist11:04:333
339950,0migration/322186-21timerandwakeup09:34:573
339950,0migration/320042-21diskmemload11:20:463
339950,0migration/317427-21kthreadcore10:59:513
339950,0migration/313818-21taskset09:50:323
339950,0migration/311471-21munin-node10:29:563
28415050,0irq/27-eth0-tx-6372-21iceccd06:46:320
28415050,0irq/27-eth0-tx-24108-21munin-node09:39:552
28415050,0irq/27-eth0-tx-16312-21grep11:59:532
28415050,0irq/27-eth0-tx-0-21swapper/211:52:392
28415050,0irq/27-eth0-tx-0-21swapper/211:41:462
28415050,0irq/27-eth0-tx-0-21swapper/211:35:212
28415050,0irq/27-eth0-tx-0-21swapper/211:17:182
28415050,0irq/27-eth0-tx-0-21swapper/211:12:212
28415050,0irq/27-eth0-tx-0-21swapper/211:04:422
28415050,0irq/27-eth0-tx-0-21swapper/210:40:492
28415050,0irq/27-eth0-tx-0-21swapper/210:21:482
28415050,0irq/27-eth0-tx-0-21swapper/210:16:302
28415050,0irq/27-eth0-tx-0-21swapper/210:14:002
28415050,0irq/27-eth0-tx-0-21swapper/209:34:042
28415050,0irq/27-eth0-tx-0-21swapper/209:10:152
28415050,0irq/27-eth0-tx-0-21swapper/208:45:142
28415050,0irq/27-eth0-tx-0-21swapper/008:29:560
28415050,0irq/27-eth0-tx-0-21swapper/008:01:360
28415050,0irq/27-eth0-tx-0-21swapper/007:19:450
28405050,0irq/26-eth0-rx-5918-21ssh10:53:083
28405050,0irq/26-eth0-rx-2698-21kthreadcore10:24:533
28405050,0irq/26-eth0-rx-2698-21kthreadcore10:24:533
28405050,0irq/26-eth0-rx-25890-21hwlatdetect09:19:533
28405050,0irq/26-eth0-rx-21735-21ssh10:40:323
28405050,0irq/26-eth0-rx-21302-21if_eth1.312:04:503
28405050,0irq/26-eth0-rx-20908-21chrt06:49:043
28405050,0irq/26-eth0-rx-1202-21ssh09:04:153
28405050,0irq/26-eth0-rx-0-21swapper/310:35:473
28405050,0irq/26-eth0-rx-0-21swapper/310:11:033
28405050,0irq/26-eth0-rx-0-21swapper/309:31:303
28405050,0irq/26-eth0-rx-0-21swapper/308:54:523
28405050,0irq/26-eth0-rx-0-21swapper/308:49:523
28405050,0irq/26-eth0-rx-0-21swapper/308:47:423
27150,0ktimersoftd/20-21swapper/209:04:022
259950,0migration/25781-21sh10:29:352
259950,0migration/231040-21latency_hist10:49:342
259950,0migration/224707-21kthreadcore12:04:522
259950,0migration/221694-21kthreadcore08:04:512
259950,0migration/220919-21kthreadcore09:14:532
259950,0migration/215246-21latency_hist10:39:342
19150,0ktimersoftd/10-21swapper/109:18:301
19150,0ktimersoftd/10-21swapper/109:04:021
179950,0migration/19261-21kthreadcore08:19:501
179950,0migration/129881-21kthreadcore06:54:501
179950,0migration/126621-21timerandwakeup11:44:571
179950,0migration/124928-21idleruntime09:39:511
169950,0watchdog/17982-21ssh09:05:281
169950,0watchdog/14708-21df_inode11:54:441
169950,0watchdog/119557-21kthreadcore07:14:521
169950,0watchdog/113277-21fschecks_count11:39:481
169950,0watchdog/111328-21if_eth1.208:49:521
169950,0watchdog/10-21swapper/111:29:561
169950,0watchdog/10-21swapper/111:13:521
169950,0watchdog/10-21swapper/110:54:481
169950,0watchdog/10-21swapper/110:34:321
169950,0watchdog/10-21swapper/110:15:081
169950,0watchdog/10-21swapper/108:44:521
169950,0watchdog/10-21swapper/107:24:521
149950,0migration/08613-21sh08:45:220
149950,0migration/08166-21kthreadcore10:09:510
149950,0migration/07355-21kthreadcore10:29:530
149950,0migration/07041-21taskset09:49:050
149950,0migration/06510-21kthreadcore09:04:520
149950,0migration/0544-21kthreadcore10:49:500
149950,0migration/04818-21aten_r1power_cu11:34:450
149950,0migration/04590-21kthreadcore07:04:500
149950,0migration/031778-21taskset09:21:380
149950,0migration/029888-21munin-node07:44:580
149950,0migration/028759-21needreboot10:44:540
149950,0migration/028625-21kthreadcore09:39:540
149950,0migration/025546-21kthreadcore11:24:500
149950,0migration/024431-21kthreadcore12:04:520
149950,0migration/023473-21irqcore11:04:510
149950,0migration/023260-21kthreadcore08:04:510
149950,0migration/022993-21kthreadcore11:44:510
149950,0migration/022419-21sh10:19:360
149950,0migration/021794-21munin-node07:14:540
149950,0migration/020980-21sh10:15:430
149950,0migration/020086-21latency_hist12:04:330
149950,0migration/016705-21kthreadcore11:39:520
149950,0migration/01119-21kthreadcore11:49:520
149950,0migration/01018-21kthreadcore07:24:530
76079943,0cyclictest6445-21snmpd08:25:582
76079943,0cyclictest6445-21snmpd07:48:132
76079943,0cyclictest6445-21snmpd07:35:282
76079943,0cyclictest3425-21lldpd08:22:092
76079943,0cyclictest3425-21lldpd07:24:372
76079943,0cyclictest17210-21awk07:14:502
76059944,0cyclictest18553-21kworker/1:107:59:521
76059943,0cyclictest6707-21awk11:14:481
76059943,0cyclictest6445-21snmpd12:07:441
76059943,0cyclictest6445-21snmpd10:39:281
76059943,0cyclictest6445-21snmpd09:35:131
76059943,0cyclictest6445-21snmpd09:09:431
76059943,0cyclictest3425-21lldpd12:01:211
76059943,0cyclictest32092-21awk10:49:491
76059943,0cyclictest31224-21awk10:24:501
76059943,0cyclictest31224-21awk10:24:501
76059943,0cyclictest3110-21irqbalance11:21:181
76059943,0cyclictest3110-21irqbalance10:59:381
76059943,0cyclictest3110-21irqbalance10:05:081
76059943,0cyclictest3110-21irqbalance08:40:181
76059943,0cyclictest3110-21irqbalance08:34:381
76059943,0cyclictest3110-21irqbalance08:25:081
76059943,0cyclictest3110-21irqbalance08:14:481
76059943,0cyclictest3110-21irqbalance07:44:381
76059943,0cyclictest3110-21irqbalance07:39:481
76059943,0cyclictest3110-21irqbalance07:35:481
76059943,0cyclictest3110-21irqbalance07:29:581
76059943,0cyclictest3110-21irqbalance07:09:481
76059943,0cyclictest3110-21irqbalance07:04:381
76059943,0cyclictest3110-21irqbalance07:00:481
76059943,0cyclictest3110-21irqbalance06:44:381
76059943,0cyclictest24330-21awk09:59:491
76059943,0cyclictest23880-21awk07:19:481
76059943,0cyclictest14351-21awk08:29:481
76049944,0cyclictest0-21swapper/011:01:270
76049944,0cyclictest0-21swapper/010:41:540
76049943,0cyclictest3425-21lldpd11:18:130
76049943,0cyclictest3425-21lldpd09:18:320
76049943,0cyclictest32655-21awk08:14:500
76049943,0cyclictest31325-21awk10:24:500
76049943,0cyclictest31325-21awk10:24:500
76049943,0cyclictest31149-21awk11:09:490
76049943,0cyclictest21888-21sed09:34:540
76049943,0cyclictest19155-21awk08:54:510
76049943,0cyclictest1869-21sadc06:59:340
76049943,0cyclictest17363-21munin-node07:39:490
76049943,0cyclictest12040-21sadc11:59:330
76049943,0cyclictest11595-21awk07:34:480
76049943,0cyclictest10586-21awk07:09:500
339940,0migration/36416-21switchtime09:24:573
339940,0migration/35496-21fschecks_count07:54:473
339940,0migration/34797-21idleruntime07:29:473
339940,0migration/329780-21kthreadcore08:39:523
339940,0migration/329332-21missed_timers07:44:543
339940,0migration/32344-21kthreadcore11:09:523
339940,0migration/320118-21kthreadcore09:54:523
339940,0migration/319732-21kthreadcore07:14:533
339940,0migration/314364-21kthreadcore07:59:513
339940,0migration/314146-21kthreadcore09:09:533
339940,0migration/313448-21kthreadcore07:34:513
339940,0migration/31307-21kthreadcore07:49:523
339940,0migration/312672-21diskmemload11:17:243
28415040,0irq/27-eth0-tx-9845-21ssh11:54:512
28415040,0irq/27-eth0-tx-10332-21ssh08:49:422
28415040,0irq/27-eth0-tx-0-21swapper/210:59:452
28415040,0irq/27-eth0-tx-0-21swapper/008:09:540
28405040,0irq/26-eth0-rx-0-21swapper/312:00:083
28405040,0irq/26-eth0-rx-0-21swapper/311:40:023
28405040,0irq/26-eth0-rx-0-21swapper/310:54:493
28405040,0irq/26-eth0-rx-0-21swapper/309:08:123
259940,0migration/27592-21kthreadcore07:29:502
259940,0migration/230902-21kthreadcore06:54:502
259940,0migration/229948-21kthreadcore08:09:522
259940,0migration/226229-21kthreadcore07:19:512
259940,0migration/219816-21runrttasks08:32:482
259940,0migration/216882-21fschecks_time09:34:472
259940,0migration/21616-21aten_r1power_cu09:04:452
259940,0migration/215782-21df_abs09:54:472
259940,0migration/211299-21runrttasks07:57:422
249940,0watchdog/21753-21kthreadcore07:49:522
249940,0watchdog/20-21swapper/208:34:362
249940,0watchdog/20-21swapper/208:14:522
179940,0migration/16679-21latency_hist10:09:341
179940,0migration/14898-21aten_r1power_fr11:34:451
179940,0migration/13784-21kthreadcore09:24:521
179940,0migration/127466-21unixbench-2d11:24:561
179940,0migration/125903-21if_err_eth1.308:09:491
179940,0migration/124817-21ksmtuned08:56:261
179940,0migration/124112-21idleruntime10:44:491
179940,0migration/11612-21kthreadcore07:49:521
169940,0watchdog/13411-21lldpd07:54:361
169940,0watchdog/121348-21cut06:49:441
169940,0watchdog/119251-21munin-node08:04:481
169940,0watchdog/115382-21ssh09:54:361
169940,0watchdog/113647-21runrttasks09:50:001
169940,0watchdog/10-21swapper/111:49:561
169940,0watchdog/10-21swapper/111:04:521
169940,0watchdog/10-21swapper/109:44:481
169940,0watchdog/10-21swapper/109:29:561
149940,0migration/011868-21kthreadcore09:49:530
76049933,0cyclictest0-21swapper/010:05:100
76049933,0cyclictest0-21swapper/008:20:100
76049932,0cyclictest6445-21snmpd08:26:390
76049932,0cyclictest24215-21awk09:59:480
76049932,0cyclictest22064-21awk06:49:470
76049932,0cyclictest0-21swapper/007:29:450
76049930,0cyclictest10767-21diskstats08:49:500
76049930,0cyclictest0-21swapper/007:53:030
339930,0migration/327435-21runrttasks06:52:273
28405030,0irq/26-eth0-rx-0-21swapper/310:14:353
28405030,0irq/26-eth0-rx-0-21swapper/308:37:163
28405030,0irq/26-eth0-rx-0-21swapper/308:29:443
28405030,0irq/26-eth0-rx-0-21swapper/308:29:253
28405030,0irq/26-eth0-rx-0-21swapper/308:09:463
28405030,0irq/26-eth0-rx-0-21swapper/307:39:463
28405030,0irq/26-eth0-rx-0-21swapper/307:24:483
28405030,0irq/26-eth0-rx-0-21swapper/307:10:003
28405030,0irq/26-eth0-rx-0-21swapper/306:59:493
28405030,0irq/26-eth0-rx-0-21swapper/306:57:513
28405030,0irq/26-eth0-rx-0-21swapper/306:39:523
27130,0ktimersoftd/215442-21if_eth006:44:492
249930,0watchdog/20-21swapper/207:09:522
249930,0watchdog/20-21swapper/206:49:442
249930,0watchdog/20-21swapper/206:43:362
169930,0watchdog/10-21swapper/110:39:481
149930,0migration/013802-21diskmemload10:35:150
76049921,0cyclictest6685-21basename10:54:440
76049921,0cyclictest0-21swapper/006:59:450
35120,0ktimersoftd/322885-21kthreadcore08:04:513
28405020,0irq/26-eth0-rx-27238-21cut07:19:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional