You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 14:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Jul 12, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
953628180,0sleep10-21swapper/106:43:141
291350630,0irq/25-eth10-21swapper/306:41:233
960324026,0sleep00-21swapper/006:44:010
964423625,0sleep20-21swapper/206:44:272
249970,0watchdog/229348-21kthreadcore07:49:522
97259965,0cyclictest27956-21ps08:14:532
97249960,0cyclictest0-21swapper/112:14:051
97239960,0cyclictest0-21swapper/012:06:370
8225060,0irq/24-0000:00:0-21swapper/211:54:322
339960,0migration/39534-21sh11:32:583
339960,0migration/39149-21sh12:11:533
339960,0migration/327423-21sh10:24:013
339960,0migration/326122-21sh10:42:343
339960,0migration/323987-21munin-node11:39:563
339960,0migration/320613-21sh09:57:263
339960,0migration/319702-21sh10:19:093
339960,0migration/317249-21sh11:36:563
339960,0migration/316940-21sh10:55:263
339960,0migration/31179-21sh11:27:233
28415060,0irq/27-eth0-tx-25017-21ssh11:00:522
28415060,0irq/27-eth0-tx-0-21swapper/212:00:402
28415060,0irq/27-eth0-tx-0-21swapper/211:42:012
28415060,0irq/27-eth0-tx-0-21swapper/211:37:032
28415060,0irq/27-eth0-tx-0-21swapper/211:29:372
28415060,0irq/27-eth0-tx-0-21swapper/211:09:482
28415060,0irq/27-eth0-tx-0-21swapper/210:58:072
28415060,0irq/27-eth0-tx-0-21swapper/210:53:372
28415060,0irq/27-eth0-tx-0-21swapper/210:16:472
28415060,0irq/27-eth0-tx-0-21swapper/210:10:142
28415060,0irq/27-eth0-tx-0-21swapper/210:04:522
28415060,0irq/27-eth0-tx-0-21swapper/209:39:072
28415060,0irq/27-eth0-tx-0-21swapper/209:30:422
28415060,0irq/27-eth0-tx-0-21swapper/209:23:022
28415060,0irq/27-eth0-tx-0-21swapper/209:14:282
28415060,0irq/27-eth0-tx-0-21swapper/208:55:502
28415060,0irq/27-eth0-tx-0-21swapper/008:44:350
28405060,0irq/26-eth0-rx-5859-21cut11:49:503
249960,0watchdog/20-21swapper/211:52:392
249960,0watchdog/20-21swapper/211:14:472
149960,0migration/03511-21kernelversion11:49:490
149960,0migration/026109-21munin-node08:09:580
149960,0migration/014946-21kthreadcore08:04:510
97259950,0cyclictest0-21swapper/207:59:252
97249955,0cyclictest1813-21diskmemload10:34:361
97249955,0cyclictest0-21swapper/110:04:581
97249954,0cyclictest11860-21ps07:09:501
97249950,0cyclictest0-21swapper/110:56:021
97249950,0cyclictest0-21swapper/109:29:581
97239950,0cyclictest0-21swapper/010:41:000
97239950,0cyclictest0-21swapper/010:33:360
97239950,0cyclictest0-21swapper/010:18:280
97239950,0cyclictest0-21swapper/010:10:440
97239950,0cyclictest0-21swapper/009:26:360
8225050,0irq/24-0000:00:14243-21latency_hist08:29:332
339950,0migration/3477-21latency_hist09:04:333
339950,0migration/33586-21if_eth010:49:493
339950,0migration/33576-21kthreadcore08:44:553
339950,0migration/329491-21munin-node07:19:523
339950,0migration/329376-21kthreadcore07:49:523
339950,0migration/328789-21proc_pri06:54:563
339950,0migration/325628-21pmu-power08:09:553
339950,0migration/324700-21munin-node10:59:583
339950,0migration/321192-21run-parts09:35:333
339950,0migration/319862-21needreboot08:29:533
339950,0migration/318728-21taskset08:54:503
339950,0migration/317225-21taskset07:14:063
339950,0migration/316276-21timerandwakeup11:54:593
339950,0migration/316031-21taskset06:45:533
339950,0migration/315918-21missed_timers11:14:543
339950,0migration/313516-21munin-node07:39:553
339950,0migration/31340-21sh12:07:493
339950,0migration/31319-21kthreadcore09:44:523
339950,0migration/311479-21ksmtuned10:12:243
28415050,0irq/27-eth0-tx-9939-21munin-node06:44:580
28415050,0irq/27-eth0-tx-8256-21ssh08:45:022
28415050,0irq/27-eth0-tx-0-21swapper/212:11:012
28415050,0irq/27-eth0-tx-0-21swapper/212:06:282
28415050,0irq/27-eth0-tx-0-21swapper/211:25:492
28415050,0irq/27-eth0-tx-0-21swapper/210:34:452
28415050,0irq/27-eth0-tx-0-21swapper/210:19:512
28415050,0irq/27-eth0-tx-0-21swapper/209:52:222
28415050,0irq/27-eth0-tx-0-21swapper/209:39:472
28415050,0irq/27-eth0-tx-0-21swapper/209:29:072
28415050,0irq/27-eth0-tx-0-21swapper/209:05:152
28415050,0irq/27-eth0-tx-0-21swapper/209:00:452
28415050,0irq/27-eth0-tx-0-21swapper/008:29:490
28415050,0irq/27-eth0-tx-0-21swapper/008:14:490
28415050,0irq/27-eth0-tx-0-21swapper/007:39:510
28415050,0irq/27-eth0-tx-0-21swapper/007:22:230
28415050,0irq/27-eth0-tx-0-21swapper/006:59:460
28405050,0irq/26-eth0-rx-9475-21grep08:24:513
28405050,0irq/26-eth0-rx-3726-21tune2fs10:29:483
28405050,0irq/26-eth0-rx-29210-21kthreadcore11:04:563
28405050,0irq/26-eth0-rx-26404-21memory08:34:533
28405050,0irq/26-eth0-rx-24227-21kthreadcore09:59:543
28405050,0irq/26-eth0-rx-24010-21ntp_offset11:59:573
28405050,0irq/26-eth0-rx-11776-21fschecks_time10:34:503
28405050,0irq/26-eth0-rx-0-21swapper/310:47:063
28405050,0irq/26-eth0-rx-0-21swapper/310:28:573
28405050,0irq/26-eth0-rx-0-21swapper/310:08:373
28405050,0irq/26-eth0-rx-0-21swapper/309:54:333
28405050,0irq/26-eth0-rx-0-21swapper/309:23:063
28405050,0irq/26-eth0-rx-0-21swapper/309:15:013
259950,0migration/2466-21sh11:45:492
259950,0migration/24652-21munin-node07:29:582
259950,0migration/232068-21missed_timers11:04:572
259950,0migration/21971-21sh10:48:312
259950,0migration/219703-21irqcore11:19:512
249950,0watchdog/219053-21munin-node08:04:562
249950,0watchdog/20-21swapper/210:39:392
249950,0watchdog/20-21swapper/210:28:432
249950,0watchdog/20-21swapper/210:00:112
249950,0watchdog/20-21swapper/209:55:032
249950,0watchdog/20-21swapper/209:46:432
249950,0watchdog/20-21swapper/209:17:192
249950,0watchdog/20-21swapper/208:49:512
249950,0watchdog/20-21swapper/208:28:232
249950,0watchdog/20-21swapper/208:02:522
249950,0watchdog/20-21swapper/207:43:402
249950,0watchdog/20-21swapper/207:03:242
19150,0ktimersoftd/10-21swapper/109:04:061
179950,0migration/15831-21latency_hist09:49:331
179950,0migration/13123-21idleruntime11:09:501
179950,0migration/122999-21kthreadcore09:59:541
179950,0migration/121776-21runrttasks09:37:261
179950,0migration/12077-21latency_hist12:09:321
179950,0migration/120365-21munin-node10:19:491
179950,0migration/118125-21ksmtuned11:39:251
179950,0migration/11510-21kthreadcore07:54:531
179950,0migration/113919-21missed_timers09:09:541
179950,0migration/113181-21missed_timers08:24:531
179950,0migration/11300-21unixbench_multi08:40:011
179950,0migration/111512-21munin-node11:14:521
169950,0watchdog/128307-21kthreadcore11:24:511
149950,0migration/09952-21sh10:53:160
149950,0migration/08072-21munin-node11:29:570
149950,0migration/05038-21timerwakeupswit07:30:030
149950,0migration/03816-21sh09:44:550
149950,0migration/028679-21kthreadcore11:04:560
149950,0migration/028233-21taskset10:02:270
149950,0migration/025215-21latency_hist08:59:330
149950,0migration/024020-21kthreadcore08:34:520
149950,0migration/023968-21sh11:19:540
149950,0migration/023016-21kthreadcore07:44:530
149950,0migration/022412-21kthreadcore10:59:520
149950,0migration/018450-21fschecks_time08:54:490
149950,0migration/018323-21idleruntime07:14:490
149950,0migration/015804-21munin-node11:34:540
149950,0migration/011927-21kthreadcore08:49:510
149950,0migration/011843-21gltestperf10:34:510
149950,0migration/011497-21fschecks_time07:09:470
97259943,0cyclictest6445-21snmpd07:26:272
97259943,0cyclictest28757-21awk08:39:502
97259943,0cyclictest24157-21awk06:54:502
97259943,0cyclictest21075-21awk08:09:502
97249944,0cyclictest0-21swapper/111:45:521
97249943,0cyclictest3093-21awk08:44:531
97249943,0cyclictest27668-21awk08:14:511
97249943,0cyclictest25008-21awk09:19:531
97249943,0cyclictest24251-21kworker/1:112:03:121
97249943,0cyclictest17363-21kworker/1:011:04:541
97249943,0cyclictest17363-21kworker/1:010:27:581
97249943,0cyclictest17363-21kworker/1:009:17:031
97249943,0cyclictest15407-21awk08:29:501
97249943,0cyclictest13158-21awk10:14:491
97249943,0cyclictest11196-21awk11:54:511
97239944,0cyclictest0-21swapper/010:28:330
97239943,0cyclictest7560-21sadc08:24:330
97239943,0cyclictest6445-21snmpd11:42:580
97239943,0cyclictest6445-21snmpd11:09:570
97239943,0cyclictest4925-21sed07:04:510
97239943,0cyclictest3425-21lldpd09:34:250
97239943,0cyclictest3110-21irqbalance11:48:570
97239943,0cyclictest3110-21irqbalance10:58:270
97239943,0cyclictest3110-21irqbalance10:04:470
97239943,0cyclictest3110-21irqbalance09:50:570
97239943,0cyclictest3110-21irqbalance09:04:470
97239943,0cyclictest3110-21irqbalance07:59:370
97239943,0cyclictest3110-21irqbalance07:55:470
97239943,0cyclictest3110-21irqbalance07:50:470
97239943,0cyclictest3110-21irqbalance07:35:470
97239943,0cyclictest3110-21irqbalance07:25:170
97239943,0cyclictest20934-21awk10:19:510
97239943,0cyclictest14637-21awk09:54:490
339940,0migration/33563-21kthreadcore08:19:523
339940,0migration/331098-21taskset07:26:563
339940,0migration/329812-21kthreadcore08:14:543
339940,0migration/324904-21switchtime07:44:583
28415040,0irq/27-eth0-tx-6037-21cut10:29:512
28415040,0irq/27-eth0-tx-0-21swapper/008:39:510
28405040,0irq/26-eth0-rx-0-21swapper/311:45:143
28405040,0irq/26-eth0-rx-0-21swapper/311:20:193
28405040,0irq/26-eth0-rx-0-21swapper/311:09:573
28405040,0irq/26-eth0-rx-0-21swapper/309:39:393
28405040,0irq/26-eth0-rx-0-21swapper/309:29:453
28405040,0irq/26-eth0-rx-0-21swapper/309:29:253
28405040,0irq/26-eth0-rx-0-21swapper/309:09:363
28405040,0irq/26-eth0-rx-0-21swapper/309:04:563
28405040,0irq/26-eth0-rx-0-21swapper/308:50:103
28405040,0irq/26-eth0-rx-0-21swapper/307:14:463
259940,0migration/220399-21irqrtprio07:44:522
259940,0migration/212889-21kthreadcore07:09:512
249940,0watchdog/224844-21grep07:19:482
249940,0watchdog/20-21swapper/208:34:352
249940,0watchdog/20-21swapper/208:20:072
249940,0watchdog/20-21swapper/207:34:482
249940,0watchdog/20-21swapper/207:15:042
249940,0watchdog/20-21swapper/207:04:482
249940,0watchdog/20-21swapper/206:52:282
249940,0watchdog/20-21swapper/206:45:002
179940,0migration/17720-21kthreadcore07:04:541
179940,0migration/17015-21kthreadcore10:49:521
179940,0migration/14079-21kthreadcore09:24:531
179940,0migration/127155-21aten_r1power_fr10:44:481
179940,0migration/124567-21timerwakeupswit10:59:571
179940,0migration/123052-21kthreadcore08:34:521
179940,0migration/122890-21kthreadcore11:19:531
179940,0migration/120254-21kthreadcore11:39:541
179940,0migration/110087-21kthreadcore07:59:531
169940,0watchdog/19014-21ssh11:31:311
169940,0watchdog/18035-21python11:49:551
169940,0watchdog/15645-21kthreadcore10:29:511
169940,0watchdog/1463-21kthreadcore09:44:511
169940,0watchdog/124843-21tune2fs07:19:481
169940,0watchdog/124454-21ssh08:56:471
169940,0watchdog/122923-21kthreadcore10:39:511
169940,0watchdog/121447-21kthreadcore08:09:521
169940,0watchdog/120620-21kthreadcore07:14:521
169940,0watchdog/115220-21cut08:04:521
169940,0watchdog/111239-21kthreadcore08:49:511
169940,0watchdog/10-21swapper/110:09:511
169940,0watchdog/10-21swapper/109:04:591
169940,0watchdog/10-21swapper/107:49:481
169940,0watchdog/10-21swapper/106:45:001
149940,0migration/0544-21timerwakeupswit10:44:590
149940,0migration/027958-21kthreadcore09:39:540
149940,0migration/020002-21kthreadcore09:34:530
149940,0migration/019742-21kthreadcore06:49:500
149940,0migration/016762-21runrttasks11:15:200
97249933,0cyclictest17494-21awk06:49:481
97249933,0cyclictest17363-21kworker/1:009:54:371
97249933,0cyclictest17363-21kworker/1:009:39:391
97249933,0cyclictest17363-21kworker/1:007:44:491
97249933,0cyclictest17363-21kworker/1:007:39:351
97249933,0cyclictest17363-21kworker/1:007:34:351
97249933,0cyclictest17363-21kworker/1:007:29:351
97249933,0cyclictest17363-21kworker/1:007:24:431
97249932,0cyclictest2027-21awk08:19:511
97239932,0cyclictest6547-21runrttasks12:01:460
97239932,0cyclictest6445-21snmpd11:57:280
97239932,0cyclictest3110-21irqbalance09:09:470
97239932,0cyclictest3110-21irqbalance06:54:370
97239932,0cyclictest17340-21awk09:14:590
97239930,0cyclictest27029-21diskstats11:24:470
97239930,0cyclictest2674-21df_inode12:09:480
97239930,0cyclictest0-21swapper/008:24:480
28405030,0irq/26-eth0-rx-0-21swapper/308:40:233
28405030,0irq/26-eth0-rx-0-21swapper/308:04:473
28405030,0irq/26-eth0-rx-0-21swapper/307:59:463
28405030,0irq/26-eth0-rx-0-21swapper/306:59:453
28405030,0irq/26-eth0-rx-0-21swapper/306:49:453
169930,0watchdog/10-21swapper/106:59:481
169930,0watchdog/10-21swapper/106:54:521
149930,0migration/023963-21runrttasks09:19:350
28405020,0irq/26-eth0-rx-0-21swapper/307:54:483
28405020,0irq/26-eth0-rx-0-21swapper/307:34:373
28405020,0irq/26-eth0-rx-0-21swapper/307:30:243
28405020,0irq/26-eth0-rx-0-21swapper/307:04:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional