You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-26 - 17:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Sun Oct 26, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2626826463,0sleep00-21swapper/006:37:470
2642723828,0sleep30-21swapper/306:39:383
2646323725,0sleep20-21swapper/206:40:012
2626423223,0sleep10-21swapper/106:37:441
26582991414,0cyclictest31718-21sensors10:45:581
265829988,0cyclictest6445-21snmpd10:53:231
249970,0watchdog/220981-21perf09:45:302
35160,0ktimersoftd/30-21swapper/311:12:393
339960,0migration/39633-21sh10:35:113
339960,0migration/37616-21sh11:53:233
339960,0migration/328890-21munin-node07:51:023
339960,0migration/325549-21sh10:23:393
339960,0migration/323842-21sh11:43:363
339960,0migration/316528-21sh10:58:323
339960,0migration/316041-21sh11:18:303
339960,0migration/311424-21sh09:53:593
28415060,0irq/27-eth0-tx-26493-21munin-node10:25:512
28415060,0irq/27-eth0-tx-0-21swapper/212:00:422
28415060,0irq/27-eth0-tx-0-21swapper/211:59:352
28415060,0irq/27-eth0-tx-0-21swapper/211:43:592
28415060,0irq/27-eth0-tx-0-21swapper/211:37:162
28415060,0irq/27-eth0-tx-0-21swapper/211:35:082
28415060,0irq/27-eth0-tx-0-21swapper/210:56:072
28415060,0irq/27-eth0-tx-0-21swapper/210:50:512
28415060,0irq/27-eth0-tx-0-21swapper/210:45:462
28415060,0irq/27-eth0-tx-0-21swapper/210:35:042
28415060,0irq/27-eth0-tx-0-21swapper/210:18:592
28415060,0irq/27-eth0-tx-0-21swapper/210:07:092
28415060,0irq/27-eth0-tx-0-21swapper/210:04:092
28415060,0irq/27-eth0-tx-0-21swapper/209:16:192
28415060,0irq/27-eth0-tx-0-21swapper/209:10:022
28415060,0irq/27-eth0-tx-0-21swapper/208:55:512
28415060,0irq/27-eth0-tx-0-21swapper/208:46:402
28405060,0irq/26-eth0-rx-16342-21rt-features10:16:003
28405060,0irq/26-eth0-rx-14559-21switchtime11:55:593
28405060,0irq/26-eth0-rx-0-21swapper/306:45:573
265839960,0cyclictest0-21swapper/210:43:262
265829966,0cyclictest0-21swapper/109:46:551
265829960,0cyclictest0-21swapper/111:24:211
265829960,0cyclictest0-21swapper/109:34:281
249960,0watchdog/20-21swapper/211:18:342
249960,0watchdog/20-21swapper/211:01:382
249960,0watchdog/20-21swapper/209:30:582
149960,0migration/012204-21munin-node11:15:540
8225050,0irq/24-0000:00:28427-21latency_hist07:30:362
35150,0ktimersoftd/30-21swapper/310:06:363
339950,0migration/38277-21kthreadcore09:30:553
339950,0migration/37486-21ksmtuned10:10:563
339950,0migration/35973-21kthreadcore07:10:543
339950,0migration/35599-21kthreadcore08:45:553
339950,0migration/34857-21latency_hist08:25:363
339950,0migration/33565-21fschecks_time07:35:493
339950,0migration/331805-21kthreadcore06:40:553
339950,0migration/331464-21munin-node09:25:533
339950,0migration/328585-21kthreadcore10:25:553
339950,0migration/328201-21kthreadcore08:40:543
339950,0migration/328052-21kthreadcore12:05:543
339950,0migration/326697-21munin-node09:40:563
339950,0migration/325431-21fschecks_time11:45:493
339950,0migration/324046-21sh10:40:593
339950,0migration/323283-21munin-node11:00:583
339950,0migration/320555-21munin-node09:15:583
339950,0migration/317408-21kthreadcore09:55:553
339950,0migration/316568-21munin-node08:06:033
339950,0migration/316318-21taskset07:45:463
339950,0migration/315900-21fschecks_time07:20:493
339950,0migration/31513-21latency07:30:553
339950,0migration/314415-21taskset06:55:433
339950,0migration/314090-21missed_timers11:35:543
339950,0migration/313200-21taskset06:50:563
339950,0migration/311310-21kthreadcore07:15:563
28415050,0irq/27-eth0-tx-18828-21sed09:35:542
28415050,0irq/27-eth0-tx-14421-21munin-node06:55:540
28415050,0irq/27-eth0-tx-0-21swapper/212:06:232
28415050,0irq/27-eth0-tx-0-21swapper/211:11:172
28415050,0irq/27-eth0-tx-0-21swapper/209:25:582
28415050,0irq/27-eth0-tx-0-21swapper/209:20:522
28415050,0irq/27-eth0-tx-0-21swapper/007:15:500
28415050,0irq/27-eth0-tx-0-21swapper/007:10:480
28415050,0irq/27-eth0-tx-0-21swapper/006:45:490
28405050,0irq/26-eth0-rx-32466-21gltestperf09:05:523
28405050,0irq/26-eth0-rx-29001-21ssh09:21:273
28405050,0irq/26-eth0-rx-28060-21kthreadcore11:05:563
28405050,0irq/26-eth0-rx-26050-21munin-node10:45:483
28405050,0irq/26-eth0-rx-2516-21iostat10:50:523
28405050,0irq/26-eth0-rx-1984-21netstat09:45:553
28405050,0irq/26-eth0-rx-19797-21kthreadcore08:55:563
28405050,0irq/26-eth0-rx-16583-21latency_hist12:00:353
28405050,0irq/26-eth0-rx-15869-21python10:35:593
28405050,0irq/26-eth0-rx-0-21swapper/311:30:473
28405050,0irq/26-eth0-rx-0-21swapper/311:25:503
28405050,0irq/26-eth0-rx-0-21swapper/311:20:543
28405050,0irq/26-eth0-rx-0-21swapper/310:00:373
28405050,0irq/26-eth0-rx-0-21swapper/309:14:213
28405050,0irq/26-eth0-rx-0-21swapper/309:01:153
265839955,0cyclictest24716-21diskmemload11:23:482
265839955,0cyclictest24716-21diskmemload11:05:592
265839955,0cyclictest0-21swapper/209:52:212
265839955,0cyclictest0-21swapper/209:00:572
265839955,0cyclictest0-21swapper/208:41:432
265839955,0cyclictest0-21swapper/208:25:382
265839954,0cyclictest19459-21awk10:20:532
265829955,0cyclictest0-21swapper/112:09:081
265829955,0cyclictest0-21swapper/109:51:581
265829950,0cyclictest0-21swapper/111:03:511
265829950,0cyclictest0-21swapper/108:49:411
265829950,0cyclictest0-21swapper/106:45:561
265819955,0cyclictest0-21swapper/009:51:020
265819954,0cyclictest0-21swapper/012:06:560
265819950,0cyclictest0-21swapper/012:00:380
265819950,0cyclictest0-21swapper/009:37:190
259950,0migration/22989-21kthreadcore06:45:542
259950,0migration/216812-21kthreadcore09:55:552
259950,0migration/216019-21fschecks_time07:20:502
259950,0migration/214399-21switchtime07:16:022
249950,0watchdog/2284150irq/27-eth0-tx-09:47:342
249950,0watchdog/20-21swapper/211:54:262
249950,0watchdog/20-21swapper/211:50:022
249950,0watchdog/20-21swapper/210:35:462
249950,0watchdog/20-21swapper/210:12:182
249950,0watchdog/20-21swapper/209:13:382
249950,0watchdog/20-21swapper/208:52:342
249950,0watchdog/20-21swapper/208:09:582
249950,0watchdog/20-21swapper/208:00:422
249950,0watchdog/20-21swapper/207:55:382
249950,0watchdog/20-21swapper/207:53:022
249950,0watchdog/20-21swapper/207:40:382
249950,0watchdog/20-21swapper/206:40:462
19150,0ktimersoftd/10-21swapper/111:58:351
19150,0ktimersoftd/10-21swapper/110:19:051
19150,0ktimersoftd/10-21swapper/109:27:031
179950,0migration/132201-21rt-features10:25:591
179950,0migration/129968-21kthreadcore07:05:531
179950,0migration/12786-21aten_r1power_en10:10:491
179950,0migration/117835-21latency_hist10:40:351
179950,0migration/115923-21taskset10:56:441
149950,0migration/09903-21latency_hist10:35:350
149950,0migration/06571-21sh11:51:000
149950,0migration/05920-21latency11:30:530
149950,0migration/032449-21kthreadcore09:25:540
149950,0migration/030619-21sh11:45:580
149950,0migration/029198-21kthreadcore09:00:590
149950,0migration/027412-21munin-node09:41:010
149950,0migration/022861-21sh08:57:310
149950,0migration/021278-21kthreadcore11:20:540
149950,0migration/016717-21munin-node07:45:490
149950,0migration/015245-21kthreadcore09:55:550
149950,0migration/014130-21needreboot11:35:540
149950,0migration/011737-21irqcore10:15:550
339940,0migration/327981-21unixbench_singl07:26:003
339940,0migration/32365-21kthreadcore07:55:583
339940,0migration/319648-21taskset09:36:093
28415040,0irq/27-eth0-tx-0-21swapper/211:25:532
28415040,0irq/27-eth0-tx-0-21swapper/008:20:490
28415040,0irq/27-eth0-tx-0-21swapper/007:55:540
28405040,0irq/26-eth0-rx-17876-21taskset08:33:433
28405040,0irq/26-eth0-rx-0-21swapper/308:50:473
265839943,0cyclictest6445-21snmpd08:35:412
265839943,0cyclictest6445-21snmpd08:22:562
265839943,0cyclictest2809-21awk07:10:512
265839943,0cyclictest24911-21awk08:15:542
265839943,0cyclictest21956-21awk07:00:542
265829943,0cyclictest6445-21snmpd09:18:561
265829943,0cyclictest6445-21snmpd09:15:261
265829943,0cyclictest6445-21snmpd09:07:261
265829943,0cyclictest6445-21snmpd08:30:561
265829943,0cyclictest6445-21snmpd08:21:111
265829943,0cyclictest6445-21snmpd08:01:251
265829943,0cyclictest6445-21snmpd07:57:551
265829943,0cyclictest6445-21snmpd07:53:401
265829943,0cyclictest6445-21snmpd07:38:401
265829943,0cyclictest6445-21snmpd07:01:401
265829943,0cyclictest2882-21awk10:30:511
265829943,0cyclictest27752-21awk06:40:531
265829943,0cyclictest26474-21awk11:05:541
265829943,0cyclictest24859-21awk09:00:571
265829943,0cyclictest23603-21awk09:20:541
265829943,0cyclictest22154-21awk09:40:531
265829943,0cyclictest1928-21awk11:10:531
265829943,0cyclictest17293-21ps08:55:541
265829943,0cyclictest14415-21awk09:35:511
265829943,0cyclictest1381-21awk11:30:501
265829943,0cyclictest1120-21kworker/1:110:40:411
265829943,0cyclictest1120-21kworker/1:110:05:471
265829943,0cyclictest1120-21kworker/1:110:01:031
265819944,0cyclictest0-21swapper/010:57:340
265819943,0cyclictest6445-21snmpd10:12:410
265819943,0cyclictest3812-21awk07:35:510
265819943,0cyclictest3425-21lldpd11:25:590
265819943,0cyclictest3425-21lldpd08:04:410
265819943,0cyclictest3425-21lldpd07:04:570
265819943,0cyclictest3110-21irqbalance10:02:300
265819943,0cyclictest3110-21irqbalance09:12:200
265819943,0cyclictest27399-21awk10:25:530
265819943,0cyclictest23635-21awk09:20:540
265819943,0cyclictest19426-21awk10:20:530
265819943,0cyclictest18343-21awk11:00:510
259940,0migration/211775-21kthreadcore06:50:542
259940,0migration/211744-21aten_r1power_vo08:30:482
249940,0watchdog/22933-21munin-node07:35:542
249940,0watchdog/20-21swapper/208:10:542
249940,0watchdog/20-21swapper/207:45:502
249940,0watchdog/20-21swapper/207:30:582
249940,0watchdog/20-21swapper/207:05:542
249940,0watchdog/20-21swapper/206:55:582
179940,0migration/19300-21fschecks_time07:15:521
179940,0migration/15043-21aten_r1power_vo08:25:491
179940,0migration/126811-21kthreadcore11:25:531
179940,0migration/119525-21kthreadcore08:10:521
179940,0migration/118055-21kthreadcore06:55:541
179940,0migration/116928-21kthreadcore07:20:531
179940,0migration/115990-21kthreadcore09:55:551
179940,0migration/114609-21ntp_kernel_err11:15:561
179940,0migration/11444-21munin-node11:50:521
169940,0watchdog/16964-21kthreadcore07:10:541
169940,0watchdog/131887-21ssh11:48:341
169940,0watchdog/131866-21cut07:30:541
169940,0watchdog/126591-21kthreadcore07:25:541
169940,0watchdog/1247752sleep110:21:181
169940,0watchdog/121250-21cut07:45:541
169940,0watchdog/10-21swapper/108:40:541
149940,0migration/09398-21irqcore08:50:520
149940,0migration/0677-21kthreadcore07:30:550
149940,0migration/032386-21kthreadcore09:45:540
149940,0migration/03186-21kthreadcore11:10:540
149940,0migration/029284-21kthreadcore10:05:570
149940,0migration/029202-21kthreadcore08:40:550
149940,0migration/029138-21kthreadcore06:40:540
149940,0migration/025748-21kthreadcore07:25:540
149940,0migration/025616-21sh10:45:240
149940,0migration/018650-21kthreadcore11:40:520
149940,0migration/015911-21kthreadcore08:30:530
149940,0migration/012694-21kthreadcore10:35:540
149940,0migration/012680-21runrttasks09:33:350
149940,0migration/011682-21kthreadcore11:55:530
28405030,0irq/26-eth0-rx-0-21swapper/308:37:073
28405030,0irq/26-eth0-rx-0-21swapper/308:22:013
28405030,0irq/26-eth0-rx-0-21swapper/308:15:563
28405030,0irq/26-eth0-rx-0-21swapper/308:10:463
28405030,0irq/26-eth0-rx-0-21swapper/308:01:033
28405030,0irq/26-eth0-rx-0-21swapper/307:05:543
28405030,0irq/26-eth0-rx-0-21swapper/307:01:543
265829933,0cyclictest23031-21kworker/1:008:35:351
265829933,0cyclictest23031-21kworker/1:008:05:511
265829933,0cyclictest1120-21kworker/1:112:00:391
265829933,0cyclictest1120-21kworker/1:111:40:391
265829933,0cyclictest1120-21kworker/1:111:35:431
265829933,0cyclictest1120-21kworker/1:108:50:351
265829932,0cyclictest6445-21snmpd08:15:411
265819933,0cyclictest3425-21lldpd07:05:580
265819932,0cyclictest8747-21awk06:50:520
265819932,0cyclictest5807-21awk08:25:520
265819932,0cyclictest3110-21irqbalance10:45:400
265819932,0cyclictest25463-21ssh11:05:430
265819932,0cyclictest18016-21gltestperf08:10:490
265819932,0cyclictest15856-21awk09:15:520
265819932,0cyclictest10479-21awk07:40:510
265819931,0cyclictest0-21swapper/009:05:560
265819930,0cyclictest23930-21munin-node08:16:010
265819930,0cyclictest22961-21munin-node07:51:010
265819930,0cyclictest16029-21perl07:20:500
265819930,0cyclictest11241-21perl08:05:520
265819930,0cyclictest0-21swapper/008:40:350
169930,0watchdog/115791-21runrttasks07:42:101
169930,0watchdog/10-21swapper/106:50:541
329920,0watchdog/30-21swapper/307:40:463
265819921,0cyclictest0-21swapper/010:50:480
265819920,0cyclictest0-21swapper/008:45:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional