You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 00:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Mon Jul 14, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2374925339,0sleep00-21swapper/006:42:200
2389523829,0sleep20-21swapper/206:44:032
2386223727,0sleep30-21swapper/306:43:443
2385023323,0sleep10-21swapper/106:43:331
28415070,0irq/27-eth0-tx-26162-21munin-node07:34:530
239989970,0cyclictest11267-21kthreadcore12:04:490
339960,0migration/323370-21munin-node10:49:543
339960,0migration/322488-21aten_r1power_fr09:29:403
339960,0migration/319825-21munin-node07:29:413
339960,0migration/313269-21sh09:20:523
339960,0migration/311771-21taskset10:02:183
28415060,0irq/27-eth0-tx-5330-21ssh09:39:192
28415060,0irq/27-eth0-tx-25335-21tail07:09:440
28415060,0irq/27-eth0-tx-0-21swapper/211:42:132
28415060,0irq/27-eth0-tx-0-21swapper/211:18:462
28415060,0irq/27-eth0-tx-0-21swapper/211:06:422
28415060,0irq/27-eth0-tx-0-21swapper/210:59:172
28415060,0irq/27-eth0-tx-0-21swapper/210:44:452
28415060,0irq/27-eth0-tx-0-21swapper/210:30:372
28415060,0irq/27-eth0-tx-0-21swapper/210:26:592
28415060,0irq/27-eth0-tx-0-21swapper/210:20:172
28415060,0irq/27-eth0-tx-0-21swapper/210:17:092
28415060,0irq/27-eth0-tx-0-21swapper/209:46:162
28415060,0irq/27-eth0-tx-0-21swapper/209:32:372
28415060,0irq/27-eth0-tx-0-21swapper/209:24:132
28415060,0irq/27-eth0-tx-0-21swapper/209:12:582
28415060,0irq/27-eth0-tx-0-21swapper/209:08:272
28415060,0irq/27-eth0-tx-0-21swapper/208:58:482
28415060,0irq/27-eth0-tx-0-21swapper/006:52:050
28405060,0irq/26-eth0-rx-5307-21pluginstate08:54:503
28405060,0irq/26-eth0-rx-32648-21irqstats09:14:473
28405060,0irq/26-eth0-rx-31849-21ssh11:19:043
28405060,0irq/26-eth0-rx-22433-21sh11:31:403
28405060,0irq/26-eth0-rx-12224-21sh09:41:023
240009960,0cyclictest0-21swapper/207:49:522
249960,0watchdog/229844-21ssh11:56:352
249960,0watchdog/229844-21ssh11:56:352
249960,0watchdog/20-21swapper/210:10:112
249960,0watchdog/20-21swapper/209:55:552
239999965,0cyclictest18344-21kworker/1:010:12:481
239999960,0cyclictest0-21swapper/111:05:061
149960,0migration/025092-21kthreadcore09:29:450
149960,0migration/023713-21taskset08:48:300
149960,0migration/022179-21idleruntime-cro09:29:280
149960,0migration/021130-21missed_timers11:29:490
149960,0migration/011509-21sendmail_mailtr09:39:530
339950,0migration/37532-21df_abs07:44:443
339950,0migration/3554-21switchtime07:59:523
339950,0migration/34732-21taskset06:52:053
339950,0migration/3354-21irqrtprio11:39:453
339950,0migration/33297-21missed_timers09:34:483
339950,0migration/33043-21fschecks_count08:29:433
339950,0migration/329411-21kthreadcore07:09:503
339950,0migration/32808-21fschecks_time10:39:443
339950,0migration/325990-21kthreadcore06:44:443
339950,0migration/325616-21kernelversion08:49:463
339950,0migration/31851-21kthreadcore11:19:473
339950,0migration/317401-21taskset10:49:253
339950,0migration/316323-21hwlatdetect09:04:463
339950,0migration/314728-21aten2.4-expect12:09:403
339950,0migration/313579-21gltestperf07:24:423
339950,0migration/310286-21kthreadcore11:24:473
28415050,0irq/27-eth0-tx-7642-21munin-node08:59:542
28415050,0irq/27-eth0-tx-6372-21iceccd09:50:082
28415050,0irq/27-eth0-tx-27283-21munin-node07:59:420
28415050,0irq/27-eth0-tx-20244-21open_files09:24:502
28415050,0irq/27-eth0-tx-0-21swapper/212:02:232
28415050,0irq/27-eth0-tx-0-21swapper/211:45:342
28415050,0irq/27-eth0-tx-0-21swapper/211:35:322
28415050,0irq/27-eth0-tx-0-21swapper/211:29:332
28415050,0irq/27-eth0-tx-0-21swapper/211:10:082
28415050,0irq/27-eth0-tx-0-21swapper/211:02:352
28415050,0irq/27-eth0-tx-0-21swapper/210:39:432
28415050,0irq/27-eth0-tx-0-21swapper/210:34:492
28415050,0irq/27-eth0-tx-0-21swapper/210:07:242
28415050,0irq/27-eth0-tx-0-21swapper/209:41:472
28415050,0irq/27-eth0-tx-0-21swapper/008:39:430
28415050,0irq/27-eth0-tx-0-21swapper/008:34:430
28415050,0irq/27-eth0-tx-0-21swapper/006:45:290
28405050,0irq/26-eth0-rx-9484-21kthreadcore11:44:493
28405050,0irq/26-eth0-rx-32392-21uname11:59:463
28405050,0irq/26-eth0-rx-31526-21sed10:54:553
28405050,0irq/26-eth0-rx-31340-21kthreadcore09:54:453
28405050,0irq/26-eth0-rx-30276-21kthreadcore07:34:473
28405050,0irq/26-eth0-rx-19534-21timerandwakeup09:44:533
28405050,0irq/26-eth0-rx-19462-21sh10:07:043
28405050,0irq/26-eth0-rx-1296-21df_abs10:59:443
28405050,0irq/26-eth0-rx-12292-21hwlatdetect10:24:443
28405050,0irq/26-eth0-rx-0-21swapper/311:50:493
28405050,0irq/26-eth0-rx-0-21swapper/310:38:433
28405050,0irq/26-eth0-rx-0-21swapper/310:32:093
28405050,0irq/26-eth0-rx-0-21swapper/310:19:493
28405050,0irq/26-eth0-rx-0-21swapper/310:14:373
28405050,0irq/26-eth0-rx-0-21swapper/309:53:103
28405050,0irq/26-eth0-rx-0-21swapper/309:12:153
28405050,0irq/26-eth0-rx-0-21swapper/309:01:163
28405050,0irq/26-eth0-rx-0-21swapper/308:45:233
259950,0migration/24986-21runrttasks09:14:532
259950,0migration/231796-21latency_hist07:14:282
259950,0migration/225400-21taskset07:09:442
259950,0migration/212871-21munin-node06:59:482
240009955,0cyclictest0-21swapper/210:51:192
249950,0watchdog/212808-21grep12:04:552
249950,0watchdog/20-21swapper/211:29:232
249950,0watchdog/20-21swapper/211:21:472
249950,0watchdog/20-21swapper/210:04:032
249950,0watchdog/20-21swapper/208:50:512
249950,0watchdog/20-21swapper/208:07:072
249950,0watchdog/20-21swapper/207:48:032
249950,0watchdog/20-21swapper/207:41:072
249950,0watchdog/20-21swapper/207:06:272
239999955,0cyclictest19870-21kworker/1:112:13:541
239999955,0cyclictest19870-21kworker/1:111:50:121
239999955,0cyclictest19870-21kworker/1:111:45:341
239999955,0cyclictest19870-21kworker/1:111:39:441
239999955,0cyclictest19870-21kworker/1:111:29:501
239999955,0cyclictest19870-21kworker/1:111:24:501
239999955,0cyclictest19870-21kworker/1:111:21:341
239999955,0cyclictest19870-21kworker/1:111:03:501
239999955,0cyclictest19870-21kworker/1:110:56:281
239999955,0cyclictest19870-21kworker/1:110:48:421
239999955,0cyclictest19870-21kworker/1:110:41:541
239999955,0cyclictest19870-21kworker/1:110:34:441
239999955,0cyclictest19870-21kworker/1:110:30:541
239999955,0cyclictest18344-21kworker/1:010:20:281
239999955,0cyclictest18344-21kworker/1:010:15:401
239999955,0cyclictest18344-21kworker/1:010:04:301
239999955,0cyclictest18344-21kworker/1:010:03:081
239999955,0cyclictest18344-21kworker/1:009:55:261
239999955,0cyclictest18344-21kworker/1:009:47:501
239999955,0cyclictest18344-21kworker/1:009:41:081
239999955,0cyclictest18344-21kworker/1:009:37:561
239999955,0cyclictest18344-21kworker/1:009:33:361
239999955,0cyclictest18344-21kworker/1:009:19:461
239999955,0cyclictest18344-21kworker/1:009:14:061
239999955,0cyclictest18344-21kworker/1:008:59:521
239999955,0cyclictest18344-21kworker/1:008:55:141
239999955,0cyclictest18344-21kworker/1:008:45:561
239999955,0cyclictest18344-21kworker/1:008:42:101
239999955,0cyclictest18344-21kworker/1:008:27:481
239999955,0cyclictest18344-21kworker/1:008:22:541
239999955,0cyclictest18344-21kworker/1:008:15:041
239999955,0cyclictest18344-21kworker/1:008:09:461
239999955,0cyclictest18344-21kworker/1:008:05:441
239999955,0cyclictest18344-21kworker/1:007:55:121
239999955,0cyclictest18344-21kworker/1:007:53:481
239999955,0cyclictest18344-21kworker/1:007:48:481
239999955,0cyclictest18344-21kworker/1:007:41:561
239999955,0cyclictest18344-21kworker/1:007:24:301
239999955,0cyclictest18344-21kworker/1:007:24:161
239999955,0cyclictest18344-21kworker/1:007:11:161
239999955,0cyclictest18344-21kworker/1:007:09:041
239999955,0cyclictest18344-21kworker/1:007:01:121
239999955,0cyclictest18344-21kworker/1:006:55:041
239999955,0cyclictest18344-21kworker/1:006:49:301
239999955,0cyclictest18344-21kworker/1:006:46:221
239999955,0cyclictest0-21swapper/109:04:581
239999954,0cyclictest18344-21kworker/1:010:24:401
239999954,0cyclictest18344-21kworker/1:009:24:521
239989950,0cyclictest0-21swapper/011:01:070
179950,0migration/13102-21kthreadcore09:14:481
179950,0migration/126276-21kthreadcore08:49:461
179950,0migration/111447-21kthreadcore08:34:471
169950,0watchdog/124812-21kthreadcore09:49:471
169950,0watchdog/112809-21grep12:04:551
149950,0migration/09485-21kthreadcore11:24:460
149950,0migration/08246-21kthreadcore09:19:450
149950,0migration/07145-21kthreadcore09:59:490
149950,0migration/0451-21idleruntime08:54:440
149950,0migration/030813-21sh11:37:520
149950,0migration/03028-21kthreadcore11:39:460
149950,0migration/030003-21pmu-power11:14:510
149950,0migration/028935-21kthreadcore10:34:460
149950,0migration/027506-21kthreadcore10:54:490
149950,0migration/027048-21munin-node09:49:500
149950,0migration/026803-21taskset07:56:430
149950,0migration/02574-21irqcore08:04:460
149950,0migration/022379-21kthreadcore10:49:510
149950,0migration/02116-21kthreadcore09:14:480
149950,0migration/01894-21taskset08:25:050
149950,0migration/018878-21taskset07:24:560
149950,0migration/016936-21kthreadcore10:04:520
149950,0migration/015084-21aten_r1power_po08:14:430
149950,0migration/014352-21taskset09:01:360
149950,0migration/011693-21kthreadcore08:09:500
339940,0migration/327061-21kthreadcore11:34:473
339940,0migration/325780-21kthreadcore07:54:573
339940,0migration/32327-21kthreadcore07:14:463
339940,0migration/322604-21kthreadcore10:09:503
339940,0migration/321257-21diskmemload09:26:163
339940,0migration/320548-21kthreadcore08:39:483
339940,0migration/316513-21diskmemload11:09:343
339940,0migration/310929-21kthreadcore08:09:503
28415040,0irq/27-eth0-tx-14665-21munin-node12:09:502
28415040,0irq/27-eth0-tx-0-21swapper/006:54:550
28405040,0irq/26-eth0-rx-18391-21kthreadcore08:14:493
28405040,0irq/26-eth0-rx-0-21swapper/312:06:013
28405040,0irq/26-eth0-rx-0-21swapper/311:54:473
28405040,0irq/26-eth0-rx-0-21swapper/311:54:473
28405040,0irq/26-eth0-rx-0-21swapper/311:05:353
259940,0migration/28051-21munin-node08:29:492
259940,0migration/26129-21latency_hist07:19:282
259940,0migration/226388-21kthreadcore08:19:492
240009943,0cyclictest6445-21snmpd08:14:362
240009943,0cyclictest6445-21snmpd08:11:212
240009943,0cyclictest6445-21snmpd06:46:212
240009943,0cyclictest3425-21lldpd08:46:382
240009943,0cyclictest16176-21awk11:49:482
240009940,0cyclictest31450-21perl06:49:462
249940,0watchdog/230884-21kthreadcore08:24:472
249940,0watchdog/20-21swapper/208:39:432
249940,0watchdog/20-21swapper/208:00:352
249940,0watchdog/20-21swapper/207:54:432
249940,0watchdog/20-21swapper/207:30:072
249940,0watchdog/20-21swapper/206:55:032
239999944,0cyclictest19870-21kworker/1:111:34:481
239999944,0cyclictest19870-21kworker/1:110:49:481
239999944,0cyclictest18344-21kworker/1:007:35:181
239999944,0cyclictest18344-21kworker/1:007:29:301
239999944,0cyclictest18344-21kworker/1:007:15:461
239999943,0cyclictest18344-21kworker/1:007:59:441
239989944,0cyclictest0-21swapper/012:03:240
239989943,0cyclictest9645-21awk11:04:440
239989943,0cyclictest6445-21snmpd07:45:210
239989943,0cyclictest6445-21snmpd07:14:510
239989943,0cyclictest19981-21awk10:29:460
239989943,0cyclictest10803-21awk10:44:440
239989940,0cyclictest18865-21df_abs07:04:410
239989940,0cyclictest11866-21munin-node06:59:450
179940,0migration/16754-21kthreadcore08:29:471
179940,0migration/124656-21kthreadcore11:54:461
179940,0migration/124656-21kthreadcore11:54:461
179940,0migration/120475-21kthreadcore11:09:501
169940,0watchdog/13420-21tr11:59:471
169940,0watchdog/125659-21kthreadcore11:14:471
149940,0migration/027874-21kthreadcore11:54:470
149940,0migration/027874-21kthreadcore11:54:470
149940,0migration/026600-21unixbench_multi10:09:560
149940,0migration/023728-21kthreadcore07:29:460
149940,0migration/020250-21kthreadcore11:09:500
149940,0migration/019726-21kthreadcore12:09:470
149940,0migration/015640-21kthreadcore07:49:510
28415030,0irq/27-eth0-tx-2641-21aten2.4-expect08:29:410
28405030,0irq/26-eth0-rx-0-21swapper/308:24:423
28405030,0irq/26-eth0-rx-0-21swapper/308:04:413
28405030,0irq/26-eth0-rx-0-21swapper/307:49:453
28405030,0irq/26-eth0-rx-0-21swapper/307:22:013
28405030,0irq/26-eth0-rx-0-21swapper/307:04:403
28405030,0irq/26-eth0-rx-0-21swapper/306:59:473
28405030,0irq/26-eth0-rx-0-21swapper/306:54:543
240009932,0cyclictest10118-21awk08:34:452
240009930,0cyclictest26162-21munin-node07:34:402
249930,0watchdog/20-21swapper/207:26:072
239989932,0cyclictest9450-21kthreadcore11:44:490
239989932,0cyclictest668-21awk11:19:450
239989932,0cyclictest6547-21runrttasks08:19:340
239989932,0cyclictest6192-21sensors07:39:520
239989932,0cyclictest3203-21awk10:39:450
239989932,0cyclictest30173-21awk09:54:430
239989932,0cyclictest24354-21awk09:09:480
239989932,0cyclictest16525-21awk09:04:480
239989930,0cyclictest6180-21munin-node07:19:440
239989930,0cyclictest21715-21diskmemload11:51:010
239989930,0cyclictest14230-21perl09:44:430
239989930,0cyclictest0-21swapper/010:19:470
149930,0migration/02142-21sh10:15:110
4120,0ktimersoftd/024248-21latency_hist08:49:280
28405020,0irq/26-eth0-rx-0-21swapper/308:34:473
28405020,0irq/26-eth0-rx-0-21swapper/308:19:313
28405020,0irq/26-eth0-rx-0-21swapper/307:39:303
239989921,0cyclictest30000-21latency_hist09:34:280
239989920,0cyclictest0-21swapper/010:24:420
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional