You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-21 - 16:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Aug 21, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350560,0irq/25-eth10-21swapper/306:40:163
2902724231,0sleep20-21swapper/206:42:572
2884524140,0sleep10-21swapper/106:40:481
2265722622,0sleep00-21swapper/006:38:110
339960,0migration/39998-21munin-node09:33:213
339960,0migration/3636-21munin-node07:58:203
339960,0migration/3612-21sh11:50:223
339960,0migration/36083-21taskset09:48:473
339960,0migration/326887-21kthreadcore10:23:233
339960,0migration/324328-21munin-node11:03:213
339960,0migration/318844-21sh08:58:063
339960,0migration/316340-21sh11:58:453
339960,0migration/31553-21sh09:06:593
339960,0migration/314262-21sh09:55:283
339960,0migration/313936-21sh10:18:103
291099960,0cyclictest27311-21diskmemload11:19:452
291099960,0cyclictest27088-21fschecks_time08:18:182
291099960,0cyclictest0-21swapper/210:04:052
291069960,0cyclictest0-21swapper/011:46:210
291069960,0cyclictest0-21swapper/009:59:280
291069960,0cyclictest0-21swapper/008:23:270
28415060,0irq/27-eth0-tx-17371-21ssh09:16:452
28415060,0irq/27-eth0-tx-0-21swapper/212:05:012
28415060,0irq/27-eth0-tx-0-21swapper/211:49:492
28415060,0irq/27-eth0-tx-0-21swapper/211:39:222
28415060,0irq/27-eth0-tx-0-21swapper/211:33:062
28415060,0irq/27-eth0-tx-0-21swapper/210:43:572
28415060,0irq/27-eth0-tx-0-21swapper/210:36:072
28415060,0irq/27-eth0-tx-0-21swapper/210:36:072
28415060,0irq/27-eth0-tx-0-21swapper/210:13:332
28415060,0irq/27-eth0-tx-0-21swapper/209:54:522
28415060,0irq/27-eth0-tx-0-21swapper/209:49:462
28415060,0irq/27-eth0-tx-0-21swapper/209:41:422
28415060,0irq/27-eth0-tx-0-21swapper/209:11:202
28415060,0irq/27-eth0-tx-0-21swapper/209:06:482
28415060,0irq/27-eth0-tx-0-21swapper/208:48:042
28415060,0irq/27-eth0-tx-0-21swapper/208:46:092
28405060,0irq/26-eth0-rx-24024-21grep07:48:233
28405060,0irq/26-eth0-rx-23775-21gltestperf10:03:193
249960,0watchdog/24366-21grep11:53:212
249960,0watchdog/211000-21kthreadcore08:03:212
249960,0watchdog/21044-21ssh11:31:172
249960,0watchdog/20-21swapper/209:18:212
249960,0watchdog/20-21swapper/208:55:372
8225050,0irq/24-0000:00:25353-21perf07:53:032
8225050,0irq/24-0000:00:10109-21idleruntime-cro06:53:032
4150,0ktimersoftd/00-21swapper/010:38:340
4150,0ktimersoftd/00-21swapper/010:21:130
339950,0migration/37457-21kthreadcore07:38:213
339950,0migration/37155-21kthreadcore10:33:223
339950,0migration/37155-21kthreadcore10:33:223
339950,0migration/36416-21cpuspeed_turbos10:13:163
339950,0migration/35293-21taskset10:11:013
339950,0migration/34724-21kthreadcore08:23:213
339950,0migration/330916-21taskset07:32:103
339950,0migration/329058-21kthreadcore08:43:223
339950,0migration/329006-21kthreadcore11:28:203
339950,0migration/320260-21kthreadcore06:58:223
339950,0migration/318498-21kthreadcore09:38:233
339950,0migration/317421-21kthreadcore09:58:203
339950,0migration/313876-21taskset08:31:593
339950,0migration/312460-21aten_r1power_po10:38:163
339950,0migration/311800-21kthreadcore11:38:163
291099955,0cyclictest0-21swapper/211:25:452
291069955,0cyclictest0-21swapper/011:25:330
291069955,0cyclictest0-21swapper/010:14:070
291069950,0cyclictest0-21swapper/012:10:190
291069950,0cyclictest0-21swapper/009:47:100
28415050,0irq/27-eth0-tx-9833-21ssh11:17:112
28415050,0irq/27-eth0-tx-6372-21iceccd09:34:532
28415050,0irq/27-eth0-tx-3425-21lldpd11:00:462
28415050,0irq/27-eth0-tx-26523-21munin-node08:18:170
28415050,0irq/27-eth0-tx-20875-21munin-node08:38:260
28415050,0irq/27-eth0-tx-20255-21ssh11:03:192
28415050,0irq/27-eth0-tx-0-21swapper/212:08:492
28415050,0irq/27-eth0-tx-0-21swapper/211:58:202
28415050,0irq/27-eth0-tx-0-21swapper/210:54:102
28415050,0irq/27-eth0-tx-0-21swapper/210:38:482
28415050,0irq/27-eth0-tx-0-21swapper/210:18:062
28415050,0irq/27-eth0-tx-0-21swapper/210:00:232
28415050,0irq/27-eth0-tx-0-21swapper/209:44:072
28415050,0irq/27-eth0-tx-0-21swapper/007:22:230
28405050,0irq/26-eth0-rx-9439-21ssh11:36:553
28405050,0irq/26-eth0-rx-3439-21grep11:13:193
28405050,0irq/26-eth0-rx-31078-21taskset09:44:543
28405050,0irq/26-eth0-rx-26905-21munin-node11:08:133
28405050,0irq/26-eth0-rx-18946-21kthreadcore08:33:213
28405050,0irq/26-eth0-rx-0-21swapper/312:03:213
28405050,0irq/26-eth0-rx-0-21swapper/311:57:383
28405050,0irq/26-eth0-rx-0-21swapper/311:45:363
28405050,0irq/26-eth0-rx-0-21swapper/311:25:543
28405050,0irq/26-eth0-rx-0-21swapper/310:31:213
28405050,0irq/26-eth0-rx-0-21swapper/309:31:153
28405050,0irq/26-eth0-rx-0-21swapper/309:23:373
28405050,0irq/26-eth0-rx-0-21swapper/309:09:073
259950,0migration/25388-21sh10:11:192
259950,0migration/232432-21kthreadcore10:48:212
259950,0migration/225003-21unixbench_singl08:58:262
249950,0watchdog/20-21swapper/211:43:252
249950,0watchdog/20-21swapper/211:09:212
249950,0watchdog/20-21swapper/210:28:252
249950,0watchdog/20-21swapper/210:24:252
249950,0watchdog/20-21swapper/209:28:452
249950,0watchdog/20-21swapper/209:25:412
249950,0watchdog/20-21swapper/208:23:332
249950,0watchdog/20-21swapper/208:17:132
249950,0watchdog/20-21swapper/207:58:262
249950,0watchdog/20-21swapper/207:43:462
249950,0watchdog/20-21swapper/207:27:422
179950,0migration/15185-21kthreadcore10:53:201
179950,0migration/13410-21turbostat06:48:031
179950,0migration/132593-21timerandwakeup09:03:271
179950,0migration/128712-21kthreadcore11:48:201
179950,0migration/128544-21sh10:26:041
179950,0migration/126799-21kthreadcore10:03:221
179950,0migration/124962-21unixbench_multi08:58:261
179950,0migration/124372-21unixbench_singl11:23:281
179950,0migration/121181-21kthreadcore09:38:241
179950,0migration/120894-21kthreadcore07:23:211
179950,0migration/118530-21sh11:43:151
179950,0migration/113267-21if_eth010:38:201
179950,0migration/110739-21kthreadcore09:33:231
169950,0watchdog/13024-21grep10:08:211
149950,0migration/09399-21taskset11:36:470
149950,0migration/07836-21kthreadcore06:48:230
149950,0migration/04665-21aten_r1power_po10:33:160
149950,0migration/04665-21aten_r1power_po10:33:160
149950,0migration/04036-21irqcore08:48:200
149950,0migration/03962-21taskset07:08:460
149950,0migration/03465-21aten_r1power_fr10:53:150
149950,0migration/03392-21kthreadcore09:48:210
149950,0migration/030954-21kthreadcore10:28:200
149950,0migration/02849-21fschecks_time09:08:180
149950,0migration/028305-21kthreadcore07:28:210
149950,0migration/026969-21fschecks_time11:28:160
149950,0migration/025219-21kthreadcore10:03:220
149950,0migration/024484-21taskset09:19:280
149950,0migration/023399-21kthreadcore08:58:200
149950,0migration/022763-21kthreadcore11:03:200
149950,0migration/022517-21kthreadcore07:48:230
149950,0migration/021902-21kthreadcore06:58:230
149950,0migration/021496-21kthreadcore08:13:210
149950,0migration/019279-21irqcore12:03:190
149950,0migration/01472-21kthreadcore10:08:210
149950,0migration/014539-21kthreadcore10:58:240
149950,0migration/012758-21fschecks_count07:43:160
149950,0migration/010533-21kthreadcore08:28:270
339940,0migration/35403-21kthreadcore06:48:223
339940,0migration/329185-21kthreadcore07:53:203
339940,0migration/323182-21latency07:23:223
339940,0migration/31936-21diskmemload10:49:243
339940,0migration/314571-21kthreadcore07:43:203
29758243,0sleep36445-21snmpd07:04:403
291099943,0cyclictest3110-21irqbalance07:53:172
291099943,0cyclictest3110-21irqbalance07:28:372
291099943,0cyclictest3110-21irqbalance06:43:172
291099943,0cyclictest14067-21awk08:08:172
291079943,0cyclictest7179-21awk10:13:191
291079943,0cyclictest647-21awk09:48:181
291079943,0cyclictest3425-21lldpd08:51:291
291079943,0cyclictest3289-21awk11:13:181
291079943,0cyclictest24519-21awk07:03:201
291079943,0cyclictest24143-21kworker/1:111:08:241
291079943,0cyclictest24143-21kworker/1:110:50:481
291079943,0cyclictest24143-21kworker/1:110:18:161
291079943,0cyclictest19789-21awk07:48:201
291079943,0cyclictest11025-21awk11:38:151
291079943,0cyclictest11016-21awk11:58:191
291079942,0cyclictest24143-21kworker/1:111:06:521
291069944,0cyclictest0-21swapper/009:23:590
291069944,0cyclictest0-21swapper/009:06:200
291069943,0cyclictest6537-21awk07:38:200
291069943,0cyclictest6445-21snmpd11:56:110
291069943,0cyclictest6445-21snmpd10:23:550
291069943,0cyclictest3425-21lldpd09:57:490
291069943,0cyclictest3269-21awk11:13:180
291069943,0cyclictest28563-21awk08:43:210
291069943,0cyclictest27159-21awk11:48:180
291069943,0cyclictest18008-21sadc11:23:020
291069943,0cyclictest11100-21awk11:38:150
291069943,0cyclictest10399-21irqstats09:33:230
28415040,0irq/27-eth0-tx-0-21swapper/007:58:170
28415040,0irq/27-eth0-tx-0-21swapper/007:38:000
28415040,0irq/27-eth0-tx-0-21swapper/007:24:550
28415040,0irq/27-eth0-tx-0-21swapper/007:03:200
28405040,0irq/26-eth0-rx-1225-21kthreadcore06:43:213
28405040,0irq/26-eth0-rx-10082-21kthreadcore08:03:213
28405040,0irq/26-eth0-rx-0-21swapper/312:08:153
28405040,0irq/26-eth0-rx-0-21swapper/311:19:123
28405040,0irq/26-eth0-rx-0-21swapper/310:59:593
28405040,0irq/26-eth0-rx-0-21swapper/310:53:593
28405040,0irq/26-eth0-rx-0-21swapper/310:43:143
28405040,0irq/26-eth0-rx-0-21swapper/309:21:473
28405040,0irq/26-eth0-rx-0-21swapper/309:13:193
28405040,0irq/26-eth0-rx-0-21swapper/308:53:363
28405040,0irq/26-eth0-rx-0-21swapper/308:48:233
28405040,0irq/26-eth0-rx-0-21swapper/308:18:213
259940,0migration/25253-21fschecks_time07:13:192
259940,0migration/231464-21cpuspeed_turbos07:33:212
259940,0migration/230252-21latency_hist07:08:032
259940,0migration/213180-21kthreadcore07:18:222
249940,0watchdog/20-21swapper/208:38:052
249940,0watchdog/20-21swapper/208:33:332
249940,0watchdog/20-21swapper/208:28:052
249940,0watchdog/20-21swapper/207:38:062
249940,0watchdog/20-21swapper/207:03:142
249940,0watchdog/20-21swapper/206:58:262
249940,0watchdog/20-21swapper/206:53:262
179940,0migration/1453-21taskset09:26:301
179940,0migration/13650-21kthreadcore07:58:231
179940,0migration/132543-21kernelversion07:33:251
179940,0migration/130888-21taskset07:53:211
179940,0migration/126178-21taskset08:16:481
179940,0migration/124336-21kthreadcore08:38:211
179940,0migration/123528-21needreboot09:18:231
179940,0migration/12149-21cpuspeed_turbos11:53:161
179940,0migration/119724-21kthreadcore09:58:211
179940,0migration/118038-21timerandwakeup07:43:251
179940,0migration/113507-21timerwakeupswit09:53:261
179940,0migration/112693-21kthreadcore11:18:191
179940,0migration/111248-21aten_r1power_cu10:58:171
169940,0watchdog/16923-21kthreadcore07:38:221
169940,0watchdog/132166-21sensors_fan08:18:251
169940,0watchdog/130611-21df07:08:181
169940,0watchdog/127286-21kthreadcore09:43:211
169940,0watchdog/124371-21ssh12:03:371
169940,0watchdog/118290-21kernelversion06:58:221
169940,0watchdog/116157-21kthreadcore08:53:211
169940,0watchdog/115763-21sh09:13:251
169940,0watchdog/112711-21cut07:18:221
169940,0watchdog/10-21swapper/111:30:451
169940,0watchdog/10-21swapper/110:33:211
169940,0watchdog/10-21swapper/110:33:211
169940,0watchdog/10-21swapper/110:28:171
169940,0watchdog/10-21swapper/109:11:091
169940,0watchdog/10-21swapper/108:03:211
149940,0migration/09560-21kthreadcore07:13:230
291079933,0cyclictest24143-21kworker/1:111:33:021
291079933,0cyclictest24143-21kworker/1:110:43:041
291079933,0cyclictest24143-21kworker/1:109:28:071
291079933,0cyclictest24143-21kworker/1:108:43:291
291079933,0cyclictest24143-21kworker/1:108:23:251
291079933,0cyclictest24143-21kworker/1:108:08:131
291079933,0cyclictest24143-21kworker/1:107:28:171
291079933,0cyclictest24143-21kworker/1:106:48:191
291079932,0cyclictest5458-21awk07:13:201
291079932,0cyclictest14517-21tr08:33:161
291079932,0cyclictest12286-21grep08:28:271
291079932,0cyclictest11246-21awk06:53:181
291069933,0cyclictest0-21swapper/009:30:270
291069932,0cyclictest6445-21snmpd09:38:550
291069932,0cyclictest25578-21munin-node07:53:150
291069932,0cyclictest20653-21perf08:38:030
291069932,0cyclictest16777-21python08:53:260
291069932,0cyclictest12546-21grep09:13:230
291069932,0cyclictest11090-21awk11:58:190
291069930,0cyclictest32573-21munin-node11:08:230
291069930,0cyclictest20521-21diskstats10:43:160
291069930,0cyclictest0-21swapper/010:48:200
291069930,0cyclictest0-21swapper/008:08:150
28415030,0irq/27-eth0-tx-10426-21aten2.4-expect06:53:150
28415030,0irq/27-eth0-tx-0-21swapper/008:03:160
28415030,0irq/27-eth0-tx-0-21swapper/006:43:200
28405030,0irq/26-eth0-rx-0-21swapper/308:40:233
28405030,0irq/26-eth0-rx-0-21swapper/307:34:453
28405030,0irq/26-eth0-rx-0-21swapper/307:13:173
28405030,0irq/26-eth0-rx-0-21swapper/307:08:173
169930,0watchdog/10-21swapper/112:08:171
28405020,0irq/26-eth0-rx-0-21swapper/308:13:033
28405020,0irq/26-eth0-rx-0-21swapper/308:08:133
28405020,0irq/26-eth0-rx-0-21swapper/307:18:283
28405020,0irq/26-eth0-rx-0-21swapper/306:53:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional