You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-02 - 16:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Sun Nov 02, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2566924131,0sleep20-21swapper/206:40:142
291350350,0irq/25-eth10-21swapper/306:36:053
2558123421,0sleep00-21swapper/006:39:120
2550223425,0sleep10-21swapper/106:38:171
339970,0migration/324102-21unixbench-2d10:40:463
257489976,0cyclictest0-21swapper/108:50:381
257479976,0cyclictest7467-21dump-pmu-power08:50:200
339960,0migration/38084-21sh10:52:313
339960,0migration/36148-21munin-node09:50:373
339960,0migration/34611-21kernelversion08:00:383
339960,0migration/330352-21sh11:48:483
339960,0migration/329514-21sh09:45:193
339960,0migration/329206-21munin-node09:00:413
28415060,0irq/27-eth0-tx-4380-21munin-node08:25:420
28415060,0irq/27-eth0-tx-3297-21munin-node08:00:440
28415060,0irq/27-eth0-tx-25257-21ssh10:44:082
28415060,0irq/27-eth0-tx-0-21swapper/212:08:272
28415060,0irq/27-eth0-tx-0-21swapper/212:04:192
28415060,0irq/27-eth0-tx-0-21swapper/211:58:022
28415060,0irq/27-eth0-tx-0-21swapper/211:53:372
28415060,0irq/27-eth0-tx-0-21swapper/211:50:092
28415060,0irq/27-eth0-tx-0-21swapper/211:39:572
28415060,0irq/27-eth0-tx-0-21swapper/211:18:582
28415060,0irq/27-eth0-tx-0-21swapper/210:28:262
28415060,0irq/27-eth0-tx-0-21swapper/210:06:522
28415060,0irq/27-eth0-tx-0-21swapper/210:03:032
28415060,0irq/27-eth0-tx-0-21swapper/209:57:122
28415060,0irq/27-eth0-tx-0-21swapper/209:50:262
28415060,0irq/27-eth0-tx-0-21swapper/209:49:312
28415060,0irq/27-eth0-tx-0-21swapper/209:12:222
28415060,0irq/27-eth0-tx-0-21swapper/209:05:372
28415060,0irq/27-eth0-tx-0-21swapper/209:00:322
28415060,0irq/27-eth0-tx-0-21swapper/208:59:112
28415060,0irq/27-eth0-tx-0-21swapper/008:20:320
28415060,0irq/27-eth0-tx-0-21swapper/007:09:280
28415060,0irq/27-eth0-tx-0-21swapper/006:57:170
28405060,0irq/26-eth0-rx-6445-21snmpd09:57:423
28405060,0irq/26-eth0-rx-15867-21idleruntime11:40:353
28405060,0irq/26-eth0-rx-10549-21fschecks_time08:05:343
257499960,0cyclictest0-21swapper/211:21:042
257489965,0cyclictest0-21swapper/111:57:461
257479965,0cyclictest3-21ksoftirqd/011:22:550
257479960,0cyclictest0-21swapper/008:12:370
249960,0watchdog/229554-21grep11:05:382
249960,0watchdog/221077-21swap09:15:462
249960,0watchdog/20-21swapper/210:20:382
249960,0watchdog/20-21swapper/209:40:022
149960,0migration/09965-21latency_hist08:05:200
149960,0migration/09069-21sh10:32:090
149960,0migration/06823-21latency_hist11:55:200
6691250,0sleep30-21swapper/306:50:203
339950,0migration/39653-21kthreadcore08:50:383
339950,0migration/35978-21sh11:32:183
339950,0migration/35460-21runrttasks09:28:093
339950,0migration/3404-21kthreadcore08:20:363
339950,0migration/33940-21taskset09:46:513
339950,0migration/32529-21kthreadcore07:10:393
339950,0migration/324854-21munin-node08:40:343
339950,0migration/321178-21munin-node08:55:413
339950,0migration/320773-21taskset07:23:153
339950,0migration/320660-21aten_r1power_en10:00:343
339950,0migration/316689-21aten2.4-expect08:10:313
339950,0migration/316350-21munin-node11:20:333
339950,0migration/31620-21taskset07:32:043
339950,0migration/314596-21kthreadcore10:35:373
339950,0migration/313319-21munin-node07:15:383
339950,0migration/311546-21taskset10:15:313
339950,0migration/311530-21fschecks_count08:30:333
28415050,0irq/27-eth0-tx-23897-21munin-node11:25:362
28415050,0irq/27-eth0-tx-18101-21chrt10:17:162
28415050,0irq/27-eth0-tx-12043-21cut10:55:402
28415050,0irq/27-eth0-tx-10193-21ssh10:35:272
28415050,0irq/27-eth0-tx-0-21swapper/210:52:342
28415050,0irq/27-eth0-tx-0-21swapper/210:47:232
28415050,0irq/27-eth0-tx-0-21swapper/210:11:412
28415050,0irq/27-eth0-tx-0-21swapper/209:40:272
28415050,0irq/27-eth0-tx-0-21swapper/209:30:202
28415050,0irq/27-eth0-tx-0-21swapper/209:25:352
28415050,0irq/27-eth0-tx-0-21swapper/209:24:152
28415050,0irq/27-eth0-tx-0-21swapper/208:51:402
28415050,0irq/27-eth0-tx-0-21swapper/208:45:342
28415050,0irq/27-eth0-tx-0-21swapper/008:40:270
28415050,0irq/27-eth0-tx-0-21swapper/007:50:520
28405050,0irq/26-eth0-rx-8193-21munin-node11:15:383
28405050,0irq/26-eth0-rx-7943-21cut10:10:393
28405050,0irq/26-eth0-rx-7645-21perl11:35:333
28405050,0irq/26-eth0-rx-5562-21python08:45:443
28405050,0irq/26-eth0-rx-31447-21cpuspeed_turbos11:50:333
28405050,0irq/26-eth0-rx-31126-21cut10:05:383
28405050,0irq/26-eth0-rx-26832-21kthreadcore11:05:373
28405050,0irq/26-eth0-rx-24742-21kthreadcore12:05:373
28405050,0irq/26-eth0-rx-22734-21rt-features11:00:453
28405050,0irq/26-eth0-rx-18933-21kthreadcore12:00:393
28405050,0irq/26-eth0-rx-16284-21interrupts09:15:403
28405050,0irq/26-eth0-rx-12226-21sendmail_mailtr09:30:423
28405050,0irq/26-eth0-rx-0-21swapper/311:55:403
28405050,0irq/26-eth0-rx-0-21swapper/311:26:043
28405050,0irq/26-eth0-rx-0-21swapper/311:10:403
28405050,0irq/26-eth0-rx-0-21swapper/310:58:543
28405050,0irq/26-eth0-rx-0-21swapper/310:31:383
28405050,0irq/26-eth0-rx-0-21swapper/310:20:383
28405050,0irq/26-eth0-rx-0-21swapper/309:36:223
28405050,0irq/26-eth0-rx-0-21swapper/309:24:543
28405050,0irq/26-eth0-rx-0-21swapper/309:14:153
28405050,0irq/26-eth0-rx-0-21swapper/309:09:523
257499955,0cyclictest32294-21strings07:05:402
257489955,0cyclictest0-21swapper/111:45:381
257489955,0cyclictest0-21swapper/108:46:321
257489954,0cyclictest6445-21snmpd08:36:111
257479955,0cyclictest0-21swapper/010:11:390
257479955,0cyclictest0-21swapper/009:59:120
257479955,0cyclictest0-21swapper/009:30:400
257479955,0cyclictest0-21swapper/009:08:450
257479950,0cyclictest0-21swapper/011:43:150
259950,0migration/232144-21runrttasks11:30:332
259950,0migration/229705-21gltestperf07:55:342
259950,0migration/22957-21kthreadcore06:45:402
259950,0migration/228763-21dump-pmu-power07:55:202
259950,0migration/224578-21aten_r1power_vo08:40:322
259950,0migration/220715-21kthreadcore11:00:422
259950,0migration/218191-21kthreadcore08:10:372
259950,0migration/215410-21latency_hist07:45:202
259950,0migration/211643-21fschecks_time08:30:332
249950,0watchdog/20-21swapper/211:44:382
249950,0watchdog/20-21swapper/211:11:102
249950,0watchdog/20-21swapper/210:32:502
249950,0watchdog/20-21swapper/208:28:072
249950,0watchdog/20-21swapper/207:00:432
249950,0watchdog/20-21swapper/206:52:192
179950,0migration/15349-21kthreadcore10:10:381
179950,0migration/15208-21sh11:51:171
179950,0migration/129188-21kthreadcore07:05:381
179950,0migration/120323-21aten2.4-expect07:00:321
179950,0migration/119970-21kthreadcore12:00:391
179950,0migration/119726-21munin-node10:20:331
179950,0migration/116116-21idleruntime09:15:391
179950,0migration/113165-21taskset09:32:281
169950,0watchdog/118748-21rm10:19:181
149950,0migration/08213-21irqcore11:35:360
149950,0migration/04931-21taskset09:50:170
149950,0migration/03963-21kthreadcore11:10:400
149950,0migration/031575-21fschecks_time09:25:330
149950,0migration/030982-21sh11:05:470
149950,0migration/030208-21kthreadcore07:30:370
149950,0migration/028344-21sh09:41:160
149950,0migration/028170-21if_eth1.310:25:360
149950,0migration/026127-21kthreadcore09:20:370
149950,0migration/025532-21latency07:00:390
149950,0migration/025332-21kthreadcore11:45:390
149950,0migration/025188-21kthreadcore08:15:540
149950,0migration/024176-21munin-node09:00:340
149950,0migration/023061-21kthreadcore10:40:410
149950,0migration/020812-21sh09:36:570
149950,0migration/018716-21kthreadcore12:00:390
149950,0migration/017335-21taskset08:33:250
149950,0migration/016157-21fschecks_time07:45:350
149950,0migration/015610-21kthreadcore10:15:390
149950,0migration/012443-21munin-node11:55:390
149950,0migration/010633-21kthreadcore10:55:400
339940,0migration/3677-21runrttasks07:08:113
339940,0migration/326395-21kthreadcore07:50:473
339940,0migration/324270-21if_eth108:15:523
339940,0migration/321940-21kthreadcore07:00:373
339940,0migration/319975-21kthreadcore08:35:363
28415040,0irq/27-eth0-tx-0-21swapper/008:35:350
28415040,0irq/27-eth0-tx-0-21swapper/007:20:330
28405040,0irq/26-eth0-rx-8916-21aten_r1power_en07:40:323
28405040,0irq/26-eth0-rx-0-21swapper/310:45:343
28405040,0irq/26-eth0-rx-0-21swapper/310:25:203
28405040,0irq/26-eth0-rx-0-21swapper/306:55:323
257499943,0cyclictest18598-21awk08:35:342
257499943,0cyclictest16509-21awk07:45:372
257489944,0cyclictest0-21swapper/111:38:131
257489944,0cyclictest0-21swapper/110:52:111
257489944,0cyclictest0-21swapper/109:57:291
257489944,0cyclictest0-21swapper/108:25:341
257489943,0cyclictest6445-21snmpd08:08:411
257489943,0cyclictest6167-21sadc09:30:201
257489943,0cyclictest28028-21awk10:25:351
257489943,0cyclictest18929-21awk10:40:381
257489943,0cyclictest17009-21awk11:20:361
257489943,0cyclictest16748-21kworker/1:011:02:591
257489943,0cyclictest14527-21awk06:55:351
257489943,0cyclictest1171-21awk06:45:381
257489943,0cyclictest11142-21awk10:35:351
257479944,0cyclictest0-21swapper/010:22:410
257479944,0cyclictest0-21swapper/007:37:280
257479943,0cyclictest8814-21awk07:15:340
257479943,0cyclictest6445-21snmpd11:17:120
257479943,0cyclictest6445-21snmpd11:01:420
257479943,0cyclictest6252-21awk09:50:380
257479943,0cyclictest24798-21awk11:25:370
257479943,0cyclictest16136-21awk09:15:390
257479940,0cyclictest21201-21munin-node07:25:350
257479940,0cyclictest0-21swapper/007:13:400
259940,0migration/23143-21if_eth1.307:35:352
259940,0migration/230686-21kthreadcore06:40:382
259940,0migration/229954-21latency_hist08:20:202
259940,0migration/220523-21unixbench_singl07:20:432
259940,0migration/215907-21kthreadcore06:55:372
249940,0watchdog/26547-21runrttasks08:00:392
249940,0watchdog/20-21swapper/208:16:112
249940,0watchdog/20-21swapper/207:30:272
249940,0watchdog/20-21swapper/207:26:032
249940,0watchdog/20-21swapper/207:19:512
179940,0migration/13447-21aten_r1power_vo08:00:331
179940,0migration/132488-21latency_hist11:10:201
179940,0migration/130665-21fschecks_time08:20:321
179940,0migration/130537-21kthreadcore07:30:371
179940,0migration/127262-21taskset07:26:131
179940,0migration/12385-21taskset10:30:301
179940,0migration/122482-21fschecks_count09:40:341
179940,0migration/121289-21taskset11:41:031
179940,0migration/116585-21latency_hist08:10:201
179940,0migration/115998-21kthreadcore09:35:381
179940,0migration/111607-21kthreadcore07:40:431
179940,0migration/111123-21kthreadcore11:15:401
169940,0watchdog/16547-21runrttasks11:27:181
169940,0watchdog/129406-21python09:00:431
169940,0watchdog/126918-21ssh10:01:141
169940,0watchdog/11953-21kthreadcore09:45:381
169940,0watchdog/116018-21cpuspeed_turbos08:55:351
169940,0watchdog/1142062chrt07:19:191
169940,0watchdog/110026-21kthreadcore06:50:391
169940,0watchdog/10-21swapper/111:05:381
169940,0watchdog/10-21swapper/110:45:381
169940,0watchdog/10-21swapper/110:05:381
169940,0watchdog/10-21swapper/109:50:381
169940,0watchdog/10-21swapper/109:20:301
149940,0migration/08874-21irqcore08:50:370
149940,0migration/0802-21kthreadcore10:05:390
149940,0migration/06746-21latency_hist06:50:200
149940,0migration/06690-21kthreadcore10:50:380
149940,0migration/0409-21irqrtprio11:30:350
149940,0migration/026777-21idleruntime06:40:350
149940,0migration/01478-21kthreadcore07:55:380
149940,0migration/013121-21kthreadcore10:35:370
35130,0ktimersoftd/30-21swapper/308:26:403
28415030,0irq/27-eth0-tx-0-21swapper/007:40:390
28415030,0irq/27-eth0-tx-0-21swapper/006:50:320
28405030,0irq/26-eth0-rx-0-21swapper/307:55:363
28405030,0irq/26-eth0-rx-0-21swapper/307:46:533
28405030,0irq/26-eth0-rx-0-21swapper/307:28:143
28405030,0irq/26-eth0-rx-0-21swapper/306:50:343
28405030,0irq/26-eth0-rx-0-21swapper/306:40:323
257499932,0cyclictest11958-21kthreadcore08:05:372
257489933,0cyclictest7739-21kworker/1:208:30:241
257489933,0cyclictest7739-21kworker/1:208:15:261
257489933,0cyclictest7739-21kworker/1:207:55:241
257489933,0cyclictest7739-21kworker/1:207:50:221
257489933,0cyclictest7739-21kworker/1:207:35:221
257489933,0cyclictest7739-21kworker/1:207:20:241
257489933,0cyclictest7739-21kworker/1:207:10:301
257489933,0cyclictest7739-21kworker/1:206:40:221
257489933,0cyclictest16748-21kworker/1:012:05:231
257489933,0cyclictest16748-21kworker/1:011:30:291
257489933,0cyclictest16748-21kworker/1:009:10:281
257489933,0cyclictest16748-21kworker/1:009:05:301
257489933,0cyclictest16748-21kworker/1:008:40:261
257489932,0cyclictest10102-21awk10:55:381
257479932,0cyclictest6445-21snmpd09:14:260
257479932,0cyclictest23330-21sadc09:00:200
257479931,0cyclictest29461-21kthreadcore10:45:380
257479930,0cyclictest25917-21cut12:05:370
249930,0watchdog/20-21swapper/207:11:592
179930,0migration/115578-21latency_hist07:45:201
149930,0migration/027722-21sh10:04:060
28405020,0irq/26-eth0-rx-0-21swapper/307:35:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional