You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 10:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Mon Nov 03, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350850,0irq/25-eth10-21swapper/318:36:483
1491327473,0sleep20-21swapper/218:38:052
1491225251,0sleep10-21swapper/118:38:041
1503323228,0sleep00-21swapper/018:39:320
151819977,0cyclictest13448-21diskmemload23:10:362
339960,0migration/39350-21sh23:59:413
339960,0migration/34488-21sh21:32:503
339960,0migration/33949-21sh20:51:183
339960,0migration/329171-21sh21:49:373
339960,0migration/319841-21sh22:01:273
28415060,0irq/27-eth0-tx-0-21swapper/223:52:012
28415060,0irq/27-eth0-tx-0-21swapper/223:30:532
28415060,0irq/27-eth0-tx-0-21swapper/223:23:422
28415060,0irq/27-eth0-tx-0-21swapper/223:16:102
28415060,0irq/27-eth0-tx-0-21swapper/223:08:042
28415060,0irq/27-eth0-tx-0-21swapper/223:02:042
28415060,0irq/27-eth0-tx-0-21swapper/222:51:262
28415060,0irq/27-eth0-tx-0-21swapper/222:47:212
28415060,0irq/27-eth0-tx-0-21swapper/222:29:482
28415060,0irq/27-eth0-tx-0-21swapper/222:17:452
28415060,0irq/27-eth0-tx-0-21swapper/221:59:012
28415060,0irq/27-eth0-tx-0-21swapper/221:48:272
28415060,0irq/27-eth0-tx-0-21swapper/221:35:472
28415060,0irq/27-eth0-tx-0-21swapper/221:09:212
28415060,0irq/27-eth0-tx-0-21swapper/221:03:012
28415060,0irq/27-eth0-tx-0-21swapper/220:55:552
28415060,0irq/27-eth0-tx-0-21swapper/220:46:492
28415060,0irq/27-eth0-tx-0-21swapper/200:05:512
28405060,0irq/26-eth0-rx-23164-21tr20:45:383
28405060,0irq/26-eth0-rx-21855-21diskstats22:05:323
28405060,0irq/26-eth0-rx-19493-21sh22:21:353
28405060,0irq/26-eth0-rx-15615-21which00:00:433
249960,0watchdog/20-21swapper/223:28:572
249960,0watchdog/20-21swapper/222:30:372
151819960,0cyclictest0-21swapper/223:55:322
151819960,0cyclictest0-21swapper/219:40:362
151729965,0cyclictest10856-21sed19:00:350
149960,0migration/06743-21idleruntime21:15:350
149960,0migration/031839-21latency23:50:370
149960,0migration/012987-21gltestperf23:00:340
35150,0ktimersoftd/30-21swapper/321:11:463
339950,0migration/39140-21kthreadcore21:15:383
339950,0migration/37742-21kthreadcore18:55:393
339950,0migration/36165-21timerwakeupswit20:30:423
339950,0migration/36016-21munin-node22:15:343
339950,0migration/34826-21latency_hist19:45:193
339950,0migration/331346-21kthreadcore23:10:353
339950,0migration/328291-21munin-node21:05:443
339950,0migration/327642-21irqcore23:50:353
339950,0migration/323649-21timerwakeupswit00:05:453
339950,0migration/323141-21aten_r1power_po19:10:323
339950,0migration/319553-21sh23:04:023
339950,0migration/319092-21kthreadcore21:20:373
339950,0migration/314039-21sh21:00:323
339950,0migration/313055-21taskset20:35:553
339950,0migration/312828-21kthreadcore19:00:363
339950,0migration/310128-21switchtime22:55:453
28415050,0irq/27-eth0-tx-23990-21munin-node19:35:370
28415050,0irq/27-eth0-tx-17395-21ssh23:41:472
28415050,0irq/27-eth0-tx-0-21swapper/222:20:522
28415050,0irq/27-eth0-tx-0-21swapper/221:52:072
28415050,0irq/27-eth0-tx-0-21swapper/221:25:482
28415050,0irq/27-eth0-tx-0-21swapper/221:21:152
28415050,0irq/27-eth0-tx-0-21swapper/220:54:102
28415050,0irq/27-eth0-tx-0-21swapper/200:00:472
28405050,0irq/26-eth0-rx-8831-21sensors23:35:423
28405050,0irq/26-eth0-rx-30412-21if_eth1.222:10:363
28405050,0irq/26-eth0-rx-26264-21cut22:25:403
28405050,0irq/26-eth0-rx-22475-21kthreadcore22:45:383
28405050,0irq/26-eth0-rx-11049-21latency21:35:423
28405050,0irq/26-eth0-rx-10390-21kthreadcore21:55:403
28405050,0irq/26-eth0-rx-0-21swapper/323:45:373
28405050,0irq/26-eth0-rx-0-21swapper/323:45:373
28405050,0irq/26-eth0-rx-0-21swapper/323:40:283
28405050,0irq/26-eth0-rx-0-21swapper/323:26:203
28405050,0irq/26-eth0-rx-0-21swapper/323:06:063
28405050,0irq/26-eth0-rx-0-21swapper/321:50:363
28405050,0irq/26-eth0-rx-0-21swapper/321:30:023
28405050,0irq/26-eth0-rx-0-21swapper/320:56:063
28405050,0irq/26-eth0-rx-0-21swapper/320:44:133
28405050,0irq/26-eth0-rx-0-21swapper/320:25:363
27150,0ktimersoftd/213448-21diskmemload21:11:462
259950,0migration/26494-21fschecks_time21:15:342
259950,0migration/227670-21sendmail_mailqu18:45:402
259950,0migration/210245-21timerwakeupswit22:55:452
249950,0watchdog/222323-21unixbench_singl19:05:422
249950,0watchdog/219162-21needreboot20:40:372
249950,0watchdog/20-21swapper/223:47:292
249950,0watchdog/20-21swapper/223:47:292
249950,0watchdog/20-21swapper/223:38:172
249950,0watchdog/20-21swapper/222:41:212
249950,0watchdog/20-21swapper/222:35:332
249950,0watchdog/20-21swapper/222:10:292
249950,0watchdog/20-21swapper/222:01:172
249950,0watchdog/20-21swapper/221:40:532
249950,0watchdog/20-21swapper/221:33:172
249950,0watchdog/20-21swapper/220:38:132
249950,0watchdog/20-21swapper/220:15:212
249950,0watchdog/20-21swapper/219:55:222
249950,0watchdog/20-21swapper/219:30:382
249950,0watchdog/20-21swapper/218:54:182
169950,0watchdog/123771-21grep19:10:341
151819955,0cyclictest0-21swapper/222:05:392
151739955,0cyclictest13448-21diskmemload21:50:401
151739955,0cyclictest0-21swapper/123:46:421
151739955,0cyclictest0-21swapper/123:46:421
151739955,0cyclictest0-21swapper/120:48:231
151739955,0cyclictest0-21swapper/100:06:051
151739950,0cyclictest0-21swapper/123:41:391
151739950,0cyclictest0-21swapper/123:27:091
151739950,0cyclictest0-21swapper/122:30:521
151739950,0cyclictest0-21swapper/122:15:061
151739950,0cyclictest0-21swapper/121:30:191
151729955,0cyclictest8257-21kthreadcore22:15:370
151729955,0cyclictest0-21swapper/023:21:220
151729955,0cyclictest0-21swapper/022:21:270
151729952,0cyclictest12386-21munin-run22:40:190
151729950,0cyclictest0-21swapper/023:35:330
151729950,0cyclictest0-21swapper/021:31:470
149950,0migration/09721-21kthreadcore20:55:390
149950,0migration/06004-21kthreadcore22:55:390
149950,0migration/04110-21sh22:34:360
149950,0migration/03916-21aten_r1power_en23:15:330
149950,0migration/03673-21taskset19:19:070
149950,0migration/03240-21kthreadcore20:05:370
149950,0migration/031150-21taskset20:02:220
149950,0migration/031073-21munin-node21:10:340
149950,0migration/0308-21aten_r1power_vo20:30:310
149950,0migration/029616-21taskset21:30:060
149950,0migration/029448-21sh22:10:300
149950,0migration/028706-21munin-node18:50:320
149950,0migration/026174-21kthreadcore22:25:400
149950,0migration/025375-21latency_hist00:10:190
149950,0migration/024549-21kthreadcore23:25:400
149950,0migration/024064-21kthreadcore23:05:370
149950,0migration/023022-21taskset21:05:350
149950,0migration/020857-21taskset21:43:440
149950,0migration/020727-21sh21:22:390
149950,0migration/019939-21kthreadcore19:55:360
149950,0migration/016111-21kthreadcore20:40:360
149950,0migration/015522-21kthreadcore21:00:380
149950,0migration/01509-21kthreadcore20:50:410
149950,0migration/013079-21taskset20:36:130
149950,0migration/012333-21sh21:57:570
149950,0migration/01227-21kthreadcore19:40:360
149950,0migration/012022-21kernelversion19:25:400
149950,0migration/010013-21kthreadcore19:45:360
339940,0migration/36490-21kthreadcore19:20:363
339940,0migration/332528-21taskset20:05:353
339940,0migration/331062-21kthreadcore18:50:383
339940,0migration/330493-21latency_hist19:40:193
339940,0migration/330239-21irqrtprio22:30:363
339940,0migration/325710-21kthreadcore18:45:373
339940,0migration/325590-21runrttasks20:23:363
339940,0migration/318517-21kthreadcore19:05:353
339940,0migration/314913-21kthreadcore22:40:383
339940,0migration/314001-21kthreadcore23:20:383
339940,0migration/31378-21sh23:30:493
28405040,0irq/26-eth0-rx-0-21swapper/323:16:243
28405040,0irq/26-eth0-rx-0-21swapper/322:50:503
28405040,0irq/26-eth0-rx-0-21swapper/322:36:133
28405040,0irq/26-eth0-rx-0-21swapper/321:40:553
259940,0migration/232730-21aten_r1power_fr20:30:312
259940,0migration/224254-21kthreadcore20:20:382
259940,0migration/216897-21kthreadcore18:40:362
259940,0migration/212931-21kthreadcore19:00:362
249940,0watchdog/20-21swapper/220:06:252
249940,0watchdog/20-21swapper/219:50:342
249940,0watchdog/20-21swapper/219:45:302
249940,0watchdog/20-21swapper/219:16:062
249940,0watchdog/20-21swapper/219:10:222
249940,0watchdog/20-21swapper/218:56:182
18793240,0sleep00-21swapper/023:45:300
18793240,0sleep00-21swapper/023:45:300
179940,0migration/18874-21kthreadcore20:55:381
179940,0migration/129109-21kthreadcore20:25:361
179940,0migration/123230-21kthreadcore23:05:361
179940,0migration/120748-21needreboot18:40:381
179940,0migration/120481-21taskset22:03:201
179940,0migration/114806-21idleruntime21:00:351
179940,0migration/114689-21kthreadcore19:25:411
169940,0watchdog/19690-21munin-node19:00:421
169940,0watchdog/14341-21aten_r1power_fr22:55:331
169940,0watchdog/14097-21ssh20:51:411
169940,0watchdog/128287-21cut23:10:291
169940,0watchdog/125515-21fschecks_count20:00:331
169940,0watchdog/124485-21kthreadcore21:25:371
169940,0watchdog/122822-21cut20:20:371
169940,0watchdog/121211-21ssh21:24:051
169940,0watchdog/120442-21ssh20:42:451
169940,0watchdog/118217-21ntp_states22:40:411
169940,0watchdog/114197-21kthreadcore23:00:371
169940,0watchdog/10-21swapper/122:20:371
169940,0watchdog/10-21swapper/100:02:091
151819943,0cyclictest6965-21awk20:10:352
151819943,0cyclictest3110-21irqbalance20:25:332
151819943,0cyclictest3110-21irqbalance20:02:432
151819943,0cyclictest3110-21irqbalance19:36:032
151819943,0cyclictest3110-21irqbalance19:24:032
151819943,0cyclictest11598-21awk19:25:382
151739944,0cyclictest0-21swapper/121:10:371
151739944,0cyclictest0-21swapper/119:50:411
151739943,0cyclictest7887-21awk20:35:361
151739943,0cyclictest6445-21snmpd22:17:181
151739943,0cyclictest6445-21snmpd22:09:031
151739943,0cyclictest6445-21snmpd21:57:331
151739943,0cyclictest6445-21snmpd21:35:481
151739943,0cyclictest6445-21snmpd00:00:191
151739943,0cyclictest3425-21lldpd22:37:231
151739943,0cyclictest3425-21lldpd18:51:381
151739943,0cyclictest30642-21awk19:15:361
151739943,0cyclictest24898-21awk19:35:341
151739943,0cyclictest23064-21awk21:05:351
151739943,0cyclictest17304-21awk19:05:341
151739943,0cyclictest12818-21awk23:20:361
151729944,0cyclictest0-21swapper/022:43:030
151729943,0cyclictest3425-21lldpd23:10:540
151729943,0cyclictest22625-21awk21:45:350
151729943,0cyclictest22512-21sed20:45:360
149940,0migration/07262-21kthreadcore23:55:390
149940,0migration/029260-21runrttasks19:12:520
149940,0migration/028179-21fschecks_count23:30:320
149940,0migration/027042-21sh22:46:300
149940,0migration/021567-21aten_r1power_po22:05:310
149940,0migration/018939-21kthreadcore19:30:360
149940,0migration/016798-21sh00:03:220
149940,0migration/011025-21aten_r1power_fr23:40:340
339930,0migration/311684-21smart_sdb20:10:413
28415030,0irq/27-eth0-tx-0-21swapper/019:50:340
28405030,0irq/26-eth0-rx-0-21swapper/320:15:343
28405030,0irq/26-eth0-rx-0-21swapper/320:00:323
28405030,0irq/26-eth0-rx-0-21swapper/319:55:313
28405030,0irq/26-eth0-rx-0-21swapper/319:45:313
28405030,0irq/26-eth0-rx-0-21swapper/319:30:373
28405030,0irq/26-eth0-rx-0-21swapper/319:25:353
28405030,0irq/26-eth0-rx-0-21swapper/319:17:113
28405030,0irq/26-eth0-rx-0-21swapper/318:45:033
179930,0migration/12128-21kthreadcore19:40:361
169930,0watchdog/111283-21latency20:10:371
169930,0watchdog/10-21swapper/123:50:371
169930,0watchdog/10-21swapper/123:35:331
169930,0watchdog/10-21swapper/122:51:251
169930,0watchdog/10-21swapper/122:45:411
169930,0watchdog/10-21swapper/120:30:411
169930,0watchdog/10-21swapper/118:55:421
169930,0watchdog/10-21swapper/118:45:341
151739933,0cyclictest0-21swapper/119:20:251
151739932,0cyclictest7110-21sed21:15:371
151739932,0cyclictest5914-21awk19:45:341
151739932,0cyclictest4852-21awk23:15:381
151739932,0cyclictest32624-21awk20:05:351
151739932,0cyclictest28530-21awk23:30:341
151739932,0cyclictest22699-21awk21:45:361
151739932,0cyclictest18316-21sensors20:15:441
151739932,0cyclictest14643-21awk21:40:361
151739931,0cyclictest25220-21kthreadcore22:25:401
151739930,0cyclictest17741-21diskstats19:30:321
151739930,0cyclictest0-21swapper/119:57:371
151729933,0cyclictest0-21swapper/022:50:560
151729933,0cyclictest0-21swapper/022:00:480
151729933,0cyclictest0-21swapper/021:53:330
151729933,0cyclictest0-21swapper/021:37:180
151729932,0cyclictest6445-21snmpd20:24:170
151729932,0cyclictest17225-21awk19:05:330
151729932,0cyclictest14848-21kthreadcore20:15:390
151729930,0cyclictest5979-21munin-node20:10:320
151729930,0cyclictest0-21swapper/018:55:350
151729930,0cyclictest0-21swapper/018:40:420
28405020,0irq/26-eth0-rx-0-21swapper/319:51:063
151729921,0cyclictest25915-21sed20:25:190
151729920,0cyclictest3780-21crond19:20:190
151729920,0cyclictest0-21swapper/018:45:320
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional