You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 09:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Fri Nov 28, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
390628782,0sleep00-21swapper/018:34:270
291350620,0irq/25-eth10-21swapper/318:35:453
1017923626,0sleep20-21swapper/218:38:012
1003723323,0sleep10-21swapper/118:36:241
8225060,0irq/24-0000:00:0-21swapper/219:39:242
339960,0migration/3356-21sh20:53:283
339960,0migration/315684-21sh21:40:483
339960,0migration/31143-21aten2.4-expect21:34:383
28415060,0irq/27-eth0-tx-31465-21ssh21:50:102
28415060,0irq/27-eth0-tx-25743-21taskset22:49:412
28415060,0irq/27-eth0-tx-23590-21ssh22:06:272
28415060,0irq/27-eth0-tx-0-21swapper/223:44:252
28415060,0irq/27-eth0-tx-0-21swapper/223:37:102
28415060,0irq/27-eth0-tx-0-21swapper/223:30:032
28415060,0irq/27-eth0-tx-0-21swapper/223:22:182
28415060,0irq/27-eth0-tx-0-21swapper/222:19:302
28415060,0irq/27-eth0-tx-0-21swapper/222:19:302
28415060,0irq/27-eth0-tx-0-21swapper/222:01:322
28415060,0irq/27-eth0-tx-0-21swapper/222:01:322
28415060,0irq/27-eth0-tx-0-21swapper/221:35:232
28415060,0irq/27-eth0-tx-0-21swapper/221:29:032
28415060,0irq/27-eth0-tx-0-21swapper/221:01:492
28415060,0irq/27-eth0-tx-0-21swapper/220:56:492
28415060,0irq/27-eth0-tx-0-21swapper/200:06:032
28415060,0irq/27-eth0-tx-0-21swapper/020:24:460
28405060,0irq/26-eth0-rx-877-21diskstats22:14:383
28405060,0irq/26-eth0-rx-27579-21kthreadcore22:49:423
28405060,0irq/26-eth0-rx-15388-21diskstats23:44:393
28405060,0irq/26-eth0-rx-14391-21python22:19:473
28405060,0irq/26-eth0-rx-14391-21python22:19:473
169960,0watchdog/10-21swapper/122:40:341
149960,0migration/027961-21aten_r1power_cu20:29:400
8225050,0irq/24-0000:00:7836-21cat20:14:242
8225050,0irq/24-0000:00:0-21swapper/221:04:502
4150,0ktimersoftd/00-21swapper/021:27:320
339950,0migration/36310-21munin-node19:24:363
339950,0migration/33461-21kthreadcore21:54:423
339950,0migration/33426-21kthreadcore19:19:443
339950,0migration/332462-21latency_hist19:44:243
339950,0migration/331942-21sh21:10:013
339950,0migration/331678-21sh21:29:593
339950,0migration/328910-21kthreadcore23:29:443
339950,0migration/328721-21timerandwakeup23:49:493
339950,0migration/323475-21unixbench_multi21:24:483
339950,0migration/32329-21irqrtprio20:09:403
339950,0migration/322958-21kthreadcore19:59:423
339950,0migration/32134-21munin-node20:54:433
339950,0migration/320278-21kthreadcore19:09:403
339950,0migration/318467-21irqrtprio18:44:413
339950,0migration/318406-21switchtime19:29:473
339950,0migration/318353-21irqcore21:44:403
339950,0migration/31688-21kthreadcore23:34:413
339950,0migration/315738-21taskset21:00:223
339950,0migration/314305-21timerandwakeup22:39:473
339950,0migration/313467-21taskset00:03:513
28415050,0irq/27-eth0-tx-5971-21munin-node19:24:430
28415050,0irq/27-eth0-tx-479-21df_abs22:34:372
28415050,0irq/27-eth0-tx-3737-21mdstat23:54:452
28415050,0irq/27-eth0-tx-18324-21munin-node19:09:380
28415050,0irq/27-eth0-tx-0-21swapper/223:26:552
28415050,0irq/27-eth0-tx-0-21swapper/223:18:092
28415050,0irq/27-eth0-tx-0-21swapper/223:08:072
28415050,0irq/27-eth0-tx-0-21swapper/223:03:332
28415050,0irq/27-eth0-tx-0-21swapper/222:55:312
28415050,0irq/27-eth0-tx-0-21swapper/222:45:572
28415050,0irq/27-eth0-tx-0-21swapper/222:39:272
28415050,0irq/27-eth0-tx-0-21swapper/222:32:232
28415050,0irq/27-eth0-tx-0-21swapper/222:18:262
28415050,0irq/27-eth0-tx-0-21swapper/221:54:592
28415050,0irq/27-eth0-tx-0-21swapper/221:49:072
28415050,0irq/27-eth0-tx-0-21swapper/221:44:032
28415050,0irq/27-eth0-tx-0-21swapper/221:30:312
28415050,0irq/27-eth0-tx-0-21swapper/221:23:342
28415050,0irq/27-eth0-tx-0-21swapper/221:17:022
28415050,0irq/27-eth0-tx-0-21swapper/221:10:072
28415050,0irq/27-eth0-tx-0-21swapper/220:49:502
28415050,0irq/27-eth0-tx-0-21swapper/220:45:282
28415050,0irq/27-eth0-tx-0-21swapper/220:40:092
28415050,0irq/27-eth0-tx-0-21swapper/019:51:400
28415050,0irq/27-eth0-tx-0-21swapper/019:14:340
28405050,0irq/26-eth0-rx-9821-21df_inode21:19:393
28405050,0irq/26-eth0-rx-3130-21kthreadcore21:14:413
28405050,0irq/26-eth0-rx-28344-21kthreadcore21:49:423
28405050,0irq/26-eth0-rx-24510-21grep22:29:243
28405050,0irq/26-eth0-rx-22203-21kthreadcore19:34:423
28405050,0irq/26-eth0-rx-20826-21grep21:04:433
28405050,0irq/26-eth0-rx-18723-21kthreadcore23:04:403
28405050,0irq/26-eth0-rx-17332-21df_inode22:04:383
28405050,0irq/26-eth0-rx-14441-21missed_timers20:39:433
28405050,0irq/26-eth0-rx-12502-21munin-node23:39:433
28405050,0irq/26-eth0-rx-0-21swapper/323:21:183
28405050,0irq/26-eth0-rx-0-21swapper/323:15:373
28405050,0irq/26-eth0-rx-0-21swapper/323:04:223
28405050,0irq/26-eth0-rx-0-21swapper/322:14:213
28405050,0irq/26-eth0-rx-0-21swapper/322:04:073
28405050,0irq/26-eth0-rx-0-21swapper/322:04:073
28405050,0irq/26-eth0-rx-0-21swapper/320:46:003
28405050,0irq/26-eth0-rx-0-21swapper/319:49:433
28405050,0irq/26-eth0-rx-0-21swapper/318:59:403
27150,0ktimersoftd/20-21swapper/223:10:122
259950,0migration/25422-21fschecks_count18:59:372
259950,0migration/225317-21munin-node22:09:372
249950,0watchdog/26623-21needreboot20:09:432
249950,0watchdog/20-21swapper/223:40:462
169950,0watchdog/19797-21kthreadcore23:39:421
169950,0watchdog/1158062sleep121:00:301
169950,0watchdog/10-21swapper/123:16:101
169950,0watchdog/10-21swapper/121:54:381
169950,0watchdog/10-21swapper/121:33:421
169950,0watchdog/10-21swapper/120:29:461
169950,0watchdog/10-21swapper/119:05:431
149950,0migration/08039-21kthreadcore23:59:470
149950,0migration/07839-21taskset22:18:000
149950,0migration/06055-21timerwakeupswit22:54:480
149950,0migration/04895-21swap23:34:460
149950,0migration/04749-21kthreadcore21:34:440
149950,0migration/031924-21cpuspeed_turbos19:19:390
149950,0migration/0313-21taskset19:44:360
149950,0migration/03117-21kthreadcore20:54:440
149950,0migration/030336-21timerandwakeup22:49:470
149950,0migration/029894-21kthreadcore21:29:420
149950,0migration/028390-21kthreadcore20:04:410
149950,0migration/02696-21kthreadcore23:54:440
149950,0migration/026085-21kthreadcore23:49:430
149950,0migration/025930-21kthreadcore18:49:440
149950,0migration/024205-21aten_r1power_en23:09:360
149950,0migration/023673-21sh23:08:500
149950,0migration/021941-21munin-node22:44:440
149950,0migration/01994-21kthreadcore18:54:410
149950,0migration/017703-21gltestperf22:04:390
149950,0migration/017511-21kthreadcore23:44:430
149950,0migration/017015-21kthreadcore23:24:400
149950,0migration/016090-21kthreadcore20:19:430
149950,0migration/015830-21sh22:02:070
149950,0migration/015830-21sh22:02:070
149950,0migration/015328-21kthreadcore18:39:420
149950,0migration/014223-21kthreadcore19:29:410
149950,0migration/014106-21kthreadcore21:19:440
149950,0migration/013945-21cpuspeed_turbos19:54:370
149950,0migration/011796-21kthreadcore22:39:420
149950,0migration/011505-21kthreadcore23:39:430
103709955,0cyclictest0-21swapper/223:53:152
103709954,0cyclictest6445-21snmpd20:35:112
103709953,0cyclictest17468-21awk22:24:392
103669954,0cyclictest6547-21runrttasks21:16:081
103669950,0cyclictest0-21swapper/122:01:331
103669950,0cyclictest0-21swapper/122:01:331
103659955,0cyclictest0-21swapper/023:02:550
103659955,0cyclictest0-21swapper/021:51:390
103659950,0cyclictest0-21swapper/022:31:140
103659950,0cyclictest0-21swapper/022:22:520
103659950,0cyclictest0-21swapper/022:22:520
103659950,0cyclictest0-21swapper/021:43:560
103659950,0cyclictest0-21swapper/020:52:420
103659950,0cyclictest0-21swapper/020:44:580
339940,0migration/3477-21kthreadcore20:29:463
339940,0migration/324034-21aten_r1power_po18:49:393
339940,0migration/320370-21kthreadcore23:24:413
28415040,0irq/27-eth0-tx-0-21swapper/020:39:260
28415040,0irq/27-eth0-tx-0-21swapper/020:34:380
28415040,0irq/27-eth0-tx-0-21swapper/019:04:480
28405040,0irq/26-eth0-rx-24925-21latency_hist19:14:243
28405040,0irq/26-eth0-rx-0-21swapper/323:57:313
28405040,0irq/26-eth0-rx-0-21swapper/323:10:253
28405040,0irq/26-eth0-rx-0-21swapper/322:54:293
28405040,0irq/26-eth0-rx-0-21swapper/322:49:143
28405040,0irq/26-eth0-rx-0-21swapper/322:35:023
28405040,0irq/26-eth0-rx-0-21swapper/322:24:483
28405040,0irq/26-eth0-rx-0-21swapper/319:54:413
28405040,0irq/26-eth0-rx-0-21swapper/300:04:363
259940,0migration/28508-21runrttasks19:49:422
259940,0migration/231632-21runrttasks19:19:332
259940,0migration/229470-21kthreadcore20:29:452
259940,0migration/22775-21kthreadcore18:54:412
259940,0migration/217108-21munin-node19:04:442
259940,0migration/215697-21irqcore20:19:422
259940,0migration/210626-21kthreadcore19:24:412
179940,0migration/18486-21gltestperf23:19:401
179940,0migration/18224-21kernelversion19:49:421
179940,0migration/15592-21fschecks_time18:59:371
179940,0migration/1418-21kthreadcore23:34:401
179940,0migration/128380-21kthreadcore21:49:411
179940,0migration/127318-21kthreadcore22:29:421
179940,0migration/127251-21kthreadcore21:09:411
179940,0migration/125012-21munin-node19:14:341
179940,0migration/124904-21if_eth1.318:49:431
179940,0migration/124445-21kthreadcore20:24:421
179940,0migration/122647-21kthreadcore19:09:411
179940,0migration/121210-21if_eth119:59:401
169940,0watchdog/119328-21kthreadcore22:04:421
169940,0watchdog/116123-21kthreadcore20:19:431
169940,0watchdog/10-21swapper/123:59:461
169940,0watchdog/10-21swapper/123:44:501
169940,0watchdog/10-21swapper/123:34:021
169940,0watchdog/10-21swapper/123:24:381
169940,0watchdog/10-21swapper/123:09:381
169940,0watchdog/10-21swapper/122:46:141
169940,0watchdog/10-21swapper/122:36:221
169940,0watchdog/10-21swapper/122:23:141
169940,0watchdog/10-21swapper/122:23:141
169940,0watchdog/10-21swapper/121:39:381
169940,0watchdog/10-21swapper/120:39:421
169940,0watchdog/10-21swapper/120:34:421
169940,0watchdog/10-21swapper/120:04:431
169940,0watchdog/10-21swapper/100:06:421
149940,0migration/030513-21kthreadcore19:39:420
149940,0migration/029974-21kthreadcore22:09:420
149940,0migration/028135-21kthreadcore23:29:440
149940,0migration/018728-21kthreadcore22:24:420
149940,0migration/012861-21kthreadcore23:19:440
103709943,0cyclictest7037-21awk23:59:452
103709943,0cyclictest6705-21sadc19:49:242
103709943,0cyclictest6445-21snmpd20:01:102
103709943,0cyclictest6445-21snmpd19:43:102
103709943,0cyclictest3425-21lldpd18:44:402
103709943,0cyclictest3110-21irqbalance18:49:382
103709943,0cyclictest27791-21awk20:04:392
103709943,0cyclictest27694-21sadc20:29:242
103709943,0cyclictest25955-21awk19:14:382
103709943,0cyclictest19207-21awk19:09:382
103709943,0cyclictest11469-21awk18:39:392
103669943,0cyclictest786-21kworker/1:018:39:381
103669943,0cyclictest3425-21lldpd21:37:121
103669943,0cyclictest3425-21lldpd21:04:411
103669943,0cyclictest3425-21lldpd20:45:111
103669943,0cyclictest3425-21lldpd19:45:401
103669943,0cyclictest32548-21awk19:19:411
103669943,0cyclictest1898-21kworker/1:022:14:561
103669943,0cyclictest1898-21kworker/1:021:21:321
103669943,0cyclictest18370-21awk21:24:401
103669943,0cyclictest17151-21awk23:04:381
103669943,0cyclictest15375-21kworker/1:120:54:281
103669943,0cyclictest15375-21kworker/1:120:53:461
103669943,0cyclictest14188-21kworker/1:023:55:411
103669943,0cyclictest14188-21kworker/1:023:04:031
103669943,0cyclictest13617-21awk19:29:391
103669940,0cyclictest0-21swapper/123:49:321
103669940,0cyclictest0-21swapper/122:24:341
103669940,0cyclictest0-21swapper/122:13:031
103659944,0cyclictest0-21swapper/021:59:230
103659944,0cyclictest0-21swapper/021:13:400
103659943,0cyclictest6445-21snmpd18:47:550
103659943,0cyclictest32204-21cut23:14:360
103659943,0cyclictest21383-21sadc00:09:230
103659943,0cyclictest2117-21awk20:09:390
103659943,0cyclictest21151-21awk19:59:390
103659943,0cyclictest20200-21awk19:34:400
103659943,0cyclictest18476-21awk21:04:410
339930,0migration/38359-21runrttasks20:35:093
339930,0migration/313241-21runrttasks20:14:433
28415030,0irq/27-eth0-tx-0-21swapper/020:14:460
28415030,0irq/27-eth0-tx-0-21swapper/018:59:350
28405030,0irq/26-eth0-rx-0-21swapper/320:25:003
28405030,0irq/26-eth0-rx-0-21swapper/320:04:283
28405030,0irq/26-eth0-rx-0-21swapper/319:04:353
28405030,0irq/26-eth0-rx-0-21swapper/318:56:293
28405030,0irq/26-eth0-rx-0-21swapper/318:39:363
249930,0watchdog/20-21swapper/219:29:432
169930,0watchdog/10-21swapper/121:44:341
169930,0watchdog/10-21swapper/119:24:311
103709930,0cyclictest19084-21ntp_states19:54:452
103669933,0cyclictest786-21kworker/1:020:09:381
103669933,0cyclictest786-21kworker/1:019:34:381
103669933,0cyclictest786-21kworker/1:018:54:441
103669933,0cyclictest786-21kworker/1:018:44:281
103669933,0cyclictest14188-21kworker/1:022:54:231
103669932,0cyclictest6445-21snmpd20:14:251
103669932,0cyclictest26221-21cut19:39:371
103669932,0cyclictest25371-21awk22:49:391
103669932,0cyclictest13659-21aten_r1power_en19:54:361
103659933,0cyclictest6445-21snmpd22:36:270
103659933,0cyclictest0-21swapper/020:59:280
103659932,0cyclictest6445-21snmpd21:47:260
103659930,0cyclictest7180-21python21:14:470
28405020,0irq/26-eth0-rx-0-21swapper/320:19:343
28405020,0irq/26-eth0-rx-0-21swapper/319:44:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional