You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 07:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Jul 19, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2141223523,0sleep20-21swapper/218:41:342
2158223422,0sleep00-21swapper/018:43:320
2127823324,0sleep10-21swapper/118:39:571
291350320,0irq/25-eth10-21swapper/318:40:363
217229993,0cyclictest3425-21lldpd23:51:342
217229993,0cyclictest3425-21lldpd23:51:342
217229992,0cyclictest0-21swapper/221:40:142
217229991,0cyclictest0-21swapper/222:52:192
217229991,0cyclictest0-21swapper/221:50:222
217229991,0cyclictest0-21swapper/220:51:402
217229991,0cyclictest0-21swapper/200:05:072
217229990,0cyclictest0-21swapper/222:39:082
217229990,0cyclictest0-21swapper/222:07:422
217229990,0cyclictest0-21swapper/221:47:412
217229983,0cyclictest31504-21awk19:39:362
217229983,0cyclictest10519-21awk18:59:342
217229982,0cyclictest14452-21awk22:19:342
217229982,0cyclictest0-21swapper/223:44:562
217229982,0cyclictest0-21swapper/221:59:452
217229982,0cyclictest0-21swapper/221:57:142
217229982,0cyclictest0-21swapper/221:25:092
217229981,0cyclictest0-21swapper/222:54:382
217229981,0cyclictest0-21swapper/221:16:332
217229980,0cyclictest3907-21munin-node20:54:422
217229980,0cyclictest22533-21fschecks_time22:44:292
217229980,0cyclictest20775-21aten2.4-expect21:04:302
217229980,0cyclictest12154-21cat20:59:182
217229980,0cyclictest0-21swapper/223:55:472
217229980,0cyclictest0-21swapper/223:34:302
217229980,0cyclictest0-21swapper/223:30:362
217229980,0cyclictest0-21swapper/223:26:592
217229980,0cyclictest0-21swapper/223:19:582
217229980,0cyclictest0-21swapper/223:15:272
217229980,0cyclictest0-21swapper/223:10:342
217229980,0cyclictest0-21swapper/223:01:582
217229980,0cyclictest0-21swapper/222:41:502
217229980,0cyclictest0-21swapper/222:29:362
217229980,0cyclictest0-21swapper/222:25:272
217229980,0cyclictest0-21swapper/222:14:422
217229980,0cyclictest0-21swapper/222:14:132
217229980,0cyclictest0-21swapper/221:30:482
217229980,0cyclictest0-21swapper/221:19:262
217229980,0cyclictest0-21swapper/221:09:432
217229980,0cyclictest0-21swapper/220:49:022
217229980,0cyclictest0-21swapper/219:59:422
339970,0migration/35216-21sh23:54:133
339970,0migration/35216-21sh23:54:133
217229973,0cyclictest30497-21awk19:14:352
217229972,0cyclictest0-21swapper/219:59:022
217229971,0cyclictest28425-21kthreadcore19:34:372
217229970,0cyclictest6813-21iostat_ios20:09:382
217229970,0cyclictest5042-21munin-node21:34:352
217229970,0cyclictest38872sleep219:19:282
217229970,0cyclictest31843-21diskstats20:04:322
217229970,0cyclictest315-21df_inode20:29:332
217229970,0cyclictest30706-21kernelversion00:09:362
217229970,0cyclictest28441-21port_sshpriv19:09:412
217229970,0cyclictest27770-21munin-node18:44:422
217229970,0cyclictest26618-21munin-node20:24:372
217229970,0cyclictest22216-21ssh23:04:252
217229970,0cyclictest20089-21iostat_ios20:19:352
217229970,0cyclictest17808-21perl19:29:312
217229970,0cyclictest17512-21irqstats19:04:352
217229970,0cyclictest1511-21missed_timers18:49:382
217229970,0cyclictest14317-21munin-node23:39:342
217229970,0cyclictest14164-21hddtemp_smartct20:39:322
217229970,0cyclictest13931-21diskstats23:59:332
217229970,0cyclictest13437-21iostat20:14:352
217229970,0cyclictest12745-21munin-node20:34:432
217229970,0cyclictest12172-21perl19:49:312
217229970,0cyclictest11557-21iostat19:24:352
217229970,0cyclictest10873-21yum19:44:422
217229970,0cyclictest0-21swapper/218:54:352
217209975,0cyclictest0-21swapper/121:07:161
217209973,0cyclictest3425-21lldpd21:46:041
217209973,0cyclictest3425-21lldpd00:08:341
217209973,0cyclictest31104-21awk22:49:351
217209973,0cyclictest14711-21awk22:39:341
217209973,0cyclictest10326-21sadc19:24:181
339960,0migration/34828-21sh23:11:383
339960,0migration/34056-21sh21:32:253
339960,0migration/33529-21sh22:09:533
339960,0migration/328069-21sh22:25:103
339960,0migration/321308-21sh22:43:143
339960,0migration/313641-21aten2.4-expect22:19:303
339960,0migration/313574-21sh23:18:143
339960,0migration/313082-21sh23:36:353
28415060,0irq/27-eth0-tx-6372-21iceccd20:19:590
28415060,0irq/27-eth0-tx-24808-21munin-node19:59:370
28415060,0irq/27-eth0-tx-0-21swapper/019:04:330
28415060,0irq/27-eth0-tx-0-21swapper/018:54:470
28405060,0irq/26-eth0-rx-27558-21ntp_states22:44:363
217209965,0cyclictest0-21swapper/100:02:161
217209963,0cyclictest3425-21lldpd22:29:041
217209963,0cyclictest3425-21lldpd21:51:341
217209962,0cyclictest6564-21awk22:34:331
217209962,0cyclictest32284-21awk20:04:341
217209962,0cyclictest16814-21sshd19:27:121
217209962,0cyclictest15355-21kthreadcore20:14:371
217209961,0cyclictest31432-21awk19:39:351
217209960,0cyclictest8086-21kthreadcore23:14:361
217209960,0cyclictest6393-21tune2fs20:09:351
217209960,0cyclictest5763-21egrep23:54:321
217209960,0cyclictest4926-21cut21:34:181
217209960,0cyclictest27106-21tr19:59:371
217209960,0cyclictest24853-21cut19:09:371
217209960,0cyclictest2459-21needreboot00:09:381
217209960,0cyclictest24091-21unixbench_singl19:54:411
217209960,0cyclictest23824-21kthreadcore20:44:371
217209960,0cyclictest2378-21kthreadcore23:29:371
217209960,0cyclictest21292-21ssh23:02:331
217209960,0cyclictest19971-21if_eth020:19:341
217209960,0cyclictest18802-21pmu-power21:39:381
217209960,0cyclictest16843-21kthreadcore21:59:331
217209960,0cyclictest14804-21munin-node23:19:311
217209960,0cyclictest14147-21sed20:39:321
217209960,0cyclictest13129-21aten2.4-expect21:19:321
217209960,0cyclictest12717-21ssh22:56:201
217209960,0cyclictest10969-21awk21:14:401
217209960,0cyclictest0-21swapper/123:24:381
217209960,0cyclictest0-21swapper/123:08:581
217209960,0cyclictest0-21swapper/122:19:371
217199960,0cyclictest0-21swapper/023:29:050
217199960,0cyclictest0-21swapper/000:05:590
149960,0migration/03772-21sh00:11:300
149960,0migration/028531-21sh21:47:300
149960,0migration/010100-21sh20:54:470
339950,0migration/37227-21kthreadcore21:14:343
339950,0migration/332736-21kthreadcore23:29:363
339950,0migration/326008-21irqrtprio19:59:373
339950,0migration/321836-21irqrtprio21:04:343
339950,0migration/31637-21kthreadcore22:29:383
339950,0migration/313925-21kthreadcore19:24:363
339950,0migration/313749-21kthreadcore19:49:343
339950,0migration/310887-21kernelversion18:59:363
339950,0migration/310398-21kthreadcore20:09:393
28415050,0irq/27-eth0-tx-6704-21munin-node20:34:330
28415050,0irq/27-eth0-tx-10556-21munin-node19:24:330
28415050,0irq/27-eth0-tx-0-21swapper/020:29:330
28405050,0irq/26-eth0-rx-8620-21kthreadcore22:14:373
28405050,0irq/26-eth0-rx-7418-21grep21:54:373
28405050,0irq/26-eth0-rx-7215-21cut19:19:353
28405050,0irq/26-eth0-rx-32230-21kthreadcore21:49:363
28405050,0irq/26-eth0-rx-28205-21ssh22:06:083
28405050,0irq/26-eth0-rx-25021-21kthreadcore21:44:403
28405050,0irq/26-eth0-rx-20889-21sh23:21:073
28405050,0irq/26-eth0-rx-1899-21kthreadcore00:09:383
28405050,0irq/26-eth0-rx-15726-21cut23:39:403
28405050,0irq/26-eth0-rx-14285-21cut21:39:343
28405050,0irq/26-eth0-rx-11800-21sort22:54:413
28405050,0irq/26-eth0-rx-0-21swapper/323:54:323
28405050,0irq/26-eth0-rx-0-21swapper/322:59:193
28405050,0irq/26-eth0-rx-0-21swapper/322:36:123
28405050,0irq/26-eth0-rx-0-21swapper/321:19:233
28405050,0irq/26-eth0-rx-0-21swapper/320:54:183
28405050,0irq/26-eth0-rx-0-21swapper/320:52:223
28405050,0irq/26-eth0-rx-0-21swapper/320:47:203
28405050,0irq/26-eth0-rx-0-21swapper/320:39:363
28405050,0irq/26-eth0-rx-0-21swapper/320:29:313
217209951,0cyclictest9739-21aten_r1power_po18:59:311
217209951,0cyclictest5398-21grep19:44:311
217209951,0cyclictest29867-21ssh22:29:241
217209951,0cyclictest29098-21aten_r1power_po21:09:321
217209951,0cyclictest25734-21munin-node20:24:331
217209951,0cyclictest21605-21diskmemload22:04:331
217209951,0cyclictest0-21swapper/120:54:311
217209951,0cyclictest0-21swapper/119:14:181
217209950,0cyclictest7121-21fschecks_count20:34:321
217209950,0cyclictest6218-21ssh23:34:201
217209950,0cyclictest5551-21telnet22:14:311
217209950,0cyclictest5463-21awk21:54:321
217209950,0cyclictest32349-21cat20:29:181
217209950,0cyclictest30345-21chrt23:09:301
217209950,0cyclictest30156-21aten2.4-expect23:49:301
217209950,0cyclictest30156-21aten2.4-expect23:49:301
217209950,0cyclictest29727-21sh22:09:311
217209950,0cyclictest29148-21date21:29:181
217209950,0cyclictest2864-21munin-node18:54:341
217209950,0cyclictest28616-21munin-node18:49:311
217209950,0cyclictest27829-21date20:49:181
217209950,0cyclictest23960-21aten_r1power_fr19:34:311
217209950,0cyclictest22415-21cut23:44:331
217209950,0cyclictest22339-21entropy18:44:321
217209950,0cyclictest22110-21sh22:44:271
217209950,0cyclictest21071-21sed21:24:181
217209950,0cyclictest17288-21aten_r1power_en19:29:291
217209950,0cyclictest16459-21cpu19:04:311
217209950,0cyclictest14400-21stty23:39:341
217209950,0cyclictest12152-21sed20:59:181
217209950,0cyclictest11740-21cpuspeed_turbos19:49:301
217199955,0cyclictest0-21swapper/023:59:310
217199955,0cyclictest0-21swapper/020:53:380
217199950,0cyclictest0-21swapper/023:21:220
217199950,0cyclictest0-21swapper/023:00:330
217199950,0cyclictest0-21swapper/022:30:450
217199950,0cyclictest0-21swapper/022:22:390
149950,0migration/09870-21cpuspeed_turbos18:59:320
149950,0migration/09552-21kthreadcore22:14:370
149950,0migration/07682-21kthreadcore23:14:360
149950,0migration/07638-21kthreadcore22:54:350
149950,0migration/07138-21kthreadcore23:54:360
149950,0migration/05437-21kthreadcore19:19:340
149950,0migration/05201-21taskset21:54:250
149950,0migration/05180-21taskset20:05:110
149950,0migration/03730-21sendmail_mailqu23:09:400
149950,0migration/03298-21needreboot22:49:390
149950,0migration/031382-21kthreadcore23:49:350
149950,0migration/031382-21kthreadcore23:49:350
149950,0migration/030491-21latency_hist19:39:180
149950,0migration/028797-21sh23:05:570
149950,0migration/028615-21taskset18:49:300
149950,0migration/027531-21kthreadcore19:09:380
149950,0migration/026898-21irqrtprio20:24:380
149950,0migration/026026-21kthreadcore21:24:360
149950,0migration/023535-21needreboot19:54:370
149950,0migration/022203-21munin-node22:44:280
149950,0migration/019358-21kthreadcore19:29:350
149950,0migration/018382-21kthreadcore21:59:340
149950,0migration/018130-21kthreadcore23:39:410
149950,0migration/014792-21kthreadcore20:14:360
149950,0migration/013116-21latency_hist21:39:180
149950,0migration/01088-21kthreadcore23:29:370
339940,0migration/36839-21kthreadcore21:34:373
339940,0migration/3506-21kthreadcore22:49:373
339940,0migration/329263-21needreboot19:34:383
339940,0migration/327823-21pmu-power23:04:393
339940,0migration/32669-21kthreadcore19:39:383
339940,0migration/325038-21kthreadcore23:24:383
339940,0migration/324444-21kthreadcore18:44:363
339940,0migration/312284-21proc_pri20:34:393
28415040,0irq/27-eth0-tx-0-21swapper/020:09:380
28405040,0irq/26-eth0-rx-26873-21munin-node21:24:403
28405040,0irq/26-eth0-rx-0-21swapper/323:44:323
28405040,0irq/26-eth0-rx-0-21swapper/321:59:433
28405040,0irq/26-eth0-rx-0-21swapper/321:09:383
28405040,0irq/26-eth0-rx-0-21swapper/320:59:343
28405040,0irq/26-eth0-rx-0-21swapper/319:54:313
28405040,0irq/26-eth0-rx-0-21swapper/319:10:553
28405040,0irq/26-eth0-rx-0-21swapper/319:04:183
28405040,0irq/26-eth0-rx-0-21swapper/318:52:183
28405040,0irq/26-eth0-rx-0-21swapper/300:04:433
28405040,0irq/26-eth0-rx-0-21swapper/300:02:373
217199944,0cyclictest0-21swapper/021:37:280
217199944,0cyclictest0-21swapper/021:04:120
217199943,0cyclictest6445-21snmpd23:37:430
217199943,0cyclictest5827-21awk21:14:320
217199943,0cyclictest5758-21awk19:44:330
217199943,0cyclictest3425-21lldpd22:27:340
217199943,0cyclictest30736-21ps19:14:360
217199943,0cyclictest30349-21awk21:49:330
217199943,0cyclictest20998-21awk20:44:350
217199943,0cyclictest14741-21awk22:39:340
217199943,0cyclictest14329-21awk20:39:340
217199943,0cyclictest12469-21awk19:49:330
149940,0migration/028059-21kthreadcore19:34:370
149940,0migration/025407-21kthreadcore21:04:360
149940,0migration/02403-21ntp_offset21:29:380
339930,0migration/32083-21kthreadcore20:04:363
28405030,0irq/26-eth0-rx-0-21swapper/320:24:293
28405030,0irq/26-eth0-rx-0-21swapper/320:19:333
28405030,0irq/26-eth0-rx-0-21swapper/320:14:313
28405030,0irq/26-eth0-rx-0-21swapper/319:31:463
28405030,0irq/26-eth0-rx-0-21swapper/319:18:053
28405030,0irq/26-eth0-rx-0-21swapper/318:54:303
217199933,0cyclictest0-21swapper/022:08:410
217199932,0cyclictest30592-21awk22:09:350
217199932,0cyclictest2376-21sensors21:09:430
217199932,0cyclictest22771-21awk18:44:340
217199932,0cyclictest22565-21forks23:44:340
217199932,0cyclictest13962-21awk21:19:360
28405020,0irq/26-eth0-rx-0-21swapper/319:44:473
217199920,0cyclictest5394-21latency_hist22:34:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional