You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 11:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Jul 12, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350780,0irq/25-eth10-21swapper/318:44:163
2886723430,0sleep10-21swapper/118:42:211
2885223324,0sleep20-21swapper/218:42:142
2882722824,0sleep00-21swapper/018:41:560
291259976,0cyclictest0-21swapper/021:34:480
339960,0migration/330588-21sh00:08:513
339960,0migration/328476-21munin-node21:04:493
339960,0migration/324988-21sh22:42:413
339960,0migration/324568-21sh22:20:063
339960,0migration/317037-21pmu-power20:54:593
339960,0migration/315672-21sh23:16:013
339960,0migration/31340-21sh21:46:143
291309960,0cyclictest0-21swapper/222:19:592
291269966,0cyclictest0-21swapper/123:20:341
291269960,0cyclictest0-21swapper/122:56:261
28415060,0irq/27-eth0-tx-25802-21ssh21:00:162
28415060,0irq/27-eth0-tx-25456-21munin-node19:54:530
28415060,0irq/27-eth0-tx-19742-21munin-node20:14:550
28415060,0irq/27-eth0-tx-16206-21sensors22:14:592
28415060,0irq/27-eth0-tx-0-21swapper/223:52:022
28415060,0irq/27-eth0-tx-0-21swapper/223:15:322
28415060,0irq/27-eth0-tx-0-21swapper/222:53:112
28415060,0irq/27-eth0-tx-0-21swapper/222:40:562
28415060,0irq/27-eth0-tx-0-21swapper/222:37:142
28415060,0irq/27-eth0-tx-0-21swapper/221:50:462
28415060,0irq/27-eth0-tx-0-21swapper/221:39:402
28415060,0irq/27-eth0-tx-0-21swapper/221:28:412
28415060,0irq/27-eth0-tx-0-21swapper/221:09:062
27160,0ktimersoftd/20-21swapper/222:24:532
249960,0watchdog/20-21swapper/219:24:532
149960,0migration/032207-21munin-node19:34:520
149960,0migration/024590-21munin-node21:39:530
149960,0migration/02028-21kthreadcore23:49:530
149960,0migration/016358-21sh22:56:520
339950,0migration/39489-21latency_hist22:54:343
339950,0migration/35932-21kthreadcore19:14:553
339950,0migration/35463-21mdstat23:49:543
339950,0migration/332552-21taskset20:21:503
339950,0migration/332328-21irqcore19:34:523
339950,0migration/32839-21kthreadcore19:09:523
339950,0migration/327549-21kthreadcore19:54:543
339950,0migration/32739-21kthreadcore00:09:523
339950,0migration/325828-21kthreadcore19:04:513
339950,0migration/325771-21munin-node23:24:503
339950,0migration/321280-21kthreadcore21:39:523
339950,0migration/319731-21taskset20:14:383
339950,0migration/31530-21kthreadcore18:44:513
339950,0migration/314191-21kthreadcore19:44:513
339950,0migration/312080-21rt-features20:04:553
291309955,0cyclictest0-21swapper/223:33:562
291309955,0cyclictest0-21swapper/220:10:052
291309955,0cyclictest0-21swapper/219:59:502
291309950,0cyclictest0-21swapper/221:19:362
291269955,0cyclictest18-21rcuc/123:57:461
291269955,0cyclictest0-21swapper/123:17:531
291269955,0cyclictest0-21swapper/122:08:501
291269955,0cyclictest0-21swapper/121:52:261
291269955,0cyclictest0-21swapper/120:48:061
291269950,0cyclictest0-21swapper/122:34:031
291269950,0cyclictest0-21swapper/121:44:051
291269950,0cyclictest0-21swapper/121:32:251
291269950,0cyclictest0-21swapper/100:03:081
291259955,0cyclictest0-21swapper/020:56:290
291259950,0cyclictest26449-21ssh22:04:260
28415050,0irq/27-eth0-tx-3643-21munin-node18:49:490
28415050,0irq/27-eth0-tx-32112-21ssh23:06:222
28415050,0irq/27-eth0-tx-17255-21ssh21:55:482
28415050,0irq/27-eth0-tx-12172-21munin-node19:44:570
28415050,0irq/27-eth0-tx-11242-21munin-node19:19:570
28415050,0irq/27-eth0-tx-0-21swapper/223:56:112
28415050,0irq/27-eth0-tx-0-21swapper/223:37:092
28415050,0irq/27-eth0-tx-0-21swapper/223:28:432
28415050,0irq/27-eth0-tx-0-21swapper/223:20:162
28415050,0irq/27-eth0-tx-0-21swapper/223:11:302
28415050,0irq/27-eth0-tx-0-21swapper/223:11:302
28415050,0irq/27-eth0-tx-0-21swapper/223:00:022
28415050,0irq/27-eth0-tx-0-21swapper/222:55:212
28415050,0irq/27-eth0-tx-0-21swapper/222:04:372
28415050,0irq/27-eth0-tx-0-21swapper/222:03:012
28415050,0irq/27-eth0-tx-0-21swapper/221:45:522
28415050,0irq/27-eth0-tx-0-21swapper/221:10:002
28415050,0irq/27-eth0-tx-0-21swapper/220:55:002
28415050,0irq/27-eth0-tx-0-21swapper/220:52:542
28415050,0irq/27-eth0-tx-0-21swapper/220:45:352
28415050,0irq/27-eth0-tx-0-21swapper/200:11:582
28415050,0irq/27-eth0-tx-0-21swapper/200:05:482
28415050,0irq/27-eth0-tx-0-21swapper/200:02:532
28415050,0irq/27-eth0-tx-0-21swapper/020:44:440
28405050,0irq/26-eth0-rx-5849-21kthreadcore22:29:523
28405050,0irq/26-eth0-rx-28655-21kthreadcore22:04:513
28405050,0irq/26-eth0-rx-21616-21sed23:59:583
28405050,0irq/26-eth0-rx-15722-21tr21:54:523
28405050,0irq/26-eth0-rx-0-21swapper/323:58:553
28405050,0irq/26-eth0-rx-0-21swapper/323:44:373
28405050,0irq/26-eth0-rx-0-21swapper/323:34:563
28405050,0irq/26-eth0-rx-0-21swapper/323:29:403
28405050,0irq/26-eth0-rx-0-21swapper/323:19:343
28405050,0irq/26-eth0-rx-0-21swapper/322:59:513
28405050,0irq/26-eth0-rx-0-21swapper/322:46:043
28405050,0irq/26-eth0-rx-0-21swapper/322:24:533
28405050,0irq/26-eth0-rx-0-21swapper/322:17:513
28405050,0irq/26-eth0-rx-0-21swapper/321:52:153
28405050,0irq/26-eth0-rx-0-21swapper/321:27:023
28405050,0irq/26-eth0-rx-0-21swapper/321:19:413
28405050,0irq/26-eth0-rx-0-21swapper/320:49:493
28405050,0irq/26-eth0-rx-0-21swapper/320:44:533
259950,0migration/2488-21sh22:47:272
259950,0migration/229101-21kthreadcore20:19:522
259950,0migration/220528-21hwlatdetect20:14:502
249950,0watchdog/20-21swapper/223:40:122
249950,0watchdog/20-21swapper/221:32:482
249950,0watchdog/20-21swapper/220:32:212
179950,0migration/125381-21kthreadcore19:04:511
149950,0migration/09657-21kthreadcore19:39:530
149950,0migration/08533-21irqcore20:29:500
149950,0migration/08216-21sendmail_mailqu22:09:560
149950,0migration/08144-21kthreadcore21:09:540
149950,0migration/07477-21sh23:10:250
149950,0migration/07477-21sh23:10:250
149950,0migration/0538-21latency_hist20:24:340
149950,0migration/04310-21kthreadcore22:49:520
149950,0migration/030816-21kthreadcore23:04:520
149950,0migration/030676-21munin-node23:24:550
149950,0migration/027893-21cpuspeed_turbos21:24:500
149950,0migration/027187-21irqrtprio22:44:540
149950,0migration/025349-21sh22:43:440
149950,0migration/02417-21fschecks_count22:29:480
149950,0migration/018740-21kthreadcore23:19:500
149950,0migration/018525-21kthreadcore18:59:520
149950,0migration/01752-21kthreadcore19:59:510
149950,0migration/015975-21munin-node22:14:570
149950,0migration/015817-21aten_r1power_fr23:59:460
149950,0migration/015413-21kthreadcore20:34:530
149950,0migration/011096-21latency_hist19:19:340
149950,0migration/010777-21kthreadcore23:14:500
8225040,0irq/24-0000:00:5278-21latency_hist19:39:342
339940,0migration/36508-21timerandwakeup20:24:573
339940,0migration/321577-21kthreadcore18:59:533
339940,0migration/31727-21kthreadcore19:59:513
339940,0migration/315512-21kthreadcore20:09:523
339940,0migration/313973-21kthreadcore21:34:533
339940,0migration/312144-21kthreadcore20:29:513
339940,0migration/310853-21fschecks_count18:54:463
291309943,0cyclictest6445-21snmpd19:44:542
291309943,0cyclictest6445-21snmpd19:01:242
291309943,0cyclictest4600-21awk18:49:512
291309943,0cyclictest3425-21lldpd22:09:502
291309943,0cyclictest3425-21lldpd20:04:422
291309943,0cyclictest31227-21awk19:09:502
291309943,0cyclictest3110-21irqbalance19:54:482
291309943,0cyclictest3110-21irqbalance19:34:382
291309943,0cyclictest3110-21irqbalance19:29:382
291269944,0cyclictest0-21swapper/122:45:241
291269944,0cyclictest0-21swapper/121:55:071
291269944,0cyclictest0-21swapper/121:21:031
291269943,0cyclictest843-21awk23:49:511
291269943,0cyclictest6445-21snmpd22:27:551
291269943,0cyclictest6445-21snmpd19:30:541
291269943,0cyclictest6433-21awk19:39:511
291269943,0cyclictest4148-21awk20:49:511
291269943,0cyclictest3425-21lldpd22:09:501
291269943,0cyclictest30292-21awk18:44:491
291269943,0cyclictest28044-21awk21:44:501
291269943,0cyclictest27373-21awk20:19:511
291269943,0cyclictest25456-21awk23:44:501
291269943,0cyclictest21590-21awk20:39:491
291269943,0cyclictest18928-21awk22:39:491
291269943,0cyclictest17642-21awk23:39:491
291269943,0cyclictest1596-21awk23:29:501
291269942,0cyclictest4510-21awk21:09:521
291269942,0cyclictest27595-21sed21:04:341
291269942,0cyclictest11741-21df_abs20:54:511
291269941,0cyclictest0-21swapper/120:24:521
291259944,0cyclictest0-21swapper/023:37:180
291259944,0cyclictest0-21swapper/021:16:310
291259944,0cyclictest0-21swapper/021:09:250
291259943,0cyclictest6445-21snmpd22:37:550
291259943,0cyclictest6445-21snmpd21:56:250
291259943,0cyclictest6445-21snmpd20:04:540
291259943,0cyclictest6445-21snmpd19:31:240
291259943,0cyclictest3801-21awk21:49:530
291259943,0cyclictest30268-21awk18:44:490
291259943,0cyclictest21620-21awk20:39:490
291259943,0cyclictest20525-21awk20:59:520
291259943,0cyclictest18485-21awk22:59:480
28415040,0irq/27-eth0-tx-0-21swapper/223:45:302
28415040,0irq/27-eth0-tx-0-21swapper/222:29:482
28415040,0irq/27-eth0-tx-0-21swapper/020:09:550
28405040,0irq/26-eth0-rx-0-21swapper/323:39:543
28405040,0irq/26-eth0-rx-0-21swapper/323:13:123
28405040,0irq/26-eth0-rx-0-21swapper/323:13:123
28405040,0irq/26-eth0-rx-0-21swapper/323:05:333
28405040,0irq/26-eth0-rx-0-21swapper/322:36:593
28405040,0irq/26-eth0-rx-0-21swapper/322:13:103
28405040,0irq/26-eth0-rx-0-21swapper/321:59:453
28405040,0irq/26-eth0-rx-0-21swapper/321:14:343
28405040,0irq/26-eth0-rx-0-21swapper/321:11:243
28405040,0irq/26-eth0-rx-0-21swapper/320:59:413
259940,0migration/24574-21kthreadcore20:24:522
249940,0watchdog/228994-21head19:04:532
249940,0watchdog/212266-21cut21:14:492
249940,0watchdog/211565-21irqrtprio18:54:492
249940,0watchdog/20-21swapper/221:34:442
249940,0watchdog/20-21swapper/220:39:492
249940,0watchdog/20-21swapper/220:34:532
249940,0watchdog/20-21swapper/219:49:412
249940,0watchdog/20-21swapper/219:19:532
179940,0migration/18239-21kthreadcore18:49:531
179940,0migration/16649-21kthreadcore19:14:551
179940,0migration/16606-21munin-node23:09:571
179940,0migration/16606-21munin-node23:09:571
179940,0migration/131234-21aten_r1power_en19:34:471
179940,0migration/130412-21aten_r1power_vo19:09:471
179940,0migration/130104-21kthreadcore23:24:541
179940,0migration/117361-21taskset22:17:211
179940,0migration/114503-21kthreadcore18:54:501
179940,0migration/114296-21kthreadcore21:14:521
179940,0migration/112432-21irqrtprio19:19:541
179940,0migration/111818-21kthreadcore23:34:531
149940,0migration/026960-21fschecks_time20:19:490
149940,0migration/024049-21forks19:04:470
149940,0migration/020895-21irqrtprio21:19:500
149940,0migration/019006-21kthreadcore23:39:510
149940,0migration/015372-21runrttasks23:59:210
291309930,0cyclictest29733-21diskstats18:44:462
291269932,0cyclictest8254-21awk20:29:481
291269932,0cyclictest7341-21awk20:04:491
291269932,0cyclictest31325-21tr00:09:471
291269932,0cyclictest27956-21df21:24:501
291269932,0cyclictest23810-21diskstats00:04:481
291269932,0cyclictest21659-21kthreadcore22:59:511
291269932,0cyclictest21339-21kthreadcore20:14:531
291269932,0cyclictest20260-21perl20:59:501
291269932,0cyclictest19065-21tr19:24:521
291269932,0cyclictest18991-21tune2fs22:19:481
291269932,0cyclictest1810-21cut19:59:511
291269932,0cyclictest16903-21latency_hist18:59:341
291269932,0cyclictest13980-21latency_hist20:34:341
291269932,0cyclictest13833-21grep20:09:491
291269932,0cyclictest12191-21awk21:34:521
291269931,0cyclictest4311-21kthreadcore22:49:521
291269931,0cyclictest15259-21kthreadcore19:44:521
291269931,0cyclictest10621-21perl22:34:471
291269930,0cyclictest19154-21df19:49:461
291259933,0cyclictest0-21swapper/020:52:040
291259932,0cyclictest6445-21snmpd22:24:460
291259932,0cyclictest6028-21basename00:14:330
291259932,0cyclictest30342-21kthreadcore21:44:530
291259932,0cyclictest30272-21munin-node19:09:470
291259932,0cyclictest19385-21awk22:19:500
291259930,0cyclictest28947-21grep22:04:510
291259930,0cyclictest24020-21perl00:04:490
291259930,0cyclictest0-21swapper/019:49:480
28415030,0irq/27-eth0-tx-0-21swapper/019:24:470
28405030,0irq/26-eth0-rx-0-21swapper/322:49:363
28405030,0irq/26-eth0-rx-0-21swapper/321:30:013
28405030,0irq/26-eth0-rx-0-21swapper/320:39:453
28405030,0irq/26-eth0-rx-0-21swapper/320:34:483
28405030,0irq/26-eth0-rx-0-21swapper/319:50:313
28405030,0irq/26-eth0-rx-0-21swapper/319:26:363
28405030,0irq/26-eth0-rx-0-21swapper/319:19:483
249930,0watchdog/20-21swapper/219:15:052
169930,0watchdog/127145-21diskmemload23:04:481
169930,0watchdog/127145-21diskmemload21:59:481
329920,0watchdog/30-21swapper/319:39:363
291269921,0cyclictest25369-21ls19:54:341
291259921,0cyclictest10247-21cat18:54:340
291259920,0cyclictest6445-21snmpd21:29:360
291259920,0cyclictest529-21sed23:29:340
291259920,0cyclictest24561-21ssh23:44:370
28405020,0irq/26-eth0-rx-0-21swapper/319:29:553
28405020,0irq/26-eth0-rx-0-21swapper/318:49:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional