You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 02:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Dec 27, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450670,0irq/25-eth10-21swapper/319:04:223
993123925,0sleep00-21swapper/019:05:580
1008123523,0sleep10-21swapper/119:07:471
991123425,0sleep20-21swapper/219:05:482
339960,0migration/34374-21sh00:15:183
339960,0migration/332322-21sh21:19:513
339960,0migration/329751-21sh22:17:183
339960,0migration/329192-21munin-node23:53:553
339960,0migration/325520-21sh23:12:463
339960,0migration/321724-21sh00:26:253
339960,0migration/317547-21latency_hist23:08:413
339960,0migration/315674-21fschecks_count22:28:563
29715060,0irq/26-eth0-rx-9131-21iostat_ios22:44:003
29715060,0irq/26-eth0-rx-3751-21kthreadcore21:23:583
29715060,0irq/26-eth0-rx-25879-21grep00:28:593
29715060,0irq/26-eth0-rx-21675-21perl21:53:553
149960,0migration/026676-21sendmail_mailqu20:54:040
149960,0migration/015649-21sh23:04:190
102339960,0cyclictest0-21swapper/123:49:051
102339960,0cyclictest0-21swapper/100:35:361
102329960,0cyclictest0-21swapper/023:15:430
102329960,0cyclictest0-21swapper/021:41:450
102329960,0cyclictest0-21swapper/000:35:100
8215050,0irq/24-0000:00:0-21swapper/223:08:412
339950,0migration/39262-21kthreadcore23:58:583
339950,0migration/37596-21kthreadcore21:43:593
339950,0migration/37107-21kthreadcore21:04:003
339950,0migration/332087-21fschecks_time00:33:533
339950,0migration/331484-21munin-node19:23:573
339950,0migration/326943-21kthreadcore22:53:583
339950,0migration/324395-21fschecks_time19:18:553
339950,0migration/321279-21kthreadcore22:48:593
339950,0migration/319133-21kthreadcore21:13:583
339950,0migration/315137-21kthreadcore19:33:583
339950,0migration/314506-21munin-node20:48:543
339950,0migration/313695-21diskmemload00:22:193
339950,0migration/313446-21munin-node19:58:583
339950,0migration/31343-21munin-node20:38:553
339950,0migration/312866-21kthreadcore20:44:003
339950,0migration/312639-21aten_r1power_po21:48:513
339950,0migration/311739-21munin-node23:23:573
339950,0migration/310309-21pmu-power19:29:013
29725050,0irq/27-eth0-tx-31137-21ssh22:37:282
29725050,0irq/27-eth0-tx-30476-21munin-node19:23:572
29725050,0irq/27-eth0-tx-27544-21munin-node23:33:512
29725050,0irq/27-eth0-tx-24899-21munin-node22:54:032
29725050,0irq/27-eth0-tx-22847-21ssh22:49:322
29725050,0irq/27-eth0-tx-1945-21iostat_ios22:58:582
29725050,0irq/27-eth0-tx-18374-21ssh21:13:582
29725050,0irq/27-eth0-tx-17786-21ssh23:26:102
29725050,0irq/27-eth0-tx-14992-21ssh22:45:552
29725050,0irq/27-eth0-tx-0-21swapper/222:00:352
29725050,0irq/27-eth0-tx-0-21swapper/221:37:452
29725050,0irq/27-eth0-tx-0-21swapper/221:19:192
29715050,0irq/26-eth0-rx-29357-21python22:34:043
29715050,0irq/26-eth0-rx-26135-21seq23:31:013
29715050,0irq/26-eth0-rx-2379-21munin-node23:18:513
29715050,0irq/26-eth0-rx-21446-21cut21:34:013
29715050,0irq/26-eth0-rx-10963-21grep21:09:003
29715050,0irq/26-eth0-rx-0-21swapper/323:13:583
29715050,0irq/26-eth0-rx-0-21swapper/323:02:033
29715050,0irq/26-eth0-rx-0-21swapper/322:19:003
29715050,0irq/26-eth0-rx-0-21swapper/322:03:573
29715050,0irq/26-eth0-rx-0-21swapper/321:59:583
29715050,0irq/26-eth0-rx-0-21swapper/321:38:573
29715050,0irq/26-eth0-rx-0-21swapper/319:48:533
29715050,0irq/26-eth0-rx-0-21swapper/300:03:523
259950,0migration/29007-21cpuspeed_turbos21:08:562
259950,0migration/28778-21kthreadcore22:23:592
259950,0migration/24698-21sh00:16:032
259950,0migration/23286-21sh21:42:272
259950,0migration/232072-21sh23:14:042
259950,0migration/226023-21sh23:30:462
259950,0migration/217412-21sh21:29:582
259950,0migration/214376-21latency_hist20:48:422
259950,0migration/21277-21df_abs20:38:542
249950,0watchdog/27912-21ntp_states23:18:592
249950,0watchdog/26219-21aten_r1power_po00:18:512
249950,0watchdog/227635-21kthreadcore00:28:592
249950,0watchdog/223393-21kthreadcore23:48:592
249950,0watchdog/220847-21ssh00:24:192
249950,0watchdog/20-21swapper/222:08:442
179950,0migration/19553-21idleruntime21:08:581
179950,0migration/15897-21kthreadcore21:04:001
179950,0migration/11737-21idleruntime22:58:571
179950,0migration/112561-21aten2.4-expect19:58:541
169950,0watchdog/118625-21tr20:49:001
169950,0watchdog/116026-21kthreadcore21:29:001
149950,0migration/08322-21kthreadcore20:18:580
149950,0migration/0603-21kthreadcore21:58:590
149950,0migration/05883-21taskset22:39:170
149950,0migration/05782-21kthreadcore22:59:000
149950,0migration/04245-21kthreadcore20:13:590
149950,0migration/031161-21fschecks_count22:18:580
149950,0migration/030109-21kernelversion23:53:580
149950,0migration/029049-21sh22:15:390
149950,0migration/028559-21kthreadcore22:34:000
149950,0migration/028448-21kthreadcore20:33:580
149950,0migration/028414-21kthreadcore19:18:590
149950,0migration/026042-21kthreadcore00:28:590
149950,0migration/025521-21gltestperf19:43:580
149950,0migration/022775-21kthreadcore21:53:580
149950,0migration/021135-21kthreadcore22:48:590
149950,0migration/02048-21kthreadcore19:48:580
149950,0migration/018725-21taskset20:01:510
149950,0migration/018393-21kernelversion21:13:580
149950,0migration/018208-21irqcore19:13:570
149950,0migration/016403-21pmu-power23:24:020
149950,0migration/015983-21kthreadcore00:23:580
149950,0migration/013553-21sh22:08:400
149950,0migration/013186-21latency_hist00:03:410
149950,0migration/012730-21kthreadcore19:08:590
149950,0migration/011312-21sh21:46:290
149950,0migration/010976-21sh23:42:000
149950,0migration/010862-21kthreadcore21:09:000
102349950,0cyclictest0-21swapper/200:10:542
102339955,0cyclictest0-21swapper/123:09:021
102339955,0cyclictest0-21swapper/123:08:151
102339955,0cyclictest0-21swapper/122:41:551
102339955,0cyclictest0-21swapper/122:20:191
102339950,0cyclictest0-21swapper/122:32:181
102339950,0cyclictest0-21swapper/121:49:351
102329955,0cyclictest0-21swapper/022:43:520
102329953,0cyclictest9394-21kworker/0:221:28:510
102329953,0cyclictest9394-21kworker/0:221:22:310
102329953,0cyclictest14647-21kworker/0:223:12:410
102329953,0cyclictest14647-21kworker/0:222:53:590
102329953,0cyclictest14647-21kworker/0:200:02:000
102329952,0cyclictest9394-21kworker/0:219:53:590
102329952,0cyclictest14647-21kworker/0:223:28:590
102329952,0cyclictest14647-21kworker/0:223:21:370
102329952,0cyclictest14647-21kworker/0:222:29:010
8215040,0irq/24-0000:00:0-21swapper/220:43:422
339940,0migration/330418-21sensors_fan19:44:053
339940,0migration/328026-21fschecks_count23:33:523
339940,0migration/324435-21kthreadcore20:54:003
339940,0migration/32377-21kthreadcore20:13:593
339940,0migration/322065-21kthreadcore20:03:573
339940,0migration/321538-21kthreadcore19:38:593
339940,0migration/317207-21kthreadcore20:23:593
339940,0migration/310535-21aten_r1power_fr19:08:543
29725040,0irq/27-eth0-tx-4109-21ssh21:43:472
29725040,0irq/27-eth0-tx-30624-21ssh22:18:512
29725040,0irq/27-eth0-tx-22262-21ssh22:13:502
29725040,0irq/27-eth0-tx-1944-21munin-node21:03:562
29725040,0irq/27-eth0-tx-13269-21munin-node00:03:582
29725040,0irq/27-eth0-tx-0-21swapper/223:54:092
29725040,0irq/27-eth0-tx-0-21swapper/223:38:542
29725040,0irq/27-eth0-tx-0-21swapper/223:09:302
29725040,0irq/27-eth0-tx-0-21swapper/222:28:572
29725040,0irq/27-eth0-tx-0-21swapper/222:03:502
29725040,0irq/27-eth0-tx-0-21swapper/220:29:042
29725040,0irq/27-eth0-tx-0-21swapper/220:13:532
29715040,0irq/26-eth0-rx-0-21swapper/323:48:593
29715040,0irq/26-eth0-rx-0-21swapper/323:46:133
29715040,0irq/26-eth0-rx-0-21swapper/323:38:563
29715040,0irq/26-eth0-rx-0-21swapper/322:38:573
29715040,0irq/26-eth0-rx-0-21swapper/322:23:553
29715040,0irq/26-eth0-rx-0-21swapper/322:09:163
29715040,0irq/26-eth0-rx-0-21swapper/321:28:423
29715040,0irq/26-eth0-rx-0-21swapper/300:08:593
259940,0migration/25173-21df_abs23:58:532
259940,0migration/23395-21kernelversion21:23:582
259940,0migration/227331-21fschecks_time20:33:552
259940,0migration/226419-21kthreadcore19:44:002
259940,0migration/221033-21aten_r1power_en21:53:522
259940,0migration/218095-21sendmail_mailqu19:59:032
259940,0migration/212632-21fschecks_time23:43:552
249940,0watchdog/26776-21aten_r1power_cu20:18:522
249940,0watchdog/24380-21missed_timers19:49:002
249940,0watchdog/214247-21kthreadcore21:48:562
249940,0watchdog/20-21swapper/220:23:562
249940,0watchdog/20-21swapper/219:33:522
249940,0watchdog/20-21swapper/219:19:042
179940,0migration/19689-21kthreadcore20:43:591
179940,0migration/18131-21diskmemload23:19:011
179940,0migration/16672-21runrttasks22:23:461
179940,0migration/14705-21munin-node19:49:041
179940,0migration/13395-21sh23:56:081
179940,0migration/132334-21kthreadcore21:38:581
179940,0migration/129721-21kthreadcore23:33:551
179940,0migration/125323-21kthreadcore20:54:001
179940,0migration/124689-21latency_hist19:43:431
179940,0migration/122433-21aten_r1power_vo22:13:541
179940,0migration/119152-21aten2.4-expect20:03:511
179940,0migration/117216-21aten_r1power_fr19:13:531
179940,0migration/115111-21irqrtprio22:08:571
169940,0watchdog/1690-21kthreadcore00:13:591
169940,0watchdog/131342-21munin-node21:19:001
169940,0watchdog/129612-21timerandwakeup19:19:041
169940,0watchdog/12930-21awk21:23:561
169940,0watchdog/128407-21ssh21:57:081
169940,0watchdog/12764-21python21:59:041
169940,0watchdog/127634-21kthreadcore00:28:591
169940,0watchdog/126718-21sort22:34:001
169940,0watchdog/116566-21python23:24:031
169940,0watchdog/112438-21awk19:33:561
169940,0watchdog/11243-21ssh23:16:591
169940,0watchdog/10-21swapper/123:40:351
169940,0watchdog/10-21swapper/123:30:151
169940,0watchdog/10-21swapper/120:18:521
169940,0watchdog/10-21swapper/100:06:391
149940,0migration/030956-21kthreadcore20:08:580
149940,0migration/022489-21kthreadcore19:39:000
102349943,0cyclictest394-21awk22:38:572
102349943,0cyclictest32243-21awk00:33:542
102349943,0cyclictest18970-21awk19:38:572
102339943,0cyclictest6358-21awk22:03:581
102339943,0cyclictest3556-21lldpd22:45:431
102339943,0cyclictest3514-21snmpd23:44:051
102339943,0cyclictest3514-21snmpd21:43:491
102339943,0cyclictest3514-21snmpd20:40:191
102339943,0cyclictest3514-21snmpd20:36:191
102339943,0cyclictest3514-21snmpd20:28:491
102339943,0cyclictest3514-21snmpd19:42:491
102339943,0cyclictest3514-21snmpd19:27:491
102339943,0cyclictest3514-21snmpd19:11:191
102339943,0cyclictest3241-21irqbalance00:22:291
102339943,0cyclictest26684-21awk20:08:561
102339943,0cyclictest25759-21awk22:53:561
102339943,0cyclictest22678-21awk00:08:581
102339943,0cyclictest20491-21awk21:34:001
102329943,0cyclictest7062-21awk00:18:550
102329943,0cyclictest3556-21lldpd21:52:130
102329943,0cyclictest31479-21iostat19:23:570
102329943,0cyclictest22621-21awk00:08:570
102329943,0cyclictest20995-21awk20:28:580
102329943,0cyclictest14647-21kworker/0:223:49:510
102329942,0cyclictest9394-21kworker/0:220:48:570
102329942,0cyclictest9394-21kworker/0:219:34:220
102329942,0cyclictest14647-21kworker/0:223:46:070
102329942,0cyclictest14647-21kworker/0:200:17:060
102329941,0cyclictest9394-21kworker/0:221:25:210
102329941,0cyclictest9394-21kworker/0:220:58:590
102329941,0cyclictest9394-21kworker/0:220:38:590
102329941,0cyclictest14647-21kworker/0:222:23:590
35130,0ktimersoftd/30-21swapper/320:30:503
29725030,0irq/27-eth0-tx-0-21swapper/219:54:032
29715030,0irq/26-eth0-rx-0-21swapper/320:58:533
29715030,0irq/26-eth0-rx-0-21swapper/320:35:343
29715030,0irq/26-eth0-rx-0-21swapper/320:11:283
29715030,0irq/26-eth0-rx-0-21swapper/319:53:533
29715030,0irq/26-eth0-rx-0-21swapper/319:13:563
249930,0watchdog/20-21swapper/220:58:562
249930,0watchdog/20-21swapper/220:54:002
179930,0migration/17602-21kthreadcore23:58:581
179930,0migration/128467-21sh20:58:561
179930,0migration/120818-21kthreadcore21:13:591
169930,0watchdog/10-21swapper/122:48:521
169930,0watchdog/10-21swapper/119:53:561
169930,0watchdog/10-21swapper/100:24:031
102349932,0cyclictest5525-21sh19:28:552
102349932,0cyclictest26708-21awk20:08:562
102349930,0cyclictest19125-21munin-node20:03:532
102349930,0cyclictest0-21swapper/219:16:142
102339933,0cyclictest3556-21lldpd20:14:421
102339932,0cyclictest3514-21snmpd20:26:191
102339932,0cyclictest3514-21snmpd19:29:491
102329933,0cyclictest9394-21kworker/0:221:03:510
102329933,0cyclictest9394-21kworker/0:220:43:490
102329933,0cyclictest9394-21kworker/0:220:23:450
102329933,0cyclictest9394-21kworker/0:220:03:430
102329933,0cyclictest9394-21kworker/0:219:28:580
102329933,0cyclictest20295-21kworker/0:022:08:450
102329933,0cyclictest20295-21kworker/0:021:33:590
102329933,0cyclictest14647-21kworker/0:223:33:450
7780220,0sleep39212-21kthreadcore20:18:583
102349921,0cyclictest10170-21crond19:08:432
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional