You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 01:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Wed Nov 19, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350710,0irq/25-eth10-21swapper/318:38:283
2153224645,0sleep10-21swapper/118:35:471
2156423727,0sleep20-21swapper/218:36:122
2181423523,0sleep00-21swapper/018:39:070
21943991515,0cyclictest29691-21awk22:05:011
21941991111,0cyclictest30778-21sensors19:55:050
339960,0migration/34485-21aten2.4-expect22:09:563
339960,0migration/328420-21sh22:44:583
339960,0migration/326291-21sh23:43:013
339960,0migration/322874-21needreboot19:25:033
339960,0migration/310617-21munin-node18:54:593
28415060,0irq/27-eth0-tx-0-21swapper/223:40:592
28415060,0irq/27-eth0-tx-0-21swapper/223:36:392
28415060,0irq/27-eth0-tx-0-21swapper/223:30:052
28415060,0irq/27-eth0-tx-0-21swapper/223:25:042
28415060,0irq/27-eth0-tx-0-21swapper/223:15:342
28415060,0irq/27-eth0-tx-0-21swapper/223:09:002
28415060,0irq/27-eth0-tx-0-21swapper/223:04:152
28415060,0irq/27-eth0-tx-0-21swapper/222:59:112
28415060,0irq/27-eth0-tx-0-21swapper/222:33:192
28415060,0irq/27-eth0-tx-0-21swapper/222:26:042
28415060,0irq/27-eth0-tx-0-21swapper/222:21:462
28415060,0irq/27-eth0-tx-0-21swapper/222:10:492
28415060,0irq/27-eth0-tx-0-21swapper/222:06:342
28415060,0irq/27-eth0-tx-0-21swapper/221:59:432
28415060,0irq/27-eth0-tx-0-21swapper/221:51:092
28415060,0irq/27-eth0-tx-0-21swapper/221:48:532
28415060,0irq/27-eth0-tx-0-21swapper/221:39:532
28415060,0irq/27-eth0-tx-0-21swapper/221:34:552
28415060,0irq/27-eth0-tx-0-21swapper/221:17:262
28415060,0irq/27-eth0-tx-0-21swapper/220:57:412
28415060,0irq/27-eth0-tx-0-21swapper/220:49:452
28415060,0irq/27-eth0-tx-0-21swapper/220:42:182
28415060,0irq/27-eth0-tx-0-21swapper/200:06:352
28415060,0irq/27-eth0-tx-0-21swapper/200:00:362
28405060,0irq/26-eth0-rx-6445-21snmpd22:58:143
219439966,0cyclictest20263-21diskmemload20:42:381
219419960,0cyclictest0-21swapper/022:55:400
219419960,0cyclictest0-21swapper/000:05:310
169960,0watchdog/121859-21needreboot19:00:061
169960,0watchdog/10-21swapper/122:16:141
149960,0migration/012820-21crond21:14:430
8225050,0irq/24-0000:00:30602-21munin-run19:34:442
339950,0migration/37907-21kthreadcore21:30:043
339950,0migration/37126-21kthreadcore23:30:023
339950,0migration/36982-21kthreadcore20:50:023
339950,0migration/35247-21timerandwakeup20:00:073
339950,0migration/34920-21irqcore22:50:013
339950,0migration/34210-21fschecks_time23:10:003
339950,0migration/332635-21kthreadcore20:45:033
339950,0migration/329512-21fschecks_count21:44:593
339950,0migration/327991-21latency_hist22:44:433
339950,0migration/327831-21cpuspeed_turbos23:24:583
339950,0migration/327448-21runrttasks22:01:523
339950,0migration/326295-21munin-node20:40:063
339950,0migration/325582-21unixbench-2d23:20:073
339950,0migration/321521-21kthreadcore19:50:023
339950,0migration/319040-21taskset22:16:263
339950,0migration/318024-21kthreadcore20:35:033
339950,0migration/314401-21kthreadcore20:55:033
339950,0migration/31432-21kthreadcore21:25:033
339950,0migration/31421-21runrttasks23:46:573
339950,0migration/313416-21taskset20:32:543
339950,0migration/313252-21munin-node21:34:583
339950,0migration/31313-21munin-node23:05:103
339950,0migration/312340-21gltestperf23:14:583
339950,0migration/310513-21munin-node23:54:573
28415050,0irq/27-eth0-tx-31781-21munin-node20:00:000
28415050,0irq/27-eth0-tx-26102-21munin-node20:20:070
28415050,0irq/27-eth0-tx-22776-21cut21:00:032
28415050,0irq/27-eth0-tx-0-21swapper/223:59:062
28415050,0irq/27-eth0-tx-0-21swapper/223:53:402
28415050,0irq/27-eth0-tx-0-21swapper/223:45:152
28415050,0irq/27-eth0-tx-0-21swapper/223:19:442
28415050,0irq/27-eth0-tx-0-21swapper/222:45:292
28415050,0irq/27-eth0-tx-0-21swapper/222:41:222
28415050,0irq/27-eth0-tx-0-21swapper/222:39:292
28415050,0irq/27-eth0-tx-0-21swapper/222:15:502
28415050,0irq/27-eth0-tx-0-21swapper/222:04:032
28415050,0irq/27-eth0-tx-0-21swapper/221:34:262
28415050,0irq/27-eth0-tx-0-21swapper/221:24:522
28415050,0irq/27-eth0-tx-0-21swapper/221:22:462
28415050,0irq/27-eth0-tx-0-21swapper/221:12:322
28415050,0irq/27-eth0-tx-0-21swapper/221:05:472
28415050,0irq/27-eth0-tx-0-21swapper/220:45:082
28415050,0irq/27-eth0-tx-0-21swapper/019:15:220
28415050,0irq/27-eth0-tx-0-21swapper/019:04:030
28405050,0irq/26-eth0-rx-9993-21kthreadcore20:05:033
28405050,0irq/26-eth0-rx-8866-21swap18:50:083
28405050,0irq/26-eth0-rx-4054-21ssh22:29:523
28405050,0irq/26-eth0-rx-19836-21munin-run22:39:433
28405050,0irq/26-eth0-rx-18138-21latency_hist23:59:433
28405050,0irq/26-eth0-rx-14068-21if_eth1.221:15:013
28405050,0irq/26-eth0-rx-1364-21tr21:05:033
28405050,0irq/26-eth0-rx-0-21swapper/323:53:443
28405050,0irq/26-eth0-rx-0-21swapper/322:19:503
28405050,0irq/26-eth0-rx-0-21swapper/322:09:023
28405050,0irq/26-eth0-rx-0-21swapper/321:50:093
28405050,0irq/26-eth0-rx-0-21swapper/321:41:013
28405050,0irq/26-eth0-rx-0-21swapper/321:20:563
28405050,0irq/26-eth0-rx-0-21swapper/321:13:483
28405050,0irq/26-eth0-rx-0-21swapper/300:04:533
259950,0migration/2605-21kernelversion20:00:012
259950,0migration/225027-21kthreadcore19:05:022
249950,0watchdog/20-21swapper/222:51:302
219439955,0cyclictest20263-21diskmemload23:58:081
219439955,0cyclictest0-21swapper/122:45:171
219439955,0cyclictest0-21swapper/122:28:351
219439950,0cyclictest0-21swapper/122:13:271
219439950,0cyclictest0-21swapper/121:19:321
219439950,0cyclictest0-21swapper/121:05:581
219419955,0cyclictest0-21swapper/023:52:160
219419955,0cyclictest0-21swapper/022:42:540
219419955,0cyclictest0-21swapper/020:39:560
219419950,0cyclictest0-21swapper/023:34:080
219419950,0cyclictest0-21swapper/022:51:250
19150,0ktimersoftd/10-21swapper/120:53:021
179950,0migration/18155-21sh22:50:031
179950,0migration/126556-21missed_timers21:20:051
179950,0migration/122565-21kthreadcore19:50:021
179950,0migration/121365-21fschecks_time21:59:591
179950,0migration/120641-21gltestperf22:59:581
179950,0migration/117340-21munin-node19:45:071
179950,0migration/113218-21kthreadcore23:15:011
179950,0migration/11102-21gltestperf20:24:591
169950,0watchdog/17807-21kthreadcore20:05:021
169950,0watchdog/132498-21expr19:59:581
169950,0watchdog/132453-21chrt20:23:261
169950,0watchdog/128558-21ssh21:03:541
169950,0watchdog/120263-21diskmemload00:08:061
169950,0watchdog/115902-21tr21:55:021
169950,0watchdog/10-21swapper/123:47:061
169950,0watchdog/10-21swapper/123:40:501
169950,0watchdog/10-21swapper/123:36:581
169950,0watchdog/10-21swapper/123:27:061
169950,0watchdog/10-21swapper/122:41:221
169950,0watchdog/10-21swapper/122:38:301
169950,0watchdog/10-21swapper/122:31:221
149950,0migration/09184-21kthreadcore21:30:050
149950,0migration/06430-21kthreadcore18:50:030
149950,0migration/05209-21gltestperf20:49:580
149950,0migration/032193-21kthreadcore21:25:020
149950,0migration/031479-21fschecks_time19:34:590
149950,0migration/030211-21df_abs19:09:570
149950,0migration/02959-21taskset21:05:150
149950,0migration/028811-21latency_hist18:44:440
149950,0migration/028218-21taskset21:03:030
149950,0migration/02820-21taskset20:46:200
149950,0migration/027438-21latency18:40:010
149950,0migration/026938-21munin-node21:20:090
149950,0migration/02350-21sh23:07:180
149950,0migration/023270-21kthreadcore22:00:020
149950,0migration/023004-21pmu-power19:25:040
149950,0migration/019933-21taskset20:57:500
149950,0migration/019217-21sh22:38:010
149950,0migration/017219-21kthreadcore20:10:050
149950,0migration/015897-21missed_timers23:55:050
149950,0migration/015376-21kthreadcore22:15:050
149950,0migration/014859-21kthreadcore19:45:040
149950,0migration/013315-21fschecks_time21:54:580
149950,0migration/013171-21aten_r1power_en21:14:570
149950,0migration/011581-21runrttasks22:12:590
149950,0migration/010755-21latency_hist19:19:440
149950,0migration/010614-21kthreadcore20:30:020
8225040,0irq/24-0000:00:0-21swapper/218:39:572
339940,0migration/36117-21kthreadcore19:15:033
339940,0migration/34246-21kthreadcore20:25:023
339940,0migration/329358-21taskset19:07:013
339940,0migration/327799-21kthreadcore19:55:013
339940,0migration/316868-21df_inode19:00:003
28415040,0irq/27-eth0-tx-0-21swapper/019:29:590
28405040,0irq/26-eth0-rx-6445-21snmpd19:31:043
28405040,0irq/26-eth0-rx-1992-21processes18:45:073
28405040,0irq/26-eth0-rx-0-21swapper/323:35:453
28405040,0irq/26-eth0-rx-0-21swapper/323:00:003
28405040,0irq/26-eth0-rx-0-21swapper/322:24:463
28405040,0irq/26-eth0-rx-0-21swapper/321:55:113
28405040,0irq/26-eth0-rx-0-21swapper/320:59:523
28405040,0irq/26-eth0-rx-0-21swapper/320:20:153
28405040,0irq/26-eth0-rx-0-21swapper/319:45:043
259940,0migration/27697-21kthreadcore18:50:042
259940,0migration/23101-21sendmail_mailtr19:10:052
259940,0migration/229355-21fschecks_count18:44:592
259940,0migration/225278-21timerwakeupswit20:15:082
259940,0migration/220186-21kthreadcore19:50:012
259940,0migration/218576-21irqcore19:25:002
259940,0migration/216777-21cpuspeed_turbos19:00:002
249940,0watchdog/227266-21irqrtprio20:20:022
249940,0watchdog/217130-21ps19:23:222
249940,0watchdog/20-21swapper/219:40:022
219459943,0cyclictest8008-21awk20:30:002
219459943,0cyclictest6445-21snmpd20:10:042
219459943,0cyclictest5016-21awk19:15:012
219459943,0cyclictest4463-21awk23:10:012
219459943,0cyclictest3425-21lldpd20:04:522
219459943,0cyclictest3425-21lldpd19:56:522
219459943,0cyclictest3425-21lldpd19:49:222
219459943,0cyclictest3425-21lldpd19:36:232
219459943,0cyclictest10719-21awk18:55:002
219459940,0cyclictest0-21swapper/220:38:262
219439944,0cyclictest0-21swapper/121:31:021
219439944,0cyclictest0-21swapper/119:35:001
219439943,0cyclictest6445-21snmpd23:49:511
219439943,0cyclictest6445-21snmpd20:45:351
219439943,0cyclictest6445-21snmpd20:31:501
219439943,0cyclictest6042-21awk19:40:021
219439943,0cyclictest4984-21awk19:15:011
219439943,0cyclictest28851-21awk23:05:021
219439943,0cyclictest26160-21kworker/1:020:54:591
219439943,0cyclictest26160-21kworker/1:020:35:191
219439943,0cyclictest26160-21kworker/1:000:00:231
219439943,0cyclictest10689-21awk18:55:001
219419944,0cyclictest0-21swapper/023:45:290
219419944,0cyclictest0-21swapper/023:24:030
219419944,0cyclictest0-21swapper/021:51:110
219419944,0cyclictest0-21swapper/021:42:220
219419943,0cyclictest10704-21awk18:55:000
19140,0ktimersoftd/129835-21munin-node19:09:571
179940,0migration/18289-21kthreadcore21:50:031
179940,0migration/16122-21kthreadcore23:10:031
179940,0migration/15785-21kthreadcore23:30:021
179940,0migration/132040-21kthreadcore21:45:031
179940,0migration/130133-21kthreadcore19:55:011
179940,0migration/125669-21kthreadcore19:30:041
169940,0watchdog/17379-21kthreadcore21:10:021
169940,0watchdog/127140-21ssh23:24:181
169940,0watchdog/123478-21kthreadcore22:20:021
169940,0watchdog/118945-21ssh22:57:541
169940,0watchdog/112815-21ssh21:34:461
169940,0watchdog/10-21swapper/121:25:021
169940,0watchdog/10-21swapper/120:10:061
169940,0watchdog/10-21swapper/119:20:021
169940,0watchdog/10-21swapper/119:05:021
169940,0watchdog/10-21swapper/118:50:061
149940,0migration/028368-21hddtemp_smartct23:25:000
149940,0migration/022169-21kthreadcore23:00:010
149940,0migration/021760-21kthreadcore19:50:020
149940,0migration/019428-21sh21:36:160
149940,0migration/01521-21irqcore20:25:010
149940,0migration/011547-21diskmemload23:14:480
149940,0migration/010012-21kthreadcore20:05:030
339930,0migration/331002-21irqrtprio19:10:003
339930,0migration/327499-21missed_timers18:40:023
28415030,0irq/27-eth0-tx-0-21swapper/020:14:590
28415030,0irq/27-eth0-tx-0-21swapper/019:40:020
28405030,0irq/26-eth0-rx-0-21swapper/319:39:593
28405030,0irq/26-eth0-rx-0-21swapper/319:36:273
28405030,0irq/26-eth0-rx-0-21swapper/319:20:343
219459933,0cyclictest1323-21awk20:25:002
219439933,0cyclictest26160-21kworker/1:021:39:551
219439933,0cyclictest26160-21kworker/1:020:14:451
219439933,0cyclictest26160-21kworker/1:019:24:451
219439933,0cyclictest26160-21kworker/1:018:44:451
219419933,0cyclictest6445-21snmpd22:19:510
219419932,0cyclictest5096-21awk22:30:020
219419932,0cyclictest4487-21awk23:10:010
219419932,0cyclictest3520-21sadc22:49:430
219419932,0cyclictest29866-21awk21:45:010
219419932,0cyclictest29716-21awk22:05:010
219419932,0cyclictest20100-21awk23:40:010
219419932,0cyclictest18371-21awk23:59:560
219419932,0cyclictest13816-21munin-node20:35:010
219419932,0cyclictest11294-21munin-node23:34:580
219419930,0cyclictest29042-21perl22:24:580
169930,0watchdog/10-21swapper/118:39:461
28405020,0irq/26-eth0-rx-0-21swapper/320:15:003
28405020,0irq/26-eth0-rx-0-21swapper/320:09:583
219419920,0cyclictest23190-21aten_r1power_en19:04:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional