You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-28 - 13:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Thu Aug 28, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2078129288,0sleep10-21swapper/118:39:501
2093126665,0sleep00-21swapper/018:41:340
291350350,0irq/25-eth10-21swapper/318:40:513
2076223228,0sleep20-21swapper/218:39:352
28415070,0irq/27-eth0-tx-0-21swapper/223:28:352
28415070,0irq/27-eth0-tx-0-21swapper/222:52:462
249970,0watchdog/20-21swapper/221:54:032
4160,0ktimersoftd/00-21swapper/021:57:480
339960,0migration/38485-21sh21:34:383
339960,0migration/34808-21kthreadcore20:53:073
339960,0migration/3385-21sh21:29:203
339960,0migration/330364-21sh22:29:543
339960,0migration/326292-21sh20:45:233
339960,0migration/325323-21aten_r1power_en20:23:033
339960,0migration/312985-21sh22:59:503
28415060,0irq/27-eth0-tx-23543-21kthreadcore23:48:082
28415060,0irq/27-eth0-tx-15813-21munin-node22:23:052
28415060,0irq/27-eth0-tx-15002-21ssh22:21:132
28415060,0irq/27-eth0-tx-11968-21munin-node20:13:100
28415060,0irq/27-eth0-tx-0-21swapper/223:56:152
28415060,0irq/27-eth0-tx-0-21swapper/223:41:102
28415060,0irq/27-eth0-tx-0-21swapper/223:32:572
28415060,0irq/27-eth0-tx-0-21swapper/223:22:552
28415060,0irq/27-eth0-tx-0-21swapper/223:13:562
28415060,0irq/27-eth0-tx-0-21swapper/223:09:362
28415060,0irq/27-eth0-tx-0-21swapper/223:05:322
28415060,0irq/27-eth0-tx-0-21swapper/223:02:322
28415060,0irq/27-eth0-tx-0-21swapper/222:36:032
28415060,0irq/27-eth0-tx-0-21swapper/222:06:002
28415060,0irq/27-eth0-tx-0-21swapper/222:00:022
28415060,0irq/27-eth0-tx-0-21swapper/221:51:022
28415060,0irq/27-eth0-tx-0-21swapper/221:43:352
28415060,0irq/27-eth0-tx-0-21swapper/221:38:492
28415060,0irq/27-eth0-tx-0-21swapper/221:37:012
28415060,0irq/27-eth0-tx-0-21swapper/221:24:532
28415060,0irq/27-eth0-tx-0-21swapper/221:15:312
28415060,0irq/27-eth0-tx-0-21swapper/221:01:232
28415060,0irq/27-eth0-tx-0-21swapper/200:12:402
28415060,0irq/27-eth0-tx-0-21swapper/200:04:072
28415060,0irq/27-eth0-tx-0-21swapper/020:33:080
28415060,0irq/27-eth0-tx-0-21swapper/019:29:190
28415060,0irq/27-eth0-tx-0-21swapper/019:23:010
28405060,0irq/26-eth0-rx-7276-21ssh21:53:403
28405060,0irq/26-eth0-rx-30143-21iostat_ios23:33:053
28405060,0irq/26-eth0-rx-27545-21diskstats21:08:043
28405060,0irq/26-eth0-rx-26399-21cut23:28:113
28405060,0irq/26-eth0-rx-22915-21idleruntime23:08:053
28405060,0irq/26-eth0-rx-22339-21port_sshpriv22:03:093
28405060,0irq/26-eth0-rx-11850-21perl20:58:053
249960,0watchdog/20-21swapper/223:58:112
249960,0watchdog/20-21swapper/223:17:552
249960,0watchdog/20-21swapper/219:10:552
211099966,0cyclictest6445-21snmpd19:22:461
211099960,0cyclictest1274-21ssh20:48:381
211099960,0cyclictest0-21swapper/123:10:031
149960,0migration/0369-21sh21:29:160
149960,0migration/030965-21if_eth1.319:38:040
149960,0migration/027568-21kthreadcore21:48:050
149960,0migration/019301-21hddtemp_smartct21:23:030
4150,0ktimersoftd/00-21swapper/023:33:450
4150,0ktimersoftd/00-21swapper/022:25:580
4150,0ktimersoftd/00-21swapper/020:43:110
4150,0ktimersoftd/00-21swapper/019:52:490
35150,0ktimersoftd/30-21swapper/323:17:343
339950,0migration/38573-21taskset18:55:183
339950,0migration/35675-21cpuspeed_turbos23:18:033
339950,0migration/34291-21munin-node22:53:103
339950,0migration/330363-21taskset19:58:553
339950,0migration/329486-21kthreadcore18:48:053
339950,0migration/329068-21sh19:12:583
339950,0migration/328640-21missed_timers19:33:073
339950,0migration/322263-21latency_hist19:07:493
339950,0migration/321983-21missed_timers19:28:083
339950,0migration/318827-21cpuspeed_turbos20:18:013
339950,0migration/318097-21fschecks_count21:43:053
339950,0migration/317125-21munin-node00:03:143
339950,0migration/316191-21taskset19:25:083
339950,0migration/314439-21kthreadcore23:43:113
339950,0migration/314318-21ksmtuned21:58:083
339950,0migration/312140-21kthreadcore22:38:063
339950,0migration/311980-21munin-node20:13:003
339950,0migration/310879-21latency_hist19:47:493
28415050,0irq/27-eth0-tx-31590-21cut21:08:092
28415050,0irq/27-eth0-tx-30946-21munin-node20:03:050
28415050,0irq/27-eth0-tx-16426-21grep21:18:122
28415050,0irq/27-eth0-tx-12913-21munin-node20:38:110
28415050,0irq/27-eth0-tx-12913-21munin-node20:38:110
28415050,0irq/27-eth0-tx-0-21swapper/222:44:042
28415050,0irq/27-eth0-tx-0-21swapper/221:04:362
28415050,0irq/27-eth0-tx-0-21swapper/020:23:050
28415050,0irq/27-eth0-tx-0-21swapper/020:17:590
28415050,0irq/27-eth0-tx-0-21swapper/020:10:490
28415050,0irq/27-eth0-tx-0-21swapper/019:13:000
28405050,0irq/26-eth0-rx-9567-21kthreadcore20:08:073
28405050,0irq/26-eth0-rx-3430-21grep22:33:083
28405050,0irq/26-eth0-rx-30366-21munin-node22:08:173
28405050,0irq/26-eth0-rx-26692-21kthreadcore22:48:063
28405050,0irq/26-eth0-rx-26330-21if_eth121:48:043
28405050,0irq/26-eth0-rx-14521-21tr21:38:103
28405050,0irq/26-eth0-rx-0-21swapper/323:57:503
28405050,0irq/26-eth0-rx-0-21swapper/323:53:013
28405050,0irq/26-eth0-rx-0-21swapper/323:02:593
28405050,0irq/26-eth0-rx-0-21swapper/322:46:323
28405050,0irq/26-eth0-rx-0-21swapper/322:23:083
28405050,0irq/26-eth0-rx-0-21swapper/322:15:523
28405050,0irq/26-eth0-rx-0-21swapper/321:18:083
28405050,0irq/26-eth0-rx-0-21swapper/319:48:073
28405050,0irq/26-eth0-rx-0-21swapper/300:10:443
27150,0ktimersoftd/20-21swapper/219:52:492
259950,0migration/29001-21munin-node19:18:132
259950,0migration/232357-21kthreadcore20:03:072
259950,0migration/232257-21kthreadcore22:53:052
259950,0migration/229252-21unixbench_multi19:33:122
259950,0migration/228587-21kthreadcore20:23:092
259950,0migration/225393-21idleruntime22:08:092
259950,0migration/216504-21latency_hist19:27:492
249950,0watchdog/20-21swapper/223:44:192
249950,0watchdog/20-21swapper/222:39:392
249950,0watchdog/20-21swapper/222:28:512
249950,0watchdog/20-21swapper/222:13:112
249950,0watchdog/20-21swapper/221:27:512
249950,0watchdog/20-21swapper/220:53:152
249950,0watchdog/20-21swapper/220:48:592
249950,0watchdog/20-21swapper/220:43:272
249950,0watchdog/20-21swapper/219:37:512
249950,0watchdog/20-21swapper/219:28:072
249950,0watchdog/20-21swapper/219:14:032
249950,0watchdog/20-21swapper/219:03:072
249950,0watchdog/20-21swapper/218:48:112
249950,0watchdog/20-21swapper/218:42:512
211099955,0cyclictest0-21swapper/123:56:151
211099955,0cyclictest0-21swapper/123:05:011
211099955,0cyclictest0-21swapper/122:09:211
211099955,0cyclictest0-21swapper/121:35:561
211099955,0cyclictest0-21swapper/121:23:381
211099955,0cyclictest0-21swapper/100:07:081
211099950,0cyclictest0-21swapper/123:51:311
211089955,0cyclictest0-21swapper/023:00:580
211089955,0cyclictest0-21swapper/021:09:490
211089955,0cyclictest0-21swapper/019:48:090
211089954,0cyclictest19755-21sh21:03:010
211089950,0cyclictest0-21swapper/021:22:260
19150,0ktimersoftd/10-21swapper/123:38:071
19150,0ktimersoftd/10-21swapper/123:17:341
179950,0migration/1856-21taskset21:52:571
179950,0migration/130894-21sh22:31:301
179950,0migration/124655-21kthreadcore19:33:051
179950,0migration/121747-21taskset19:03:411
179950,0migration/11663-21kthreadcore22:13:091
179950,0migration/113927-21munin-node22:18:121
169950,0watchdog/117203-21tail20:58:151
149950,0migration/08601-21sendmail_mailtr21:13:100
149950,0migration/08507-21kthreadcore22:38:050
149950,0migration/07414-21kthreadcore23:18:070
149950,0migration/06273-21kthreadcore23:58:110
149950,0migration/05043-21munin-node22:33:120
149950,0migration/04896-21sh23:17:070
149950,0migration/04380-21aten_r1power_fr19:43:010
149950,0migration/04211-21kthreadcore22:13:100
149950,0migration/032039-21kthreadcore20:48:070
149950,0migration/029029-21munin-node23:53:030
149950,0migration/027473-21kthreadcore19:58:070
149950,0migration/026063-21kthreadcore22:48:050
149950,0migration/025175-21kthreadcore23:28:110
149950,0migration/023086-21gltestperf19:08:050
149950,0migration/022090-21kthreadcore00:08:120
149950,0migration/021311-21irqcore23:48:070
149950,0migration/020809-21sh23:04:370
149950,0migration/020622-21kthreadcore21:43:090
149950,0migration/01978-21ksmtuned20:28:070
149950,0migration/01773-21kthreadcore22:53:060
149950,0migration/011894-21kthreadcore22:18:070
339940,0migration/331367-21fschecks_count20:03:043
339940,0migration/323778-21taskset19:54:233
339940,0migration/322889-21kthreadcore18:43:083
28415040,0irq/27-eth0-tx-0-21swapper/019:33:010
28415040,0irq/27-eth0-tx-0-21swapper/019:19:190
28415040,0irq/27-eth0-tx-0-21swapper/018:48:060
28405040,0irq/26-eth0-rx-0-21swapper/323:48:403
28405040,0irq/26-eth0-rx-0-21swapper/323:42:383
28405040,0irq/26-eth0-rx-0-21swapper/323:22:513
28405040,0irq/26-eth0-rx-0-21swapper/321:24:483
28405040,0irq/26-eth0-rx-0-21swapper/321:13:063
28405040,0irq/26-eth0-rx-0-21swapper/321:03:063
28405040,0irq/26-eth0-rx-0-21swapper/320:47:533
28405040,0irq/26-eth0-rx-0-21swapper/320:38:033
28405040,0irq/26-eth0-rx-0-21swapper/320:38:033
28405040,0irq/26-eth0-rx-0-21swapper/320:33:003
259940,0migration/25855-21kthreadcore18:53:062
259940,0migration/24851-21fschecks_time19:43:032
259940,0migration/212230-21irqrtprio19:48:062
249940,0watchdog/20-21swapper/220:38:152
249940,0watchdog/20-21swapper/220:38:152
249940,0watchdog/20-21swapper/220:33:472
249940,0watchdog/20-21swapper/220:29:232
249940,0watchdog/20-21swapper/220:17:552
249940,0watchdog/20-21swapper/220:13:032
249940,0watchdog/20-21swapper/220:07:592
249940,0watchdog/20-21swapper/219:58:072
249940,0watchdog/20-21swapper/218:58:072
211099943,0cyclictest783-21kworker/1:123:46:301
211099943,0cyclictest783-21kworker/1:123:17:541
211099943,0cyclictest783-21kworker/1:118:43:161
211099943,0cyclictest6445-21snmpd20:55:501
211099943,0cyclictest3425-21lldpd22:58:031
211099943,0cyclictest3425-21lldpd22:24:251
211099943,0cyclictest3425-21lldpd00:08:011
211099943,0cyclictest3425-21lldpd00:01:301
211099943,0cyclictest32561-21awk22:33:061
211099943,0cyclictest31697-21awk22:53:031
211099943,0cyclictest3164-21awk18:53:031
211099943,0cyclictest28146-21awk21:08:071
211099943,0cyclictest26378-21awk21:48:041
211099943,0cyclictest10833-21awk19:23:041
211099943,0cyclictest10687-21awk21:38:081
211099940,0cyclictest0-21swapper/120:29:151
211089944,0cyclictest0-21swapper/022:08:090
211089943,0cyclictest6445-21snmpd22:44:050
211089943,0cyclictest3110-21irqbalance20:58:130
211089943,0cyclictest3105-21awk18:53:030
211089943,0cyclictest1784-21awk21:53:060
211089943,0cyclictest13349-21awk23:43:090
211089943,0cyclictest10634-21awk21:38:070
179940,0migration/18846-21kthreadcore19:43:071
179940,0migration/131076-21aten_r1power_po20:03:021
179940,0migration/130258-21unixbench_singl19:58:121
179940,0migration/129510-21fschecks_count19:13:001
179940,0migration/129273-21kthreadcore20:23:091
179940,0migration/124047-21ntp_kernel_pll_20:18:081
179940,0migration/123042-21kthreadcore23:28:101
179940,0migration/118852-21kthreadcore22:03:051
179940,0migration/118741-21irqcore19:53:071
179940,0migration/116839-21timerwakeupswit19:48:131
179940,0migration/114250-21taskset22:41:091
179940,0migration/111133-21timerwakeupswit20:08:121
169940,0watchdog/128805-21cat22:48:111
169940,0watchdog/128299-21grep19:08:151
169940,0watchdog/127977-21munin-node18:48:111
169940,0watchdog/127561-21irqrtprio21:28:071
169940,0watchdog/120914-21munin-node20:43:071
169940,0watchdog/120840-21python22:43:111
169940,0watchdog/119157-21aten2.4-expect21:02:591
169940,0watchdog/118913-21ntp_kernel_pll_23:23:111
169940,0watchdog/11602-21kthreadcore23:33:071
169940,0watchdog/114915-21unixbench_multi18:58:151
169940,0watchdog/113309-21diskstats20:38:031
169940,0watchdog/113309-21diskstats20:38:031
169940,0watchdog/111733-21iostat_ios21:18:071
169940,0watchdog/10-21swapper/119:38:111
149940,0migration/021097-21kthreadcore22:03:060
149940,0migration/019811-21kthreadcore19:03:070
149940,0migration/011918-21kthreadcore18:58:090
28405030,0irq/26-eth0-rx-0-21swapper/322:17:563
28405030,0irq/26-eth0-rx-0-21swapper/320:28:023
28405030,0irq/26-eth0-rx-0-21swapper/319:38:473
28405030,0irq/26-eth0-rx-0-21swapper/319:18:123
211099933,0cyclictest783-21kworker/1:121:42:521
211099933,0cyclictest783-21kworker/1:121:12:501
211099933,0cyclictest783-21kworker/1:120:13:001
211099933,0cyclictest783-21kworker/1:119:27:561
211099932,0cyclictest783-21kworker/1:121:57:581
211099931,0cyclictest5977-21latency_hist20:32:491
211089932,0cyclictest9322-21sh20:53:230
211089932,0cyclictest6547-21runrttasks23:08:450
211089932,0cyclictest27209-21grep18:43:140
211089932,0cyclictest14668-21tr23:23:080
211089932,0cyclictest13407-21kthreadcore00:03:080
211089930,0cyclictest305532sleep022:30:290
211089930,0cyclictest11213-21diskmemload23:39:330
28405020,0irq/26-eth0-rx-0-21swapper/319:02:573
28405020,0irq/26-eth0-rx-0-21swapper/318:57:553
211089920,0cyclictest1522-21latency_hist21:32:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional