You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-02 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Wed Jul 02, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350460,0irq/25-eth10-21swapper/318:44:233
401223526,0sleep20-21swapper/218:43:192
401023430,0sleep00-21swapper/018:43:170
408723021,0sleep10-21swapper/118:44:101
249970,0watchdog/20-21swapper/223:47:312
42289960,0cyclictest2446-21diskmemload23:15:112
42289960,0cyclictest0-21swapper/221:27:402
42279960,0cyclictest0-21swapper/123:22:371
339960,0migration/325827-21sh22:19:273
339960,0migration/325169-21idleruntime-cro19:24:573
339960,0migration/31364-21sh22:24:323
339960,0migration/313571-21sh00:11:573
28415060,0irq/27-eth0-tx-2446-21diskmemload00:08:522
28415060,0irq/27-eth0-tx-0-21swapper/223:30:392
28415060,0irq/27-eth0-tx-0-21swapper/223:09:112
28415060,0irq/27-eth0-tx-0-21swapper/223:04:552
28415060,0irq/27-eth0-tx-0-21swapper/222:46:392
28415060,0irq/27-eth0-tx-0-21swapper/222:32:362
28415060,0irq/27-eth0-tx-0-21swapper/222:19:092
28415060,0irq/27-eth0-tx-0-21swapper/222:05:132
28415060,0irq/27-eth0-tx-0-21swapper/221:57:242
28415060,0irq/27-eth0-tx-0-21swapper/221:47:182
28415060,0irq/27-eth0-tx-0-21swapper/221:20:432
28415060,0irq/27-eth0-tx-0-21swapper/200:12:312
28415060,0irq/27-eth0-tx-0-21swapper/020:45:100
28405060,0irq/26-eth0-rx-25132-21perl19:00:123
249960,0watchdog/20-21swapper/222:22:152
249960,0watchdog/20-21swapper/200:00:232
149960,0migration/032553-21latency_hist23:44:560
149960,0migration/015785-21sh23:54:560
42289953,0cyclictest17525-21sadc18:54:572
42289950,0cyclictest0-21swapper/221:50:412
42289950,0cyclictest0-21swapper/221:16:382
42269955,0cyclictest0-21swapper/023:23:240
42269950,0cyclictest0-21swapper/023:36:290
42269950,0cyclictest0-21swapper/000:12:160
339950,0migration/37509-21aten2.4-expect20:00:103
339950,0migration/36458-21kthreadcore21:05:153
339950,0migration/34731-21kthreadcore20:20:153
339950,0migration/34436-21taskset18:45:043
339950,0migration/34268-21kthreadcore20:45:153
339950,0migration/3355-21munin-node21:40:183
339950,0migration/330659-21kthreadcore20:40:143
339950,0migration/327619-21munin-node21:00:113
339950,0migration/326170-21idleruntime-cro19:49:573
339950,0migration/320862-21kthreadcore20:55:153
339950,0migration/320394-21kthreadcore21:55:173
339950,0migration/317084-21kthreadcore20:05:153
339950,0migration/316446-21missed_timers20:50:183
339950,0migration/313613-21kthreadcore20:25:133
339950,0migration/312731-21kthreadcore22:30:163
339950,0migration/311173-21kthreadcore23:50:143
28415050,0irq/27-eth0-tx-984-21ssh22:02:062
28415050,0irq/27-eth0-tx-658-21munin-node23:25:132
28415050,0irq/27-eth0-tx-31327-21ssh23:20:292
28415050,0irq/27-eth0-tx-22963-21ssh23:35:232
28415050,0irq/27-eth0-tx-15785-21latency_hist23:54:562
28415050,0irq/27-eth0-tx-0-21swapper/222:38:292
28415050,0irq/27-eth0-tx-0-21swapper/221:42:012
28415050,0irq/27-eth0-tx-0-21swapper/221:10:422
28415050,0irq/27-eth0-tx-0-21swapper/221:05:312
28415050,0irq/27-eth0-tx-0-21swapper/221:00:002
28415050,0irq/27-eth0-tx-0-21swapper/220:55:252
28415050,0irq/27-eth0-tx-0-21swapper/220:50:422
28405050,0irq/26-eth0-rx-7556-21head22:25:193
28405050,0irq/26-eth0-rx-4329-21kthreadcore23:05:183
28405050,0irq/26-eth0-rx-32316-21if_err_eth100:05:123
28405050,0irq/26-eth0-rx-32280-21chrt23:02:123
28405050,0irq/26-eth0-rx-2643-21ssh21:24:473
28405050,0irq/26-eth0-rx-23607-21sed18:55:203
28405050,0irq/26-eth0-rx-19182-21sed22:35:163
28405050,0irq/26-eth0-rx-18325-21ssh21:12:523
28405050,0irq/26-eth0-rx-17535-21df23:15:123
28405050,0irq/26-eth0-rx-10237-21grep22:09:563
28405050,0irq/26-eth0-rx-10205-21kthreadcore23:30:183
28405050,0irq/26-eth0-rx-0-21swapper/323:44:573
28405050,0irq/26-eth0-rx-0-21swapper/323:44:083
28405050,0irq/26-eth0-rx-0-21swapper/323:36:423
28405050,0irq/26-eth0-rx-0-21swapper/323:25:043
28405050,0irq/26-eth0-rx-0-21swapper/323:10:193
28405050,0irq/26-eth0-rx-0-21swapper/322:45:563
28405050,0irq/26-eth0-rx-0-21swapper/322:40:223
28405050,0irq/26-eth0-rx-0-21swapper/322:07:333
28405050,0irq/26-eth0-rx-0-21swapper/322:00:133
28405050,0irq/26-eth0-rx-0-21swapper/321:47:343
28405050,0irq/26-eth0-rx-0-21swapper/321:39:533
27150,0ktimersoftd/20-21swapper/223:39:562
259950,0migration/26321-21kthreadcore20:45:162
259950,0migration/22929-21kthreadcore19:30:152
249950,0watchdog/28826-21kthreadcore18:45:152
249950,0watchdog/20-21swapper/223:50:192
249950,0watchdog/20-21swapper/223:11:592
249950,0watchdog/20-21swapper/222:55:592
249950,0watchdog/20-21swapper/222:51:192
249950,0watchdog/20-21swapper/222:40:232
249950,0watchdog/20-21swapper/222:25:312
249950,0watchdog/20-21swapper/221:36:312
249950,0watchdog/20-21swapper/221:32:112
249950,0watchdog/20-21swapper/220:26:592
249950,0watchdog/20-21swapper/220:22:352
249950,0watchdog/20-21swapper/220:08:592
249950,0watchdog/20-21swapper/219:40:392
249950,0watchdog/20-21swapper/219:34:592
249950,0watchdog/20-21swapper/219:25:192
249950,0watchdog/20-21swapper/219:20:432
249950,0watchdog/20-21swapper/219:01:432
179950,0migration/15724-21kthreadcore23:05:181
179950,0migration/15187-21kthreadcore21:05:141
179950,0migration/130569-21runrttasks00:02:551
179950,0migration/128371-21latency_hist20:39:571
179950,0migration/127027-21cpuspeed_turbos22:00:131
179950,0migration/124814-21missed_timers21:15:161
179950,0migration/122062-21kthreadcore20:55:151
179950,0migration/12176-21taskset21:23:281
179950,0migration/120860-21timerandwakeup20:30:201
179950,0migration/117596-21taskset20:51:361
179950,0migration/11474-21munin-node19:55:121
179950,0migration/110689-21sh22:30:121
169950,0watchdog/18761-21kthreadcore20:00:151
169950,0watchdog/132528-21grep20:15:151
169950,0watchdog/132528-21grep20:15:151
169950,0watchdog/129126-21tr22:40:191
169950,0watchdog/12545-21kthreadcore19:30:151
169950,0watchdog/111505-21df_inode21:30:111
149950,0migration/06908-21fschecks_count19:35:120
149950,0migration/06885-21proc_pri23:05:210
149950,0migration/04895-21kthreadcore23:25:130
149950,0migration/04290-21kthreadcore22:25:150
149950,0migration/032111-21fschecks_time00:05:110
149950,0migration/02938-21taskset21:24:580
149950,0migration/029027-21kthreadcore00:00:170
149950,0migration/029017-21kthreadcore21:20:130
149950,0migration/027628-21kthreadcore23:40:150
149950,0migration/025329-21kthreadcore20:35:140
149950,0migration/023086-21kthreadcore21:35:150
149950,0migration/023068-21ntp_kernel_pll_22:55:170
149950,0migration/022371-21kthreadcore20:55:150
149950,0migration/020490-21munin-run20:09:570
149950,0migration/020331-21fschecks_time19:45:120
149950,0migration/019462-21gltestperf19:20:120
149950,0migration/017370-21sh23:15:110
149950,0migration/016762-21taskset23:14:320
149950,0migration/016210-21sh22:30:480
149950,0migration/015625-21kthreadcore19:40:150
149950,0migration/014251-21kthreadcore21:10:130
149950,0migration/013889-21pmu-power20:25:160
42289943,0cyclictest3425-21lldpd19:50:182
42289943,0cyclictest3110-21irqbalance20:36:012
42289943,0cyclictest3110-21irqbalance19:10:112
42289943,0cyclictest3110-21irqbalance19:05:512
42289943,0cyclictest29262-21awk20:40:122
42279944,0cyclictest0-21swapper/122:23:201
42279943,0cyclictest9264-21awk20:25:101
42279943,0cyclictest8167-21sadc20:24:571
42279943,0cyclictest6445-21snmpd21:59:311
42279943,0cyclictest6445-21snmpd20:09:461
42279943,0cyclictest6445-21snmpd19:14:461
42279943,0cyclictest6445-21snmpd18:54:461
42279943,0cyclictest5887-21kworker/1:023:00:161
42279943,0cyclictest5887-21kworker/1:022:15:281
42279943,0cyclictest5887-21kworker/1:021:50:161
42279943,0cyclictest25289-21awk19:00:131
42279943,0cyclictest22734-21ps20:35:131
42279943,0cyclictest13994-21awk19:40:131
42279943,0cyclictest10099-21awk23:10:151
42279942,0cyclictest5887-21kworker/1:022:05:341
42269944,0cyclictest0-21swapper/021:59:500
42269943,0cyclictest8396-21awk20:00:140
42269943,0cyclictest6445-21snmpd22:36:310
42269943,0cyclictest3425-21lldpd22:54:540
42269943,0cyclictest2353-21sadc21:44:560
42269943,0cyclictest11358-21awk22:10:150
42269943,0cyclictest0-21swapper/018:50:090
42269942,0cyclictest0-21swapper/020:20:190
42269940,0cyclictest0-21swapper/019:15:220
339940,0migration/38841-21kthreadcore19:10:153
339940,0migration/35802-21taskset19:30:593
339940,0migration/326130-21if_eth1.223:20:153
339940,0migration/324422-21fschecks_time00:00:123
339940,0migration/322445-21gltestperf20:35:113
339940,0migration/312577-21taskset19:36:523
339940,0migration/311221-21kthreadcore22:50:133
28415040,0irq/27-eth0-tx-20724-21munin-node20:10:160
28415040,0irq/27-eth0-tx-0-21swapper/222:09:562
28415040,0irq/27-eth0-tx-0-21swapper/019:55:080
28415040,0irq/27-eth0-tx-0-21swapper/019:10:100
28415040,0irq/27-eth0-tx-0-21swapper/019:00:120
28415040,0irq/27-eth0-tx-0-21swapper/018:55:120
28415040,0irq/27-eth0-tx-0-21swapper/018:45:120
28405040,0irq/26-eth0-rx-0-21swapper/323:55:123
28405040,0irq/26-eth0-rx-0-21swapper/322:55:063
28405040,0irq/26-eth0-rx-0-21swapper/321:50:043
28405040,0irq/26-eth0-rx-0-21swapper/321:30:053
28405040,0irq/26-eth0-rx-0-21swapper/321:25:153
28405040,0irq/26-eth0-rx-0-21swapper/321:15:083
28405040,0irq/26-eth0-rx-0-21swapper/320:30:213
259940,0migration/222885-21kthreadcore19:45:162
259940,0migration/216531-21kthreadcore20:30:142
259940,0migration/212321-21cpuspeed_turbos19:15:112
249940,0watchdog/26412-21python19:55:192
249940,0watchdog/220761-21aten_r1power_en20:10:112
249940,0watchdog/20-21swapper/220:19:192
249940,0watchdog/20-21swapper/220:19:192
249940,0watchdog/20-21swapper/220:00:192
249940,0watchdog/20-21swapper/218:54:592
179940,0migration/19538-21taskset21:47:181
179940,0migration/19518-21fschecks_time23:30:141
179940,0migration/14581-21kthreadcore20:45:151
179940,0migration/13814-21kthreadcore23:25:131
179940,0migration/130443-21kthreadcore21:00:161
179940,0migration/129718-21kthreadcore23:40:161
179940,0migration/129609-21kthreadcore19:25:141
179940,0migration/127388-21munin-node19:50:121
179940,0migration/123958-21sshd23:17:191
179940,0migration/118984-21latency_hist21:34:561
169940,0watchdog/17338-21kthreadcore21:25:151
169940,0watchdog/17263-21cpuspeed_turbos00:10:111
169940,0watchdog/16739-21ssh23:46:071
169940,0watchdog/16732-21cpuspeed_turbos19:35:111
169940,0watchdog/12611-21kthreadcore00:05:151
169940,0watchdog/12446-21diskmemload21:09:591
169940,0watchdog/119097-21kthreadcore23:55:151
169940,0watchdog/119027-21expr19:20:111
169940,0watchdog/116714-21ssh22:53:231
169940,0watchdog/114728-21ssh23:51:231
169940,0watchdog/10-21swapper/123:37:311
169940,0watchdog/10-21swapper/122:25:151
169940,0watchdog/10-21swapper/121:40:431
169940,0watchdog/10-21swapper/120:10:111
149940,0migration/029729-21kthreadcore22:20:170
149940,0migration/027368-21kthreadcore23:00:180
149940,0migration/017450-21kthreadcore20:30:140
42279933,0cyclictest5887-21kworker/1:022:46:141
42279933,0cyclictest5887-21kworker/1:022:34:561
42279933,0cyclictest5887-21kworker/1:022:09:561
42279933,0cyclictest5887-21kworker/1:019:45:021
42279933,0cyclictest5887-21kworker/1:019:15:361
42279933,0cyclictest5887-21kworker/1:018:45:141
42279932,0cyclictest6445-21snmpd18:55:371
42279932,0cyclictest31937-21awk19:05:121
42269932,0cyclictest6445-21snmpd20:44:010
42269932,0cyclictest6445-21snmpd20:19:460
42269932,0cyclictest6445-21snmpd20:19:460
42269932,0cyclictest3995-21awk21:05:130
42269932,0cyclictest3275-21awk22:05:140
42269932,0cyclictest27518-21sed22:00:150
42269932,0cyclictest27119-21cut19:25:140
42269932,0cyclictest26752-21awk22:40:160
42269932,0cyclictest26696-21expr19:50:090
42269932,0cyclictest2666-21sadc21:04:560
42269932,0cyclictest20053-21grep21:15:110
42269932,0cyclictest16962-21awk23:55:120
42269932,0cyclictest13027-21kthreadcore23:30:190
42269932,0cyclictest11815-21awk20:50:140
42269932,0cyclictest0-21swapper/019:30:410
28415030,0irq/27-eth0-tx-0-21swapper/019:05:160
28405030,0irq/26-eth0-rx-0-21swapper/320:15:093
28405030,0irq/26-eth0-rx-0-21swapper/320:15:093
28405030,0irq/26-eth0-rx-0-21swapper/320:10:113
28405030,0irq/26-eth0-rx-0-21swapper/319:55:093
28405030,0irq/26-eth0-rx-0-21swapper/319:15:103
28405030,0irq/26-eth0-rx-0-21swapper/318:50:363
169930,0watchdog/10-21swapper/122:54:591
149930,0migration/03892-21diskmemload21:45:150
42269920,0cyclictest1342-21date22:44:560
42269920,0cyclictest0-21swapper/022:15:000
42269920,0cyclictest0-21swapper/021:49:560
42269920,0cyclictest0-21swapper/021:29:580
28405020,0irq/26-eth0-rx-4069-21sed19:05:163
28405020,0irq/26-eth0-rx-0-21swapper/319:50:173
28405020,0irq/26-eth0-rx-0-21swapper/319:40:043
28405020,0irq/26-eth0-rx-0-21swapper/319:25:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional