You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 10:38
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Sat Jul 12, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
351740,0ktimersoftd/312231-21perf18:27:493
1911624230,0sleep20-21swapper/218:24:512
1912524031,0sleep10-21swapper/118:24:581
1916222925,0sleep00-21swapper/018:25:250
19453992222,0cyclictest23871-21snmpd20:29:090
19453991818,0cyclictest23871-21snmpd21:04:150
8315070,0irq/26-0000:00:0-21swapper/318:38:123
29925070,0irq/32-eth0-rx-0-21swapper/322:48:133
29925070,0irq/32-eth0-rx-0-21swapper/322:13:213
149970,0migration/026558-21kthreadcore22:33:180
8315060,0irq/26-0000:00:3302-21if_eth1.219:23:173
35160,0ktimersoftd/30-21swapper/321:03:353
339960,0migration/36326-21fschecks_time21:08:163
339960,0migration/331462-21sh21:22:503
339960,0migration/31355-21munin-node22:38:123
329960,0watchdog/327824-21telnet20:03:123
329960,0watchdog/327824-21telnet20:03:123
329960,0watchdog/30-21swapper/323:53:193
329960,0watchdog/30-21swapper/323:38:273
329960,0watchdog/30-21swapper/323:19:153
329960,0watchdog/30-21swapper/323:12:593
329960,0watchdog/30-21swapper/323:07:593
329960,0watchdog/30-21swapper/323:06:313
329960,0watchdog/30-21swapper/322:42:593
329960,0watchdog/30-21swapper/322:24:473
329960,0watchdog/30-21swapper/322:18:433
329960,0watchdog/30-21swapper/322:08:153
329960,0watchdog/30-21swapper/321:59:153
329960,0watchdog/30-21swapper/321:44:193
329960,0watchdog/30-21swapper/321:26:473
329960,0watchdog/30-21swapper/321:00:003
329960,0watchdog/30-21swapper/320:53:003
329960,0watchdog/30-21swapper/320:29:043
329960,0watchdog/30-21swapper/320:10:323
329960,0watchdog/30-21swapper/319:38:283
29925060,0irq/32-eth0-rx-25488-21taskset23:48:133
29925060,0irq/32-eth0-rx-18030-21fschecks_time20:38:173
29925060,0irq/32-eth0-rx-0-21swapper/323:28:123
29925060,0irq/32-eth0-rx-0-21swapper/323:23:163
29925060,0irq/32-eth0-rx-0-21swapper/322:58:213
29925060,0irq/32-eth0-rx-0-21swapper/322:53:533
29925060,0irq/32-eth0-rx-0-21swapper/321:52:593
29925060,0irq/32-eth0-rx-0-21swapper/321:48:083
29925060,0irq/32-eth0-rx-0-21swapper/321:40:343
29925060,0irq/32-eth0-rx-0-21swapper/321:28:123
29925060,0irq/32-eth0-rx-0-21swapper/321:13:553
29925060,0irq/32-eth0-rx-0-21swapper/320:48:183
29925060,0irq/32-eth0-rx-0-21swapper/320:33:163
149960,0migration/06001-21timerandwakeup21:23:250
8315050,0irq/26-0000:00:0-21swapper/318:58:003
339950,0migration/332681-21kthreadcore19:18:193
339950,0migration/331463-21kthreadcore18:33:203
339950,0migration/327298-21munin-node20:43:183
329950,0watchdog/36368-21timerwakeupswit23:33:273
329950,0watchdog/30-21swapper/322:33:073
329950,0watchdog/30-21swapper/322:28:153
329950,0watchdog/30-21swapper/320:23:083
329950,0watchdog/30-21swapper/320:13:043
329950,0watchdog/30-21swapper/319:58:083
329950,0watchdog/30-21swapper/319:53:123
329950,0watchdog/30-21swapper/319:48:323
329950,0watchdog/30-21swapper/319:45:523
329950,0watchdog/30-21swapper/319:34:483
329950,0watchdog/30-21swapper/319:28:003
329950,0watchdog/30-21swapper/319:13:163
329950,0watchdog/30-21swapper/319:08:163
329950,0watchdog/30-21swapper/319:03:003
329950,0watchdog/30-21swapper/318:53:123
329950,0watchdog/30-21swapper/318:48:163
329950,0watchdog/30-21swapper/318:43:203
329950,0watchdog/30-21swapper/318:28:123
29935050,0irq/33-eth0-tx-5208-21ssh21:07:552
29935050,0irq/33-eth0-tx-3546-21sshd18:46:200
29935050,0irq/33-eth0-tx-3173-21kthreadcore23:53:172
29935050,0irq/33-eth0-tx-24339-21taskset20:39:292
29935050,0irq/33-eth0-tx-2116-21munin-node19:23:200
29935050,0irq/33-eth0-tx-20264-21ssh22:08:352
29935050,0irq/33-eth0-tx-17163-21munin-node20:18:270
29935050,0irq/33-eth0-tx-14345-21munin-node21:13:232
29935050,0irq/33-eth0-tx-14038-21munin-run21:12:582
29935050,0irq/33-eth0-tx-0-21swapper/223:00:032
29935050,0irq/33-eth0-tx-0-21swapper/222:00:522
29935050,0irq/33-eth0-tx-0-21swapper/221:38:422
29935050,0irq/33-eth0-tx-0-21swapper/221:19:412
29935050,0irq/33-eth0-tx-0-21swapper/020:08:210
29935050,0irq/33-eth0-tx-0-21swapper/019:53:160
29935050,0irq/33-eth0-tx-0-21swapper/019:49:010
29935050,0irq/33-eth0-tx-0-21swapper/019:18:160
29925050,0irq/32-eth0-rx-20361-21tr23:43:183
29925050,0irq/32-eth0-rx-0-21swapper/322:03:173
29925050,0irq/32-eth0-rx-0-21swapper/321:32:593
29925050,0irq/32-eth0-rx-0-21swapper/320:18:233
27150,0ktimersoftd/20-21swapper/220:58:232
259950,0migration/28378-21munin-node21:28:132
259950,0migration/230927-21kthreadcore20:03:182
259950,0migration/230927-21kthreadcore20:03:182
259950,0migration/22995-21irqcore18:38:152
259950,0migration/228028-21fschecks_count22:53:172
259950,0migration/226451-21kthreadcore23:28:182
259950,0migration/225238-21kthreadcore22:13:172
259950,0migration/223901-21kthreadcore19:13:212
259950,0migration/22264-21munin-node22:38:152
259950,0migration/218665-21munin-node23:03:182
259950,0migration/217949-21kthreadcore19:08:192
259950,0migration/214062-21kthreadcore20:53:192
259950,0migration/210805-21kthreadcore22:23:162
194559955,0cyclictest0-21swapper/219:48:232
194539955,0cyclictest0-21swapper/020:36:580
179950,0migration/19242-21fschecks_time23:38:121
179950,0migration/1583-21sh21:40:021
179950,0migration/13499-21fschecks_time20:08:141
179950,0migration/130858-21kthreadcore20:03:181
179950,0migration/130858-21kthreadcore20:03:181
179950,0migration/126666-21munin-node20:58:211
179950,0migration/126612-21fschecks_count20:43:151
179950,0migration/123165-21kthreadcore19:58:191
179950,0migration/122973-21kthreadcore23:08:191
179950,0migration/121766-21unixbench_singl23:23:241
179950,0migration/120149-21irqrtprio22:48:161
179950,0migration/117320-21munin-node20:38:161
179950,0migration/111347-21idleruntime-cro20:52:581
169950,0watchdog/112495-21perl20:53:161
149950,0migration/09314-21kthreadcore22:23:160
149950,0migration/07705-21irqcore23:18:170
149950,0migration/05758-21kthreadcore18:38:160
149950,0migration/04468-21latency_hist22:02:580
149950,0migration/031411-21sh23:31:260
149950,0migration/031239-21munin-node23:48:190
149950,0migration/030186-21sh22:16:540
149950,0migration/030107-21kthreadcore21:38:180
149950,0migration/028466-21kernelversion18:33:190
149950,0migration/027329-21kthreadcore21:18:200
149950,0migration/024947-21kthreadcore20:58:180
149950,0migration/022112-21kthreadcore19:13:200
149950,0migration/021844-21fschecks_count23:08:150
149950,0migration/021733-21kthreadcore22:48:170
149950,0migration/019392-21kthreadcore22:28:200
149950,0migration/018815-21kthreadcore21:33:200
149950,0migration/018409-21kthreadcore21:13:220
149950,0migration/018312-21kthreadcore19:33:160
149950,0migration/018159-21sh20:38:180
149950,0migration/017724-21kthreadcore22:08:200
149950,0migration/01424-21kthreadcore19:43:170
149950,0migration/011205-21kthreadcore21:08:210
149950,0migration/010910-21sh22:59:140
149950,0migration/010628-21fschecks_time20:13:130
149950,0migration/010614-21sh20:51:180
8315040,0irq/26-0000:00:2065-21md0_raid120:28:002
29935040,0irq/33-eth0-tx-4736-21munin-node22:03:132
29935040,0irq/33-eth0-tx-18686-21munin-node22:48:222
29935040,0irq/33-eth0-tx-0-21swapper/223:48:332
29935040,0irq/33-eth0-tx-0-21swapper/223:44:042
29935040,0irq/33-eth0-tx-0-21swapper/223:24:152
29935040,0irq/33-eth0-tx-0-21swapper/223:08:452
29935040,0irq/33-eth0-tx-0-21swapper/222:43:192
29935040,0irq/33-eth0-tx-0-21swapper/222:36:072
29935040,0irq/33-eth0-tx-0-21swapper/222:28:092
29935040,0irq/33-eth0-tx-0-21swapper/221:48:002
29935040,0irq/33-eth0-tx-0-21swapper/221:33:122
29935040,0irq/33-eth0-tx-0-21swapper/221:23:182
29935040,0irq/33-eth0-tx-0-21swapper/220:48:052
29935040,0irq/33-eth0-tx-0-21swapper/018:58:150
259940,0migration/26945-21fschecks_count23:18:142
259940,0migration/25938-21kthreadcore20:08:172
259940,0migration/2379-21kthreadcore19:43:172
259940,0migration/232090-21latency_hist23:32:592
259940,0migration/229969-21sh23:13:052
259940,0migration/227680-21munin-node20:43:192
259940,0migration/224775-21kthreadcore21:53:192
259940,0migration/222128-21kthreadcore20:18:192
259940,0migration/221723-21kthreadcore18:48:182
259940,0migration/21892-21latency_hist21:42:592
259940,0migration/216606-21kthreadcore19:53:182
259940,0migration/215625-21munin-node19:28:212
259940,0migration/21527-21kthreadcore22:18:162
259940,0migration/214629-21proc_pri23:38:202
249940,0watchdog/226859-21cut19:38:202
249940,0watchdog/223462-21chrt18:53:042
249940,0watchdog/20-21swapper/218:28:242
194559943,0cyclictest9359-21awk20:33:162
194559943,0cyclictest3599-21lldpd20:25:192
194559943,0cyclictest28434-21awk19:18:172
194549943,0cyclictest5883-21sadc19:47:591
194549943,0cyclictest4891-21awk22:58:131
194549943,0cyclictest3599-21lldpd19:13:181
194549943,0cyclictest3262-21irqbalance23:53:431
194549943,0cyclictest3262-21irqbalance23:28:431
194549943,0cyclictest3262-21irqbalance21:19:131
194549943,0cyclictest3262-21irqbalance21:13:131
194549943,0cyclictest3262-21irqbalance19:03:541
194549943,0cyclictest3262-21irqbalance18:58:141
194549943,0cyclictest3262-21irqbalance18:49:141
194549943,0cyclictest3262-21irqbalance18:39:041
194549943,0cyclictest3196-21awk21:43:151
194549943,0cyclictest29576-21tr18:53:211
194549943,0cyclictest28482-21awk19:18:171
194549943,0cyclictest20816-21awk18:28:151
194549943,0cyclictest20588-21awk21:53:161
194549943,0cyclictest16927-21sadc20:17:591
194549943,0cyclictest15740-21sadc22:27:581
194549943,0cyclictest14016-21awk19:08:171
194549943,0cyclictest11804-21cut21:48:131
194539944,0cyclictest0-21swapper/019:03:000
194539943,0cyclictest9423-21awk21:28:170
194539943,0cyclictest31088-21awk23:13:160
194539943,0cyclictest28665-21awk22:53:200
194539943,0cyclictest27133-21awk20:43:170
194539943,0cyclictest25537-21cat21:53:200
194539943,0cyclictest23871-21snmpd20:06:480
194539943,0cyclictest23871-21snmpd20:06:480
194539943,0cyclictest2254-21awk23:53:150
194539942,0cyclictest3060-21kworker/0:121:43:360
194539940,0cyclictest0-21swapper/022:47:020
179940,0migration/19710-21cpuspeed_turbos19:28:121
179940,0migration/19125-21aten_r1power_po18:43:121
179940,0migration/17382-21hwlatdetect23:18:151
179940,0migration/16759-21fschecks_count19:48:131
179940,0migration/16165-21kthreadcore22:38:181
179940,0migration/15142-21kthreadcore21:23:181
179940,0migration/14200-21kthreadcore19:23:191
179940,0migration/130728-21fschecks_time23:13:141
179940,0migration/130252-21kthreadcore22:53:221
179940,0migration/129605-21kthreadcore23:48:191
179940,0migration/128382-21munin-node21:58:141
179940,0migration/127299-21kthreadcore20:23:171
179940,0migration/124584-21kthreadcore22:13:171
179940,0migration/124036-21taskset20:21:231
179940,0migration/123333-21switchtime23:43:251
179940,0migration/119244-21kthreadcore19:33:171
179940,0migration/115923-21mdstat22:43:171
179940,0migration/114628-21sensors_fan20:33:231
169940,0watchdog/15905-21df_abs21:08:151
169940,0watchdog/14850-21ssh22:03:111
169940,0watchdog/132436-21kthreadcore18:33:201
169940,0watchdog/129306-21aten_r1power_vo21:03:111
169940,0watchdog/117611-21irqstats22:28:191
169940,0watchdog/10-21swapper/123:33:191
169940,0watchdog/10-21swapper/122:08:191
169940,0watchdog/10-21swapper/119:38:201
149940,0migration/028475-21kthreadcore19:38:200
149940,0migration/027321-21kthreadcore18:53:180
149940,0migration/020380-21kthreadcore23:23:170
149940,0migration/019322-21ksmtuned20:55:480
149940,0migration/018076-21kthreadcore19:08:190
149940,0migration/014212-21kthreadcore23:03:160
259930,0migration/230838-21kthreadcore18:33:192
259930,0migration/210326-21kthreadcore19:03:212
194559933,0cyclictest3599-21lldpd18:45:182
194559932,0cyclictest3599-21lldpd20:13:192
194559932,0cyclictest3335-21awk19:23:182
194559932,0cyclictest31834-21awk18:58:192
194559932,0cyclictest23888-21kthreadcore19:58:192
194559932,0cyclictest17821-21awk19:33:152
194549932,0cyclictest9398-21awk21:28:171
194549932,0cyclictest31930-21awk22:18:141
194549932,0cyclictest17611-21grep21:33:161
194549932,0cyclictest13772-21awk23:03:141
194539933,0cyclictest4927-21kworker/0:222:38:110
194539933,0cyclictest3120-21kworker/0:120:23:100
194539933,0cyclictest3120-21kworker/0:119:58:020
194539933,0cyclictest3120-21kworker/0:119:28:040
194539933,0cyclictest3060-21kworker/0:122:18:120
194539933,0cyclictest3060-21kworker/0:121:49:020
194539933,0cyclictest18004-21kworker/0:023:43:070
194539933,0cyclictest18004-21kworker/0:023:38:150
194539933,0cyclictest18004-21kworker/0:023:33:310
194539932,0cyclictest17340-21if_eth118:48:150
194539932,0cyclictest16969-21kworker/0:218:28:100
194539931,0cyclictest8697-21tr22:03:170
169930,0watchdog/116844-21kworker/u8:120:28:121
169930,0watchdog/10-21swapper/122:33:151
169930,0watchdog/10-21swapper/119:57:521
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional