You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-29 - 01:58
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2.osadl.org (updated Sun Jun 29, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3036324435,0sleep10-21swapper/118:23:581
467724130,0sleep20-21swapper/218:25:432
480923727,0sleep30-21swapper/318:27:123
419623228,0sleep00-21swapper/018:24:130
8315070,0irq/26-0000:00:0-21swapper/318:29:063
35170,0ktimersoftd/314416-21cpuspeed_turbos23:19:003
329970,0watchdog/30-21swapper/321:05:393
29925070,0irq/32-eth0-rx-0-21swapper/320:39:123
50129965,0cyclictest0-21swapper/123:22:511
50129960,0cyclictest0-21swapper/122:41:041
50129960,0cyclictest0-21swapper/122:27:491
35160,0ktimersoftd/30-21swapper/322:20:103
329960,0watchdog/362922chrt23:30:393
329960,0watchdog/324143-21ssh21:29:113
329960,0watchdog/318871-21cut22:24:113
329960,0watchdog/30-21swapper/323:48:473
329960,0watchdog/30-21swapper/323:48:273
329960,0watchdog/30-21swapper/323:41:353
329960,0watchdog/30-21swapper/323:23:473
329960,0watchdog/30-21swapper/323:11:553
329960,0watchdog/30-21swapper/323:04:113
329960,0watchdog/30-21swapper/322:58:473
329960,0watchdog/30-21swapper/322:49:073
329960,0watchdog/30-21swapper/322:39:073
329960,0watchdog/30-21swapper/322:34:593
329960,0watchdog/30-21swapper/322:31:193
329960,0watchdog/30-21swapper/322:09:193
329960,0watchdog/30-21swapper/322:04:353
329960,0watchdog/30-21swapper/321:58:473
329960,0watchdog/30-21swapper/321:24:073
329960,0watchdog/30-21swapper/321:17:393
329960,0watchdog/30-21swapper/321:12:313
329960,0watchdog/30-21swapper/320:59:113
329960,0watchdog/30-21swapper/320:44:113
329960,0watchdog/30-21swapper/320:36:273
329960,0watchdog/30-21swapper/320:15:433
329960,0watchdog/30-21swapper/319:49:073
329960,0watchdog/30-21swapper/319:44:073
329960,0watchdog/30-21swapper/319:24:043
329960,0watchdog/30-21swapper/318:33:483
29925060,0irq/32-eth0-rx-0-21swapper/323:56:123
29925060,0irq/32-eth0-rx-0-21swapper/323:35:523
29925060,0irq/32-eth0-rx-0-21swapper/323:14:093
29925060,0irq/32-eth0-rx-0-21swapper/322:53:503
29925060,0irq/32-eth0-rx-0-21swapper/322:45:033
29925060,0irq/32-eth0-rx-0-21swapper/322:14:043
29925060,0irq/32-eth0-rx-0-21swapper/321:44:053
29925060,0irq/32-eth0-rx-0-21swapper/321:43:093
29925060,0irq/32-eth0-rx-0-21swapper/321:37:523
29925060,0irq/32-eth0-rx-0-21swapper/321:19:113
29925060,0irq/32-eth0-rx-0-21swapper/320:54:163
29925060,0irq/32-eth0-rx-0-21swapper/320:50:433
27160,0ktimersoftd/20-21swapper/222:20:102
27160,0ktimersoftd/20-21swapper/220:28:542
149960,0migration/06637-21sh23:31:340
149960,0migration/032023-21munin-node20:59:060
149960,0migration/024519-21sh23:43:190
8315050,0irq/26-0000:00:6318-21kthreadcore21:39:052
8315050,0irq/26-0000:00:19657-21latency_hist19:23:472
8315050,0irq/26-0000:00:0-21swapper/221:09:022
50139950,0cyclictest0-21swapper/223:19:252
50129955,0cyclictest0-21swapper/123:14:011
50129955,0cyclictest0-21swapper/122:58:151
50129955,0cyclictest0-21swapper/121:53:271
50129955,0cyclictest0-21swapper/121:35:571
50129955,0cyclictest0-21swapper/121:19:521
50129950,0cyclictest0-21swapper/122:32:141
50119955,0cyclictest6877-21kworker/0:021:24:050
50119955,0cyclictest0-21swapper/021:31:180
339950,0migration/38133-21timerandwakeup18:49:113
339950,0migration/330772-21aten_r1power_en19:54:003
339950,0migration/31870-21unixbench-2d20:19:133
339950,0migration/315252-21kthreadcore20:04:063
339950,0migration/314122-21kthreadcore19:39:073
329950,0watchdog/37914-21kthreadcore19:14:083
329950,0watchdog/36020-21diskstats19:59:033
329950,0watchdog/316162-21grep19:19:083
329950,0watchdog/31538-21tr21:54:073
329950,0watchdog/310549-21kernelversion18:54:083
329950,0watchdog/30-21swapper/321:48:473
329950,0watchdog/30-21swapper/320:28:553
329950,0watchdog/30-21swapper/320:23:473
329950,0watchdog/30-21swapper/320:11:033
329950,0watchdog/30-21swapper/319:33:513
329950,0watchdog/30-21swapper/319:28:523
329950,0watchdog/30-21swapper/319:09:043
329950,0watchdog/30-21swapper/319:05:083
329950,0watchdog/30-21swapper/318:43:523
329950,0watchdog/30-21swapper/318:38:483
29935050,0irq/33-eth0-tx-6521-21ssh22:54:102
29935050,0irq/33-eth0-tx-4426-21ssh20:59:112
29935050,0irq/33-eth0-tx-367-21tr20:39:072
29935050,0irq/33-eth0-tx-2671-21munin-node20:24:120
29935050,0irq/33-eth0-tx-26447-21munin-node21:34:072
29935050,0irq/33-eth0-tx-23752-21memory21:29:072
29935050,0irq/33-eth0-tx-23361-21munin-node19:04:020
29935050,0irq/33-eth0-tx-20607-21munin-node23:04:052
29935050,0irq/33-eth0-tx-10585-21irqrtprio21:24:062
29935050,0irq/33-eth0-tx-0-21swapper/223:51:012
29935050,0irq/33-eth0-tx-0-21swapper/223:26:032
29935050,0irq/33-eth0-tx-0-21swapper/222:39:032
29935050,0irq/33-eth0-tx-0-21swapper/222:12:062
29935050,0irq/33-eth0-tx-0-21swapper/220:57:352
29935050,0irq/33-eth0-tx-0-21swapper/220:49:012
29935050,0irq/33-eth0-tx-0-21swapper/020:19:100
29935050,0irq/33-eth0-tx-0-21swapper/019:34:130
29935050,0irq/33-eth0-tx-0-21swapper/019:09:080
29935050,0irq/33-eth0-tx-0-21swapper/018:34:150
29925050,0irq/32-eth0-rx-0-21swapper/318:58:493
27150,0ktimersoftd/20-21swapper/222:43:562
259950,0migration/22970-21munin-node23:09:142
259950,0migration/226893-21sh22:48:592
259950,0migration/217683-21irqcore23:39:062
259950,0migration/215485-21kthreadcore18:34:072
259950,0migration/214279-21kthreadcore18:54:092
259950,0migration/211566-21kthreadcore23:34:072
259950,0migration/211566-21aten_r1power_fr21:44:032
249950,0watchdog/220458-21tr19:44:072
249950,0watchdog/215461-21cut22:59:112
19150,0ktimersoftd/10-21swapper/122:20:101
19150,0ktimersoftd/10-21swapper/121:31:321
19150,0ktimersoftd/10-21swapper/120:28:541
179950,0migration/131986-21kthreadcore18:44:091
179950,0migration/130256-21gltestperf23:09:081
179950,0migration/129368-21sh22:10:121
179950,0migration/124192-21munin-node22:44:081
179950,0migration/122323-21munin-node19:44:101
179950,0migration/116443-21sh21:25:451
179950,0migration/110241-21if_eth119:39:051
169950,0watchdog/117062-21cut22:04:111
169950,0watchdog/112985-21sh21:04:111
149950,0migration/09244-21kthreadcore18:29:060
149950,0migration/06764-21kthreadcore21:59:100
149950,0migration/06330-21fschecks_time19:59:040
149950,0migration/05053-21kthreadcore22:34:100
149950,0migration/04921-21smart_sda22:14:120
149950,0migration/04471-21kthreadcore21:19:120
149950,0migration/03578-21kthreadcore23:49:050
149950,0migration/031188-21kthreadcore22:49:070
149950,0migration/030662-21latency_hist19:53:470
149950,0migration/030448-21munin-node23:09:100
149950,0migration/028696-21kthreadcore19:29:060
149950,0migration/028531-21taskset20:54:360
149950,0migration/028127-21kthreadcore21:34:080
149950,0migration/026729-21kthreadcore22:09:090
149950,0migration/026664-21latency_hist22:48:470
149950,0migration/025128-21kthreadcore20:14:030
149950,0migration/024545-21kthreadcore23:04:060
149950,0migration/024510-21sh23:24:040
149950,0migration/024484-21kthreadcore21:49:090
149950,0migration/021873-21taskset22:08:200
149950,0migration/020321-21taskset20:12:530
149950,0migration/020273-21irqcore20:34:050
149950,0migration/019974-21kthreadcore21:09:060
149950,0migration/019310-21unixbench_singl20:04:140
149950,0migration/014765-21latency18:54:090
149950,0migration/014685-21kthreadcore20:49:080
149950,0migration/013694-21latency23:34:080
50139944,0cyclictest0-21swapper/219:39:012
50139943,0cyclictest6658-21awk19:59:062
50139943,0cyclictest6175-21awk21:59:082
50139943,0cyclictest3599-21lldpd19:07:572
50139943,0cyclictest31693-21awk19:09:042
50139943,0cyclictest2953-21awk19:34:032
50139943,0cyclictest24573-21awk19:49:032
50139943,0cyclictest23871-21snmpd19:23:562
50139943,0cyclictest20081-21awk20:34:042
50139943,0cyclictest0-21swapper/221:04:112
50129944,0cyclictest0-21swapper/123:47:171
50129944,0cyclictest0-21swapper/121:09:091
50129944,0cyclictest0-21swapper/120:52:361
50129943,0cyclictest6190-21awk21:59:081
50129943,0cyclictest3599-21lldpd23:33:471
50129943,0cyclictest3599-21lldpd23:28:471
50129943,0cyclictest3599-21lldpd20:10:291
50129943,0cyclictest3599-21lldpd19:59:281
50129943,0cyclictest23871-21snmpd20:59:121
50129943,0cyclictest17504-21awk23:39:051
50129940,0cyclictest0-21swapper/119:03:021
50119944,0cyclictest0-21swapper/023:56:580
50119944,0cyclictest0-21swapper/022:39:240
50119944,0cyclictest0-21swapper/021:42:010
50119943,0cyclictest23871-21snmpd19:53:260
50119943,0cyclictest12550-21awk21:44:070
50119943,0cyclictest12413-21sadc19:18:470
4140,0ktimersoftd/00-21swapper/019:40:280
31839240,0sleep00-21swapper/023:44:500
29935040,0irq/33-eth0-tx-9740-21diskmemload22:33:562
29935040,0irq/33-eth0-tx-27113-21kthreadcore21:14:062
29935040,0irq/33-eth0-tx-0-21swapper/223:54:042
29935040,0irq/33-eth0-tx-0-21swapper/223:46:022
29935040,0irq/33-eth0-tx-0-21swapper/223:28:582
29935040,0irq/33-eth0-tx-0-21swapper/223:14:062
29935040,0irq/33-eth0-tx-0-21swapper/222:30:012
29935040,0irq/33-eth0-tx-0-21swapper/222:24:062
29935040,0irq/33-eth0-tx-0-21swapper/222:13:582
29935040,0irq/33-eth0-tx-0-21swapper/221:51:252
29935040,0irq/33-eth0-tx-0-21swapper/221:19:012
29935040,0irq/33-eth0-tx-0-21swapper/020:29:000
29935040,0irq/33-eth0-tx-0-21swapper/019:24:030
29935040,0irq/33-eth0-tx-0-21swapper/018:59:000
259940,0migration/25621-21irqrtprio20:44:062
259940,0migration/25201-21latency_hist18:28:472
259940,0migration/231425-21fschecks_time19:54:032
259940,0migration/229687-21kthreadcore18:44:082
259940,0migration/228630-21latency_hist21:53:472
259940,0migration/220644-21hwlatdetect18:39:042
259940,0migration/218396-21kthreadcore20:04:062
259940,0migration/218241-21kthreadcore18:59:052
259940,0migration/21795-21timerwakeupswit20:19:132
249940,0watchdog/26195-21kthreadcore20:24:072
249940,0watchdog/22311-21fschecks_time18:49:002
249940,0watchdog/217063-21tr22:04:112
249940,0watchdog/20-21swapper/220:14:032
249940,0watchdog/20-21swapper/219:14:042
179940,0migration/17783-21unixbench-2d23:49:131
179940,0migration/17010-21kthreadcore19:34:061
179940,0migration/14940-21vmstat19:54:151
179940,0migration/1411-21munin-node19:29:081
179940,0migration/13950-21latency_hist20:43:471
179940,0migration/13630-21kthreadcore19:09:061
179940,0migration/13263-21fschecks_count20:24:031
179940,0migration/130321-21idleruntime-cro19:08:471
179940,0migration/128136-21irqcore22:49:061
179940,0migration/128077-21switchtime20:54:141
179940,0migration/126621-21kthreadcore19:49:061
179940,0migration/124935-21fschecks_time21:14:021
179940,0migration/123096-21kthreadcore18:39:061
179940,0migration/122915-21aten_r1power_vo23:23:591
179940,0migration/122189-21kernelversion23:04:061
179940,0migration/113565-21gltestperf19:19:041
179940,0migration/112491-21hwlatdetect21:44:071
179940,0migration/111087-21sh21:43:351
169940,0watchdog/15319-21ssh22:14:191
169940,0watchdog/12699-21ssh20:41:111
169940,0watchdog/12313-21fschecks_time18:49:001
169940,0watchdog/112844-21perl22:59:071
169940,0watchdog/10-21swapper/123:54:071
169940,0watchdog/10-21swapper/120:14:031
169940,0watchdog/10-21swapper/118:54:081
149940,0migration/07729-21fschecks_time21:04:040
149940,0migration/04601-21runrttasks21:58:240
149940,0migration/031555-21ksmtuned22:30:010
149940,0migration/022743-21kthreadcore18:39:060
149940,0migration/019496-21kthreadcore23:19:080
149940,0migration/013937-21kthreadcore22:59:110
149940,0migration/012663-21aten_r1power_cu19:19:010
149940,0migration/011320-21kthreadcore22:19:110
50139932,0cyclictest28085-21awk19:29:042
50129933,0cyclictest0-21swapper/120:05:091
50129932,0cyclictest3599-21lldpd18:32:561
50129932,0cyclictest28974-21awk20:19:031
50129932,0cyclictest23871-21snmpd22:35:271
50129932,0cyclictest13515-21awk18:34:041
50119933,0cyclictest6877-21kworker/0:022:24:110
50119933,0cyclictest25880-21kworker/0:019:43:520
50119933,0cyclictest11813-21kworker/0:221:14:210
50119933,0cyclictest11813-21kworker/0:220:44:380
50119933,0cyclictest10545-21kworker/0:123:13:570
50119933,0cyclictest10545-21kworker/0:122:53:530
50119932,0cyclictest29033-21tr20:39:060
50119932,0cyclictest23871-21snmpd18:48:480
29935030,0irq/33-eth0-tx-0-21swapper/018:44:020
259930,0migration/220109-21runrttasks20:10:242
169930,0watchdog/19740-21diskmemload20:36:191
169930,0watchdog/10-21swapper/121:54:071
169930,0watchdog/10-21swapper/119:14:041
169920,0watchdog/10-21swapper/119:23:481
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional