You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-02 - 07:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Wed Jul 02, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1054924229,0sleep30-21swapper/318:28:073
1036023723,0sleep00-21swapper/018:25:520
1040523627,0sleep10-21swapper/118:26:251
1039823526,0sleep20-21swapper/218:26:222
329980,0watchdog/30-21swapper/321:34:563
329970,0watchdog/31795-21ssh20:53:443
329970,0watchdog/30-21swapper/321:01:203
329970,0watchdog/30-21swapper/320:48:523
329970,0watchdog/30-21swapper/319:43:563
29925070,0irq/32-eth0-rx-0-21swapper/322:52:113
29925070,0irq/32-eth0-rx-0-21swapper/321:53:583
29925070,0irq/32-eth0-rx-0-21swapper/321:08:543
29925070,0irq/32-eth0-rx-0-21swapper/320:43:553
106749970,0cyclictest1241-21kthreadcore19:48:561
339960,0migration/315961-21sh20:43:023
329960,0watchdog/30-21swapper/323:48:363
329960,0watchdog/30-21swapper/323:43:363
329960,0watchdog/30-21swapper/323:38:043
329960,0watchdog/30-21swapper/323:28:403
329960,0watchdog/30-21swapper/323:25:523
329960,0watchdog/30-21swapper/323:18:483
329960,0watchdog/30-21swapper/323:09:123
329960,0watchdog/30-21swapper/323:04:283
329960,0watchdog/30-21swapper/323:01:363
329960,0watchdog/30-21swapper/322:56:523
329960,0watchdog/30-21swapper/322:19:003
329960,0watchdog/30-21swapper/321:32:083
329960,0watchdog/30-21swapper/321:25:083
329960,0watchdog/30-21swapper/321:20:323
329960,0watchdog/30-21swapper/321:17:043
329960,0watchdog/30-21swapper/320:21:403
329960,0watchdog/30-21swapper/319:57:403
329960,0watchdog/30-21swapper/319:57:403
329960,0watchdog/30-21swapper/319:08:563
329960,0watchdog/30-21swapper/318:53:563
29925060,0irq/32-eth0-rx-929-21cut21:48:523
29925060,0irq/32-eth0-rx-28304-21tr20:28:573
29925060,0irq/32-eth0-rx-0-21swapper/323:39:443
29925060,0irq/32-eth0-rx-0-21swapper/322:44:533
29925060,0irq/32-eth0-rx-0-21swapper/322:36:463
29925060,0irq/32-eth0-rx-0-21swapper/322:28:553
29925060,0irq/32-eth0-rx-0-21swapper/322:14:233
29925060,0irq/32-eth0-rx-0-21swapper/322:09:103
29925060,0irq/32-eth0-rx-0-21swapper/322:03:563
29925060,0irq/32-eth0-rx-0-21swapper/322:02:383
29925060,0irq/32-eth0-rx-0-21swapper/321:43:473
19160,0ktimersoftd/10-21swapper/121:46:231
106749960,0cyclictest0-21swapper/121:30:471
8315050,0irq/26-0000:00:11165-21awk23:28:352
4150,0ktimersoftd/00-21swapper/021:26:080
339950,0migration/327906-21kthreadcore20:08:533
339950,0migration/325383-21munin-node18:38:493
339950,0migration/322928-21kthreadcore20:03:553
339950,0migration/322671-21runrttasks21:03:533
339950,0migration/318180-21munin-node18:33:493
339950,0migration/316734-21missed_timers18:28:573
339950,0migration/312037-21munin-node19:13:533
329950,0watchdog/321048-21kthreadcore19:18:563
329950,0watchdog/30-21swapper/323:53:363
329950,0watchdog/30-21swapper/323:13:403
329950,0watchdog/30-21swapper/322:23:483
329950,0watchdog/30-21swapper/321:39:043
329950,0watchdog/30-21swapper/320:23:483
329950,0watchdog/30-21swapper/320:13:483
329950,0watchdog/30-21swapper/320:01:043
329950,0watchdog/30-21swapper/319:50:123
329950,0watchdog/30-21swapper/319:38:483
329950,0watchdog/30-21swapper/319:33:363
329950,0watchdog/30-21swapper/319:28:363
329950,0watchdog/30-21swapper/319:23:443
329950,0watchdog/30-21swapper/319:03:483
329950,0watchdog/30-21swapper/318:59:443
329950,0watchdog/30-21swapper/318:49:013
329950,0watchdog/30-21swapper/318:44:293
29935050,0irq/33-eth0-tx-32598-21munin-node19:28:590
29935050,0irq/33-eth0-tx-21088-21kthreadcore21:59:012
29935050,0irq/33-eth0-tx-0-21swapper/020:28:410
29935050,0irq/33-eth0-tx-0-21swapper/020:08:490
29925050,0irq/32-eth0-rx-0-21swapper/322:38:493
29925050,0irq/32-eth0-rx-0-21swapper/320:33:543
249950,0watchdog/221045-21kthreadcore19:18:562
179950,0migration/119348-21sh21:19:001
179950,0migration/118275-21fschecks_time21:58:561
179950,0migration/113573-21switchtime18:49:031
179950,0migration/11236-21irqcore18:43:551
169950,0watchdog/118903-21perl19:18:521
149950,0migration/09980-21kthreadcore20:18:540
149950,0migration/09281-21kthreadcore22:48:580
149950,0migration/08367-21fschecks_time20:38:510
149950,0migration/07011-21kthreadcore21:13:550
149950,0migration/04256-21fschecks_count19:08:500
149950,0migration/03820-21sh21:12:590
149950,0migration/03460-21kthreadcore21:28:570
149950,0migration/029212-21fschecks_time23:38:500
149950,0migration/028829-21kthreadcore23:18:540
149950,0migration/026070-21kthreadcore22:58:570
149950,0migration/025899-21nfs_client22:18:580
149950,0migration/023369-21kthreadcore23:33:530
149950,0migration/02245-21kthreadcore21:48:560
149950,0migration/022292-21cpuspeed_turbos22:38:500
149950,0migration/021597-21latency_hist18:58:360
149950,0migration/02140-21ntp_offset22:24:010
149950,0migration/021234-21kthreadcore23:13:590
149950,0migration/020864-21sh22:36:100
149950,0migration/020705-21irqcore21:03:520
149950,0migration/018768-21fschecks_count19:18:520
149950,0migration/015620-21smart_sdb23:08:580
149950,0migration/014810-21sh21:38:090
149950,0migration/014711-21kthreadcore23:48:540
149950,0migration/012714-21munin-node22:13:550
149950,0migration/011055-21latency_hist19:58:360
149950,0migration/011055-21latency_hist19:58:360
106789954,0cyclictest32712-21ps20:33:572
106789950,0cyclictest0-21swapper/222:44:532
106789950,0cyclictest0-21swapper/220:51:412
106749955,0cyclictest0-21swapper/122:22:481
106749955,0cyclictest0-21swapper/121:17:411
106749954,0cyclictest3599-21lldpd18:54:531
106749950,0cyclictest0-21swapper/121:39:171
106749950,0cyclictest0-21swapper/119:04:001
106669955,0cyclictest0-21swapper/023:57:100
106669955,0cyclictest0-21swapper/022:46:410
106669955,0cyclictest0-21swapper/021:45:500
106669954,0cyclictest3262-21irqbalance22:09:000
106669954,0cyclictest16411-21awk21:38:550
106669950,0cyclictest0-21swapper/023:32:440
106669950,0cyclictest0-21swapper/021:00:160
106669950,0cyclictest0-21swapper/020:07:270
29935040,0irq/33-eth0-tx-21924-21diskmemload23:38:552
29935040,0irq/33-eth0-tx-21924-21diskmemload22:13:412
29935040,0irq/33-eth0-tx-21924-21diskmemload21:33:402
29935040,0irq/33-eth0-tx-21751-21ssh21:23:472
29935040,0irq/33-eth0-tx-19976-21ssh23:50:412
29935040,0irq/33-eth0-tx-10518-21iostat_ios23:08:512
29935040,0irq/33-eth0-tx-0-21swapper/223:54:502
29935040,0irq/33-eth0-tx-0-21swapper/223:18:392
29935040,0irq/33-eth0-tx-0-21swapper/222:54:052
29935040,0irq/33-eth0-tx-0-21swapper/222:49:212
29935040,0irq/33-eth0-tx-0-21swapper/222:34:212
29935040,0irq/33-eth0-tx-0-21swapper/222:30:262
29935040,0irq/33-eth0-tx-0-21swapper/222:19:162
29935040,0irq/33-eth0-tx-0-21swapper/221:49:252
29935040,0irq/33-eth0-tx-0-21swapper/018:43:540
29935040,0irq/33-eth0-tx-0-21swapper/018:39:160
29935040,0irq/33-eth0-tx-0-21swapper/018:28:590
259940,0migration/27810-21runrttasks23:05:162
259940,0migration/27686-21kthreadcore20:53:532
259940,0migration/26046-21kthreadcore23:43:552
259940,0migration/25662-21kthreadcore22:08:552
259940,0migration/25373-21idleruntime21:13:522
259940,0migration/231930-21kthreadcore19:48:552
259940,0migration/230099-21diskmemload21:44:012
259940,0migration/221425-21kthreadcore21:03:532
259940,0migration/218878-21fschecks_count20:03:512
259940,0migration/213348-21kthreadcore20:38:562
249940,0watchdog/22694-21sendmail_mailst19:04:002
249940,0watchdog/21980-21memory22:24:002
249940,0watchdog/2176072sleep219:14:362
249940,0watchdog/211395-21ksmtuned23:28:402
249940,0watchdog/20-21swapper/223:36:562
249940,0watchdog/20-21swapper/222:05:562
249940,0watchdog/20-21swapper/220:59:002
249940,0watchdog/20-21swapper/219:08:482
179940,0migration/16567-21smart_sda21:49:011
179940,0migration/13886-21hddtemp_smartct23:23:521
179940,0migration/131798-21sh23:01:131
179940,0migration/131773-21kthreadcore21:08:551
179940,0migration/131593-21kthreadcore22:23:591
179940,0migration/131464-21mdstat20:08:541
179940,0migration/130475-21kthreadcore23:38:541
179940,0migration/130006-21kthreadcore20:48:551
179940,0migration/128069-21kthreadcore19:23:551
179940,0migration/123015-21hddtemp_smartct22:38:521
179940,0migration/12151-21sh23:22:271
179940,0migration/115526-21kthreadcore20:58:551
179940,0migration/114350-21runrttasks19:38:001
179940,0migration/114344-21gltestperf22:33:501
179940,0migration/11319-21kthreadcore22:43:591
179940,0migration/111265-21kthreadcore20:18:551
179940,0migration/110453-21kthreadcore21:34:011
169940,0watchdog/19461-21ssh20:56:121
169940,0watchdog/14895-21ntp_states20:34:001
169940,0watchdog/14327-21df_abs19:53:521
169940,0watchdog/14327-21df_abs19:53:521
169940,0watchdog/116900-21cat22:53:561
169940,0watchdog/113934-21ssh22:49:321
169940,0watchdog/111311-21ssh23:44:441
169940,0watchdog/10-21swapper/123:53:561
169940,0watchdog/10-21swapper/122:13:521
169940,0watchdog/10-21swapper/118:28:531
149940,0migration/09838-21runrttasks23:24:400
149940,0migration/07273-21sh20:38:280
149940,0migration/05693-21kthreadcore20:53:530
149940,0migration/024646-21taskset18:34:340
149940,0migration/018394-21kthreadcore20:43:550
149940,0migration/016902-21if_eth1.222:53:560
149940,0migration/01080-21kthreadcore19:03:570
106789944,0cyclictest0-21swapper/223:13:442
106789944,0cyclictest0-21swapper/222:59:522
106789944,0cyclictest0-21swapper/221:19:262
106789943,0cyclictest3599-21lldpd20:28:542
106789943,0cyclictest3599-21lldpd19:58:532
106789943,0cyclictest3599-21lldpd19:28:532
106789943,0cyclictest3599-21lldpd18:51:232
106789943,0cyclictest3599-21lldpd18:39:232
106789943,0cyclictest31402-21awk21:28:542
106789943,0cyclictest23871-21snmpd21:42:012
106789943,0cyclictest23871-21snmpd21:09:462
106789943,0cyclictest23871-21snmpd20:25:312
106789943,0cyclictest23871-21snmpd20:19:162
106789943,0cyclictest23871-21snmpd20:13:462
106789943,0cyclictest23871-21snmpd20:10:312
106789943,0cyclictest23871-21snmpd19:44:462
106789943,0cyclictest23871-21snmpd19:35:312
106789943,0cyclictest23871-21snmpd19:24:162
106789943,0cyclictest23871-21snmpd18:45:012
106789943,0cyclictest23138-21awk22:38:532
106789943,0cyclictest22759-21awk18:58:542
106789943,0cyclictest17581-21awk20:43:542
106789943,0cyclictest11943-21awk18:28:532
106789943,0cyclictest10067-21awk21:53:552
106789940,0cyclictest0-21swapper/219:54:282
106789940,0cyclictest0-21swapper/219:54:282
106749944,0cyclictest0-21swapper/122:07:021
106749943,0cyclictest9977-21awk21:53:551
106749943,0cyclictest3599-21lldpd23:51:101
106749943,0cyclictest3599-21lldpd23:34:091
106749943,0cyclictest3599-21lldpd21:04:241
106749943,0cyclictest3599-21lldpd20:47:251
106749943,0cyclictest3599-21lldpd20:31:241
106749943,0cyclictest3599-21lldpd20:25:241
106749943,0cyclictest3599-21lldpd19:14:221
106749943,0cyclictest22728-21awk21:23:511
106749943,0cyclictest18978-21awk23:13:561
106749943,0cyclictest10470-21awk23:08:511
106749940,0cyclictest0-21swapper/123:06:291
106749940,0cyclictest0-21swapper/120:06:351
106669944,0cyclictest0-21swapper/021:59:430
106669943,0cyclictest3599-21lldpd23:44:090
106669943,0cyclictest3599-21lldpd22:03:580
106669943,0cyclictest3599-21lldpd21:19:260
106669943,0cyclictest3262-21irqbalance22:30:100
106669940,0cyclictest0-21swapper/018:52:250
8315030,0irq/26-0000:00:0-21swapper/219:39:032
4130,0ktimersoftd/00-21swapper/019:43:490
29935030,0irq/33-eth0-tx-0-21swapper/019:58:480
29935030,0irq/33-eth0-tx-0-21swapper/019:16:230
169930,0watchdog/13894-21sh19:08:481
169930,0watchdog/10-21swapper/120:13:481
169930,0watchdog/10-21swapper/119:58:361
169930,0watchdog/10-21swapper/119:43:561
169930,0watchdog/10-21swapper/118:33:491
149930,0migration/0468-21sh23:03:450
106789933,0cyclictest3599-21lldpd18:53:532
106789932,0cyclictest3599-21lldpd18:33:532
106749932,0cyclictest8182-21tune2fs20:38:511
106749932,0cyclictest5835-21awk22:28:531
106749932,0cyclictest3599-21lldpd23:28:381
106749932,0cyclictest31710-21tail18:39:031
106749932,0cyclictest23871-21snmpd22:09:531
106749932,0cyclictest22871-21awk18:58:551
106749932,0cyclictest1858-21kthreadcore19:28:561
106749932,0cyclictest16808-21kthreadcore19:38:541
106669932,0cyclictest7467-21munin-node19:33:510
106669932,0cyclictest3262-21irqbalance19:38:500
106669932,0cyclictest31591-21sensors19:24:000
106669932,0cyclictest30277-21awk19:48:540
106669932,0cyclictest266002sleep020:48:550
106669932,0cyclictest17201-21sadc21:58:350
106669930,0cyclictest14941-21diskstats18:53:500
106669920,0cyclictest0-21swapper/020:23:410
106669920,0cyclictest0-21swapper/020:13:440
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional