You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 08:40
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Sun Apr 28, 2024 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
700524031,0sleep10-21swapper/118:52:131
714723927,0sleep20-21swapper/218:53:502
718623825,0sleep00-21swapper/018:54:200
697723728,0sleep30-21swapper/318:51:513
72859966,0cyclictest0-21swapper/123:54:521
35160,0ktimersoftd/30-21swapper/323:06:073
339960,0migration/38238-21fschecks_time18:54:423
339960,0migration/3681-21sh22:05:573
339960,0migration/35118-21sh23:40:343
339960,0migration/332732-21runrttasks23:02:013
339960,0migration/311894-21sh22:30:113
339960,0migration/310182-21sh00:07:073
29935060,0irq/33-eth0-tx-6734-21memory20:54:562
29925060,0irq/32-eth0-rx-11267-21kthreadcore21:54:513
29925060,0irq/32-eth0-rx-0-21swapper/321:59:463
149960,0migration/029242-21sh23:37:010
149960,0migration/029172-21latency_hist20:29:320
149960,0migration/028033-21munin-node21:09:510
149960,0migration/026713-21sh22:20:130
149960,0migration/01326-21sh00:01:520
8315050,0irq/26-0000:00:4645-21perl19:49:452
8315050,0irq/26-0000:00:28663-21perf19:44:322
8315050,0irq/26-0000:00:28164-21python19:05:032
8315050,0irq/26-0000:00:25986-21latency_hist22:04:322
72869954,0cyclictest23871-21snmpd19:01:352
72859955,0cyclictest0-21swapper/123:18:461
72859955,0cyclictest0-21swapper/123:06:181
72859954,0cyclictest20472-21ps19:29:461
72859953,0cyclictest18556-21awk21:59:491
72859950,0cyclictest0-21swapper/122:39:401
72859950,0cyclictest0-21swapper/121:39:261
72859950,0cyclictest0-21swapper/100:17:241
339950,0migration/39926-21kthreadcore20:14:493
339950,0migration/38984-21kthreadcore21:34:503
339950,0migration/38142-21kthreadcore23:24:543
339950,0migration/331786-21munin-node20:49:563
339950,0migration/331299-21kthreadcore19:44:573
339950,0migration/328460-21fschecks_count23:19:443
339950,0migration/327849-21sh22:23:133
339950,0migration/32451-21kthreadcore21:49:523
339950,0migration/320611-21kthreadcore23:54:513
339950,0migration/319057-21ksmtuned21:20:373
339950,0migration/318578-21munin-node19:59:453
339950,0migration/318307-21kthreadcore18:59:513
339950,0migration/313642-21kthreadcore00:09:523
339950,0migration/310770-21taskset21:16:503
329950,0watchdog/328790-21diskmemload00:19:443
329950,0watchdog/328790-21diskmemload00:19:443
329950,0watchdog/328041-21nfs_client20:24:573
329950,0watchdog/319865-21cut20:44:493
329950,0watchdog/314332-21grep22:34:453
329950,0watchdog/310895-21irqrtprio20:59:493
29935050,0irq/33-eth0-tx-5756-21ssh22:45:102
29935050,0irq/33-eth0-tx-4350-21ssh22:28:272
29935050,0irq/33-eth0-tx-29658-21open_inodes21:44:562
29935050,0irq/33-eth0-tx-25124-21ssh22:02:442
29935050,0irq/33-eth0-tx-13081-21hddtemp_smartct21:19:482
29935050,0irq/33-eth0-tx-12103-21ssh22:30:442
29935050,0irq/33-eth0-tx-0-21swapper/223:53:312
29935050,0irq/33-eth0-tx-0-21swapper/223:34:532
29935050,0irq/33-eth0-tx-0-21swapper/223:07:212
29935050,0irq/33-eth0-tx-0-21swapper/222:21:312
29935050,0irq/33-eth0-tx-0-21swapper/221:25:032
29935050,0irq/33-eth0-tx-0-21swapper/221:09:272
29935050,0irq/33-eth0-tx-0-21swapper/200:20:592
29935050,0irq/33-eth0-tx-0-21swapper/200:20:592
29935050,0irq/33-eth0-tx-0-21swapper/200:05:512
29935050,0irq/33-eth0-tx-0-21swapper/019:54:360
29935050,0irq/33-eth0-tx-0-21swapper/019:14:570
29925050,0irq/32-eth0-rx-7393-21taskset20:14:163
29925050,0irq/32-eth0-rx-735-21ssh20:54:473
29925050,0irq/32-eth0-rx-7025-21ssh22:48:263
29925050,0irq/32-eth0-rx-4619-21fschecks_count19:14:493
29925050,0irq/32-eth0-rx-27905-21ssh00:18:143
29925050,0irq/32-eth0-rx-27629-21cut23:59:503
29925050,0irq/32-eth0-rx-22153-21diskstats23:34:453
29925050,0irq/32-eth0-rx-21315-21kthreadcore19:29:473
29925050,0irq/32-eth0-rx-20154-21basename23:14:493
29925050,0irq/32-eth0-rx-16519-21munin-node22:54:463
29925050,0irq/32-eth0-rx-11875-21fschecks_time22:14:473
29925050,0irq/32-eth0-rx-11830-21kthreadcore22:49:523
29925050,0irq/32-eth0-rx-0-21swapper/323:48:523
29925050,0irq/32-eth0-rx-0-21swapper/323:29:443
29925050,0irq/32-eth0-rx-0-21swapper/322:41:313
29925050,0irq/32-eth0-rx-0-21swapper/322:28:153
29925050,0irq/32-eth0-rx-0-21swapper/322:09:593
29925050,0irq/32-eth0-rx-0-21swapper/321:41:073
29925050,0irq/32-eth0-rx-0-21swapper/321:33:593
29925050,0irq/32-eth0-rx-0-21swapper/321:25:153
29925050,0irq/32-eth0-rx-0-21swapper/321:13:393
29925050,0irq/32-eth0-rx-0-21swapper/321:05:243
259950,0migration/29688-21sh22:11:152
259950,0migration/25667-21latency_hist21:34:322
259950,0migration/22968-21sh19:09:582
249950,0watchdog/26474-21kthreadcore21:14:532
249950,0watchdog/217256-21perl22:54:492
149950,0migration/08914-21latency_hist23:49:320
149950,0migration/08031-21sh23:05:010
149950,0migration/07006-21kthreadcore19:49:500
149950,0migration/05600-21irqrtprio20:34:560
149950,0migration/05341-21sh22:44:590
149950,0migration/03809-21fschecks_count21:14:460
149950,0migration/032692-21latency_hist20:09:320
149950,0migration/032525-21kthreadcore19:44:580
149950,0migration/031134-21taskset21:47:230
149950,0migration/031095-21kthreadcore22:59:490
149950,0migration/030278-21kthreadcore23:19:520
149950,0migration/029593-21kthreadcore22:04:510
149950,0migration/029274-21aten_r1power_vo19:09:440
149950,0migration/028148-21kthreadcore20:49:530
149950,0migration/027756-21sh00:17:490
149950,0migration/027227-21kthreadcore20:04:540
149950,0migration/026565-21kthreadcore20:24:550
149950,0migration/026448-21kthreadcore19:04:540
149950,0migration/023178-21kthreadcore21:59:520
149950,0migration/023050-21sh21:44:080
149950,0migration/022761-21kthreadcore21:24:500
149950,0migration/022409-21kthreadcore23:14:540
149950,0migration/019386-21kthreadcore23:54:500
149950,0migration/019167-21fschecks_count20:44:450
149950,0migration/017006-21kthreadcore22:14:550
149950,0migration/016776-21kthreadcore18:59:510
149950,0migration/015213-21taskset22:52:010
149950,0migration/015097-21cpuspeed_turbos20:19:470
149950,0migration/013951-21aten_r1power_po22:34:430
149950,0migration/011965-21fschecks_count00:09:470
149950,0migration/011284-21latency_hist20:39:320
149950,0migration/010314-21sh23:09:340
72869943,0cyclictest8814-21awk20:14:482
72869943,0cyclictest3599-21lldpd23:54:402
72869943,0cyclictest3599-21lldpd23:44:402
72869943,0cyclictest3599-21lldpd21:40:382
72869943,0cyclictest3599-21lldpd20:50:072
72869943,0cyclictest3599-21lldpd20:34:372
72869943,0cyclictest3599-21lldpd20:04:372
72869943,0cyclictest3599-21lldpd19:36:372
72869943,0cyclictest3599-21lldpd00:14:402
72869943,0cyclictest3262-21irqbalance20:59:372
72869943,0cyclictest28933-21awk23:19:472
72869943,0cyclictest25233-21sadc20:04:322
72869943,0cyclictest23193-21awk20:24:522
72869943,0cyclictest20246-21awk19:29:452
72869943,0cyclictest14165-21awk23:29:482
72869940,0cyclictest0-21swapper/223:24:462
72859944,0cyclictest0-21swapper/122:16:201
72859944,0cyclictest0-21swapper/121:23:121
72859944,0cyclictest0-21swapper/119:05:051
72859944,0cyclictest0-21swapper/100:21:211
72859944,0cyclictest0-21swapper/100:21:211
72859943,0cyclictest3599-21lldpd00:03:101
72859943,0cyclictest26911-21awk20:49:511
72859943,0cyclictest26584-21awk20:04:511
72859943,0cyclictest23871-21snmpd22:30:061
72859943,0cyclictest23871-21snmpd22:19:361
72859943,0cyclictest23871-21snmpd21:17:511
72859943,0cyclictest23871-21snmpd20:13:051
72859943,0cyclictest23871-21snmpd18:58:501
72859943,0cyclictest21245-21diskstats21:24:431
72859943,0cyclictest12372-21awk19:19:521
72859943,0cyclictest0-21swapper/123:31:131
72859942,0cyclictest8982-21iostat_ios22:49:501
72859942,0cyclictest7152-21python20:55:041
72859942,0cyclictest29970-21df_inode21:29:441
72859942,0cyclictest2858-21ntp_kernel_pll_20:29:551
72859942,0cyclictest28245-21timerandwakeup23:35:011
72859942,0cyclictest28115-21grep22:59:481
72859942,0cyclictest18563-21ssh23:13:501
72859942,0cyclictest15145-21sed21:39:461
72859942,0cyclictest11118-21aten2.4-expect19:54:461
72859941,0cyclictest741-21rt-features21:09:581
72859941,0cyclictest5300-21ssh23:41:011
72859941,0cyclictest32374-21rt-features22:04:561
72859941,0cyclictest32186-21ssh21:49:351
72859941,0cyclictest29747-21if_err_eth1.222:24:521
72859941,0cyclictest19301-21ssh00:13:191
72859941,0cyclictest15526-21ntp_states20:59:541
72859940,0cyclictest0-21swapper/121:08:321
72849943,0cyclictest23871-21snmpd19:41:500
72849943,0cyclictest23625-21awk22:39:480
72849943,0cyclictest10001-21kworker/0:021:29:490
72849943,0cyclictest10001-21kworker/0:020:00:070
72849940,0cyclictest0-21swapper/021:38:090
4140,0ktimersoftd/00-21swapper/020:59:410
339940,0migration/39046-21diskmemload23:49:423
329940,0watchdog/311820-21diskstats20:39:573
329940,0watchdog/30-21swapper/323:09:493
329940,0watchdog/30-21swapper/320:34:573
329940,0watchdog/30-21swapper/319:09:533
329940,0watchdog/30-21swapper/319:04:533
29935040,0irq/33-eth0-tx-7178-21ssh21:34:492
29935040,0irq/33-eth0-tx-25176-21ssh22:59:382
29935040,0irq/33-eth0-tx-17240-21kthreadcore00:09:542
29935040,0irq/33-eth0-tx-13857-21aten_r1power_en22:34:422
29935040,0irq/33-eth0-tx-0-21swapper/223:59:512
29935040,0irq/33-eth0-tx-0-21swapper/223:40:052
29935040,0irq/33-eth0-tx-0-21swapper/222:49:322
29935040,0irq/33-eth0-tx-0-21swapper/222:39:402
29935040,0irq/33-eth0-tx-0-21swapper/221:54:422
29935040,0irq/33-eth0-tx-0-21swapper/221:49:382
29935040,0irq/33-eth0-tx-0-21swapper/221:09:362
29925040,0irq/32-eth0-rx-0-21swapper/321:44:443
29925040,0irq/32-eth0-rx-0-21swapper/319:24:123
259940,0migration/24810-21fschecks_time19:14:492
259940,0migration/217404-21kthreadcore20:19:562
259940,0migration/215087-21kthreadcore22:14:542
259940,0migration/210264-21latency_hist23:09:322
249940,0watchdog/225508-21sensors23:15:012
249940,0watchdog/225110-21awk20:45:012
249940,0watchdog/20-21swapper/220:29:452
249940,0watchdog/20-21swapper/219:44:572
149940,0migration/07680-21kthreadcore23:24:540
149940,0migration/04440-21proc_pri23:39:580
149940,0migration/024451-21sh22:58:190
149940,0migration/022714-21kthreadcore19:29:470
149940,0migration/014264-21kthreadcore21:54:520
149940,0migration/012700-21fschecks_count21:19:480
149940,0migration/010204-21kthreadcore18:54:460
72869933,0cyclictest3599-21lldpd19:57:062
72869933,0cyclictest3599-21lldpd18:54:342
72869932,0cyclictest3599-21lldpd20:10:372
72869932,0cyclictest3599-21lldpd19:19:362
72869930,0cyclictest0-21swapper/219:26:022
72859932,0cyclictest23871-21snmpd20:19:391
72859932,0cyclictest11454-21aten2.4-expect20:39:531
72859931,0cyclictest8950-21grep23:49:321
72859931,0cyclictest8597-21date21:54:321
72859931,0cyclictest7587-21latency_hist20:14:321
72859931,0cyclictest6901-21seq23:44:371
72859931,0cyclictest4125-21cat23:24:321
72859931,0cyclictest4110-21chrt20:34:341
72859931,0cyclictest4042-21basename19:14:451
72859931,0cyclictest3678-21cat19:49:321
72859931,0cyclictest31130-21wc22:44:321
72859931,0cyclictest29069-21latency_hist19:09:331
72859931,0cyclictest28841-21latency_hist19:44:321
72859931,0cyclictest27874-21aten_r1power_cu23:19:411
72859931,0cyclictest26989-21latency_hist19:39:321
72859931,0cyclictest264232sleep119:35:181
72859931,0cyclictest2517-21ssh00:04:341
72859931,0cyclictest23393-21sort21:44:321
72859931,0cyclictest2311-21date22:09:321
72859931,0cyclictest22021-21cat20:24:321
72859931,0cyclictest19007-21diskstats20:44:441
72859931,0cyclictest18383-21basename19:59:441
72859931,0cyclictest18345-21cut19:24:321
72859931,0cyclictest16410-21ssh22:54:361
72859931,0cyclictest14641-21wc18:59:331
72859931,0cyclictest13788-21ssh22:34:361
72849933,0cyclictest3627-21kworker/0:200:19:330
72849933,0cyclictest3627-21kworker/0:200:19:330
72849933,0cyclictest3414-21kworker/0:222:29:350
72849933,0cyclictest3414-21kworker/0:222:24:370
72849933,0cyclictest19228-21kworker/0:219:24:330
72849933,0cyclictest17473-21kworker/0:023:44:330
72849933,0cyclictest17473-21kworker/0:023:29:370
72849933,0cyclictest17473-21kworker/0:000:04:330
72849933,0cyclictest10001-21kworker/0:022:09:330
72849933,0cyclictest10001-21kworker/0:021:49:330
72849933,0cyclictest10001-21kworker/0:021:04:390
72849933,0cyclictest10001-21kworker/0:020:54:330
72849933,0cyclictest10001-21kworker/0:020:14:330
329930,0watchdog/30-21swapper/320:29:453
329930,0watchdog/30-21swapper/320:04:453
329930,0watchdog/30-21swapper/319:49:493
329930,0watchdog/30-21swapper/319:34:413
29935030,0irq/33-eth0-tx-0-21swapper/019:34:320
249930,0watchdog/20-21swapper/220:40:052
149930,0migration/011208-21runrttasks19:19:390
329920,0watchdog/30-21swapper/320:19:373
329920,0watchdog/30-21swapper/319:54:453
329920,0watchdog/30-21swapper/319:39:333
329920,0watchdog/30-21swapper/319:24:453
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional