You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-16 - 08:17
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Tue Apr 16, 2024 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1750223928,0sleep00-21swapper/018:53:060
1744923724,0sleep10-21swapper/118:52:311
1738123727,0sleep20-21swapper/218:51:442
1757423228,0sleep30-21swapper/318:54:013
8315060,0irq/26-0000:00:28627-21crond19:25:163
339960,0migration/39200-21sh22:15:333
339960,0migration/34998-21sh23:25:393
339960,0migration/330325-21sh23:23:123
339960,0migration/323320-21sh23:36:133
339960,0migration/317863-21sh21:42:483
29925060,0irq/32-eth0-rx-4764-21perl21:55:293
29925060,0irq/32-eth0-rx-26964-21cut22:25:363
29925060,0irq/32-eth0-rx-18455-21df19:40:223
29925060,0irq/32-eth0-rx-15705-21kthreadcore22:55:353
29925060,0irq/32-eth0-rx-14322-21kthreadcore23:50:373
19160,0ktimersoftd/10-21swapper/121:23:181
177589960,0cyclictest0-21swapper/223:30:342
177589960,0cyclictest0-21swapper/220:00:342
177579960,0cyclictest0-21swapper/123:00:061
177579960,0cyclictest0-21swapper/121:52:281
8315050,0irq/26-0000:00:2065-21md0_raid119:00:163
339950,0migration/39317-21kthreadcore20:15:413
339950,0migration/330315-21kthreadcore00:00:363
339950,0migration/328582-21taskset20:07:463
339950,0migration/327154-21kthreadcore23:40:343
339950,0migration/325135-21latency_hist19:00:163
339950,0migration/324013-21kthreadcore00:15:363
339950,0migration/323085-21kthreadcore23:00:333
339950,0migration/31945-21kthreadcore00:20:363
339950,0migration/318560-21kthreadcore20:45:343
339950,0migration/317100-21munin-node20:20:383
339950,0migration/314716-21taskset23:31:263
339950,0migration/313656-21kthreadcore00:10:343
339950,0migration/313309-21missed_timers19:55:393
339950,0migration/312873-21aten_r1power_cu22:00:293
329950,0watchdog/37773-21cpu20:40:283
329950,0watchdog/330261-21irqrtprio19:25:293
329950,0watchdog/32108-21grep20:35:363
329950,0watchdog/319837-21perl21:45:323
329950,0watchdog/30-21swapper/323:45:403
329950,0watchdog/30-21swapper/319:05:293
29935050,0irq/33-eth0-tx-7525-21cat21:00:362
29935050,0irq/33-eth0-tx-7228-21grep23:10:362
29935050,0irq/33-eth0-tx-3714-21munin-node19:30:240
29935050,0irq/33-eth0-tx-31401-21munin-node23:25:442
29935050,0irq/33-eth0-tx-27104-21ssh21:49:202
29935050,0irq/33-eth0-tx-23871-21snmpd21:25:092
29935050,0irq/33-eth0-tx-18740-21grep23:35:352
29935050,0irq/33-eth0-tx-1522-21ssh00:01:192
29935050,0irq/33-eth0-tx-14782-21munin-node20:45:290
29935050,0irq/33-eth0-tx-11729-21munin-node00:10:392
29935050,0irq/33-eth0-tx-0-21swapper/223:50:472
29935050,0irq/33-eth0-tx-0-21swapper/223:22:162
29935050,0irq/33-eth0-tx-0-21swapper/223:18:172
29935050,0irq/33-eth0-tx-0-21swapper/221:55:442
29935050,0irq/33-eth0-tx-0-21swapper/200:09:452
29925050,0irq/32-eth0-rx-6424-21python19:50:353
29925050,0irq/32-eth0-rx-6386-21runrttasks21:06:453
29925050,0irq/32-eth0-rx-32758-21aten_r1power_en22:30:283
29925050,0irq/32-eth0-rx-29236-21kthreadcore22:45:343
29925050,0irq/32-eth0-rx-25667-21munin-node21:30:303
29925050,0irq/32-eth0-rx-22746-21ssh22:22:303
29925050,0irq/32-eth0-rx-21969-21chrt23:19:113
29925050,0irq/32-eth0-rx-11744-21expr23:10:443
29925050,0irq/32-eth0-rx-0-21swapper/323:55:313
29925050,0irq/32-eth0-rx-0-21swapper/323:05:293
29925050,0irq/32-eth0-rx-0-21swapper/322:35:353
29925050,0irq/32-eth0-rx-0-21swapper/322:05:303
29925050,0irq/32-eth0-rx-0-21swapper/321:35:473
29925050,0irq/32-eth0-rx-0-21swapper/321:21:263
29925050,0irq/32-eth0-rx-0-21swapper/321:10:383
259950,0migration/2949-21munin-node19:05:262
259950,0migration/232436-21kthreadcore21:50:382
259950,0migration/226502-21hddtemp_smartct20:30:332
259950,0migration/216193-21irqrtprio20:45:332
259950,0migration/212563-21kthreadcore19:35:282
249950,0watchdog/227144-21ssh23:01:082
249950,0watchdog/219101-21fschecks_count23:55:322
19150,0ktimersoftd/10-21swapper/122:49:221
17841250,0sleep318022-21cpuspeed_turbos22:40:283
177579955,0cyclictest0-21swapper/122:39:081
177579954,0cyclictest0-21swapper/119:35:371
177579950,0cyclictest0-21swapper/123:23:071
177579950,0cyclictest0-21swapper/122:54:251
177579950,0cyclictest0-21swapper/122:32:211
177579950,0cyclictest0-21swapper/121:00:451
177579950,0cyclictest0-21swapper/119:11:011
177579950,0cyclictest0-21swapper/119:01:521
177579950,0cyclictest0-21swapper/100:17:211
149950,0migration/09790-21sh22:51:110
149950,0migration/08102-21df_abs23:30:290
149950,0migration/07323-21latency_hist19:55:160
149950,0migration/0494-21irqrtprio23:25:360
149950,0migration/04117-21fschecks_time00:05:310
149950,0migration/03855-21kthreadcore22:30:360
149950,0migration/0368-21aten_r1power_vo20:35:290
149950,0migration/03206-21unixbench_singl23:05:450
149950,0migration/030639-21kthreadcore19:00:310
149950,0migration/030379-21irqrtprio19:25:290
149950,0migration/029075-21kthreadcore22:25:370
149950,0migration/027381-21kthreadcore19:20:330
149950,0migration/027214-21kthreadcore23:40:340
149950,0migration/024392-21kthreadcore20:50:370
149950,0migration/024352-21kthreadcore21:45:380
149950,0migration/024154-21sendmail_mailqu18:55:370
149950,0migration/023398-21fschecks_time23:20:300
149950,0migration/023128-21kthreadcore19:40:280
149950,0migration/022179-21kthreadcore20:25:350
149950,0migration/02059-21munin-node21:35:320
149950,0migration/019954-21kthreadcore20:00:350
149950,0migration/019641-21kthreadcore22:40:340
149950,0migration/015188-21taskset21:05:310
149950,0migration/011569-21kthreadcore22:15:330
149950,0migration/010687-21cpuspeed_turbos21:40:310
149950,0migration/010062-21rt-features20:15:440
339940,0migration/330520-21irqcore20:55:363
329940,0watchdog/39847-21proc_pri19:30:373
329940,0watchdog/320606-21munin-node20:00:363
329940,0watchdog/30-21swapper/320:33:483
329940,0watchdog/30-21swapper/319:45:283
329940,0watchdog/30-21swapper/319:35:283
29935040,0irq/33-eth0-tx-29272-21munin-node20:55:362
29935040,0irq/33-eth0-tx-28979-21diskmemload23:05:362
29935040,0irq/33-eth0-tx-25331-21munin-node21:30:332
29935040,0irq/33-eth0-tx-19819-21ssh22:01:212
29935040,0irq/33-eth0-tx-15635-40yum-updatesd-he19:15:280
29935040,0irq/33-eth0-tx-10409-21munin-node21:40:492
29935040,0irq/33-eth0-tx-0-21swapper/223:40:232
29935040,0irq/33-eth0-tx-0-21swapper/222:55:332
29935040,0irq/33-eth0-tx-0-21swapper/222:25:192
29935040,0irq/33-eth0-tx-0-21swapper/222:20:362
29935040,0irq/33-eth0-tx-0-21swapper/222:10:532
29935040,0irq/33-eth0-tx-0-21swapper/221:15:252
29935040,0irq/33-eth0-tx-0-21swapper/221:11:082
29935040,0irq/33-eth0-tx-0-21swapper/200:20:332
29925040,0irq/32-eth0-rx-0-21swapper/322:51:383
29925040,0irq/32-eth0-rx-0-21swapper/322:10:243
29925040,0irq/32-eth0-rx-0-21swapper/321:51:073
29925040,0irq/32-eth0-rx-0-21swapper/321:26:053
29925040,0irq/32-eth0-rx-0-21swapper/321:15:343
29925040,0irq/32-eth0-rx-0-21swapper/321:00:293
29925040,0irq/32-eth0-rx-0-21swapper/320:10:263
29925040,0irq/32-eth0-rx-0-21swapper/319:10:223
29925040,0irq/32-eth0-rx-0-21swapper/300:06:083
259940,0migration/25948-21kthreadcore22:50:342
259940,0migration/24504-21kthreadcore20:35:372
259940,0migration/23826-21aten_r1power_po19:30:222
259940,0migration/23500-21kthreadcore22:30:362
259940,0migration/231442-21kthreadcore19:25:302
259940,0migration/229667-21kthreadcore22:45:342
259940,0migration/229132-21kthreadcore19:45:292
259940,0migration/221583-21aten_r1power_cu22:05:282
259940,0migration/21842-21cpu21:35:302
259940,0migration/214892-21kthreadcore22:35:352
259940,0migration/211462-21kthreadcore20:40:352
249940,0watchdog/229452-21df_inode20:10:282
249940,0watchdog/2107682sleep220:19:242
249940,0watchdog/20-21swapper/222:40:482
249940,0watchdog/20-21swapper/219:20:332
249940,0watchdog/20-21swapper/200:15:362
177589944,0cyclictest0-21swapper/221:05:312
177589943,0cyclictest3599-21lldpd19:58:592
177589943,0cyclictest3599-21lldpd19:12:592
177589943,0cyclictest2512-21awk23:45:372
177589943,0cyclictest22919-21awk20:05:312
177589943,0cyclictest17733-21awk21:25:342
177589940,0cyclictest0-21swapper/222:15:292
177589940,0cyclictest0-21swapper/219:15:282
177579944,0cyclictest0-21swapper/123:43:181
177579944,0cyclictest0-21swapper/121:46:091
177579944,0cyclictest0-21swapper/120:08:051
177579944,0cyclictest0-21swapper/119:46:201
177579943,0cyclictest3599-21lldpd22:00:301
177579943,0cyclictest3599-21lldpd20:56:281
177579943,0cyclictest3599-21lldpd18:57:591
177579943,0cyclictest32505-21awk23:25:341
177579943,0cyclictest28159-21awk00:00:331
177579943,0cyclictest26689-21awk20:30:361
177579943,0cyclictest23871-21snmpd21:56:091
177579943,0cyclictest23871-21snmpd21:18:091
177579943,0cyclictest23871-21snmpd20:11:381
177579943,0cyclictest23871-21snmpd19:59:081
177579943,0cyclictest23122-21awk20:50:351
177579943,0cyclictest22697-21awk22:05:351
177579943,0cyclictest17889-21awk21:25:351
177579943,0cyclictest16601-21awk22:20:361
177579943,0cyclictest15252-21awk21:05:311
177579943,0cyclictest1355-21awk20:35:341
177579940,0cyclictest0-21swapper/119:30:151
177569944,0cyclictest0-21swapper/022:10:280
177569944,0cyclictest0-21swapper/022:08:440
177569943,0cyclictest5198-21awk21:55:310
177569943,0cyclictest3599-21lldpd22:20:320
177569943,0cyclictest3599-21lldpd21:12:590
177569943,0cyclictest3599-21lldpd21:01:290
177569943,0cyclictest3262-21irqbalance23:46:500
177569943,0cyclictest3262-21irqbalance23:16:500
177569943,0cyclictest3262-21irqbalance22:45:200
177569943,0cyclictest3262-21irqbalance22:36:100
177569943,0cyclictest3262-21irqbalance21:50:500
177569943,0cyclictest3262-21irqbalance21:17:200
177569943,0cyclictest3262-21irqbalance20:55:200
177569943,0cyclictest3262-21irqbalance20:42:500
177569943,0cyclictest3262-21irqbalance20:30:200
177569943,0cyclictest3262-21irqbalance20:22:100
177569943,0cyclictest3262-21irqbalance20:05:300
177569943,0cyclictest3262-21irqbalance19:55:200
177569943,0cyclictest3262-21irqbalance19:46:400
177569943,0cyclictest3262-21irqbalance19:35:200
177569943,0cyclictest3262-21irqbalance19:10:500
177569943,0cyclictest3262-21irqbalance19:05:300
177569943,0cyclictest29854-21awk00:20:330
177569943,0cyclictest23871-21snmpd21:32:540
177569943,0cyclictest21285-21sadc22:05:150
177569943,0cyclictest17513-21awk23:35:330
177569943,0cyclictest11183-21awk23:50:350
179940,0migration/13216-21aten2.4-expect00:05:271
179940,0migration/129101-21aten2.4-expect23:05:281
179940,0migration/128697-21sh00:20:221
179940,0migration/125781-21kthreadcore23:00:341
179940,0migration/122892-21switchtime23:35:451
179940,0migration/119662-21kthreadcore19:15:301
179940,0migration/112537-21taskset23:12:121
179940,0migration/111113-21idleruntime23:50:341
169940,0watchdog/15258-21ssh22:12:201
169940,0watchdog/130919-21ssh21:14:161
169940,0watchdog/12459-21fschecks_time21:35:321
169940,0watchdog/123195-21kthreadcore19:40:281
169940,0watchdog/121545-21ssh23:18:041
169940,0watchdog/119174-21fschecks_count23:55:321
169940,0watchdog/10-21swapper/119:05:291
169940,0watchdog/10-21swapper/100:11:321
149940,0migration/032023-21kthreadcore20:10:340
149940,0migration/019278-21fschecks_time23:55:320
149940,0migration/013824-21kthreadcore22:55:340
149940,0migration/010938-21kthreadcore21:20:360
8315030,0irq/26-0000:00:0-21swapper/220:25:152
329930,0watchdog/328016-21ntpq20:50:403
329930,0watchdog/30-21swapper/320:26:323
329930,0watchdog/30-21swapper/319:15:293
249930,0watchdog/20-21swapper/219:50:282
177589933,0cyclictest3599-21lldpd20:22:592
177589932,0cyclictest3599-21lldpd19:00:292
177589932,0cyclictest23026-21awk20:50:342
177589932,0cyclictest19935-21grep18:55:312
177589932,0cyclictest18909-21tune2fs19:40:242
177579933,0cyclictest3599-21lldpd20:45:581
177579932,0cyclictest8622-21awk20:40:321
177579932,0cyclictest28979-21diskmemload23:30:251
177579932,0cyclictest25394-21stty21:30:291
177579932,0cyclictest25222-21awk22:25:341
177579932,0cyclictest15293-21tune2fs20:00:301
177579932,0cyclictest14379-21sh22:18:211
177579932,0cyclictest12236-21awk20:20:331
177579932,0cyclictest11187-21fschecks_time21:40:331
177579930,0cyclictest22396-21perl19:20:271
177569932,0cyclictest3599-21lldpd21:27:000
177569932,0cyclictest26543-21sensors00:15:420
177569932,0cyclictest23871-21snmpd23:10:290
177569932,0cyclictest21364-21awk23:00:310
177569932,0cyclictest11831-21sh00:10:260
177569930,0cyclictest0-21swapper/000:00:350
179930,0migration/12873-21kthreadcore19:50:291
179930,0migration/118736-21entropy20:25:291
169930,0watchdog/10-21swapper/123:45:201
169930,0watchdog/10-21swapper/122:40:241
169930,0watchdog/10-21swapper/120:15:401
169930,0watchdog/10-21swapper/119:30:371
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional