You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 11:57
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2.osadl.org (updated Fri May 02, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2534824028,0sleep30-21swapper/318:28:573
2535723728,0sleep20-21swapper/218:29:042
2539923524,0sleep00-21swapper/018:29:340
2551222925,0sleep10-21swapper/118:30:511
329980,0watchdog/317283-21sh21:52:383
329970,0watchdog/320282-21kthreadcore22:12:383
329970,0watchdog/30-21swapper/322:37:423
329970,0watchdog/30-21swapper/322:36:423
29925070,0irq/32-eth0-rx-0-21swapper/323:46:573
29925070,0irq/32-eth0-rx-0-21swapper/321:49:013
257059970,0cyclictest0-21swapper/222:19:452
257049977,0cyclictest2546-21nvme18:57:291
339960,0migration/38253-21sh23:38:073
339960,0migration/324410-21kthreadcore21:37:373
339960,0migration/322170-21idleruntime-cro18:52:183
339960,0migration/310992-21taskset22:44:323
339960,0migration/310738-21kthreadcore22:07:363
329960,0watchdog/30-21swapper/323:53:423
329960,0watchdog/30-21swapper/323:34:143
329960,0watchdog/30-21swapper/323:23:423
329960,0watchdog/30-21swapper/323:18:223
329960,0watchdog/30-21swapper/323:07:263
329960,0watchdog/30-21swapper/322:57:263
329960,0watchdog/30-21swapper/321:59:263
329960,0watchdog/30-21swapper/321:42:503
329960,0watchdog/30-21swapper/321:24:023
329960,0watchdog/30-21swapper/321:19:143
329960,0watchdog/30-21swapper/321:13:143
329960,0watchdog/30-21swapper/321:05:183
329960,0watchdog/30-21swapper/320:59:223
329960,0watchdog/30-21swapper/320:50:063
329960,0watchdog/30-21swapper/320:42:423
329960,0watchdog/30-21swapper/320:37:423
329960,0watchdog/30-21swapper/320:12:423
329960,0watchdog/30-21swapper/319:57:303
329960,0watchdog/30-21swapper/319:52:343
329960,0watchdog/30-21swapper/319:38:023
329960,0watchdog/30-21swapper/319:07:393
329960,0watchdog/30-21swapper/318:54:553
29935060,0irq/33-eth0-tx-19162-21munin-node19:57:410
29925060,0irq/32-eth0-rx-8053-40yum-updatesd-he21:07:443
29925060,0irq/32-eth0-rx-24245-21munin-node20:22:403
29925060,0irq/32-eth0-rx-22382-21turbostat.cron19:37:173
29925060,0irq/32-eth0-rx-21936-21hostname22:52:353
29925060,0irq/32-eth0-rx-18571-21open_inodes21:32:383
29925060,0irq/32-eth0-rx-15046-21tr22:27:383
29925060,0irq/32-eth0-rx-0-21swapper/323:30:023
29925060,0irq/32-eth0-rx-0-21swapper/323:12:343
29925060,0irq/32-eth0-rx-0-21swapper/323:02:353
29925060,0irq/32-eth0-rx-0-21swapper/322:51:303
29925060,0irq/32-eth0-rx-0-21swapper/322:05:033
29925060,0irq/32-eth0-rx-0-21swapper/321:29:103
29925060,0irq/32-eth0-rx-0-21swapper/320:52:373
27160,0ktimersoftd/20-21swapper/221:41:492
257059960,0cyclictest0-21swapper/221:43:302
257059960,0cyclictest0-21swapper/219:02:502
257039960,0cyclictest0-21swapper/022:02:440
19160,0ktimersoftd/10-21swapper/121:41:491
149960,0migration/029352-21pmu-power21:57:390
149960,0migration/02605-21gltestperf21:07:340
149960,0migration/022547-21sh21:56:230
149960,0migration/019782-21kthreadcore23:27:370
8315050,0irq/26-0000:00:0-21swapper/223:57:182
339950,0migration/39930-21kthreadcore18:42:363
339950,0migration/331906-21munin-node18:32:403
339950,0migration/318588-21cpuspeed_turbos23:47:313
339950,0migration/31376-21cpu20:07:363
339950,0migration/310770-21kthreadcore19:27:393
329950,0watchdog/324252-21cut19:12:383
329950,0watchdog/30-21swapper/322:22:263
329950,0watchdog/30-21swapper/320:28:343
329950,0watchdog/30-21swapper/320:18:343
329950,0watchdog/30-21swapper/320:02:343
329950,0watchdog/30-21swapper/319:47:183
329950,0watchdog/30-21swapper/319:42:343
329950,0watchdog/30-21swapper/319:22:463
329950,0watchdog/30-21swapper/319:17:263
329950,0watchdog/30-21swapper/319:02:393
329950,0watchdog/30-21swapper/318:38:073
29935050,0irq/33-eth0-tx-653-21munin-node18:37:370
29935050,0irq/33-eth0-tx-22385-21munin-node18:52:320
29935050,0irq/33-eth0-tx-19308-21diskstats21:17:322
29935050,0irq/33-eth0-tx-13408-21if_eth1.222:47:352
29935050,0irq/33-eth0-tx-10941-21kthreadcore21:47:392
29935050,0irq/33-eth0-tx-0-21swapper/223:00:062
29935050,0irq/33-eth0-tx-0-21swapper/222:32:372
29935050,0irq/33-eth0-tx-0-21swapper/222:22:372
29935050,0irq/33-eth0-tx-0-21swapper/222:07:482
29925050,0irq/32-eth0-rx-0-21swapper/323:58:403
29925050,0irq/32-eth0-rx-0-21swapper/322:17:343
29925050,0irq/32-eth0-rx-0-21swapper/320:32:353
29925050,0irq/32-eth0-rx-0-21swapper/318:57:313
257059955,0cyclictest0-21swapper/223:46:282
257059955,0cyclictest0-21swapper/223:39:032
257059955,0cyclictest0-21swapper/223:16:122
257059955,0cyclictest0-21swapper/221:34:402
257059950,0cyclictest0-21swapper/223:05:192
257049955,0cyclictest2546-21nvme21:22:321
257049955,0cyclictest2546-21nvme21:06:301
257049955,0cyclictest0-21swapper/120:52:441
259950,0migration/28678-21fschecks_time20:52:382
259950,0migration/28080-21latency_hist19:27:172
259950,0migration/26609-21sh23:19:092
259950,0migration/26259-21sh22:03:122
259950,0migration/230483-21munin-node18:57:322
259950,0migration/228385-21idleruntime22:37:342
259950,0migration/22221-21irqrtprio18:37:362
259950,0migration/219726-21missed_timers20:37:382
259950,0migration/219088-21kthreadcore23:27:372
259950,0migration/218865-21kthreadcore18:47:372
249950,0watchdog/227037-21irqstats21:02:382
179950,0migration/14322-21fschecks_count22:42:331
179950,0migration/126744-21sh22:36:221
179950,0migration/126487-21taskset21:20:211
179950,0migration/116420-21latency_hist22:12:171
179950,0migration/11533-21kthreadcore22:02:361
169950,0watchdog/127527-21sed19:17:381
169950,0watchdog/117286-21tr21:52:381
149950,0migration/09133-21fschecks_time20:12:350
149950,0migration/08971-21runrttasks23:40:250
149950,0migration/07703-21kthreadcore19:02:430
149950,0migration/07353-21kthreadcore22:42:380
149950,0migration/0481-21kthreadcore21:42:370
149950,0migration/04520-21fschecks_time21:27:350
149950,0migration/04093-21kthreadcore19:22:380
149950,0migration/032404-21kthreadcore22:37:380
149950,0migration/03053-21df_abs23:57:310
149950,0migration/029500-21kthreadcore21:02:390
149950,0migration/027439-21sh21:22:270
149950,0migration/022641-21aten_r1power_cu19:37:330
149950,0migration/022155-21taskset19:34:390
149950,0migration/019717-21kthreadcore18:47:370
149950,0migration/019513-21fschecks_time22:32:330
149950,0migration/019079-21kthreadcore22:12:380
149950,0migration/018738-21latency_hist21:17:170
149950,0migration/017772-21latency19:52:380
149950,0migration/017391-21kthreadcore21:32:350
149950,0migration/016361-21latency_hist23:27:170
149950,0migration/016023-21kthreadcore20:37:360
149950,0migration/015862-21missed_timers23:42:400
149950,0migration/015462-21sh20:54:560
149950,0migration/014178-21kthreadcore21:12:380
8315040,0irq/26-0000:00:19079-21sed19:57:172
29935040,0irq/33-eth0-tx-8301-21kthreadcore22:42:382
29935040,0irq/33-eth0-tx-16929-21ssh22:27:532
29935040,0irq/33-eth0-tx-16582-21ssh22:12:232
29935040,0irq/33-eth0-tx-12151-21ssh20:33:522
29935040,0irq/33-eth0-tx-11029-21fschecks_time21:12:332
29935040,0irq/33-eth0-tx-0-21swapper/223:53:342
29935040,0irq/33-eth0-tx-0-21swapper/223:22:352
29935040,0irq/33-eth0-tx-0-21swapper/222:52:262
29935040,0irq/33-eth0-tx-0-21swapper/221:22:452
29935040,0irq/33-eth0-tx-0-21swapper/220:48:222
29935040,0irq/33-eth0-tx-0-21swapper/220:42:372
29935040,0irq/33-eth0-tx-0-21swapper/020:22:400
29935040,0irq/33-eth0-tx-0-21swapper/020:07:350
29935040,0irq/33-eth0-tx-0-21swapper/019:42:330
29935040,0irq/33-eth0-tx-0-21swapper/018:57:300
257059943,0cyclictest9479-21awk20:12:372
257059943,0cyclictest29608-21sadc19:42:172
257059943,0cyclictest23902-21awk20:22:382
257059943,0cyclictest23871-21snmpd20:21:352
257059943,0cyclictest23871-21snmpd18:32:352
257059943,0cyclictest19354-21awk23:47:342
257059943,0cyclictest11728-21sadc19:52:172
257059940,0cyclictest0-21swapper/219:34:412
257049944,0cyclictest0-21swapper/122:59:481
257049944,0cyclictest0-21swapper/122:22:451
257049943,0cyclictest4925-21awk21:27:371
257049943,0cyclictest3599-21lldpd23:42:201
257049943,0cyclictest3599-21lldpd23:07:191
257049943,0cyclictest23871-21snmpd23:27:501
257049943,0cyclictest23871-21snmpd22:48:201
257049943,0cyclictest23871-21snmpd19:57:501
257049943,0cyclictest22144-21awk22:52:371
257049943,0cyclictest19336-21awk23:47:341
257049943,0cyclictest1735-21awk18:37:351
257049943,0cyclictest16361-21awk18:47:351
257049943,0cyclictest11465-21awk21:12:361
257039943,0cyclictest9237-21awk19:27:370
257039943,0cyclictest5869-21awk19:47:370
257039943,0cyclictest27653-21awk23:52:340
257039943,0cyclictest26266-21awk23:32:340
257039943,0cyclictest23871-21snmpd23:14:200
257039943,0cyclictest23871-21snmpd23:08:500
257039943,0cyclictest23871-21snmpd20:42:350
257039943,0cyclictest21999-21awk22:52:360
257039943,0cyclictest18821-21grep23:47:320
257039940,0cyclictest0-21swapper/019:07:460
259940,0migration/28272-21aten_r1power_fr19:27:332
259940,0migration/25605-21kthreadcore21:27:392
259940,0migration/232289-21sendmail_mailtr19:17:442
259940,0migration/230523-21fschecks_time19:42:352
259940,0migration/226835-21kthreadcore23:32:362
259940,0migration/226433-21kthreadcore21:57:362
259940,0migration/221425-21kthreadcore19:12:382
259940,0migration/220641-21kthreadcore20:57:392
259940,0migration/215703-21munin-node21:52:362
259940,0migration/213666-21kthreadcore19:07:412
249940,0watchdog/20-21swapper/223:07:382
249940,0watchdog/20-21swapper/221:07:382
249940,0watchdog/20-21swapper/220:27:382
179940,0migration/18740-21fschecks_count19:27:351
179940,0migration/15394-21missed_timers23:17:371
179940,0migration/14482-21aten_r1power_cu19:02:371
179940,0migration/131698-21kthreadcore21:42:371
179940,0migration/130643-21fschecks_time19:42:351
179940,0migration/129765-21kthreadcore22:37:371
179940,0migration/127525-21kthreadcore21:57:361
179940,0migration/126810-21irqrtprio22:17:361
179940,0migration/125213-21kthreadcore18:52:371
179940,0migration/124562-21kthreadcore20:42:421
179940,0migration/116233-21sh23:27:031
179940,0migration/115101-21runrttasks20:13:521
179940,0migration/11494-21latency_hist21:07:171
179940,0migration/114321-21timerandwakeup18:42:441
179940,0migration/110747-21kthreadcore23:02:371
179940,0migration/11069-21latency_hist20:07:171
169940,0watchdog/16386-21runrttasks20:34:381
169940,0watchdog/14737-21diskmemload00:01:181
169940,0watchdog/13072-21kthreadcore20:27:381
169940,0watchdog/123581-21ssh22:13:261
169940,0watchdog/117127-21df_inode20:57:341
169940,0watchdog/114873-21awk20:37:341
169940,0watchdog/10-21swapper/123:12:381
169940,0watchdog/10-21swapper/121:47:381
169940,0watchdog/10-21swapper/121:32:341
169940,0watchdog/10-21swapper/119:22:381
149940,0migration/029076-21kthreadcore22:17:370
149940,0migration/028671-21kthreadcore20:02:430
149940,0migration/028459-21kthreadcore19:17:390
149940,0migration/027798-21sh21:37:500
149940,0migration/024390-21sh21:00:180
149940,0migration/02304-21kthreadcore20:47:400
149940,0migration/012296-21sh20:34:170
149940,0migration/01097-21kthreadcore20:27:380
149940,0migration/010602-21kthreadcore21:47:390
8315030,0irq/26-0000:00:0-21swapper/218:42:242
257059932,0cyclictest338-21grep20:02:512
257059932,0cyclictest28601-21sensors18:52:422
257059932,0cyclictest23871-21snmpd19:56:352
257049933,0cyclictest0-21swapper/123:52:471
257049932,0cyclictest27458-21awk20:02:411
257049932,0cyclictest23933-21awk20:22:391
257049932,0cyclictest19896-21awk19:12:351
257049932,0cyclictest16489-21awk19:32:371
257049930,0cyclictest26775-21perl18:32:341
257049930,0cyclictest10948-21perl22:27:331
257039933,0cyclictest3730-21kworker/0:220:17:350
257039933,0cyclictest28568-21kworker/0:119:12:210
257039933,0cyclictest28568-21kworker/0:118:42:190
257039933,0cyclictest28568-21kworker/0:118:32:190
257039933,0cyclictest20064-21kworker/0:023:17:200
257039933,0cyclictest20064-21kworker/0:023:02:180
257039933,0cyclictest20064-21kworker/0:022:57:240
257039933,0cyclictest20064-21kworker/0:022:47:200
257039933,0cyclictest20064-21kworker/0:022:27:180
257039933,0cyclictest20064-21kworker/0:022:22:280
257039933,0cyclictest20064-21kworker/0:022:07:240
259930,0migration/21154-21runrttasks20:07:322
169930,0watchdog/18180-21kthreadcore19:47:381
169930,0watchdog/16386-21runrttasks19:52:341
169930,0watchdog/14737-21diskmemload23:37:421
169930,0watchdog/112710-21munin-node19:07:391
169930,0watchdog/10-21swapper/123:32:301
169930,0watchdog/10-21swapper/120:47:461
169930,0watchdog/10-21swapper/119:37:341
257049921,0cyclictest0-21swapper/120:17:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional