You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 08:20
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Mon Nov 17, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1144225645,0sleep30-21swapper/318:19:293
1144623627,0sleep20-21swapper/218:19:322
1128923524,0sleep10-21swapper/118:17:431
1120723523,0sleep00-21swapper/018:16:450
329980,0watchdog/30-21swapper/321:07:203
329980,0watchdog/30-21swapper/319:35:333
29925080,0irq/32-eth0-rx-0-21swapper/322:15:333
329970,0watchdog/321950-21kthreadcore20:15:323
329970,0watchdog/30-21swapper/323:15:363
29935070,0irq/33-eth0-tx-0-21swapper/019:45:300
29925070,0irq/32-eth0-rx-5574-21kthreadcore20:45:353
4160,0ktimersoftd/00-21swapper/020:50:280
339960,0migration/325507-21munin-node19:10:363
339960,0migration/324137-21sh20:20:163
329960,0watchdog/330227-21chrt22:51:083
329960,0watchdog/325319-21python18:25:373
329960,0watchdog/30-21swapper/323:45:163
329960,0watchdog/30-21swapper/323:27:363
329960,0watchdog/30-21swapper/323:20:283
329960,0watchdog/30-21swapper/322:37:203
329960,0watchdog/30-21swapper/322:04:003
329960,0watchdog/30-21swapper/321:43:083
329960,0watchdog/30-21swapper/321:37:003
329960,0watchdog/30-21swapper/321:34:003
329960,0watchdog/30-21swapper/321:29:523
329960,0watchdog/30-21swapper/321:25:083
329960,0watchdog/30-21swapper/321:15:403
329960,0watchdog/30-21swapper/321:01:203
329960,0watchdog/30-21swapper/320:55:163
329960,0watchdog/30-21swapper/320:51:083
329960,0watchdog/30-21swapper/320:40:563
329960,0watchdog/30-21swapper/320:36:523
329960,0watchdog/30-21swapper/320:31:403
329960,0watchdog/30-21swapper/320:25:283
329960,0watchdog/30-21swapper/320:10:323
329960,0watchdog/30-21swapper/319:45:283
329960,0watchdog/30-21swapper/319:30:293
329960,0watchdog/30-21swapper/319:15:413
329960,0watchdog/30-21swapper/318:30:253
29935060,0irq/33-eth0-tx-0-21swapper/019:00:330
29925060,0irq/32-eth0-rx-5367-21iostat_ios22:20:323
29925060,0irq/32-eth0-rx-0-21swapper/323:37:533
29925060,0irq/32-eth0-rx-0-21swapper/323:33:263
29925060,0irq/32-eth0-rx-0-21swapper/322:55:213
29925060,0irq/32-eth0-rx-0-21swapper/322:27:023
29925060,0irq/32-eth0-rx-0-21swapper/321:51:463
29925060,0irq/32-eth0-rx-0-21swapper/321:45:253
29925060,0irq/32-eth0-rx-0-21swapper/318:20:343
149960,0migration/023621-21pmu-power21:50:340
8315050,0irq/26-0000:00:7900-21ls23:00:112
8315050,0irq/26-0000:00:1019-21vmstat19:15:422
8315050,0irq/26-0000:00:0-21swapper/219:20:122
339950,0migration/39981-21kthreadcore23:00:293
339950,0migration/34101-21kthreadcore19:20:313
339950,0migration/322285-21gltestperf22:30:273
339950,0migration/319301-21kthreadcore18:45:323
339950,0migration/310254-21kthreadcore18:40:323
329950,0watchdog/32600-21munin-node20:05:243
329950,0watchdog/311566-21kthreadcore22:40:363
329950,0watchdog/30-21swapper/323:43:043
329950,0watchdog/30-21swapper/323:10:203
329950,0watchdog/30-21swapper/322:45:163
329950,0watchdog/30-21swapper/322:05:123
329950,0watchdog/30-21swapper/321:56:003
329950,0watchdog/30-21swapper/321:10:203
329950,0watchdog/30-21swapper/320:01:443
329950,0watchdog/30-21swapper/319:57:523
329950,0watchdog/30-21swapper/319:50:523
329950,0watchdog/30-21swapper/319:25:573
329950,0watchdog/30-21swapper/319:00:293
329950,0watchdog/30-21swapper/318:55:133
329950,0watchdog/30-21swapper/318:52:133
329950,0watchdog/30-21swapper/318:35:173
29935050,0irq/33-eth0-tx-5163-21ssh21:20:532
29935050,0irq/33-eth0-tx-32572-21ssh21:40:222
29935050,0irq/33-eth0-tx-25445-21munin-node23:10:312
29935050,0irq/33-eth0-tx-23871-21snmpd20:34:002
29935050,0irq/33-eth0-tx-22387-21kthreadcore22:10:312
29935050,0irq/33-eth0-tx-20736-21munin-node23:45:322
29935050,0irq/33-eth0-tx-17371-21munin-node21:50:262
29935050,0irq/33-eth0-tx-16825-21munin-node23:05:402
29935050,0irq/33-eth0-tx-1641-21munin-node23:15:382
29935050,0irq/33-eth0-tx-12063-21ssh21:06:252
29935050,0irq/33-eth0-tx-0-21swapper/222:53:512
29935050,0irq/33-eth0-tx-0-21swapper/222:18:462
29935050,0irq/33-eth0-tx-0-21swapper/221:02:432
29935050,0irq/33-eth0-tx-0-21swapper/019:20:250
29935050,0irq/33-eth0-tx-0-21swapper/018:40:420
29925050,0irq/32-eth0-rx-524-21kthreadcore19:40:313
29925050,0irq/32-eth0-rx-16861-21kthreadcore19:05:323
29925050,0irq/32-eth0-rx-0-21swapper/323:05:303
29925050,0irq/32-eth0-rx-0-21swapper/322:10:293
259950,0migration/26391-21kthreadcore20:45:362
259950,0migration/229175-21munin-node18:50:402
259950,0migration/226102-21kthreadcore19:35:322
259950,0migration/219662-21kthreadcore19:30:312
259950,0migration/219012-21kthreadcore21:30:312
259950,0migration/217504-21fschecks_count20:15:272
259950,0migration/216080-21latency_hist19:30:122
259950,0migration/215908-21kthreadcore23:40:352
259950,0migration/210745-21sh23:36:012
249950,0watchdog/220846-21needreboot22:45:322
249950,0watchdog/215838-21cat21:10:322
19150,0ktimersoftd/10-21swapper/121:13:331
179950,0migration/19679-21kthreadcore23:35:311
179950,0migration/17655-21kthreadcore20:45:361
179950,0migration/129138-21kthreadcore22:50:311
179950,0migration/12697-21fschecks_count22:00:271
179950,0migration/120643-21munin-node19:10:311
179950,0migration/119003-21aten_r1power_fr23:25:251
149950,0migration/09586-21kthreadcore23:35:310
149950,0migration/07006-21kthreadcore23:15:320
149950,0migration/04309-21kthreadcore18:35:300
149950,0migration/04082-21smart_sda18:55:370
149950,0migration/031822-21taskset22:55:150
149950,0migration/030697-21sh21:20:210
149950,0migration/024309-21kthreadcore22:30:310
149950,0migration/023932-21missed_timers20:35:340
149950,0migration/023770-21cpuspeed_turbos19:35:270
149950,0migration/021268-21kthreadcore22:10:300
149950,0migration/020158-21latency_hist20:55:120
149950,0migration/015339-21fschecks_count22:45:260
149950,0migration/014485-21kthreadcore22:25:320
149950,0migration/011569-21kthreadcore21:45:310
149950,0migration/01132-21taskset19:16:510
149950,0migration/010876-21kthreadcore21:25:310
115789955,0cyclictest0-21swapper/223:35:042
115789955,0cyclictest0-21swapper/223:03:222
115789955,0cyclictest0-21swapper/222:43:592
115789955,0cyclictest0-21swapper/221:15:322
115789950,0cyclictest0-21swapper/222:28:512
115789950,0cyclictest0-21swapper/222:04:342
115789950,0cyclictest0-21swapper/220:43:032
115779955,0cyclictest0-21swapper/123:18:531
115779955,0cyclictest0-21swapper/122:15:021
115779955,0cyclictest0-21swapper/121:35:091
115779950,0cyclictest0-21swapper/123:02:011
115779950,0cyclictest0-21swapper/122:44:121
115779950,0cyclictest0-21swapper/121:07:431
115779950,0cyclictest0-21swapper/119:05:411
115779950,0cyclictest0-21swapper/119:00:291
115769954,0cyclictest0-21swapper/021:20:120
8315040,0irq/26-0000:00:0-21swapper/218:50:122
29935040,0irq/33-eth0-tx-0-21swapper/222:21:092
29935040,0irq/33-eth0-tx-0-21swapper/221:45:272
29935040,0irq/33-eth0-tx-0-21swapper/220:55:192
29935040,0irq/33-eth0-tx-0-21swapper/220:52:022
29935040,0irq/33-eth0-tx-0-21swapper/220:25:322
29935040,0irq/33-eth0-tx-0-21swapper/220:21:272
29935040,0irq/33-eth0-tx-0-21swapper/019:34:130
29935040,0irq/33-eth0-tx-0-21swapper/019:05:400
259940,0migration/26953-21aten_r1power_cu21:25:232
259940,0migration/23518-21latency22:35:322
259940,0migration/231264-21fschecks_count19:40:272
259940,0migration/230941-21kthreadcore21:55:302
259940,0migration/22540-21munin-node18:35:292
259940,0migration/224585-21kthreadcore19:55:302
259940,0migration/224474-21munin-node21:35:252
259940,0migration/223966-21kthreadcore22:30:312
259940,0migration/222832-21kthreadcore19:10:332
259940,0migration/219811-21kthreadcore20:35:322
259940,0migration/219788-21fschecks_time23:25:272
259940,0migration/21445-21timerandwakeup20:00:392
259940,0migration/213111-21fschecks_time19:05:282
259940,0migration/210976-21sh22:05:262
249940,0watchdog/226726-21diskstats18:30:252
249940,0watchdog/210079-21grep19:45:322
179940,0migration/18530-21runrttasks20:29:061
179940,0migration/15769-21kthreadcore20:05:301
179940,0migration/130190-21aten_r1power_cu18:55:251
179940,0migration/129561-21ntp_offset19:35:351
179940,0migration/120976-21cpuspeed_turbos23:45:281
179940,0migration/115351-21kthreadcore22:25:321
179940,0migration/111628-21mdstat19:45:331
179940,0migration/110914-21kthreadcore18:40:321
179940,0migration/110884-21aten_r1power_en22:05:251
179940,0migration/110567-21kthreadcore21:25:311
169940,0watchdog/17402-21ssh21:41:481
169940,0watchdog/15368-21iostat_ios22:20:321
169940,0watchdog/1412-21chrt21:56:121
169940,0watchdog/128912-21sh21:00:241
169940,0watchdog/12651-21ssh20:45:001
169940,0watchdog/125825-21ssh20:38:441
169940,0watchdog/123162-21egrep18:50:291
169940,0watchdog/121951-21kthreadcore20:15:321
169940,0watchdog/121313-21munin-node22:30:281
169940,0watchdog/120844-21needreboot22:45:321
169940,0watchdog/115584-21cut20:50:321
169940,0watchdog/110567-21cpuspeed_turbos23:20:281
169940,0watchdog/10-21swapper/122:35:321
169940,0watchdog/10-21swapper/120:10:321
169940,0watchdog/10-21swapper/118:45:331
149940,0migration/029964-21if_eth021:00:290
149940,0migration/029717-21kthreadcore20:40:310
149940,0migration/028736-21kthreadcore21:35:300
149940,0migration/027119-21kthreadcore20:20:340
149940,0migration/026663-21timerwakeupswit19:55:380
149940,0migration/019927-21kthreadcore23:05:310
149940,0migration/016892-21irqcore21:30:300
149940,0migration/015919-21kthreadcore19:50:320
115789944,0cyclictest0-21swapper/223:20:142
115789944,0cyclictest0-21swapper/218:20:132
115789943,0cyclictest9521-21awk18:40:302
115779944,0cyclictest0-21swapper/123:33:421
115779944,0cyclictest0-21swapper/122:58:521
115779944,0cyclictest0-21swapper/122:18:401
115779944,0cyclictest0-21swapper/121:50:171
115779944,0cyclictest0-21swapper/119:25:141
115779943,0cyclictest3599-21lldpd20:01:141
115779943,0cyclictest31832-21awk19:40:291
115779943,0cyclictest31682-21gltestperf21:20:271
115779943,0cyclictest2879-21awk19:20:301
115779943,0cyclictest27867-21awk19:15:311
115779943,0cyclictest26596-21awk23:10:291
115779943,0cyclictest25197-21awk21:35:281
115779943,0cyclictest23871-21snmpd21:45:381
115779943,0cyclictest23871-21snmpd20:34:381
115779943,0cyclictest23871-21snmpd19:33:081
115779943,0cyclictest23293-21awk21:15:311
115779943,0cyclictest2249-21awk18:35:281
115779943,0cyclictest21351-21awk20:55:301
115779943,0cyclictest21296-21awk19:55:281
115779943,0cyclictest20101-21awk18:25:281
115779943,0cyclictest17914-21awk23:05:291
115769943,0cyclictest9247-21awk23:00:270
115769943,0cyclictest4399-21head19:40:330
115769943,0cyclictest3599-21lldpd20:48:500
115769943,0cyclictest3262-21irqbalance23:31:060
115769943,0cyclictest3262-21irqbalance23:27:060
115769943,0cyclictest3262-21irqbalance22:50:360
115769943,0cyclictest3262-21irqbalance22:42:060
115769943,0cyclictest3262-21irqbalance22:05:160
115769943,0cyclictest3262-21irqbalance22:03:460
115769943,0cyclictest3262-21irqbalance21:10:160
115769943,0cyclictest3262-21irqbalance21:05:160
115769943,0cyclictest3262-21irqbalance20:30:160
115769943,0cyclictest3262-21irqbalance20:27:260
115769943,0cyclictest3262-21irqbalance20:15:260
115769943,0cyclictest3262-21irqbalance18:45:260
115769943,0cyclictest3262-21irqbalance18:25:470
115769943,0cyclictest29020-21awk22:15:330
115769943,0cyclictest28556-21awk20:00:290
115769943,0cyclictest27355-21awk18:30:270
115769943,0cyclictest24051-21awk18:50:320
115769943,0cyclictest1256-21awk21:40:320
115769943,0cyclictest10752-21awk20:10:300
115769943,0cyclictest10204-21awk19:25:300
115769941,0cyclictest0-21swapper/020:05:230
8315030,0irq/26-0000:00:0-21swapper/219:00:122
29935030,0irq/33-eth0-tx-0-21swapper/019:10:270
249930,0watchdog/20-21swapper/219:50:322
179930,0migration/127981-21kthreadcore18:30:291
179930,0migration/112338-21sh23:40:141
169930,0watchdog/10-21swapper/120:22:041
115789932,0cyclictest3560-21awk20:05:282
115789930,0cyclictest9665-21runrttasks20:10:202
115789930,0cyclictest22650-21kthreadcore18:25:312
115779932,0cyclictest3599-21lldpd19:50:141
115779932,0cyclictest23871-21snmpd18:24:261
115769932,0cyclictest3262-21irqbalance23:40:160
115769932,0cyclictest3262-21irqbalance22:20:160
115769932,0cyclictest31569-21awk23:10:330
115769932,0cyclictest12875-21awk18:20:320
115769932,0cyclictest12257-21cut23:20:330
115769930,0cyclictest30775-21perl22:35:270
115769930,0cyclictest26765-21perl21:55:260
115789920,0cyclictest0-21swapper/218:55:142
115769920,0cyclictest0-21swapper/023:45:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional