You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 20:56
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Tue Nov 18, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1244027069,0sleep10-21swapper/106:17:581
1249426564,0sleep00-21swapper/006:18:330
12699995858,0cyclictest26153-21md0_resync08:36:361
1239625655,0sleep20-21swapper/206:17:312
12700994646,0cyclictest26152-21md1_resync08:36:362
1255124028,0sleep30-21swapper/306:19:123
12698991312,0cyclictest29719-21ssh08:36:360
126989987,0cyclictest0-21swapper/008:20:310
8315070,0irq/26-0000:00:3056-21kthreadcore09:40:263
8315070,0irq/26-0000:00:1517-21port_sshpriv06:30:283
8315070,0irq/26-0000:00:0-21swapper/307:15:293
8315070,0irq/26-0000:00:0-21swapper/306:20:353
329970,0watchdog/388682sleep307:43:273
329970,0watchdog/30-21swapper/309:36:513
329970,0watchdog/30-21swapper/309:22:353
29925070,0irq/32-eth0-rx-30121-21cut10:15:283
29925070,0irq/32-eth0-rx-285142sleep308:18:583
29925070,0irq/32-eth0-rx-195362sleep309:14:013
126999976,0cyclictest27019-21kthreadcore11:10:271
8315060,0irq/26-0000:00:7760-21perl09:45:223
8315060,0irq/26-0000:00:6428-21iostat_ios08:25:263
8315060,0irq/26-0000:00:4986-21kthreadcore08:00:373
8315060,0irq/26-0000:00:4701-21aten2.4-expect07:20:203
8315060,0irq/26-0000:00:25738-21perl08:55:243
8315060,0irq/26-0000:00:20350-21cut06:45:273
8315060,0irq/26-0000:00:20250-21iostat_ios10:30:253
8315060,0irq/26-0000:00:13306-21load07:00:253
8315060,0irq/26-0000:00:11284-21if_eth1.206:40:233
8315060,0irq/26-0000:00:0-21swapper/311:45:283
8315060,0irq/26-0000:00:0-21swapper/311:26:293
8315060,0irq/26-0000:00:0-21swapper/311:10:273
8315060,0irq/26-0000:00:0-21swapper/310:10:223
8315060,0irq/26-0000:00:0-21swapper/309:54:583
8315060,0irq/26-0000:00:0-21swapper/309:25:283
8315060,0irq/26-0000:00:0-21swapper/308:40:303
8315060,0irq/26-0000:00:0-21swapper/308:10:263
8315060,0irq/26-0000:00:0-21swapper/307:45:293
8315060,0irq/26-0000:00:0-21swapper/307:10:243
8315060,0irq/26-0000:00:0-21swapper/306:55:193
8315060,0irq/26-0000:00:0-21swapper/306:25:243
339960,0migration/330879-21sh09:55:373
339960,0migration/330504-21sh08:38:493
339960,0migration/323279-21sh08:51:093
339960,0migration/312801-21sh11:00:473
329960,0watchdog/3299250irq/32-eth0-rx-11:35:343
329960,0watchdog/30-21swapper/311:33:223
329960,0watchdog/30-21swapper/311:20:553
329960,0watchdog/30-21swapper/311:19:033
329960,0watchdog/30-21swapper/310:58:273
329960,0watchdog/30-21swapper/310:49:233
329960,0watchdog/30-21swapper/310:41:553
329960,0watchdog/30-21swapper/310:35:153
329960,0watchdog/30-21swapper/310:35:153
329960,0watchdog/30-21swapper/310:26:393
329960,0watchdog/30-21swapper/310:23:073
329960,0watchdog/30-21swapper/309:31:313
329960,0watchdog/30-21swapper/309:16:073
329960,0watchdog/30-21swapper/309:02:033
329960,0watchdog/30-21swapper/308:45:113
329960,0watchdog/30-21swapper/308:21:433
329960,0watchdog/30-21swapper/307:58:033
329960,0watchdog/30-21swapper/307:26:153
329960,0watchdog/30-21swapper/307:06:513
329960,0watchdog/30-21swapper/306:50:513
329960,0watchdog/30-21swapper/306:35:113
29925060,0irq/32-eth0-rx-0-21swapper/311:40:153
29925060,0irq/32-eth0-rx-0-21swapper/311:05:233
29925060,0irq/32-eth0-rx-0-21swapper/310:50:153
29925060,0irq/32-eth0-rx-0-21swapper/310:05:093
29925060,0irq/32-eth0-rx-0-21swapper/310:04:453
29925060,0irq/32-eth0-rx-0-21swapper/309:05:253
29925060,0irq/32-eth0-rx-0-21swapper/308:30:133
19160,0ktimersoftd/10-21swapper/107:38:051
127009966,0cyclictest0-21swapper/211:17:562
127009960,0cyclictest0-21swapper/209:10:142
127009960,0cyclictest0-21swapper/207:48:432
126999960,0cyclictest0-21swapper/111:06:311
126989966,0cyclictest0-21swapper/011:14:030
126989966,0cyclictest0-21swapper/007:24:390
126989960,0cyclictest0-21swapper/011:25:520
126989960,0cyclictest0-21swapper/011:01:070
8315050,0irq/26-0000:00:0-21swapper/308:05:193
8315050,0irq/26-0000:00:0-21swapper/307:50:203
8315050,0irq/26-0000:00:0-21swapper/307:35:203
8315050,0irq/26-0000:00:0-21swapper/307:30:183
29935050,0irq/33-eth0-tx-6315-21iceccd09:01:152
29935050,0irq/33-eth0-tx-32441-21munin-node10:00:282
29935050,0irq/33-eth0-tx-23871-21snmpd06:57:200
29935050,0irq/33-eth0-tx-13831-21munin-node08:30:212
29935050,0irq/33-eth0-tx-0-21swapper/210:50:112
29935050,0irq/33-eth0-tx-0-21swapper/210:08:332
29935050,0irq/33-eth0-tx-0-21swapper/209:35:252
259950,0migration/29748-21kthreadcore09:45:262
259950,0migration/27984-21kthreadcore09:25:272
259950,0migration/26574-21sh08:43:052
259950,0migration/26219-21kthreadcore11:35:262
259950,0migration/24768-21sh08:24:502
259950,0migration/24302-21sh10:56:062
259950,0migration/231098-21runrttasks07:58:052
259950,0migration/230831-21sh11:13:562
259950,0migration/228509-21runrttasks07:10:332
259950,0migration/227995-21df_abs10:35:222
259950,0migration/227995-21df_abs10:35:222
259950,0migration/227011-21kthreadcore06:50:232
259950,0migration/223722-21sh11:26:192
259950,0migration/221163-21switchtime07:05:362
259950,0migration/218367-21kthreadcore09:50:292
259950,0migration/217695-21kthreadcore10:45:262
259950,0migration/213748-21kthreadcore11:40:292
259950,0migration/212595-21latency08:05:282
179950,0migration/127606-21sh10:35:131
179950,0migration/127606-21sh10:35:131
179950,0migration/127099-21sh11:46:451
179950,0migration/123981-21latency_hist07:55:071
179950,0migration/117185-21runrttasks06:44:181
179950,0migration/114637-21kthreadcore10:25:251
169950,0watchdog/121092-21gltestperf09:15:231
149950,0migration/04662-21fschecks_time10:40:190
149950,0migration/030852-21kthreadcore11:30:260
127009955,0cyclictest0-21swapper/210:30:192
127009955,0cyclictest0-21swapper/210:20:142
127009955,0cyclictest0-21swapper/209:22:512
127009955,0cyclictest0-21swapper/207:31:422
127009955,0cyclictest0-21swapper/207:22:332
127009950,0cyclictest0-21swapper/211:10:032
127009950,0cyclictest0-21swapper/210:25:162
127009950,0cyclictest0-21swapper/208:54:092
127009950,0cyclictest0-21swapper/207:27:262
126999955,0cyclictest0-21swapper/111:24:011
126999955,0cyclictest0-21swapper/110:47:261
126999955,0cyclictest0-21swapper/110:24:251
126999955,0cyclictest0-21swapper/108:23:201
126999955,0cyclictest0-21swapper/107:43:361
126999955,0cyclictest0-21swapper/107:33:501
126999955,0cyclictest0-21swapper/106:30:081
126999950,0cyclictest0-21swapper/111:02:341
126999950,0cyclictest0-21swapper/110:52:291
126999950,0cyclictest0-21swapper/110:19:221
126999950,0cyclictest0-21swapper/106:23:221
126989955,0cyclictest0-21swapper/010:19:390
126989955,0cyclictest0-21swapper/009:54:540
126989955,0cyclictest0-21swapper/009:27:180
126989955,0cyclictest0-21swapper/008:45:220
126989955,0cyclictest0-21swapper/008:01:040
126989955,0cyclictest0-21swapper/007:58:040
126989955,0cyclictest0-21swapper/006:45:050
126989950,0cyclictest0-21swapper/010:24:420
126989950,0cyclictest0-21swapper/009:41:390
126989950,0cyclictest0-21swapper/008:43:380
126989950,0cyclictest0-21swapper/007:47:500
126989950,0cyclictest0-21swapper/007:19:080
126989950,0cyclictest0-21swapper/007:02:440
29935040,0irq/33-eth0-tx-30888-21ssh09:55:392
29935040,0irq/33-eth0-tx-26152-21md1_resync06:25:240
29935040,0irq/33-eth0-tx-0-21swapper/211:45:342
29935040,0irq/33-eth0-tx-0-21swapper/211:20:162
29935040,0irq/33-eth0-tx-0-21swapper/208:45:432
29935040,0irq/33-eth0-tx-0-21swapper/008:05:210
259940,0migration/26046-21kthreadcore09:05:252
259940,0migration/25373-21diskmemload10:40:232
259940,0migration/23969-21vmstat07:15:382
259940,0migration/226326-21fschecks_count10:15:242
259940,0migration/225290-21kthreadcore09:15:272
259940,0migration/223075-21kthreadcore08:15:262
259940,0migration/221008-21kthreadcore06:45:272
259940,0migration/21800-21kthreadcore06:55:252
249940,0watchdog/28277-21tr08:25:272
249940,0watchdog/227245-21entropy07:35:232
249940,0watchdog/225388-21ssh11:30:062
249940,0watchdog/222746-21kthreadcore10:10:272
179940,0migration/19564-21timerwakeupswit09:05:331
179940,0migration/18754-21kthreadcore07:20:271
179940,0migration/17220-21cpuspeed_turbos09:45:211
179940,0migration/16937-21kthreadcore09:25:261
179940,0migration/13877-21kthreadcore11:15:301
179940,0migration/129842-21cpuspeed_turbos10:55:211
179940,0migration/129205-21kthreadcore09:55:271
179940,0migration/126679-21kthreadcore08:55:281
179940,0migration/12147-21kthreadcore06:55:251
179940,0migration/120658-21kthreadcore06:45:271
179940,0migration/118907-21kthreadcore08:30:271
179940,0migration/118473-21kthreadcore11:25:261
179940,0migration/114575-21kthreadcore11:40:291
169940,0watchdog/129305-21ssh09:36:511
169940,0watchdog/125678-21fschecks_time06:50:191
169940,0watchdog/12092-21md1_raid108:00:351
169940,0watchdog/112411-21ssh08:26:511
169940,0watchdog/110348-21munin-node07:45:271
169940,0watchdog/10-21swapper/110:30:111
169940,0watchdog/10-21swapper/108:50:271
169940,0watchdog/10-21swapper/108:05:271
149940,0migration/0651-21fschecks_time10:00:220
149940,0migration/04303-21irqrtprio09:05:240
149940,0migration/030671-21timerwakeupswit09:55:340
149940,0migration/03004-21munin-node07:40:250
149940,0migration/029784-21idleruntime09:20:230
149940,0migration/029667-21kthreadcore07:35:280
149940,0migration/025629-21timerandwakeup10:30:350
149940,0migration/024554-21kthreadcore09:35:250
149940,0migration/023052-21kthreadcore10:10:270
149940,0migration/017726-21irqrtprio08:50:260
149940,0migration/016168-21kthreadcore10:45:250
149940,0migration/013944-21fschecks_count09:30:210
127009944,0cyclictest0-21swapper/209:41:082
127009944,0cyclictest0-21swapper/209:30:152
127009944,0cyclictest0-21swapper/206:25:102
127009943,0cyclictest3791-21awk06:35:242
127009943,0cyclictest25928-21awk08:55:262
127009943,0cyclictest23871-21snmpd11:00:422
127009943,0cyclictest23871-21snmpd08:11:422
127009943,0cyclictest23871-21snmpd07:02:412
127009943,0cyclictest17778-21awk07:50:342
127009943,0cyclictest14035-21awk06:20:312
126999944,0cyclictest0-21swapper/110:11:291
126999944,0cyclictest0-21swapper/109:33:491
126999944,0cyclictest0-21swapper/106:38:491
126999943,0cyclictest3599-21lldpd08:41:231
126999943,0cyclictest29821-21awk09:20:231
126999943,0cyclictest23871-21snmpd11:35:271
126999943,0cyclictest23871-21snmpd10:00:571
126999943,0cyclictest23871-21snmpd08:45:571
126999943,0cyclictest23871-21snmpd07:59:421
126999943,0cyclictest23871-21snmpd07:04:111
126999943,0cyclictest23145-21awk07:10:241
126999943,0cyclictest21482-21awk06:25:281
126999943,0cyclictest18042-21grep07:25:281
126999943,0cyclictest15807-21awk07:05:271
126999943,0cyclictest12725-21awk09:10:251
126989944,0cyclictest0-21swapper/010:56:040
126989944,0cyclictest0-21swapper/008:56:530
126989943,0cyclictest8037-21awk09:45:240
126989943,0cyclictest3599-21lldpd11:37:080
126989943,0cyclictest3599-21lldpd11:15:380
126989943,0cyclictest3599-21lldpd08:32:470
126989943,0cyclictest3599-21lldpd07:25:140
126989943,0cyclictest3599-21lldpd06:53:420
126989943,0cyclictest3599-21lldpd06:30:100
126989943,0cyclictest23871-21snmpd11:21:120
126989943,0cyclictest23871-21snmpd10:09:270
126989943,0cyclictest23871-21snmpd07:08:560
126989943,0cyclictest22111-21awk10:50:260
126989943,0cyclictest21329-21awk09:15:250
126989943,0cyclictest17763-21awk07:50:340
249930,0watchdog/20-21swapper/208:00:352
249930,0watchdog/20-21swapper/206:40:192
249930,0watchdog/20-21swapper/206:30:072
179930,0migration/127334-21kthreadcore11:30:251
179930,0migration/110296-21python10:40:301
169930,0watchdog/126323-21cut08:15:271
169930,0watchdog/10-21swapper/109:50:231
169930,0watchdog/10-21swapper/109:00:311
169930,0watchdog/10-21swapper/107:15:231
149930,0migration/024489-21runrttasks06:48:260
127009932,0cyclictest7842-21cat07:40:272
126999932,0cyclictest31699-21awk09:40:231
126999932,0cyclictest12414-21kthreadcore10:05:271
126999931,0cyclictest14685-21cut08:10:221
126989932,0cyclictest3599-21lldpd06:22:060
126989932,0cyclictest3262-21irqbalance09:00:310
126989932,0cyclictest3262-21irqbalance08:10:110
126989932,0cyclictest28629-21diskmemload11:45:330
126989932,0cyclictest27612-21sh10:35:140
126989932,0cyclictest27612-21sh10:35:140
126989932,0cyclictest27503-21cut07:10:270
126989932,0cyclictest2698-21basename06:35:190
126989932,0cyclictest19442-21cut11:05:260
126989932,0cyclictest11669-21awk10:25:230
126989932,0cyclictest109272sleep011:40:110
126989932,0cyclictest10884-21kthreadcore08:25:280
126989931,0cyclictest13498-21kthreadcore09:10:260
126989931,0cyclictest0-21swapper/007:32:260
126989930,0cyclictest0-21swapper/008:15:280
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional