You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 18:49
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Sun Apr 28, 2024 12:50:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
63321610,6sleep20-21swapper/207:05:262
148628573,6sleep40-21swapper/407:05:384
270128451,29sleep30-21swapper/307:09:543
243528451,29sleep70-21swapper/707:06:057
255428248,29sleep60-21swapper/607:07:476
3252127944,30sleep10-21swapper/107:05:091
247327949,25sleep00-21swapper/007:06:380
244227540,30sleep50-21swapper/507:06:115
321472560,0sleep00-21swapper/010:19:490
37752530,0sleep60-21swapper/609:05:006
309112500,0sleep40-21swapper/411:14:224
187622500,0sleep50-21swapper/510:11:055
223382490,0sleep60-21swapper/609:36:066
297732470,0sleep30-21swapper/307:55:153
298322440,0sleep10-21swapper/112:27:451
298322440,0sleep10-21swapper/112:27:451
5642400,0sleep20-21swapper/209:00:002
3075991811,6cyclictest7690-21sendmail_mailqu11:00:201
3075991811,6cyclictest22669-21missed_timers08:40:161
3075991711,5cyclictest22519-21sh11:45:341
3075991710,0cyclictest0-21swapper/108:10:261
3113991615,1cyclictest27731-21ssh10:54:077
3113991615,1cyclictest13559-21ssh10:26:517
3075991615,1cyclictest24177-21ssh10:15:101
3075991611,4cyclictest9140-21sh10:43:081
3075991611,4cyclictest9140-21sh10:43:071
3108991515,0cyclictest0-21swapper/611:40:126
3083991515,0cyclictest0-21swapper/212:26:132
3083991515,0cyclictest0-21swapper/212:26:132
3083991510,5cyclictest0-21swapper/211:45:182
3083991510,5cyclictest0-21swapper/211:38:062
3083991510,5cyclictest0-21swapper/211:13:202
3083991510,0cyclictest0-21swapper/211:23:352
3075991515,0cyclictest0-21swapper/112:34:211
3075991515,0cyclictest0-21swapper/112:22:401
3075991514,1cyclictest15905-21ssh11:05:181
3075991511,4cyclictest0-21swapper/112:19:441
3075991511,3cyclictest682-21nscd09:20:311
3075991511,3cyclictest14601-21taskset08:28:051
3075991510,0cyclictest0-21swapper/112:11:481
3075991510,0cyclictest0-21swapper/110:21:331
3068991515,0cyclictest24776-21ssh11:29:220
3068991510,1cyclictest0-21swapper/012:34:410
3068991510,1cyclictest0-21swapper/007:10:050
3113991412,1cyclictest10484-21rm10:44:147
3113991412,1cyclictest10484-21rm10:44:147
3108991414,0cyclictest0-21swapper/611:51:286
3108991414,0cyclictest0-21swapper/611:25:276
3108991414,0cyclictest0-21swapper/611:20:156
3108991413,1cyclictest13742-21ssh09:31:176
3108991412,1cyclictest21084-21ssh09:17:356
3108991410,0cyclictest0-21swapper/610:35:536
3108991410,0cyclictest0-21swapper/609:24:286
3101991414,0cyclictest0-21swapper/512:17:465
3101991414,0cyclictest0-21swapper/512:13:425
3101991414,0cyclictest0-21swapper/512:06:365
3101991414,0cyclictest0-21swapper/509:43:465
3101991414,0cyclictest0-21swapper/508:10:125
3101991413,1cyclictest3060-21cyclictest11:40:305
3101991412,1cyclictest73550irq/125-eth010:24:095
3101991411,2cyclictest0-21swapper/510:00:045
3101991410,4cyclictest0-21swapper/510:20:005
3096991414,0cyclictest0-21swapper/412:33:264
3096991414,0cyclictest0-21swapper/411:57:194
3096991414,0cyclictest0-21swapper/411:30:424
3096991413,1cyclictest14951-21rm11:23:294
3096991412,1cyclictest6998-21kworker/u16:109:41:444
3096991412,1cyclictest682-21nscd10:02:184
3096991410,4cyclictest0-21swapper/409:49:054
3096991410,0cyclictest0-21swapper/411:47:174
3096991410,0cyclictest0-21swapper/409:31:594
3089991412,1cyclictest682-21nscd10:28:193
3089991412,1cyclictest682-21nscd10:12:373
3089991410,0cyclictest0-21swapper/310:02:023
3083991414,0cyclictest0-21swapper/212:00:132
3083991414,0cyclictest0-21swapper/211:17:262
3083991414,0cyclictest0-21swapper/210:28:082
3083991414,0cyclictest0-21swapper/209:52:582
3083991414,0cyclictest0-21swapper/209:52:572
3083991414,0cyclictest0-21swapper/209:48:412
3083991412,1cyclictest1651-21sshd12:24:292
3083991410,3cyclictest0-21swapper/210:40:212
3083991410,3cyclictest0-21swapper/210:40:212
3075991414,0cyclictest0-21swapper/111:36:561
3075991414,0cyclictest0-21swapper/109:19:311
3075991413,1cyclictest32125-21ssh11:15:021
3075991410,4cyclictest0-21swapper/109:57:271
3075991410,0cyclictest0-21swapper/108:18:011
3068991411,0cyclictest0-21swapper/007:54:500
3068991410,1cyclictest0-21swapper/009:31:210
3068991410,0cyclictest0-21swapper/011:52:470
3068991410,0cyclictest0-21swapper/009:12:310
3113991313,0cyclictest0-21swapper/709:58:427
3113991313,0cyclictest0-21swapper/709:40:267
3113991312,1cyclictest26681-1kworker/u17:212:21:117
3113991312,1cyclictest21097-21kworker/u16:310:10:267
3113991311,1cyclictest682-21nscd12:01:417
3113991311,1cyclictest682-21nscd11:51:357
3113991311,1cyclictest6484-21diskmemload11:55:067
3113991311,1cyclictest2841-21timerandwakeup09:00:287
3113991311,1cyclictest25520-21mailstats11:10:237
3113991311,1cyclictest18223-21fschecks_count11:25:137
3113991311,1cyclictest15735-21diskmemload11:42:067
3113991310,1cyclictest121rcu_preempt09:21:077
3108991312,1cyclictest6484-21diskmemload11:35:116
3108991312,1cyclictest6484-21diskmemload10:01:036
3108991312,1cyclictest6280-21munin-node12:28:506
3108991312,1cyclictest6280-21munin-node12:28:496
3108991312,1cyclictest5412-1kworker/u17:012:15:486
3108991312,1cyclictest30615-21ssh09:40:436
3108991312,1cyclictest1796-1kworker/u17:012:00:366
3108991311,1cyclictest701-21avahi-daemon09:45:166
3108991311,1cyclictest6484-21diskmemload10:33:536
3108991311,1cyclictest2513-21nscd11:56:096
3108991311,1cyclictest11843-21sh10:25:276
3108991310,0cyclictest0-21swapper/607:20:076
3101991313,0cyclictest0-21swapper/512:30:375
3101991313,0cyclictest0-21swapper/511:51:335
3101991313,0cyclictest0-21swapper/511:47:365
3101991312,1cyclictest73550irq/125-eth012:35:265
3101991312,1cyclictest73550irq/125-eth012:28:555
3101991312,1cyclictest73550irq/125-eth012:28:555
3101991312,1cyclictest73550irq/125-eth012:04:045
3101991312,1cyclictest73550irq/125-eth011:29:415
3101991312,1cyclictest73550irq/125-eth010:59:085
3101991312,1cyclictest73550irq/125-eth010:45:235
3101991312,1cyclictest73550irq/125-eth009:30:215
3101991312,1cyclictest73550irq/125-eth007:50:145
3101991312,1cyclictest73550irq/125-eth007:17:045
3101991311,2cyclictest73550irq/125-eth009:17:275
3101991311,1cyclictest704-21ptp4l07:37:535
3101991311,1cyclictest25097-21cpuspeed_turbos09:20:115
3101991311,1cyclictest2173-21timerandwakeup11:15:265
3101991311,1cyclictest18558-21cstates10:30:115
3101991310,3cyclictest0-21swapper/510:50:215
3101991310,3cyclictest0-21swapper/509:25:285
3101991310,3cyclictest0-21swapper/509:25:275
3101991310,3cyclictest0-21swapper/508:21:115
3101991310,2cyclictest0-21swapper/509:50:015
3101991310,2cyclictest0-21swapper/508:40:015
3101991310,1cyclictest682-21nscd10:41:585
3101991310,1cyclictest682-21nscd10:41:585
3101991310,1cyclictest32370-21nscd11:02:075
3101991310,1cyclictest0-21swapper/511:59:035
3101991310,0cyclictest0-21swapper/510:35:195
3096991313,0cyclictest0-21swapper/410:35:214
3096991313,0cyclictest0-21swapper/410:26:424
3096991313,0cyclictest0-21swapper/409:25:364
3096991313,0cyclictest0-21swapper/409:25:364
3096991312,1cyclictest6484-21diskmemload11:16:564
3096991312,1cyclictest32072-21snmpd12:28:334
3096991312,1cyclictest32072-21snmpd12:28:334
3096991312,1cyclictest28820-21kworker/u16:209:11:124
3096991311,1cyclictest15727-21switchtime12:00:244
3096991311,1cyclictest13774-21sh11:40:294
3096991310,0cyclictest0-21swapper/408:55:274
3096991310,0cyclictest0-21swapper/408:15:524
3089991313,0cyclictest0-21swapper/311:23:153
3089991313,0cyclictest0-21swapper/310:16:193
3089991312,1cyclictest779-21gmain10:40:343
3089991312,1cyclictest779-21gmain10:40:343
3089991312,1cyclictest32072-21snmpd12:13:513
3089991312,1cyclictest201-21md0_raid109:26:083
3089991312,1cyclictest201-21md0_raid109:26:073
3089991311,1cyclictest682-21nscd11:52:593
3089991311,1cyclictest522-21/usr/sbin/munin10:20:113
3089991311,1cyclictest27497-21cpuspeed_turbos08:50:093
3089991311,1cyclictest0-21swapper/311:05:003
3089991310,3cyclictest0-21swapper/312:38:073
3089991310,3cyclictest0-21swapper/308:30:203
3089991310,3cyclictest0-21swapper/307:45:403
3089991310,1cyclictest0-21swapper/312:25:223
3089991310,1cyclictest0-21swapper/312:25:213
3089991310,0cyclictest0-21swapper/309:14:123
3089991310,0cyclictest0-21swapper/307:32:123
3083991313,0cyclictest0-21swapper/212:08:172
3083991313,0cyclictest0-21swapper/210:15:252
3083991311,1cyclictest704-21ptp4l09:09:422
3083991311,1cyclictest704-21ptp4l09:09:412
3083991311,1cyclictest704-21ptp4l08:20:202
3083991311,1cyclictest692-21nscd10:20:442
3083991311,1cyclictest691-21nscd11:59:512
3083991311,1cyclictest688-21nscd12:15:162
3083991311,1cyclictest682-21nscd11:28:102
3083991311,1cyclictest682-21nscd11:07:512
3083991311,1cyclictest682-21nscd09:26:542
3083991311,1cyclictest682-21nscd09:26:542
3083991311,1cyclictest32072-21snmpd07:41:402
3083991311,1cyclictest28855-21/usr/sbin/munin07:55:122
3083991311,1cyclictest25814-1kworker/u17:009:40:282
3083991311,1cyclictest2513-21nscd10:45:442
3083991311,1cyclictest23232-21diskmemload09:36:512
3083991311,1cyclictest23052-21mailstats08:40:202
3083991311,1cyclictest18635-21nscd09:15:192
3083991311,1cyclictest16661-21if_eth010:10:122
3083991311,1cyclictest11299-21nscd10:50:022
3083991311,1cyclictest11127-21cpuspeed_turbos07:25:092
3083991311,1cyclictest11059-21nscd09:21:142
3083991310,3cyclictest0-21swapper/211:41:252
3083991310,2cyclictest0-21swapper/212:37:282
3083991310,2cyclictest0-21swapper/212:10:252
3083991310,2cyclictest0-21swapper/211:50:212
3083991310,2cyclictest0-21swapper/208:47:052
3083991310,2cyclictest0-21swapper/208:30:372
3083991310,1cyclictest0-21swapper/210:33:362
3083991310,0cyclictest0-21swapper/208:50:382
3083991310,0cyclictest0-21swapper/208:01:032
3075991313,0cyclictest0-21swapper/110:04:381
3075991313,0cyclictest0-21swapper/109:12:191
3075991312,1cyclictest27960-21fschecks_time10:35:151
3075991311,1cyclictest9120-21nscd10:56:541
3075991311,1cyclictest704-21ptp4l10:49:481
3075991311,1cyclictest704-21ptp4l10:27:121
3075991311,1cyclictest701-21avahi-daemon09:47:311
3075991311,1cyclictest688-21nscd09:25:441
3075991311,1cyclictest688-21nscd09:25:441
3075991311,1cyclictest3060-21cyclictest10:07:351
3075991311,1cyclictest1624-21runrttasks09:01:071
3075991310,0cyclictest0-21swapper/107:30:551
3068991313,0cyclictest0-21swapper/012:26:320
3068991313,0cyclictest0-21swapper/012:26:320
3068991313,0cyclictest0-21swapper/012:01:590
3068991313,0cyclictest0-21swapper/009:05:070
3068991313,0cyclictest0-21swapper/009:05:070
3068991312,1cyclictest23017-21sh09:19:070
3068991312,1cyclictest21097-21kworker/u16:309:48:510
3068991311,1cyclictest6484-21diskmemload11:10:200
3068991311,1cyclictest31459-21if_eth008:55:190
3068991311,1cyclictest21176-21sh12:22:410
3068991310,0cyclictest0-21swapper/010:09:120
3068991310,0cyclictest0-21swapper/008:51:140
3068991310,0cyclictest0-21swapper/008:37:140
3068991310,0cyclictest0-21swapper/008:30:450
3068991310,0cyclictest0-21swapper/007:20:290
2828721310,2sleep40-21swapper/407:52:304
257232130,0sleep10-21swapper/112:06:071
2052221310,2sleep70-21swapper/708:37:327
3113991211,1cyclictest9201-21cstates11:20:127
3113991211,1cyclictest779-21gmain08:27:017
3113991211,1cyclictest7300-21cstates09:10:107
3113991211,1cyclictest704-21ptp4l07:48:007
3113991211,1cyclictest4063-21/usr/sbin/munin09:45:127
3113991211,1cyclictest3263-1kworker/u17:211:00:147
3113991211,1cyclictest3060-21cyclictest08:58:027
3113991211,1cyclictest3060-21cyclictest08:42:387
3113991211,1cyclictest3060-21cyclictest07:28:237
3113991211,1cyclictest22832-21sh11:45:517
3113991211,1cyclictest21552-21ssh09:35:297
3113991211,1cyclictest20696-21kworker/u16:211:30:097
3113991211,1cyclictest15949-1kworker/u17:011:05:277
3113991211,1cyclictest1573-21fschecks_count12:30:157
3113991211,1cyclictest11565-21tr08:20:107
3113991210,2cyclictest0-21swapper/712:15:137
3113991210,2cyclictest0-21swapper/708:15:157
3113991210,2cyclictest0-21swapper/707:34:597
3113991210,1cyclictest7016-21cstates10:05:097
3113991210,1cyclictest682-21nscd12:10:017
3113991210,1cyclictest6517-21processes07:15:017
3113991210,1cyclictest6484-21diskmemload10:55:097
3113991210,1cyclictest5135-21latency09:05:187
3113991210,1cyclictest5135-21latency09:05:177
3113991210,1cyclictest3688-21latency08:05:147
3113991210,1cyclictest32370-21nscd10:00:127
3113991210,1cyclictest32072-21snmpd07:20:567
3113991210,1cyclictest3130-21df_abs11:35:147
3113991210,1cyclictest29141-21cstates07:55:137
3113991210,1cyclictest28183-21kworker/u16:111:15:057
3113991210,1cyclictest28122-21sed10:35:157
3113991210,1cyclictest27430-21tr08:50:097
3113991210,1cyclictest2704-21ssh09:25:207
3113991210,1cyclictest2704-21ssh09:25:197
3113991210,1cyclictest26937-21nscd09:51:387
3113991210,1cyclictest26937-21nscd09:51:377
3113991210,1cyclictest26257-21timerandwakeup08:45:237
3113991210,1cyclictest25872-21/usr/sbin/munin07:50:097
3113991210,1cyclictest24936-21ssh10:15:127
3113991210,1cyclictest24573-21ssh12:25:127
3113991210,1cyclictest24573-21ssh12:25:117
3113991210,1cyclictest22508-21latency_hist07:45:017
3113991210,1cyclictest22357-21/usr/sbin/munin12:05:217
3113991210,1cyclictest19613-21ls10:30:177
3113991210,1cyclictest18708-21python07:35:227
3113991210,1cyclictest18521-21ssh09:15:327
3113991210,1cyclictest17978-21latency_hist08:35:007
3113991210,1cyclictest15121-21ssh10:45:547
3113991210,1cyclictest1122-21cstates10:20:127
3113991210,1cyclictest10764-21cstates09:30:137
3113991210,1cyclictest0-21swapper/712:35:147
3108991212,0cyclictest0-21swapper/611:45:256
3108991212,0cyclictest0-21swapper/611:15:026
3108991212,0cyclictest0-21swapper/607:35:096
3108991212,0cyclictest0-21swapper/607:10:046
3108991211,1cyclictest931-21grep08:00:226
3108991211,1cyclictest8809-21pmu-power07:15:196
3108991211,1cyclictest779-21gmain10:50:066
3108991211,1cyclictest704-21ptp4l09:25:026
3108991211,1cyclictest704-21ptp4l09:25:026
3108991211,1cyclictest704-21ptp4l08:12:036
3108991211,1cyclictest704-21ptp4l07:40:016
3108991211,1cyclictest6749-21latency_hist09:10:016
3108991211,1cyclictest6749-21latency_hist09:10:006
3108991211,1cyclictest5886-1kworker/u17:010:05:396
3108991211,1cyclictest30942-1kworker/u17:210:46:206
3108991211,1cyclictest3060-21cyclictest12:05:216
3108991211,1cyclictest3060-21cyclictest10:40:276
3108991211,1cyclictest3060-21cyclictest10:40:276
3108991211,1cyclictest3060-21cyclictest08:50:096
3108991211,1cyclictest3060-21cyclictest08:46:296
3108991211,1cyclictest3060-21cyclictest07:58:436
3108991211,1cyclictest2729-21cstates08:05:106
3108991211,1cyclictest24582-1kworker/u17:109:10:516
3108991211,1cyclictest23984-21unixbench_multi09:55:256
3108991211,1cyclictest23505-21/usr/sbin/munin11:10:236
3108991211,1cyclictest21602-21cstates08:40:106
3108991211,1cyclictest20335-21unixbench_singl08:35:276
3108991211,1cyclictest20169-21ssh12:21:516
3108991211,1cyclictest17969-21ssh10:10:266
3108991211,1cyclictest17445-21runrttasks08:31:096
3108991211,1cyclictest1596-21gltestperf10:20:146
3108991211,1cyclictest1569-21fschecks_count12:30:156
3108991211,1cyclictest10630-21nscd11:05:116
3108991211,1cyclictest10112-1kworker/u17:110:55:036
3108991210,1cyclictest9120-21nscd11:30:216
3108991210,1cyclictest7752-21mailstats11:00:206
3108991210,1cyclictest6484-21diskmemload10:16:366
3108991210,1cyclictest30837-21ssh12:10:016
3108991210,1cyclictest30315-21cpu08:55:146
3108991210,1cyclictest16236-21nscd12:35:006
3108991210,1cyclictest15600-21sh09:50:416
3108991210,1cyclictest15600-21sh09:50:406
3108991210,1cyclictest15572-21proc_pri07:30:186
3108991210,1cyclictest14701-21runrttasks08:29:316
3108991210,1cyclictest12291-21missed_timers07:25:136
3101991211,1cyclictest73550irq/125-eth012:20:085
3101991211,1cyclictest73550irq/125-eth011:30:085
3101991211,1cyclictest73550irq/125-eth011:20:015
3101991211,1cyclictest73550irq/125-eth011:05:115
3101991211,1cyclictest73550irq/125-eth010:25:085
3101991211,1cyclictest73550irq/125-eth010:05:125
3101991211,1cyclictest73550irq/125-eth009:55:075
3101991211,1cyclictest73550irq/125-eth009:35:085
3101991211,1cyclictest73550irq/125-eth009:10:025
3101991211,1cyclictest73550irq/125-eth009:00:145
3101991211,1cyclictest73550irq/125-eth008:55:145
3101991211,1cyclictest73550irq/125-eth008:50:335
3101991211,1cyclictest73550irq/125-eth008:30:085
3101991211,1cyclictest73550irq/125-eth008:25:475
3101991211,1cyclictest73550irq/125-eth008:15:055
3101991211,1cyclictest73550irq/125-eth008:00:125
3101991211,1cyclictest73550irq/125-eth007:55:265
3101991211,1cyclictest73550irq/125-eth007:45:195
3101991211,1cyclictest32072-21snmpd11:35:035
3101991211,1cyclictest12650-21sh09:50:045
3101991211,1cyclictest12650-21sh09:50:035
3101991210,2cyclictest0-21swapper/507:22:205
3101991210,1cyclictest704-21ptp4l09:05:105
3101991210,1cyclictest704-21ptp4l09:05:105
3101991210,1cyclictest673-21irqbalance08:05:015
3101991210,1cyclictest6280-21munin-node08:35:075
3101991210,1cyclictest5446-21taskset07:10:435
3101991210,1cyclictest20559-21df_abs07:40:125
3101991210,1cyclictest14906-21gltestperf07:30:135
3101991210,0cyclictest0-21swapper/511:10:025
3101991210,0cyclictest0-21swapper/508:45:145
3101991210,0cyclictest0-21swapper/507:25:085
3096991212,0cyclictest0-21swapper/409:06:434
3096991212,0cyclictest0-21swapper/409:06:424
3096991212,0cyclictest0-21swapper/407:16:024
3096991211,1cyclictest689-21nscd10:30:194
3096991211,1cyclictest6484-21diskmemload11:25:064
3096991211,1cyclictest3060-21cyclictest08:27:364
3096991211,1cyclictest3060-21cyclictest08:21:574
3096991211,1cyclictest3060-21cyclictest08:05:004
3096991211,1cyclictest3060-21cyclictest07:57:094
3096991211,1cyclictest3060-21cyclictest07:46:214
3096991211,1cyclictest3060-21cyclictest07:20:324
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional