You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-15 - 12:33
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Sat Jun 15, 2024 00:50:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1771528956,29sleep60-21swapper/619:09:176
1626028641,38sleep20-21swapper/219:05:192
1761128248,29sleep50-21swapper/519:07:465
1765428147,30sleep10-21swapper/119:08:231
1762428146,30sleep70-21swapper/719:07:577
1758128147,29sleep30-21swapper/319:07:203
1516828046,29sleep40-21swapper/419:05:044
1752227844,30sleep00-21swapper/019:06:290
17932550,0sleep40-21swapper/422:01:264
131072480,0sleep20-21swapper/223:40:132
291592460,0sleep70-21swapper/719:25:177
181502450,0sleep70-21swapper/720:10:007
179332440,0sleep20-21swapper/223:57:502
18201992110,7cyclictest0-21swapper/422:07:294
18201992010,8cyclictest23291-21ssh21:56:104
18201992010,8cyclictest21179-21ssh22:41:294
18201992010,8cyclictest18893-21ssh23:11:244
18201992010,8cyclictest11720-21ssh23:08:294
18201992010,8cyclictest0-21swapper/422:20:274
18201992010,8cyclictest0-21swapper/422:10:184
18201992010,8cyclictest0-21swapper/421:45:174
18201992010,8cyclictest0-21swapper/421:45:174
18201992010,8cyclictest0-21swapper/421:20:094
18201992010,8cyclictest0-21swapper/420:34:504
18201992010,7cyclictest21218-21ssh23:28:374
18201992010,7cyclictest0-21swapper/423:46:294
18201992010,7cyclictest0-21swapper/423:46:294
18201992010,7cyclictest0-21swapper/420:10:224
18201992010,7cyclictest0-21swapper/400:27:554
18201992010,7cyclictest0-21swapper/400:16:474
18201992010,6cyclictest0-21swapper/422:33:274
18201992010,6cyclictest0-21swapper/400:21:574
18201992010,4cyclictest0-21swapper/421:26:514
18201991910,7cyclictest2759-21ssh00:05:324
18201991910,7cyclictest23530-1kworker/u17:022:27:294
18201991910,7cyclictest12137-21ssh22:37:314
18201991910,7cyclictest0-21swapper/423:35:164
18201991910,7cyclictest0-21swapper/423:31:074
18201991910,7cyclictest0-21swapper/423:20:114
18201991910,7cyclictest0-21swapper/423:15:264
18201991910,7cyclictest0-21swapper/423:00:084
18201991910,7cyclictest0-21swapper/422:55:164
18201991910,7cyclictest0-21swapper/422:50:454
18201991910,7cyclictest0-21swapper/421:51:004
18201991910,7cyclictest0-21swapper/421:12:104
18201991910,7cyclictest0-21swapper/420:05:004
18201991910,7cyclictest0-21swapper/419:40:194
18201991910,7cyclictest0-21swapper/419:15:114
18201991910,7cyclictest0-21swapper/400:37:004
18201991910,7cyclictest0-21swapper/400:32:354
18201991910,6cyclictest0-21swapper/423:40:084
18201991910,6cyclictest0-21swapper/422:15:574
18201991910,6cyclictest0-21swapper/421:38:264
18201991910,6cyclictest0-21swapper/421:33:194
18201991910,6cyclictest0-21swapper/421:15:074
18201991910,6cyclictest0-21swapper/420:58:144
18201991910,6cyclictest0-21swapper/419:31:544
18201991910,6cyclictest0-21swapper/419:24:224
18201991910,6cyclictest0-21swapper/400:11:254
18201991910,4cyclictest0-21swapper/419:12:554
1820099190,18cyclictest0-21swapper/319:10:103
18201991810,6cyclictest0-21swapper/423:57:284
18201991810,6cyclictest0-21swapper/423:51:224
18201991810,6cyclictest0-21swapper/422:46:364
18201991810,6cyclictest0-21swapper/421:40:034
18201991810,6cyclictest0-21swapper/421:05:224
18201991810,6cyclictest0-21swapper/421:01:124
18201991810,6cyclictest0-21swapper/420:52:084
18201991810,6cyclictest0-21swapper/420:45:464
18201991810,6cyclictest0-21swapper/420:40:244
18201991810,6cyclictest0-21swapper/420:35:594
18201991810,6cyclictest0-21swapper/420:26:004
18201991810,6cyclictest0-21swapper/420:23:424
18201991810,6cyclictest0-21swapper/420:15:124
18201991810,6cyclictest0-21swapper/420:06:414
18201991810,6cyclictest0-21swapper/419:57:474
18201991810,6cyclictest0-21swapper/419:47:054
18201991810,6cyclictest0-21swapper/419:35:514
18201991810,6cyclictest0-21swapper/400:00:074
18201991810,5cyclictest0-21swapper/419:51:364
18201991710,5cyclictest0-21swapper/419:25:074
257452160,0sleep00-21swapper/021:11:020
18202991615,1cyclictest13086-21ssh22:38:075
18204991510,5cyclictest0-21swapper/723:11:107
18204991510,5cyclictest0-21swapper/722:00:427
18203991515,0cyclictest0-21swapper/623:22:016
18203991515,0cyclictest0-21swapper/622:41:176
18203991513,1cyclictest682-21nscd00:26:036
18203991510,5cyclictest0-21swapper/621:27:046
18203991510,5cyclictest0-21swapper/600:18:426
18203991510,0cyclictest0-21swapper/623:57:496
18203991510,0cyclictest0-21swapper/600:34:356
18203991510,0cyclictest0-21swapper/600:14:206
18200991515,0cyclictest0-21swapper/300:18:213
18200991510,5cyclictest0-21swapper/323:51:063
18200991510,0cyclictest0-21swapper/323:03:053
18200991510,0cyclictest0-21swapper/300:00:423
18199991515,0cyclictest0-21swapper/223:52:522
18199991510,5cyclictest0-21swapper/223:48:352
18199991510,5cyclictest0-21swapper/223:48:352
18199991510,5cyclictest0-21swapper/222:56:482
18199991510,5cyclictest0-21swapper/221:33:032
18199991510,5cyclictest0-21swapper/200:24:162
18198991510,5cyclictest0-21swapper/100:14:161
18197991515,0cyclictest0-21swapper/023:00:150
18197991510,5cyclictest0-21swapper/023:56:390
18197991510,5cyclictest0-21swapper/023:22:240
18197991510,5cyclictest0-21swapper/022:47:560
18197991510,5cyclictest0-21swapper/000:32:400
751221410,3sleep70-21swapper/721:49:297
751221410,3sleep70-21swapper/721:49:297
3215921411,2sleep50-21swapper/523:49:315
3215921411,2sleep50-21swapper/523:49:305
1844221410,3sleep60-21swapper/621:03:496
18204991414,0cyclictest0-21swapper/720:25:097
18204991414,0cyclictest0-21swapper/700:19:037
18204991413,1cyclictest9728-21/usr/sbin/munin20:50:257
18204991412,1cyclictest682-21nscd21:28:247
18204991410,0cyclictest0-21swapper/723:34:097
18203991414,0cyclictest0-21swapper/623:46:396
18203991414,0cyclictest0-21swapper/623:46:396
18203991414,0cyclictest0-21swapper/622:56:396
18203991414,0cyclictest0-21swapper/622:02:156
18203991414,0cyclictest0-21swapper/621:42:496
18203991413,1cyclictest8318-21ssh00:09:096
18203991412,1cyclictest704-21ptp4l23:40:056
18202991412,1cyclictest682-21nscd21:05:175
18202991412,1cyclictest682-21nscd00:19:495
18202991412,1cyclictest21529-21diskmemload22:09:055
18202991412,1cyclictest21529-21diskmemload21:23:255
18202991412,1cyclictest1532-21cstates23:50:125
18202991410,4cyclictest0-21swapper/522:43:125
18202991410,4cyclictest0-21swapper/522:32:575
18202991410,4cyclictest0-21swapper/521:54:085
18202991410,4cyclictest0-21swapper/500:32:055
18200991414,0cyclictest0-21swapper/323:30:263
18200991414,0cyclictest0-21swapper/322:57:083
18200991414,0cyclictest0-21swapper/322:47:183
18200991414,0cyclictest0-21swapper/322:08:533
18200991414,0cyclictest0-21swapper/322:03:093
18200991414,0cyclictest0-21swapper/321:57:303
18200991414,0cyclictest0-21swapper/321:42:463
18200991414,0cyclictest0-21swapper/321:23:273
18200991410,4cyclictest0-21swapper/323:13:183
18200991410,4cyclictest0-21swapper/323:08:113
18200991410,4cyclictest0-21swapper/300:34:253
18200991410,0cyclictest0-21swapper/323:43:223
18199991414,0cyclictest0-21swapper/223:31:342
18199991414,0cyclictest0-21swapper/223:00:072
18199991414,0cyclictest0-21swapper/222:43:002
18199991414,0cyclictest0-21swapper/221:54:142
18199991412,1cyclictest682-21nscd22:34:292
18198991414,0cyclictest0-21swapper/123:21:581
18198991414,0cyclictest0-21swapper/121:59:351
18198991414,0cyclictest0-21swapper/119:10:131
18198991412,1cyclictest682-21nscd22:39:261
18198991410,4cyclictest0-21swapper/100:25:511
18198991410,0cyclictest0-21swapper/100:24:041
18197991414,0cyclictest0-21swapper/023:37:450
18197991414,0cyclictest0-21swapper/023:07:040
18197991414,0cyclictest0-21swapper/019:45:020
18197991410,1cyclictest121rcu_preempt23:34:440
18197991410,1cyclictest121rcu_preempt23:26:180
18197991410,1cyclictest121rcu_preempt00:22:070
18197991410,1cyclictest0-21swapper/021:19:010
262192130,0sleep00-21swapper/021:42:260
18204991313,0cyclictest0-21swapper/721:21:527
18204991312,1cyclictest17324-21latency22:55:197
18204991311,1cyclictest688-21nscd22:45:137
18204991311,1cyclictest682-21nscd23:09:357
18204991311,1cyclictest6117-21ssh22:19:437
18204991311,1cyclictest5476-21diskmemload21:32:367
18204991311,1cyclictest24047-21ssh21:10:227
18204991311,1cyclictest1803-21sendmail-mta00:30:027
18204991311,1cyclictest17771-21nscd23:16:077
18204991310,3cyclictest0-21swapper/720:41:397
18204991310,3cyclictest0-21swapper/719:42:347
18204991310,0cyclictest0-21swapper/723:44:247
18203991313,0cyclictest0-21swapper/623:28:486
18203991312,1cyclictest779-21gmain22:26:176
18203991312,1cyclictest682-21nscd21:53:406
18203991312,1cyclictest320-21systemd-journal22:30:026
18203991312,1cyclictest320-21systemd-journal21:38:586
18203991312,1cyclictest2177-1kworker/u17:122:49:176
18203991311,1cyclictest9296-21rm23:38:166
18203991311,1cyclictest682-21nscd23:30:006
18203991311,1cyclictest682-21nscd22:11:576
18203991311,1cyclictest682-21nscd21:13:556
18203991311,1cyclictest682-21nscd00:23:006
18203991311,1cyclictest682-21nscd00:03:196
18203991311,1cyclictest320-21systemd-journal19:55:266
18203991311,1cyclictest27855-21uptime23:15:296
18203991311,1cyclictest24235-1kworker/u17:221:15:146
18203991311,1cyclictest21529-21diskmemload23:54:486
18203991311,1cyclictest15838-21ssh22:24:146
18203991311,1cyclictest13261-21ssh21:20:246
18203991310,3cyclictest0-21swapper/619:46:496
18203991310,0cyclictest0-21swapper/620:06:346
18203991310,0cyclictest0-21swapper/619:53:116
18202991313,0cyclictest0-21swapper/522:20:015
18202991313,0cyclictest0-21swapper/522:14:535
18202991313,0cyclictest0-21swapper/519:10:055
18202991312,1cyclictest3403-1kworker/u17:021:27:005
18202991312,1cyclictest3403-1kworker/u17:021:19:455
18202991311,1cyclictest682-21nscd23:34:205
18202991311,1cyclictest682-21nscd23:16:035
18202991311,1cyclictest682-21nscd22:48:105
18202991311,1cyclictest682-21nscd22:00:555
18202991311,1cyclictest682-21nscd21:45:025
18202991311,1cyclictest682-21nscd21:45:025
18202991311,1cyclictest682-21nscd21:42:125
18202991311,1cyclictest682-21nscd21:10:415
18202991311,1cyclictest5365-21sh22:50:105
18202991311,1cyclictest24368-21nscd22:25:055
18202991311,1cyclictest13544-21ssh23:55:215
18202991311,1cyclictest12905-21cpuspeed_turbos23:40:145
18202991310,3cyclictest0-21swapper/523:36:365
18202991310,3cyclictest0-21swapper/523:04:165
18202991310,3cyclictest0-21swapper/521:58:535
18202991310,3cyclictest0-21swapper/520:51:465
18202991310,3cyclictest0-21swapper/520:25:155
18202991310,3cyclictest0-21swapper/520:07:065
18202991310,0cyclictest0-21swapper/523:26:365
18202991310,0cyclictest0-21swapper/520:56:285
18202991310,0cyclictest0-21swapper/520:35:145
18202991310,0cyclictest0-21swapper/520:24:505
18202991310,0cyclictest0-21swapper/500:29:085
18202991310,0cyclictest0-21swapper/500:06:375
18200991312,1cyclictest73550irq/125-eth023:46:393
18200991312,1cyclictest73550irq/125-eth023:46:393
18200991312,1cyclictest73550irq/125-eth023:15:553
18200991312,1cyclictest73550irq/125-eth022:33:433
18200991312,1cyclictest73550irq/125-eth022:22:313
18200991312,1cyclictest73550irq/125-eth021:30:423
18200991312,1cyclictest73550irq/125-eth021:10:213
18200991312,1cyclictest73550irq/125-eth020:56:553
18200991312,1cyclictest401ktimersoftd/300:09:483
18200991312,1cyclictest25754-21tr20:20:193
18200991311,1cyclictest73550irq/125-eth023:36:473
18200991311,1cyclictest73550irq/125-eth023:26:273
18200991311,1cyclictest682-21nscd21:17:573
18200991310,3cyclictest0-21swapper/320:30:163
18200991310,2cyclictest0-21swapper/319:16:153
18200991310,1cyclictest24368-21nscd21:46:473
18200991310,1cyclictest24368-21nscd21:46:473
18200991310,1cyclictest0-21swapper/322:15:023
18199991313,0cyclictest0-21swapper/221:44:322
18199991313,0cyclictest0-21swapper/200:16:202
18199991313,0cyclictest0-21swapper/200:02:212
18199991312,1cyclictest18732-21cstates20:10:122
18199991312,1cyclictest1811-21kworker/u16:222:18:192
18199991311,1cyclictest682-21nscd23:15:152
18199991311,1cyclictest682-21nscd22:20:422
18199991311,1cyclictest682-21nscd21:58:512
18199991311,1cyclictest682-21nscd21:11:012
18199991310,3cyclictest0-21swapper/220:52:522
18199991310,3cyclictest0-21swapper/220:46:422
18199991310,2cyclictest121rcu_preempt00:39:542
18199991310,0cyclictest0-21swapper/219:38:562
18198991313,0cyclictest0-21swapper/122:23:421
18198991313,0cyclictest0-21swapper/121:17:431
18198991312,1cyclictest1811-21kworker/u16:200:07:091
18198991312,1cyclictest15773-1kworker/u17:221:45:481
18198991312,1cyclictest15773-1kworker/u17:221:45:481
18198991311,1cyclictest704-21ptp4l22:00:091
18198991311,1cyclictest690-21nscd23:50:181
18198991311,1cyclictest682-21nscd23:42:341
18198991311,1cyclictest682-21nscd21:53:301
18198991311,1cyclictest682-21nscd21:35:371
18198991311,1cyclictest682-21nscd21:10:291
18198991311,1cyclictest682-21nscd00:30:031
18198991311,1cyclictest682-21nscd00:16:171
18198991311,1cyclictest23762-21timerwakeupswit21:25:241
18198991311,1cyclictest18635-21nscd23:25:311
18198991311,1cyclictest1811-21kworker/u16:200:00:261
18198991310,0cyclictest0-21swapper/123:57:461
18197991313,0cyclictest0-21swapper/021:58:140
18197991313,0cyclictest0-21swapper/021:50:180
18197991313,0cyclictest0-21swapper/020:35:340
18197991312,1cyclictest21529-21diskmemload21:45:250
18197991312,1cyclictest21529-21diskmemload21:45:250
18197991312,1cyclictest19369-21kworker/u16:021:31:140
18197991312,1cyclictest13577-21missed_timers20:00:220
18197991311,1cyclictest682-21nscd22:56:530
18197991311,1cyclictest32715-21cstates00:05:120
18197991310,0cyclictest0-21swapper/000:28:240
207592120,0sleep70-21swapper/722:10:267
18204991212,0cyclictest0-21swapper/723:46:037
18204991212,0cyclictest0-21swapper/723:46:027
18204991212,0cyclictest0-21swapper/722:20:257
18204991211,1cyclictest999-21sh23:50:047
18204991211,1cyclictest682-21nscd22:40:567
18204991211,1cyclictest6756-21if_err_eth022:50:187
18204991211,1cyclictest28438-21sendmail_mailtr23:00:257
18204991211,1cyclictest24368-21nscd22:05:227
18204991211,1cyclictest23341-21grep20:15:237
18204991211,1cyclictest22057-21/usr/sbin/munin19:15:257
18204991211,1cyclictest19706-21fschecks_count21:05:167
18204991211,1cyclictest19117-21fschecks_time20:10:157
18204991211,1cyclictest18195-21cyclictest23:56:007
18204991211,1cyclictest18195-21cyclictest23:20:347
18204991211,1cyclictest18195-21cyclictest22:35:037
18204991211,1cyclictest18195-21cyclictest22:25:037
18204991211,1cyclictest18195-21cyclictest21:15:097
18204991211,1cyclictest18195-21cyclictest20:45:097
18204991211,1cyclictest18195-21cyclictest20:21:017
18204991211,1cyclictest18195-21cyclictest20:01:197
18204991211,1cyclictest18195-21cyclictest19:55:167
18204991211,1cyclictest18195-21cyclictest19:51:077
18204991211,1cyclictest18195-21cyclictest19:37:197
18204991211,1cyclictest18195-21cyclictest19:33:467
18204991211,1cyclictest18195-21cyclictest19:22:057
18204991211,1cyclictest1811-21kworker/u16:200:00:017
18204991211,1cyclictest1624-21runrttasks19:45:487
18204991211,1cyclictest16236-21nscd22:31:387
18204991211,1cyclictest13156-21cstates20:55:137
18204991211,1cyclictest11299-21nscd23:35:157
18204991211,1cyclictest0-21swapper/721:55:377
18204991210,1cyclictest9925-21diskmemload21:35:117
18204991210,1cyclictest433-21cstates00:05:127
18204991210,1cyclictest30960-21ssh00:20:057
18204991210,1cyclictest30479-21cstates20:30:147
18204991210,1cyclictest29998-21ssh00:35:017
18204991210,1cyclictest23802-21sh21:40:537
18204991210,1cyclictest1869-21idleruntime20:35:157
18204991210,1cyclictest16207-21cstates21:00:117
18204991210,1cyclictest14650-21diskmemload23:25:157
18204991210,1cyclictest10281-21if_err_eth021:50:157
18204991210,1cyclictest10221-21diskmemload00:10:087
18204991210,1cyclictest0-21swapper/700:25:047
18204991210,0cyclictest0-21swapper/719:10:267
18203991212,0cyclictest0-21swapper/620:55:066
18203991212,0cyclictest0-21swapper/620:47:546
18203991212,0cyclictest0-21swapper/619:25:066
18203991211,1cyclictest682-21nscd23:05:126
18203991211,1cyclictest682-21nscd22:05:056
18203991211,1cyclictest320-21systemd-journal20:20:006
18203991211,1cyclictest320-21systemd-journal20:15:286
18203991211,1cyclictest320-21systemd-journal19:31:186
18203991211,1cyclictest320-21systemd-journal19:20:116
18203991211,1cyclictest31803-21cpuspeed_turbos19:35:086
18203991211,1cyclictest31386-21cstates21:45:136
18203991211,1cyclictest31386-21cstates21:45:136
18203991211,1cyclictest28370-21/usr/sbin/munin20:25:146
18203991211,1cyclictest26269-21ssh23:00:086
18203991211,1cyclictest19369-21kworker/u16:022:15:096
18203991211,1cyclictest18973-21cpuspeed_turbos21:05:146
18203991211,1cyclictest18195-21cyclictest22:35:076
18203991211,1cyclictest18195-21cyclictest20:51:196
18203991211,1cyclictest18195-21cyclictest20:30:066
18203991211,1cyclictest18195-21cyclictest20:01:016
18203991211,1cyclictest18195-21cyclictest19:40:116
18203991211,1cyclictest18195-21cyclictest19:15:286
18203991211,1cyclictest17032-21grep23:10:256
18203991211,1cyclictest15060-1kworker/u17:121:55:086
18203991211,1cyclictest1043-21cstates20:35:106
18203991211,1cyclictest0-21swapper/622:50:146
18203991211,1cyclictest0-21swapper/621:30:186
18203991211,1cyclictest0-21swapper/620:40:176
18203991210,1cyclictest19589-21missed_timers20:10:206
18203991210,1cyclictest19123-21fschecks_time19:10:136
18203991210,1cyclictest1306-21ssh00:35:586
18202991212,0cyclictest0-21swapper/519:35:365
18202991211,1cyclictest704-21ptp4l21:00:415
18202991211,1cyclictest32561-21/usr/sbin/munin00:20:195
18202991211,1cyclictest29061-21ssh22:15:075
18202991211,1cyclictest21944-21/usr/sbin/munin00:00:215
18202991211,1cyclictest18195-21cyclictest20:45:035
18202991211,1cyclictest18195-21cyclictest20:40:035
18202991211,1cyclictest18195-21cyclictest19:40:105
18202991211,1cyclictest18195-21cyclictest19:30:175
18202991211,1cyclictest18195-21cyclictest00:10:135
18202991211,1cyclictest1811-21kworker/u16:223:10:095
18202991211,1cyclictest17771-21nscd23:05:145
18202991210,2cyclictest0-21swapper/521:30:005
18202991210,2cyclictest0-21swapper/520:15:325
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional