You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-04 - 12:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Tue Nov 04, 2025 00:50:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2344828550,30sleep20-21swapper/219:08:322
2350028349,30sleep70-21swapper/719:09:187
2349028349,30sleep60-21swapper/619:09:086
2328028349,30sleep50-21swapper/519:06:105
2330628147,30sleep30-21swapper/319:06:323
2337927944,30sleep00-21swapper/019:07:340
2353227643,29sleep10-21swapper/119:09:431
2352627542,29sleep40-21swapper/419:09:384
293252560,0sleep50-21swapper/500:16:365
23985995510,45cyclictest0-21swapper/522:46:575
23983995555,0cyclictest0-21swapper/300:17:343
23987995410,0cyclictest0-21swapper/723:16:257
23986995410,0cyclictest0-21swapper/623:20:126
23984995410,0cyclictest0-21swapper/423:58:304
23982995453,0cyclictest0-21swapper/200:14:282
23987995353,0cyclictest0-21swapper/723:20:297
23987995353,0cyclictest0-21swapper/721:56:267
23987995310,43cyclictest0-21swapper/722:22:497
23985995353,0cyclictest0-21swapper/523:47:225
23982995353,0cyclictest0-21swapper/221:41:022
23980995352,0cyclictest0-21swapper/023:12:450
23980995352,0cyclictest0-21swapper/023:12:450
23980995310,43cyclictest0-21swapper/023:09:500
23987995251,0cyclictest0-21swapper/722:25:157
23987995210,0cyclictest0-21swapper/719:35:137
23985995252,0cyclictest0-21swapper/500:30:275
23985995252,0cyclictest0-21swapper/500:25:555
23984995252,0cyclictest0-21swapper/400:12:524
23983995251,0cyclictest0-21swapper/300:25:043
23982995210,0cyclictest0-21swapper/222:59:562
23980995252,0cyclictest0-21swapper/019:10:150
23980995251,0cyclictest0-21swapper/022:30:340
23987995151,0cyclictest0-21swapper/700:29:367
23985995151,0cyclictest0-21swapper/500:39:285
23985995110,41cyclictest0-21swapper/522:55:435
23984995151,0cyclictest0-21swapper/422:30:194
23981995151,0cyclictest0-21swapper/122:55:041
23981995151,0cyclictest0-21swapper/100:30:351
23981995150,0cyclictest0-21swapper/123:22:581
23987995050,0cyclictest0-21swapper/723:30:217
23987995050,0cyclictest0-21swapper/722:17:427
23986995050,0cyclictest0-21swapper/600:10:126
23983995050,0cyclictest0-21swapper/321:10:013
23983995049,0cyclictest0-21swapper/322:46:003
23983995049,0cyclictest0-21swapper/300:24:403
23982995050,0cyclictest0-21swapper/200:17:052
23980995010,39cyclictest0-21swapper/023:26:370
23987994949,0cyclictest0-21swapper/723:53:307
23985994949,0cyclictest0-21swapper/523:35:555
23985994949,0cyclictest0-21swapper/500:21:235
23984994949,0cyclictest0-21swapper/400:30:264
23983994948,0cyclictest0-21swapper/321:48:223
23981994949,0cyclictest0-21swapper/123:15:591
23981994948,0cyclictest0-21swapper/122:50:421
23981994948,0cyclictest0-21swapper/122:41:511
23980994910,0cyclictest0-21swapper/022:29:450
23987994848,0cyclictest0-21swapper/700:03:477
23986994848,0cyclictest0-21swapper/623:59:256
23983994847,0cyclictest0-21swapper/300:14:513
23982994810,0cyclictest0-21swapper/221:48:282
23982994810,0cyclictest0-21swapper/200:27:112
23981994848,0cyclictest0-21swapper/100:05:371
23980994848,0cyclictest0-21swapper/022:35:420
74082470,0sleep773-21ksoftirqd/722:47:037
23985994710,37cyclictest0-21swapper/500:02:045
23985994710,36cyclictest0-21swapper/522:10:175
23984994745,1cyclictest0-21swapper/423:50:224
23982994747,0cyclictest0-21swapper/223:15:402
23982994746,0cyclictest0-21swapper/222:52:342
23987994646,0cyclictest0-21swapper/723:29:567
23985994646,0cyclictest0-21swapper/522:34:435
23983994645,0cyclictest0-21swapper/322:15:173
23981994645,0cyclictest0-21swapper/123:27:031
23986994543,1cyclictest12862-21runrttasks00:25:056
23986994510,0cyclictest0-21swapper/600:33:426
2398699450,0cyclictest0-21swapper/623:06:586
23984994545,0cyclictest0-21swapper/423:37:444
23984994545,0cyclictest0-21swapper/422:21:524
23984994544,1cyclictest0-21swapper/422:15:204
23983994510,34cyclictest0-21swapper/323:08:203
23982994543,1cyclictest0-21swapper/200:00:242
2398299450,44cyclictest0-21swapper/223:41:382
23981994542,3cyclictest0-21swapper/123:10:211
23981994542,3cyclictest0-21swapper/123:10:211
23980994545,0cyclictest0-21swapper/021:35:130
23987994444,0cyclictest0-21swapper/723:09:507
23986994444,0cyclictest0-21swapper/623:19:246
23985994444,0cyclictest0-21swapper/523:05:225
23985994444,0cyclictest0-21swapper/522:52:015
23985994444,0cyclictest0-21swapper/500:10:185
2398599440,43cyclictest21275-21awk23:25:175
23984994443,1cyclictest0-21swapper/423:40:404
23983994444,0cyclictest0-21swapper/322:59:563
23983994443,0cyclictest0-21swapper/323:14:093
23983994443,0cyclictest0-21swapper/323:14:093
23983994443,0cyclictest0-21swapper/321:41:023
23982994444,0cyclictest0-21swapper/223:35:222
23982994443,0cyclictest0-21swapper/223:50:312
2398299440,0cyclictest0-21swapper/219:20:122
23981994444,0cyclictest0-21swapper/123:51:571
23981994443,0cyclictest0-21swapper/100:11:061
23980994444,0cyclictest0-21swapper/023:57:540
23987994343,0cyclictest0-21swapper/722:30:217
23987994310,33cyclictest0-21swapper/700:31:427
23987994310,0cyclictest0-21swapper/721:41:057
23986994343,0cyclictest0-21swapper/623:13:536
23986994343,0cyclictest0-21swapper/623:13:536
23986994343,0cyclictest0-21swapper/600:24:026
23986994341,1cyclictest27602-21kworker/u16:322:53:466
2398699430,43cyclictest0-21swapper/621:54:346
23985994310,0cyclictest0-21swapper/523:51:455
2398499430,0cyclictest0-21swapper/400:20:004
2398499430,0cyclictest0-21swapper/400:15:204
23983994343,0cyclictest0-21swapper/323:39:053
23982994343,0cyclictest0-21swapper/223:29:292
23982994342,0cyclictest0-21swapper/222:30:172
23982994342,0cyclictest0-21swapper/221:55:132
23982994341,1cyclictest3472-21kworker/u16:122:25:162
23981994343,0cyclictest0-21swapper/122:45:401
23980994343,0cyclictest0-21swapper/000:01:330
23980994310,32cyclictest0-21swapper/000:14:310
23987994242,0cyclictest0-21swapper/723:12:457
23987994242,0cyclictest0-21swapper/723:12:457
23987994242,0cyclictest0-21swapper/722:35:187
2398799420,0cyclictest0-21swapper/723:02:507
23986994242,0cyclictest0-21swapper/600:01:156
23985994242,0cyclictest0-21swapper/523:58:285
23985994242,0cyclictest0-21swapper/522:42:515
2398599420,42cyclictest0-21swapper/522:01:255
23984994241,0cyclictest0-21swapper/422:59:444
23982994242,0cyclictest0-21swapper/200:05:362
23981994210,31cyclictest0-21swapper/123:46:441
23980994242,0cyclictest0-21swapper/023:52:470
23980994242,0cyclictest0-21swapper/022:40:040
23980994242,0cyclictest0-21swapper/000:34:270
2398099420,0cyclictest0-21swapper/022:45:210
23987994141,0cyclictest0-21swapper/719:10:167
23986994141,0cyclictest0-21swapper/623:33:176
23986994141,0cyclictest0-21swapper/622:40:336
23986994110,0cyclictest0-21swapper/623:53:546
23985994140,0cyclictest0-21swapper/521:40:195
23985994110,31cyclictest0-21swapper/523:30:295
23984994141,0cyclictest0-21swapper/423:05:314
23984994140,0cyclictest0-21swapper/400:37:594
23983994140,0cyclictest32450-21diskmemload22:41:563
23983994140,0cyclictest0-21swapper/322:08:023
23982994141,0cyclictest0-21swapper/222:40:062
23982994141,0cyclictest0-21swapper/200:22:442
23982994140,0cyclictest0-21swapper/223:11:162
23982994140,0cyclictest0-21swapper/223:11:162
2398299410,41cyclictest0-21swapper/222:21:352
23981994141,0cyclictest0-21swapper/123:37:361
23981994141,0cyclictest0-21swapper/123:34:581
23981994140,0cyclictest0-21swapper/122:15:221
23981994140,0cyclictest0-21swapper/100:27:591
23980994110,0cyclictest0-21swapper/022:05:220
2398099410,41cyclictest0-21swapper/000:18:180
23987994040,0cyclictest0-21swapper/723:58:357
23986994040,0cyclictest0-21swapper/623:39:526
23986994040,0cyclictest0-21swapper/622:25:156
23986994010,29cyclictest0-21swapper/622:20:406
23985994040,0cyclictest0-21swapper/523:18:065
23985994040,0cyclictest0-21swapper/523:10:005
23985994040,0cyclictest0-21swapper/523:10:005
2398499400,39cyclictest0-21swapper/423:47:404
23983994039,0cyclictest0-21swapper/300:34:023
23982994040,0cyclictest0-21swapper/200:30:532
23981994040,0cyclictest0-21swapper/123:40:201
23981994040,0cyclictest0-21swapper/122:28:531
2398199400,0cyclictest0-21swapper/122:30:221
23980994038,1cyclictest0-21swapper/023:44:370
23986993939,0cyclictest0-21swapper/623:00:186
23984993939,0cyclictest0-21swapper/423:31:384
23984993910,29cyclictest0-21swapper/422:10:154
23984993910,0cyclictest0-21swapper/423:15:284
23983993939,0cyclictest0-21swapper/323:15:133
23983993939,0cyclictest0-21swapper/322:35:383
23983993938,1cyclictest0-21swapper/322:25:173
23982993938,0cyclictest0-21swapper/223:45:372
23982993938,0cyclictest0-21swapper/221:38:232
23982993910,0cyclictest0-21swapper/223:33:562
23981993938,0cyclictest0-21swapper/123:59:201
23981993910,0cyclictest0-21swapper/123:08:341
23980993939,0cyclictest0-21swapper/023:04:450
23980993939,0cyclictest0-21swapper/022:53:010
23980993938,1cyclictest194-21md0_raid120:40:150
23980993910,28cyclictest0-21swapper/000:36:400
23987993838,0cyclictest0-21swapper/722:56:387
23987993838,0cyclictest0-21swapper/700:37:007
23986993838,0cyclictest0-21swapper/622:45:216
23986993838,0cyclictest0-21swapper/622:33:306
23985993838,0cyclictest0-21swapper/522:22:575
23985993838,0cyclictest0-21swapper/500:09:095
23984993838,0cyclictest0-21swapper/423:11:504
23984993838,0cyclictest0-21swapper/423:11:504
23984993838,0cyclictest0-21swapper/422:42:304
23984993838,0cyclictest0-21swapper/400:01:244
23984993837,1cyclictest0-21swapper/422:46:524
2398499380,38cyclictest0-21swapper/423:04:544
23983993838,0cyclictest0-21swapper/321:53:383
23983993837,1cyclictest0-21swapper/323:55:153
23982993838,0cyclictest0-21swapper/223:20:492
23982993837,0cyclictest0-21swapper/223:56:042
23981993838,0cyclictest0-21swapper/100:36:121
2355323834,2sleep20-21swapper/219:10:002
23987993737,0cyclictest0-21swapper/700:14:137
23987993736,0cyclictest0-21swapper/721:50:167
23986993737,0cyclictest0-21swapper/622:37:136
23986993736,0cyclictest0-21swapper/600:15:176
23985993737,0cyclictest0-21swapper/523:41:205
23985993737,0cyclictest0-21swapper/521:10:005
23984993737,0cyclictest0-21swapper/421:52:224
2398499370,0cyclictest0-21swapper/400:27:524
23983993736,0cyclictest0-21swapper/323:33:133
23983993736,0cyclictest0-21swapper/322:52:353
23982993737,0cyclictest0-21swapper/222:15:172
23982993736,0cyclictest0-21swapper/220:35:132
23982993736,0cyclictest0-21swapper/200:35:222
23982993735,1cyclictest23645-21sendmail22:40:002
23981993737,0cyclictest0-21swapper/122:07:321
23981993737,0cyclictest0-21swapper/121:40:121
23981993737,0cyclictest0-21swapper/100:00:181
23981993736,1cyclictest0-21swapper/122:37:021
23980993737,0cyclictest0-21swapper/023:35:140
23980993737,0cyclictest0-21swapper/023:19:020
23980993737,0cyclictest0-21swapper/022:15:230
23980993737,0cyclictest0-21swapper/000:20:150
23987993636,0cyclictest0-21swapper/722:54:567
23987993636,0cyclictest0-21swapper/700:06:287
23987993635,0cyclictest0-21swapper/721:35:137
23986993636,0cyclictest0-21swapper/600:06:036
23986993635,0cyclictest0-21swapper/623:45:226
23985993636,0cyclictest0-21swapper/520:30:025
23984993636,0cyclictest0-21swapper/400:05:134
23983993636,0cyclictest0-21swapper/323:49:023
2398299360,35cyclictest0-21swapper/223:02:462
23980993636,0cyclictest0-21swapper/021:32:360
2398099360,0cyclictest0-21swapper/022:13:580
23987993535,0cyclictest0-21swapper/723:41:217
23987993534,0cyclictest0-21swapper/723:37:367
23987993534,0cyclictest0-21swapper/720:45:137
23986993535,0cyclictest0-21swapper/623:26:296
23986993535,0cyclictest0-21swapper/621:40:436
23985993535,0cyclictest0-21swapper/522:26:525
23985993535,0cyclictest0-21swapper/521:55:185
23983993535,0cyclictest0-21swapper/319:20:133
23983993534,0cyclictest0-21swapper/323:41:383
23983993534,0cyclictest0-21swapper/322:30:183
23983993534,0cyclictest0-21swapper/319:30:133
23983993533,1cyclictest0-21swapper/300:00:263
23983993510,24cyclictest0-21swapper/300:38:253
23982993510,25cyclictest0-21swapper/223:06:042
2398299350,34cyclictest1413-21/usr/sbin/munin19:25:102
23981993534,0cyclictest0-21swapper/100:18:101
23980993535,0cyclictest0-21swapper/023:45:370
23980993535,0cyclictest0-21swapper/021:10:010
23980993535,0cyclictest0-21swapper/000:25:040
23987993434,0cyclictest0-21swapper/721:25:237
23987993410,0cyclictest0-21swapper/723:48:427
23986993434,0cyclictest0-21swapper/622:10:176
23986993434,0cyclictest0-21swapper/622:05:026
23986993434,0cyclictest0-21swapper/621:55:186
23986993434,0cyclictest0-21swapper/600:39:216
23986993410,0cyclictest0-21swapper/623:40:586
23985993434,0cyclictest0-21swapper/521:00:015
23983993433,0cyclictest0-21swapper/323:22:173
23983993432,1cyclictest0-21swapper/323:50:313
23983993432,1cyclictest0-21swapper/300:05:133
23982993434,0cyclictest0-21swapper/222:11:012
2398199340,34cyclictest0-21swapper/100:24:561
23980993434,0cyclictest0-21swapper/023:32:550
23980993434,0cyclictest0-21swapper/020:10:120
23980993433,1cyclictest0-21swapper/021:48:220
23987993333,0cyclictest0-21swapper/722:10:177
23986993333,0cyclictest0-21swapper/622:59:146
23986993333,0cyclictest0-21swapper/622:00:116
23986993333,0cyclictest0-21swapper/620:20:146
2398599330,1cyclictest0-21swapper/522:15:185
23983993333,0cyclictest0-21swapper/323:29:293
23983993332,1cyclictest0-21swapper/321:55:133
2398199330,1cyclictest0-21swapper/121:55:161
23980993333,0cyclictest0-21swapper/000:08:120
23987993231,0cyclictest0-21swapper/722:40:057
2398799320,32cyclictest0-21swapper/722:05:227
23986993232,0cyclictest0-21swapper/621:36:546
23985993232,0cyclictest0-21swapper/523:00:485
23985993232,0cyclictest0-21swapper/522:37:145
23984993232,0cyclictest0-21swapper/422:01:504
23984993230,1cyclictest0-21swapper/422:25:144
2398499320,0cyclictest0-21swapper/422:39:554
23983993232,0cyclictest0-21swapper/322:10:163
23981993232,0cyclictest0-21swapper/123:00:181
2398199320,31cyclictest0-21swapper/122:15:001
23987993131,0cyclictest0-21swapper/700:24:407
23987993131,0cyclictest0-21swapper/700:18:187
23985993131,0cyclictest32450-21diskmemload21:25:135
23985993131,0cyclictest0-21swapper/522:08:355
23985993131,0cyclictest0-21swapper/521:48:355
23985993130,0cyclictest0-21swapper/519:35:135
23984993131,0cyclictest0-21swapper/423:20:204
23983993131,0cyclictest0-21swapper/322:21:353
23980993131,0cyclictest0-21swapper/022:55:350
23980993131,0cyclictest0-21swapper/021:20:170
23984993030,0cyclictest0-21swapper/420:00:134
23983993029,0cyclictest0-21swapper/321:38:243
23981993030,0cyclictest0-21swapper/119:55:131
23981993029,0cyclictest0-21swapper/119:15:131
23980993030,0cyclictest0-21swapper/021:56:420
23986992929,0cyclictest0-21swapper/622:15:126
23986992929,0cyclictest0-21swapper/621:20:166
23984992929,0cyclictest0-21swapper/423:27:554
23984992929,0cyclictest0-21swapper/421:31:154
23983992926,2cyclictest11495-21/usr/sbin/munin20:35:133
23982992929,0cyclictest0-21swapper/222:00:242
23982992928,0cyclictest0-21swapper/222:45:402
23981992929,0cyclictest0-21swapper/120:45:131
23981992928,0cyclictest0-21swapper/119:25:091
23980992929,0cyclictest0-21swapper/021:52:130
23986992828,0cyclictest0-21swapper/619:35:136
23982992828,0cyclictest0-21swapper/221:20:182
23981992826,1cyclictest0-21swapper/121:20:241
23981992823,1cyclictest121rcu_preempt21:10:221
23980992828,0cyclictest0-21swapper/022:21:410
23980992828,0cyclictest0-21swapper/021:15:150
23980992827,0cyclictest0-21swapper/021:41:020
23987992727,0cyclictest0-21swapper/720:40:147
23984992727,0cyclictest0-21swapper/421:43:464
23984992724,2cyclictest28892-21latency_hist21:40:004
23983992725,1cyclictest0-21swapper/320:55:123
2398199270,26cyclictest0-21swapper/122:24:061
2398199270,26cyclictest0-21swapper/121:15:151
23987992626,0cyclictest0-21swapper/721:32:377
23987992624,1cyclictest0-21swapper/721:10:017
23986992626,0cyclictest32450-21diskmemload21:45:276
23986992626,0cyclictest0-21swapper/621:30:166
23983992626,0cyclictest0-21swapper/323:02:463
23982992626,0cyclictest19790-21ls21:50:162
23981992622,1cyclictest121rcu_preempt21:05:181
23981992618,5cyclictest25-21ksoftirqd/121:31:401
23984992525,0cyclictest0-21swapper/422:09:454
23984992523,1cyclictest0-21swapper/419:20:124
23982992525,0cyclictest0-21swapper/221:25:232
2398199259,6cyclictest121rcu_preempt21:50:001
23981992525,0cyclictest0-21swapper/122:01:511
23987992424,0cyclictest0-21swapper/720:10:127
23987992424,0cyclictest0-21swapper/719:45:367
23986992424,0cyclictest0-21swapper/619:33:256
23983992424,0cyclictest0-21swapper/319:25:113
23983992423,0cyclictest0-21swapper/320:10:123
23981992418,5cyclictest25-21ksoftirqd/121:05:001
23980992424,0cyclictest0-21swapper/023:24:080
23987992322,1cyclictest0-21swapper/721:48:237
23985992310,0cyclictest0-21swapper/523:22:505
23984992323,0cyclictest0-21swapper/421:49:464
23983992321,1cyclictest658-21ptp4l21:20:333
23982992310,13cyclictest0-21swapper/219:38:262
2398199238,13cyclictest0-21swapper/120:40:141
23981992323,0cyclictest0-21swapper/120:25:131
23981992319,3cyclictest20788-21ls21:35:171
23981992319,1cyclictest121rcu_preempt21:51:371
23980992323,0cyclictest0-21swapper/021:05:130
23980992310,3cyclictest0-21swapper/022:03:140
23980992310,13cyclictest0-21swapper/019:30:190
23987992222,0cyclictest0-21swapper/721:16:357
23985992222,0cyclictest0-21swapper/521:40:005
23985992210,4cyclictest0-21swapper/521:05:015
23985992210,3cyclictest0-21swapper/520:35:225
23985992210,3cyclictest0-21swapper/519:15:225
23984992222,0cyclictest0-21swapper/419:45:144
23984992210,4cyclictest0-21swapper/419:35:174
23983992222,0cyclictest0-21swapper/322:00:233
23982992222,0cyclictest0-21swapper/222:07:322
23982992210,4cyclictest0-21swapper/220:20:232
23981992217,4cyclictest121rcu_preempt20:30:131
23981992217,3cyclictest121rcu_preempt20:10:141
23981992217,3cyclictest121rcu_preempt19:35:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional