You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 04:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Nov 28, 2025 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2830622580,13sleep028309-21latency19:04:330
293679911855,12cyclictest9695-21ssh21:18:220
293679911855,12cyclictest9695-21ssh21:18:220
29367991153,76cyclictest10783-21apt-get23:09:590
2920621156,16sleep10-21swapper/119:08:591
29367991053,76cyclictest28406-21apt-get22:44:190
2937999983,73cyclictest28472-21ssh22:44:232
280602989,62sleep228115-21fschecks_time19:04:272
2937999953,68cyclictest7291-21apt-get21:14:162
2937999953,68cyclictest7291-21apt-get21:14:162
29367999561,10cyclictest181rcu_preempt20:59:200
29367999339,29cyclictest29082-21rm21:49:400
29367999266,18cyclictest26132-21kworker/u19:3+events_unbound23:53:590
29379999158,8cyclictest181rcu_preempt21:09:222
2936799912,79cyclictest25418-21ssh22:39:330
29379999055,9cyclictest181rcu_preempt23:14:002
29379999039,15cyclictest181rcu_preempt20:14:242
29379998936,15cyclictest181rcu_preempt23:49:272
29367998945,7cyclictest181rcu_preempt23:02:540
29371998870,12cyclictest292-21dbus-daemon20:24:011
2936799882,35cyclictest29765-21grep19:09:220
29379998632,22cyclictest19289-21ssh23:24:022
29379998632,22cyclictest19289-21ssh23:24:022
29367998668,13cyclictest18604-21latency_hist00:19:020
29367998653,13cyclictest181rcu_preempt20:29:340
2936799864,29cyclictest19685-21cstates23:24:180
29379998554,10cyclictest181rcu_preempt22:26:542
29379998554,10cyclictest181rcu_preempt22:26:542
29379998453,8cyclictest181rcu_preempt19:39:432
29379998452,8cyclictest181rcu_preempt23:38:572
29379998449,8cyclictest181rcu_preempt20:44:022
29379998448,8cyclictest181rcu_preempt23:54:102
2937999844,61cyclictest21980-21apt-get20:24:222
29379998445,11cyclictest29355-21cyclictest19:20:272
29379998445,11cyclictest29355-21cyclictest19:20:272
2936799844,36cyclictest3802-21diskmemload00:09:230
29379998358,16cyclictest10511-21kworker/u17:2-xprtiod00:09:302
29379998350,8cyclictest181rcu_preempt22:19:012
29379998349,9cyclictest181rcu_preempt00:31:042
29367998354,21cyclictest0-21swapper/019:29:350
29367998354,21cyclictest0-21swapper/019:29:350
29367998336,11cyclictest181rcu_preempt20:24:300
29379998255,8cyclictest181rcu_preempt22:04:272
29379998253,8cyclictest181rcu_preempt23:49:002
29379998253,8cyclictest181rcu_preempt19:28:092
29379998253,8cyclictest181rcu_preempt19:28:092
29379998241,9cyclictest181rcu_preempt19:09:122
29371998264,11cyclictest20247-21unixbench_singl22:29:381
29367998249,7cyclictest181rcu_preempt23:40:390
29379998147,8cyclictest181rcu_preempt22:40:092
29379998141,17cyclictest30675-21ssh21:54:012
29367998147,12cyclictest181rcu_preempt21:29:230
29379998054,12cyclictest181rcu_preempt22:34:372
29379998054,12cyclictest181rcu_preempt22:34:372
29379998051,8cyclictest181rcu_preempt23:07:572
29379998051,10cyclictest181rcu_preempt22:59:362
29379998048,9cyclictest181rcu_preempt00:19:112
29379998048,10cyclictest181rcu_preempt00:05:142
29379998045,9cyclictest181rcu_preempt22:29:162
29367998050,11cyclictest181rcu_preempt20:04:170
29367998047,13cyclictest207-21systemd-journal20:04:010
29367998046,13cyclictest181rcu_preempt22:49:160
2936799802,42cyclictest14019-21fschecks_time23:14:220
29379997951,9cyclictest181rcu_preempt23:19:022
29379997950,8cyclictest181rcu_preempt21:56:272
29379997948,8cyclictest181rcu_preempt20:39:022
29379997947,9cyclictest181rcu_preempt21:09:032
29379997947,8cyclictest181rcu_preempt23:59:132
29379997947,8cyclictest181rcu_preempt23:59:132
29379997947,11cyclictest181rcu_preempt21:34:192
29379997943,23cyclictest30273-21cut20:49:232
29379997943,23cyclictest30273-21cut20:49:232
29379997934,10cyclictest181rcu_preempt20:49:022
29379997930,10cyclictest181rcu_preempt19:39:312
29371997958,14cyclictest11187-21threads19:49:371
29367997954,18cyclictest29355-21cyclictest22:34:260
29367997954,18cyclictest29355-21cyclictest22:34:260
29367997950,11cyclictest181rcu_preempt20:49:230
29367997950,11cyclictest181rcu_preempt20:49:230
29367997949,21cyclictest24821-21latency_hist21:44:020
29379997840,13cyclictest181rcu_preempt21:29:262
29379997830,12cyclictest0-21swapper/221:46:522
29371997860,11cyclictest309-21systemd-logind23:01:041
29371997857,14cyclictest26832-21latency_hist00:34:011
29367997852,17cyclictest17645-21threads20:09:350
29367997847,26cyclictest29355-21cyclictest00:04:240
29367997824,9cyclictest181rcu_preempt22:09:030
29379997751,11cyclictest3620-21ssh22:56:502
29379997748,9cyclictest181rcu_preempt20:14:002
29379997745,8cyclictest181rcu_preempt19:14:092
29379997732,13cyclictest1480-21apt-get21:59:102
2937199775,62cyclictest7291-21apt-get21:14:291
2937199775,62cyclictest7291-21apt-get21:14:291
29371997756,13cyclictest21861-21fschecks_count00:24:231
29371997745,25cyclictest31859-21kworker/u18:1+nfsiod23:05:451
29371997733,33cyclictest14056-21grep19:59:241
29371997730,24cyclictest3802-21diskmemload23:10:011
29367997757,14cyclictest17630-21kworker/u18:3+events_unbound19:38:590
29367997743,19cyclictest4004-21ssh22:57:570
29379997641,10cyclictest181rcu_preempt19:33:592
29379997641,10cyclictest181rcu_preempt19:33:592
29379997634,18cyclictest181rcu_preempt00:17:232
29367997654,12cyclictest0-21swapper/023:59:030
29367997649,9cyclictest181rcu_preempt19:54:020
29367997648,13cyclictest31321-21ssh21:54:240
29367997630,17cyclictest181rcu_preempt23:48:020
2936799762,39cyclictest24166-21latency_hist00:29:020
29367997621,15cyclictest0-21swapper/021:44:470
29379997543,10cyclictest181rcu_preempt20:54:172
29379997538,9cyclictest181rcu_preempt19:49:272
29379997528,13cyclictest181rcu_preempt23:40:262
2937199753,35cyclictest13579-21cut22:19:211
29367997546,9cyclictest181rcu_preempt22:14:140
29367997522,9cyclictest181rcu_preempt20:34:350
29379997452,8cyclictest181rcu_preempt22:53:452
29379997451,7cyclictest181rcu_preempt20:04:012
29379997443,15cyclictest181rcu_preempt22:20:042
29379997441,16cyclictest181rcu_preempt00:26:032
29379997437,9cyclictest181rcu_preempt20:29:122
29371997458,9cyclictest0-21swapper/121:34:241
29371997447,21cyclictest0-21swapper/122:40:141
29379997352,7cyclictest181rcu_preempt21:24:032
29379997348,8cyclictest181rcu_preempt22:12:462
29379997346,9cyclictest181rcu_preempt19:54:262
29379997346,9cyclictest181rcu_preempt19:54:262
29379997335,10cyclictest181rcu_preempt00:39:032
29379997335,10cyclictest181rcu_preempt00:39:032
29371997356,12cyclictest29355-21cyclictest21:28:041
29371997355,12cyclictest29892-21kernelversion19:09:251
29371997355,11cyclictest3798-21latency_hist21:09:031
29371997349,16cyclictest29355-21cyclictest19:44:331
29371997345,20cyclictest355-21runrttasks19:29:321
29371997345,20cyclictest355-21runrttasks19:29:321
29371997324,11cyclictest301ktimers/123:49:021
29371997323,11cyclictest301ktimers/120:34:291
29367997354,7cyclictest181rcu_preempt23:04:240
29367997347,8cyclictest181rcu_preempt19:14:330
29379997255,10cyclictest0-21swapper/220:04:372
29379997237,11cyclictest0-21swapper/219:44:332
29371997255,11cyclictest17115-21ssh22:24:361
29371997255,11cyclictest17115-21ssh22:24:361
29371997251,16cyclictest22181-21kworker/u17:4+events_unbound23:34:461
2937199724,45cyclictest15846-21latency_hist00:14:021
29371997238,26cyclictest7853-21processes19:39:541
2937199723,48cyclictest355-21runrttasks22:03:391
29371997226,9cyclictest301ktimers/121:31:561
29371997224,13cyclictest181rcu_preempt21:44:221
29367997256,10cyclictest24134-21ssh23:31:180
29367997244,9cyclictest181rcu_preempt21:24:140
29367997244,21cyclictest29355-21cyclictest20:14:160
29367997234,10cyclictest848-21cpuspeed19:19:200
29367997234,10cyclictest848-21cpuspeed19:19:200
29367997224,12cyclictest181rcu_preempt21:59:210
2936799722,32cyclictest7807-21fschecks_time23:59:240
2936799722,32cyclictest7807-21fschecks_time23:59:240
286652720,8sleep10-21swapper/123:39:181
29379997129,17cyclictest181rcu_preempt21:44:032
29371997149,13cyclictest0-21swapper/100:14:341
29371997148,15cyclictest0-21swapper/123:49:401
29371997147,16cyclictest0-21swapper/120:09:021
29371997144,18cyclictest0-21swapper/122:17:491
29371997144,17cyclictest0-21swapper/121:24:021
2936799718,53cyclictest25804-21ssh23:34:420
29367997144,12cyclictest181rcu_preempt22:29:110
29367997136,28cyclictest0-21swapper/000:19:360
29371997051,12cyclictest4576-21ssh22:59:021
29371997051,12cyclictest1-21systemd20:19:001
29371997050,14cyclictest1-21systemd23:34:011
29371997049,16cyclictest29355-21cyclictest22:07:011
29371997048,15cyclictest28979-21sh21:49:391
29371997027,8cyclictest301ktimers/120:44:191
29371997022,9cyclictest301ktimers/122:44:201
29367997040,11cyclictest181rcu_preempt20:24:030
29367997035,21cyclictest181rcu_preempt00:29:290
2936799703,32cyclictest21912-21ssh21:39:020
29367997032,12cyclictest181rcu_preempt19:24:250
29367997032,12cyclictest181rcu_preempt19:24:250
29367997027,12cyclictest181rcu_preempt22:09:330
29379996948,8cyclictest181rcu_preempt21:20:512
29379996945,8cyclictest181rcu_preempt23:34:002
29379996942,18cyclictest181rcu_preempt23:24:392
29379996928,11cyclictest181rcu_preempt20:59:222
29371996945,16cyclictest7234-40mandb23:59:001
2937199694,49cyclictest10642-21fschecks_time00:04:211
29371996926,8cyclictest301ktimers/120:24:361
29371996922,7cyclictest301ktimers/119:29:021
29371996922,7cyclictest301ktimers/119:29:021
29371996920,11cyclictest301ktimers/121:39:181
29371996915,26cyclictest2236-21latency_hist19:24:031
29371996915,26cyclictest2236-21latency_hist19:24:031
2936799698,23cyclictest15-21ksoftirqd/020:39:360
2936799693,34cyclictest3402-21grep21:04:380
29367996931,10cyclictest0-21swapper/022:28:330
29367996931,10cyclictest0-21swapper/022:28:330
29367996929,11cyclictest181rcu_preempt23:22:080
29367996929,11cyclictest181rcu_preempt23:22:080
29367996928,16cyclictest181rcu_preempt20:49:030
2937199684,40cyclictest23766-21fschecks_count20:29:221
29371996843,17cyclictest5131-21unixbench_multi21:09:381
29371996838,9cyclictest181rcu_preempt22:09:341
29371996836,21cyclictest19363-21nfs4_client00:19:281
2937199683,42cyclictest727-21latency_hist19:19:041
2937199682,44cyclictest10022-21idleruntime-cro00:03:591
2937199682,44cyclictest10022-21idleruntime-cro00:03:591
29367996850,9cyclictest11276-21rm21:19:360
29367996840,10cyclictest181rcu_preempt19:39:470
29367996839,11cyclictest181rcu_preempt19:49:020
29367996838,23cyclictest0-21swapper/021:09:360
2936799682,34cyclictest355-21runrttasks22:20:040
29367996821,13cyclictest181rcu_preempt20:54:210
29367996812,20cyclictest15-21ksoftirqd/019:54:210
29367996812,20cyclictest15-21ksoftirqd/019:54:210
29379996747,14cyclictest314-21in:imuxsock20:23:332
29371996745,15cyclictest27619-21timerandwakeup20:39:351
29371996724,7cyclictest301ktimers/122:34:221
29371996724,7cyclictest301ktimers/122:34:221
29371996721,34cyclictest30757-21switchtime20:49:361
29371996721,34cyclictest30757-21switchtime20:49:361
29367996722,25cyclictest27321-21fschecks_time00:34:210
29367996722,25cyclictest27321-21fschecks_time00:34:210
29371996634,12cyclictest181rcu_preempt23:19:221
29371996634,12cyclictest181rcu_preempt23:19:221
29371996622,7cyclictest301ktimers/122:51:591
29371996622,7cyclictest301ktimers/119:54:231
29371996622,7cyclictest301ktimers/119:54:231
2937199653,44cyclictest5772-21fschecks_count19:34:221
2937199653,42cyclictest32055-21latency20:54:251
2937199653,35cyclictest13685-21/usr/sbin/munin23:14:191
29386996410,38cyclictest10863-21chrt23:09:253
2858326415,32sleep30-21swapper/319:04:453
29371996347,10cyclictest17273-21latency20:09:261
2937199634,31cyclictest355-21runrttasks21:54:421
2937199633,30cyclictest826-21/usr/sbin/munin20:59:211
29371996325,10cyclictest301ktimers/100:34:261
29371996325,10cyclictest301ktimers/100:34:261
29371996214,12cyclictest20139-21sh23:24:291
29386996014,28cyclictest0-21swapper/322:44:233
29386995920,23cyclictest0-21swapper/321:14:303
29386995920,23cyclictest0-21swapper/321:14:303
2938699572,43cyclictest20383-21taskset21:35:063
2938699553,38cyclictest22005-21chrt20:24:253
2938699543,39cyclictest29074-21chrt00:37:543
2938699543,39cyclictest29074-21chrt00:37:543
2938699543,36cyclictest31520-21chrt20:54:023
2938699543,36cyclictest31520-21chrt20:54:023
2938699542,41cyclictest927-21taskset23:47:143
2938699533,37cyclictest4577-21chrt22:04:203
2938699532,39cyclictest5484-21taskset21:10:223
2938699532,37cyclictest23343-21chrt00:26:483
2938699522,38cyclictest27755-21chrt23:37:383
2938699522,37cyclictest21679-21chrt23:27:403
2938699522,37cyclictest11319-21chrt19:49:383
2938699513,38cyclictest14694-21taskset19:59:593
2938699513,37cyclictest31819-21taskset22:49:263
2938699513,34cyclictest16606-21taskset23:19:053
2938699513,34cyclictest16606-21taskset23:19:053
2938699512,36cyclictest14820-21taskset21:26:033
2938699503,37cyclictest26276-21taskset00:32:323
2938699503,35cyclictest21513-21taskset20:24:003
2938699503,33cyclictest3831-21chrt21:09:003
2938699502,36cyclictest4104-21chrt19:29:173
2938699502,36cyclictest4104-21chrt19:29:173
2938699502,35cyclictest16241-21chrt23:18:493
2938699502,35cyclictest15156-21chrt22:21:463
2938699502,34cyclictest23939-21chrt22:36:413
2938699502,34cyclictest23939-21chrt22:36:413
2938699493,33cyclictest3339-21chrt23:50:593
2938699493,32cyclictest24620-21chrt23:32:433
2938699493,31cyclictest995-21chrt19:19:253
2938699493,31cyclictest995-21chrt19:19:253
2938699482,33cyclictest29067-21chrt23:39:283
2938699482,33cyclictest27746-21chrt20:39:363
2938699482,32cyclictest9781-21chrt23:07:423
2938699482,31cyclictest6751-21chrt23:02:233
2938699473,32cyclictest4372-21chrt22:58:593
2938699473,31cyclictest460-21chrt21:57:063
2938699472,33cyclictest12856-21taskset22:18:423
2938699463,32cyclictest15308-21chrt00:12:553
2938699462,34cyclictest25386-21taskset20:34:213
2938699462,34cyclictest23082-21chrt21:39:363
2938699462,32cyclictest31356-21taskset19:14:103
2938699462,32cyclictest21005-21chrt22:31:433
2938699462,32cyclictest11507-21chrt21:19:593
2938699454,31cyclictest7256-21taskset23:59:013
2938699453,31cyclictest20887-21chrt00:22:393
2938699452,30cyclictest26208-21taskset22:40:303
2938699443,31cyclictest27699-21chrt21:49:003
2938699443,31cyclictest17429-21taskset00:15:463
2938699443,28cyclictest11766-21chrt00:05:413
2938699442,30cyclictest16430-21taskset21:29:233
2938699432,30cyclictest2668-21taskset19:24:233
2938699432,30cyclictest2668-21taskset19:24:233
2938699426,25cyclictest16498-21chrt20:07:323
29386994226,10cyclictest0-21swapper/319:39:473
2938699422,30cyclictest8909-21chrt00:00:313
2938699422,30cyclictest8909-21chrt00:00:313
2938699422,30cyclictest16897-21taskset20:09:143
2938699422,28cyclictest5558-21taskset19:34:033
2938699422,28cyclictest28011-21chrt21:49:163
2938699422,28cyclictest16192-21chrt22:24:053
2938699422,28cyclictest16192-21chrt22:24:053
2938699413,29cyclictest30593-21taskset19:10:263
2938699413,27cyclictest31662-21taskset20:54:143
2938699413,27cyclictest18502-21taskset20:14:123
2938699413,27cyclictest1436-21taskset21:59:053
2938699412,28cyclictest812-21chrt20:59:043
2938699412,27cyclictest23565-21taskset20:29:073
2938699412,26cyclictest7336-21taskset22:09:063
2938699404,27cyclictest9353-21taskset19:44:283
2938699402,28cyclictest12041-21taskset19:54:123
2938699402,28cyclictest12041-21taskset19:54:123
2938699402,27cyclictest28411-21taskset20:44:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional