You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 07:01
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sat May 02, 2026 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
253922310,21sleep20-21swapper/218:57:462
1803213811,83sleep11820-21cstates18:57:211
3042211513,71sleep03045-21cron19:02:010
3202991133,79cyclictest11507-21apt-get21:57:190
3209991062,27cyclictest20158-21taskset22:11:562
32099910536,38cyclictest352-21runrttasks22:21:092
32099910536,38cyclictest352-21runrttasks22:21:092
3209991012,27cyclictest32540-21sh00:18:552
3209991012,27cyclictest24581-21sh21:24:522
32099910079,15cyclictest15708-21systemd-run23:52:022
3209991002,33cyclictest11642-21chrt19:24:372
106202990,6sleep10-21swapper/121:02:201
106202990,6sleep10-21swapper/121:02:201
320999973,30cyclictest22226-21taskset21:21:402
3209999531,37cyclictest0-21swapper/223:16:092
320999952,30cyclictest7102-21sh21:49:452
320999942,55cyclictest21969-21apt-get00:02:122
320999942,55cyclictest21969-21apt-get00:02:122
3202999367,18cyclictest565-21latency_hist20:32:040
3209999231,40cyclictest0-21swapper/219:51:362
3204999229,39cyclictest31-21ksoftirqd/122:35:051
3204999229,39cyclictest31-21ksoftirqd/122:35:051
3209999049,34cyclictest2990-21kworker/u17:1-xprtiod22:47:402
3204998963,19cyclictest7065-21latency_hist19:12:031
321699887,60cyclictest0-21swapper/322:27:243
320999882,31cyclictest352-21runrttasks19:02:292
3202998858,8cyclictest283742modprobe00:12:400
320999872,74cyclictest0-21swapper/220:22:122
320299872,70cyclictest6008-21cron22:42:000
320999863,6cyclictest0-21swapper/220:17:142
320999863,6cyclictest0-21swapper/220:17:142
320999862,36cyclictest5825-21if_eth020:47:252
320499863,59cyclictest30225-21diskstats22:28:061
320999852,35cyclictest12451-21/usr/sbin/munin21:58:462
320999852,26cyclictest31009-21runrttasks21:36:022
3204998567,12cyclictest8477-21ssh21:52:221
320299857,62cyclictest29733-21/usr/sbin/munin22:28:020
3202998549,15cyclictest181rcu_preempt19:02:250
3202998541,13cyclictest181rcu_preempt19:42:330
3202998541,13cyclictest181rcu_preempt19:42:330
321699842,34cyclictest8792-21chrt19:17:043
320999844,69cyclictest0-21swapper/220:57:162
320999843,70cyclictest0-21swapper/220:37:132
320999843,6cyclictest0-21swapper/200:27:122
320999843,69cyclictest0-21swapper/219:17:122
320999843,57cyclictest0-21swapper/219:32:172
320999842,38cyclictest15527-21rm22:57:182
320999842,26cyclictest352-21runrttasks20:56:152
3204998445,12cyclictest181rcu_preempt23:30:341
3204998442,8cyclictest181rcu_preempt21:09:301
3204998435,12cyclictest181rcu_preempt00:28:271
320999836,71cyclictest0-21swapper/223:17:162
320999834,69cyclictest0-21swapper/220:07:162
3209998340,38cyclictest0-21swapper/222:12:382
320999833,74cyclictest0-21swapper/222:52:182
320999833,6cyclictest0-21swapper/220:42:122
3209998336,40cyclictest0-21swapper/222:02:412
3209998335,40cyclictest0-21swapper/222:32:352
3209998335,40cyclictest0-21swapper/222:32:352
320999833,35cyclictest5891-21fschecks_time19:07:432
320999833,32cyclictest32241-21/usr/sbin/munin21:37:282
320999832,36cyclictest18688-21awk23:57:012
3209998312,61cyclictest29756-21apt-get22:27:242
3204998338,8cyclictest181rcu_preempt19:27:021
3202998359,18cyclictest943-21cpu23:27:160
3202998349,12cyclictest181rcu_preempt22:24:010
3202998349,12cyclictest181rcu_preempt22:24:010
320299833,54cyclictest20246-21modprobe22:12:020
3209998263,13cyclictest23439-21ssh23:10:232
320999825,71cyclictest0-21swapper/221:27:152
3209998254,20cyclictest28417-21sh22:24:512
3209998254,20cyclictest28417-21sh22:24:512
320999823,6cyclictest0-21swapper/223:27:162
3209998235,39cyclictest0-21swapper/221:08:232
320999823,37cyclictest21074-21ssh23:07:022
320999823,32cyclictest32141-21/usr/sbin/munin20:27:332
320999822,7cyclictest0-21swapper/220:02:122
320999822,35cyclictest7320-21cpu19:12:152
320999822,34cyclictest20208-21runrttasks19:52:172
320999822,34cyclictest10532-21/usr/sbin/munin23:42:282
3204998252,10cyclictest181rcu_preempt22:59:071
3204998249,9cyclictest181rcu_preempt21:34:381
3204998243,7cyclictest181rcu_preempt21:39:261
3202998263,13cyclictest24848-21kworker/u17:0+rpciod00:22:010
3202998257,18cyclictest16817-21latency_hist19:42:050
3202998256,11cyclictest181rcu_preempt20:37:110
3202998253,10cyclictest181rcu_preempt20:02:130
3202998240,26cyclictest17764-21ssh23:54:310
3202998232,13cyclictest181rcu_preempt00:27:390
3209998163,12cyclictest6041-21systemctl22:42:012
320999814,72cyclictest0-21swapper/221:12:112
320999813,73cyclictest0-21swapper/223:57:192
3209998135,39cyclictest0-21swapper/221:02:262
3209998135,39cyclictest0-21swapper/221:02:262
320999813,38cyclictest3975-21ssh23:32:162
320999813,34cyclictest16032-21/usr/sbin/munin19:37:342
320999813,32cyclictest26856-21grep20:12:222
320999812,35cyclictest7148-21seq23:37:212
320999812,35cyclictest7148-21seq23:37:212
320999812,32cyclictest12951-21sed19:27:372
3204998110,47cyclictest31-21ksoftirqd/122:02:261
3202998160,15cyclictest7980-21timerwakeupswit20:52:370
3202998152,9cyclictest181rcu_preempt22:59:190
3202998143,28cyclictest0-21swapper/023:12:190
3202998130,11cyclictest181rcu_preempt21:07:060
3202998130,11cyclictest181rcu_preempt21:07:060
320999803,64cyclictest0-21swapper/220:32:172
320999803,33cyclictest25903-21cat00:07:382
320999803,33cyclictest25903-21cat00:07:382
320999803,32cyclictest17306-21/usr/sbin/munin19:42:262
320999803,32cyclictest17306-21/usr/sbin/munin19:42:262
320999802,29cyclictest8340-21grep21:52:172
3204998058,8cyclictest181rcu_preempt23:02:201
3204998049,9cyclictest181rcu_preempt23:48:121
3204998045,13cyclictest181rcu_preempt22:15:551
3202998063,11cyclictest13547-21ssh21:07:060
3202998047,12cyclictest181rcu_preempt19:47:140
3202998032,11cyclictest181rcu_preempt23:40:010
3202998032,11cyclictest181rcu_preempt23:40:010
3202998029,16cyclictest181rcu_preempt22:06:260
321699793,52cyclictest0-21swapper/321:07:193
321699793,29cyclictest10276-21taskset23:42:223
321699792,38cyclictest24531-21taskset00:06:423
321699792,38cyclictest24531-21taskset00:06:423
3209997937,34cyclictest0-21swapper/200:12:562
320999792,35cyclictest23103-21taskset20:01:342
3204997938,13cyclictest181rcu_preempt21:13:491
3202997957,14cyclictest19676-21snmpd19:29:000
3202997955,11cyclictest181rcu_preempt20:07:370
3202997951,11cyclictest181rcu_preempt19:55:570
3202997950,9cyclictest181rcu_preempt19:57:170
3202997943,12cyclictest181rcu_preempt23:05:430
3202997939,12cyclictest181rcu_preempt23:49:320
321699783,43cyclictest0-21swapper/319:07:253
320999782,37cyclictest7235-21sed22:42:352
3204997834,13cyclictest181rcu_preempt23:13:521
3202997858,14cyclictest26273-21latency_hist22:22:030
3202997858,14cyclictest26273-21latency_hist22:22:030
3202997833,12cyclictest181rcu_preempt19:07:110
3209997738,34cyclictest31035-21kworker/u17:2+xprtiod21:42:092
320999773,27cyclictest1837-21fschecks_time00:22:232
320999772,34cyclictest30614-21/usr/sbin/munin23:22:222
320499775,34cyclictest31-21ksoftirqd/122:08:291
3204997752,7cyclictest181rcu_preempt00:03:041
3204997752,7cyclictest181rcu_preempt00:03:041
3204997729,36cyclictest25789-21sort00:07:371
3204997729,36cyclictest25789-21sort00:07:371
3202997740,9cyclictest181rcu_preempt21:25:490
321699763,31cyclictest4635-21taskset00:27:163
321699763,30cyclictest25684-21chrt23:13:133
3204997639,15cyclictest181rcu_preempt19:27:301
3204997637,13cyclictest181rcu_preempt21:24:011
3204997634,12cyclictest181rcu_preempt23:25:121
3202997659,11cyclictest10764-21fschecks_time19:22:230
3202997654,9cyclictest181rcu_preempt23:57:120
3202997650,18cyclictest11237-21apt-key21:57:050
321699753,33cyclictest12023-21chrt19:27:053
321699753,29cyclictest6126-21taskset23:35:373
3204997551,9cyclictest181rcu_preempt00:22:381
3204997535,10cyclictest181rcu_preempt23:37:181
3204997535,10cyclictest181rcu_preempt23:37:181
3202997557,12cyclictest211rcuc/021:27:140
3202997544,9cyclictest181rcu_preempt21:14:540
3202997542,13cyclictest181rcu_preempt19:17:360
3202997530,7cyclictest181rcu_preempt23:17:200
3202997527,12cyclictest181rcu_preempt21:17:290
3204997455,13cyclictest5864-21kernelversion20:47:261
3204997453,8cyclictest181rcu_preempt20:03:161
3204997443,8cyclictest181rcu_preempt22:37:261
3204997442,7cyclictest181rcu_preempt21:47:501
3204997436,10cyclictest181rcu_preempt21:30:501
3202997460,10cyclictest3186-21cyclictest20:20:200
3202997460,10cyclictest3186-21cyclictest20:20:200
3202997449,13cyclictest0-21swapper/023:32:400
3202997436,8cyclictest181rcu_preempt21:02:030
3202997426,8cyclictest181rcu_preempt00:02:260
3202997426,8cyclictest181rcu_preempt00:02:260
3202997422,11cyclictest181rcu_preempt22:44:420
321699733,32cyclictest9419-21chrt22:47:193
321699733,30cyclictest17804-21chrt23:54:383
3204997351,8cyclictest181rcu_preempt20:52:391
3204997349,16cyclictest9088-21latency_hist22:47:031
3204997345,19cyclictest0-21swapper/122:22:221
3204997345,19cyclictest0-21swapper/122:22:221
3204997341,11cyclictest181rcu_preempt23:20:571
3204997337,10cyclictest181rcu_preempt21:17:231
3204997336,16cyclictest181rcu_preempt19:57:021
320499732,35cyclictest31-21ksoftirqd/120:07:371
3202997351,9cyclictest181rcu_preempt21:42:320
3202997331,12cyclictest181rcu_preempt23:22:380
3204997252,7cyclictest181rcu_preempt23:47:011
3204997242,7cyclictest181rcu_preempt00:15:241
3204997237,8cyclictest181rcu_preempt23:57:041
3204997235,20cyclictest19423-21ssh23:57:311
3204997234,19cyclictest181rcu_preempt19:37:161
3204997231,16cyclictest181rcu_preempt22:50:361
3202997238,21cyclictest181rcu_preempt22:51:170
3202997232,12cyclictest181rcu_preempt21:47:290
3204997121,25cyclictest31-21ksoftirqd/122:18:311
3204997121,25cyclictest31-21ksoftirqd/122:18:311
3202997145,20cyclictest21502-21timerwakeupswit22:12:370
3202997136,9cyclictest181rcu_preempt00:22:170
3202997135,16cyclictest181rcu_preempt20:12:140
3202997130,27cyclictest181rcu_preempt00:10:120
3202997130,27cyclictest181rcu_preempt00:10:120
3204997035,10cyclictest181rcu_preempt19:57:161
3204997033,15cyclictest181rcu_preempt23:08:281
3204997031,8cyclictest181rcu_preempt00:19:331
3204997030,31cyclictest8618-21modprobe19:17:011
3202997040,9cyclictest181rcu_preempt22:52:310
3202997040,10cyclictest181rcu_preempt20:24:570
3202997039,11cyclictest181rcu_preempt20:42:110
320299703,24cyclictest10806-21find23:42:350
3202997030,8cyclictest181rcu_preempt23:07:230
3202997027,8cyclictest181rcu_preempt22:32:470
3202997027,8cyclictest181rcu_preempt22:32:470
3202997024,12cyclictest181rcu_preempt21:40:010
320299702,33cyclictest29363-21ps21:32:330
3204996946,8cyclictest181rcu_preempt21:57:291
3204996930,12cyclictest181rcu_preempt20:31:291
3202996944,10cyclictest181rcu_preempt19:32:170
3202996930,8cyclictest2073-21latency_hist20:37:030
320499687,45cyclictest2481-21cut21:42:221
320499686,23cyclictest31-21ksoftirqd/123:32:301
320499685,31cyclictest301ktimers/119:02:171
3204996852,10cyclictest3186-21cyclictest22:56:171
3204996844,10cyclictest181rcu_preempt20:32:171
3204996844,10cyclictest181rcu_preempt19:47:221
3204996830,12cyclictest181rcu_preempt19:19:421
3204996738,8cyclictest181rcu_preempt19:46:161
3204996738,8cyclictest181rcu_preempt19:46:161
3204996733,13cyclictest181rcu_preempt20:57:441
3202996747,12cyclictest0-21swapper/019:12:220
321699667,28cyclictest0-21swapper/321:58:413
3204996640,9cyclictest181rcu_preempt20:37:381
3204996637,14cyclictest181rcu_preempt20:17:311
3204996637,14cyclictest181rcu_preempt20:17:311
320499663,40cyclictest31-21ksoftirqd/120:16:021
3204996619,19cyclictest31-21ksoftirqd/119:36:171
3216996415,38cyclictest6006-21chrt22:42:003
3204996437,9cyclictest181rcu_preempt20:42:341
320299643,45cyclictest6370-21timerwakeupswit20:47:370
321699633,43cyclictest6849-21taskset20:52:013
321699633,42cyclictest21028-21taskset23:07:003
320499639,20cyclictest31-21ksoftirqd/120:22:371
321699617,41cyclictest16468-21taskset22:57:413
321699583,40cyclictest1522-21taskset21:40:573
321699573,38cyclictest23031-21taskset21:22:253
321699564,40cyclictest29600-21taskset22:27:053
321699564,40cyclictest29600-21taskset22:27:053
321699562,38cyclictest25568-21taskset00:07:313
321699562,38cyclictest25568-21taskset00:07:313
321699553,37cyclictest2555-21chrt23:29:093
321699553,37cyclictest1009-21chrt00:21:293
321699553,35cyclictest18703-21chrt21:15:113
321699553,32cyclictest917-21kworker/3:1+events19:57:353
321699552,41cyclictest15711-21chrt23:52:033
321699543,35cyclictest9638-21chrt21:53:223
321699533,39cyclictest2905-21taskset00:23:233
321699533,37cyclictest28511-21taskset20:17:233
321699533,37cyclictest28511-21taskset20:17:233
321699533,35cyclictest7471-21chrt23:37:283
321699533,35cyclictest7471-21chrt23:37:283
321699533,34cyclictest31558-21taskset21:37:043
321699533,34cyclictest27603-21chrt20:12:373
321699532,36cyclictest4150-21chrt20:42:213
321699532,36cyclictest25576-21chrt22:20:163
321699532,36cyclictest25576-21chrt22:20:163
321699532,36cyclictest20792-21taskset23:59:363
321699532,35cyclictest13917-21taskset22:53:563
321699526,34cyclictest3946-21chrt21:44:183
321699523,37cyclictest27449-21chrt23:17:183
321699522,39cyclictest22937-21taskset23:08:593
321699522,34cyclictest5261-21chrt21:47:143
321699522,29cyclictest917-21kworker/3:1+events19:02:123
3216995125,14cyclictest239942chrt20:02:323
321699512,34cyclictest23369-21chrt22:17:043
321699512,33cyclictest22576-21chrt21:22:043
321699512,32cyclictest7474-21taskset22:42:453
321699512,31cyclictest965-21taskset22:32:323
321699512,31cyclictest965-21taskset22:32:323
321699503,35cyclictest31027-21chrt23:22:333
321699502,35cyclictest17498-21taskset22:07:113
321699502,34cyclictest27668-21taskset21:30:073
321699493,32cyclictest12235-21taskset21:04:373
321699493,32cyclictest12235-21taskset21:04:373
321699492,33cyclictest13033-21chrt19:27:383
321699484,34cyclictest15943-21chrt22:03:353
321699483,33cyclictest624-21chrt20:32:053
321699483,33cyclictest20937-21taskset19:52:363
321699483,30cyclictest9611-21chrt20:57:363
321699482,35cyclictest2415-21taskset20:37:193
321699474,31cyclictest25256-21chrt20:07:223
321699474,29cyclictest29263-21chrt00:13:063
321699473,32cyclictest20037-21chrt19:52:043
321699453,31cyclictest15513-21chrt19:37:213
26182459,21sleep30-21swapper/318:57:483
321699442,31cyclictest15278-21taskset19:37:053
321699433,28cyclictest7299-21taskset20:52:223
321699422,27cyclictest9862-21chrt19:17:373
321699412,30cyclictest29920-21taskset20:22:173
321699403,25cyclictest17217-21chrt19:42:243
321699403,25cyclictest17217-21chrt19:42:243
321699393,26cyclictest1017-21taskset20:32:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional