You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 06:50
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Mar 14, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21218212510,84sleep221308-21unixbench_multi19:01:032
21833991233,27cyclictest21840-21sh23:33:480
21833991213,45cyclictest1552-21apt-get19:40:280
218339911933,46cyclictest0-21swapper/000:26:500
218339911515,23cyclictest6164-21sh22:14:120
21833991133,26cyclictest21320-21apt-get21:45:410
218339910924,55cyclictest0-21swapper/023:29:550
21833991087,76cyclictest16314-21apt-get20:25:460
542521070,7sleep20-21swapper/222:12:092
218339910542,56cyclictest4955-21ssh00:00:170
21833991044,31cyclictest8494-21fschecks_count00:05:380
21833991043,32cyclictest30199-21taskset22:53:380
21833991043,32cyclictest30199-21taskset22:53:380
21833991043,23cyclictest31354-21seq23:50:140
218339910430,53cyclictest1202-21ssh22:59:500
218339910415,21cyclictest5652-21sh21:19:560
218339910415,21cyclictest5652-21sh21:19:560
21833991032,24cyclictest17407-21apt-key00:20:210
218339910240,55cyclictest0-21swapper/021:05:390
218339910240,55cyclictest0-21swapper/021:05:390
21833991023,26cyclictest16856-21mailstats22:30:510
21833991023,17cyclictest23517-21apt-get00:30:270
21833991013,46cyclictest336-21ssh21:10:390
21833991003,21cyclictest32381-21/usr/sbin/munin19:35:560
21833991003,20cyclictest10150-21fschecks_count22:20:390
2183399995,14cyclictest31702sleep019:45:340
2183399994,35cyclictest10027-21/usr/sbin/munin20:05:430
2183399994,29cyclictest25799-21rm22:45:500
2183399992,36cyclictest352-21runrttasks23:02:290
2183399987,12cyclictest0-21swapper/023:05:480
2183399982,31cyclictest352-21runrttasks23:15:540
2092329819,43sleep00-21swapper/019:00:560
2183399974,36cyclictest18325-21sort21:40:390
2183399974,36cyclictest18325-21sort21:40:390
2183399973,33cyclictest25382-21grep21:50:530
2183399973,20cyclictest352-21runrttasks20:40:290
2183399973,20cyclictest352-21runrttasks20:40:290
2183399965,28cyclictest24305-21apt-get19:11:140
2183399963,37cyclictest12301-21sshd21:30:380
2183399963,27cyclictest1-21systemd23:38:430
2183399963,20cyclictest15955-21ssh22:30:200
2183399962,33cyclictest20212-21latency_hist00:25:200
2183399962,25cyclictest10368-21sendmail23:15:170
21833999617,7cyclictest0-21swapper/023:22:480
21833999617,7cyclictest0-21swapper/023:22:480
21838999558,10cyclictest181rcu_preempt20:25:371
21833999573,15cyclictest23376-21idleruntime-cro19:10:180
2183399956,10cyclictest0-21swapper/019:35:200
2183399954,33cyclictest27854-21/usr/sbin/munin21:00:350
2183399954,28cyclictest6865-21mailstats21:20:510
2183399953,37cyclictest9079-21/usr/sbin/munin21:25:340
2183399953,32cyclictest19544-21/usr/sbin/munin20:35:390
2183399953,32cyclictest19544-21/usr/sbin/munin20:35:390
2183399953,28cyclictest352-21runrttasks23:55:010
2183399953,20cyclictest18177-21cut20:30:380
2183399953,20cyclictest18177-21cut20:30:380
21833999472,15cyclictest30661-21latency_hist19:30:210
2183399945,28cyclictest22497-21idleruntime22:40:430
2183399944,29cyclictest27372-21latency_hist19:20:200
2183399943,32cyclictest27538-21/usr/sbin/munin19:20:460
2183399943,31cyclictest14454-21/usr/sbin/munin20:20:490
2183399943,30cyclictest352-21runrttasks20:10:520
2183399943,29cyclictest6218-21latency_hist19:55:190
2183399943,29cyclictest6218-21latency_hist19:55:190
2183399943,27cyclictest11521-21fschecks_count00:10:360
2183399943,27cyclictest11521-21fschecks_count00:10:360
2183399943,25cyclictest27653-21uniq21:55:390
2183399942,31cyclictest13055-21tune2fs20:15:370
2183399942,28cyclictest8996-21ssh22:18:400
2183399935,31cyclictest15528-21kernelversion21:35:420
2183399934,33cyclictest352-21runrttasks22:01:400
2183399934,32cyclictest26243-21/usr/sbin/munin20:55:420
2183399934,32cyclictest26243-21/usr/sbin/munin20:55:420
2183399934,26cyclictest26039-21latency_hist20:55:190
21833999336,50cyclictest5256-21/usr/sbin/munin00:00:360
2183399933,34cyclictest23291-21ntpq20:45:450
2183399933,32cyclictest26578-21timerandwakeup23:40:520
2183399933,28cyclictest912-21ssh22:05:310
21838999271,15cyclictest14521-21unixbench_singl22:26:501
21838999262,22cyclictest352-21runrttasks21:37:511
2183399923,28cyclictest7860-21latency_hist20:00:200
2183399913,31cyclictest8902-21grep20:00:530
2183399913,25cyclictest20095-21cat22:35:530
21838999011,70cyclictest19676-21snmpd00:20:591
21843998740,18cyclictest181rcu_preempt22:30:402
21843998658,20cyclictest2561-21threads23:00:532
21843998640,12cyclictest181rcu_preempt23:55:522
2183899863,59cyclictest21358-21taskset21:45:331
21843998567,12cyclictest297-21systemd-logind19:10:212
21843998458,18cyclictest27561-21fschecks_count21:55:382
2184399842,71cyclictest21397-21ssh21:45:402
2183899848,59cyclictest24623-21apt-get20:50:381
21838998459,18cyclictest23666-21apt-config19:10:211
21838998455,8cyclictest181rcu_preempt22:00:211
2183899842,48cyclictest31-21ksoftirqd/123:40:481
21843998328,10cyclictest181rcu_preempt22:45:192
21838998346,14cyclictest181rcu_preempt21:25:211
21838998336,21cyclictest31-21ksoftirqd/123:35:441
21838998326,13cyclictest31-21ksoftirqd/122:17:251
2183899832,50cyclictest31-21ksoftirqd/122:55:481
2169728311,46sleep121738-21awk19:05:171
21843998261,15cyclictest1721-21kworker/u18:2-nfsiod00:18:042
21843998253,8cyclictest181rcu_preempt20:26:562
21843998126,8cyclictest181rcu_preempt19:50:522
21843998126,8cyclictest181rcu_preempt19:50:522
2183899816,26cyclictest31-21ksoftirqd/122:05:331
2183899814,53cyclictest31-21ksoftirqd/123:50:241
2185299808,48cyclictest0-21swapper/320:50:403
2184399803,52cyclictest24623-21apt-get20:50:322
2184399802,42cyclictest39-21ksoftirqd/221:06:092
2184399802,42cyclictest39-21ksoftirqd/221:06:092
21838998059,12cyclictest24417-21latency_hist20:50:201
21838998055,17cyclictest15932-21kworker/u17:3+events_unbound21:15:171
21838998038,17cyclictest181rcu_preempt21:05:401
21838998038,17cyclictest181rcu_preempt21:05:401
21838998015,28cyclictest31-21ksoftirqd/121:55:211
2184399794,34cyclictest6699-21ssh21:20:462
21838997955,18cyclictest5267-21kworker/u18:2+nfsiod21:41:361
21838997955,18cyclictest5267-21kworker/u18:2+nfsiod21:41:361
21838997945,10cyclictest181rcu_preempt22:41:011
21843997862,9cyclictest21461-21diskmemload00:25:522
2184399784,64cyclictest16986-21apt-get00:20:312
21843997845,9cyclictest15035-21kworker/u19:1+rpciod22:10:192
21843997843,9cyclictest9512-21ssh00:06:172
21843997835,17cyclictest181rcu_preempt21:35:452
21843997827,13cyclictest181rcu_preempt23:20:492
21843997827,13cyclictest181rcu_preempt23:20:492
21838997857,14cyclictest200-21systemd-journal23:22:371
21838997857,14cyclictest200-21systemd-journal23:22:371
21838997856,15cyclictest21824-21cyclictest00:19:351
21838997836,32cyclictest31165-21kworker/u18:0+nfsiod21:17:521
21838997836,32cyclictest31165-21kworker/u18:0+nfsiod21:17:521
21838997813,28cyclictest301ktimers/123:00:291
21843997754,17cyclictest6735-21interrupts19:55:402
21843997734,13cyclictest181rcu_preempt22:15:422
21843997731,15cyclictest21228-21ls20:40:332
21843997731,15cyclictest21228-21ls20:40:332
21838997748,20cyclictest4631-21sendmail_mailtr22:10:521
2183899773,36cyclictest31-21ksoftirqd/122:03:001
2184399768,28cyclictest22566-21kworker/2:2-events_power_efficient22:53:532
2184399768,28cyclictest22566-21kworker/2:2-events_power_efficient22:53:532
21843997632,8cyclictest181rcu_preempt23:39:092
2184399762,37cyclictest20045-21ls23:30:402
21838997656,14cyclictest16132-21latency_hist20:25:211
21838997647,20cyclictest20909-21sh00:25:401
21838997636,21cyclictest21824-21cyclictest00:07:171
2084927615,36sleep30-21swapper/319:00:513
21843997557,12cyclictest18966-21timerandwakeup21:40:532
21843997557,12cyclictest18966-21timerandwakeup21:40:532
21843997548,13cyclictest181rcu_preempt19:40:212
21843997545,15cyclictest181rcu_preempt22:28:522
21843997539,12cyclictest181rcu_preempt19:40:522
21838997539,16cyclictest181rcu_preempt23:09:051
21838997538,12cyclictest181rcu_preempt20:20:211
21838997531,19cyclictest291rcuc/123:57:161
21843997450,7cyclictest181rcu_preempt20:10:322
21843997448,12cyclictest181rcu_preempt20:05:192
21843997446,8cyclictest25697-21ssh22:45:452
21843997446,12cyclictest181rcu_preempt19:12:342
2183899742,32cyclictest31-21ksoftirqd/123:45:451
21843997356,12cyclictest26606-21timerandwakeup23:40:522
21843997341,8cyclictest181rcu_preempt23:53:022
21843997340,13cyclictest181rcu_preempt21:52:382
2184399733,41cyclictest13199-21ssh23:19:492
21838997349,10cyclictest181rcu_preempt23:30:391
21838997323,30cyclictest31-21ksoftirqd/119:57:081
21838997312,29cyclictest31-21ksoftirqd/119:20:341
2184399725,14cyclictest381ktimers/219:25:292
21843997234,12cyclictest181rcu_preempt19:46:352
21843997231,12cyclictest181rcu_preempt23:10:322
21843997229,17cyclictest0-21swapper/219:22:462
21843997224,30cyclictest0-21swapper/219:30:432
21843997212,29cyclictest39-21ksoftirqd/221:19:222
21843997212,29cyclictest39-21ksoftirqd/221:19:222
21838997243,17cyclictest16911-21switchtime22:30:521
21838997241,12cyclictest181rcu_preempt23:16:171
21838997229,12cyclictest0-21swapper/121:29:491
2183899722,37cyclictest8053-21grep00:05:191
21852997110,34cyclictest0-21swapper/323:55:173
2184399714,36cyclictest5583-21fschecks_time00:00:382
21843997137,8cyclictest181rcu_preempt00:30:272
21843997137,14cyclictest181rcu_preempt20:45:392
21843997133,17cyclictest181rcu_preempt00:15:022
21843997133,17cyclictest181rcu_preempt00:15:022
21843997125,13cyclictest181rcu_preempt22:58:322
2184399712,29cyclictest32755-21fschecks_count21:10:382
21843997116,23cyclictest0-21swapper/221:00:322
21838997154,11cyclictest30115-21grep19:25:531
2183899714,45cyclictest27407-21latency_hist19:20:201
21838997136,8cyclictest181rcu_preempt00:13:491
21838997136,8cyclictest181rcu_preempt00:13:491
21838997136,12cyclictest181rcu_preempt22:21:001
2183899713,29cyclictest31-21ksoftirqd/123:28:541
21838997126,28cyclictest31-21ksoftirqd/120:10:531
21843997045,16cyclictest9963-21ssh21:25:522
21843997043,11cyclictest181rcu_preempt23:25:402
21843997036,9cyclictest181rcu_preempt19:15:312
21843997029,12cyclictest181rcu_preempt20:30:452
21843997029,12cyclictest181rcu_preempt20:30:452
2184399702,43cyclictest39-21ksoftirqd/223:49:362
2183899707,29cyclictest31-21ksoftirqd/123:10:471
21838997048,8cyclictest181rcu_preempt00:30:421
21838997042,12cyclictest181rcu_preempt22:36:591
21838997039,9cyclictest181rcu_preempt19:11:111
21838997015,38cyclictest31-21ksoftirqd/122:50:451
21838997015,38cyclictest31-21ksoftirqd/122:50:451
21843996940,8cyclictest181rcu_preempt20:09:102
21843996939,14cyclictest181rcu_preempt23:07:152
21843996936,13cyclictest181rcu_preempt20:25:222
21838996947,15cyclictest21824-21cyclictest20:30:461
21838996947,15cyclictest21824-21cyclictest20:30:461
21838996942,8cyclictest181rcu_preempt21:34:251
21838996935,12cyclictest181rcu_preempt22:47:481
21838996934,16cyclictest181rcu_preempt20:01:251
21843996847,7cyclictest181rcu_preempt20:57:562
21843996847,7cyclictest181rcu_preempt20:57:562
21843996823,11cyclictest0-21swapper/222:00:322
2184399682,30cyclictest19206-21/usr/sbin/munin22:35:332
21838996841,19cyclictest29223-21diskmemload21:05:191
2183899683,42cyclictest31-21ksoftirqd/119:35:381
2183899682,28cyclictest31-21ksoftirqd/119:42:001
21843996733,8cyclictest181rcu_preempt22:20:342
21838996741,11cyclictest181rcu_preempt19:55:161
21838996741,11cyclictest181rcu_preempt19:55:161
21838996731,13cyclictest181rcu_preempt20:58:221
21838996731,13cyclictest181rcu_preempt20:58:221
21838996730,13cyclictest181rcu_preempt20:41:181
21838996730,13cyclictest181rcu_preempt20:41:181
21838996729,13cyclictest181rcu_preempt19:48:581
2183899672,21cyclictest31-21ksoftirqd/120:05:341
21852996617,42cyclictest0-21swapper/321:45:363
2184399663,35cyclictest20980-21latency_hist20:40:192
2184399663,35cyclictest20980-21latency_hist20:40:192
21843996543,7cyclictest181rcu_preempt21:34:302
21843996531,12cyclictest0-21swapper/220:20:212
2183899654,36cyclictest20473-21unixbench_multi20:35:541
2183899654,36cyclictest20473-21unixbench_multi20:35:541
21838996540,8cyclictest181rcu_preempt19:30:391
2185299633,45cyclictest0-21swapper/300:21:023
21852996321,37cyclictest0-21swapper/322:25:463
2185299603,42cyclictest8016-21taskset22:15:553
21852995843,10cyclictest451rcuc/320:19:513
21852995841,12cyclictest451rcuc/323:21:093
21852995841,12cyclictest451rcuc/323:21:093
21852995739,13cyclictest451rcuc/319:45:253
2185299564,40cyclictest1781-21chrt19:40:383
2185299563,37cyclictest20979-21chrt21:45:203
2185299563,37cyclictest20979-21chrt21:45:203
2185299554,34cyclictest19555-21chrt23:30:203
21852995537,11cyclictest451rcuc/323:42:003
2185299553,39cyclictest22872-21chrt20:45:343
2185299552,42cyclictest9220-21taskset23:12:083
2185299552,38cyclictest1172-21chrt22:05:383
21852995443,7cyclictest451rcuc/321:13:103
2185299543,37cyclictest4326-21taskset23:58:323
2185299543,34cyclictest9537-21chrt20:05:213
2185299543,33cyclictest16500-21chrt20:26:183
2185299542,38cyclictest26510-21chrt20:55:393
2185299542,38cyclictest26510-21chrt20:55:393
2185299542,36cyclictest22975-21chrt23:35:393
2185299533,38cyclictest5077-21chrt23:05:403
2185299533,38cyclictest29978-21chrt22:00:113
2185299533,36cyclictest11127-21taskset23:15:413
2185299533,36cyclictest10459-21chrt22:20:453
2185299532,35cyclictest7205-21chrt21:20:553
2185299523,37cyclictest32451-21chrt21:10:213
2185299523,37cyclictest32451-21chrt21:10:213
2185299523,36cyclictest26559-21chrt19:15:483
2185299523,36cyclictest21762-21chrt20:40:483
2185299523,36cyclictest21762-21chrt20:40:483
2185299523,36cyclictest18485-21chrt22:34:273
2185299523,36cyclictest18024-21chrt21:40:213
2185299523,35cyclictest9938-21chrt20:05:413
2185299522,35cyclictest9532-21chrt21:25:423
21852995212,26cyclictest0-21swapper/319:10:343
2185299514,37cyclictest28150-21chrt19:20:503
2185299514,34cyclictest21037-21chrt20:40:203
2185299514,34cyclictest21037-21chrt20:40:203
2185299513,38cyclictest23083-21chrt22:40:563
2185299513,37cyclictest18408-21chrt20:30:443
2185299513,37cyclictest18408-21chrt20:30:443
2185299513,32cyclictest6313-21chrt19:55:203
2185299513,32cyclictest6313-21chrt19:55:203
2185299513,32cyclictest3791-21chrt23:03:423
2185299513,32cyclictest13745-21chrt21:32:273
21852995129,15cyclictest451rcuc/323:49:093
2185299512,40cyclictest6392-21chrt22:14:503
2185299512,35cyclictest16850-21chrt00:19:553
2185299512,33cyclictest21877-21chrt22:40:193
21852995038,8cyclictest451rcuc/300:13:203
21852995038,8cyclictest451rcuc/300:13:203
2185299503,36cyclictest23977-21chrt00:30:423
2185299503,34cyclictest32074-21chrt22:55:523
2185299502,36cyclictest820-21chrt19:35:523
2185299502,36cyclictest4623-21chrt21:17:023
2185299502,36cyclictest4623-21chrt21:17:023
2185299502,36cyclictest30658-21taskset22:54:543
2185299502,36cyclictest30658-21taskset22:54:543
2185299494,33cyclictest25269-21chrt22:45:343
2185299493,32cyclictest29996-21taskset19:25:523
2185299493,32cyclictest28593-21chrt21:00:493
2185299483,34cyclictest21417-21chrt00:25:513
2185299483,33cyclictest11486-21chrt20:10:393
2185299483,32cyclictest5603-21chrt00:00:393
2185299482,32cyclictest20657-21chrt23:30:533
2185299473,34cyclictest22003-21taskset19:05:233
2185299472,32cyclictest9961-21chrt00:07:323
2185299452,31cyclictest31721-21chrt22:01:413
21852994431,9cyclictest451rcuc/319:58:253
21852994424,13cyclictest451rcuc/321:53:483
21852994423,15cyclictest451rcuc/320:21:543
21852994331,8cyclictest451rcuc/319:31:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional