You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 10:09
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Mar 26, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
95679911633,41cyclictest0-21swapper/022:45:220
95819910783,16cyclictest9556-21cyclictest23:09:282
9581991053,75cyclictest22925-21rm23:00:102
9581991053,75cyclictest22925-21rm23:00:102
95679910534,28cyclictest21680-21ssh23:50:510
95679910437,28cyclictest1750-21seq00:12:110
95679910231,37cyclictest29742-21irqstats00:04:540
95679910231,37cyclictest29742-21irqstats00:04:540
95679910035,26cyclictest27512-21sh00:00:190
84702998,63sleep18561-21load19:00:011
9567999836,24cyclictest17728-21irqstats23:44:540
9567999834,32cyclictest994-21apt-get22:24:420
9567999834,32cyclictest994-21apt-get22:24:420
9567999834,26cyclictest7686-21ssh00:21:580
9567999833,29cyclictest24427-21apt-get22:09:530
9567999833,29cyclictest24427-21apt-get22:09:530
9567999832,26cyclictest23504-21seq19:44:490
9567999830,31cyclictest19676-21snmpd21:17:130
9567999830,31cyclictest19676-21snmpd21:17:130
9567999824,44cyclictest22529-21ssh22:05:450
9567999824,44cyclictest22529-21ssh22:05:450
9581999770,18cyclictest9162-21fschecks_time20:39:502
9567999737,23cyclictest31064-21/usr/sbin/munin22:19:570
9567999728,31cyclictest32410-21/usr/sbin/munin23:15:000
9567999727,28cyclictest29547-21idleruntime-cro20:04:290
9567999727,28cyclictest29547-21idleruntime-cro20:04:290
9567999632,27cyclictest18698-21rm22:53:090
9567999628,27cyclictest15288-21apt-get19:19:590
9567999628,27cyclictest15288-21apt-get19:19:590
9567999535,26cyclictest11245-21rm22:40:220
9567999535,22cyclictest8317-21ssh23:29:400
9567999534,29cyclictest28125-21ssh23:08:410
9567999533,27cyclictest15195-21ssh21:54:420
9567999533,23cyclictest20446-21ssh21:09:550
9567999532,25cyclictest12206-21xargs23:35:040
9567999436,21cyclictest29423-21ssh22:17:290
9567999434,23cyclictest29249-21memory23:09:550
9567999433,30cyclictest9544-21ssh22:39:010
9567999433,25cyclictest21118-21runrttasks22:56:270
9567999433,25cyclictest21118-21runrttasks22:56:270
9567999432,28cyclictest11502-21ssh00:29:160
9567999432,28cyclictest11502-21ssh00:29:160
9567999431,27cyclictest3620-21rm23:20:390
9567999430,27cyclictest0-21swapper/022:32:030
9567999429,31cyclictest23332-21/usr/sbin/munin23:54:380
9567999429,31cyclictest23332-21/usr/sbin/munin23:54:380
9581999356,13cyclictest181rcu_preempt22:20:052
9567999334,29cyclictest18804-21rm21:06:570
9567999334,29cyclictest18804-21rm21:06:570
9567999334,28cyclictest5283-21rm23:24:380
9567999334,26cyclictest884-21fschecks_time20:14:520
9567999334,21cyclictest28751-21apt-get21:24:470
9567999334,21cyclictest28751-21apt-get21:24:470
9567999331,26cyclictest0-21swapper/021:20:020
9567999331,22cyclictest4471-21ssh21:36:220
9567999327,31cyclictest14332-21awk23:39:400
9567999235,18cyclictest20477-21/usr/sbin/munin19:34:530
9567999232,35cyclictest19861-21taskset22:01:410
9567999232,22cyclictest11857-21ssh00:29:340
9567999231,24cyclictest13805-21rm21:51:390
9567999231,24cyclictest10719-21ssh21:46:250
9567999231,24cyclictest10719-21ssh21:46:250
9567999228,30cyclictest32405-21awk20:10:080
9567999228,27cyclictest23355-21/usr/sbin/munin23:00:360
9567999145,36cyclictest2929-21apt-get00:14:410
9567999145,36cyclictest2929-21apt-get00:14:410
9567999132,27cyclictest7155-21ssh21:40:300
9567999128,28cyclictest13204-21taskset20:51:210
9581999064,9cyclictest181rcu_preempt22:06:192
9581999064,9cyclictest181rcu_preempt22:06:192
9581999055,11cyclictest181rcu_preempt20:14:492
9573999012,60cyclictest13270-21apt-get22:44:491
9567999030,23cyclictest11022-21seq19:08:320
9567999020,36cyclictest0-21swapper/020:09:300
9567999020,36cyclictest0-21swapper/020:09:300
9581998953,13cyclictest181rcu_preempt22:39:592
958199893,74cyclictest24427-21apt-get22:09:452
958199893,74cyclictest24427-21apt-get22:09:452
9567998935,19cyclictest8621-21cron20:39:290
9567998932,25cyclictest1627-21ssh21:31:520
93972896,64sleep00-21swapper/019:03:590
9581998851,13cyclictest181rcu_preempt00:14:522
9581998851,13cyclictest181rcu_preempt00:14:522
9567998833,19cyclictest5385-21cron20:29:290
9567998833,19cyclictest5385-21cron20:29:290
9567998831,19cyclictest16883-21/usr/sbin/munin19:24:530
9567998831,19cyclictest16883-21/usr/sbin/munin19:24:530
9567998827,29cyclictest14073-21awk19:15:030
943628812,47sleep29465-21munin-run19:04:302
9567998732,19cyclictest2356-21df20:19:480
9567998731,22cyclictest12986-21taskset19:12:280
9581998651,13cyclictest181rcu_preempt21:05:522
9581998651,13cyclictest181rcu_preempt21:05:522
9581998642,8cyclictest181rcu_preempt00:20:032
9581998559,9cyclictest15488-21ssh22:48:132
9581998548,16cyclictest181rcu_preempt21:15:212
9581998548,16cyclictest181rcu_preempt21:15:212
9581998544,18cyclictest181rcu_preempt20:59:322
9581998544,18cyclictest181rcu_preempt20:59:322
9581998542,11cyclictest181rcu_preempt20:34:392
958199853,53cyclictest9131-21apt-get21:44:462
958199853,53cyclictest9131-21apt-get21:44:462
9567998531,20cyclictest0-21swapper/020:45:060
9567998530,18cyclictest14900-21taskset20:57:000
9567998530,18cyclictest14900-21taskset20:57:000
9567998528,21cyclictest27830-21taskset19:58:190
958199846,54cyclictest23005-21modprobe23:00:222
9581998450,12cyclictest181rcu_preempt21:20:072
9581998441,10cyclictest181rcu_preempt21:41:252
9581998438,10cyclictest181rcu_preempt21:01:262
9567998430,21cyclictest26207-21taskset19:53:320
9567998429,20cyclictest19664-21taskset19:32:190
9581998358,15cyclictest2223-21kworker/u17:0-xprtiod23:10:032
9581998355,8cyclictest181rcu_preempt23:50:052
9581998344,18cyclictest32354-21ssh23:14:572
9573998359,15cyclictest936-21rm22:24:331
9573998355,18cyclictest29069-21apt-key21:24:341
957399833,68cyclictest4055-21apt-get20:25:181
957399833,68cyclictest4055-21apt-get20:25:181
9567998338,35cyclictest5646-21/usr/sbin/munin20:29:520
9567998330,19cyclictest10203-21taskset20:44:030
9567998330,18cyclictest16590-21taskset21:03:080
9581998259,8cyclictest181rcu_preempt21:59:512
9581998244,14cyclictest181rcu_preempt21:26:192
9581998244,14cyclictest181rcu_preempt21:26:192
958199823,55cyclictest16605-21ssh22:49:562
9581998232,17cyclictest181rcu_preempt22:35:372
9581998232,12cyclictest181rcu_preempt23:34:522
9581998229,13cyclictest181rcu_preempt22:18:532
9567998236,11cyclictest22283-21/usr/sbin/munin19:40:000
9581998154,11cyclictest181rcu_preempt22:25:282
9581998154,11cyclictest181rcu_preempt22:25:282
9581998141,17cyclictest181rcu_preempt21:10:292
9581998140,13cyclictest181rcu_preempt21:39:302
958199804,52cyclictest20519-21latency19:34:542
9573998020,27cyclictest31-21ksoftirqd/123:54:061
9573998014,33cyclictest31-21ksoftirqd/121:17:291
9573998014,33cyclictest31-21ksoftirqd/121:17:291
9581997956,16cyclictest16724-21diskmemload00:24:542
9581997956,16cyclictest16724-21diskmemload00:24:542
9581997951,21cyclictest9556-21cyclictest00:12:442
9581997945,12cyclictest181rcu_preempt23:27:382
9581997943,13cyclictest181rcu_preempt20:25:502
9581997943,13cyclictest181rcu_preempt20:25:502
9581997942,12cyclictest181rcu_preempt20:14:322
958199792,48cyclictest39-21ksoftirqd/219:39:452
9573997959,14cyclictest20003-21idleruntime-cro23:49:281
9573997927,28cyclictest16724-21diskmemload21:04:311
9581997848,13cyclictest181rcu_preempt21:54:522
9581997843,12cyclictest181rcu_preempt23:44:502
9581997821,34cyclictest0-21swapper/200:34:332
9573997858,12cyclictest32113-21latency_hist00:09:321
9573997858,12cyclictest32113-21latency_hist00:09:321
9581997746,13cyclictest181rcu_preempt20:09:312
9581997746,13cyclictest181rcu_preempt20:09:312
9581997744,13cyclictest181rcu_preempt21:30:032
9581997743,11cyclictest181rcu_preempt22:34:212
9581997739,14cyclictest181rcu_preempt23:55:052
9581997739,14cyclictest181rcu_preempt23:55:052
9581997733,14cyclictest181rcu_preempt00:03:002
9581997718,12cyclictest0-21swapper/219:34:322
9573997758,13cyclictest3776-21threads21:35:041
9573997721,20cyclictest31-21ksoftirqd/100:27:131
9573997721,20cyclictest31-21ksoftirqd/100:27:131
9573997719,27cyclictest31-21ksoftirqd/123:14:551
9581997650,9cyclictest181rcu_preempt19:55:082
9581997648,18cyclictest0-21swapper/219:49:332
9573997654,14cyclictest2952-21systemd-journal00:02:441
957399762,30cyclictest31-21ksoftirqd/122:44:251
9581997559,9cyclictest170-21jbd2/mmcblk2p2-819:09:482
9581997553,12cyclictest9556-21cyclictest19:49:532
9581997547,11cyclictest181rcu_preempt20:44:422
9581997540,12cyclictest181rcu_preempt21:51:592
9581997537,8cyclictest181rcu_preempt20:54:322
9581997524,21cyclictest0-21swapper/223:30:052
9573997558,8cyclictest0-21swapper/119:49:291
9573997558,11cyclictest5340-21sh00:18:461
9573997558,11cyclictest5340-21sh00:18:461
9573997554,14cyclictest16724-21diskmemload21:43:251
957399752,32cyclictest31-21ksoftirqd/123:41:141
957399752,32cyclictest31-21ksoftirqd/121:26:071
957399752,32cyclictest31-21ksoftirqd/121:26:071
9573997517,35cyclictest31-21ksoftirqd/122:08:101
9573997517,35cyclictest31-21ksoftirqd/122:08:101
896727515,38sleep30-21swapper/319:00:153
9581997441,9cyclictest181rcu_preempt00:09:332
9581997441,9cyclictest181rcu_preempt00:09:332
9581997422,26cyclictest0-21swapper/219:19:332
9573997455,14cyclictest7128-21latency_hist20:34:321
9573997440,24cyclictest8183-21sort23:29:321
9573997439,19cyclictest27636-21df_inode22:14:461
957399742,29cyclictest31-21ksoftirqd/123:39:101
957399742,24cyclictest31-21ksoftirqd/123:22:261
9573997421,21cyclictest31-21ksoftirqd/121:55:171
9573997411,30cyclictest31-21ksoftirqd/122:33:171
9581997348,19cyclictest3556-21latency_hist20:24:332
9573997358,9cyclictest9556-21cyclictest21:08:051
9573997358,9cyclictest9556-21cyclictest21:08:051
9573997354,12cyclictest0-21swapper/122:11:041
9573997354,12cyclictest0-21swapper/122:11:041
957399734,25cyclictest31-21ksoftirqd/120:14:301
9573997337,15cyclictest181rcu_preempt22:56:091
9573997337,15cyclictest181rcu_preempt22:56:091
957399732,29cyclictest26056-21fschecks_time23:04:521
958699723,55cyclictest9882-21taskset20:40:043
9581997241,10cyclictest181rcu_preempt23:20:032
9581997239,21cyclictest11271-21latency_hist19:09:332
9581997239,14cyclictest181rcu_preempt20:01:542
9581997239,14cyclictest181rcu_preempt20:01:542
9581997233,11cyclictest181rcu_preempt23:43:242
9573997258,10cyclictest9556-21cyclictest21:45:231
9573997258,10cyclictest9556-21cyclictest21:45:231
9573997252,14cyclictest25315-21kernelversion19:49:551
9573997238,9cyclictest9556-21cyclictest22:02:561
9573997233,9cyclictest181rcu_preempt23:56:131
9573997233,9cyclictest181rcu_preempt23:56:131
9573997230,35cyclictest3131-21kworker/u18:2+nfsiod00:19:401
9573997215,34cyclictest31-21ksoftirqd/119:36:311
9581997148,9cyclictest181rcu_preempt19:24:512
9581997148,9cyclictest181rcu_preempt19:24:512
9573997156,10cyclictest9556-21cyclictest19:09:541
9573997152,14cyclictest28033-21latency_hist19:59:321
957399713,35cyclictest31-21ksoftirqd/119:15:451
957399709,31cyclictest31-21ksoftirqd/122:35:541
957399705,37cyclictest31-21ksoftirqd/119:29:591
9573997048,11cyclictest0-21swapper/119:24:521
9573997048,11cyclictest0-21swapper/119:24:521
9573997019,18cyclictest31-21ksoftirqd/120:44:571
958699693,50cyclictest19908-21taskset19:34:313
958699692,54cyclictest30567-21chrt22:19:453
9581996943,12cyclictest181rcu_preempt19:23:352
9581996943,12cyclictest181rcu_preempt19:23:352
957399693,26cyclictest31-21ksoftirqd/123:00:471
9573996912,22cyclictest31-21ksoftirqd/119:08:201
958699684,51cyclictest22890-21chrt23:00:093
958699684,51cyclictest22890-21chrt23:00:093
958699683,53cyclictest32530-21taskset21:29:483
958699683,50cyclictest22502-21taskset23:53:103
9586996833,28cyclictest0-21swapper/322:50:003
958699682,55cyclictest1071-21taskset23:16:553
9581996816,28cyclictest0-21swapper/220:29:542
9573996854,10cyclictest9556-21cyclictest23:30:011
9573996852,11cyclictest9556-21cyclictest20:20:071
9573996846,7cyclictest181rcu_preempt21:12:101
957399683,35cyclictest31-21ksoftirqd/120:15:061
957399682,34cyclictest31-21ksoftirqd/122:24:491
957399682,34cyclictest31-21ksoftirqd/122:24:491
958699674,49cyclictest14694-21chrt23:39:523
958699673,52cyclictest17066-21chrt21:57:163
958699673,49cyclictest28015-21chrt21:22:263
958699673,49cyclictest18168-21chrt21:59:393
958699672,52cyclictest8031-21taskset21:42:573
958699672,50cyclictest11814-21chrt19:09:443
9573996753,8cyclictest0-21swapper/122:49:511
9573996738,8cyclictest181rcu_preempt23:12:101
9573996737,8cyclictest181rcu_preempt20:34:591
9573996737,7cyclictest181rcu_preempt00:29:531
9573996733,8cyclictest181rcu_preempt21:52:471
958699663,51cyclictest5475-21taskset22:31:073
958699663,50cyclictest4495-21taskset00:16:243
958699663,50cyclictest4495-21taskset00:16:243
958699663,48cyclictest1758-21chrt20:17:163
958699662,51cyclictest21702-21taskset21:11:403
958699662,51cyclictest13331-21chrt22:44:493
958699662,50cyclictest8246-21taskset22:35:253
958699662,50cyclictest31169-21taskset00:07:083
958699662,50cyclictest31169-21taskset00:07:083
958699662,50cyclictest14770-21taskset19:19:283
957399667,33cyclictest31-21ksoftirqd/120:00:011
957399667,33cyclictest31-21ksoftirqd/120:00:011
9573996646,7cyclictest181rcu_preempt00:11:221
957399664,26cyclictest31-21ksoftirqd/121:29:421
958699653,49cyclictest7681-21taskset20:34:543
958699653,49cyclictest25742-21taskset23:04:393
958699653,49cyclictest18135-21chrt23:45:033
958699653,49cyclictest12388-21chrt21:49:543
958699653,48cyclictest22465-21taskset22:05:343
958699653,48cyclictest22465-21taskset22:05:343
958699652,50cyclictest7571-21taskset00:21:383
958699652,50cyclictest498-21chrt00:09:593
958699652,49cyclictest24630-21taskset19:48:573
958699652,49cyclictest12660-21chrt20:50:003
9573996515,12cyclictest301ktimers/120:05:041
9573996515,12cyclictest301ktimers/120:05:041
958699643,49cyclictest10187-21chrt23:32:153
958699643,47cyclictest4089-21taskset23:21:583
958699643,47cyclictest30987-21chrt21:26:363
958699643,47cyclictest30987-21chrt21:26:363
958699643,47cyclictest25191-21chrt23:57:113
958699643,47cyclictest25191-21chrt23:57:113
958699642,50cyclictest31096-21taskset20:09:023
958699642,50cyclictest31096-21taskset20:09:023
958699642,48cyclictest9115-21chrt00:24:503
958699642,48cyclictest9115-21chrt00:24:503
9573996428,17cyclictest9556-21cyclictest20:41:581
958699633,47cyclictest3191-21chrt20:20:073
958699633,46cyclictest15372-21chrt20:59:403
958699633,45cyclictest10522-21chrt20:44:423
958699632,48cyclictest4193-21chrt21:35:363
958699632,48cyclictest29211-21chrt22:16:533
958699632,48cyclictest28753-21chrt23:09:423
958699632,48cyclictest22815-21taskset19:41:153
958699632,48cyclictest11199-21chrt22:40:153
958699632,47cyclictest11913-21chrt00:29:413
9573996348,10cyclictest11337-21munin-node19:22:091
9573996348,10cyclictest11337-21munin-node19:22:091
9573996329,9cyclictest181rcu_preempt19:40:101
958699623,46cyclictest20583-21chrt19:34:563
958699623,46cyclictest1190-21chrt22:24:473
958699623,46cyclictest1190-21chrt22:24:473
958699622,48cyclictest24211-21chrt21:15:123
958699622,48cyclictest24211-21chrt21:15:123
958699622,47cyclictest26137-21chrt22:11:473
958699622,47cyclictest26137-21chrt22:11:473
958699622,46cyclictest14846-21chrt20:56:203
958699622,46cyclictest14846-21chrt20:56:203
957399621,29cyclictest31-21ksoftirqd/120:54:441
957399621,29cyclictest31-21ksoftirqd/120:54:441
958699613,48cyclictest10801-21taskset19:05:393
958699612,47cyclictest27034-21chrt19:54:563
958699612,47cyclictest12686-21taskset23:35:473
958699612,46cyclictest6658-21chrt20:30:123
958699612,46cyclictest29205-21taskset20:00:193
958699612,46cyclictest29205-21taskset20:00:193
958699612,45cyclictest27587-21chrt00:00:313
957399613,33cyclictest12105-21/usr/sbin/munin20:49:571
958699603,45cyclictest16880-21chrt19:24:393
958699603,45cyclictest16880-21chrt19:24:393
958699603,43cyclictest25978-21chrt19:50:353
958699603,43cyclictest17196-21chrt21:04:463
958699603,43cyclictest17196-21chrt21:04:463
958699602,48cyclictest23032-21chrt23:00:263
958699602,47cyclictest4099-21chrt20:25:143
958699602,47cyclictest4099-21chrt20:25:143
958699602,46cyclictest15266-21chrt19:19:373
958699602,46cyclictest15266-21chrt19:19:373
958699602,44cyclictest31758-21chrt20:09:553
958699593,43cyclictest5886-21chrt23:24:573
958699583,44cyclictest11798-21chrt21:49:253
958699583,44cyclictest11798-21chrt21:49:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional