You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-01 - 12:49
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Mon Jun 01, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30948212512,76sleep130952-21ls19:00:041
30227211564,31sleep20-21swapper/218:55:452
31015991133,71cyclictest4931-21kworker/u19:0+ext4-rsv-conversion22:10:341
31015991006,72cyclictest2145-21rm22:41:061
31015991006,72cyclictest2145-21rm22:41:061
1045221000,6sleep110451-21open_files22:00:331
1045221000,6sleep110451-21open_files22:00:331
3101199993,70cyclictest25366-21kworker/u19:2+ext4-rsv-conversion23:35:240
3101199993,70cyclictest25366-21kworker/u19:2+ext4-rsv-conversion23:35:240
3101199988,44cyclictest15-21ksoftirqd/019:00:150
31015999618,66cyclictest1807-21apt-get23:35:241
31015999618,66cyclictest1807-21apt-get23:35:241
3101199952,64cyclictest1819-21apt-get22:41:060
3101199952,64cyclictest1819-21apt-get22:41:060
31019999432,52cyclictest0-21swapper/223:25:372
31011999216,52cyclictest15-21ksoftirqd/020:17:210
31011999216,52cyclictest15-21ksoftirqd/020:17:210
31019999163,22cyclictest2456-21sendmail20:50:002
3101999914,39cyclictest8994-21grep19:30:032
3101199914,49cyclictest15-21ksoftirqd/023:10:150
3101199913,27cyclictest7121-21fschecks_time21:55:230
3101999904,40cyclictest6980-21latency_hist19:25:032
3101999904,39cyclictest12483-21rm23:50:592
31019998967,16cyclictest544-21cron21:54:592
3101999894,40cyclictest8864-21ntp_kernel_pll_23:45:292
3101999894,36cyclictest12236-21/usr/sbin/munin21:11:172
3101999893,31cyclictest15926-21ssh23:03:412
3101199893,34cyclictest15-21ksoftirqd/021:46:220
3101999884,38cyclictest11098-21fschecks_time22:55:222
3101999883,41cyclictest9448-21seq22:52:042
3101999883,41cyclictest9448-21seq22:52:042
3101999883,41cyclictest28295-21latency_hist23:25:022
31019998766,13cyclictest0-21swapper/222:30:272
31019998766,13cyclictest0-21swapper/222:30:272
3101999874,40cyclictest26866-21latency_hist20:25:022
3101999874,40cyclictest26866-21latency_hist20:25:022
3101999874,38cyclictest7520-21awk22:50:002
31019998734,45cyclictest0-21swapper/221:18:032
3101999873,42cyclictest10920-21fschecks_count19:35:232
3101999873,41cyclictest22974-21fschecks_time23:15:232
3101999873,41cyclictest22974-21fschecks_time23:15:232
3101999873,38cyclictest352-21runrttasks22:21:352
3101999873,36cyclictest14436-21fschecks_count19:46:262
3101999873,33cyclictest32211-21tune2fs20:40:242
3101999873,31cyclictest9213-21ssh21:05:222
3101999873,30cyclictest15401-21runrttasks23:58:292
31019998730,49cyclictest0-21swapper/222:11:322
3101999872,30cyclictest1579-21latency_hist22:40:032
3101999872,30cyclictest1579-21latency_hist22:40:032
31019998710,33cyclictest20575-21cut20:05:042
3101199876,46cyclictest15-21ksoftirqd/019:17:520
3101199872,52cyclictest15-21ksoftirqd/022:37:480
3101199872,52cyclictest15-21ksoftirqd/022:37:480
3101999864,41cyclictest352-21runrttasks21:02:332
3101999864,41cyclictest352-21runrttasks21:02:332
3101999864,41cyclictest23001-21ps00:10:352
3101999864,36cyclictest352-21runrttasks23:07:192
3101999864,32cyclictest30187-21latency_hist21:40:042
3101999864,30cyclictest21259-21taskset21:25:122
3101999863,40cyclictest3291-21runrttasks23:36:232
3101999863,40cyclictest3291-21runrttasks23:36:232
3101999863,38cyclictest28400-21latency_hist00:20:032
3101999863,37cyclictest5967-21open_inodes23:40:332
3101999863,36cyclictest19849-21cstates23:10:172
3101999862,28cyclictest6794-21/usr/sbin/munin21:55:292
31011998664,15cyclictest15915-21latency_hist22:10:040
3101199863,55cyclictest15-21ksoftirqd/021:40:560
3101199862,45cyclictest15-21ksoftirqd/022:02:280
3101199862,45cyclictest15-21ksoftirqd/022:02:280
3101999854,40cyclictest14037-21latency_hist19:45:042
3101999853,66cyclictest1260-21apt-get22:40:142
3101999853,66cyclictest1260-21apt-get22:40:142
3101999853,40cyclictest853-21cpuspeed19:05:192
3101999853,27cyclictest5695-21ssh21:00:002
3101999853,26cyclictest30221-21/usr/sbin/munin21:40:272
3101999853,25cyclictest30036-21cron20:35:002
3101999852,30cyclictest31576-21cpuspeed00:25:172
31015998563,16cyclictest28448-21sendmail20:30:001
31011998563,9cyclictest181rcu_preempt23:25:450
3101199853,55cyclictest16265-21sh22:10:460
3101199853,53cyclictest15-21ksoftirqd/000:27:230
3101999844,38cyclictest352-21runrttasks00:07:372
3101999844,30cyclictest4185-21latency_hist20:55:022
31019998434,43cyclictest0-21swapper/219:55:402
3101999843,39cyclictest32434-21switchtime23:30:362
3101999843,32cyclictest2651-21grep19:10:232
3101999843,31cyclictest13469-21grep22:05:322
3101999843,31cyclictest12621-21latency_hist22:05:032
3101999843,31cyclictest12621-21latency_hist22:05:032
3101999843,30cyclictest15771-21/usr/sbin/munin19:50:242
3101999843,28cyclictest28486-21/usr/sbin/munin00:20:322
3101999843,20cyclictest28002-21unixbench_multi20:25:362
3101999842,29cyclictest31460-21grep19:00:202
31015998458,16cyclictest1-21systemd21:10:011
3101199846,46cyclictest15-21ksoftirqd/022:31:000
3101199846,46cyclictest15-21ksoftirqd/022:31:000
3101199843,35cyclictest15-21ksoftirqd/022:19:290
3101199842,52cyclictest15-21ksoftirqd/019:08:080
31011998417,40cyclictest15-21ksoftirqd/023:50:260
31011998413,47cyclictest15-21ksoftirqd/023:05:200
3101999835,16cyclictest0-21swapper/222:25:132
3101999834,32cyclictest20599-21/usr/sbin/munin20:05:222
3101999834,32cyclictest18595-21idleruntime21:20:232
3101999834,27cyclictest24490-21/usr/sbin/munin21:30:222
3101999834,22cyclictest16525-21fschecks_time00:00:182
3101999833,39cyclictest1386-21ntp_kernel_pll_21:45:292
3101999833,31cyclictest10442-21munin-run19:35:002
3101999832,29cyclictest27503-21grep21:35:222
3101999832,29cyclictest22324-21cpuspeed20:10:192
3101199838,37cyclictest15-21ksoftirqd/023:24:520
3101199833,67cyclictest5672-21modprobe20:59:590
3101199833,55cyclictest15-21ksoftirqd/021:15:170
31011998326,39cyclictest15-21ksoftirqd/020:08:420
3101999823,22cyclictest31045-21/usr/sbin/munin20:35:352
3101599829,35cyclictest983-21latency_hist20:45:031
31011998232,28cyclictest15-21ksoftirqd/022:25:070
31011998218,37cyclictest15-21ksoftirqd/020:44:180
3101999814,29cyclictest4394-21kernelversion19:15:272
3101999813,26cyclictest24114-21tune2fs20:15:232
3101999813,26cyclictest24114-21tune2fs20:15:232
3101999812,27cyclictest19079-21cpu22:15:172
31015998160,13cyclictest31002-21cyclictest19:05:041
31011998152,11cyclictest181rcu_preempt21:52:210
31011998145,16cyclictest181rcu_preempt19:39:220
3101199812,47cyclictest15-21ksoftirqd/023:40:180
31015998059,14cyclictest29941-21idleruntime-cro21:40:001
31015998059,14cyclictest29941-21idleruntime-cro21:40:001
3101199802,42cyclictest15-21ksoftirqd/000:23:180
31011997955,15cyclictest300-21rs:main0
31011997953,19cyclictest25713-21fschecks_time20:20:220
31011997953,19cyclictest25713-21fschecks_time20:20:220
31011997949,23cyclictest0-21swapper/022:25:000
3101199794,46cyclictest15-21ksoftirqd/019:27:110
31011997938,31cyclictest0-21swapper/023:20:030
31011997938,31cyclictest0-21swapper/023:20:030
31011997913,26cyclictest15-21ksoftirqd/019:50:240
31015997815,22cyclictest30215-21seq23:27:041
3101199783,35cyclictest15-21ksoftirqd/000:15:380
31015997715,25cyclictest2720-21seq20:50:131
3101199773,43cyclictest15-21ksoftirqd/020:47:420
3101199772,42cyclictest15-21ksoftirqd/022:51:540
3101199772,42cyclictest15-21ksoftirqd/022:51:540
31015997656,11cyclictest0-21swapper/122:10:031
3101599764,54cyclictest6979-21apt-get19:25:141
31011997639,21cyclictest15-21ksoftirqd/021:30:300
3101199763,49cyclictest15-21ksoftirqd/021:28:330
31011997613,36cyclictest15-21ksoftirqd/021:05:160
31011997612,35cyclictest15-21ksoftirqd/023:57:300
31015997558,10cyclictest2395-21kworker/u19:1+flush-179:000:25:341
31015997554,15cyclictest6494-21munin-plugin-st21:55:011
31015997553,16cyclictest8694-21kernelversion23:45:261
31015997536,26cyclictest3764-21modprobe19:15:001
31015997515,25cyclictest30338-21ssh00:22:411
31015997514,27cyclictest0-21swapper/123:52:571
3101199755,24cyclictest15-21ksoftirqd/021:00:190
3101199755,24cyclictest15-21ksoftirqd/021:00:190
3101199753,58cyclictest13637-21cron19:45:000
31011997523,35cyclictest15-21ksoftirqd/020:10:210
31011997513,35cyclictest15-21ksoftirqd/023:45:160
3101199746,46cyclictest15-21ksoftirqd/023:33:350
3101199745,39cyclictest12191-21modprobe21:10:050
31011997452,9cyclictest181rcu_preempt20:39:240
31011997452,8cyclictest181rcu_preempt22:56:240
3101199743,41cyclictest15-21ksoftirqd/019:49:340
3101199743,39cyclictest15-21ksoftirqd/021:21:150
3101199743,23cyclictest5679-21diskmemload00:05:240
31011997429,17cyclictest181rcu_preempt20:27:460
3101199742,37cyclictest15-21ksoftirqd/000:11:350
31015997352,15cyclictest2130-21apt-key23:35:041
31015997351,16cyclictest17375-21proc_pri23:05:321
31015997314,22cyclictest20836-21seq23:10:391
31015997312,23cyclictest5679-21diskmemload00:14:251
31015997257,10cyclictest31002-21cyclictest20:20:291
31015997257,10cyclictest31002-21cyclictest20:20:291
31015997218,22cyclictest24942-21seq20:16:481
31015997218,22cyclictest24942-21seq20:16:481
31015997213,20cyclictest22724-21seq21:26:401
31015997210,35cyclictest4752-21ssh22:45:041
31011997249,12cyclictest0-21swapper/019:25:040
31011997235,15cyclictest181rcu_preempt00:01:220
3101199723,44cyclictest15-21ksoftirqd/020:00:390
3101199723,36cyclictest30899-21latency_hist19:00:050
31015997157,10cyclictest31002-21cyclictest22:30:211
31015997157,10cyclictest31002-21cyclictest22:30:211
3101599715,26cyclictest0-21swapper/123:00:011
31015997135,27cyclictest10232-21sh22:54:221
31015997135,27cyclictest10232-21sh22:54:221
31015997124,9cyclictest0-21swapper/120:06:561
31015997115,23cyclictest12285-21seq19:40:071
31015997114,24cyclictest5429-21seq20:57:061
31015997114,23cyclictest0-21swapper/119:38:151
31015997114,19cyclictest16847-21seq19:51:251
31015997113,22cyclictest6877-21seq19:23:521
3101199716,43cyclictest15-21ksoftirqd/022:49:250
31024997058,8cyclictest451rcuc/319:00:393
3101599708,26cyclictest0-21swapper/121:21:171
31015997051,12cyclictest6250-21timerwakeupswit23:40:371
31015997050,14cyclictest31067-21switchtime20:35:351
3101599703,29cyclictest0-21swapper/122:35:321
3101599703,29cyclictest0-21swapper/122:35:321
31015997011,26cyclictest352-21runrttasks19:15:541
31011997053,10cyclictest0-21swapper/020:30:230
31011997028,26cyclictest15-21ksoftirqd/020:51:150
3101599699,28cyclictest14513-21grep23:00:331
31015996928,22cyclictest0-21swapper/100:16:081
31015996920,14cyclictest181rcu_preempt21:35:031
31015996918,12cyclictest0-21swapper/122:23:591
31015996913,23cyclictest23395-21seq20:12:311
31015996913,20cyclictest20285-21seq20:04:211
31015996912,20cyclictest18657-21seq19:59:021
3101199694,32cyclictest13721-21/usr/sbin/munin23:00:330
3101199692,33cyclictest15-21ksoftirqd/021:39:530
3101199692,33cyclictest15-21ksoftirqd/021:39:530
3101599689,22cyclictest0-21swapper/100:07:041
3101599687,26cyclictest30871-21ntp_states21:40:311
31015996852,10cyclictest12414-21kworker/u17:3-writeback23:18:231
31015996852,10cyclictest12414-21kworker/u17:3-writeback23:18:231
31015996838,20cyclictest483-21fschecks_count19:05:081
31015996822,9cyclictest0-21swapper/100:02:071
31015996812,23cyclictest14177-21seq19:45:251
31011996818,24cyclictest15-21ksoftirqd/019:55:280
3101599672,22cyclictest8229-21runrttasks21:03:461
3101599672,22cyclictest8229-21runrttasks21:03:461
31015996715,19cyclictest12212-21runrttasks21:10:071
31015996713,20cyclictest10241-21seq19:32:381
3101199672,39cyclictest15-21ksoftirqd/019:10:200
31015996620,10cyclictest0-21swapper/122:17:201
31015996612,20cyclictest352-21runrttasks20:30:411
31015996612,19cyclictest27286-21taskset23:22:091
31015996610,26cyclictest0-21swapper/120:46:081
3101599659,26cyclictest352-21runrttasks21:58:171
31015996511,30cyclictest27955-21latency_hist22:30:031
3101599647,20cyclictest2098-21sh21:46:111
31015996321,8cyclictest0-21swapper/121:15:081
31024996213,36cyclictest0-21swapper/322:41:213
31024996213,36cyclictest0-21swapper/322:41:213
31015996211,18cyclictest152962ssh23:58:121
3102499605,22cyclictest0-21swapper/322:10:343
31024995843,10cyclictest451rcuc/322:52:163
31024995843,10cyclictest451rcuc/322:52:163
31024995542,8cyclictest451rcuc/320:49:323
31024995542,8cyclictest451rcuc/320:36:493
31024995541,8cyclictest451rcuc/321:26:363
31024995540,10cyclictest451rcuc/319:38:533
31024995538,10cyclictest451rcuc/319:57:233
31024995538,10cyclictest451rcuc/319:49:013
31024995536,14cyclictest451rcuc/321:52:213
31024995441,8cyclictest451rcuc/322:57:393
3102499543,40cyclictest24693-21taskset22:24:593
3102499542,40cyclictest11793-21taskset22:03:063
3102499542,40cyclictest11793-21taskset22:03:063
3102499542,33cyclictest12206-21chrt21:10:063
2981825410,26sleep30-21swapper/318:55:303
31024995339,9cyclictest451rcuc/319:43:313
31024995339,9cyclictest451rcuc/319:12:133
31024995335,10cyclictest451rcuc/322:16:483
31024995324,21cyclictest10822-21chrt23:49:453
3102499532,37cyclictest13920-21taskset23:55:003
31024995239,8cyclictest451rcuc/323:10:563
31024995237,11cyclictest451rcuc/321:02:533
31024995237,11cyclictest451rcuc/321:02:533
31024995237,10cyclictest451rcuc/323:42:083
31024995237,10cyclictest451rcuc/323:03:303
31024995237,10cyclictest451rcuc/300:17:133
3102499523,39cyclictest3688-21chrt20:50:373
3102499523,38cyclictest6820-21chrt21:55:123
3102499523,38cyclictest10531-21taskset21:06:493
3102499523,35cyclictest27527-21taskset22:29:333
3102499523,34cyclictest31798-21chrt23:30:203
31024995136,8cyclictest451rcuc/300:23:323
31024995135,9cyclictest451rcuc/320:13:353
3102499513,39cyclictest5091-21chrt22:45:223
3102499513,35cyclictest29848-21chrt21:39:483
3102499513,35cyclictest29848-21chrt21:39:483
3102499513,33cyclictest29258-21chrt22:30:553
3102499513,33cyclictest29258-21chrt22:30:553
31024995128,15cyclictest451rcuc/321:42:493
31024995037,8cyclictest451rcuc/323:22:433
31024995036,9cyclictest451rcuc/300:14:343
3102499503,35cyclictest26280-21taskset20:20:363
3102499503,35cyclictest26280-21taskset20:20:363
3102499503,33cyclictest32138-21chrt22:35:373
3102499503,33cyclictest32138-21chrt22:35:373
31024994936,9cyclictest451rcuc/321:49:343
31024994935,10cyclictest451rcuc/320:31:383
3102499493,33cyclictest24575-21chrt23:17:523
3102499493,33cyclictest24575-21chrt23:17:523
3102499493,31cyclictest22195-21chrt00:10:033
31024994933,11cyclictest451rcuc/323:06:063
3102499492,35cyclictest18584-21chrt21:20:233
31024994835,9cyclictest451rcuc/320:27:373
31024994835,8cyclictest451rcuc/320:59:083
3102499483,32cyclictest31009-21chrt23:29:183
3102499483,29cyclictest7324-21chrt19:25:043
31024994829,12cyclictest451rcuc/320:03:363
3102499482,30cyclictest4302-21chrt19:15:243
31024994735,8cyclictest451rcuc/319:31:283
3102499473,34cyclictest7619-21chrt19:26:003
3102499473,34cyclictest32222-21taskset20:40:243
31024994732,10cyclictest451rcuc/321:32:283
3102499472,30cyclictest16779-21taskset19:50:393
31024994712,23cyclictest0-21swapper/323:35:243
31024994712,23cyclictest0-21swapper/323:35:243
3102499463,34cyclictest16069-21chrt00:00:023
3102499463,32cyclictest16578-21chrt00:00:183
31024994630,11cyclictest451rcuc/300:25:163
3102499462,33cyclictest17057-21chrt21:17:363
31024994426,11cyclictest451rcuc/320:06:073
3102499442,30cyclictest12940-21chrt22:05:173
3102499442,27cyclictest24372-21chrt20:15:303
3102499442,27cyclictest24372-21chrt20:15:303
31024994125,9cyclictest451rcuc/319:06:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional