You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 20:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Nov 25, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
91799910830,44cyclictest0-21swapper/111:39:431
893621077,80sleep00-21swapper/007:08:060
2294721070,5sleep222949-21cut07:49:312
827221038,62sleep18321-21processes07:04:531
9168999573,15cyclictest32113-21systemctl08:19:090
9168999522,31cyclictest0-21swapper/010:29:220
9168999522,31cyclictest0-21swapper/010:29:220
917999943,42cyclictest9415-21sort07:09:261
917999933,28cyclictest24878-21cron10:24:081
916899928,73cyclictest7629-21ssh11:49:440
9168999244,37cyclictest9324-21apt-get10:54:220
9168999165,17cyclictest14450-21ssh11:04:110
9168999148,35cyclictest1809-21apt-get11:39:220
917999903,31cyclictest18302-21grep11:09:431
917999903,31cyclictest18302-21grep11:09:431
917999893,26cyclictest8605-21cpu09:54:251
917999892,42cyclictest29150-21apt-get08:09:211
917999883,40cyclictest22567-21unixbench_multi12:14:461
917999883,33cyclictest28487-21grep12:25:031
9179998810,10cyclictest0-21swapper/110:54:131
9179998810,10cyclictest0-21swapper/110:54:131
9168998821,30cyclictest22690-21apt-get07:49:240
9179998759,18cyclictest12421-21sh11:58:131
9179998759,18cyclictest12421-21sh11:58:131
917999874,32cyclictest23751-21runrttasks11:19:421
9179998742,39cyclictest19425-21fschecks_count09:14:311
917999873,30cyclictest25198-21apt-get10:24:191
9179998714,36cyclictest355-21runrttasks08:45:021
9179998714,36cyclictest355-21runrttasks08:45:021
9168998721,30cyclictest6405-21apt-get08:39:270
9168998720,32cyclictest27426-21apt-get09:29:360
9179998640,41cyclictest0-21swapper/110:59:451
9179998640,39cyclictest0-21swapper/109:49:121
9179998640,39cyclictest0-21swapper/109:49:121
9179998638,39cyclictest0-21swapper/110:34:271
917999863,30cyclictest1241-21/usr/sbin/munin12:35:041
917999863,30cyclictest1241-21/usr/sbin/munin12:35:041
917999863,25cyclictest15820-21/usr/sbin/munin12:04:411
9179998626,52cyclictest13781-21cut08:59:451
917999862,32cyclictest11303-21timerwakeupswit07:14:221
917999862,31cyclictest22175-21ls10:19:121
9168998657,19cyclictest27747-21fschecks_count08:04:320
9168998619,33cyclictest355-21runrttasks10:49:100
9168998618,24cyclictest26519-21ps11:24:420
9168998618,24cyclictest26519-21ps11:24:420
9168998612,38cyclictest19268-21munin-run10:14:090
918999853,60cyclictest847-21apt-get12:34:262
918999853,60cyclictest847-21apt-get12:34:262
917999856,33cyclictest355-21runrttasks09:40:151
917999854,40cyclictest8204-21latency11:50:231
9179998536,40cyclictest0-21swapper/109:24:361
9179998536,40cyclictest0-21swapper/109:24:361
917999853,34cyclictest29326-21processes11:29:411
917999853,30cyclictest30024-21cat09:34:121
917999853,29cyclictest4501-21cron08:34:091
917999853,27cyclictest13258-21fschecks_time07:19:311
917999853,26cyclictest17392-21/usr/sbin/munin10:09:391
9168998526,25cyclictest0-21swapper/010:34:470
9168998515,30cyclictest736-21/usr/sbin/munin08:19:460
9168998510,43cyclictest26280-21rm09:27:150
9168998510,43cyclictest26280-21rm09:27:150
917999844,38cyclictest13058-21/usr/sbin/munin11:59:311
917999843,40cyclictest31160-21timerwakeupswit12:29:461
917999843,40cyclictest19465-21acpi07:39:211
917999843,30cyclictest24497-21fschecks_count07:54:321
917999843,29cyclictest15492-21ntp_kernel_pll_09:04:521
917999843,29cyclictest15492-21ntp_kernel_pll_09:04:521
917999842,29cyclictest22677-21taskset09:19:451
917999842,29cyclictest22677-21taskset09:19:451
9179998417,28cyclictest25862-21awk11:24:211
9179998417,28cyclictest25862-21awk11:24:211
916899845,59cyclictest959-21modprobe12:34:390
916899845,59cyclictest959-21modprobe12:34:390
9168998422,23cyclictest25223-21ssh10:24:230
9168998421,31cyclictest28893-21tune2fs11:29:300
9168998416,25cyclictest9597-21latency_hist08:49:110
9168998416,25cyclictest9597-21latency_hist08:49:110
9189998360,16cyclictest691-21cron10:29:092
9189998360,16cyclictest10972-21sendmail09:59:092
9189998358,19cyclictest31717-21unixbench_multi10:34:472
9189998353,21cyclictest15667-21latency_hist12:04:112
917999834,29cyclictest27827-21/usr/sbin/munin08:04:331
917999833,39cyclictest26558-21sendmail_mailst07:59:471
917999833,33cyclictest17642-21unixbench_multi09:09:471
917999833,33cyclictest17642-21unixbench_multi09:09:471
917999833,30cyclictest16707-21grep10:09:121
917999833,29cyclictest1068-21/usr/sbin/munin10:39:261
917999832,31cyclictest6325-21fschecks_count09:50:041
917999832,31cyclictest6325-21fschecks_count09:50:041
9168998322,26cyclictest3686-21runrttasks08:29:380
9168998321,25cyclictest17408-21/usr/sbin/munin09:09:440
9168998321,25cyclictest17408-21/usr/sbin/munin09:09:440
786428324,33sleep20-21swapper/207:04:372
9195998210,56cyclictest0-21swapper/312:34:383
9195998210,56cyclictest0-21swapper/312:34:383
9189998257,16cyclictest9107-21latency_hist10:54:122
9189998257,16cyclictest9107-21latency_hist10:54:122
918999823,69cyclictest27478-21ssh09:29:292
918999822,30cyclictest20866-21latency_hist07:44:112
917999824,39cyclictest6927-21ssh11:49:031
917999824,39cyclictest6927-21ssh11:49:031
917999824,36cyclictest11764-21idleruntime08:54:331
917999823,38cyclictest15584-21unixbench_singl07:24:471
917999823,37cyclictest17091-21timerwakeupswit07:29:461
917999823,32cyclictest24027-21latency_hist07:54:121
917999823,31cyclictest24683-21/usr/sbin/munin12:19:581
917999823,31cyclictest20289-21/usr/sbin/munin11:14:291
917999823,29cyclictest18125-21fschecks_time07:34:331
917999823,29cyclictest18125-21fschecks_time07:34:331
917999823,28cyclictest5685-21unixbench_multi08:34:471
917999822,30cyclictest10442-21processes08:49:411
917999822,29cyclictest30879-21mailstats09:34:461
9168998226,16cyclictest19580-21apt-get10:14:260
9168998223,24cyclictest14265-21/usr/sbin/munin10:04:310
9168998219,25cyclictest26382-21seq07:59:410
9168998215,32cyclictest0-21swapper/009:19:210
9168998215,32cyclictest0-21swapper/009:19:210
9168998213,28cyclictest0-21swapper/010:49:340
9168998213,28cyclictest0-21swapper/010:49:340
917999814,31cyclictest355-21runrttasks11:08:081
917999813,31cyclictest11710-21latency_hist10:59:111
917999813,27cyclictest31511-21cstates11:34:261
917999813,26cyclictest2031-21awk08:24:371
917999812,30cyclictest21438-21latency_hist12:14:121
9168998128,23cyclictest23092-21mailstats10:19:420
9168998125,26cyclictest0-21swapper/007:09:450
9168998113,25cyclictest13880-21timerwakeupswit07:19:440
9168998112,27cyclictest18064-21ntpq11:09:370
9168998112,27cyclictest18064-21ntpq11:09:370
918999807,61cyclictest7702-21ssh11:49:582
9189998060,14cyclictest22674-21timerandwakeup09:19:452
9189998060,14cyclictest22674-21timerandwakeup09:19:452
9189998055,16cyclictest1-21systemd12:34:102
917999803,39cyclictest3989-21ssh10:44:261
917999803,32cyclictest11284-21/usr/sbin/munin09:59:441
917999803,32cyclictest11284-21/usr/sbin/munin09:59:441
917999803,30cyclictest21497-21grep07:44:341
917999802,37cyclictest7308-21grep08:39:481
917999802,30cyclictest32670-21fschecks_time08:19:341
917999802,30cyclictest30756-21apt-get08:14:211
916899808,30cyclictest0-21swapper/008:54:090
916899808,25cyclictest0-21swapper/009:54:120
916899808,25cyclictest0-21swapper/009:54:120
916899803,41cyclictest21669-21latency_hist09:19:120
9168998023,34cyclictest6243-21cat08:39:120
9168998020,20cyclictest12452-21who08:54:470
9168998018,40cyclictest13086-21/usr/sbin/munin08:59:320
9168998015,29cyclictest9248-21/usr/sbin/munin09:54:410
9168998013,26cyclictest24611-21latency_hist12:19:130
9168998011,30cyclictest22473-21latency_hist07:49:110
9168998011,29cyclictest207-21systemd-journal11:35:580
9189997956,15cyclictest10945-21munin-node12:14:182
917999793,30cyclictest30622-21latency_hist10:34:131
917999793,30cyclictest30622-21latency_hist10:34:131
916899798,30cyclictest16123-21cat07:29:130
9168997959,14cyclictest13845-21latency_hist10:04:120
9168997959,14cyclictest13845-21latency_hist10:04:120
9168997946,26cyclictest0-21swapper/009:39:090
9168997929,21cyclictest1130-21timerandwakeup09:39:460
9168997919,22cyclictest26358-21ssh12:21:330
9168997913,24cyclictest13661-21ntpq11:59:400
916899787,35cyclictest16100-21fschecks_time12:04:300
9168997854,18cyclictest4555-21/usr/sbin/munin11:44:260
9168997854,18cyclictest4555-21/usr/sbin/munin11:44:260
9168997838,30cyclictest0-21swapper/012:30:200
9168997821,23cyclictest19106-21/usr/sbin/munin12:09:330
9168997816,23cyclictest0-21swapper/008:09:450
9168997815,27cyclictest12111-21sshd07:14:470
9168997813,25cyclictest14958-21dpkg09:04:210
9168997813,25cyclictest14958-21dpkg09:04:210
9168997812,29cyclictest22998-21cat11:19:120
9168997810,28cyclictest0-21swapper/012:24:260
9189997747,21cyclictest22663-21fschecks_time10:19:312
9168997714,43cyclictest0-21swapper/007:54:210
9168997714,30cyclictest23226-21awk11:19:270
9168997713,28cyclictest12998-21date11:59:130
9168997711,43cyclictest17101-21rm11:08:520
9189997647,10cyclictest181rcu_preempt10:15:132
916899769,25cyclictest1088-21apt-get10:39:260
9168997612,20cyclictest10849-21grep11:54:390
9168997610,28cyclictest1687-21cstates08:24:280
9189997559,11cyclictest9157-21cyclictest12:14:082
916899759,44cyclictest207-21systemd-journal07:35:470
916899759,44cyclictest207-21systemd-journal07:35:470
916899752,25cyclictest3480-21memory09:44:370
916899752,25cyclictest3480-21memory09:44:370
9189997458,9cyclictest0-21swapper/211:24:132
9189997448,20cyclictest12886-21latency_hist07:19:122
9189997444,12cyclictest181rcu_preempt09:39:462
9168997410,31cyclictest17763-21cat07:34:130
9189997355,11cyclictest17133-21runrttasks11:08:562
9189997330,25cyclictest13251-21fschecks_time07:19:312
918999732,59cyclictest24712-21sh12:19:242
9189997249,8cyclictest181rcu_preempt09:19:102
9189997212,35cyclictest26783-21wc11:24:462
9189997212,35cyclictest26783-21wc11:24:462
916899727,28cyclictest19801-21cat07:39:340
9189997154,11cyclictest17499-21/usr/sbin/munin11:09:462
9189997154,11cyclictest17499-21/usr/sbin/munin11:09:462
9189997152,12cyclictest0-21swapper/209:24:322
9189997152,12cyclictest0-21swapper/209:24:322
9189997151,12cyclictest10932-21processes11:54:432
9189997151,12cyclictest10932-21processes11:54:432
9189997148,15cyclictest32415-21sh09:39:092
9189997147,16cyclictest207-21systemd-journal09:09:112
9189997147,16cyclictest207-21systemd-journal09:09:112
9189997133,10cyclictest181rcu_preempt08:34:092
9189997132,13cyclictest181rcu_preempt09:49:122
9189997132,13cyclictest181rcu_preempt09:49:122
9189997112,20cyclictest28684-21sshd08:06:462
9189997055,10cyclictest9157-21cyclictest12:25:162
9189997055,10cyclictest32166-21switchtime11:34:432
9189997051,13cyclictest32602-21fschecks_count08:19:332
9189997051,11cyclictest0-21swapper/208:59:132
918999703,45cyclictest1149-21cpu10:39:262
9189997034,14cyclictest181rcu_preempt08:36:592
918999703,13cyclictest16651-21ssh09:09:252
918999703,13cyclictest16651-21ssh09:09:252
9189997020,26cyclictest0-21swapper/210:04:382
9189996946,17cyclictest28847-21sh11:29:272
9189996946,11cyclictest0-21swapper/207:39:122
9189996946,11cyclictest0-21swapper/207:39:122
9189996943,16cyclictest0-21swapper/208:14:462
918999691,43cyclictest11367-21latency_hist08:54:122
9189996911,15cyclictest7746-21awk08:44:092
9189996840,12cyclictest181rcu_preempt07:54:242
9189996832,13cyclictest181rcu_preempt10:59:092
918999682,52cyclictest13743-21modprobe10:04:092
918999682,52cyclictest13743-21modprobe10:04:092
918999682,45cyclictest4873-21fschecks_time11:44:322
918999682,45cyclictest4873-21fschecks_time11:44:322
918999672,22cyclictest6990-21/usr/sbin/munin09:50:192
918999672,22cyclictest6990-21/usr/sbin/munin09:50:192
203152670,5sleep220314-21ssh11:14:212
9189996651,9cyclictest0-21swapper/212:04:402
9189996644,15cyclictest0-21swapper/210:44:282
9189996642,9cyclictest181rcu_preempt07:29:452
9189996638,20cyclictest27266-21modprobe08:04:092
918999663,44cyclictest355-21runrttasks07:24:442
918999663,43cyclictest2082-21fschecks_time11:39:312
9189996632,15cyclictest181rcu_preempt10:29:132
9189996632,15cyclictest181rcu_preempt10:29:132
918999662,25cyclictest30626-21latency_hist08:14:122
88972667,43sleep30-21swapper/307:07:363
9189996543,15cyclictest10697-21idleruntime-cro07:14:092
918999653,51cyclictest19243-21ssh10:14:092
918999652,44cyclictest8492-21fschecks_count08:45:192
918999652,44cyclictest8492-21fschecks_count08:45:192
919599643,35cyclictest0-21swapper/309:49:333
919599643,35cyclictest0-21swapper/309:49:333
9189996444,14cyclictest22033-21head07:44:452
9189996441,12cyclictest0-21swapper/208:59:482
9189996437,9cyclictest13101-21rm10:59:552
919599635,47cyclictest0-21swapper/311:49:593
918999633,40cyclictest355-21runrttasks08:26:582
919599614,41cyclictest27061-21chrt09:29:113
919599614,41cyclictest27061-21chrt09:29:113
919599603,39cyclictest5788-21taskset09:49:143
919599603,39cyclictest5788-21taskset09:49:143
919599603,38cyclictest0-21swapper/312:19:243
919599563,39cyclictest25981-21taskset11:24:273
919599563,39cyclictest25981-21taskset11:24:273
919599563,39cyclictest10263-21taskset07:09:473
919599563,38cyclictest7063-21chrt10:49:353
919599563,38cyclictest7063-21chrt10:49:353
919599553,36cyclictest26638-21chrt07:59:483
919599553,36cyclictest22256-21chrt10:19:133
919599552,36cyclictest24506-21taskset12:19:123
919599549,34cyclictest0-21swapper/309:29:253
919599544,35cyclictest21260-21taskset07:44:293
919599533,39cyclictest12196-21chrt08:54:443
919599533,38cyclictest19116-21chrt09:14:123
919599533,38cyclictest19116-21chrt09:14:123
919599533,38cyclictest17232-21taskset12:05:393
919599533,36cyclictest1925-21taskset11:39:263
919599533,36cyclictest10644-21taskset09:58:063
919599533,33cyclictest24342-21chrt09:24:133
919599533,33cyclictest24342-21chrt09:24:133
919599532,37cyclictest3737-21taskset10:44:123
919599524,38cyclictest5217-21taskset10:45:463
919599523,37cyclictest1262-21chrt11:38:333
919599523,34cyclictest27893-21chrt12:24:353
919599523,32cyclictest30671-21taskset09:34:383
919599513,39cyclictest895-21kworker/3:1+events10:54:473
919599513,36cyclictest29771-21taskset11:29:503
919599513,36cyclictest16686-21chrt10:09:123
919599513,36cyclictest12212-21chrt10:59:323
919599513,35cyclictest19510-21chrt07:39:273
919599513,34cyclictest20033-21chrt09:14:453
919599512,35cyclictest17333-21taskset11:09:123
919599503,37cyclictest6613-21taskset08:39:333
919599503,34cyclictest7028-21taskset11:49:103
919599503,34cyclictest7028-21taskset11:49:103
919599503,33cyclictest26556-21chrt10:25:373
919599502,35cyclictest20620-21chrt12:12:123
919599502,34cyclictest28465-21chrt08:04:463
919599502,33cyclictest32696-21chrt12:34:033
919599493,35cyclictest29332-21chrt10:30:353
919599493,35cyclictest29332-21chrt10:30:353
919599493,33cyclictest6316-21chrt08:39:133
919599493,32cyclictest11909-21chrt07:14:413
919599492,33cyclictest24790-21taskset11:21:443
919599492,32cyclictest11265-21chrt08:54:113
9195994832,9cyclictest451rcuc/310:00:333
9195994832,9cyclictest451rcuc/310:00:333
919599482,34cyclictest13758-21chrt11:59:443
919599482,33cyclictest22689-21chrt07:49:193
919599482,32cyclictest2369-21chrt09:43:103
919599482,31cyclictest32710-21chrt10:37:363
919599473,35cyclictest19133-21taskset11:11:273
919599473,35cyclictest19133-21taskset11:11:273
919599472,33cyclictest15417-21chrt07:24:453
919599472,31cyclictest29966-21chrt08:09:443
919599464,32cyclictest24296-21chrt07:54:213
9195994634,8cyclictest451rcuc/308:44:363
9195994634,8cyclictest451rcuc/308:44:363
919599463,34cyclictest3915-21chrt08:29:453
919599463,33cyclictest21685-21chrt11:15:403
919599463,29cyclictest18617-21chrt07:34:453
919599463,29cyclictest18617-21chrt07:34:453
919599455,29cyclictest0-21swapper/310:11:563
919599453,28cyclictest24270-21chrt10:22:173
919599452,28cyclictest17720-21chrt07:34:133
919599444,31cyclictest934-21taskset08:19:493
919599444,29cyclictest16323-21chrt09:09:103
919599444,29cyclictest16323-21chrt09:09:103
919599443,30cyclictest13472-21taskset07:19:353
919599422,31cyclictest14573-21chrt09:04:113
919599422,26cyclictest12133-21taskset11:57:173
919599422,26cyclictest12133-21taskset11:57:173
919599393,24cyclictest1532-21taskset08:24:143
919599383,26cyclictest30824-21chrt08:14:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional