You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 17:30
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Wed Nov 05, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2127221166,16sleep00-21swapper/007:08:020
214959910277,18cyclictest9324-21sh11:34:092
214959910277,18cyclictest9324-21sh11:34:092
21492991013,45cyclictest26309-21unixbench_multi10:04:251
2149299993,38cyclictest25649-21munin-run08:58:461
21492999679,11cyclictest995-21tune2fs10:19:091
2148599963,67cyclictest15946-21apt-get08:29:000
21495999572,17cyclictest13723-21apt-key08:23:492
21495999572,17cyclictest13723-21apt-key08:23:492
21492999370,16cyclictest299-21systemd-logind07:58:471
21492999370,16cyclictest299-21systemd-logind07:58:471
21495999262,20cyclictest19258-21apt-key08:38:502
21485999239,43cyclictest0-21swapper/011:49:010
21485999239,43cyclictest0-21swapper/011:49:010
21485999232,24cyclictest15191-21/usr/sbin/munin10:44:410
21495999167,14cyclictest0-21swapper/208:28:492
21485999129,54cyclictest352-21runrttasks09:47:090
21485999127,30cyclictest4188-21/usr/sbin/munin10:24:270
21485999127,30cyclictest4188-21/usr/sbin/munin10:24:270
21492999063,21cyclictest11052-21latency_hist12:33:501
21492999063,21cyclictest11052-21latency_hist12:33:501
21485999033,20cyclictest17712-21taskset08:34:070
21492998960,19cyclictest3005-21unixbench_singl09:19:241
21485998929,27cyclictest10223-21ssh09:34:190
21485998928,22cyclictest8062-21grep08:04:230
207902897,53sleep120883-21timerwakeupswit07:04:331
21485998831,26cyclictest30705-21ssh11:14:080
21485998826,25cyclictest12552-21rm12:35:150
21485998825,28cyclictest0-21swapper/011:59:080
21495998770,10cyclictest300-21in:imuxsock11:23:462
21495998762,15cyclictest0-21swapper/209:49:172
21495998738,31cyclictest9189-21cut08:09:102
21485998751,23cyclictest352-21runrttasks12:29:010
21485998751,23cyclictest352-21runrttasks12:29:010
21485998727,23cyclictest12386-21tr11:39:160
21495998663,12cyclictest0-21swapper/208:14:232
21495998662,12cyclictest0-21swapper/208:58:492
21492998662,18cyclictest8055-21kworker/u17:3+events_unbound12:18:451
21492998659,18cyclictest3801-21sh12:20:321
21492998655,10cyclictest1160-21sshd11:19:091
21495998567,11cyclictest0-21swapper/212:14:112
21495998564,15cyclictest21472-21cyclictest10:14:252
21495998561,17cyclictest22261-21kworker/u18:3+events_unbound11:08:452
21495998561,13cyclictest0-21swapper/208:39:392
21495998561,13cyclictest0-21swapper/208:39:392
21485998527,27cyclictest957-21taskset07:44:010
21485998527,25cyclictest5648-21/usr/sbin/munin12:24:130
21485998525,23cyclictest24441-21rm10:02:000
21485998525,23cyclictest24441-21rm10:02:000
21485998525,23cyclictest23971-21taskset11:01:000
21485998525,23cyclictest23971-21taskset11:01:000
21492998438,37cyclictest17978-21vmstat10:49:261
21492998438,37cyclictest17978-21vmstat10:49:261
21485998429,24cyclictest28495-21/usr/sbin/munin11:09:170
21485998425,29cyclictest27780-21ssh12:05:560
21485998425,26cyclictest25539-21taskset11:04:170
21485998424,27cyclictest6184-21sh11:28:560
21485998422,28cyclictest0-21swapper/011:25:560
21485998421,24cyclictest0-21swapper/007:09:110
21495998358,13cyclictest0-21swapper/211:03:492
21495998358,13cyclictest0-21swapper/211:03:492
21492998358,17cyclictest27746-21timerandwakeup07:24:221
21485998329,19cyclictest25351-21chrt07:18:590
21485998324,24cyclictest28694-21chrt09:06:200
21485998225,28cyclictest6603-21sh09:27:530
21485998223,22cyclictest31486-21chrt07:38:430
2088528220,42sleep220959-21grep07:04:342
2149599819,63cyclictest15448-21apt-get08:29:002
21495998158,13cyclictest0-21swapper/210:09:122
21495998157,12cyclictest0-21swapper/211:28:502
21485998129,22cyclictest32529-21ssh12:14:190
21485998127,21cyclictest14699-21sh11:44:020
21485998124,27cyclictest736-21sh10:18:550
21485998113,33cyclictest14108-21apt-get08:24:030
21495998052,17cyclictest0-21swapper/210:58:492
21492998019,41cyclictest5419-21switchtime09:24:201
21485998028,27cyclictest19405-21rm09:52:240
21485998028,23cyclictest8910-21runrttasks10:34:030
21485998028,23cyclictest8910-21runrttasks10:34:030
21485998025,18cyclictest11633-21/usr/sbin/munin10:39:200
21495997963,9cyclictest0-21swapper/208:53:482
21495997961,8cyclictest0-21swapper/209:59:222
21495997961,8cyclictest0-21swapper/209:59:222
21495997956,11cyclictest0-21swapper/210:48:492
21495997955,14cyclictest0-21swapper/211:18:482
21495997951,19cyclictest28903-21diskmemload12:28:592
21495997951,19cyclictest28903-21diskmemload12:28:592
2149599792,35cyclictest27572-21grep07:24:202
2149299796,33cyclictest17553-21ssh11:49:011
2149299796,33cyclictest17553-21ssh11:49:011
21492997959,13cyclictest11224-21kworker/u17:1-rpciod09:48:491
21492997942,13cyclictest0-21swapper/108:29:401
21485997927,45cyclictest31040-21chrt12:12:110
21485997927,45cyclictest31040-21chrt12:12:110
21485997927,24cyclictest1030-21tune2fs11:19:070
21485997926,23cyclictest24288-21/usr/sbin/munin08:54:070
21485997925,27cyclictest17040-21ssh10:49:060
21485997925,27cyclictest17040-21ssh10:49:060
21485997924,24cyclictest12979-21switchtime09:39:250
21485997924,24cyclictest12979-21switchtime09:39:250
21485997923,19cyclictest7658-21chrt10:30:500
21485997825,25cyclictest3933-21ssh09:22:250
21485997824,24cyclictest29618-21irqstats09:09:130
21495997759,8cyclictest0-21swapper/209:29:182
21492997763,10cyclictest21472-21cyclictest10:29:261
21492997756,13cyclictest21472-21cyclictest12:00:551
21492997747,21cyclictest22886-21cut09:59:091
21492997747,21cyclictest22886-21cut09:59:091
2149299772,43cyclictest13605-21apt-get08:24:001
21485997757,16cyclictest11224-21kworker/u17:1+xprtiod09:57:060
21485997743,25cyclictest23707-21apt-get07:14:010
21485997743,25cyclictest23707-21apt-get07:14:010
21485997728,38cyclictest20863-21latency11:54:120
21485997728,20cyclictest9031-21taskset11:34:020
21485997728,20cyclictest9031-21taskset11:34:020
21485997726,43cyclictest28241-21/usr/sbin/munin10:09:090
21485997724,24cyclictest21044-21chrt10:55:300
21485997724,23cyclictest27427-21sh10:07:440
21485997721,25cyclictest0-21swapper/007:29:110
21485997718,21cyclictest0-21swapper/008:14:170
21495997657,8cyclictest0-21swapper/208:44:042
21495997657,8cyclictest0-21swapper/208:44:042
21495997654,11cyclictest0-21swapper/207:43:502
21492997654,15cyclictest21472-21cyclictest11:12:391
21492997647,19cyclictest5776-21load12:24:151
21485997637,26cyclictest31798-21apt-get07:38:590
21485997627,40cyclictest499-21unin-run10:18:480
21485997627,23cyclictest513-21sh09:15:080
21485997624,18cyclictest2709-21df_inode07:49:070
21485997618,20cyclictest0-21swapper/007:28:490
21495997557,13cyclictest4638-21acpi09:23:562
21495997552,14cyclictest25999-21cpuspeed08:59:022
2149599753,33cyclictest21889-21cut07:09:092
21492997554,12cyclictest14328-21cron11:43:451
21492997548,11cyclictest181rcu_preempt10:38:521
2149299754,21cyclictest14689-21ssh11:44:011
21485997520,21cyclictest0-21swapper/008:39:010
21485997520,21cyclictest0-21swapper/008:39:010
21485997517,19cyclictest0-21swapper/008:59:130
21495997416,25cyclictest2961-21sh10:23:042
21485997424,22cyclictest7080-21taskset08:03:110
21495997351,16cyclictest1087-21kworker/u17:0-rpciod09:19:042
21495997346,18cyclictest13955-21latency_hist12:38:482
21492997352,11cyclictest0-21swapper/110:54:231
2148599733,45cyclictest2332-21acpi12:18:560
21485997320,22cyclictest0-21swapper/009:30:090
21501997213,44cyclictest15963-21chrt08:29:053
21495997254,12cyclictest28903-21diskmemload12:22:532
21495997251,12cyclictest26876-21rm12:04:172
21495997250,16cyclictest11300-21cron10:38:462
21495997250,16cyclictest11300-21cron10:38:462
21495997249,13cyclictest0-21swapper/211:45:172
21495997248,13cyclictest0-21swapper/209:18:492
21495997246,19cyclictest25573-21fschecks_time07:19:102
21495997246,15cyclictest8752-21latency_hist08:08:502
21495997244,22cyclictest2819-21fschecks_time07:49:092
21492997241,13cyclictest181rcu_preempt10:14:031
21485997216,20cyclictest0-21swapper/008:44:250
21485997216,20cyclictest0-21swapper/008:44:250
73902712,23sleep17395-21ssh11:29:411
21495997157,7cyclictest0-21swapper/207:44:122
21495997154,10cyclictest0-21swapper/210:04:212
21495997148,15cyclictest28198-21kernelversion11:09:092
21495997131,11cyclictest181rcu_preempt09:44:102
2149299715,49cyclictest28893-21modprobe09:08:461
21492997150,11cyclictest0-21swapper/108:39:541
21492997150,11cyclictest0-21swapper/108:39:541
21492997146,19cyclictest21472-21cyclictest09:50:301
2149299714,27cyclictest0-21swapper/107:09:221
21492997142,17cyclictest0-21swapper/110:59:031
21492997142,17cyclictest0-21swapper/110:59:031
21492997130,14cyclictest181rcu_preempt09:42:571
21492997130,14cyclictest181rcu_preempt09:42:571
21492997110,36cyclictest0-21swapper/110:44:441
2148599714,42cyclictest5654-21latency_hist07:58:490
2148599714,42cyclictest5654-21latency_hist07:58:490
21495997054,11cyclictest366-21ntpd10:33:482
21495997045,13cyclictest0-21swapper/207:29:082
21492997048,11cyclictest0-21swapper/110:24:231
21492997048,11cyclictest0-21swapper/110:24:231
21492997042,22cyclictest26388-21fschecks_count12:04:051
21495996949,14cyclictest17565-21sh11:49:032
21495996949,14cyclictest17565-21sh11:49:032
21495996947,12cyclictest0-21swapper/210:24:162
21495996947,12cyclictest0-21swapper/210:24:162
21495996945,14cyclictest0-21swapper/212:03:472
21495996922,9cyclictest0-21swapper/211:39:172
21495996911,22cyclictest0-21swapper/212:13:472
21495996911,22cyclictest0-21swapper/212:13:472
2149299694,36cyclictest352-21runrttasks10:10:021
21492996938,23cyclictest3854-21sh11:24:091
21492996933,10cyclictest181rcu_preempt09:34:291
21492996932,13cyclictest181rcu_preempt10:35:201
21492996932,13cyclictest181rcu_preempt10:35:201
21492996925,31cyclictest8095-21cat09:29:271
21495996850,11cyclictest299-21systemd-logind10:43:472
21495996846,16cyclictest28903-21diskmemload09:43:402
21495996846,16cyclictest28903-21diskmemload09:43:402
21495996846,13cyclictest0-21swapper/207:34:282
21495996845,12cyclictest0-21swapper/209:08:492
21495996841,19cyclictest28903-21diskmemload10:51:312
21495996841,19cyclictest28903-21diskmemload10:51:312
2149599683,30cyclictest20410-21grep09:54:092
21492996851,11cyclictest207-21systemd-journal08:48:461
21492996851,11cyclictest207-21systemd-journal08:48:461
21492996846,8cyclictest31-21ksoftirqd/109:55:561
21492996843,15cyclictest0-21swapper/107:39:041
2149299683,41cyclictest27472-21latency_hist09:03:501
21485996829,32cyclictest13181-21grep08:19:240
21485996829,32cyclictest13181-21grep08:19:240
21495996740,17cyclictest21472-21cyclictest09:13:492
21495996740,12cyclictest0-21swapper/208:03:502
2149599673,43cyclictest7880-21latency_hist12:28:492
2149299679,43cyclictest26683-21sh07:23:461
2149299673,37cyclictest6562grep11:18:491
21492996733,25cyclictest18101-21systemctl08:34:181
21492996719,14cyclictest181rcu_preempt12:09:151
21492996719,14cyclictest181rcu_preempt12:09:151
21492996711,39cyclictest31446-21sh09:13:461
21485996730,28cyclictest8997-21/usr/sbin/munin08:09:090
21495996639,19cyclictest9735-21sort09:34:032
2149299667,43cyclictest20394-21/usr/sbin/munin11:54:091
21492996649,11cyclictest13830-21sendmail12:38:461
21492996646,14cyclictest352-21runrttasks09:18:201
2149299664,41cyclictest3133-21grep07:49:191
2149299663,32cyclictest11117-21ntp_kernel_pll_08:14:171
21501996511,34cyclictest14119-21chrt08:24:033
21495996549,10cyclictest207-21systemd-journal11:56:182
21495996547,9cyclictest0-21swapper/207:54:062
21495996547,9cyclictest0-21swapper/207:54:062
2149599653,41cyclictest8889-21latency_hist11:33:492
2149299659,17cyclictest0-21swapper/108:04:261
21492996547,10cyclictest0-21swapper/111:04:131
2149299654,33cyclictest352-21runrttasks07:45:261
21492996536,10cyclictest181rcu_preempt11:34:141
21492996536,10cyclictest181rcu_preempt11:34:141
2149299653,39cyclictest23897-21fschecks_count07:14:111
2149299653,39cyclictest23897-21fschecks_count07:14:111
21492996525,29cyclictest24120-21cat08:53:481
21492996516,27cyclictest13942-21sort08:23:501
21492996516,27cyclictest13942-21sort08:23:501
21485996512,19cyclictest0-21swapper/008:49:060
2149299644,40cyclictest30389-21grep07:34:121
2149299643,41cyclictest10491-21latency_hist08:13:491
2149299643,34cyclictest352-21runrttasks08:02:191
21495996318,10cyclictest0-21swapper/207:14:052
21495996318,10cyclictest0-21swapper/207:14:052
21492996232,8cyclictest0-21swapper/107:29:181
2150199614,38cyclictest23199-21chrt08:49:143
21501996044,8cyclictest91182sleep312:29:253
21501996044,8cyclictest91182sleep312:29:253
2150199584,37cyclictest12431-21chrt11:39:193
2150199584,36cyclictest17768-21taskset09:49:073
2150199573,36cyclictest23363-21taskset11:59:053
2150199572,39cyclictest3838-21taskset10:24:173
2150199572,39cyclictest3838-21taskset10:24:173
2150199564,39cyclictest22439-21chrt08:48:463
2150199564,39cyclictest22439-21chrt08:48:463
2150199563,40cyclictest22064-21taskset10:58:463
2150199563,38cyclictest5374-21chrt12:24:053
2150199563,35cyclictest10759-21taskset08:14:063
2150199562,40cyclictest29076-21taskset07:29:153
2150199562,38cyclictest26516-21chrt11:05:263
2150199553,35cyclictest27811-21chrt09:04:093
2150199553,34cyclictest23201-21taskset10:59:223
2150199553,34cyclictest23201-21taskset10:59:223
2150199553,33cyclictest31290-21chrt11:14:223
2150199553,33cyclictest2482-21taskset12:19:033
2150199552,38cyclictest8733-21chrt10:33:493
2150199552,36cyclictest29620-21chrt11:11:593
2150199543,36cyclictest23476-21chrt07:13:563
2150199543,36cyclictest23476-21chrt07:13:563
2150199543,36cyclictest14009-21chrt12:38:493
2150199543,35cyclictest18128-21chrt11:49:163
2150199543,35cyclictest18128-21chrt11:49:163
2150199543,34cyclictest15637-21taskset11:44:253
2150199542,37cyclictest6613-21chrt07:59:203
2150199533,38cyclictest30290-21chrt12:09:553
2150199533,38cyclictest30290-21chrt12:09:553
2150199533,36cyclictest11492-21chrt11:38:463
2150199533,36cyclictest11492-21chrt11:38:463
2150199533,33cyclictest1704-21taskset07:44:223
2150199533,32cyclictest7341-21chrt09:29:093
2150199532,40cyclictest30780-21taskset09:11:103
2150199532,38cyclictest8020-21chrt08:04:223
2150199532,35cyclictest20970-21chrt11:54:143
2150199523,33cyclictest3611-21chrt11:24:033
2150199522,35cyclictest9600-21taskset10:34:223
2150199522,35cyclictest9600-21taskset10:34:223
2150199522,35cyclictest16673-21chrt10:48:493
2150199522,34cyclictest14119-21chrt09:42:183
2150199522,34cyclictest14119-21chrt09:42:183
2150199522,33cyclictest20124-21chrt09:53:573
2150199522,32cyclictest27023-21chrt12:04:203
2150199513,35cyclictest29980-21chrt10:12:503
2150199512,31cyclictest1076-21chrt10:19:103
2150199503,31cyclictest12886-21taskset08:19:203
2150199503,31cyclictest12886-21taskset08:19:203
2150199502,37cyclictest19259-21chrt08:38:493
2150199502,37cyclictest10976-21chrt09:35:333
2150199502,32cyclictest27091-21chrt07:24:053
2150199493,33cyclictest17093-21chrt10:49:093
2150199493,33cyclictest17093-21chrt10:49:093
2150199493,32cyclictest27722-21chrt10:08:483
2150199493,30cyclictest3400-21chrt09:20:383
2150199493,28cyclictest22728-21chrt09:59:043
2150199493,28cyclictest22728-21chrt09:59:043
2150199487,27cyclictest30605-21chrt10:13:593
2150199482,33cyclictest31535-21chrt07:38:473
2150199482,33cyclictest2838-21chrt11:22:413
2150199482,33cyclictest1689-21taskset09:18:473
2150199473,31cyclictest10340-21chrt08:13:463
21501994713,23cyclictest3977-21chrt07:53:493
21501994712,22cyclictest4758-21chrt09:24:023
2150199463,29cyclictest32607-21chrt12:14:213
21501994612,22cyclictest19661-21taskset08:39:353
21501994612,22cyclictest19661-21taskset08:39:353
21501994611,24cyclictest22491-21taskset07:09:223
21501994515,19cyclictest11602-21chrt10:38:513
21501994513,23cyclictest4717-21chrt07:54:223
21501994513,23cyclictest4717-21chrt07:54:223
21501994312,21cyclictest16621-21chrt09:46:523
21501994312,19cyclictest7423-21chrt11:29:463
21501994311,22cyclictest336-21chrt07:39:363
2150199423,27cyclictest25354-21taskset07:19:023
21501994213,19cyclictest26619-21chrt08:59:193
21501994211,21cyclictest25530-21chrt08:57:283
205592418,18sleep30-21swapper/307:04:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional