You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 18:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Wed Dec 03, 2025 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1451721309,91sleep10-21swapper/107:04:201
15402991097,62cyclictest14210-21apt-get11:53:511
154109910753,19cyclictest314-21in:imuxsock10:43:382
14501210711,70sleep014516-21/usr/sbin/munin07:04:190
15402999648,12cyclictest181rcu_preempt10:04:081
1541099943,82cyclictest31354-21sed11:24:152
15402999444,15cyclictest0-21swapper/107:23:531
1541099903,68cyclictest31891-21apt-get07:59:032
1539899902,47cyclictest15-21ksoftirqd/008:20:250
15402998950,9cyclictest181rcu_preempt10:42:581
15402998950,8cyclictest181rcu_preempt08:28:531
15402998950,8cyclictest181rcu_preempt08:28:531
1539899892,54cyclictest15-21ksoftirqd/008:54:060
1534228912,47sleep239-21ksoftirqd/207:08:412
1540299882,61cyclictest355-21runrttasks10:09:301
15402998754,9cyclictest181rcu_preempt11:16:081
15402998656,12cyclictest181rcu_preempt09:33:571
15398998645,8cyclictest181rcu_preempt11:20:080
15410998569,10cyclictest27317-21/usr/sbin/munin12:19:172
15410998554,22cyclictest19300-21modprobe10:03:372
15410998554,22cyclictest19300-21modprobe10:03:372
15402998540,12cyclictest181rcu_preempt12:34:201
15398998524,50cyclictest13662-21apt-get11:53:520
1541099843,59cyclictest5036-21apt-get08:13:562
15402998457,19cyclictest22874-21latency07:29:041
15402998441,16cyclictest181rcu_preempt11:30:301
15402998441,16cyclictest181rcu_preempt11:30:301
15398998417,54cyclictest31137-21/usr/sbin/munin11:24:120
1541099835,54cyclictest22407-21apt-get10:08:542
1539899833,48cyclictest15-21ksoftirqd/010:04:080
15410998267,10cyclictest15389-21cyclictest11:38:392
15410998238,35cyclictest7541-21grep08:19:162
15402998255,18cyclictest22803-21diskmemload09:11:541
15402998254,18cyclictest31080-21latency_hist09:23:411
15402998247,12cyclictest181rcu_preempt12:03:521
15402998243,9cyclictest181rcu_preempt11:23:371
15402998238,20cyclictest181rcu_preempt09:54:031
15402998233,12cyclictest181rcu_preempt10:24:121
1539899823,47cyclictest15-21ksoftirqd/010:39:000
15410998122,25cyclictest17732-21sh10:59:152
15410998121,28cyclictest0-21swapper/210:34:042
15402998155,10cyclictest181rcu_preempt11:48:531
15402998149,13cyclictest181rcu_preempt10:18:411
15402998149,13cyclictest181rcu_preempt10:18:411
15398998136,12cyclictest181rcu_preempt08:32:250
15398998136,12cyclictest181rcu_preempt08:32:250
15402998057,16cyclictest0-21swapper/111:24:081
15402998044,13cyclictest181rcu_preempt11:34:391
15398998046,23cyclictest12536-21cat08:34:130
15410997963,11cyclictest22803-21diskmemload11:55:292
15410997960,13cyclictest15389-21cyclictest11:04:112
15402997954,17cyclictest4649-21apt-key08:13:401
15402997951,10cyclictest181rcu_preempt12:19:031
15402997927,11cyclictest181rcu_preempt10:34:151
15398997959,11cyclictest22803-21diskmemload12:27:280
1539899794,42cyclictest15-21ksoftirqd/010:01:390
1539899794,42cyclictest15-21ksoftirqd/010:01:390
15398997937,25cyclictest31412-21snmpd11:13:400
15402997849,20cyclictest19422-21latency_hist10:03:401
15402997849,20cyclictest19422-21latency_hist10:03:401
15402997848,14cyclictest8658-21ssh10:43:521
15402997839,8cyclictest181rcu_preempt09:53:391
1540299782,37cyclictest31-21ksoftirqd/109:39:091
1539899784,50cyclictest15-21ksoftirqd/009:33:570
15398997839,13cyclictest181rcu_preempt08:49:080
1539899783,47cyclictest15-21ksoftirqd/007:34:090
1541099774,50cyclictest3836-21latency_hist09:33:412
1541099774,50cyclictest3836-21latency_hist09:33:412
15402997747,11cyclictest181rcu_preempt12:28:421
15402997744,9cyclictest181rcu_preempt08:48:401
15402997738,7cyclictest181rcu_preempt12:29:151
15402997727,12cyclictest181rcu_preempt12:04:021
1540299772,44cyclictest19527-21fschecks_time07:19:011
1540299772,44cyclictest19527-21fschecks_time07:19:011
1539899778,36cyclictest15-21ksoftirqd/008:02:220
15398997756,15cyclictest9371-21sh09:44:040
15398997756,15cyclictest9371-21sh09:44:040
15398997756,13cyclictest11710-21/usr/sbin/munin09:48:590
15398997740,8cyclictest181rcu_preempt09:31:220
15398997740,8cyclictest181rcu_preempt09:31:220
15398997739,17cyclictest181rcu_preempt08:25:170
15398997739,17cyclictest181rcu_preempt08:25:170
15398997737,16cyclictest181rcu_preempt12:00:360
1539899772,40cyclictest15-21ksoftirqd/010:10:280
15410997656,14cyclictest864-21cstates11:28:562
15410997656,14cyclictest864-21cstates11:28:562
15410997653,15cyclictest15389-21cyclictest10:53:412
15402997648,9cyclictest181rcu_preempt07:48:551
15402997646,9cyclictest181rcu_preempt09:03:531
15402997642,13cyclictest181rcu_preempt07:38:411
15402997623,29cyclictest0-21swapper/108:48:491
15398997656,7cyclictest181rcu_preempt10:14:300
15398997656,7cyclictest181rcu_preempt10:14:300
15398997640,29cyclictest0-21swapper/009:14:180
15410997553,12cyclictest0-21swapper/212:13:372
15410997552,16cyclictest15389-21cyclictest12:03:512
15410997550,16cyclictest1-21systemd11:18:382
15410997540,26cyclictest0-21swapper/212:29:102
15410997523,19cyclictest15332-21sh10:54:582
15402997546,8cyclictest181rcu_preempt11:03:151
15402997542,15cyclictest30530-21/usr/sbin/munin07:54:101
15402997540,8cyclictest181rcu_preempt09:26:421
15402997539,8cyclictest181rcu_preempt10:19:481
15402997539,8cyclictest181rcu_preempt07:59:481
15402997528,33cyclictest5078-21dpkg08:14:131
15402997519,18cyclictest0-21swapper/107:44:041
15398997548,20cyclictest0-21swapper/011:18:410
1539899754,55cyclictest355-21runrttasks12:31:190
15398997540,10cyclictest181rcu_preempt09:41:310
15410997453,14cyclictest15389-21cyclictest08:38:292
15410997449,14cyclictest0-21swapper/212:33:562
1541099743,49cyclictest16515-21latency_hist08:48:412
15410997422,24cyclictest0-21swapper/210:23:382
15402997445,12cyclictest181rcu_preempt10:58:411
15402997442,13cyclictest181rcu_preempt11:04:061
15402997429,12cyclictest181rcu_preempt08:25:061
15402997429,12cyclictest181rcu_preempt08:25:061
1540299742,36cyclictest8328-21apt-config08:23:411
15402997338,12cyclictest181rcu_preempt08:54:031
15398997354,14cyclictest16592-21latency_hist10:58:400
1539899734,41cyclictest15-21ksoftirqd/007:14:040
1539899734,41cyclictest15-21ksoftirqd/007:14:040
15398997343,12cyclictest181rcu_preempt10:31:160
15398997341,23cyclictest16943-21cron07:13:380
15398997336,20cyclictest176-21jbd2/mmcblk2p2-812:12:400
1539899733,57cyclictest28807-21latency_hist07:48:410
15398997334,9cyclictest181rcu_preempt09:23:580
1539899732,42cyclictest15-21ksoftirqd/009:58:400
1539899732,26cyclictest27730-21ssh10:18:550
15410997255,10cyclictest18550-21kernelversion08:54:032
15410997242,22cyclictest2924-21latency_hist08:08:372
1541099722,58cyclictest25857-21ssh09:13:512
15402997254,12cyclictest10356-21ssh09:45:271
15402997254,12cyclictest10356-21ssh09:45:271
15402997240,15cyclictest181rcu_preempt12:18:101
15402997240,15cyclictest181rcu_preempt12:18:101
15402997233,8cyclictest181rcu_preempt07:39:001
1540299723,21cyclictest1671-21fschecks_time09:29:001
1540299723,21cyclictest1671-21fschecks_time09:29:001
15402997228,16cyclictest181rcu_preempt07:11:521
15398997256,9cyclictest17639-21ssh10:59:140
15398997251,14cyclictest22387-21latency_hist07:28:410
15398997237,17cyclictest181rcu_preempt08:40:390
1539899723,27cyclictest15-21ksoftirqd/007:22:140
1539899723,27cyclictest15-21ksoftirqd/007:22:140
15398997231,23cyclictest15-21ksoftirqd/007:40:460
15398997227,13cyclictest0-21swapper/011:43:570
1541699715,49cyclictest0-21swapper/310:08:573
1541099716,33cyclictest0-21swapper/210:49:002
15410997150,14cyclictest207-21systemd-journal12:18:432
15410997150,14cyclictest207-21systemd-journal12:18:432
15410997140,24cyclictest0-21swapper/212:04:082
15410997136,25cyclictest0-21swapper/210:24:142
15410997135,13cyclictest181rcu_preempt11:48:582
1541099713,43cyclictest28837-21latency_hist07:48:412
1541099712,49cyclictest3812-21sendmail_mailqu08:09:112
15402997136,9cyclictest181rcu_preempt11:09:011
1540299713,45cyclictest11863-21latency10:49:031
1540299713,45cyclictest11863-21latency10:49:031
15402997128,8cyclictest181rcu_preempt11:45:071
15402997121,23cyclictest0-21swapper/107:17:371
15402997121,23cyclictest0-21swapper/107:17:371
15398997156,9cyclictest21332-21ssh12:07:130
1539899715,37cyclictest15-21ksoftirqd/012:35:170
1539899715,37cyclictest15-21ksoftirqd/007:48:570
15398997145,7cyclictest181rcu_preempt11:05:130
15398997136,13cyclictest181rcu_preempt11:34:010
15398997135,14cyclictest181rcu_preempt08:46:060
15398997131,6cyclictest181rcu_preempt11:33:250
15398997131,6cyclictest181rcu_preempt11:33:250
1539899712,42cyclictest15-21ksoftirqd/012:13:440
1539899712,42cyclictest15-21ksoftirqd/012:13:440
15410997053,11cyclictest20200-21ntp_offset10:04:072
15410997049,15cyclictest355-21runrttasks09:53:562
15410997049,13cyclictest6943-21ssh11:39:132
1541099704,44cyclictest31889-21latency_hist07:58:382
15410997041,20cyclictest792-21fschecks_count10:29:002
15410997015,22cyclictest30773-21sh09:23:212
15402997042,8cyclictest181rcu_preempt09:14:151
15402997041,13cyclictest181rcu_preempt08:59:021
15402997041,13cyclictest181rcu_preempt08:59:021
15402997027,14cyclictest181rcu_preempt10:29:061
15398997055,9cyclictest29127-21ntp_offset09:19:070
15398997050,11cyclictest30246-21ssh10:23:490
15398997037,11cyclictest181rcu_preempt07:33:140
1539899703,49cyclictest15-21ksoftirqd/009:04:110
15398997024,7cyclictest0-21swapper/007:53:590
1539899702,35cyclictest23880-21grep09:09:130
15398997018,31cyclictest15-21ksoftirqd/008:04:140
15410996955,10cyclictest15389-21cyclictest07:53:412
15410996947,16cyclictest18436-21threads07:14:462
15410996947,16cyclictest18436-21threads07:14:462
15410996946,17cyclictest309-21systemd-logind11:48:382
1541099694,43cyclictest18037-21latency_hist08:53:402
15410996942,18cyclictest27229-21latency_hist07:43:412
15410996940,20cyclictest27558-21/usr/sbin/munin11:18:562
15410996923,37cyclictest14091-21latency_hist09:53:402
15410996916,26cyclictest8538-21sh09:43:212
15402996950,13cyclictest15389-21cyclictest12:12:491
15402996942,14cyclictest181rcu_preempt08:40:561
1539899696,38cyclictest15-21ksoftirqd/008:14:200
15398996947,14cyclictest6514-21fschecks_time11:39:010
1539899693,33cyclictest15-21ksoftirqd/008:59:140
1539899693,33cyclictest15-21ksoftirqd/008:59:140
15398996928,13cyclictest181rcu_preempt12:21:370
1539899692,39cyclictest15-21ksoftirqd/010:36:440
1541699685,37cyclictest0-21swapper/307:59:003
15410996845,15cyclictest21086-21fschecks_count07:24:012
1541099684,42cyclictest20238-21unixbench_singl07:19:152
1541099684,42cyclictest20238-21unixbench_singl07:19:152
1541099683,44cyclictest30940-21mailstats12:24:162
1541099683,43cyclictest10994-21cat08:29:152
1541099683,43cyclictest10994-21cat08:29:152
15410996834,13cyclictest181rcu_preempt09:08:382
1541099683,35cyclictest24805-21switchtime07:34:152
15402996837,23cyclictest2923-21sh08:08:371
1539899682,44cyclictest15-21ksoftirqd/011:48:530
1539899682,35cyclictest4779-21latency_hist08:13:410
15410996748,9cyclictest0-21swapper/211:09:052
15410996740,18cyclictest23758-21sh09:09:132
15402996743,10cyclictest181rcu_preempt08:33:521
15398996746,7cyclictest181rcu_preempt10:44:540
15410996635,24cyclictest0-21swapper/210:44:092
1541099663,44cyclictest17063-21latency_hist07:13:412
15410996627,29cyclictest25065-21diskmemload10:13:482
15410996627,29cyclictest25065-21diskmemload10:13:482
15398996625,13cyclictest0-21swapper/010:50:030
15398996625,13cyclictest0-21swapper/010:50:030
15410996545,14cyclictest9732-21switchtime09:44:132
15410996545,14cyclictest9732-21switchtime09:44:132
1541099653,43cyclictest14277-21grep08:39:132
15402996531,8cyclictest181rcu_preempt11:38:551
15410996443,12cyclictest0-21swapper/209:34:022
1541099643,41cyclictest8802-21fschecks_count08:24:512
1541099643,41cyclictest8802-21fschecks_count08:24:512
1541099643,35cyclictest21225-21latency_hist09:03:392
1541099643,35cyclictest21225-21latency_hist09:03:392
15410996341,12cyclictest0-21swapper/209:24:002
15410996332,12cyclictest181rcu_preempt07:32:032
1541699613,40cyclictest24724-21chrt11:13:403
139622609,27sleep30-21swapper/307:03:573
1541699583,39cyclictest32651-21taskset12:28:393
1541699583,37cyclictest8466-21taskset08:23:413
15416995816,30cyclictest23466-21chrt12:10:083
1541699574,36cyclictest4840-21chrt08:13:413
1541699573,38cyclictest19809-21chrt10:04:003
15416995725,22cyclictest25137-21chrt12:14:013
15416995725,22cyclictest25137-21chrt12:14:013
1541699563,39cyclictest25473-21chrt07:38:383
15416995622,21cyclictest0-21swapper/311:24:163
15416995621,25cyclictest28188-21chrt11:19:113
1541699554,38cyclictest16990-21chrt09:58:493
1541699554,38cyclictest16990-21chrt09:58:493
1541699553,37cyclictest15497-21chrt09:54:483
1541699553,37cyclictest14323-21chrt10:54:003
1541699553,34cyclictest16448-21chrt08:48:403
15416995530,18cyclictest0-21swapper/308:13:543
15416995519,23cyclictest25028-21taskset10:13:443
15416995519,23cyclictest25028-21taskset10:13:443
15416995519,22cyclictest28050-21taskset10:19:033
1541699544,36cyclictest5675-21chrt10:38:403
1541699543,39cyclictest13010-21taskset11:51:163
1541699542,36cyclictest31708-21taskset09:24:053
15416995420,22cyclictest29623-21chrt07:49:113
15416995417,25cyclictest18480-21taskset11:01:213
15416995416,26cyclictest8549-21taskset08:24:023
15416995416,26cyclictest8549-21taskset08:24:023
1541699533,39cyclictest28625-21taskset09:18:543
1541699533,38cyclictest16812-21chrt11:58:583
1541699533,37cyclictest9571-21chrt11:44:113
1541699533,37cyclictest2700-21chrt12:33:133
1541699533,36cyclictest21256-21chrt07:24:043
1541699533,36cyclictest10017-21chrt10:45:123
1541699533,33cyclictest31342-21chrt07:54:133
15416995321,20cyclictest15140-21chrt11:55:213
15416995320,22cyclictest11372-21chrt10:48:453
15416995320,22cyclictest11372-21chrt10:48:453
15416995319,21cyclictest5085-21chrt12:37:063
15416995318,23cyclictest4620-21chrt11:34:353
15416995318,21cyclictest9953-21chrt09:44:153
15416995318,21cyclictest9953-21chrt09:44:153
15416995312,28cyclictest18733-21taskset07:15:383
15416995312,28cyclictest18733-21taskset07:15:383
1541699523,35cyclictest20973-21chrt11:05:423
1541699522,39cyclictest23896-21taskset07:33:403
1541699522,37cyclictest11861-21chrt09:49:013
1541699522,36cyclictest26096-21chrt09:14:003
1541699522,35cyclictest7799-21chrt10:41:563
15416995219,21cyclictest13374-21chrt08:38:483
15416995217,23cyclictest16387-21chrt07:09:143
15416995216,21cyclictest20375-21chrt08:59:093
15416995216,21cyclictest20375-21chrt08:59:093
1541699513,36cyclictest28013-21chrt12:19:073
1541699513,36cyclictest20668-21chrt07:23:393
1541699513,36cyclictest20668-21chrt07:23:393
1541699513,36cyclictest20307-21chrt12:04:213
1541699513,35cyclictest6689-21chrt09:38:563
1541699513,35cyclictest26334-21chrt11:15:433
15416995121,19cyclictest8080-21chrt11:42:003
15416995119,22cyclictest30253-21taskset10:23:493
15416995118,22cyclictest12898-21chrt08:36:253
15416995117,21cyclictest2848-21chrt09:30:343
15416995117,21cyclictest2848-21chrt09:30:343
15416995116,23cyclictest2310-21taskset11:30:543
15416995116,23cyclictest2310-21taskset11:30:543
15416995116,22cyclictest11451-21chrt08:33:303
15416995116,22cyclictest11451-21chrt08:33:303
1541699503,35cyclictest2309-21chrt10:31:523
1541699503,33cyclictest21592-21chrt09:03:553
15416995020,19cyclictest18831-21chrt08:54:113
15416995015,23cyclictest28416-21chrt07:45:263
15416994919,19cyclictest16740-21chrt08:48:573
15416994917,21cyclictest23281-21chrt09:08:573
15416994916,21cyclictest2614-21chrt08:04:463
1541699483,32cyclictest3981-21taskset09:33:463
15416994814,21cyclictest26935-21chrt07:42:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional