You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 20:11
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Mon Dec 01, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266319914067,17cyclictest497-21latency_hist12:03:470
2562521379,102sleep00-21swapper/007:04:210
266419912260,10cyclictest0-21swapper/209:39:202
266419911866,45cyclictest30573-21latency_hist07:18:492
266419911866,45cyclictest30573-21latency_hist07:18:492
2618821166,16sleep10-21swapper/107:04:431
266339911232,41cyclictest0-21swapper/108:39:071
266339911232,41cyclictest0-21swapper/108:39:071
266419911055,12cyclictest0-21swapper/212:38:482
266419911055,12cyclictest0-21swapper/212:38:482
266419910854,13cyclictest1-21systemd10:00:452
266419910743,21cyclictest29190-21cut08:53:472
26633991054,92cyclictest5685-21apt-get11:14:491
2622121056,16sleep20-21swapper/207:05:092
266339910433,40cyclictest0-21swapper/109:54:041
26633999828,34cyclictest0-21swapper/110:04:031
26631999466,19cyclictest11343-21latency_hist07:58:480
26631999466,19cyclictest11343-21latency_hist07:58:480
26631999454,21cyclictest4764-21ssh12:09:420
26641999372,13cyclictest7311-21sh10:18:462
26633999362,21cyclictest1598-21diskmemload11:08:481
2664199923,56cyclictest5685-21apt-get11:14:072
26641999048,11cyclictest181rcu_preempt08:19:142
2663399902,51cyclictest28336-21grep07:13:491
2663399902,51cyclictest28336-21grep07:13:491
2663399902,50cyclictest31530-21find07:19:401
26641998961,9cyclictest181rcu_preempt08:04:242
26641998961,9cyclictest181rcu_preempt08:04:242
26633998942,40cyclictest3593-21cpu12:09:021
2663399892,51cyclictest14451-21wc11:29:121
2663399892,51cyclictest14451-21wc11:29:121
26641998863,10cyclictest181rcu_preempt10:29:312
26641998863,10cyclictest181rcu_preempt10:29:312
2663399887,72cyclictest19053-21grep12:34:231
2663399887,72cyclictest19053-21grep12:34:231
2663399882,63cyclictest23915-21apt-get10:48:591
26631998865,9cyclictest181rcu_preempt09:28:470
26631998850,11cyclictest26622-21cyclictest09:19:200
2665199873,37cyclictest4521-21chrt10:13:453
26641998765,12cyclictest0-21swapper/210:54:222
2665199863,70cyclictest24512-21chrt10:49:063
26641998665,15cyclictest12964-21latency_hist08:03:482
26641998665,15cyclictest12964-21latency_hist08:03:482
2664199865,44cyclictest27912-21fschecks_time11:54:072
26633998663,17cyclictest12890-21apt-key08:03:471
26633998663,17cyclictest12890-21apt-key08:03:471
2663399862,52cyclictest1598-21diskmemload11:59:101
26641998563,12cyclictest0-21swapper/211:08:482
26641998557,16cyclictest181rcu_preempt12:25:362
2663399856,45cyclictest0-21swapper/109:37:531
26633998555,21cyclictest9456-21latency_hist09:23:481
26631998545,15cyclictest58-21kcompactd010:13:490
2663199853,73cyclictest26276-21apt-get09:54:060
26641998460,8cyclictest181rcu_preempt10:53:372
26633998442,35cyclictest0-21swapper/109:49:201
26633998442,35cyclictest0-21swapper/109:49:201
26633998440,38cyclictest0-21swapper/110:19:111
26633998439,39cyclictest19577-21sh11:38:571
2663399843,49cyclictest10043-21/usr/sbin/munin12:19:181
26631998454,24cyclictest18447-21latency12:34:120
26631998454,24cyclictest18447-21latency12:34:120
26641998360,16cyclictest691-21cron11:48:452
26641998360,16cyclictest691-21cron11:48:452
26641998360,13cyclictest0-21swapper/208:48:492
26641998359,14cyclictest0-21swapper/211:29:092
26641998359,14cyclictest0-21swapper/211:29:092
26641998357,19cyclictest691-21cron09:18:462
26641998357,19cyclictest691-21cron09:18:462
26641998264,12cyclictest23841-21latency09:49:122
26641998264,12cyclictest23841-21latency09:49:122
26641998262,14cyclictest30035-21sh10:59:082
26641998259,16cyclictest355-21runrttasks08:14:212
26641998250,15cyclictest0-21swapper/209:33:452
26633998249,23cyclictest21011-21latency09:44:131
2663399822,47cyclictest3013-21fschecks_count11:09:081
2663399822,47cyclictest3013-21fschecks_count11:09:081
26633998212,41cyclictest8789-21diskmemload12:18:041
26631998253,21cyclictest1690-21latency_hist09:08:480
26631998244,9cyclictest181rcu_preempt09:44:100
2663199823,68cyclictest30742-21apt-get07:19:120
2663199823,28cyclictest15-21ksoftirqd/011:04:200
2665199812,48cyclictest0-21swapper/311:14:073
26641998157,14cyclictest0-21swapper/207:43:482
26633998161,14cyclictest1598-21diskmemload09:08:491
2663399814,66cyclictest0-21swapper/110:29:031
2663399814,66cyclictest0-21swapper/110:29:031
26633998142,32cyclictest11782-21ssh10:25:191
2663399812,63cyclictest27188-21apt-get10:53:581
26631998156,16cyclictest369-21ntpd11:29:220
26631998156,16cyclictest369-21ntpd11:29:220
26641998068,8cyclictest0-21swapper/207:54:252
26641998068,8cyclictest0-21swapper/207:54:252
26641998064,12cyclictest26622-21cyclictest10:04:162
26641998059,11cyclictest0-21swapper/209:13:482
26641998055,18cyclictest32207-21apt-key07:23:492
26633998051,20cyclictest691-21cron07:38:461
2663399803,71cyclictest0-21swapper/108:58:571
2663399803,64cyclictest0-21swapper/108:53:571
26631998059,15cyclictest26622-21cyclictest11:39:100
26631998012,28cyclictest5140-21apt-get11:14:010
26641997956,13cyclictest0-21swapper/208:43:472
26641997956,13cyclictest0-21swapper/208:43:472
26641997955,9cyclictest181rcu_preempt08:33:492
2664199794,44cyclictest4346-21grep12:09:202
2663399795,45cyclictest0-21swapper/109:59:051
2663399792,65cyclictest0-21swapper/108:24:011
26631997957,17cyclictest14499-21kworker/u18:3+events_unbound09:38:460
26631997936,13cyclictest0-21swapper/012:24:350
26631997934,13cyclictest181rcu_preempt11:38:480
26641997860,11cyclictest17662-21munin-run12:33:452
26641997858,14cyclictest25910-21timerandwakeup11:49:252
26641997850,11cyclictest181rcu_preempt09:27:212
26641997843,16cyclictest0-21swapper/209:04:112
26641997838,13cyclictest0-21swapper/207:44:172
26633997858,14cyclictest2128-21cpu10:09:021
26633997833,37cyclictest0-21swapper/109:24:241
2663399782,61cyclictest18166-21apt-get08:19:031
2663199787,37cyclictest11969-21mailstats11:24:200
26631997855,15cyclictest12852-21systemd-run08:03:470
26631997855,15cyclictest12852-21systemd-run08:03:470
26631997854,17cyclictest29114-21fschecks_time07:14:100
26631997854,17cyclictest29114-21fschecks_time07:14:100
26631997847,12cyclictest58-21kcompactd010:08:240
26631997843,13cyclictest24460-21apt-get10:49:100
26641997764,8cyclictest0-21swapper/209:54:012
26641997744,22cyclictest27059-21fschecks_time07:09:092
26641997744,22cyclictest27059-21fschecks_time07:09:092
26641997744,12cyclictest181rcu_preempt12:15:532
26633997740,30cyclictest0-21swapper/112:04:211
26633997731,40cyclictest0-21swapper/110:14:261
26633997728,41cyclictest0-21swapper/111:21:371
26633997728,41cyclictest0-21swapper/111:21:371
2663399772,64cyclictest0-21swapper/108:48:571
26631997754,14cyclictest20371-21sh10:41:210
26631997754,14cyclictest20371-21sh10:41:210
26631997730,14cyclictest16493-21ntpq10:34:150
26641997655,10cyclictest0-21swapper/210:43:492
26641997655,10cyclictest0-21swapper/210:43:492
26641997648,12cyclictest181rcu_preempt07:49:182
26633997653,16cyclictest26622-21cyclictest07:53:491
26631997652,15cyclictest17446-21ssh12:33:090
26631997650,18cyclictest19159-21ssh09:40:560
26641997550,18cyclictest3231-21cron12:08:452
26641997536,14cyclictest0-21swapper/208:09:092
2664199752,46cyclictest5462-21cat11:13:492
26633997537,32cyclictest355-21runrttasks12:31:361
26633997532,32cyclictest0-21swapper/108:36:541
26633997532,32cyclictest0-21swapper/108:36:541
26633997529,39cyclictest0-21swapper/111:43:491
26633997529,39cyclictest0-21swapper/111:43:491
2663399752,63cyclictest6621-21awk09:18:461
2663399752,63cyclictest6621-21awk09:18:461
2663399752,59cyclictest0-21swapper/107:54:021
2663399752,59cyclictest0-21swapper/107:54:021
26631997557,11cyclictest26622-21cyclictest10:30:120
26631997557,11cyclictest26622-21cyclictest10:30:120
26631997554,15cyclictest22920-21latency_hist08:33:490
26631997551,17cyclictest26622-21cyclictest12:06:530
26631997544,10cyclictest181rcu_preempt10:19:190
26631997527,13cyclictest181rcu_preempt09:08:540
26641997454,14cyclictest3555-21fschecks_count07:34:072
2663399744,64cyclictest0-21swapper/108:29:021
2663399743,65cyclictest0-21swapper/107:29:021
26633997433,36cyclictest0-21swapper/110:59:231
2663399742,67cyclictest0-21swapper/111:54:001
2663399742,66cyclictest0-21swapper/109:38:591
2663399742,63cyclictest12557-21fschecks_time12:24:091
2663399742,62cyclictest21449-21apt-get10:43:581
2663399742,62cyclictest21449-21apt-get10:43:581
26631997452,15cyclictest8464-21ssh11:18:540
26631997452,15cyclictest8464-21ssh11:18:540
26631997452,14cyclictest24808-21ssh11:48:480
26631997452,14cyclictest24808-21ssh11:48:480
26631997450,18cyclictest691-21cron07:53:460
26631997426,9cyclictest0-21swapper/009:00:050
26631997420,17cyclictest181rcu_preempt11:52:050
26641997346,20cyclictest19819-21fschecks_count11:39:092
26641997345,13cyclictest181rcu_preempt10:21:182
26641997343,11cyclictest181rcu_preempt10:24:202
26641997339,8cyclictest181rcu_preempt11:28:332
2664199732,35cyclictest30022-21proc_pri08:54:182
2663399733,60cyclictest6347-21basename07:43:461
26633997332,34cyclictest0-21swapper/109:09:231
2663399732,61cyclictest1489-21idleruntime-cro07:28:451
2663399732,60cyclictest0-21swapper/108:09:011
26633997323,42cyclictest27781-21latency_hist08:48:491
26631997347,8cyclictest181rcu_preempt10:23:580
26631997337,14cyclictest207-21systemd-journal07:38:480
26641997255,9cyclictest0-21swapper/208:38:482
26641997255,9cyclictest0-21swapper/208:38:482
26641997244,18cyclictest0-21swapper/210:37:562
2663399722,59cyclictest16707-21ssh11:33:491
2663199728,33cyclictest29497-21cron10:58:450
2663199722,45cyclictest23922-21unixbench_singl08:34:250
2663199722,45cyclictest23922-21unixbench_singl08:34:250
259032720,4sleep025901-21cat09:53:480
259032720,4sleep025901-21cat09:53:480
26641997153,8cyclictest0-21swapper/207:25:042
26641997135,19cyclictest8783-21kworker/u18:0-rpciod11:23:482
26641997135,19cyclictest8783-21kworker/u18:0-rpciod11:23:482
2664199713,37cyclictest2846-21cpuspeed11:09:032
26633997130,35cyclictest0-21swapper/111:28:391
2663399712,59cyclictest30617-21sed07:18:501
2663399712,59cyclictest30617-21sed07:18:501
2663399712,58cyclictest13482-21/usr/sbin/munin08:04:241
2663399712,58cyclictest13482-21/usr/sbin/munin08:04:241
26631997154,11cyclictest207-21systemd-journal12:23:460
26631997149,16cyclictest27549-21journal-offline08:48:370
26631997133,28cyclictest0-21swapper/011:58:200
26641997042,21cyclictest17002-21fschecks_count11:34:062
2663399702,60cyclictest18559-21latency_hist10:38:491
2663399702,58cyclictest18723-21ssh10:39:061
2663399702,58cyclictest18723-21ssh10:39:061
26631997050,11cyclictest0-21swapper/012:14:080
26631997048,16cyclictest316-21rs:main0
26631997048,16cyclictest316-21rs:main0
26631997048,16cyclictest2336-21processes07:29:190
26631997042,19cyclictest461-21fschecks_time07:24:510
2663199702,38cyclictest355-21runrttasks08:51:510
26641996941,21cyclictest2506-21threads07:29:212
2664199693,32cyclictest30463-21fschecks_count11:59:062
2663399696,44cyclictest0-21swapper/108:14:201
2663399695,43cyclictest0-21swapper/109:33:491
26631996947,12cyclictest0-21swapper/010:59:010
2663199694,48cyclictest15015-21cstates08:09:020
26631996932,8cyclictest181rcu_preempt11:09:230
26631996932,8cyclictest181rcu_preempt11:09:230
2568726916,29sleep30-21swapper/307:04:243
2665199688,46cyclictest26356-21taskset09:54:123
2663399682,52cyclictest6774-21/usr/sbin/munin07:44:091
26631996833,7cyclictest181rcu_preempt07:48:480
26631996818,27cyclictest181rcu_preempt08:19:040
26641996745,16cyclictest26622-21cyclictest09:19:182
2664199673,42cyclictest19961-21fschecks_count08:24:072
26641996724,10cyclictest0-21swapper/212:20:372
2663399672,51cyclictest25211-21ssh11:49:101
26631996737,21cyclictest25948-21modprobe08:43:450
26631996737,21cyclictest25948-21modprobe08:43:450
2663199673,54cyclictest14782-21latency_hist08:08:480
2663199673,54cyclictest14782-21latency_hist08:08:480
26631996713,20cyclictest24084-21latency_hist10:48:470
26631996713,20cyclictest24084-21latency_hist10:48:470
2664199665,13cyclictest31085-21apt-get08:59:012
26641996638,8cyclictest181rcu_preempt10:48:482
26641996638,8cyclictest181rcu_preempt10:48:482
26641996636,9cyclictest181rcu_preempt09:44:522
26641996630,9cyclictest0-21swapper/210:11:432
26641996619,8cyclictest181rcu_preempt09:33:582
2663199665,31cyclictest12960-21/usr/sbin/munin09:29:200
26631996647,13cyclictest5424-21ntp_states10:14:170
2663199664,43cyclictest5189-21fschecks_count07:39:080
2663199664,41cyclictest29360-21runrttasks09:59:080
26631996625,31cyclictest17274-21find08:14:220
2665199653,30cyclictest0-21swapper/309:42:343
2663199653,45cyclictest30493-21sshd08:55:030
2663199653,41cyclictest26847-21grep07:09:040
2663199653,41cyclictest26847-21grep07:09:040
26631996335,10cyclictest58-21kcompactd008:24:030
2665199573,40cyclictest20475-21taskset10:41:413
2665199573,40cyclictest20475-21taskset10:41:413
2665199573,38cyclictest7524-21chrt07:44:243
2665199573,35cyclictest16148-21taskset12:29:233
2665199564,37cyclictest27360-21taskset07:09:163
2665199564,37cyclictest27360-21taskset07:09:163
2665199563,39cyclictest23463-21chrt11:44:333
2665199563,39cyclictest23463-21chrt11:44:333
2665199563,36cyclictest18648-21chrt08:19:153
2665199562,38cyclictest13621-21chrt08:04:373
2665199562,38cyclictest13621-21chrt08:04:373
238862560,7sleep30-21swapper/310:48:423
238862560,7sleep30-21swapper/310:48:423
2665199553,41cyclictest3889-21chrt12:09:093
2665199553,39cyclictest577-21taskset10:04:503
2665199553,38cyclictest9077-21chrt12:18:453
2665199552,37cyclictest20611-21chrt08:24:213
2665199552,35cyclictest22361-21chrt09:46:383
2665199543,35cyclictest7820-21chrt09:19:223
2665199543,35cyclictest2507-21chrt07:29:213
2665199533,39cyclictest14859-21taskset11:29:223
2665199533,39cyclictest14859-21taskset11:29:223
2665199533,37cyclictest31369-21chrt08:59:083
2665199533,37cyclictest23335-21chrt08:34:143
2665199533,37cyclictest23335-21chrt08:34:143
2665199533,35cyclictest12940-21chrt08:03:483
2665199533,35cyclictest12940-21chrt08:03:483
2665199532,36cyclictest14713-21chrt09:33:493
2665199523,37cyclictest10600-21chrt10:24:063
2665199523,35cyclictest30771-21taskset07:19:133
2665199523,35cyclictest12299-21chrt11:24:243
2665199523,32cyclictest16402-21chrt08:13:483
2665199513,38cyclictest9385-21chrt12:18:573
2665199513,36cyclictest30410-21chrt10:59:153
2665199513,35cyclictest21758-21chrt08:29:113
2665199513,35cyclictest13759-21taskset10:29:173
2665199513,35cyclictest13759-21taskset10:29:173
2665199513,34cyclictest17904-21chrt11:34:323
2665199513,33cyclictest6101-21chrt10:15:033
2665199513,33cyclictest20417-21chrt11:39:233
2665199513,32cyclictest30200-21chrt11:58:513
2665199513,31cyclictest2551-21chrt11:08:473
2665199512,34cyclictest3604-21chrt09:12:053
2665199504,29cyclictest3418-21chrt12:08:483
2665199503,34cyclictest17129-21chrt10:34:483
2665199503,33cyclictest5852-21chrt07:39:223
2665199493,35cyclictest24223-21chrt09:49:223
2665199493,35cyclictest24223-21chrt09:49:223
2665199493,34cyclictest5944-21chrt09:16:023
2665199493,34cyclictest5944-21chrt09:16:023
2665199493,28cyclictest895-21kworker/3:1+events07:13:513
2665199493,28cyclictest895-21kworker/3:1+events07:13:513
2665199492,34cyclictest8758-21chrt07:49:183
2665199492,34cyclictest24878-21chrt08:39:083
2665199492,34cyclictest24878-21chrt08:39:083
2665199492,32cyclictest1687-21chrt09:08:483
2665199474,31cyclictest29803-21taskset09:59:193
2665199473,35cyclictest16539-21taskset09:36:243
2665199473,31cyclictest9492-21chrt11:19:213
2665199473,31cyclictest9492-21chrt11:19:213
2665199463,32cyclictest29497-21chrt08:54:023
2665199463,32cyclictest11413-21chrt09:27:043
2665199463,31cyclictest3863-21taskset11:09:283
2665199463,31cyclictest3863-21taskset11:09:283
2665199452,32cyclictest26561-21chrt11:50:573
26651994428,9cyclictest451rcuc/308:47:463
2665199423,27cyclictest7606-21taskset10:19:003
2665199423,27cyclictest27849-21taskset08:49:003
2665199422,30cyclictest674-21chrt07:24:563
2665199422,30cyclictest14843-21taskset12:28:463
2665199413,26cyclictest9821-21taskset07:54:003
2665199413,26cyclictest9821-21taskset07:54:003
2665199413,26cyclictest17975-21chrt12:33:563
2665199413,26cyclictest17975-21chrt12:33:563
2665199402,27cyclictest27162-21taskset10:53:573
2665199402,27cyclictest16663-21chrt08:14:053
2665199393,25cyclictest3364-21taskset07:33:593
2665199392,24cyclictest28016-21chrt11:54:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional