You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 12:54
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Nov 14, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1804799131115,10cyclictest0-21swapper/100:05:111
180379910753,46cyclictest0-21swapper/021:25:140
1756421066,16sleep20-21swapper/219:05:492
18037991045,24cyclictest27096-21apt-get22:05:200
18037991044,33cyclictest18027-21cyclictest19:09:590
180379910441,54cyclictest0-21swapper/022:47:320
18037991034,34cyclictest20888-21grep23:40:310
18037991034,34cyclictest20888-21grep23:40:310
18037991034,29cyclictest4665-21latency_hist23:14:570
18037991032,21cyclictest24204-21/usr/sbin/munin22:00:210
18037991032,21cyclictest24204-21/usr/sbin/munin22:00:210
18037991023,23cyclictest9881-21grep20:20:300
180379910222,10cyclictest10990-21ssh23:25:140
180529910157,35cyclictest0-21swapper/220:55:172
180379910145,48cyclictest0-21swapper/022:40:180
180379910145,48cyclictest0-21swapper/022:40:180
18037991014,34cyclictest6942-21cpuspeed22:25:110
18037991014,34cyclictest6942-21cpuspeed22:25:110
18037991013,40cyclictest28285-21ssh23:00:110
18037991013,25cyclictest413-21latency_hist22:14:580
18037991013,25cyclictest413-21latency_hist22:14:580
18037991013,23cyclictest19941-21sort19:14:590
180529910061,32cyclictest0-21swapper/221:34:582
180529910061,32cyclictest0-21swapper/221:34:582
18037991005,30cyclictest5770-21/usr/sbin/munin23:15:320
18037991005,30cyclictest5770-21/usr/sbin/munin23:15:320
18037991003,87cyclictest10442-21awk22:30:240
1805899993,85cyclictest20498-21chrt19:15:113
1803799994,23cyclictest31143-21sendmail-msp21:19:550
1803799993,44cyclictest8809-21modprobe20:19:550
18037999920,21cyclictest25478-21ssh23:48:510
18037999918,23cyclictest18734-21apt-get20:50:090
18037999918,23cyclictest18734-21apt-get20:50:090
18037999883,10cyclictest18027-21cyclictest20:49:590
1803799984,35cyclictest1667-21latency_hist23:09:580
1803799983,29cyclictest355-21runrttasks21:44:000
1803799983,27cyclictest22117-21/usr/sbin/munin19:20:310
1803799983,23cyclictest20574-21ls19:15:260
1803799983,21cyclictest31569-21rm23:58:560
1803799983,21cyclictest1170-21sh21:22:470
1803799982,31cyclictest22364-21fschecks_time22:50:190
18037999818,6cyclictest0-21swapper/022:20:170
1803799974,31cyclictest17040-21latency_hist20:44:580
1803799974,29cyclictest14817-21/usr/sbin/munin00:25:360
1803799974,29cyclictest14817-21/usr/sbin/munin00:25:360
1803799973,31cyclictest25615-21ssh21:10:190
1803799973,31cyclictest25615-21ssh21:10:190
18037999728,61cyclictest0-21swapper/022:16:370
1803799972,22cyclictest8440-21ssh00:14:140
1804799964,57cyclictest20494-21apt-get19:15:111
18037999640,51cyclictest0-21swapper/000:20:320
18037999640,51cyclictest0-21swapper/000:20:320
18037999637,50cyclictest15439-21latency_hist20:39:580
18037999626,63cyclictest0-21swapper/000:30:120
18037999621,26cyclictest6252-21sed20:10:250
18037999618,29cyclictest22015-21cpu21:00:140
172882969,59sleep117351-21timerandwakeup19:05:391
18052999551,39cyclictest18027-21cyclictest00:25:232
18052999551,39cyclictest18027-21cyclictest00:25:232
18037999559,26cyclictest23846-21grep21:05:180
1803799954,33cyclictest355-21runrttasks19:50:120
1803799954,32cyclictest25362-21grep22:55:170
1803799953,35cyclictest3782-21/usr/sbin/munin00:05:370
1803799953,35cyclictest26279-21grep19:30:330
1803799952,30cyclictest23719-21/usr/sbin/munin19:25:310
1803799952,28cyclictest8553-21cstates21:35:120
18052999475,11cyclictest0-21swapper/220:09:552
18052999440,46cyclictest0-21swapper/222:15:302
1803799944,32cyclictest5110-21unixbench_multi20:05:350
1803799944,31cyclictest10011-21unixbench_multi00:15:310
1803799943,27cyclictest14950-21timerwakeupswit23:30:340
1803799943,19cyclictest13119-21/usr/sbin/munin22:35:150
1803799942,30cyclictest12002-21latency_hist20:29:580
1803799942,27cyclictest12448-21uniq20:30:180
1804799933,81cyclictest31678-21apt-get21:20:091
1803799934,82cyclictest26391-21df23:50:150
1803799934,32cyclictest18002-21wc23:35:320
1803799934,30cyclictest30172-21/usr/sbin/munin19:45:310
1803799934,24cyclictest0-21swapper/021:59:550
1803799934,20cyclictest28584-21apt-get19:40:070
1803799934,20cyclictest28584-21apt-get19:40:070
1803799933,29cyclictest8110-21fschecks_count23:20:160
1803799933,28cyclictest355-21runrttasks21:45:210
1803799933,28cyclictest355-21runrttasks21:45:210
18052999234,50cyclictest24369-21systemctl19:25:282
18037999237,49cyclictest0-21swapper/000:35:190
1803799923,30cyclictest7818-21ssh21:34:090
1803799923,30cyclictest7818-21ssh21:34:090
18052999140,43cyclictest0-21swapper/223:05:282
1803799913,30cyclictest20321-21/usr/sbin/munin20:55:270
1803799912,29cyclictest20546-21latency_hist21:54:570
18052999067,12cyclictest0-21swapper/220:34:582
18052999046,29cyclictest16421-21grep20:40:342
18047999025,42cyclictest31-21ksoftirqd/123:16:561
18047999025,42cyclictest31-21ksoftirqd/123:16:561
1803799909,10cyclictest0-21swapper/019:55:170
1803799905,31cyclictest2979-21chrt20:00:200
1803799903,28cyclictest26952-21apt-get19:35:070
1803799903,28cyclictest26952-21apt-get19:35:070
18037999026,57cyclictest0-21swapper/000:00:300
18037999026,57cyclictest0-21swapper/000:00:300
18052998941,41cyclictest0-21swapper/219:15:012
1805299893,33cyclictest18508-21latency_hist20:49:572
18047998970,12cyclictest292-21dbus-daemon23:39:561
18052998834,46cyclictest0-21swapper/219:43:162
18052998834,46cyclictest0-21swapper/219:43:162
18052998833,47cyclictest0-21swapper/220:00:162
1805299883,29cyclictest355-21runrttasks23:39:092
18052998818,33cyclictest18526-21timerandwakeup21:50:332
1805299873,31cyclictest24565-21mailstats22:00:312
1805299873,31cyclictest24565-21mailstats22:00:312
1805299872,29cyclictest5816-21apt-get20:10:112
1804799873,32cyclictest31-21ksoftirqd/119:52:341
18052998667,11cyclictest0-21swapper/221:55:182
18052998665,12cyclictest0-21swapper/223:50:182
1805299864,41cyclictest11534-21sshd21:40:082
18052998643,36cyclictest0-21swapper/200:05:092
18052998637,42cyclictest0-21swapper/222:25:172
18052998637,42cyclictest0-21swapper/222:25:172
1805299863,30cyclictest16218-21fschecks_count22:40:172
1805299863,30cyclictest16218-21fschecks_count22:40:172
1805299862,43cyclictest8395-21rm00:14:062
1805299862,32cyclictest5214-21latency_hist21:29:582
1804799868,34cyclictest31-21ksoftirqd/100:13:351
1805299857,35cyclictest1918-21expr23:10:132
18052998566,13cyclictest23240-21sh23:45:122
18052998564,9cyclictest0-21swapper/219:54:582
18052998562,12cyclictest0-21swapper/219:39:572
18052998562,12cyclictest0-21swapper/219:39:572
1805299854,32cyclictest10274-21if_eth022:30:202
1805299853,33cyclictest23474-21awk19:24:552
1805299853,32cyclictest9181-21mailstats21:35:282
1805299853,32cyclictest10396-21latency_hist20:24:572
1805299853,27cyclictest22652-21munin-run00:39:542
1805299852,33cyclictest2420-21latency_hist19:59:572
1805299852,31cyclictest22607-21needreboot22:50:242
1805299852,30cyclictest11432-21timerandwakeup20:25:302
1804799853,52cyclictest31-21ksoftirqd/122:19:381
1804799852,52cyclictest31-21ksoftirqd/123:00:471
1804799852,45cyclictest31-21ksoftirqd/121:05:131
1805299844,40cyclictest32193-21tune2fs21:20:522
1805299844,28cyclictest26984-21apt-key22:04:592
1805299843,32cyclictest14761-21diskstats21:45:152
1805299843,32cyclictest14761-21diskstats21:45:152
1805299843,30cyclictest31613-21latency_hist19:49:572
1805299843,30cyclictest22178-21fschecks_count21:00:192
1805299842,44cyclictest25922-21/usr/sbin/munin21:10:282
1805299842,44cyclictest25922-21/usr/sbin/munin21:10:282
1805299842,33cyclictest29443-21users21:15:342
1805299842,30cyclictest19380-21grep20:50:312
1805299842,30cyclictest19380-21grep20:50:312
18052998410,11cyclictest0-21swapper/200:01:562
18052998410,11cyclictest0-21swapper/200:01:562
18047998447,12cyclictest181rcu_preempt19:48:331
1804799842,54cyclictest31-21ksoftirqd/122:37:421
18052998350,28cyclictest0-21swapper/200:30:122
1805299833,31cyclictest27976-21ssh22:59:572
1805299833,28cyclictest355-21runrttasks22:22:382
1805299833,28cyclictest20117-21/usr/sbin/munin23:40:352
1805299833,28cyclictest20117-21/usr/sbin/munin23:40:352
18052998330,46cyclictest14531-21ssh23:30:252
18052998323,53cyclictest30266-21unixbench_multi23:55:332
18052998312,24cyclictest14274-21diskmemload22:36:112
18047998368,9cyclictest31430-21fschecks_count23:05:171
18047998361,16cyclictest314-21in:imuxsock19:55:091
18047998358,18cyclictest31105-21latency_hist21:19:591
18047998350,9cyclictest181rcu_preempt20:59:411
1805299824,29cyclictest13864-21/usr/sbin/munin20:35:212
1805299823,34cyclictest7662-21cut20:15:202
1805299823,30cyclictest28264-21cpuspeed23:00:112
1805299823,29cyclictest9119-21ssh23:20:542
1805299823,29cyclictest26197-21cat19:30:322
18052998223,51cyclictest0-21swapper/200:19:562
1805299822,30cyclictest25112-21latency_hist21:09:582
18047998255,9cyclictest181rcu_preempt22:20:181
18047998215,53cyclictest32009-21unin-run23:59:561
18052998161,14cyclictest20714-21ssh22:47:082
18052998154,19cyclictest19600-21sh19:14:562
1805299813,38cyclictest30824-21grep22:10:292
1805299813,38cyclictest30824-21grep22:10:292
1805299813,37cyclictest7744-21grep23:19:582
1805299813,37cyclictest7744-21grep23:19:582
1804799819,30cyclictest31-21ksoftirqd/123:21:541
18047998137,12cyclictest181rcu_preempt00:34:001
18047998124,33cyclictest31-21ksoftirqd/123:52:451
18052998057,12cyclictest0-21swapper/200:24:502
18052998057,12cyclictest0-21swapper/200:24:502
1805299803,31cyclictest11277-21fschecks_time23:25:212
18047998063,11cyclictest8302-21latency_hist21:34:581
18047998063,11cyclictest8302-21latency_hist21:34:581
18047998055,8cyclictest181rcu_preempt00:00:441
18047998055,8cyclictest181rcu_preempt00:00:441
18047998044,14cyclictest181rcu_preempt20:51:431
18047998044,14cyclictest181rcu_preempt20:51:431
18047998034,17cyclictest18027-21cyclictest00:17:491
1804799802,50cyclictest31-21ksoftirqd/120:15:271
1804799802,41cyclictest31-21ksoftirqd/120:05:261
1804799801,36cyclictest31-21ksoftirqd/123:47:331
18047997962,12cyclictest24903-21latency_hist22:54:571
18047997959,12cyclictest29811-21latency_hist22:09:581
18047997948,12cyclictest181rcu_preempt20:29:581
1804799794,44cyclictest31-21ksoftirqd/119:13:581
18047997920,27cyclictest0-21swapper/121:26:251
18047997864,10cyclictest18027-21cyclictest20:20:311
18047997861,11cyclictest207-21systemd-journal22:29:541
18047997861,11cyclictest207-21systemd-journal22:29:541
18047997860,12cyclictest24941-21diskmemload21:41:511
18047997855,17cyclictest15032-21latency21:45:211
18047997855,17cyclictest15032-21latency21:45:211
1804799785,40cyclictest31-21ksoftirqd/120:33:121
18047997853,8cyclictest181rcu_preempt22:48:391
18047997846,13cyclictest181rcu_preempt23:10:521
18047997845,8cyclictest181rcu_preempt20:40:131
18047997837,13cyclictest181rcu_preempt19:20:151
18047997753,10cyclictest181rcu_preempt22:30:181
1804799775,21cyclictest31-21ksoftirqd/100:37:511
18047997734,11cyclictest181rcu_preempt21:53:291
18047997725,32cyclictest31-21ksoftirqd/120:00:361
1804799772,52cyclictest31-21ksoftirqd/123:27:131
18047997660,10cyclictest309-21systemd-logind23:34:551
18047997645,13cyclictest181rcu_preempt21:37:471
18047997640,7cyclictest181rcu_preempt22:01:061
18047997640,7cyclictest181rcu_preempt22:01:061
18047997610,26cyclictest31-21ksoftirqd/123:43:011
18047997610,26cyclictest31-21ksoftirqd/123:43:011
18047997555,15cyclictest18027-21cyclictest21:10:311
18047997555,15cyclictest18027-21cyclictest21:10:311
18047997548,8cyclictest181rcu_preempt20:14:141
18047997534,7cyclictest181rcu_preempt19:39:551
18047997534,7cyclictest181rcu_preempt19:39:551
1804799753,29cyclictest31-21ksoftirqd/120:35:031
18047997525,15cyclictest0-21swapper/120:45:221
18047997519,32cyclictest31-21ksoftirqd/119:25:001
18047997514,22cyclictest301ktimers/122:55:331
1804799742,39cyclictest31-21ksoftirqd/100:21:431
1804799742,39cyclictest31-21ksoftirqd/100:21:431
18047997354,12cyclictest21030-21ssh21:55:181
18047997339,8cyclictest181rcu_preempt22:43:131
18047997339,8cyclictest181rcu_preempt22:43:131
1804799723,43cyclictest31-21ksoftirqd/119:34:031
1804799719,31cyclictest31-21ksoftirqd/100:27:011
1804799719,31cyclictest31-21ksoftirqd/100:27:011
1804799712,27cyclictest31-21ksoftirqd/122:09:591
1804799712,27cyclictest31-21ksoftirqd/122:09:591
18047996828,12cyclictest181rcu_preempt21:03:391
18047996651,8cyclictest0-21swapper/119:40:321
18047996651,8cyclictest0-21swapper/119:40:321
18058996413,32cyclictest0-21swapper/323:59:563
18058996210,33cyclictest9500-21chrt22:29:463
18058996210,33cyclictest9500-21chrt22:29:463
18058996126,20cyclictest0-21swapper/321:20:153
18058996011,32cyclictest0-21swapper/300:05:143
18058996010,30cyclictest0-21swapper/322:05:143
1805899573,41cyclictest3199-21taskset20:00:273
1805899563,39cyclictest18873-21chrt19:10:293
1656625613,23sleep30-21swapper/319:05:123
1805899553,38cyclictest7545-21chrt23:19:543
1805899553,38cyclictest7545-21chrt23:19:543
1805899553,37cyclictest1741-21chrt22:15:323
1805899545,38cyclictest17684-21chrt00:30:123
1805899544,37cyclictest32200-21taskset19:50:233
18058995440,9cyclictest451rcuc/320:43:093
18058995437,8cyclictest451rcuc/323:29:393
18058995434,14cyclictest451rcuc/300:15:163
1805899543,40cyclictest28926-21chrt21:15:243
1805899543,38cyclictest14841-21taskset00:25:103
1805899543,38cyclictest14841-21taskset00:25:103
1805899534,39cyclictest13958-21chrt23:30:033
1805899534,37cyclictest13091-21taskset22:35:143
1805899533,39cyclictest19195-21chrt20:50:253
1805899533,39cyclictest19195-21chrt20:50:253
1805899533,37cyclictest22166-21taskset19:20:133
1805899533,36cyclictest24072-21chrt21:05:233
1805899532,39cyclictest28113-21chrt23:00:013
1805899532,39cyclictest12280-21chrt20:30:123
1805899524,34cyclictest23338-21chrt21:04:553
1805899523,38cyclictest9162-21chrt23:21:003
1805899523,37cyclictest22332-21taskset23:43:333
1805899523,37cyclictest22332-21taskset23:43:333
1805899523,37cyclictest16032-21chrt22:40:123
1805899523,37cyclictest16032-21chrt22:40:123
1805899523,35cyclictest18371-21chrt21:50:313
1805899522,36cyclictest8836-21chrt00:14:573
1805899522,36cyclictest13866-21chrt20:35:103
1805899513,35cyclictest1800-21taskset23:10:103
1805899512,36cyclictest22340-21chrt21:56:593
1805899512,35cyclictest14666-21chrt21:45:123
1805899512,35cyclictest14666-21chrt21:45:123
1805899507,31cyclictest7765-21chrt21:34:013
1805899507,31cyclictest7765-21chrt21:34:013
1805899505,31cyclictest13111-21chrt21:41:513
1805899503,36cyclictest26411-21chrt21:10:383
1805899503,36cyclictest26411-21chrt21:10:383
1805899503,36cyclictest19628-21chrt22:45:273
1805899503,35cyclictest26250-21chrt23:50:133
1805899503,35cyclictest24264-21chrt19:25:243
1805899503,34cyclictest24489-21chrt23:46:083
1805899503,33cyclictest26964-21chrt19:35:123
1805899503,33cyclictest26964-21chrt19:35:123
1805899503,33cyclictest2322-21taskset21:25:093
1805899503,31cyclictest31738-21taskset19:49:583
1805899502,34cyclictest26038-21chrt22:55:313
1805899493,33cyclictest11990-21chrt00:20:123
1805899493,33cyclictest11990-21chrt00:20:123
1805899493,31cyclictest8519-21chrt21:35:103
1805899492,34cyclictest20554-21taskset00:35:193
18058994833,8cyclictest451rcuc/319:41:033
18058994833,8cyclictest451rcuc/319:41:033
1805899483,36cyclictest21193-21chrt20:55:313
1805899483,35cyclictest2302-21chrt00:04:153
1805899483,35cyclictest2302-21chrt00:04:153
1805899483,33cyclictest537-21taskset23:07:273
1805899483,31cyclictest25236-21chrt22:01:253
1805899483,31cyclictest25236-21chrt22:01:253
1805899482,35cyclictest10713-21taskset22:30:313
1805899474,34cyclictest431-21chrt22:14:553
1805899474,34cyclictest431-21chrt22:14:553
1805899473,35cyclictest4490-21taskset20:05:213
1805899473,31cyclictest23047-21taskset22:50:333
1805899473,30cyclictest17218-21taskset23:35:143
1805899472,34cyclictest5395-21chrt22:21:483
1805899472,33cyclictest10511-21chrt20:24:573
1805899463,30cyclictest7904-21chrt20:15:263
1805899463,29cyclictest1380-21chrt19:55:213
1805899444,28cyclictest17624-21taskset20:45:273
18058994429,8cyclictest451rcuc/319:31:153
18058994331,8cyclictest451rcuc/320:12:023
1805899393,27cyclictest11847-21taskset20:28:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional