You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 08:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Nov 22, 2025 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4895212515,80sleep24924-21/usr/sbin/munin19:04:502
60579912142,69cyclictest0-21swapper/023:54:590
60579912142,69cyclictest0-21swapper/023:54:590
6057991143,22cyclictest18989-21/usr/sbin/munin23:04:570
6057991135,36cyclictest6180-21apt-get22:44:390
746521090,7sleep30-21swapper/322:45:503
60579910647,52cyclictest8012-21cut19:14:310
6057991063,24cyclictest28604-21runrttasks21:34:440
60579910537,59cyclictest0-21swapper/000:35:010
60579910537,59cyclictest0-21swapper/000:35:010
6057991052,22cyclictest31530-21cpuspeed21:39:390
6057991052,22cyclictest31530-21cpuspeed21:39:390
60579910446,52cyclictest6048-21cyclictest23:12:460
60579910428,71cyclictest0-21swapper/021:14:360
60579910337,59cyclictest0-21swapper/020:04:270
6057991033,20cyclictest17985-21find19:44:330
60579910328,67cyclictest0-21swapper/022:14:570
60579910314,29cyclictest20448-21grep21:20:010
60739910266,8cyclictest181rcu_preempt20:09:242
6057991023,38cyclictest18785-21find23:59:330
6057991023,31cyclictest25414-21sh21:29:370
6057991022,29cyclictest11021-21latency_hist21:59:260
6057991014,31cyclictest30536-21runrttasks00:19:400
6057991013,28cyclictest7665-21grep20:49:570
6057991013,21cyclictest355-21runrttasks23:29:100
6057991003,33cyclictest29740-21kernelversion22:29:480
6057991003,33cyclictest29740-21kernelversion22:29:480
6057991002,20cyclictest27660-21awk20:14:230
607399996,49cyclictest18538-21latency_hist23:59:252
607399996,49cyclictest18538-21latency_hist23:59:252
6057999940,50cyclictest0-21swapper/000:29:460
605799993,31cyclictest24939-21rm23:14:560
605799993,29cyclictest7738-21modprobe21:54:230
605799993,29cyclictest3945-21cstates23:34:400
605799993,28cyclictest5611-21latency_hist22:44:270
605799992,25cyclictest3615-21cat23:34:250
605799992,25cyclictest3615-21cat23:34:250
605799992,24cyclictest2419-21ssh22:38:590
605799992,20cyclictest28883-21unixbench_singl20:14:590
605799984,29cyclictest9778-21/usr/sbin/munin23:44:470
605799983,30cyclictest24661-21/usr/sbin/munin00:09:490
605799983,28cyclictest29252-21latency_hist22:29:270
605799983,28cyclictest23172-21switchtime21:24:590
605799983,28cyclictest23172-21switchtime21:24:590
605799983,20cyclictest28002-21kernelversion00:14:500
605799982,26cyclictest6245-21/usr/sbin/munin19:09:490
605799974,31cyclictest26042-21latency_hist20:09:260
605799974,30cyclictest355-21runrttasks22:12:330
605799974,30cyclictest3537-21/usr/sbin/munin20:39:520
605799973,40cyclictest23004-21ssh00:05:190
605799973,30cyclictest2127-21grep20:34:470
605799973,30cyclictest2127-21grep20:34:470
605799973,27cyclictest14657-21/usr/sbin/munin22:04:460
605799973,26cyclictest29540-21/usr/sbin/munin20:20:020
605799972,30cyclictest27542-21fschecks_time23:19:440
605799972,22cyclictest13841-21netstat21:09:510
605799972,22cyclictest13841-21netstat21:09:510
6057999719,23cyclictest19779-21fschecks_count19:49:450
605799964,32cyclictest355-21runrttasks21:46:130
605799964,32cyclictest355-21runrttasks21:46:130
605799964,30cyclictest9168-21/usr/sbin/munin22:50:000
605799962,22cyclictest8359-21ssh23:41:160
605799954,35cyclictest12178-21mailstats19:24:580
605799954,34cyclictest8711-21if_eth020:54:470
605799953,29cyclictest21427-21/usr/sbin/munin19:55:340
605799953,28cyclictest556-21fschecks_time20:29:450
605799953,27cyclictest355-21runrttasks19:32:160
605799953,25cyclictest11485-21cat19:24:270
605799952,31cyclictest23562-21grep22:19:430
605799952,21cyclictest16235-21grep23:00:020
605799952,20cyclictest1577-21timerwakeupswit00:24:570
605799952,13cyclictest368-21snmpd22:54:350
605799952,13cyclictest368-21snmpd22:54:350
605799944,31cyclictest5148-21/usr/sbin/munin20:44:490
605799943,30cyclictest355-21runrttasks19:40:570
605799943,28cyclictest11309-21apt-get21:59:380
605799942,30cyclictest31142-21/usr/sbin/munin20:24:510
605799942,24cyclictest6048-21cyclictest19:09:270
605799923,27cyclictest10432-21cat20:59:490
605799923,23cyclictest13547-21mailstats23:49:550
605799923,23cyclictest13547-21mailstats23:49:550
605799923,17cyclictest11634-21apt-get21:04:340
605799922,28cyclictest14741-21/usr/sbin/munin19:34:490
6065999063,8cyclictest181rcu_preempt21:54:521
510828911,45sleep15141-21ntpq19:04:571
6065998741,13cyclictest181rcu_preempt21:39:491
6065998741,13cyclictest181rcu_preempt21:39:491
6065998671,8cyclictest0-21swapper/121:29:471
607399857,59cyclictest355-21runrttasks22:00:082
606599852,63cyclictest6216-21ssh22:44:411
6073998458,17cyclictest7911-21latency_hist21:54:262
6065998462,7cyclictest181rcu_preempt21:53:241
6073998351,8cyclictest181rcu_preempt23:49:372
6073998351,8cyclictest181rcu_preempt23:49:372
6065998361,14cyclictest1-21systemd23:26:461
6073998261,14cyclictest12023-21netstat19:24:522
6065998239,17cyclictest181rcu_preempt21:19:121
6073998157,16cyclictest316-21rs:main2
6073998157,16cyclictest316-21rs:main2
6065998163,13cyclictest355-21runrttasks00:14:521
6065998140,10cyclictest181rcu_preempt19:24:571
6065998137,16cyclictest181rcu_preempt22:40:241
6065998056,15cyclictest207-21systemd-journal22:08:471
6065998044,12cyclictest181rcu_preempt20:59:071
6065998043,12cyclictest181rcu_preempt19:49:251
6073997956,16cyclictest6048-21cyclictest21:39:452
6073997956,16cyclictest6048-21cyclictest21:39:452
6065997963,10cyclictest15454-21munin-run23:54:221
6065997963,10cyclictest15454-21munin-run23:54:221
6065997955,15cyclictest492-21kworker/u17:2-rpciod23:46:241
6065997953,10cyclictest181rcu_preempt22:24:251
6065997940,15cyclictest181rcu_preempt00:13:171
606599793,37cyclictest31-21ksoftirqd/119:41:101
606599793,34cyclictest31-21ksoftirqd/123:31:171
606599793,34cyclictest31-21ksoftirqd/123:31:171
6073997858,14cyclictest314-21in:imuxsock00:09:222
6073997857,15cyclictest691-21cron23:44:222
607399784,62cyclictest6542-21cpu22:45:302
6065997839,16cyclictest181rcu_preempt22:50:241
6065997838,15cyclictest181rcu_preempt23:14:241
6065997836,17cyclictest181rcu_preempt00:01:171
6065997829,12cyclictest181rcu_preempt22:38:401
6073997759,12cyclictest20884-21latency_hist23:09:262
6065997743,13cyclictest181rcu_preempt22:02:221
6065997743,10cyclictest181rcu_preempt22:25:451
6065997739,15cyclictest181rcu_preempt20:30:411
6065997738,9cyclictest181rcu_preempt21:14:261
6065997738,13cyclictest181rcu_preempt22:10:041
606599772,44cyclictest31-21ksoftirqd/100:25:571
6065997655,15cyclictest314-21in:imuxsock19:19:431
6065997653,10cyclictest181rcu_preempt00:38:001
6065997653,10cyclictest181rcu_preempt00:38:001
6065997646,15cyclictest181rcu_preempt00:33:211
6065997632,12cyclictest181rcu_preempt22:32:361
6065997632,12cyclictest181rcu_preempt22:32:361
6065997631,8cyclictest181rcu_preempt19:36:271
606599762,39cyclictest31-21ksoftirqd/123:04:381
6073997554,14cyclictest0-21swapper/221:30:012
6065997555,7cyclictest181rcu_preempt21:20:031
6065997554,7cyclictest181rcu_preempt22:56:461
6065997554,7cyclictest181rcu_preempt22:56:461
6065997541,12cyclictest181rcu_preempt20:59:451
6065997540,12cyclictest181rcu_preempt20:24:051
6065997534,14cyclictest181rcu_preempt23:18:261
6065997514,18cyclictest31-21ksoftirqd/121:09:541
6073997457,11cyclictest28980-21ntp_kernel_err21:34:532
6073997450,16cyclictest1-21systemd20:22:442
6073997441,12cyclictest181rcu_preempt00:11:032
6073997432,14cyclictest12036-21ssh23:48:132
606599746,51cyclictest20946-21apt-get19:54:391
6065997452,8cyclictest181rcu_preempt21:39:151
6065997452,16cyclictest12985-21diskmemload00:04:581
6065997448,13cyclictest181rcu_preempt23:55:291
6065997448,13cyclictest181rcu_preempt23:55:291
6065997434,12cyclictest181rcu_preempt23:00:461
6077997310,44cyclictest0-21swapper/321:59:593
6073997354,13cyclictest21225-21latency_hist19:54:262
6073997353,14cyclictest16135-21sendmail19:39:232
6073997348,8cyclictest181rcu_preempt23:31:172
6073997348,8cyclictest181rcu_preempt23:31:172
607399734,48cyclictest16164-21latency_hist21:14:262
607399734,48cyclictest16164-21latency_hist21:14:262
6065997354,9cyclictest181rcu_preempt23:40:041
6065997352,15cyclictest0-21swapper/119:14:471
6065997331,12cyclictest181rcu_preempt20:38:461
6065997331,12cyclictest181rcu_preempt20:38:461
16962730,5sleep30-21swapper/321:44:233
16962730,5sleep30-21swapper/321:44:233
6073997255,10cyclictest30265-21latency_hist00:19:262
6073997238,8cyclictest181rcu_preempt23:59:482
607399723,34cyclictest8564-21fschecks_count20:54:442
6065997250,9cyclictest181rcu_preempt21:26:591
6065997250,9cyclictest181rcu_preempt21:26:591
6065997210,27cyclictest31-21ksoftirqd/121:04:481
543527210,37sleep30-21swapper/319:05:083
6073997153,12cyclictest314-21in:imuxsock23:24:222
6073997150,15cyclictest12985-21diskmemload22:44:232
6073997137,11cyclictest181rcu_preempt23:00:162
607399713,50cyclictest355-21runrttasks20:34:432
607399713,50cyclictest355-21runrttasks20:34:432
6065997143,11cyclictest181rcu_preempt00:20:461
6065997127,8cyclictest181rcu_preempt23:21:401
6065997124,25cyclictest0-21swapper/122:14:591
6073997053,11cyclictest10471-21ntp_offset19:19:542
607399703,36cyclictest20138-21latency_hist22:14:262
607399703,34cyclictest31654-21ntpq20:24:502
6065997046,10cyclictest181rcu_preempt20:54:231
6065997031,13cyclictest181rcu_preempt20:27:541
606599702,50cyclictest28860-21head20:14:591
607399695,43cyclictest5750-21runrttasks00:34:002
6073996953,10cyclictest10785-21/usr/sbin/munin21:59:242
6073996948,15cyclictest6048-21cyclictest00:34:462
6073996948,15cyclictest6048-21cyclictest00:34:462
6073996942,11cyclictest0-21swapper/222:04:542
6073996929,9cyclictest26924-21find20:09:542
6065996932,12cyclictest181rcu_preempt23:36:231
606599692,33cyclictest21278-21wc19:54:271
6073996841,18cyclictest12985-21diskmemload23:15:382
6073996837,13cyclictest32047-21kworker/u19:3+rpciod22:50:212
6065996836,12cyclictest181rcu_preempt20:41:341
6065996835,12cyclictest181rcu_preempt20:04:531
6065996834,12cyclictest181rcu_preempt21:44:541
6065996834,12cyclictest181rcu_preempt21:44:541
6065996810,28cyclictest31-21ksoftirqd/120:49:251
295082680,6sleep30-21swapper/323:23:133
6073996750,10cyclictest0-21swapper/221:14:582
6073996749,12cyclictest20722-21ssh22:14:482
6073996749,12cyclictest10783-21timerandwakeup20:59:582
6073996748,11cyclictest0-21swapper/223:37:512
6073996748,11cyclictest0-21swapper/219:39:482
6073996738,9cyclictest30811-21ssh00:19:482
607399673,29cyclictest6049-21unixbench_multi20:45:002
6073996725,10cyclictest181rcu_preempt22:54:402
6073996725,10cyclictest181rcu_preempt22:54:402
6065996737,8cyclictest181rcu_preempt19:30:211
6065996729,19cyclictest31-21ksoftirqd/119:59:481
6073996635,11cyclictest181rcu_preempt20:29:522
607399663,36cyclictest24433-21vmstat22:20:002
6073996633,11cyclictest181rcu_preempt22:30:192
6073996633,11cyclictest181rcu_preempt22:30:192
6073996611,21cyclictest8314-21cpu19:14:412
6065996638,20cyclictest26546-21grep20:09:431
6073996552,9cyclictest58-21kcompactd021:22:232
607399654,23cyclictest0-21swapper/219:32:162
607399653,39cyclictest24070-21latency_hist23:14:262
6073996528,12cyclictest0-21swapper/222:25:452
607399652,44cyclictest32285-21chrt23:27:342
6065996543,8cyclictest181rcu_preempt19:14:231
6073996447,10cyclictest3827-21ssh21:46:592
6073996447,10cyclictest3827-21ssh21:46:592
6073996439,8cyclictest181rcu_preempt20:53:202
6073996439,10cyclictest181rcu_preempt19:11:142
607399643,47cyclictest28875-21unixbench_singl20:14:592
607399643,31cyclictest32314-21/usr/sbin/munin22:34:482
6073996418,8cyclictest181rcu_preempt19:44:432
6073996333,13cyclictest181rcu_preempt19:55:312
6073996248,10cyclictest6048-21cyclictest19:59:362
6073996236,11cyclictest181rcu_preempt20:41:182
607399622,26cyclictest12113-21ntp_kernel_err21:04:502
607399622,25cyclictest3145-21latency_hist00:29:242
6077995821,19cyclictest0-21swapper/319:54:443
6077995743,9cyclictest451rcuc/300:22:383
607799573,43cyclictest19371-21taskset22:12:433
607799573,38cyclictest2850-21chrt20:35:013
607799573,38cyclictest2850-21chrt20:35:013
607799564,36cyclictest5829-21taskset21:49:583
607799563,38cyclictest6729-21chrt23:39:263
607799553,39cyclictest27272-21chrt00:14:233
607799553,37cyclictest9967-21chrt20:59:263
607799552,39cyclictest15634-21chrt23:54:253
607799552,39cyclictest15634-21chrt23:54:253
607799545,35cyclictest23653-21chrt22:19:443
607799544,38cyclictest24266-21chrt23:14:383
607799544,35cyclictest32009-21chrt22:34:233
607799544,35cyclictest32009-21chrt22:34:233
607799543,39cyclictest18517-21taskset23:59:243
607799543,39cyclictest18517-21taskset23:59:243
607799543,37cyclictest26975-21taskset22:24:553
607799542,38cyclictest23096-21taskset00:05:363
607799534,37cyclictest22137-21taskset21:24:253
607799534,34cyclictest29991-21chrt20:19:513
607799533,38cyclictest27029-21chrt20:09:553
607799533,38cyclictest15053-21chrt19:34:453
607799533,38cyclictest14827-21taskset22:04:493
607799532,39cyclictest17932-21taskset21:16:233
607799532,38cyclictest31654-21chrt23:25:493
607799532,37cyclictest18748-21chrt23:59:283
6077995238,9cyclictest451rcuc/323:42:593
607799523,36cyclictest5928-21chrt00:34:243
607799522,34cyclictest29467-21chrt20:19:273
607799514,35cyclictest4499-21chrt21:48:533
607799514,35cyclictest4499-21chrt21:48:533
607799514,34cyclictest28888-21chrt21:34:513
6077995137,10cyclictest451rcuc/319:11:303
607799513,40cyclictest5216-21taskset22:43:183
607799513,38cyclictest18007-21chrt19:44:373
607799513,31cyclictest32673-21chrt20:29:273
607799512,37cyclictest1280-21chrt22:35:493
607799504,30cyclictest9061-21taskset00:39:263
607799504,30cyclictest9061-21taskset00:39:263
6077995037,8cyclictest451rcuc/323:45:513
6077995036,8cyclictest451rcuc/322:57:053
6077995036,8cyclictest451rcuc/322:57:053
607799503,37cyclictest24810-21taskset20:04:413
607799503,35cyclictest29331-21chrt00:17:103
607799503,35cyclictest22855-21chrt21:24:503
607799503,35cyclictest22855-21chrt21:24:503
607799503,34cyclictest23366-21chrt23:12:563
607799503,33cyclictest27025-21chrt21:31:313
607799502,36cyclictest10946-21chrt20:59:593
607799502,35cyclictest6865-21chrt20:49:393
607799502,34cyclictest5239-21taskset20:44:413
607799493,34cyclictest19915-21chrt19:49:473
607799493,30cyclictest13163-21taskset21:09:263
607799492,35cyclictest3270-21chrt00:29:263
607799492,33cyclictest17614-21chrt23:03:413
108752490,7sleep30-21swapper/322:51:413
607799484,34cyclictest20641-21chrt22:14:463
6077994835,8cyclictest451rcuc/319:43:163
607799483,35cyclictest8680-21chrt21:54:573
607799474,32cyclictest13124-21chrt19:29:273
6077994733,9cyclictest451rcuc/323:33:443
6077994733,9cyclictest451rcuc/323:33:443
607799472,29cyclictest13834-21chrt21:09:513
607799472,29cyclictest13834-21chrt21:09:513
607799463,28cyclictest18844-21chrt23:04:543
6077994430,10cyclictest451rcuc/320:02:033
607799442,28cyclictest3802-21chrt20:39:433
607799434,30cyclictest8649-21taskset19:14:493
6077994331,8cyclictest451rcuc/320:30:413
6077994329,9cyclictest451rcuc/319:26:453
607799413,28cyclictest11319-21taskset19:24:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional