You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 12:49
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Dec 03, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
82539911752,7cyclictest0-21swapper/019:30:150
82729911366,41cyclictest9107-21sh22:45:042
82729911366,41cyclictest9107-21sh22:45:042
82539911338,15cyclictest0-21swapper/019:28:420
82539911338,15cyclictest0-21swapper/019:28:420
82539911034,20cyclictest15869-21modprobe21:08:390
8253991103,33cyclictest32554-21ps23:24:110
8253991103,33cyclictest32554-21ps23:24:110
82539910654,44cyclictest0-21swapper/021:09:220
82539910645,13cyclictest20393-21unixbench_multi21:14:160
82539910440,12cyclictest0-21swapper/020:44:160
8253991023,35cyclictest8245-21cyclictest19:08:440
8272991003,50cyclictest10996-21ssh22:48:562
827299983,37cyclictest24138-21fschecks_time19:54:032
827299972,39cyclictest24747-21ssh21:22:562
827299972,39cyclictest24747-21ssh21:22:562
8265999773,14cyclictest0-21swapper/122:38:421
825399964,44cyclictest11246-21ntpq21:54:100
825399964,44cyclictest11246-21ntpq21:54:100
752729611,58sleep17581-21timerandwakeup19:04:271
827299947,80cyclictest0-21swapper/220:53:512
827299922,70cyclictest2218-21ssh00:23:582
758329210,40sleep20-21swapper/219:04:282
827299913,34cyclictest22477-21/usr/sbin/munin22:14:182
825399914,30cyclictest1681-21ssh00:23:400
8272999045,37cyclictest15890-21diskmemload21:14:112
827299893,38cyclictest28457-21munin-run23:18:412
827299892,40cyclictest8052-21sendmail-msp23:38:402
827299892,40cyclictest8052-21sendmail-msp23:38:402
8272998847,36cyclictest8245-21cyclictest22:32:472
827299876,56cyclictest6241-21sshd20:38:592
8272998734,45cyclictest0-21swapper/219:43:422
827299873,37cyclictest17577-21wc19:34:072
8272998639,40cyclictest1769-21ssh00:23:422
827299863,35cyclictest1542-21latency20:24:072
827299863,33cyclictest27624-21/usr/sbin/munin20:04:092
827299863,31cyclictest355-21runrttasks21:02:092
827299863,29cyclictest355-21runrttasks19:52:192
827299862,32cyclictest30264-21ssh00:15:562
827299862,26cyclictest1733-21kworker/u19:3+rpciod21:58:512
827299856,62cyclictest28947-21apt-get23:18:562
827299853,39cyclictest30963-21taskset22:28:292
827299853,38cyclictest19226-21modprobe22:08:392
827299853,35cyclictest19967-21/usr/sbin/munin23:03:582
827299853,35cyclictest19967-21/usr/sbin/munin23:03:582
827299853,34cyclictest14677-21unixbench_singl19:24:172
827299853,34cyclictest14677-21unixbench_singl19:24:172
827299853,34cyclictest11487-21df23:43:582
827299853,28cyclictest14697-21cstates21:04:402
827299852,37cyclictest14980-21grep22:54:172
827299852,37cyclictest14980-21grep22:54:172
827299852,29cyclictest3224-21grep23:29:162
8272998521,51cyclictest0-21swapper/220:13:422
827299844,32cyclictest13274-21/usr/sbin/munin21:58:412
827299844,32cyclictest13274-21/usr/sbin/munin21:58:412
827299843,38cyclictest4554-21rm22:38:412
827299843,35cyclictest31422-21taskset21:33:592
827299843,24cyclictest8069-21fschecks_time20:44:042
827299842,37cyclictest20965-21fschecks_count19:45:032
8272998423,54cyclictest0-21swapper/220:14:052
827299842,30cyclictest22755-21latency_hist00:03:412
8265998468,9cyclictest0-21swapper/123:34:131
8265998468,9cyclictest0-21swapper/123:34:131
827299833,37cyclictest5906-21chrt20:38:422
827299833,37cyclictest5906-21chrt20:38:422
827299833,35cyclictest7827-21latency_hist22:43:432
827299833,34cyclictest7580-21latency_hist00:33:432
827299833,34cyclictest15344-21grep23:49:192
827299833,32cyclictest15879-21apt-get21:08:452
827299823,37cyclictest8981-21head00:34:172
827299823,34cyclictest548-21rm23:24:192
827299823,34cyclictest548-21rm23:24:192
827299823,31cyclictest23199-21fschecks_count00:04:032
827299822,35cyclictest8709-21unixbench_multi21:49:192
124302820,6sleep00-21swapper/022:50:210
827299813,37cyclictest17209-21cut22:59:042
827299813,34cyclictest9945-21ssh23:40:412
827299813,23cyclictest32495-21/usr/sbin/munin20:19:092
827299812,33cyclictest17282-21cpu23:53:562
827299812,33cyclictest17282-21cpu23:53:562
827299812,28cyclictest9798-21latency_hist19:13:442
8265998165,12cyclictest22106-21kworker/u17:0-rpciod22:29:011
8265998160,11cyclictest0-21swapper/100:03:401
8253998162,13cyclictest9862-21ssh23:40:260
8253998159,15cyclictest7772-21latency_hist00:33:430
827299803,27cyclictest23301-21/usr/sbin/munin23:09:062
827299803,27cyclictest10641-21/usr/sbin/munin19:14:002
8253998065,9cyclictest15890-21diskmemload00:04:040
8253998040,23cyclictest19905-21ssh23:03:570
8253998040,23cyclictest19905-21ssh23:03:570
8272997958,14cyclictest28559-21fschecks_time21:29:032
827299793,38cyclictest25669-21ssh22:19:002
827299793,38cyclictest25669-21ssh22:19:002
827299793,36cyclictest2244-21grep21:39:072
827299793,22cyclictest4820-21ssh21:43:572
827299793,22cyclictest4820-21ssh21:43:572
827299792,34cyclictest12119-21/usr/sbin/munin19:19:112
827299792,28cyclictest2926-21grep20:29:022
827299792,28cyclictest2926-21grep20:29:022
827299792,26cyclictest20128-21ntpq22:09:092
827299792,26cyclictest20128-21ntpq22:09:092
827299792,22cyclictest10104-21/usr/sbin/munin20:49:152
8265997960,8cyclictest0-21swapper/121:09:211
8253997960,13cyclictest23139-21idleruntime23:09:040
827299783,35cyclictest25770-21/usr/sbin/munin00:09:172
827299783,35cyclictest25770-21/usr/sbin/munin00:09:172
827299783,34cyclictest25460-21cstates21:23:592
827299782,29cyclictest26863-21cron20:03:392
827299782,29cyclictest26863-21cron20:03:392
827299782,29cyclictest16933-21/usr/sbin/munin19:33:422
8265997864,10cyclictest8245-21cyclictest21:48:591
8265997857,10cyclictest0-21swapper/122:59:071
8253997837,30cyclictest0-21swapper/000:14:160
8253997750,17cyclictest1038-21latency_hist20:23:430
8253997750,10cyclictest181rcu_preempt21:20:210
8253997750,10cyclictest181rcu_preempt21:20:210
8253997734,34cyclictest17980-21systemctl23:54:140
8253997734,34cyclictest17980-21systemctl23:54:140
8265997660,9cyclictest28972-21kworker/u17:1-writeback23:29:231
8265997656,14cyclictest207-21systemd-journal21:47:001
8265997656,14cyclictest207-21systemd-journal21:47:001
8265997656,13cyclictest20427-21ssh21:14:171
8253997659,8cyclictest0-21swapper/020:49:140
8253997648,8cyclictest181rcu_preempt21:39:040
8253997631,27cyclictest181rcu_preempt23:19:580
8265997559,7cyclictest0-21swapper/100:19:151
8265997557,13cyclictest15890-21diskmemload22:08:571
8265997557,13cyclictest15890-21diskmemload22:08:571
8265997545,21cyclictest20219-21systemd-run19:43:411
8253997562,7cyclictest0-21swapper/023:46:010
8253997554,14cyclictest28359-21cpuspeed21:28:570
8253997554,11cyclictest9812-21idleruntime-cro19:13:400
8253997550,8cyclictest181rcu_preempt22:12:480
8253997550,8cyclictest181rcu_preempt22:12:480
8253997536,28cyclictest0-21swapper/022:54:150
8253997536,28cyclictest0-21swapper/022:54:150
8282997417,47cyclictest0-21swapper/319:28:593
8265997457,8cyclictest0-21swapper/122:45:001
8265997457,8cyclictest0-21swapper/122:45:001
8265997449,19cyclictest25285-21kworker/u18:1+nfsiod21:40:331
8265997441,10cyclictest58-21kcompactd020:18:431
8253997459,8cyclictest0-21swapper/020:38:420
8253997459,8cyclictest0-21swapper/020:38:420
8253997457,8cyclictest0-21swapper/020:14:140
8253997457,7cyclictest0-21swapper/020:08:580
8253997457,11cyclictest26435-21kworker/u18:1-nfsiod23:33:010
8253997453,10cyclictest0-21swapper/019:49:040
8265997360,9cyclictest8245-21cyclictest23:46:071
8265997360,9cyclictest8245-21cyclictest23:10:241
8265997359,8cyclictest0-21swapper/123:04:171
8265997359,8cyclictest0-21swapper/123:04:171
8265997257,10cyclictest14336-21sh23:48:591
8265997253,13cyclictest2737-21fschecks_count00:24:571
8265997251,15cyclictest8587-21fschecks_count23:39:011
8265997251,15cyclictest314-21in:imuxsock20:38:421
8265997251,15cyclictest314-21in:imuxsock20:38:421
8265997250,15cyclictest15152-21apt-key19:28:411
8265997250,15cyclictest15152-21apt-key19:28:411
8265997238,14cyclictest1-21systemd19:13:501
8265997156,11cyclictest8245-21cyclictest22:14:101
8265997152,13cyclictest27391-21fschecks_time20:04:031
8265997142,9cyclictest58-21kcompactd000:30:361
8265997129,11cyclictest181rcu_preempt21:35:551
8253997159,8cyclictest8245-21cyclictest00:12:160
8253997159,8cyclictest8245-21cyclictest00:12:160
8253997149,14cyclictest8245-21cyclictest19:33:590
8253997148,17cyclictest22106-21kworker/u17:0+xprtiod22:34:050
8253997142,11cyclictest25937-21latency22:19:060
8253997142,11cyclictest25937-21latency22:19:060
8265997054,10cyclictest292-21dbus-daemon22:43:411
8265997054,10cyclictest28474-21ssh21:29:011
8265997054,10cyclictest26876-21ssh00:09:161
8265997054,10cyclictest26876-21ssh00:09:161
8253997047,15cyclictest26867-21idleruntime-cro20:03:390
8253997047,15cyclictest26867-21idleruntime-cro20:03:390
8253997037,15cyclictest31413-21df21:33:580
8265996950,14cyclictest0-21swapper/121:03:551
8253996954,8cyclictest0-21swapper/020:24:090
8253996954,8cyclictest0-21swapper/019:23:420
8253996953,11cyclictest17147-21ntp_kernel_pll_22:04:070
8253996948,14cyclictest0-21swapper/022:14:100
8253996946,15cyclictest4863-21ssh22:38:570
8253996944,18cyclictest16414-21kworker/u17:0+nfsiod21:46:410
8253996944,18cyclictest16414-21kworker/u17:0+nfsiod21:46:410
828299688,42cyclictest0-21swapper/322:44:093
828299688,42cyclictest0-21swapper/322:44:093
828299686,52cyclictest0-21swapper/300:23:573
8253996853,11cyclictest8245-21cyclictest21:59:170
8253996851,11cyclictest8237-21ssh22:44:290
8253996851,11cyclictest8237-21ssh22:44:290
8253996846,8cyclictest181rcu_preempt00:27:460
8253996846,15cyclictest8245-21cyclictest00:34:280
8253996843,7cyclictest181rcu_preempt23:33:590
8253996843,7cyclictest181rcu_preempt23:33:590
8253996839,20cyclictest19239-21cat19:39:070
8265996754,9cyclictest58-21kcompactd022:57:461
8265996754,9cyclictest58-21kcompactd022:57:461
8265996750,11cyclictest13197-21ntp_kernel_pll_20:59:091
8265996737,21cyclictest17897-21grep23:54:131
8265996737,21cyclictest17897-21grep23:54:131
8253996740,20cyclictest16414-21kworker/u17:0+nfsiod21:51:200
8253996736,22cyclictest1532-21modprobe22:33:390
8265996650,12cyclictest8245-21cyclictest22:50:211
8265996648,11cyclictest8245-21cyclictest22:05:041
8265996646,14cyclictest14060-21latency21:59:051
8253996653,6cyclictest0-21swapper/021:03:430
8253996649,9cyclictest0-21swapper/000:03:420
8253996649,11cyclictest19549-21ssh23:03:390
8253996645,15cyclictest6450-21fschecks_count20:39:160
8265996547,9cyclictest0-21swapper/123:19:591
8265996543,16cyclictest13483-21cut19:23:411
8265996535,21cyclictest16916-21modprobe19:33:401
8253996535,20cyclictest28686-21sort22:24:020
828299649,47cyclictest0-21swapper/321:09:033
826599646,21cyclictest0-21swapper/121:53:511
826599646,21cyclictest0-21swapper/121:53:511
8265996447,11cyclictest29132-21systemctl22:24:131
8265996436,19cyclictest9785-21cron19:13:401
8265996434,21cyclictest29449-21grep00:14:141
825399643,35cyclictest25848-21/usr/sbin/munin23:13:560
8265996343,14cyclictest10791-21sh20:53:401
826599633,45cyclictest3599-21timerwakeupswit20:29:161
826599633,45cyclictest3599-21timerwakeupswit20:29:161
826599633,28cyclictest24499-21find19:54:131
8265996332,10cyclictest0-21swapper/120:48:431
826599632,46cyclictest22393-21ssh21:18:531
826599632,46cyclictest22393-21ssh21:18:531
8253996339,19cyclictest8245-21cyclictest21:24:060
8265996241,14cyclictest0-21swapper/123:18:011
826599623,45cyclictest25728-21fschecks_count19:59:021
826599623,45cyclictest25728-21fschecks_count19:59:021
826599623,39cyclictest355-21runrttasks19:35:341
8253996249,7cyclictest0-21swapper/019:58:430
8253996240,14cyclictest8245-21cyclictest23:48:490
828299616,50cyclictest0-21swapper/320:39:003
826599613,44cyclictest6347-21cstates20:39:111
826599613,43cyclictest11423-21kernelversion20:54:041
826599612,44cyclictest32376-21latency23:24:051
826599612,44cyclictest32376-21latency23:24:051
826599612,36cyclictest29702-21grep20:09:181
8253996148,10cyclictest8245-21cyclictest20:29:150
8253996148,10cyclictest8245-21cyclictest20:29:150
826599606,28cyclictest8518-21runrttasks00:34:101
8265996042,12cyclictest23970-21unixbench_multi00:04:171
8265996041,14cyclictest0-21swapper/119:48:431
826599603,42cyclictest355-21runrttasks20:28:291
8265996029,23cyclictest22863-21processes19:49:151
826599602,43cyclictest25714-21cat21:24:061
8253996046,9cyclictest18355-21kworker/u17:1+flush-179:019:43:540
8253996042,13cyclictest12019-21latency_hist19:18:420
8253996012,19cyclictest11914-21sort20:54:150
828299595,37cyclictest0-21swapper/323:19:343
826599593,42cyclictest32571-21wc20:19:131
8253995940,14cyclictest27302-21forks20:04:020
828299584,40cyclictest5700-21taskset23:34:043
828299584,40cyclictest5700-21taskset23:34:043
8265995814,9cyclictest181rcu_preempt22:19:141
8265995814,9cyclictest181rcu_preempt22:19:141
828299573,39cyclictest2610-21chrt20:28:433
828299573,39cyclictest20444-21taskset19:43:433
828299565,36cyclictest15211-21chrt21:04:563
828299564,27cyclictest0-21swapper/323:58:443
828299564,27cyclictest0-21swapper/323:58:443
8282995637,13cyclictest451rcuc/321:00:433
828299562,41cyclictest22398-21taskset21:18:543
828299562,41cyclictest22398-21taskset21:18:543
828299554,38cyclictest5417-21chrt22:39:093
828299554,34cyclictest28540-21chrt00:13:433
828299554,34cyclictest28540-21chrt00:13:433
828299552,35cyclictest6057-21chrt20:38:433
828299552,35cyclictest6057-21chrt20:38:433
828299544,39cyclictest24752-21chrt19:54:163
828299544,38cyclictest23710-21taskset23:09:163
828299543,40cyclictest28027-21chrt20:04:173
828299543,36cyclictest32122-21taskset20:18:593
828299542,37cyclictest22846-21taskset00:03:423
812325411,26sleep30-21swapper/319:08:413
828299534,37cyclictest17391-21chrt22:59:073
828299534,34cyclictest10499-21chrt21:53:423
828299533,37cyclictest16622-21taskset23:52:523
828299533,37cyclictest10594-21chrt23:42:313
828299524,38cyclictest31948-21taskset21:34:133
828299524,33cyclictest19765-21chrt21:14:043
828299524,33cyclictest17175-21chrt22:04:083
828299523,38cyclictest24444-21chrt00:05:333
828299523,36cyclictest9176-21chrt20:48:393
828299523,36cyclictest20882-21taskset22:10:003
828299523,36cyclictest20882-21taskset22:10:003
828299523,36cyclictest17574-21chrt19:34:073
828299523,33cyclictest25268-21taskset21:23:443
828299522,39cyclictest11596-21chrt20:54:103
828299514,32cyclictest9154-21chrt00:34:463
8282995135,9cyclictest451rcuc/320:29:033
8282995135,9cyclictest451rcuc/320:29:033
828299513,37cyclictest4443-21chrt23:31:573
828299513,36cyclictest27771-21chrt23:16:583
828299513,36cyclictest13936-21taskset23:48:393
828299513,35cyclictest3753-21taskset21:41:473
828299513,35cyclictest29512-21chrt22:24:183
828299513,35cyclictest2454-21chrt22:34:113
828299513,34cyclictest30898-21chrt20:14:133
8282995132,14cyclictest451rcuc/322:23:423
8282995132,14cyclictest451rcuc/322:23:423
828299512,39cyclictest20531-21chrt23:04:143
828299512,39cyclictest20531-21chrt23:04:143
828299512,38cyclictest5577-21chrt00:29:073
828299512,36cyclictest13604-21chrt19:23:423
828299503,37cyclictest880-21chrt22:31:473
828299503,37cyclictest28906-21taskset20:09:033
828299503,35cyclictest15356-21taskset22:55:183
828299503,35cyclictest15356-21taskset22:55:183
828299503,31cyclictest25637-21chrt19:58:583
828299503,31cyclictest25637-21chrt19:58:583
828299502,36cyclictest13328-21chrt21:58:413
828299502,36cyclictest13328-21chrt21:58:413
8282995020,22cyclictest0-21swapper/319:44:003
828299494,33cyclictest10919-21chrt22:48:453
828299493,31cyclictest29909-21chrt21:30:493
828299492,35cyclictest16172-21chrt22:03:273
828299483,31cyclictest23027-21chrt22:14:093
828299482,35cyclictest31805-21chrt00:19:023
828299482,33cyclictest10835-21chrt20:53:423
828299474,31cyclictest8938-21chrt19:09:083
828299473,34cyclictest30911-21chrt00:17:523
828299472,36cyclictest551-21chrt23:24:203
828299472,36cyclictest551-21chrt23:24:203
8282994523,15cyclictest451rcuc/319:24:083
8282994523,15cyclictest451rcuc/319:24:083
828299442,27cyclictest6910-21taskset21:47:223
828299442,27cyclictest6910-21taskset21:47:223
828299403,25cyclictest22749-21chrt19:49:103
8282994027,9cyclictest451rcuc/319:18:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional