You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 06:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Mon Nov 17, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28960991143,31cyclictest27046-21cstates20:41:220
289729911056,12cyclictest3804-21diskmemload23:54:451
28960991083,36cyclictest1-21systemd23:59:420
28960991083,36cyclictest1-21systemd23:59:420
1404621070,5sleep014045-21ssh00:14:150
289769910516,80cyclictest3419-21apt-get23:00:322
28960991054,29cyclictest3294-21modprobe19:29:430
28960991053,33cyclictest15242-21wc00:15:180
289609910445,52cyclictest0-21swapper/022:30:240
28960991043,33cyclictest355-21runrttasks23:41:000
28960991043,31cyclictest5775-21/usr/sbin/munin00:00:170
28960991042,27cyclictest26753-21latency_hist22:44:460
28960991033,26cyclictest368-21snmpd22:26:060
289609910237,55cyclictest0-21swapper/023:35:070
28960991023,32cyclictest20052-21latency_hist20:19:450
28960991023,32cyclictest20052-21latency_hist20:19:450
28960991023,30cyclictest355-21runrttasks22:48:390
28960991023,30cyclictest355-21runrttasks22:48:390
28960991022,31cyclictest6789-21idleruntime-cro21:14:430
28960991012,27cyclictest355-21runrttasks22:20:530
28960991003,31cyclictest31473-21ssh21:55:280
28960991003,29cyclictest24329-21modprobe21:44:470
28960991003,29cyclictest22503-21switchtime20:25:190
28960991003,20cyclictest30425-21/usr/sbin/munin22:50:130
28960991002,29cyclictest21381-21chrt21:39:450
2801521009,61sleep228018-21ntp_states19:05:232
2801521009,61sleep228018-21ntp_states19:05:232
2896099994,19cyclictest15784-21ssh21:29:530
2896099994,19cyclictest15784-21ssh21:29:530
2896099993,36cyclictest9731-21unixbench_multi00:05:210
2896099993,32cyclictest21531-21modprobe20:24:420
2896099993,20cyclictest3335-21gpgv22:59:470
2896099993,20cyclictest3335-21gpgv22:59:470
2896099992,33cyclictest23392-21/usr/sbin/munin20:30:090
2896099992,31cyclictest28421-21switchtime21:50:360
2896099983,41cyclictest22158-21/usr/sbin/munin21:40:130
2896099983,32cyclictest355-21runrttasks23:21:210
2896099983,31cyclictest29464-21/usr/sbin/munin23:45:150
2896099983,31cyclictest29464-21/usr/sbin/munin23:45:150
2896099983,30cyclictest1085-21fschecks_time22:00:080
2896099983,28cyclictest2450-21/usr/sbin/munin21:05:040
2896099983,27cyclictest29839-21latency_hist20:49:450
2896099982,44cyclictest13216-21fschecks_time21:25:070
2896099982,25cyclictest355-21runrttasks23:09:400
2896099973,33cyclictest16565-21latency_hist20:09:450
2896099973,32cyclictest10918-21grep19:50:210
2896099973,32cyclictest10918-21grep19:50:210
2896099973,21cyclictest9173-21apt-get23:10:010
2896099973,21cyclictest9173-21apt-get23:10:010
2896099972,31cyclictest355-21runrttasks23:14:570
2896099972,21cyclictest21446-21ps23:30:180
28972999645,17cyclictest181rcu_preempt22:55:011
28972999645,17cyclictest181rcu_preempt22:55:011
2896099965,22cyclictest13488-21acpi20:00:040
2896099963,35cyclictest7214-21processes22:10:170
2896099963,28cyclictest5158-21/usr/sbin/munin19:35:100
2896099963,21cyclictest21174-21ssh22:34:590
28960999628,62cyclictest0-21swapper/019:09:480
28960999628,62cyclictest0-21swapper/019:09:480
2896099962,34cyclictest3754-21ssh23:00:400
2896099962,32cyclictest21306-21switchtime00:25:200
2896099962,29cyclictest17500-21/usr/sbin/munin00:20:200
28976999565,20cyclictest3884-21sh21:09:462
2896099954,30cyclictest3734-21cpu22:05:020
2896099953,33cyclictest9401-21/usr/sbin/munin22:15:040
2896099953,33cyclictest7484-21grep19:40:210
2896099953,33cyclictest7484-21grep19:40:210
2896099953,32cyclictest27079-21sed00:35:220
2896099953,25cyclictest23530-21tune2fs00:30:090
2896099952,30cyclictest355-21runrttasks19:14:470
2896099952,30cyclictest25190-21grep20:35:070
2896099952,30cyclictest25190-21grep20:35:070
28972999469,15cyclictest29369-21latency_hist23:44:461
2896099949,11cyclictest0-21swapper/021:15:040
2896099949,11cyclictest0-21swapper/021:15:040
2896099943,30cyclictest3981-21cat19:30:110
2896099943,30cyclictest16691-21/usr/sbin/munin20:10:180
2896099943,30cyclictest16691-21/usr/sbin/munin20:10:180
2896099943,29cyclictest30629-21wc19:14:460
2896099943,28cyclictest18311-21latency23:25:090
2896099943,28cyclictest18311-21latency23:25:090
2896099942,32cyclictest9738-21modprobe19:49:420
2896099942,32cyclictest9738-21modprobe19:49:420
2896099933,32cyclictest30046-21/usr/sbin/munin20:50:080
2896099933,32cyclictest11606-21/usr/sbin/munin19:55:100
2896099933,32cyclictest11606-21/usr/sbin/munin19:55:100
2896099933,30cyclictest1898-21latency_hist19:24:460
28960999315,10cyclictest0-21swapper/021:00:060
28960999315,10cyclictest0-21swapper/021:00:060
2896099922,31cyclictest10371-21fschecks_time21:20:090
2896099913,28cyclictest32322-21/usr/sbin/munin23:49:540
2896099912,31cyclictest31650-21/usr/sbin/munin20:55:070
28976999062,21cyclictest27221-21apt-key21:49:462
28976998967,16cyclictest691-21cron00:29:422
2782528928,38sleep30-21swapper/319:05:153
2782528928,38sleep30-21swapper/319:05:153
282662888,55sleep128358-21cat19:05:301
282662888,55sleep128358-21cat19:05:301
28976998758,21cyclictest3804-21diskmemload21:10:082
28972998653,22cyclictest28708-21cut20:45:061
2897699857,67cyclictest13512-21taskset20:00:092
28976998461,14cyclictest21777-21open_inodes22:35:162
28976998212,57cyclictest18552-21apt-get20:14:582
28976998212,57cyclictest18552-21apt-get20:14:582
28976998153,17cyclictest7334-21ssh22:10:192
28976998117,26cyclictest3804-21diskmemload22:23:182
28976998058,14cyclictest207-21systemd-journal23:54:302
28976998054,17cyclictest355-21runrttasks19:35:102
28972998055,10cyclictest181rcu_preempt00:20:031
28972997957,15cyclictest9099-21latency_hist22:14:461
28972997950,14cyclictest207-21systemd-journal00:06:581
28976997859,11cyclictest566-21systemctl20:59:432
2897699785,35cyclictest17143-21modprobe00:19:422
2897299784,45cyclictest23507-21grep23:34:451
28976997751,17cyclictest3804-21diskmemload00:10:022
28972997756,14cyclictest24619-21fschecks_count21:45:061
28976997650,17cyclictest355-21runrttasks21:39:462
28972997646,21cyclictest32601-21grep20:55:191
28976997550,18cyclictest21550-21kworker/u18:1+events_unbound21:59:432
28972997559,11cyclictest444-21cron21:59:431
28972997543,10cyclictest181rcu_preempt20:05:001
28976997440,12cyclictest181rcu_preempt23:35:112
28976997410,50cyclictest21536-21idleruntime-cro20:24:422
28972997455,8cyclictest0-21swapper/100:35:071
28972997454,14cyclictest27834-21unixbench_multi20:41:411
28972997452,16cyclictest15779-21switchtime23:20:181
28972997445,20cyclictest9703-21kworker/u19:0+events_unbound00:14:421
28972997429,14cyclictest181rcu_preempt20:29:571
28972997417,43cyclictest27504-21apt-get21:50:071
28976997354,12cyclictest28951-21cyclictest20:40:252
28976997348,18cyclictest27671-21cpuspeed21:50:142
28976997345,20cyclictest3651-21sh22:04:562
28972997355,13cyclictest3804-21diskmemload00:00:161
28985997211,26cyclictest0-21swapper/320:15:113
28985997211,26cyclictest0-21swapper/320:15:113
28976997242,21cyclictest15839-21threads23:20:192
28972997252,14cyclictest23028-21sh21:41:311
28972997233,12cyclictest181rcu_preempt22:30:011
28972997227,13cyclictest181rcu_preempt19:10:251
28972997224,37cyclictest19007-21fschecks_time21:35:091
28972997215,27cyclictest2222-21runrttasks22:01:171
2897699714,46cyclictest7349-21fschecks_count21:15:082
2897699714,46cyclictest7349-21fschecks_count21:15:082
28976997118,13cyclictest20252-21ssh22:33:002
28972997139,16cyclictest14142-21ssh23:18:151
2897699709,49cyclictest368-21snmpd22:25:182
28976997054,10cyclictest207-21systemd-journal23:59:422
28976997054,10cyclictest207-21systemd-journal23:59:422
28976997049,14cyclictest18049-21idleruntime-cro20:14:432
28976997049,14cyclictest18049-21idleruntime-cro20:14:432
28976997047,16cyclictest13354-21kernelversion21:25:102
2897699703,46cyclictest30055-21acpi20:49:552
28972997051,11cyclictest29488-21cron22:49:431
28972997051,11cyclictest29488-21cron22:49:431
28976996953,11cyclictest28951-21cyclictest00:32:482
2897699695,28cyclictest0-21swapper/220:39:462
2897699695,28cyclictest0-21swapper/220:39:462
28976996949,14cyclictest16371-21ntp_kernel_err21:30:122
28976996949,14cyclictest16371-21ntp_kernel_err21:30:122
2897699693,50cyclictest20758-21/usr/sbin/munin23:30:212
28972996946,17cyclictest14245-21kworker/u17:1-xprtiod22:24:441
28972996946,16cyclictest23714-21kworker/u17:3-ext4-rsv-conversion20:39:471
28972996946,12cyclictest0-21swapper/122:39:541
28972996945,9cyclictest0-21swapper/100:34:451
28972996934,10cyclictest181rcu_preempt22:34:551
28985996818,34cyclictest0-21swapper/323:00:323
2897699685,39cyclictest355-21runrttasks19:11:162
28976996850,12cyclictest1140-21timerandwakeup22:55:202
28976996850,12cyclictest1140-21timerandwakeup22:55:202
28976996847,15cyclictest8968-21fschecks_count00:05:062
28976996847,13cyclictest7533-21switchtime19:40:222
28976996847,13cyclictest7533-21switchtime19:40:222
28976996845,15cyclictest28951-21cyclictest23:25:202
28976996845,15cyclictest28951-21cyclictest23:25:202
2897699683,46cyclictest1334-21users19:20:202
28972996852,11cyclictest11829-21ssh23:14:441
28972996852,11cyclictest11829-21ssh23:14:441
28972996850,12cyclictest16677-21threads21:30:201
28972996850,12cyclictest16677-21threads21:30:201
28972996848,14cyclictest1-21systemd23:27:471
28972996848,14cyclictest1-21systemd23:27:471
28972996846,13cyclictest0-21swapper/100:15:171
2897299684,41cyclictest355-21runrttasks21:01:361
2897299684,41cyclictest355-21runrttasks21:01:361
28972996843,15cyclictest0-21swapper/119:35:061
2897299683,47cyclictest4343-21ps22:05:191
2897299683,44cyclictest9861-21sed19:49:451
2897299683,44cyclictest9861-21sed19:49:451
2897299683,44cyclictest31345-21fschecks_count19:15:071
2897299683,40cyclictest23265-21latency_hist20:29:451
2897699675,39cyclictest9817-21latency_hist19:49:452
2897699675,39cyclictest9817-21latency_hist19:49:452
28976996745,14cyclictest355-21runrttasks19:54:552
28976996745,14cyclictest355-21runrttasks19:54:552
28976996744,15cyclictest0-21swapper/222:15:092
28976996742,19cyclictest6945-21proc_pri23:05:202
28976996740,18cyclictest31431-21fschecks_time19:15:082
2897699673,51cyclictest23141-21apt20:29:422
2897699673,44cyclictest5775-21/usr/sbin/munin00:00:052
2897699672,43cyclictest30266-21latency22:50:102
28976996712,20cyclictest0-21swapper/222:04:052
28972996750,11cyclictest369-21ntpd21:10:061
28972996749,12cyclictest292-21dbus-daemon20:14:441
28972996749,12cyclictest292-21dbus-daemon20:14:441
28972996748,12cyclictest1-21systemd21:15:471
28972996748,12cyclictest1-21systemd21:15:471
28972996746,11cyclictest0-21swapper/119:30:221
28972996743,8cyclictest181rcu_preempt21:29:461
2897299673,55cyclictest13478-21taskset19:59:571
2897299673,44cyclictest10816-21timerandwakeup19:50:201
2897299673,44cyclictest10816-21timerandwakeup19:50:201
2897299673,43cyclictest1945-21/usr/sbin/munin19:25:001
28972996733,21cyclictest17939-21modprobe22:29:431
28972996720,34cyclictest1-21systemd20:39:441
28972996712,8cyclictest0-21swapper/120:19:591
28976996646,15cyclictest25545-21kworker/u18:2+events_unbound23:44:422
28976996641,15cyclictest0-21swapper/222:40:152
2897699662,44cyclictest355-21runrttasks22:48:542
2897699662,44cyclictest355-21runrttasks22:48:542
28972996655,7cyclictest0-21swapper/119:59:471
28972996655,7cyclictest0-21swapper/119:59:471
28972996648,12cyclictest30722-21munin-node00:26:131
28972996646,9cyclictest0-21swapper/123:00:051
2897299664,44cyclictest2726-21grep21:05:071
2897299664,42cyclictest376-21sshd20:54:311
2897299663,44cyclictest6791-21/usr/sbin/munin19:40:221
2897299663,44cyclictest6791-21/usr/sbin/munin19:40:221
2897699654,39cyclictest355-21runrttasks19:50:262
2897699654,39cyclictest355-21runrttasks19:50:262
28976996541,11cyclictest0-21swapper/200:20:042
2897699653,43cyclictest3855-21idleruntime19:30:082
28972996528,13cyclictest181rcu_preempt22:54:421
28976996444,14cyclictest29602-21cstates23:45:042
28976996444,14cyclictest29602-21cstates23:45:042
28976996437,18cyclictest1672-21timerwakeupswit21:00:202
28976996437,18cyclictest1672-21timerwakeupswit21:00:202
2897699643,45cyclictest24832-21cat20:34:452
2897699643,43cyclictest13705-21ssh23:16:562
2897699643,41cyclictest9616-21latency23:10:112
2897699643,41cyclictest9616-21latency23:10:112
28972996448,8cyclictest0-21swapper/122:19:421
28972996445,13cyclictest0-21swapper/123:45:121
28972996445,13cyclictest0-21swapper/123:45:121
28972996439,13cyclictest0-21swapper/123:05:131
28972996431,12cyclictest181rcu_preempt21:20:461
2897299642,38cyclictest1899-21cat19:24:471
28976996342,10cyclictest0-21swapper/219:25:052
28976996341,12cyclictest0-21swapper/220:05:192
2897699633,47cyclictest28719-21ssh00:39:452
28976996328,25cyclictest10070-21apt-get21:20:002
28976996321,37cyclictest0-21swapper/221:39:562
28972996340,16cyclictest28951-21cyclictest23:59:471
28972996340,16cyclictest28951-21cyclictest23:59:471
28972996318,10cyclictest181rcu_preempt20:15:361
28972996318,10cyclictest181rcu_preempt20:15:361
28972996311,7cyclictest181rcu_preempt23:36:541
2898599623,26cyclictest0-21swapper/320:00:043
2898599622,50cyclictest15585-21chrt22:25:243
2897699612,26cyclictest28986-21ntp_states20:45:122
2898599563,40cyclictest780-21chrt19:20:083
2898599563,39cyclictest19052-21taskset21:35:103
2898599553,38cyclictest31340-21chrt23:47:413
2898599553,38cyclictest31340-21chrt23:47:413
2898599543,38cyclictest23304-21taskset00:29:583
10222540,8sleep30-21swapper/322:00:073
2898599533,39cyclictest25346-21taskset22:41:043
2898599533,36cyclictest14895-21taskset22:24:433
2898599533,35cyclictest27360-21taskset20:41:303
2898599532,39cyclictest8622-21chrt19:45:063
2898599532,39cyclictest8622-21chrt19:45:063
2898599532,38cyclictest3080-21chrt23:55:063
2898599532,38cyclictest3080-21chrt23:55:063
2898599523,37cyclictest10258-21chrt21:20:063
2898599523,36cyclictest16352-21taskset21:30:123
2898599523,36cyclictest16352-21taskset21:30:123
2898599523,34cyclictest24330-21chrt21:44:473
2898599523,34cyclictest14844-21chrt23:19:463
2898599522,37cyclictest9404-21chrt21:18:533
2898599522,37cyclictest9404-21chrt21:18:533
2898599522,37cyclictest19568-21taskset23:27:113
2898599522,37cyclictest19568-21taskset23:27:113
2898599513,37cyclictest613-21chrt23:50:163
2898599513,34cyclictest21823-21chrt20:25:013
2898599512,32cyclictest8355-21chrt19:44:463
2898599512,32cyclictest8355-21chrt19:44:463
2898599503,35cyclictest24071-21taskset21:44:423
2898599503,34cyclictest9358-21chrt23:10:053
2898599503,34cyclictest9358-21chrt23:10:053
2898599503,33cyclictest5534-21chrt19:35:103
2898599503,32cyclictest31561-21taskset21:55:413
2898599502,39cyclictest10688-21taskset00:07:593
2898599502,38cyclictest22275-21taskset22:35:293
2898599502,38cyclictest21642-21chrt20:24:453
2898599502,35cyclictest28845-21chrt21:51:073
2898599502,32cyclictest3680-21chrt22:04:573
28985994936,8cyclictest451rcuc/321:03:183
28985994936,8cyclictest451rcuc/321:03:183
28985994936,8cyclictest451rcuc/320:10:493
28985994936,8cyclictest451rcuc/320:10:493
2898599493,36cyclictest4514-21chrt21:10:093
2898599493,35cyclictest25954-21taskset00:34:483
2898599493,34cyclictest31682-21chrt19:15:143
2898599493,34cyclictest27194-21chrt22:45:053
2898599493,34cyclictest27194-21chrt22:45:053
2898599493,33cyclictest11806-21chrt22:19:123
2898599492,38cyclictest6483-21chrt00:00:173
2898599492,31cyclictest23206-21taskset00:29:463
28985994916,18cyclictest0-21swapper/322:54:423
2898599483,36cyclictest28667-21chrt23:43:233
2898599483,35cyclictest21937-21chrt23:30:353
2898599483,33cyclictest18388-21chrt00:20:223
2898599482,31cyclictest25640-21chrt20:35:183
2898599482,31cyclictest25640-21chrt20:35:183
2898599475,33cyclictest2736-21chrt19:25:183
2898599473,32cyclictest2941-21chrt22:59:443
2898599473,32cyclictest2941-21chrt22:59:443
2898599473,31cyclictest7178-21chrt22:10:163
2898599472,36cyclictest29186-21chrt19:10:033
2898599472,31cyclictest24097-21taskset23:35:093
2898599463,33cyclictest11589-21taskset19:54:463
2898599463,31cyclictest8913-21taskset23:09:453
2898599462,36cyclictest15397-21chrt00:15:203
2898599462,35cyclictest13275-21chrt00:12:003
2898599454,32cyclictest20419-21chrt22:33:303
2898599443,32cyclictest16536-21chrt23:21:443
28985994431,9cyclictest451rcuc/320:52:533
2898599433,31cyclictest2747-21chrt21:05:073
2898599423,28cyclictest15476-21taskset21:29:393
2898599412,24cyclictest11099-21chrt19:51:343
2898599403,27cyclictest32248-21taskset20:55:153
28985994027,9cyclictest451rcuc/319:58:283
28985994027,9cyclictest451rcuc/319:58:283
2898599402,29cyclictest3853-21chrt19:30:083
2898599383,24cyclictest23383-21taskset20:29:473
2898599383,23cyclictest29334-21chrt20:45:193
2898599382,27cyclictest15643-21taskset20:05:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional