You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-06 - 13:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Mon Apr 06, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2530322720,60sleep015-21ksoftirqd/019:00:260
258589912453,50cyclictest0-21swapper/022:06:130
2511421220,13chrt25113-21sendmail_mailtr19:00:231
258589911559,50cyclictest7603-21latency_hist23:03:500
25858991123,25cyclictest11051-21fschecks_time22:15:040
25858991123,25cyclictest11051-21fschecks_time22:15:040
25863991113,77cyclictest352-21runrttasks22:14:071
25863991113,77cyclictest352-21runrttasks22:14:071
258589911149,53cyclictest0-21swapper/020:34:060
25858991113,31cyclictest17907-21fschecks_count00:14:050
25858991113,31cyclictest17907-21fschecks_count00:14:050
25858991092,32cyclictest8650-21/usr/sbin/munin23:59:170
25863991083,82cyclictest27421-21df00:30:071
258589910727,11cyclictest0-21swapper/021:28:470
258589910636,11cyclictest0-21swapper/022:43:470
25858991063,48cyclictest20139-21/usr/sbin/munin23:24:220
258589910629,69cyclictest0-21swapper/023:39:170
258699910560,39cyclictest6734-21cpu21:14:032
258589910545,51cyclictest0-21swapper/021:18:490
258589910543,56cyclictest25845-21cyclictest21:20:440
258589910543,56cyclictest25845-21cyclictest21:20:440
258589910542,7cyclictest0-21swapper/023:09:080
258589910528,69cyclictest0-21swapper/022:02:000
258589910526,13cyclictest0-21swapper/021:49:090
258589910446,52cyclictest25845-21cyclictest19:44:210
25858991043,35cyclictest22618-21cut22:34:070
25858991043,35cyclictest22618-21cut22:34:070
258589910426,16cyclictest0-21swapper/020:09:220
258699910337,59cyclictest0-21swapper/221:04:222
25858991032,42cyclictest352-21runrttasks21:06:000
25858991032,33cyclictest25921-21/usr/sbin/munin20:39:220
258589910266,29cyclictest10169-21apt-key22:13:490
258589910245,50cyclictest0-21swapper/019:38:480
25858991014,34cyclictest17179-21/usr/sbin/munin22:24:200
258589910135,9cyclictest0-21swapper/020:49:070
258589910135,9cyclictest0-21swapper/020:49:070
25858991013,43cyclictest31303-21/usr/sbin/munin19:19:110
25858991013,38cyclictest26874-21/usr/sbin/munin00:28:570
25858991012,33cyclictest5663-21/usr/sbin/munin23:54:230
25858991012,33cyclictest5663-21/usr/sbin/munin23:54:230
258589910034,60cyclictest0-21swapper/021:54:110
25858991003,34cyclictest31614-21grep20:59:070
25858991003,34cyclictest31614-21grep20:59:070
25858991003,28cyclictest352-21runrttasks23:05:160
25858991002,41cyclictest11515-21latency_hist19:58:490
25858991002,36cyclictest19494-21/usr/sbin/munin21:34:210
25858991002,31cyclictest29441-21run-parts19:13:460
2585899994,31cyclictest22530-21timerandwakeup21:39:220
2585899994,31cyclictest22530-21timerandwakeup21:39:220
2585899993,37cyclictest1336-21ssh22:53:040
2585899993,37cyclictest1336-21ssh22:53:040
2585899993,36cyclictest19659-21fschecks_time22:29:070
2585899993,30cyclictest27611-21munin-node22:53:540
2585899993,30cyclictest18533-21latency_hist21:33:490
2585899993,30cyclictest18533-21latency_hist21:33:490
2585899993,29cyclictest1128-21mailstats19:24:180
2585899992,32cyclictest16193-21latency_hist22:23:480
25858999838,10cyclictest31395-21kworker/u17:3+rpciod23:14:200
25858999838,10cyclictest31395-21kworker/u17:3+rpciod23:14:200
2585899983,37cyclictest29769-21/usr/sbin/munin20:54:210
2585899983,33cyclictest4423-21awk21:09:220
2585899983,33cyclictest4423-21awk21:09:220
2585899983,31cyclictest23617-21/usr/sbin/munin00:23:560
2585899982,31cyclictest26227-21fschecks_count19:04:060
25858999814,31cyclictest23372-21latency_hist00:23:470
2585899973,41cyclictest25793-21ssh21:44:360
2585899973,38cyclictest14539-21ssh00:08:490
2585899973,38cyclictest14539-21ssh00:08:490
2585899973,38cyclictest13816-21latency20:04:120
2585899973,36cyclictest18785-21fschecks_count20:20:150
2585899973,34cyclictest12163-21cat19:59:110
2585899973,33cyclictest27177-21sed23:34:220
2585899973,30cyclictest2540-21latency_hist23:48:490
2585899972,36cyclictest23515-21if_eth023:29:090
2585899972,36cyclictest23515-21if_eth023:29:090
25869999648,41cyclictest0-21swapper/219:04:212
25858999672,18cyclictest17433-21latency_hist00:13:470
2585899963,34cyclictest352-21runrttasks20:33:480
2585899963,30cyclictest28361-21/usr/sbin/munin22:44:080
2585899963,28cyclictest2655-21/usr/sbin/munin23:49:120
25869999549,39cyclictest0-21swapper/223:19:082
25869999547,41cyclictest0-21swapper/220:20:002
2585899953,37cyclictest352-21runrttasks19:14:100
2585899953,36cyclictest17977-21latency_hist20:18:490
2585899953,33cyclictest21496-21latency_hist20:28:480
2586999943,35cyclictest29605-21latency_hist20:53:482
2586999943,35cyclictest29605-21latency_hist20:53:482
2586999943,35cyclictest22964-21awk22:34:132
2586999943,35cyclictest22964-21awk22:34:132
2586999943,33cyclictest28844-21munin-plugin-st23:38:452
2586999943,32cyclictest5525-21ssh22:59:142
2586399942,73cyclictest8220-21sh23:04:071
2585899942,33cyclictest9978-21latency_hist19:53:480
2585899942,33cyclictest9978-21latency_hist19:53:480
2586999933,34cyclictest7181-21fschecks_time19:44:072
2586999933,34cyclictest5633-21/usr/sbin/munin19:39:092
25869999333,11cyclictest0-21swapper/221:58:482
25869999319,25cyclictest27796-21rm21:49:032
2585899933,33cyclictest5409-21cstates19:39:030
2586999924,34cyclictest11156-21runrttasks22:15:072
2586999924,34cyclictest11156-21runrttasks22:15:072
2586999923,35cyclictest13959-21latency_hist23:13:472
2586999923,34cyclictest31051-21modprobe22:48:452
2586999922,34cyclictest26782-21fschecks_time20:44:082
2586999922,31cyclictest25411-21ntp_kernel_err20:39:112
2585899923,33cyclictest17471-21perl23:19:070
25869999134,11cyclictest0-21swapper/219:09:132
2586999913,39cyclictest31045-21modprobe19:18:462
2586999913,36cyclictest1361-21grep19:24:212
2586999913,34cyclictest30258-21netstat20:54:142
2586999913,34cyclictest20270-21fschecks_count20:24:062
2586999913,33cyclictest32163-21grep20:59:202
2586999913,33cyclictest32163-21grep20:59:202
2586999913,33cyclictest17559-21grep20:14:212
2586999913,33cyclictest13379-21/usr/sbin/munin22:19:212
2586999913,31cyclictest1-21systemd00:23:452
2586999912,39cyclictest10439-21fschecks_time19:54:082
2586999912,37cyclictest24667-21latency_hist20:38:492
2586999912,34cyclictest9810-21ssh22:13:442
2585899914,32cyclictest3422-21latency_hist19:33:500
2585899913,31cyclictest26528-21/usr/sbin/munin20:44:140
2586999904,38cyclictest12165-21latency19:59:112
2586999904,37cyclictest22826-21latency_hist23:28:492
2586999904,31cyclictest4138-21/usr/sbin/munin22:04:092
2586999904,31cyclictest31523-21fschecks_count19:19:082
2586999904,31cyclictest25912-21latency_hist23:33:482
2586999904,31cyclictest25912-21latency_hist23:33:482
2586999903,35cyclictest6240-21ssh21:13:372
2586999903,35cyclictest6240-21ssh21:13:372
2586999903,34cyclictest15538-21timerandwakeup00:09:202
2586999903,32cyclictest16871-21latency22:24:112
25869999027,20cyclictest2008-21cat21:59:212
2586999902,36cyclictest14313-21tail20:04:232
2586999902,35cyclictest9083-21/usr/sbin/munin19:49:162
2586999902,35cyclictest9083-21/usr/sbin/munin19:49:162
2586999902,34cyclictest22137-21/usr/sbin/munin21:39:122
2586999902,34cyclictest22137-21/usr/sbin/munin21:39:122
2586999895,33cyclictest2500-21sort23:48:482
2586999894,32cyclictest17906-21fschecks_count00:14:052
2586999894,32cyclictest17906-21fschecks_count00:14:052
2586999893,34cyclictest25167-21ntpq21:44:142
2586999893,34cyclictest22060-21kernelversion20:29:132
2586999892,32cyclictest2687-21processes19:29:182
2586999892,32cyclictest13394-21mailstats21:24:202
2438928914,48sleep224376-21apt-get18:59:072
25869998871,8cyclictest0-21swapper/200:30:262
2586999884,34cyclictest29408-21fschecks_count23:39:072
2586999884,32cyclictest8243-21ssh23:04:112
2586999883,35cyclictest352-21runrttasks22:39:522
2586999883,34cyclictest11049-21ssh21:20:142
2586999883,34cyclictest11049-21ssh21:20:142
2586999883,31cyclictest14917-21mailstats23:14:182
2586999883,31cyclictest14917-21mailstats23:14:182
25863998859,21cyclictest32751-21diskmemload21:14:211
2586999874,27cyclictest23709-21ssh00:24:022
2586999873,36cyclictest1911-21/usr/sbin/munin22:54:122
2586999873,35cyclictest18911-21fschecks_time21:34:082
2586999873,33cyclictest15214-21fschecks_time20:09:082
2586999873,32cyclictest4666-21grep19:34:212
2586999873,32cyclictest16590-21cat21:29:232
2586999873,32cyclictest16590-21cat21:29:232
25869998730,10cyclictest0-21swapper/223:49:132
2586999872,38cyclictest21397-21sshd22:31:572
25863998766,14cyclictest19676-21snmpd23:16:011
25863998766,14cyclictest19676-21snmpd23:16:011
25863998763,16cyclictest28604-21ssh22:44:061
2586999864,34cyclictest14588-21latency_hist00:08:492
2586999863,36cyclictest6695-21grep23:54:212
2586999863,36cyclictest6695-21grep23:54:212
2586999854,32cyclictest9508-21switchtime23:59:192
2586999853,30cyclictest32256-21timerandwakeup22:49:202
2586999853,30cyclictest32256-21timerandwakeup22:49:202
2587599847,44cyclictest0-21swapper/300:30:063
2587599843,64cyclictest0-21swapper/323:04:493
25869998330,10cyclictest14492-21sh00:08:492
25863998353,19cyclictest32751-21diskmemload22:24:201
2586399822,62cyclictest25796-21kworker/u17:3+xprtiod22:05:361
25863998049,22cyclictest12735-21grep00:04:251
25863998049,22cyclictest12735-21grep00:04:251
25863997812,27cyclictest0-21swapper/121:19:161
25863997812,27cyclictest0-21swapper/121:19:161
25863997717,24cyclictest20828-21ps23:24:181
25863997716,26cyclictest14540-21taskset22:19:271
25863997656,13cyclictest5052-21latency_hist19:38:501
25863997559,10cyclictest24000-21idleruntime00:24:091
25863997549,18cyclictest13321-21latency_hist20:03:491
25863997521,28cyclictest0-21swapper/122:59:201
25863997459,10cyclictest27648-21runrttasks23:35:271
25863997455,10cyclictest0-21swapper/123:53:481
2586399737,39cyclictest16020-21kworker/u17:2+ext4-rsv-conversion20:19:131
25863997345,20cyclictest6610-21timerandwakeup23:54:201
25863997345,20cyclictest6610-21timerandwakeup23:54:201
25863997310,26cyclictest0-21swapper/123:48:481
25863997245,21cyclictest32751-21diskmemload00:14:221
25863997245,21cyclictest32751-21diskmemload00:14:221
2586399722,46cyclictest352-21runrttasks23:40:291
25863997211,22cyclictest0-21swapper/119:54:141
25863997152,12cyclictest300-21rs:main1
25863997137,9cyclictest181rcu_preempt20:29:091
25863997118,22cyclictest24525-21latency_hist21:43:491
25863997118,22cyclictest24525-21latency_hist21:43:491
25863997054,11cyclictest32751-21diskmemload23:21:511
25863997038,23cyclictest16743-21kworker/u19:0-rpciod21:04:231
25863997014,23cyclictest0-21swapper/123:11:021
25863996947,16cyclictest29120-21unixbench_singl20:49:211
25863996947,16cyclictest29120-21unixbench_singl20:49:211
2586399694,43cyclictest17504-21latency_hist00:13:481
25863996940,21cyclictest9384-21cat19:49:201
25863996940,21cyclictest9384-21cat19:49:201
25863996939,16cyclictest0-21swapper/119:04:101
25863996852,8cyclictest23870-21ssh22:35:251
25863996852,8cyclictest23870-21ssh22:35:251
25863996848,11cyclictest0-21swapper/119:23:461
2586399684,29cyclictest0-21swapper/122:09:081
25863996839,21cyclictest19409-21cut21:34:201
25863996821,19cyclictest24924-21kworker/u18:0-rpciod22:57:431
25863996816,20cyclictest2895-21taskset22:01:081
2586399678,24cyclictest12620-21/usr/sbin/munin21:24:141
25863996753,10cyclictest25845-21cyclictest23:29:181
25863996753,10cyclictest25845-21cyclictest23:29:181
25863996751,9cyclictest0-21swapper/122:33:451
25863996742,13cyclictest0-21swapper/120:43:461
2586399673,46cyclictest30582-21timerandwakeup20:54:221
2586399665,52cyclictest30646-21ssh21:54:001
25863996651,8cyclictest0-21swapper/120:59:171
25863996651,8cyclictest0-21swapper/120:59:171
25863996649,13cyclictest21458-21kworker/u19:1+events_unbound21:13:461
25863996649,13cyclictest21458-21kworker/u19:1+events_unbound21:13:461
25863996644,11cyclictest0-21swapper/120:24:221
25863996640,20cyclictest14147-21timerandwakeup20:04:211
2586399663,47cyclictest16335-21modprobe20:13:461
2586399663,41cyclictest31048-21modprobe19:18:461
25863996547,11cyclictest0-21swapper/121:29:191
25863996547,11cyclictest0-21swapper/121:29:191
25863996541,16cyclictest0-21swapper/123:59:101
25863996538,17cyclictest465-21apt-get19:24:011
2586399653,47cyclictest2658-21grep19:29:171
25863996518,38cyclictest17209-21grep20:14:171
25863996513,42cyclictest31595-21grep22:49:061
25863996513,42cyclictest31595-21grep22:49:061
2439026510,35sleep30-21swapper/318:59:083
25863996437,21cyclictest27935-21ssh21:49:071
2586399643,40cyclictest27877-21modprobe20:48:461
2586399643,20cyclictest24739-21ssh21:44:031
2586399642,43cyclictest7343-21latency19:44:111
2586399642,42cyclictest26263-21ls22:39:191
25863996319,11cyclictest181rcu_preempt00:19:061
2586399623,37cyclictest24655-21modprobe20:38:461
2587599609,41cyclictest0-21swapper/322:14:073
2587599609,41cyclictest0-21swapper/322:14:073
2587599603,39cyclictest26743-21chrt20:44:083
25863996026,19cyclictest3712-21apt-get19:33:561
2587599594,38cyclictest4658-21chrt19:34:213
2587599592,41cyclictest26780-21taskset22:40:103
2587599573,37cyclictest2284-21chrt19:29:073
2587599564,37cyclictest6654-21chrt21:13:573
2587599563,36cyclictest19560-21taskset22:29:043
2587599562,39cyclictest29594-21chrt21:51:523
2587599562,36cyclictest20469-21taskset20:24:113
2587599553,38cyclictest11217-21taskset21:20:433
2587599553,38cyclictest11217-21taskset21:20:433
2587599553,37cyclictest31345-21taskset19:19:023
2587599553,36cyclictest6371-21chrt23:00:203
2587599553,36cyclictest24500-21taskset00:24:213
2587599553,36cyclictest13133-21chrt20:03:463
2587599553,35cyclictest9082-21chrt19:49:163
2587599553,35cyclictest9082-21chrt19:49:163
2587599552,39cyclictest8100-21taskset23:58:033
2587599552,39cyclictest8100-21taskset23:58:033
2587599552,38cyclictest20672-21chrt00:18:583
2587599552,37cyclictest32333-21taskset23:44:053
2587599544,36cyclictest20319-21chrt23:24:033
2587599543,38cyclictest17626-21chrt20:14:233
2587599543,37cyclictest25364-21taskset20:39:103
2587599543,37cyclictest15321-21chrt21:28:453
2587599543,36cyclictest23726-21taskset23:29:143
2587599543,36cyclictest23726-21taskset23:29:143
2587599534,34cyclictest29247-21chrt22:44:203
2587599533,36cyclictest6799-21chrt19:43:493
2587599533,35cyclictest26320-21chrt19:04:083
2587599533,34cyclictest592-21chrt21:03:563
2587599533,34cyclictest4156-21taskset21:09:153
2587599533,34cyclictest4156-21taskset21:09:153
2587599532,36cyclictest9397-21chrt22:12:343
2587599532,36cyclictest5587-21chrt23:53:493
2587599532,35cyclictest20233-21chrt21:35:473
2587599532,34cyclictest2510-21chrt22:00:023
2587599523,37cyclictest9913-21chrt23:59:453
2587599523,37cyclictest3952-21chrt22:56:563
2587599523,37cyclictest24845-21chrt20:38:493
2587599523,36cyclictest30784-21taskset21:54:063
2587599523,34cyclictest21822-21chrt20:29:053
2587599523,34cyclictest16262-21taskset21:29:183
2587599523,34cyclictest16262-21taskset21:29:183
2587599523,33cyclictest30240-21chrt23:39:233
2587599523,32cyclictest16947-21taskset00:12:423
2587599522,36cyclictest12021-21chrt00:04:113
2587599522,36cyclictest12021-21chrt00:04:113
2587599522,35cyclictest18873-21chrt22:28:153
2587599522,34cyclictest23093-21chrt21:40:253
2587599522,34cyclictest23093-21chrt21:40:253
2587599513,34cyclictest29604-21chrt20:53:483
2587599513,34cyclictest29604-21chrt20:53:483
2587599513,32cyclictest1691-21taskset22:53:483
2587599513,32cyclictest1691-21taskset22:53:483
2587599512,38cyclictest30164-21chrt20:54:123
2587599512,33cyclictest13718-21chrt20:04:103
2587599512,32cyclictest22517-21chrt22:34:023
2587599512,32cyclictest22517-21chrt22:34:023
2587599512,31cyclictest14024-21chrt22:19:153
2587599503,35cyclictest27793-21chrt19:08:543
2587599503,32cyclictest11488-21chrt23:09:083
2587599503,30cyclictest31479-21chrt20:59:023
2587599503,30cyclictest31479-21chrt20:59:023
2587599503,29cyclictest0-21swapper/320:19:093
2587599503,28cyclictest8304-21chrt19:48:473
2587599502,32cyclictest7081-21chrt22:08:503
2587599494,33cyclictest14196-21taskset23:13:583
2587599494,33cyclictest14196-21taskset23:13:583
2587599493,33cyclictest18637-21taskset00:14:203
2587599493,33cyclictest18637-21taskset00:14:203
2587599493,33cyclictest17513-21chrt23:19:093
2587599493,28cyclictest29703-21taskset19:13:563
2587599492,32cyclictest26199-21chrt21:45:443
2587599482,33cyclictest15211-21chrt20:09:083
25875994815,16cyclictest0-21swapper/323:34:113
2587599452,30cyclictest1048-21chrt19:24:173
2587599442,26cyclictest10685-21chrt19:54:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional