You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 21:26
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Sun Sep 06, 2026 12:44:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2615521389,86sleep126206-21cpuspeed07:05:051
27621991184,46cyclictest26706-21ssh10:40:000
27621991184,46cyclictest26706-21ssh10:40:000
276219911537,8cyclictest0-21swapper/012:05:110
276219911443,18cyclictest27970-21latency_hist11:34:490
27621991122,42cyclictest12256-21fschecks_count09:25:060
276219911037,13cyclictest0-21swapper/009:09:330
27621991084,43cyclictest4930-21apt-get10:05:080
27621991084,43cyclictest4930-21apt-get10:05:080
27621991083,35cyclictest12875-21cstates11:10:030
27621991083,35cyclictest12875-21cstates11:10:030
27621991083,32cyclictest3646-21unixbench_multi09:10:260
27621991083,32cyclictest3646-21unixbench_multi09:10:260
26308210552,25sleep00-21swapper/007:05:120
276379910342,54cyclictest0-21swapper/212:10:082
276379910238,10cyclictest0-21swapper/208:50:232
2621021019,62sleep226264-21cstates07:05:062
276379910042,9cyclictest0-21swapper/210:44:492
276379910042,9cyclictest0-21swapper/210:44:492
276379910040,55cyclictest27611-21cyclictest11:10:112
276379910040,55cyclictest27611-21cyclictest11:10:112
276379910034,9cyclictest0-21swapper/209:10:112
276379910034,9cyclictest0-21swapper/209:10:112
27637999979,14cyclictest27611-21cyclictest11:25:112
27637999939,14cyclictest0-21swapper/208:35:202
27637999938,8cyclictest0-21swapper/210:10:252
27637999829,12cyclictest0-21swapper/210:30:082
27637999826,13cyclictest0-21swapper/211:04:502
27637999826,13cyclictest0-21swapper/211:04:502
27637999741,8cyclictest0-21swapper/210:37:202
27637999736,9cyclictest0-21swapper/207:20:102
27637999736,9cyclictest0-21swapper/207:20:102
2763799973,36cyclictest31533-21sh11:40:112
2763799973,36cyclictest31533-21sh11:40:112
2762999973,78cyclictest14520-21sshd10:20:341
2762999973,78cyclictest14520-21sshd10:20:341
2763799962,45cyclictest18328-21munin-plugin-st08:19:462
27637999543,46cyclictest19008-21sh11:20:042
27637999543,46cyclictest19008-21sh11:20:042
27637999537,7cyclictest0-21swapper/208:55:222
27637999537,7cyclictest0-21swapper/208:55:222
27637999536,9cyclictest0-21swapper/209:00:222
27637999535,12cyclictest0-21swapper/209:44:312
27637999535,12cyclictest0-21swapper/209:44:312
2763799953,34cyclictest10733-21grep07:55:082
27637999529,12cyclictest0-21swapper/212:39:482
27621999568,18cyclictest326-21systemd-logind08:44:470
27621999568,18cyclictest326-21systemd-logind08:44:470
27637999440,8cyclictest0-21swapper/208:24:482
2763799943,34cyclictest14853-21sort12:05:202
2763799942,34cyclictest28288-21cut11:35:082
2763799942,33cyclictest14827-21fschecks_count10:20:482
2763799942,33cyclictest14827-21fschecks_count10:20:482
27637999362,23cyclictest2285-21diskmemload09:56:192
2763799933,34cyclictest6204-21latency09:15:132
27637999328,14cyclictest0-21swapper/211:45:012
2763799932,37cyclictest12529-21latency_hist11:09:482
27637999264,21cyclictest5285-21latency11:50:122
2763799923,35cyclictest29481-21fschecks_count07:14:552
2763799923,35cyclictest29481-21fschecks_count07:14:552
2763799923,33cyclictest13700-21latency_hist12:04:482
2763799922,35cyclictest11274-21diskmemload09:23:462
2763799922,34cyclictest6033-21ls10:05:412
2763799922,34cyclictest6033-21ls10:05:412
2763799922,34cyclictest15583-21cut08:10:072
2762199922,63cyclictest23731-21ssh12:20:250
2762199922,63cyclictest23731-21ssh12:20:250
27637999164,18cyclictest29247-21sh09:51:422
2763799913,36cyclictest25745-21mailstats11:30:192
27637999037,11cyclictest2285-21diskmemload10:15:332
2763799903,38cyclictest28451-21mailstats07:10:192
2763799903,35cyclictest21073-21timerwakeupswit08:25:222
2763799902,40cyclictest25111-21ssh12:21:512
2763799902,40cyclictest25111-21ssh12:21:512
2762999902,78cyclictest1291-21tr09:59:461
27637998960,20cyclictest28364-21latency_hist08:49:492
2763799894,34cyclictest2510-21fschecks_time07:30:072
2763799894,34cyclictest2510-21fschecks_time07:30:072
2763799893,38cyclictest12996-21unixbench_multi09:25:212
2763799893,36cyclictest1418-21grep10:50:222
2763799893,36cyclictest1418-21grep10:50:222
2763799893,35cyclictest1962-21sshd07:29:412
2763799883,34cyclictest26598-21/usr/sbin/munin12:25:192
2763799882,37cyclictest4388-21latency_hist10:04:502
2763799882,34cyclictest18048-21latency_hist09:34:492
2763799882,34cyclictest18048-21latency_hist09:34:492
27629998859,21cyclictest21199-21sh10:30:201
2763799874,33cyclictest16511-21grep11:15:192
2763799874,30cyclictest32728-21latency_hist10:49:492
2763799873,33cyclictest10316-21latency_hist07:54:482
27629998770,10cyclictest2417-21ssh10:00:221
2763799864,33cyclictest343-21runrttasks09:45:452
27637998637,11cyclictest22299-21ntp_kernel_err08:30:132
27637998637,11cyclictest22299-21ntp_kernel_err08:30:132
2763799863,37cyclictest32715-21cat12:34:502
2763799863,35cyclictest14329-21proc_pri08:05:182
2763799863,35cyclictest14329-21proc_pri08:05:182
2763799862,33cyclictest21139-21latency_hist09:39:482
27621998658,19cyclictest30017-21latency_hist08:54:500
2763799854,33cyclictest29695-21/usr/sbin/munin12:30:202
27629998558,19cyclictest17369-21sh09:33:171
27629998558,19cyclictest17369-21sh09:33:171
2763799844,31cyclictest25426-21kernelversion08:40:122
2763799844,31cyclictest25426-21kernelversion08:40:122
2763799843,33cyclictest5919-21fschecks_time07:40:102
2763799843,33cyclictest5919-21fschecks_time07:40:102
2763799843,33cyclictest18058-21ssh10:25:182
2764199823,35cyclictest12658-21chrt09:25:173
2763799824,36cyclictest10783-21ssh11:59:492
2763799824,36cyclictest10783-21ssh11:59:492
27629998262,12cyclictest0-21swapper/110:55:011
27629998257,18cyclictest15124-21vmstat12:05:241
2762199828,54cyclictest10091-21rm11:05:150
27629998152,23cyclictest24358-21latency_hist09:44:501
27629998152,23cyclictest24358-21latency_hist09:44:501
2762999814,50cyclictest9975-21apt-get11:05:001
27621998163,11cyclictest29902-21cpuspeed07:15:060
27621998163,11cyclictest29902-21cpuspeed07:15:060
2763799803,36cyclictest5052-21unixbench_singl07:36:112
27629998051,21cyclictest5427-21sendmail07:39:471
27629998050,22cyclictest24776-21cut08:39:481
27629998020,11cyclictest0-21swapper/108:09:571
27621998063,8cyclictest0-21swapper/010:49:560
27621998063,8cyclictest0-21swapper/010:49:560
27621998048,10cyclictest181rcu_preempt12:35:040
27629997960,13cyclictest27611-21cyclictest10:05:511
27629997960,13cyclictest27611-21cyclictest10:05:511
2762999793,52cyclictest12102-21grep10:15:231
27621997962,8cyclictest0-21swapper/011:20:040
27621997962,8cyclictest0-21swapper/011:20:040
2762199793,44cyclictest367-21cron07:24:460
2762199793,44cyclictest367-21cron07:24:460
27621997861,11cyclictest524-21in:imuxsock09:19:460
27621997861,11cyclictest524-21in:imuxsock09:04:470
27621997859,14cyclictest27611-21cyclictest11:37:340
27629997758,12cyclictest8895-21unixbench_multi11:55:231
27629997758,12cyclictest8895-21unixbench_multi11:55:231
27629997751,14cyclictest0-21swapper/107:14:581
27629997751,14cyclictest0-21swapper/107:14:581
27629997750,17cyclictest0-21swapper/108:49:491
2762999773,29cyclictest31-21ksoftirqd/112:15:231
2762999773,29cyclictest31-21ksoftirqd/112:15:231
27621997738,9cyclictest3856-21fschecks_time10:55:080
27629997652,16cyclictest2285-21diskmemload10:29:481
27621997653,16cyclictest27611-21cyclictest10:16:350
27621997649,18cyclictest8405-21netstat11:55:140
27621997649,18cyclictest8405-21netstat11:55:140
2762199763,40cyclictest23787-21taskset11:27:230
27629997557,11cyclictest27611-21cyclictest09:13:161
27629997557,11cyclictest27611-21cyclictest09:13:161
27629997512,55cyclictest7551-21apt11:54:461
27621997547,20cyclictest0-21swapper/012:34:460
27621997547,20cyclictest0-21swapper/012:34:460
67762740,13sleep3451rcuc/309:15:243
2764199743,25cyclictest27311-21chrt10:40:173
2764199743,25cyclictest27311-21chrt10:40:173
27637997453,15cyclictest23056-21apt-key12:19:472
27637997453,15cyclictest23056-21apt-key12:19:472
27629997444,20cyclictest2285-21diskmemload10:14:491
27621997445,16cyclictest0-21swapper/010:30:150
2764199733,34cyclictest3841-21taskset07:34:493
2764199733,34cyclictest3841-21taskset07:34:493
2764199733,32cyclictest18061-21chrt11:18:293
2764199732,32cyclictest22057-21chrt10:31:373
2764199732,30cyclictest7333-21chrt10:08:503
2764199732,30cyclictest7333-21chrt10:08:503
2762999736,32cyclictest31-21ksoftirqd/108:23:191
27629997351,16cyclictest27562-21timerwakeupswit12:25:221
27629997351,15cyclictest200-21systemd-journal11:31:481
27629997346,18cyclictest0-21swapper/109:04:491
27629997336,26cyclictest0-21swapper/108:25:191
27621997356,8cyclictest0-21swapper/012:10:100
27621997356,12cyclictest22996-21latency_hist12:19:480
27621997356,12cyclictest22996-21latency_hist12:19:480
27621997354,12cyclictest343-21runrttasks09:35:150
2762199733,38cyclictest17079-21awk10:24:460
2762199733,38cyclictest17079-21awk10:24:460
27621997331,12cyclictest181rcu_preempt09:24:500
2764199723,27cyclictest30546-21chrt10:45:193
27637997242,22cyclictest7649-21kernelversion07:45:122
27637997242,22cyclictest7649-21kernelversion07:45:122
27629997258,10cyclictest27611-21cyclictest12:21:391
27629997258,10cyclictest27611-21cyclictest12:21:391
27621997247,14cyclictest3775-21apt-key07:34:490
27621997247,14cyclictest3775-21apt-key07:34:490
27621997244,21cyclictest0-21swapper/007:59:490
27621997242,23cyclictest0-21swapper/011:00:230
27621997242,23cyclictest0-21swapper/011:00:230
27621997239,24cyclictest17327-21kernelversion08:15:110
27621997237,26cyclictest2651-21kworker/u17:2+nfsiod09:40:590
27621997237,26cyclictest2651-21kworker/u17:2+nfsiod09:40:590
27621997233,29cyclictest0-21swapper/011:49:490
2764199712,25cyclictest3715-21chrt10:03:033
27637997150,11cyclictest0-21swapper/209:09:502
27629997151,14cyclictest31234-21cpuspeed11:40:041
27629997151,14cyclictest31234-21cpuspeed11:40:041
27629997151,13cyclictest27611-21cyclictest07:25:101
27629997149,16cyclictest25934-21sh09:46:131
2762999712,29cyclictest31-21ksoftirqd/109:25:261
27629997115,24cyclictest31-21ksoftirqd/107:51:281
2762999711,36cyclictest31-21ksoftirqd/109:14:541
27621997147,15cyclictest200-21systemd-journal10:35:230
27621997146,16cyclictest200-21systemd-journal09:59:460
27621997138,10cyclictest0-21swapper/011:50:110
2762199713,31cyclictest4836-21apt-key10:04:510
2764199702,26cyclictest32762-21chrt11:41:473
2764199702,26cyclictest32762-21chrt11:41:473
27637997050,11cyclictest0-21swapper/208:04:502
27629997053,11cyclictest200-21systemd-journal07:40:141
27629997053,11cyclictest200-21systemd-journal07:40:141
27629997051,13cyclictest4505-21munin-plugin-st11:49:451
27629997047,14cyclictest29286-21kworker/u19:0-rpciod11:39:491
27629997043,19cyclictest0-21swapper/108:04:501
27629997042,12cyclictest0-21swapper/111:25:221
27621997053,10cyclictest0-21swapper/010:25:080
2762199703,36cyclictest5069-21grep07:36:110
27621997017,38cyclictest16697-21cron08:14:460
2764199693,28cyclictest681-21chrt12:35:123
27629996947,15cyclictest8953-21sh09:20:041
2762999693,45cyclictest3216-21latency_hist10:54:491
2762999693,45cyclictest3216-21latency_hist10:54:491
27629996921,12cyclictest291rcuc/107:30:061
27629996921,12cyclictest291rcuc/107:30:061
27621996954,10cyclictest25089-21ntp_kernel_err09:45:150
27621996948,15cyclictest526-21rs:main0
11612690,7sleep00-21swapper/007:25:150
2763799683,25cyclictest3534-21apt-get10:55:012
2762999688,10cyclictest13232-21ssh12:03:321
27629996843,14cyclictest0-21swapper/111:10:101
27629996843,14cyclictest0-21swapper/111:10:101
27629996814,31cyclictest31-21ksoftirqd/108:41:521
27629996814,31cyclictest31-21ksoftirqd/108:41:521
27621996849,14cyclictest7995-21/usr/sbin/munin10:10:180
2762199684,31cyclictest21904-21ls08:30:020
2762199684,31cyclictest21904-21ls08:30:020
2762199684,29cyclictest29765-21/usr/sbin/munin10:45:030
27621996840,19cyclictest29421-21sh12:29:460
27621996834,9cyclictest0-21swapper/012:02:200
2762199683,47cyclictest15451-21fschecks_time09:30:090
2762199683,47cyclictest15451-21fschecks_time09:30:090
202562680,6sleep30-21swapper/311:21:043
202562680,6sleep30-21swapper/311:21:043
111492680,8sleep30-21swapper/309:23:263
2762999674,41cyclictest343-21runrttasks08:52:121
27629996744,13cyclictest0-21swapper/109:50:241
27629996742,18cyclictest30185-21latency12:30:121
27629996742,18cyclictest30185-21latency12:30:121
27629996740,16cyclictest0-21swapper/107:45:121
27629996740,16cyclictest0-21swapper/107:45:121
2762999673,40cyclictest32753-21latency_hist10:49:491
27629996727,27cyclictest23150-21modprobe08:34:471
27629996727,27cyclictest23150-21modprobe08:34:471
2762999672,24cyclictest31-21ksoftirqd/111:04:201
2762999672,24cyclictest31-21ksoftirqd/111:04:201
27629996712,22cyclictest31-21ksoftirqd/111:23:331
27629996712,22cyclictest31-21ksoftirqd/111:23:331
27621996721,32cyclictest18540-21modprobe11:19:450
2764199663,24cyclictest30960-21chrt12:30:463
2764199663,24cyclictest30960-21chrt12:30:463
27629996649,10cyclictest0-21swapper/107:19:521
27629996649,10cyclictest0-21swapper/107:19:521
2762999664,43cyclictest31-21ksoftirqd/108:56:251
2762999664,43cyclictest31-21ksoftirqd/108:56:251
2762999663,42cyclictest376-21cpu12:35:051
2762999663,26cyclictest31-21ksoftirqd/111:16:291
2762999662,35cyclictest31-21ksoftirqd/109:38:511
2762999662,21cyclictest31-21ksoftirqd/107:11:581
27629996612,37cyclictest13959-21fschecks_count08:05:081
27629996612,37cyclictest13959-21fschecks_count08:05:081
2762199664,41cyclictest12372-21fschecks_time08:00:080
27621996643,15cyclictest20472-21fschecks_time08:25:090
27621996614,36cyclictest10244-21cron07:54:460
27641996510,31cyclictest0-21swapper/311:05:063
2762999656,29cyclictest31-21ksoftirqd/112:14:051
2762999652,25cyclictest26306-21ssh10:39:421
2762199653,29cyclictest19953-21idleruntime-cro08:24:460
27621996516,22cyclictest31330-21seq11:40:060
27621996516,22cyclictest31330-21seq11:40:060
137662652,53sleep013814-21awk08:05:030
137662652,53sleep013814-21awk08:05:030
2762999644,39cyclictest26669-21/usr/sbin/munin10:40:091
2762999644,39cyclictest26669-21/usr/sbin/munin10:40:091
27621996446,12cyclictest26992-21cpu08:46:020
2762199643,40cyclictest28614-21timerandwakeup07:10:210
2762199642,47cyclictest30110-21/usr/sbin/munin08:55:240
2762199642,47cyclictest30110-21/usr/sbin/munin08:55:240
2644326414,34sleep30-21swapper/307:05:153
27629996348,7cyclictest0-21swapper/109:05:051
27629996344,7cyclictest181rcu_preempt07:59:121
27621996329,27cyclictest0-21swapper/007:44:470
27621996329,27cyclictest0-21swapper/007:44:470
2762999618,21cyclictest31-21ksoftirqd/108:19:371
27621996131,8cyclictest181rcu_preempt07:45:010
27621996131,8cyclictest181rcu_preempt07:45:010
2764199609,19cyclictest0-21swapper/312:20:023
2764199609,19cyclictest0-21swapper/312:20:023
2764199584,37cyclictest13526-21taskset12:04:203
2762199575,41cyclictest23412-21/usr/sbin/munin08:35:090
2764199563,35cyclictest2829-21taskset10:53:463
2764199563,35cyclictest2829-21taskset10:53:463
2764199553,38cyclictest17777-21chrt08:15:213
2764199553,33cyclictest5705-21chrt10:58:123
2764199543,38cyclictest15555-21taskset08:10:073
2764199543,37cyclictest2057-21taskset11:45:073
2764199543,35cyclictest29195-21taskset07:14:493
2764199543,34cyclictest3584-21chrt09:10:253
2764199543,34cyclictest3584-21chrt09:10:253
2764199542,39cyclictest23903-21chrt11:27:413
2764199542,35cyclictest30342-21taskset07:15:173
2764199542,35cyclictest30342-21taskset07:15:173
2764199533,36cyclictest15256-21chrt11:14:113
2764199533,36cyclictest15256-21chrt11:14:113
2764199533,33cyclictest23920-21chrt10:35:103
2764199533,33cyclictest17760-21taskset10:25:093
2764199532,34cyclictest19294-21chrt08:20:193
2764199523,35cyclictest32067-21chrt07:20:203
2764199523,35cyclictest32067-21chrt07:20:203
2764199523,34cyclictest4775-21taskset11:49:533
2764199523,33cyclictest22728-21chrt09:40:573
2764199523,33cyclictest22728-21chrt09:40:573
2764199523,33cyclictest14370-21chrt12:05:093
2764199514,19cyclictest0-21swapper/308:45:183
27641995135,8cyclictest274832sleep309:49:493
2764199513,35cyclictest20503-21chrt08:25:093
2764199513,32cyclictest32176-21chrt09:56:423
2764199513,32cyclictest11288-21chrt10:15:073
2764199513,31cyclictest9178-21chrt10:10:333
2764199512,33cyclictest30118-21chrt09:53:593
2764199503,35cyclictest18094-21chrt09:34:493
2764199503,35cyclictest18094-21chrt09:34:493
2764199503,34cyclictest16566-21chrt10:23:263
2764199503,34cyclictest16566-21chrt10:23:263
2764199503,34cyclictest10755-21taskset07:55:093
2764199503,31cyclictest9871-21chrt07:50:233
2764199502,34cyclictest13103-21chrt08:00:233
2764199502,33cyclictest27654-21chrt12:25:233
2764199494,33cyclictest32020-21chrt09:00:083
2764199483,33cyclictest17563-21chrt12:10:113
2764199483,32cyclictest30916-21chrt11:39:483
2764199482,33cyclictest19148-21chrt09:35:223
2764199482,21cyclictest4123-21chrt07:35:213
2764199474,31cyclictest20851-21chrt12:15:173
2764199474,31cyclictest20851-21chrt12:15:173
2764199474,30cyclictest26143-21taskset11:30:293
2764199473,33cyclictest8842-21taskset11:03:153
2764199473,33cyclictest8842-21taskset11:03:153
2764199473,32cyclictest14524-21taskset08:05:213
2764199473,32cyclictest14524-21taskset08:05:213
2764199442,31cyclictest1673-21chrt07:25:543
2764199434,28cyclictest25694-21chrt08:40:203
2764199434,28cyclictest25694-21chrt08:40:203
2764199424,26cyclictest30135-21taskset08:55:023
2764199424,26cyclictest30135-21taskset08:55:023
2764199423,27cyclictest6147-21taskset07:40:143
2764199423,27cyclictest6147-21taskset07:40:143
2764199423,27cyclictest28964-21taskset08:50:123
2764199413,25cyclictest8296-21taskset11:55:123
2764199413,25cyclictest8296-21taskset11:55:123
2764199403,27cyclictest24579-21chrt08:37:333
2764199402,28cyclictest1084-21taskset09:05:063
2764199393,25cyclictest7514-21taskset07:45:083
2764199393,25cyclictest7514-21taskset07:45:083
2764199393,25cyclictest21827-21taskset08:29:593
2764199393,25cyclictest21827-21taskset08:29:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional