You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 19:28
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sun Nov 23, 2025 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1276721180,5sleep012766-21ssh10:30:020
24475991144,91cyclictest32144-21apt-get09:09:371
244759911063,39cyclictest0-21swapper/110:09:541
244659910689,13cyclictest22466-21kworker/u18:2+flush-179:011:24:360
244759910445,11cyclictest0-21swapper/109:54:511
244759910338,11cyclictest0-21swapper/111:59:501
24475991033,73cyclictest3751-21apt-get12:09:331
244759910240,13cyclictest0-21swapper/107:24:531
244759910240,13cyclictest0-21swapper/107:24:531
244759910140,12cyclictest0-21swapper/107:54:171
1523721010,6sleep30-21swapper/310:35:283
24494991003,58cyclictest12694-21chrt12:24:363
244759910044,9cyclictest0-21swapper/108:14:551
244759910044,9cyclictest0-21swapper/108:14:551
244759910038,12cyclictest0-21swapper/108:54:551
244759910031,13cyclictest0-21swapper/109:04:391
24475999936,11cyclictest0-21swapper/110:54:171
24475999836,13cyclictest0-21swapper/107:59:201
24475999834,16cyclictest0-21swapper/110:44:191
24475999834,16cyclictest0-21swapper/110:44:191
2447599982,39cyclictest11235-21ntp_kernel_err09:29:451
24475999749,11cyclictest207-21systemd-journal08:48:541
2447599973,41cyclictest10084-21apt-get11:24:321
24475999734,12cyclictest0-21swapper/110:44:381
2447599973,30cyclictest21302-21/usr/sbin/munin09:49:391
2447599972,37cyclictest24859-21sort11:49:341
24465999771,17cyclictest1-21systemd12:09:180
24475999650,39cyclictest3099-21diskmemload10:14:221
24475999636,9cyclictest0-21swapper/108:24:431
2447599953,37cyclictest30079-21latency_hist09:04:201
24475999439,12cyclictest0-21swapper/109:14:351
24465999449,10cyclictest181rcu_preempt08:49:290
24465999369,16cyclictest21153-21latency_hist09:49:190
24465999369,16cyclictest21153-21latency_hist09:49:190
24475999263,21cyclictest10711-21kworker/u17:4+events_unbound11:49:161
2447599923,42cyclictest25593-21fschecks_count08:49:381
2447599923,40cyclictest19211-21fschecks_time08:29:401
24475999221,29cyclictest7307-21grep12:14:511
2446599913,76cyclictest29539-21kworker/u19:1+ext4-rsv-conversion10:35:110
24488999059,23cyclictest20542-21kworker/u19:1+nfsiod12:04:552
24488998862,18cyclictest28058-21sendmail07:19:182
24465998833,17cyclictest181rcu_preempt11:54:340
24475998741,12cyclictest9485-21sh09:26:291
24475998741,12cyclictest9485-21sh09:26:291
2446599877,64cyclictest15735-21apt-get08:19:400
2446599874,45cyclictest24410-21kworker/0:2+events_power_efficient07:39:200
2446599874,45cyclictest24410-21kworker/0:2+events_power_efficient07:39:200
24465998665,15cyclictest24456-21cyclictest10:22:160
24488998559,17cyclictest11613-21/usr/sbin/munin10:29:352
24475998561,15cyclictest11411-21ssh10:29:191
24465998559,18cyclictest314-21in:imuxsock08:04:400
24465998559,18cyclictest314-21in:imuxsock08:04:400
24465998543,8cyclictest181rcu_preempt09:09:460
24475998462,16cyclictest24456-21cyclictest12:39:201
2447599844,36cyclictest355-21runrttasks08:07:521
2447599844,36cyclictest355-21runrttasks08:07:521
2447599842,57cyclictest14687-21apt-get10:35:131
24465998462,15cyclictest24456-21cyclictest12:23:120
24465998446,30cyclictest7595-21modprobe09:24:170
2446599843,57cyclictest12679-21ssh12:24:340
2447599839,66cyclictest12630-21apt-get12:24:341
24475998366,11cyclictest16591-21unixbench_multi12:29:551
24475998356,19cyclictest13733-21latency_hist08:14:201
24475998356,19cyclictest13733-21latency_hist08:14:201
2448899827,62cyclictest32200-21rm09:09:392
2448899812,69cyclictest13142-21/usr/sbin/munin12:25:312
24475998137,15cyclictest24456-21cyclictest11:42:321
24475998137,15cyclictest24456-21cyclictest11:42:321
2447599812,33cyclictest355-21runrttasks07:19:211
24488998056,17cyclictest309-21systemd-logind09:54:172
24488998054,13cyclictest0-21swapper/209:59:402
24475998060,15cyclictest691-21cron11:09:211
24465998057,14cyclictest2929-21latency_hist10:14:210
24465998048,23cyclictest355-21runrttasks12:18:090
2448899797,62cyclictest3200-21apt-get12:09:312
24488997952,16cyclictest58-21kcompactd010:19:382
24475997945,23cyclictest3503-21apt-key12:09:211
2446599792,54cyclictest3751-21apt-get12:09:420
241642796,54sleep00-21swapper/007:07:050
24465997854,13cyclictest0-21swapper/007:39:420
24465997853,17cyclictest24977-21ssh11:49:390
24488997736,8cyclictest2633-21kworker/u18:1+flush-179:008:09:352
24488997736,8cyclictest2633-21kworker/u18:1+flush-179:008:09:352
2448899773,53cyclictest2053-21apt-get07:39:312
24475997756,16cyclictest20756-21fschecks_count08:34:391
24475997750,19cyclictest15801-21sh11:33:501
24465997756,14cyclictest15403-21apt-key08:19:210
24465997756,14cyclictest15403-21apt-key08:19:210
24465997734,8cyclictest181rcu_preempt10:27:320
2446599769,31cyclictest22068-21/usr/sbin/munin09:49:560
24465997654,15cyclictest1-21systemd11:42:040
24465997654,15cyclictest1-21systemd11:42:040
2446599763,35cyclictest20187-21latency_hist10:44:190
2446599763,35cyclictest20187-21latency_hist10:44:190
24488997555,11cyclictest3911-21ssh10:15:252
24488997551,18cyclictest16497-21timerwakeupswit12:29:552
24488997551,16cyclictest16529-21timerandwakeup08:20:162
24475997557,10cyclictest207-21systemd-journal09:44:171
24475997553,11cyclictest0-21swapper/107:34:531
24475997553,11cyclictest0-21swapper/107:34:531
24465997553,14cyclictest5582-21idleruntime-cro10:19:170
24488997442,23cyclictest21664-21modprobe11:44:162
24488997442,23cyclictest21664-21modprobe11:44:162
24475997457,11cyclictest32427-21timerwakeupswit07:29:531
24475997437,30cyclictest26710-21mailstats10:54:521
24475997430,32cyclictest8303-21mailstats11:19:501
24465997455,14cyclictest24456-21cyclictest10:04:510
24465997452,15cyclictest13079-21latency_hist09:34:190
2433127411,36sleep124356-21munin-run07:09:181
24488997354,12cyclictest0-21swapper/211:25:132
24488997352,15cyclictest316-21rs:main2
24488997346,18cyclictest0-21swapper/211:29:342
2448899733,49cyclictest32486-21/usr/sbin/munin10:09:432
24475997356,11cyclictest32004-21runrttasks10:08:461
24475997351,16cyclictest15886-21latency_hist09:39:201
24475997350,13cyclictest19029-21latency_hist11:39:201
24475997350,13cyclictest19029-21latency_hist11:39:201
2446599738,44cyclictest24740-21ssh09:54:470
24465997333,9cyclictest181rcu_preempt11:14:550
2446599732,40cyclictest16275-21fschecks_time09:39:400
24488997256,10cyclictest24456-21cyclictest09:33:592
24488997248,17cyclictest5288-21grep11:14:462
24488997245,20cyclictest7497-21threads12:14:542
2448899723,44cyclictest10551-21latency_hist08:04:212
2448899723,44cyclictest10551-21latency_hist08:04:212
2448899723,30cyclictest14687-21apt-get10:34:322
24465997257,9cyclictest14187-21rm09:34:540
24465997257,10cyclictest24456-21cyclictest11:45:050
24465997254,13cyclictest16955-21processes11:34:510
24465997254,13cyclictest16955-21processes11:34:510
24465997251,15cyclictest24456-21cyclictest08:29:170
2446599723,55cyclictest5064-21latency_hist09:19:190
2446599723,35cyclictest8579-21ssh09:24:520
2446599723,35cyclictest8579-21ssh09:24:520
24488997148,15cyclictest369-21ntpd07:19:452
24488997148,15cyclictest369-21ntpd07:19:452
24488997147,14cyclictest0-21swapper/209:44:172
2448899713,48cyclictest24348-21latency_hist07:09:222
2448899713,45cyclictest24840-21grep09:54:512
2448899713,43cyclictest31396-21ssh11:04:192
24475997151,14cyclictest27059-21cpuspeed09:59:331
24475997142,20cyclictest355-21runrttasks07:59:391
24475997142,20cyclictest355-21runrttasks07:59:391
2447599713,25cyclictest4679-21/usr/sbin/munin11:14:541
24465997140,8cyclictest181rcu_preempt12:39:040
2446599713,57cyclictest355-21runrttasks11:12:480
24465997126,33cyclictest29033-21cat08:59:420
2364127116,38sleep30-21swapper/307:05:003
2448899704,45cyclictest355-21runrttasks09:27:342
2448899704,45cyclictest355-21runrttasks09:27:342
2448899704,38cyclictest355-21runrttasks11:50:532
24488997024,7cyclictest381ktimers/209:04:202
24488997022,8cyclictest381ktimers/210:04:372
24475997019,24cyclictest0-21swapper/112:23:161
24465997052,12cyclictest19918-21grep08:29:540
24465997027,11cyclictest181rcu_preempt10:54:400
24488996949,14cyclictest2050-21latency_hist07:39:202
24488996949,14cyclictest2050-21latency_hist07:39:202
24488996949,13cyclictest20958-21ssh10:44:452
24488996937,20cyclictest0-21swapper/209:09:222
2448899693,41cyclictest26181-21latency_hist07:14:212
24475996948,15cyclictest5594-21idleruntime09:19:401
24475996941,20cyclictest18975-21cut09:44:411
24475996941,20cyclictest18975-21cut09:44:411
2447599693,31cyclictest31845-21fschecks_time11:04:391
2446599693,46cyclictest7224-21latency_hist07:54:200
2446599693,32cyclictest30986-21ssh11:59:510
2446599692,36cyclictest27149-21/usr/sbin/munin09:59:340
24465996917,26cyclictest23530-21munin-plugin-st08:44:160
24465996913,38cyclictest15-21ksoftirqd/011:06:050
89662680,7sleep30-21swapper/311:20:543
2449499684,38cyclictest3870-21taskset12:09:493
24488996852,11cyclictest58-21kcompactd009:19:332
2448899684,50cyclictest355-21runrttasks09:16:352
2448899684,45cyclictest1406-21taskset11:08:542
24488996843,15cyclictest23835-21diskmemload10:49:502
2448899683,46cyclictest4422-21fschecks_time07:44:402
2448899683,42cyclictest19138-21fschecks_count08:29:392
2448899683,32cyclictest25781-21kernelversion08:49:422
2448899682,46cyclictest8246-21grep11:19:492
24488996820,14cyclictest0-21swapper/208:19:212
24488996820,14cyclictest0-21swapper/208:19:212
24475996843,13cyclictest0-21swapper/108:39:391
2447599683,56cyclictest2580-21apt-get07:39:311
2447599683,37cyclictest5905-21/usr/sbin/munin10:19:571
24465996851,12cyclictest32135-21processes07:29:490
24465996848,13cyclictest369-21ntpd11:29:460
2446599683,52cyclictest24195-21sed10:49:540
24465996826,31cyclictest20791-21fschecks_time08:34:390
24488996746,13cyclictest207-21systemd-journal11:46:052
2448899674,40cyclictest355-21runrttasks07:51:022
24488996741,17cyclictest11400-21threads08:04:522
24488996741,17cyclictest11400-21threads08:04:522
24488996741,13cyclictest0-21swapper/211:34:462
24488996741,13cyclictest0-21swapper/211:34:462
24488996738,17cyclictest0-21swapper/212:04:192
2448899673,39cyclictest31343-21latency_hist07:29:192
2448899673,39cyclictest31343-21latency_hist07:29:192
24475996750,10cyclictest0-21swapper/110:34:201
24475996746,14cyclictest5760-21latency_hist07:49:211
24475996740,8cyclictest181rcu_preempt10:59:231
24465996751,10cyclictest31331-21ssh11:04:170
24465996750,10cyclictest0-21swapper/007:59:380
24465996750,10cyclictest0-21swapper/007:59:380
24465996736,21cyclictest0-21swapper/010:44:390
2446599673,44cyclictest355-21runrttasks11:24:100
24488996646,14cyclictest18845-21sh12:34:432
24488996643,12cyclictest0-21swapper/212:24:202
24488996640,15cyclictest0-21swapper/208:29:212
2447599664,39cyclictest355-21runrttasks07:19:581
2447599664,39cyclictest355-21runrttasks07:19:581
24475996635,10cyclictest0-21swapper/108:20:001
24465996653,9cyclictest24456-21cyclictest09:04:510
24488996544,15cyclictest24456-21cyclictest10:25:102
2448899654,31cyclictest24011-21fschecks_count08:44:402
24488996543,16cyclictest20991-21latency_hist09:49:192
24488996543,16cyclictest20991-21latency_hist09:49:192
2448899653,43cyclictest18494-21timerwakeupswit10:39:542
2448899653,43cyclictest18494-21timerwakeupswit10:39:542
2448899653,38cyclictest25988-21apt-get10:54:302
2448899652,42cyclictest20752-21fschecks_count08:34:392
2448899650,29cyclictest22944-21find08:39:522
2446599653,49cyclictest27000-21/usr/sbin/munin08:54:490
2448899642,43cyclictest27501-21ntp_kernel_err08:54:452
24465996439,17cyclictest30142-21fschecks_count07:24:380
24465996439,17cyclictest30142-21fschecks_count07:24:380
2446599643,26cyclictest28590-21fschecks_time07:19:380
2446599643,26cyclictest28590-21fschecks_time07:19:380
24488996343,11cyclictest0-21swapper/211:59:202
24488996340,13cyclictest0-21swapper/209:39:062
2448899633,35cyclictest8094-21switchtime07:54:542
2448899632,42cyclictest489-21latency_hist07:34:192
24475996337,18cyclictest0-21swapper/107:09:401
24465996348,10cyclictest58-21kcompactd007:14:270
2446599634,31cyclictest355-21runrttasks07:12:170
24465996333,21cyclictest12513-21cut08:09:410
24465996333,21cyclictest12513-21cut08:09:410
2446599632,26cyclictest15763-21diskstats12:29:390
24475996241,10cyclictest0-21swapper/111:54:501
2446599623,30cyclictest25225-21latency_hist08:49:190
2446599613,29cyclictest7620-21fschecks_count07:54:410
24465996120,9cyclictest5749-21latency_hist07:49:210
2449499584,38cyclictest2159-21taskset11:09:403
24494995813,29cyclictest0-21swapper/311:24:323
2449499573,38cyclictest895-21kworker/3:1+events09:59:013
2449499572,39cyclictest1660-21taskset11:09:193
2449499552,39cyclictest27953-21taskset11:54:413
2449499552,39cyclictest24520-21chrt11:49:183
2449499534,35cyclictest303-21chrt09:10:013
2449499534,33cyclictest31468-21chrt07:29:203
2449499534,33cyclictest31468-21chrt07:29:203
2449499533,38cyclictest4377-21taskset09:17:393
2449499533,38cyclictest23230-21chrt10:49:343
2449499533,36cyclictest5279-21chrt11:14:463
2449499522,35cyclictest27563-21chrt08:54:503
2449499522,34cyclictest18086-21chrt10:39:463
2449499522,34cyclictest18086-21chrt10:39:463
2449499513,35cyclictest16290-21chrt09:39:403
2449499513,35cyclictest11472-21taskset10:29:203
2449499513,34cyclictest12407-21chrt12:24:193
2449499513,32cyclictest8356-21chrt07:54:563
2449499512,36cyclictest31769-21chrt10:08:043
2449499512,35cyclictest14924-21taskset09:36:443
2449499504,35cyclictest20748-21taskset11:41:293
2449499504,35cyclictest20748-21taskset11:41:293
2449499504,34cyclictest9202-21chrt07:59:383
2449499504,34cyclictest9202-21chrt07:59:383
2449499503,33cyclictest15940-21chrt12:29:423
2449499503,27cyclictest18730-21taskset09:44:303
2449499503,27cyclictest18730-21taskset09:44:303
2449499502,32cyclictest26968-21chrt07:14:423
24494995016,21cyclictest0-21swapper/310:14:173
2449499494,33cyclictest3466-21chrt12:09:203
2449499493,37cyclictest15431-21taskset11:32:413
2449499493,34cyclictest24844-21chrt07:09:403
2449499492,36cyclictest12750-21chrt08:09:463
2449499492,36cyclictest12750-21chrt08:09:463
2449499483,33cyclictest6777-21chrt12:14:353
24494994815,20cyclictest0-21swapper/310:14:413
59582470,8sleep30-21swapper/309:19:503
2449499473,30cyclictest4254-21taskset07:44:353
2449499473,28cyclictest20484-21chrt08:34:213
2449499472,34cyclictest10401-21chrt09:29:183
2449499472,34cyclictest10401-21chrt09:29:183
2449499463,34cyclictest28892-21chrt10:02:493
2449499463,32cyclictest6816-21taskset10:19:553
2449499463,32cyclictest22434-21chrt08:39:393
2449499463,31cyclictest14200-21chrt10:34:183
2449499463,30cyclictest20371-21taskset10:44:283
2449499453,33cyclictest22157-21chrt09:49:573
2449499453,32cyclictest6710-21chrt07:49:543
2449499453,30cyclictest30285-21taskset09:04:343
2449499453,30cyclictest10694-21taskset09:29:313
2449499452,32cyclictest25238-21chrt08:49:193
2449499452,31cyclictest29256-21taskset07:19:523
2449499452,31cyclictest29256-21taskset07:19:523
2449499452,31cyclictest18470-21taskset12:34:333
2449499443,31cyclictest30176-21chrt11:00:483
2449499443,30cyclictest16888-21chrt11:34:483
2449499443,30cyclictest16888-21chrt11:34:483
2449499443,28cyclictest17253-21chrt08:24:203
2449499436,27cyclictest11444-21chrt08:04:523
2449499436,27cyclictest11444-21chrt08:04:523
2449499433,31cyclictest3602-21taskset07:40:593
2449499433,31cyclictest26157-21chrt10:54:373
2449499433,29cyclictest21490-21chrt08:34:533
2449499424,29cyclictest32651-21chrt12:03:573
2449499424,27cyclictest25877-21chrt08:49:443
2449499422,31cyclictest26848-21taskset11:53:113
2449499422,27cyclictest714-21taskset07:34:313
2449499422,27cyclictest714-21taskset07:34:313
2449499422,27cyclictest28657-21chrt08:59:323
2449499413,27cyclictest17653-21taskset08:24:413
2449499403,27cyclictest14084-21taskset08:14:403
2449499403,27cyclictest14084-21taskset08:14:403
2449499382,27cyclictest31555-21chrt07:29:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional