You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-29 - 13:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Oct 29, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25910214610,24sleep20-21swapper/219:04:362
2550421200,10sleep025508-21unixbench_multi00:34:540
27378991146,73cyclictest8269-21apt-get21:19:410
27378991146,73cyclictest8269-21apt-get21:19:410
273819911163,42cyclictest32461-21sh21:04:171
273789910730,27cyclictest24610-21taskset20:39:280
273789910545,49cyclictest26213-21apt-get20:44:340
273819910183,11cyclictest12932-21kworker/u17:1-rpciod21:14:291
27388999839,49cyclictest0-21swapper/223:59:272
27388999839,49cyclictest0-21swapper/223:59:272
27381999778,13cyclictest17462-21fschecks_count22:29:521
27378999737,17cyclictest43761systemd-run23:59:170
2738899963,31cyclictest26386-21cut20:44:402
2737899956,43cyclictest17172-21apt-get22:29:380
27378999521,42cyclictest18216-21rm23:25:380
27378999521,42cyclictest18216-21rm23:25:380
27381999421,26cyclictest2775-21timerwakeupswit23:54:541
27378999431,30cyclictest29030-21ssh22:49:340
27388999350,37cyclictest0-21swapper/222:59:392
2738199933,33cyclictest16959-21latency_hist22:29:201
27378999364,19cyclictest21219-21latency_hist20:29:180
27378999364,19cyclictest21219-21latency_hist20:29:180
27378999322,50cyclictest25828-21ssh23:39:380
27381999244,39cyclictest0-21swapper/121:49:181
269542926,15sleep10-21swapper/119:05:491
27388999118,30cyclictest290-21dbus-daemon23:29:172
27388999118,30cyclictest290-21dbus-daemon23:29:172
27381999144,41cyclictest27367-21cyclictest23:39:191
27378999158,24cyclictest11690-21apt-get19:59:330
27378999129,24cyclictest15901-21ssh23:22:220
2738199903,34cyclictest2070-21modprobe22:04:151
27381999014,10cyclictest0-21swapper/122:34:581
27378999030,23cyclictest27924-21rm21:52:180
27378999025,27cyclictest5881-21grep19:39:500
27388998969,13cyclictest4548-21latency_hist23:59:202
27388998920,59cyclictest0-21swapper/200:24:182
27381998939,43cyclictest0-21swapper/119:39:541
27381998939,42cyclictest0-21swapper/123:39:281
2738199893,33cyclictest19823-21diskmemload21:39:191
2738199893,33cyclictest13580-21fschecks_time20:04:431
2738199893,32cyclictest27035-21ps21:50:541
2738199892,37cyclictest16513-21apt-get20:14:301
27381998923,23cyclictest11451-21tune2fs22:19:411
2737899899,48cyclictest467-21apt-get21:04:310
27378998929,42cyclictest31887-21apt-get21:59:340
27388998821,27cyclictest290-21dbus-daemon23:34:172
27388998813,36cyclictest29271-21grep22:49:402
2738199883,35cyclictest21839-21cstates00:29:351
2738199883,35cyclictest21839-21cstates00:29:351
27381998830,50cyclictest0-21swapper/119:59:451
27381998822,12cyclictest0-21swapper/100:19:511
27378998833,19cyclictest2138-21latency_hist22:59:180
27378998828,22cyclictest18381-21ssh00:23:290
27388998763,17cyclictest27367-21cyclictest20:24:422
27388998763,17cyclictest27367-21cyclictest20:24:422
2738899873,42cyclictest16621-21latency_hist23:24:172
2738899873,29cyclictest60-21kcompactd022:29:402
2738199873,35cyclictest9351-21grep19:49:531
2738199873,35cyclictest23194-21fschecks_time20:34:401
2738199873,34cyclictest28955-21latency_hist19:14:181
2738199873,33cyclictest18932-21/usr/sbin/munin00:24:461
2738199873,32cyclictest31473-21fschecks_time19:19:431
2738199873,32cyclictest20021-21fschecks_time20:24:391
2738199873,32cyclictest20021-21fschecks_time20:24:391
2738199873,32cyclictest10967-21cat23:14:191
27381998729,51cyclictest0-21swapper/122:14:551
2738199872,33cyclictest1034-21grep19:24:501
27378998729,22cyclictest3779-21ssh22:05:420
27378998722,28cyclictest8269-21ssh23:09:340
27378998719,28cyclictest13406-21ssh21:28:170
27388998634,45cyclictest0-21swapper/219:59:452
27388998634,44cyclictest0-21swapper/200:07:252
27388998620,30cyclictest729-21fschecks_count21:05:332
2738199864,33cyclictest32181-21fschecks_time22:54:391
2738199864,31cyclictest29961-21cat22:49:541
27381998638,40cyclictest0-21swapper/121:09:391
2738199863,34cyclictest25779-21taskset20:42:191
2738199863,33cyclictest28902-21/usr/sbin/munin21:54:421
2738199863,33cyclictest12216-21grep21:24:541
27381998628,51cyclictest0-21swapper/120:59:161
2738199862,36cyclictest10632-21proc_pri19:54:501
2738199862,36cyclictest10632-21proc_pri19:54:501
27378998660,16cyclictest7489-21apt-get00:04:350
27378998656,21cyclictest18953-21apt-get00:24:310
2738899854,30cyclictest18659-21ntp_kernel_pll_20:19:482
2738899854,30cyclictest18659-21ntp_kernel_pll_20:19:482
2738899853,38cyclictest29528-21cstates20:54:372
2738899853,32cyclictest14809-21mailstats21:29:512
2738899853,32cyclictest14809-21mailstats21:29:512
2738899853,31cyclictest19654-21taskset21:39:172
2738899853,31cyclictest12085-21grep23:14:542
2738899853,31cyclictest12085-21grep23:14:542
2738899853,30cyclictest26376-21fschecks_count22:44:402
2738899853,30cyclictest26376-21fschecks_count22:44:402
2738899853,30cyclictest19209-21fschecks_count00:24:402
2738899853,29cyclictest29259-21/usr/sbin/munin23:44:522
2738899853,29cyclictest29259-21/usr/sbin/munin23:44:522
2738899853,29cyclictest27916-21sed19:09:422
2738899853,27cyclictest23463-21sh21:44:482
27388998527,50cyclictest0-21swapper/219:49:402
2738899852,35cyclictest23232-21apt-get22:39:292
2738899852,24cyclictest2171-21rm21:09:322
27388998519,21cyclictest8539-21expr21:19:572
27388998519,21cyclictest8539-21expr21:19:572
2738199854,36cyclictest16779-21latency_hist21:34:191
2738199854,36cyclictest16779-21latency_hist21:34:191
27381998538,40cyclictest0-21swapper/123:54:151
2738199853,36cyclictest4636-21/usr/sbin/munin23:59:451
2738199853,36cyclictest4636-21/usr/sbin/munin23:59:451
2738199853,33cyclictest6872-21taskset23:06:211
2738199853,33cyclictest6872-21taskset23:06:211
2738199853,26cyclictest28225-21latency20:49:451
27381998529,49cyclictest0-21swapper/122:04:451
2738199852,28cyclictest26720-21ntp_offset22:44:471
2738199852,28cyclictest26720-21ntp_offset22:44:471
27378998556,21cyclictest307-21latency_hist19:24:190
27378998548,25cyclictest27235-21latency_hist19:09:200
27378998544,30cyclictest29594-21apt-get19:14:340
27378998538,39cyclictest0-21swapper/023:44:310
27378998538,39cyclictest0-21swapper/023:44:310
27378998536,40cyclictest21701-21/usr/sbin/munin00:29:540
27378998536,40cyclictest21701-21/usr/sbin/munin00:29:540
27378998530,22cyclictest13406-21apt-get00:14:350
27388998443,34cyclictest0-21swapper/223:34:372
2738899844,31cyclictest13709-21seq20:04:462
27388998439,37cyclictest0-21swapper/219:34:192
27388998434,42cyclictest0-21swapper/200:14:302
2738899843,30cyclictest26696-21head23:39:542
2738899843,28cyclictest11132-21/usr/sbin/munin22:19:542
2738899843,27cyclictest29434-21/usr/sbin/munin21:54:432
2738899843,25cyclictest29949-21/usr/sbin/munin19:14:432
2738899842,30cyclictest6147-21ssh22:09:532
2738899842,30cyclictest5089-21cat19:39:192
2738899842,30cyclictest31304-21fschecks_time20:59:392
2738899842,27cyclictest12928-21readlink00:14:172
27388998411,28cyclictest31572-21sh23:49:352
2738199843,34cyclictest23492-21fschecks_count22:39:401
2738199843,32cyclictest10575-21fschecks_count00:09:381
2738199843,31cyclictest5088-21latency_hist19:39:191
27381998427,51cyclictest0-21swapper/123:24:171
27381998425,7cyclictest0-21swapper/123:26:241
27381998425,7cyclictest0-21swapper/123:26:241
2738199842,33cyclictest3524-21latency_hist19:34:191
27378998464,11cyclictest6843-21apt-get19:44:310
2737899843,51cyclictest2122-21/usr/sbin/munin21:09:480
27378998431,43cyclictest10836-21basename22:19:150
27378998427,22cyclictest6978-21ssh23:06:400
27378998427,22cyclictest6978-21ssh23:06:400
27378998427,22cyclictest0-21swapper/021:04:190
27378998425,21cyclictest22653-21taskset22:38:380
2739399833,38cyclictest2069-21taskset21:09:203
27388998359,11cyclictest0-21swapper/222:04:192
27388998344,30cyclictest0-21swapper/223:04:522
27388998344,30cyclictest0-21swapper/223:04:522
2738899834,37cyclictest352-21runrttasks00:32:302
2738899834,37cyclictest352-21runrttasks00:32:302
2738899834,29cyclictest12218-21head21:24:542
27388998334,42cyclictest0-21swapper/223:14:172
2738899833,33cyclictest14454-21uname22:24:422
2738899833,31cyclictest10630-21ps19:54:502
2738899833,31cyclictest10630-21ps19:54:502
2738899833,23cyclictest17070-21taskset20:14:502
27388998318,25cyclictest7747-21apt-key21:19:162
2738199834,32cyclictest13682-21cpuspeed00:15:111
2738199833,34cyclictest15483-21ps20:09:501
2738199833,33cyclictest352-21runrttasks23:01:261
2738199833,32cyclictest28523-21/usr/sbin/munin23:44:461
2738199833,32cyclictest28523-21/usr/sbin/munin23:44:461
2738199833,32cyclictest1395-21timerwakeupswit21:05:471
27381998331,45cyclictest0-21swapper/121:39:551
2738199832,35cyclictest7566-21grep19:44:521
2738199832,35cyclictest12126-21unixbench_singl23:14:551
2738199832,35cyclictest12126-21unixbench_singl23:14:551
2738199832,34cyclictest27639-21latency_hist20:49:181
2738199832,31cyclictest7715-21fschecks_count00:04:411
27378998355,18cyclictest14836-21latency_hist20:09:190
27378998348,21cyclictest352-21runrttasks19:29:320
27378998332,42cyclictest22736-21ssh21:44:190
27378998330,20cyclictest30942-21sh21:57:330
27378998326,22cyclictest11920-21rm00:11:250
27378998325,29cyclictest30290-21cat20:54:560
27378998319,39cyclictest11110-21ssh22:19:250
2738899824,31cyclictest639-21fschecks_time19:24:392
2738899823,32cyclictest32296-21if_eth022:54:412
2738899823,31cyclictest25546-21unixbench_singl20:39:542
2738899823,29cyclictest8188-21/usr/sbin/munin22:14:482
2738899823,28cyclictest16259-21awk20:14:162
2738899823,27cyclictest2416-21taskset22:04:302
2738199823,33cyclictest5292-21/usr/sbin/munin22:09:341
2738199823,32cyclictest22289-21grep20:29:541
2738199823,32cyclictest22289-21grep20:29:541
2738199823,31cyclictest24952-21latency00:34:421
2738199823,30cyclictest18660-21ntp_kernel_pll_20:19:481
2738199823,30cyclictest18660-21ntp_kernel_pll_20:19:481
2738199823,24cyclictest30493-21grep19:14:541
27381998216,21cyclictest9142-21ssh21:20:151
27381998216,21cyclictest9142-21ssh21:20:151
27378998258,14cyclictest22846-21apt-get21:44:320
27378998248,23cyclictest26125-21apt-get22:44:340
27378998248,23cyclictest26125-21apt-get22:44:340
27378998225,26cyclictest16611-21/usr/sbin/munin20:14:360
27378998220,28cyclictest12723-21rm23:16:320
27378998220,28cyclictest12723-21rm23:16:320
27388998134,40cyclictest0-21swapper/220:29:512
27388998134,40cyclictest0-21swapper/220:29:512
2738899813,33cyclictest20641-21grep21:39:512
2738899813,29cyclictest20476-21cpu22:34:582
2738899813,28cyclictest27423-21sed21:50:592
2738899813,26cyclictest352-21runrttasks19:39:392
2738899812,36cyclictest27202-21ls00:39:192
2738199813,25cyclictest19766-21/usr/sbin/munin23:29:361
27378998130,20cyclictest15940-21rm21:32:100
27378998130,20cyclictest15940-21rm21:32:100
27378998129,23cyclictest4604-21rm23:59:210
27378998129,23cyclictest4604-21rm23:59:210
27378998128,21cyclictest24237-21ssh23:36:130
27378998126,23cyclictest5745-21processes21:14:510
2738899803,27cyclictest23657-21/usr/sbin/munin20:34:512
2738899802,33cyclictest28264-21latency20:49:452
27378998056,16cyclictest19788-21apt-get23:29:320
27378998026,23cyclictest18207-21sh21:35:160
27378998021,23cyclictest515-21expr19:24:340
2738899793,30cyclictest7738-21unixbench_multi19:44:542
2738899792,25cyclictest31396-21/usr/sbin/munin19:19:412
27378997930,42cyclictest28771-21grep20:49:560
27378997920,24cyclictest0-21swapper/020:09:420
27378997856,15cyclictest11425-21sendmail19:59:170
27378997856,15cyclictest11425-21sendmail19:59:170
27378997831,18cyclictest23852-21taskset22:39:490
27378997828,21cyclictest4842-21sh23:03:540
2739399772,35cyclictest6684-21taskset00:02:503
2739399772,35cyclictest6684-21taskset00:02:503
27378997759,9cyclictest27556-21apt-get19:09:310
27378997736,27cyclictest21368-21apt-get20:29:330
27378997736,27cyclictest21368-21apt-get20:29:330
27378997722,24cyclictest8935-21awk19:49:450
27378997722,19cyclictest0-21swapper/019:34:330
2739399765,32cyclictest8111-21chrt23:09:203
2739399764,43cyclictest29520-21taskset23:44:553
2739399764,43cyclictest29520-21taskset23:44:553
27378997644,20cyclictest18157-21apt-get20:19:360
27378997644,20cyclictest18157-21apt-get20:19:360
27378997631,38cyclictest32246-21sed23:49:530
2739399753,28cyclictest363-21chrt19:24:273
51262740,4sleep3451rcuc/321:14:343
2739399743,33cyclictest14677-21taskset21:29:473
2739399743,33cyclictest14677-21taskset21:29:473
2739399743,31cyclictest13502-21chrt00:14:463
27378997421,23cyclictest14651-21ssh22:24:470
2739399734,30cyclictest4384-21chrt23:59:173
2739399733,33cyclictest32760-21chrt21:59:533
27378997317,29cyclictest0-21swapper/022:11:000
2739399724,31cyclictest11349-21taskset21:24:363
2739399724,27cyclictest31618-21chrt22:54:163
2739399723,31cyclictest21226-21chrt23:30:563
2591127215,37sleep30-21swapper/319:04:373
2739399713,33cyclictest9382-21chrt22:15:093
2739399713,30cyclictest4473-21taskset23:02:473
2739399712,26cyclictest31941-21chrt23:49:453
2739399704,31cyclictest24337-21taskset22:39:573
2739399703,27cyclictest19512-21chrt20:24:163
2739399703,27cyclictest19512-21chrt20:24:163
27378997042,18cyclictest22973-21apt-get20:34:330
2739399693,25cyclictest22933-21chrt23:34:373
2739399693,24cyclictest21595-21chrt20:29:393
2739399693,24cyclictest21595-21chrt20:29:393
2739399674,27cyclictest13518-21taskset22:23:443
2739399673,32cyclictest12216-21chrt00:12:193
99442660,7sleep30-21swapper/300:09:063
2739399663,28cyclictest32457-21chrt22:54:443
27393996617,16cyclictest0-21swapper/323:40:183
2739399649,44cyclictest17232-21chrt22:29:393
2739399633,27cyclictest28508-21taskset22:48:493
2739399633,27cyclictest28508-21taskset22:48:493
2739399623,27cyclictest16901-21chrt23:24:323
2739399623,27cyclictest16901-21chrt23:24:323
2739399622,29cyclictest17767-21chrt00:21:313
2739399584,39cyclictest9987-21chrt19:54:203
2739399573,40cyclictest32733-21taskset19:24:193
2739399573,38cyclictest14733-21chrt20:09:183
2739399573,35cyclictest23246-21chrt21:44:423
2739399572,42cyclictest260262sleep321:49:333
2739399564,38cyclictest2806-21chrt21:09:483
2739399563,40cyclictest3496-21taskset22:04:563
2739399563,38cyclictest14864-21chrt23:19:583
2739399563,37cyclictest9102-21chrt21:20:143
2739399563,37cyclictest9102-21chrt21:20:143
2739399563,36cyclictest26681-21taskset20:44:463
2739399554,37cyclictest7458-21taskset23:08:063
2739399554,37cyclictest7458-21taskset23:08:063
2739399553,39cyclictest22849-21taskset00:30:023
2739399553,39cyclictest22849-21taskset00:30:023
2739399553,35cyclictest0-21swapper/322:34:333
2739399552,40cyclictest15073-21taskset22:24:533
2739399543,38cyclictest17541-21chrt21:34:453
2739399542,36cyclictest24525-21chrt20:39:193
2739399533,38cyclictest29700-21taskset21:54:513
2739399533,37cyclictest29940-21chrt19:14:433
2739399533,33cyclictest29064-21chrt19:14:193
2739399532,39cyclictest7637-21chrt22:13:353
2739399525,34cyclictest13708-21chrt23:19:173
2739399525,34cyclictest13708-21chrt23:19:173
2739399523,37cyclictest22107-21chrt21:43:043
2739399522,36cyclictest32498-21chrt21:04:183
2739399513,34cyclictest20343-21chrt00:25:573
2739399512,35cyclictest5879-21taskset19:39:503
2739399512,34cyclictest20258-21taskset20:24:453
2739399512,34cyclictest20258-21taskset20:24:453
2739399504,33cyclictest27031-21chrt00:39:183
2739399493,32cyclictest3516-21taskset19:34:193
2739399473,32cyclictest17727-21chrt20:17:453
2739399464,28cyclictest25637-21chrt20:40:283
2739399463,30cyclictest15433-21taskset20:09:473
2739399454,30cyclictest4454-21taskset19:34:523
2739399452,31cyclictest11970-21taskset19:59:423
2739399443,31cyclictest11340-21taskset19:58:493
2739399443,31cyclictest11340-21taskset19:58:493
2739399443,29cyclictest30582-21taskset20:57:013
2739399443,27cyclictest6984-21taskset19:44:363
2739399432,26cyclictest28844-21chrt20:50:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional