You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 21:56
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Nov 28, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1150721160,17sleep3451rcuc/309:03:583
57239910535,28cyclictest14617-21/usr/sbin/munin07:34:250
5723999834,28cyclictest6641-21ssh11:55:400
5723999834,28cyclictest6641-21ssh11:55:400
5723999832,30cyclictest9431-21rm10:01:350
5723999832,30cyclictest9431-21rm10:01:350
460429825,39sleep10-21swapper/107:04:321
188352980,8sleep20-21swapper/207:44:352
188352980,8sleep20-21swapper/207:44:352
572999974,40cyclictest19484-21seq07:49:141
572999974,40cyclictest19484-21seq07:49:141
5723999763,24cyclictest32558-21apt-get10:44:110
5723999737,27cyclictest23115-21ssh12:25:350
5723999737,27cyclictest23115-21ssh12:25:350
5723999336,35cyclictest30682-21sort10:39:310
5723999332,23cyclictest20631-21ssh09:22:010
5723999332,23cyclictest20631-21ssh09:22:010
5723999331,30cyclictest11870-21seq12:04:380
5729999250,36cyclictest5711-21cyclictest08:14:341
5723999250,25cyclictest27545-21apt-get08:14:130
5729999150,33cyclictest0-21swapper/111:24:371
5723999135,40cyclictest0-21swapper/008:09:070
5723999132,24cyclictest28848-21ssh09:37:470
5723999056,27cyclictest5275-21apt-get08:44:320
5723999052,22cyclictest29728-21apt-get11:39:140
5723999050,33cyclictest0-21swapper/007:09:020
5723999039,41cyclictest13724-21ssh12:09:170
5729998966,16cyclictest691-21cron07:38:581
5723998961,17cyclictest3001-21apt-get10:49:590
5723998959,21cyclictest32110-21ssh09:44:110
5723998959,21cyclictest32110-21ssh09:44:110
5723998933,18cyclictest13100-21diskmemload09:14:380
5723998928,23cyclictest0-21swapper/009:54:320
47892897,15sleep30-21swapper/307:04:383
5729998833,47cyclictest0-21swapper/109:39:001
5723998850,24cyclictest13164-21apt-get10:09:140
5723998836,22cyclictest23336-21ssh09:27:150
5723998834,18cyclictest29360-21ssh10:38:250
5723998833,28cyclictest12956-21sh11:06:530
5723998833,28cyclictest12956-21sh11:06:530
5729998763,18cyclictest309-21systemd-logind10:28:591
5729998763,18cyclictest309-21systemd-logind10:28:591
5723998753,25cyclictest21312-21runrttasks11:24:120
5723998750,27cyclictest8008-21apt-get11:59:130
5723998734,23cyclictest4357-21rm11:52:100
5723998734,23cyclictest30454-21ssh09:39:330
5723998733,20cyclictest28268-21ssh12:35:090
5732998663,16cyclictest8302-21latency_hist10:59:012
5732998663,16cyclictest8302-21latency_hist10:59:012
5729998664,15cyclictest31913-21sh09:44:011
5729998634,42cyclictest0-21swapper/111:39:011
572999863,30cyclictest5146-21latency_hist08:44:011
572999863,28cyclictest29585-21basename10:38:571
572999862,31cyclictest26582-21apt-key11:33:591
5723998672,8cyclictest0-21swapper/010:07:040
5723998643,31cyclictest0-21swapper/010:19:390
5723998631,23cyclictest1483-21sh11:46:290
5723998631,23cyclictest1483-21sh11:46:290
5732998562,15cyclictest5711-21cyclictest11:24:232
5732998559,19cyclictest30818-21apt-key08:24:012
5732998559,19cyclictest30818-21apt-key08:24:012
572999855,41cyclictest10475-21rm12:03:571
572999854,31cyclictest19109-21sort09:19:201
572999854,31cyclictest19109-21sort09:19:201
572999853,44cyclictest32155-21latency_hist11:44:001
572999853,44cyclictest20000-21ssh10:20:041
5729998534,46cyclictest0-21swapper/112:29:271
572999853,39cyclictest355-21runrttasks12:10:061
572999853,32cyclictest10926-21idleruntime-cro07:23:581
572999853,32cyclictest10926-21idleruntime-cro07:23:581
572999852,19cyclictest138922latency09:09:221
572999852,19cyclictest138922latency09:09:221
5723998558,16cyclictest24559-21apt-get10:30:010
5723998557,13cyclictest13100-21diskmemload11:24:000
5723998556,18cyclictest24179-21apt-get09:29:130
5723998534,23cyclictest17837-21rm11:16:330
5723998530,24cyclictest13957-21ntpq09:09:240
5723998530,24cyclictest13957-21ntpq09:09:240
5732998459,16cyclictest314-21in:imuxsock10:48:582
5732998455,20cyclictest16867-21netstat12:14:242
5732998455,20cyclictest16867-21netstat12:14:242
573299842,65cyclictest26479-21apt-get11:34:122
572999843,41cyclictest2019-21cut08:34:171
572999843,38cyclictest26031-21cstates08:09:131
572999843,30cyclictest9465-21grep10:59:411
572999843,29cyclictest25313-21/usr/sbin/munin10:30:181
5729998429,47cyclictest0-21swapper/110:54:351
5729998429,47cyclictest0-21swapper/110:54:351
572999842,32cyclictest32658-21ls10:44:161
572999842,31cyclictest22237-21ssh09:24:311
572999842,31cyclictest11861-21sendmail_mailqu11:04:311
572999842,31cyclictest11861-21sendmail_mailqu11:04:311
5723998438,33cyclictest24065-21taskset08:03:570
5723998436,39cyclictest21871-21fschecks_count10:24:210
5723998436,39cyclictest21871-21fschecks_count10:24:210
5723998436,32cyclictest18710-21apt-config10:19:020
5723998433,42cyclictest27074-21ssh11:34:320
5723998432,24cyclictest4142-21chrt09:52:180
5723998432,19cyclictest10499-21sh11:02:490
5732998361,16cyclictest10487-21sh12:03:582
572999834,72cyclictest2455-21apt-get10:49:121
572999834,31cyclictest32290-21/usr/sbin/munin11:44:341
572999834,31cyclictest32290-21/usr/sbin/munin11:44:341
572999834,26cyclictest11471-21modprobe09:03:571
572999833,32cyclictest29375-21fschecks_count08:19:201
572999833,32cyclictest29375-21fschecks_count08:19:201
572999833,30cyclictest16099-21fschecks_count10:14:181
572999833,29cyclictest16517-21grep07:39:211
572999833,29cyclictest16517-21grep07:39:211
572999833,28cyclictest2521-21fschecks_count09:49:201
572999833,26cyclictest29849-21/usr/sbin/munin10:39:251
5729998331,45cyclictest0-21swapper/109:29:151
572999832,33cyclictest24102-21latency_hist08:04:001
572999832,30cyclictest31234-21fschecks_count08:24:351
572999832,25cyclictest16095-21/usr/sbin/munin09:14:101
5723998352,21cyclictest24050-21apt-get11:29:130
5723998347,26cyclictest13846-21apt-get11:09:080
5723998336,37cyclictest20117-21uptime12:19:350
5723998333,40cyclictest11120-21latency_hist07:24:010
5723998333,40cyclictest11120-21latency_hist07:24:010
5732998257,16cyclictest871-21sendmail-mta09:39:152
572999824,24cyclictest5257-21/usr/sbin/munin08:44:351
572999823,37cyclictest6892-21apt-get08:49:091
572999823,33cyclictest24502-21cut08:04:191
572999823,31cyclictest13462-21fschecks_time10:09:211
572999823,29cyclictest4926-21/usr/sbin/munin09:54:221
572999823,29cyclictest355-21runrttasks12:19:231
572999823,26cyclictest7714-21modprobe11:58:571
572999823,26cyclictest7714-21modprobe11:58:571
572999822,41cyclictest14371-21latency_hist07:34:001
572999822,35cyclictest2909-21uname11:49:231
572999822,34cyclictest11822-21unixbench_singl12:04:361
5723998251,21cyclictest10132-21apt-get08:59:140
5723998250,21cyclictest17872-21apt-get07:44:090
5723998250,21cyclictest17872-21apt-get07:44:090
5723998249,20cyclictest21078-21apt-get07:54:070
5723998247,25cyclictest24383-21apt-get12:29:110
5732998161,14cyclictest8630-21sh10:59:202
5732998159,12cyclictest0-21swapper/209:44:372
5732998159,12cyclictest0-21swapper/209:44:372
5732998118,28cyclictest14212-21if_eth011:09:212
572999815,31cyclictest8867-21grep07:14:361
572999814,31cyclictest18134-21fschecks_time07:44:201
572999814,31cyclictest18134-21fschecks_time07:44:201
572999814,29cyclictest13287-21latency_hist09:09:011
572999813,37cyclictest13827-21/usr/sbin/munin11:09:331
572999813,33cyclictest6813-21unixbench_multi07:09:351
572999813,30cyclictest12442-21grep07:25:511
572999813,23cyclictest2107-21latency_hist09:49:001
572999813,23cyclictest2107-21latency_hist09:49:001
5729998128,48cyclictest0-21swapper/111:14:071
5729998128,46cyclictest0-21swapper/112:14:191
5729998128,46cyclictest0-21swapper/112:14:191
572999812,27cyclictest9879-21grep08:58:581
572999812,27cyclictest9879-21grep08:58:581
572999812,27cyclictest7874-21cstates09:59:151
572999812,27cyclictest7874-21cstates09:59:151
5729998120,9cyclictest0-21swapper/112:24:121
5729998120,9cyclictest0-21swapper/112:24:121
5723998149,23cyclictest11471-21apt-get07:24:410
5723998147,20cyclictest13028-21apt-get07:29:130
5723998137,15cyclictest6583-21xargs10:54:320
5723998137,15cyclictest6583-21xargs10:54:320
5723998134,40cyclictest4128-21grep08:39:310
5732998061,13cyclictest30217-21ssh10:39:202
5732998054,11cyclictest0-21swapper/208:39:012
572999803,37cyclictest21180-21cstates07:54:131
572999803,32cyclictest32763-21cstates08:29:141
572999803,28cyclictest18635-21/usr/sbin/munin11:19:181
572999803,26cyclictest355-21runrttasks10:04:271
5723998048,22cyclictest1798-21apt-get08:34:060
5723998039,33cyclictest32022-21grep08:24:510
5723998037,34cyclictest16360-21ssh12:14:110
5723998037,34cyclictest16360-21ssh12:14:110
5723998022,24cyclictest0-21swapper/008:23:580
5723998022,24cyclictest0-21swapper/008:23:580
5732997960,13cyclictest314-21in:imuxsock10:33:572
5732997914,39cyclictest14966-21if_eth007:34:212
572999793,24cyclictest26972-21/usr/sbin/munin12:34:101
5723997941,27cyclictest748-21/usr/sbin/munin08:29:280
5723997934,16cyclictest8319-21cut07:14:250
5732997856,10cyclictest0-21swapper/210:14:012
5732997855,11cyclictest0-21swapper/208:59:012
5732997855,11cyclictest0-21swapper/208:59:012
5732997854,18cyclictest24010-21apt-get10:29:022
5723997845,21cyclictest8501-21apt-get08:54:080
5723997845,21cyclictest8501-21apt-get08:54:080
5723997840,28cyclictest25679-21grep08:08:580
44102789,35sleep20-21swapper/207:04:242
5732997751,7cyclictest181rcu_preempt11:39:212
5732997750,20cyclictest21754-21fschecks_count09:24:192
5732997745,23cyclictest5344-21ls11:54:132
5732997745,23cyclictest5344-21ls11:54:132
573299772,38cyclictest355-21runrttasks08:24:392
5732997718,28cyclictest20190-21ssh11:21:092
5732997713,26cyclictest0-21swapper/212:34:312
5723997734,15cyclictest13289-21ssh09:09:010
5723997716,31cyclictest0-21swapper/007:43:590
5723997716,31cyclictest0-21swapper/007:43:590
5732997658,12cyclictest26784-21latency_hist12:33:592
5732997657,9cyclictest0-21swapper/208:09:172
5732997655,10cyclictest0-21swapper/209:34:012
5732997653,15cyclictest4768-21ssh11:53:292
5732997558,11cyclictest5711-21cyclictest10:49:102
5732997558,11cyclictest207-21systemd-journal09:51:502
5732997557,12cyclictest14154-21mailstats09:09:312
5732997557,12cyclictest14154-21mailstats09:09:312
5732997552,12cyclictest0-21swapper/212:13:462
5732997517,28cyclictest12348-21ssh10:07:192
5732997511,32cyclictest19739-21systemctl10:20:012
5723997539,26cyclictest6155-21fschecks_time07:09:210
5723997523,19cyclictest0-21swapper/007:49:100
5723997523,19cyclictest0-21swapper/007:49:100
5732997454,11cyclictest0-21swapper/209:20:032
5732997454,11cyclictest0-21swapper/209:20:032
5732997453,15cyclictest24183-21latency_hist08:04:002
5732997411,26cyclictest16313-21entropy09:14:202
5732997358,8cyclictest0-21swapper/211:44:342
5732997358,8cyclictest0-21swapper/211:44:342
5732997352,11cyclictest0-21swapper/207:18:592
5732997349,13cyclictest0-21swapper/210:28:592
5723997339,27cyclictest7863-21unixbench_singl08:49:340
5732997254,12cyclictest1-21systemd11:33:582
573299723,33cyclictest11425-21apt-get07:24:172
5732997150,13cyclictest5711-21cyclictest07:49:362
5732997150,13cyclictest5711-21cyclictest07:49:362
5732997150,10cyclictest0-21swapper/210:14:182
5732997148,13cyclictest0-21swapper/212:04:362
5732997056,8cyclictest0-21swapper/209:04:202
5732997051,8cyclictest0-21swapper/209:59:332
5732997051,8cyclictest0-21swapper/209:59:332
5732996951,12cyclictest1-21systemd08:43:592
5732996946,13cyclictest0-21swapper/211:09:002
5732996946,13cyclictest0-21swapper/211:09:002
5732996853,8cyclictest0-21swapper/209:04:002
5732996852,9cyclictest0-21swapper/208:09:002
5732996851,12cyclictest6399-21ntp_kernel_pll_07:09:272
5732996847,9cyclictest0-21swapper/210:39:002
5732996829,14cyclictest24109-21munin-run12:28:582
5732996829,14cyclictest24109-21munin-run12:28:582
5732996814,21cyclictest20667-21taskset12:21:262
5732996747,9cyclictest0-21swapper/208:14:292
5732996747,10cyclictest0-21swapper/207:24:002
5732996747,10cyclictest0-21swapper/207:24:002
5732996729,28cyclictest16269-21apt-get07:39:152
5732996729,28cyclictest16269-21apt-get07:39:152
5732996647,13cyclictest1-21systemd09:58:592
5732996642,14cyclictest16945-21open_inodes11:14:292
5732996542,15cyclictest0-21swapper/208:54:002
5732996446,8cyclictest0-21swapper/208:29:182
5732996444,10cyclictest0-21swapper/207:54:182
5732996344,9cyclictest0-21swapper/209:39:002
573299625,43cyclictest5275-21apt-get08:44:272
5732996043,8cyclictest0-21swapper/207:29:222
573899594,35cyclictest11464-21chrt07:24:363
573899593,38cyclictest3176-21taskset10:49:493
573899562,31cyclictest310572sleep308:24:253
573899553,32cyclictest0-21swapper/311:34:123
573899553,31cyclictest28867-21chrt10:36:523
573899543,31cyclictest0-21swapper/310:29:213
573899532,32cyclictest15863-21taskset12:13:283
573899523,35cyclictest22179-21taskset10:24:273
573899523,35cyclictest22179-21taskset10:24:273
573899523,32cyclictest12589-21taskset09:04:353
573899522,32cyclictest26729-21chrt09:34:013
573899513,31cyclictest8996-21chrt11:59:343
573899512,33cyclictest7157-21chrt08:49:203
573899503,30cyclictest22129-21chrt09:24:293
573899502,36cyclictest12776-21chrt11:06:183
573899502,36cyclictest12776-21chrt11:06:183
573899502,32cyclictest5531-21taskset09:54:283
573899492,31cyclictest21404-21chrt07:54:203
573899492,31cyclictest12408-21chrt12:06:233
573899483,28cyclictest27843-21chrt08:14:203
573899483,27cyclictest16269-21taskset10:14:223
573899482,30cyclictest4622-21taskset11:53:013
573899482,30cyclictest15397-21taskset09:12:293
573899482,30cyclictest15397-21taskset09:12:293
573899482,29cyclictest20046-21chrt11:20:413
573899473,30cyclictest19013-21chrt10:19:413
573899473,28cyclictest16021-21taskset11:13:563
573899473,27cyclictest2105-21taskset09:49:003
573899473,27cyclictest2105-21taskset09:49:003
573899472,32cyclictest19200-21chrt07:48:573
573899472,32cyclictest19200-21chrt07:48:573
573899472,26cyclictest5228-21chrt11:54:073
573899472,26cyclictest5228-21chrt11:54:073
573899463,30cyclictest7938-21taskset09:59:183
573899463,30cyclictest7938-21taskset09:59:183
573899462,31cyclictest25682-21taskset08:08:583
573899454,29cyclictest6022-21taskset10:54:193
573899454,29cyclictest6022-21taskset10:54:193
573899453,29cyclictest9035-21chrt10:59:293
573899453,29cyclictest3438-21chrt08:39:113
573899453,29cyclictest30125-21taskset09:39:283
573899453,29cyclictest28455-21taskset12:35:413
573899453,28cyclictest29850-21chrt10:39:073
573899452,30cyclictest24018-21taskset12:28:483
573899452,30cyclictest24018-21taskset12:28:483
573899452,26cyclictest18751-21chrt09:19:013
573899452,26cyclictest18751-21chrt09:19:013
573899444,28cyclictest12903-21chrt10:08:593
573899443,29cyclictest2116-21taskset10:47:423
573899443,29cyclictest14879-21chrt10:11:423
573899443,28cyclictest3042-21taskset08:37:533
573899443,28cyclictest19075-21chrt12:19:123
573899443,27cyclictest5911-21taskset07:09:133
573899442,29cyclictest5660-21taskset08:44:463
573899442,29cyclictest32602-21taskset11:44:223
573899442,29cyclictest32602-21taskset11:44:223
573899442,29cyclictest27340-21chrt08:14:003
573899442,29cyclictest24656-21taskset11:29:293
573899434,29cyclictest29309-21taskset09:38:593
573899433,28cyclictest2583-21chrt09:49:213
573899433,28cyclictest17438-21taskset11:14:543
573899433,27cyclictest26424-21taskset12:33:143
573899433,27cyclictest21290-21chrt11:24:103
573899433,27cyclictest16142-21taskset09:14:153
573899433,27cyclictest13566-21taskset07:29:293
573899432,29cyclictest9477-21taskset08:54:343
573899432,29cyclictest9477-21taskset08:54:343
573899432,29cyclictest30506-21taskset08:23:523
573899432,29cyclictest30506-21taskset08:23:523
573899432,28cyclictest7946-21taskset07:14:113
573899432,28cyclictest16350-21taskset12:14:093
573899432,28cyclictest16350-21taskset12:14:093
573899423,28cyclictest17538-21taskset07:43:183
573899423,28cyclictest17538-21taskset07:43:183
573899423,27cyclictest23742-21taskset07:59:593
573899423,27cyclictest20617-21taskset07:51:363
573899423,27cyclictest20617-21taskset07:51:363
573899422,28cyclictest9732-21chrt07:19:183
573899422,28cyclictest9732-21chrt07:19:183
573899413,27cyclictest29921-21chrt11:39:193
573899413,27cyclictest14640-21chrt07:34:113
573899413,26cyclictest465-21chrt08:29:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional