You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 00:02
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Mon Jun 30, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1357121317,16sleep00-21swapper/007:03:580
13778211016,65sleep113816-21munin-run07:06:441
2199621010,2sleep30-21swapper/309:07:113
1395299988,54cyclictest0-21swapper/312:12:023
13948999874,17cyclictest21627-21idleruntime-cro10:06:432
1394099983,37cyclictest21956-21apt-get12:02:011
1394099963,37cyclictest24848-21sshd07:37:181
1394099953,39cyclictest854-21fschecks_time10:27:101
13940999446,40cyclictest0-21swapper/107:11:471
13948999378,11cyclictest13922-21cyclictest09:24:052
1394099933,39cyclictest17657-21unixbench_singl08:52:261
1394099932,23cyclictest1828250ssh09:59:571
13948999265,16cyclictest0-21swapper/208:46:482
13932999211,45cyclictest20662-21/usr/sbin/munin11:02:200
13948999168,12cyclictest0-21swapper/210:11:472
13932999164,18cyclictest22338-21idleruntime-cro07:31:430
13948999064,19cyclictest17260-21kworker/u17:2+events_unbound07:46:442
1394099905,34cyclictest1255-21/usr/sbin/munin11:26:551
1394099904,38cyclictest13365-21fschecks_time11:47:091
1394099904,36cyclictest12886-21cat08:41:451
1394099904,35cyclictest7368-21/usr/sbin/munin12:32:061
1394099904,35cyclictest7368-21/usr/sbin/munin12:32:061
13940999041,42cyclictest202-21systemd-journal11:26:421
13940999038,44cyclictest0-21swapper/110:42:231
1394099903,37cyclictest14608-21grep09:52:211
1394099903,36cyclictest5764-21idleruntime09:37:131
1394099903,36cyclictest5764-21idleruntime09:37:131
1394099903,36cyclictest17516-21kernelversion10:57:111
1394099903,32cyclictest10810-21/usr/sbin/munin09:47:121
13940999029,7cyclictest0-21swapper/112:07:111
1394099895,34cyclictest6406-21latency_hist08:21:461
1394099894,34cyclictest356-21runrttasks09:10:451
1394099894,34cyclictest16800-21idleruntime-cro10:56:431
1394099893,38cyclictest3739-21grep09:32:271
1394099893,32cyclictest16539-21latency07:12:121
1394099893,32cyclictest16539-21latency07:12:121
13940998930,10cyclictest0-21swapper/111:37:021
13940998922,59cyclictest0-21swapper/112:27:081
13932998968,15cyclictest379-21cron07:56:430
1393299893,60cyclictest28381-21df12:12:300
13948998861,19cyclictest0-21swapper/212:12:232
13940998872,10cyclictest28077-21kworker/u17:2-xprtiod12:17:231
1394099886,38cyclictest5184-21timerwakeupswit11:32:231
1394099886,38cyclictest5184-21timerwakeupswit11:32:231
1394099884,34cyclictest9695-21latency_hist08:31:471
1394099883,76cyclictest18724-21apt-get10:02:081
1394099883,39cyclictest24973-21fschecks_time10:12:101
1394099883,39cyclictest24973-21fschecks_time10:12:101
1394099883,33cyclictest1496-21/usr/sbin/munin12:22:141
1394099883,33cyclictest1496-21/usr/sbin/munin12:22:141
1394099883,25cyclictest31903-21seq07:57:251
13940998828,8cyclictest0-21swapper/111:11:541
13940998825,13cyclictest0-21swapper/112:11:471
13940998822,15cyclictest0-21swapper/109:42:191
13932998865,12cyclictest0-21swapper/010:51:460
13932998865,12cyclictest0-21swapper/010:51:460
13932998860,20cyclictest25620-21fschecks_count11:12:050
13948998766,13cyclictest0-21swapper/208:31:462
13940998738,41cyclictest0-21swapper/110:51:451
13940998738,41cyclictest0-21swapper/110:51:451
1394099873,36cyclictest20037-21unixbench_singl11:57:211
1394099873,36cyclictest1704-21cpu08:07:001
1394099873,35cyclictest24476-21cpuspeed09:12:041
1394099873,35cyclictest24476-21cpuspeed09:12:041
1394099873,32cyclictest29157-21timerwakeupswit11:17:221
1394099873,32cyclictest29157-21timerwakeupswit11:17:221
1394099873,31cyclictest20881-21latency_hist07:26:461
1394099873,30cyclictest32592-21cstates09:27:081
1394099873,30cyclictest32592-21cstates09:27:081
1394099872,35cyclictest14660-21apt-key08:46:451
1394099872,33cyclictest10370-21fschecks_count11:42:091
1394099872,30cyclictest23564-21grep07:32:201
13948998665,14cyclictest27524-21cpu10:17:042
13948998665,14cyclictest27524-21cpu10:17:042
1394099864,30cyclictest5841-21latency_hist10:36:461
1394099863,37cyclictest27840-21proc_pri09:17:241
1394099863,34cyclictest30142-21fschecks_time09:22:091
1394099863,32cyclictest3296-21acpi08:11:561
1394099863,31cyclictest21311-21idleruntime-cro09:06:441
1394099863,31cyclictest21311-21idleruntime-cro09:06:441
1394099862,39cyclictest22462-21latency_hist11:06:451
1394099862,36cyclictest21060-21cpuspeed07:27:041
1394099862,35cyclictest6326-21fschecks_time10:37:131
1394099862,34cyclictest22966-21grep10:07:301
1394099862,31cyclictest17053-21grep11:52:221
1394099862,28cyclictest30314-21cpuspeed10:22:051
1394099862,28cyclictest15386-21fschecks_time08:47:221
1394899857,46cyclictest14573-21ssh10:52:252
13948998568,11cyclictest0-21swapper/209:57:112
13948998560,17cyclictest11208-21sh10:46:462
13940998537,40cyclictest0-21swapper/108:02:111
1394099853,34cyclictest18417-21sendmail_mailqu07:17:241
1394099853,34cyclictest18417-21sendmail_mailqu07:17:241
13940998529,49cyclictest0-21swapper/107:44:291
1394099852,34cyclictest30006-21latency_hist10:21:531
1394099852,34cyclictest30006-21latency_hist10:21:531
1394099852,32cyclictest29521-21cut07:52:101
13940998517,19cyclictest10436-21tr08:32:191
13940998517,19cyclictest10436-21tr08:32:191
13932998563,12cyclictest0-21swapper/008:27:060
13948998463,14cyclictest1865-21fschecks_count08:07:052
1394099844,30cyclictest19306-21unixbench_singl08:57:241
1394099843,27cyclictest6535-21/usr/sbin/munin08:21:571
13932998466,11cyclictest0-21swapper/009:22:240
13948998365,8cyclictest0-21swapper/208:41:452
13948998345,25cyclictest25559-21modprobe07:41:432
1394099833,26cyclictest27671-21/usr/sbin/munin07:47:041
1394099833,26cyclictest27671-21/usr/sbin/munin07:47:041
13932998357,16cyclictest0-21swapper/009:11:460
13948998263,9cyclictest0-21swapper/208:47:212
13948998259,13cyclictest0-21swapper/209:32:072
13948998258,13cyclictest0-21swapper/212:21:462
13948998257,19cyclictest16329-21fschecks_count07:12:082
13948998257,19cyclictest16329-21fschecks_count07:12:082
13948998257,13cyclictest0-21swapper/209:41:462
13948998257,13cyclictest0-21swapper/209:41:462
1394099823,29cyclictest23079-21ls11:07:121
1394099823,29cyclictest23079-21ls11:07:121
1393299816,25cyclictest32832sleep010:32:030
13932998162,10cyclictest0-21swapper/009:17:290
13932998160,15cyclictest9399-21processes10:42:190
13932998158,17cyclictest13922-21cyclictest11:42:210
13948998063,9cyclictest0-21swapper/208:36:472
13948998063,9cyclictest0-21swapper/208:36:472
13948998058,9cyclictest0-21swapper/210:41:462
13948998054,13cyclictest0-21swapper/212:26:472
13948998052,18cyclictest701-21processes09:27:242
13948998052,18cyclictest701-21processes09:27:242
13948997958,11cyclictest0-21swapper/208:16:462
1394899795,62cyclictest25670-21switchtime12:07:192
13932997960,10cyclictest0-21swapper/010:12:250
13932997960,10cyclictest0-21swapper/010:12:250
13948997859,9cyclictest0-21swapper/208:17:092
13948997854,13cyclictest0-21swapper/208:52:122
13948997853,13cyclictest0-21swapper/207:56:462
13948997848,19cyclictest0-21swapper/211:52:192
13932997813,27cyclictest1929-21/usr/sbin/munin12:22:100
13932997813,27cyclictest1929-21/usr/sbin/munin12:22:100
13932997813,19cyclictest11765-21wc09:47:260
13948997761,8cyclictest0-21swapper/209:17:242
13948997759,9cyclictest0-21swapper/208:22:182
13948997757,10cyclictest0-21swapper/208:02:022
13948997754,12cyclictest0-21swapper/211:42:102
13948997713,47cyclictest19876-21sort11:01:562
13932997762,10cyclictest19470-21kworker/u17:5+rpciod10:27:140
13932997755,12cyclictest0-21swapper/007:17:240
13932997755,12cyclictest0-21swapper/007:17:240
1393299772,35cyclictest11266-21latency_hist08:36:460
1393299772,35cyclictest11266-21latency_hist08:36:460
99412760,23sleep29945-21latency_hist11:41:462
13948997637,25cyclictest24213-21kworker/u19:0+nfsiod11:31:462
1393299768,39cyclictest2024-21mailstats11:27:180
1393299766,55cyclictest25750-21kworker/u18:1+ext4-rsv-conversion10:52:250
13932997657,14cyclictest0-21swapper/007:07:090
13932997655,16cyclictest13922-21cyclictest07:15:550
13932997655,16cyclictest13922-21cyclictest07:15:550
13932997628,35cyclictest13621-21netstat11:47:150
13932997617,51cyclictest244212chrt12:06:240
13932997613,29cyclictest9982-21cron12:36:420
13932997613,29cyclictest9982-21cron12:36:420
13948997553,14cyclictest202-21systemd-journal11:10:152
13948997553,14cyclictest202-21systemd-journal11:10:152
13948997552,12cyclictest0-21swapper/207:22:062
13948997550,13cyclictest0-21swapper/209:51:462
13932997554,15cyclictest16575-21ntp_kernel_pll_11:52:130
13932997551,17cyclictest22143-21sh10:07:090
1393299753,37cyclictest17486-21/usr/sbin/munin10:57:100
13932997533,32cyclictest0-21swapper/009:27:240
13932997533,32cyclictest0-21swapper/009:27:240
13948997453,12cyclictest0-21swapper/207:32:092
13948997452,15cyclictest5885-21kworker/u17:3-nfsiod12:02:072
13948997450,15cyclictest0-21swapper/209:46:452
13932997455,13cyclictest27640-21apt-key12:11:460
1323027412,37sleep30-21swapper/307:02:303
13948997355,13cyclictest13922-21cyclictest09:07:192
1393299736,24cyclictest20738-21timerandwakeup09:02:230
1393299736,24cyclictest20738-21timerandwakeup09:02:230
13932997355,11cyclictest0-21swapper/010:26:460
13932997352,15cyclictest18858-21latency_hist10:01:450
13932997351,11cyclictest0-21swapper/011:11:460
13932997351,11cyclictest0-21swapper/011:11:460
13948997255,9cyclictest0-21swapper/210:27:272
13948997249,13cyclictest0-21swapper/210:47:122
13948997249,13cyclictest0-21swapper/210:47:122
13948997247,11cyclictest0-21swapper/212:30:322
1393299729,26cyclictest7156-21rm11:36:540
13932997249,13cyclictest0-21swapper/008:02:020
13932997238,25cyclictest14582-21modprobe08:46:430
13948997155,11cyclictest13922-21cyclictest11:57:152
13948997148,16cyclictest13922-21cyclictest07:07:202
1394899713,56cyclictest24821-21latency09:12:132
1394899713,56cyclictest24821-21latency09:12:132
1394899713,48cyclictest10125-21latency_hist12:36:452
1394899713,48cyclictest10125-21latency_hist12:36:452
13932997152,10cyclictest0-21swapper/012:01:460
13932997147,17cyclictest15376-21fschecks_time08:47:220
13932997146,18cyclictest29149-21kworker/u18:0+events_unbound09:56:430
1393299714,54cyclictest19595-21grep07:22:050
1393299713,35cyclictest1278-21cut12:21:450
13948997046,13cyclictest0-21swapper/211:01:462
13932997053,7cyclictest0-21swapper/008:41:460
13932997051,8cyclictest0-21swapper/009:16:470
13932997051,8cyclictest0-21swapper/009:16:470
1393299702,47cyclictest10641-21latency_hist09:46:460
1394899698,52cyclictest18954-21find08:57:202
1394899695,24cyclictest39-21ksoftirqd/212:23:462
13948996950,10cyclictest0-21swapper/211:47:112
13948996943,14cyclictest0-21swapper/207:51:462
13948996943,14cyclictest0-21swapper/207:51:462
13948996941,20cyclictest0-21swapper/210:22:122
1394899693,25cyclictest39-21ksoftirqd/210:33:012
13932996953,11cyclictest0-21swapper/012:27:200
13932996952,10cyclictest0-21swapper/008:52:190
13932996950,8cyclictest0-21swapper/008:11:470
13932996929,28cyclictest30646-21latency_hist11:21:450
13932996929,28cyclictest30646-21latency_hist11:21:450
13948996843,13cyclictest0-21swapper/209:56:202
13932996851,10cyclictest0-21swapper/008:01:450
13932996821,25cyclictest0-21swapper/010:37:010
13932996813,22cyclictest6281-21rm09:37:250
13932996813,22cyclictest6281-21rm09:37:250
13948996746,9cyclictest0-21swapper/207:17:052
13948996746,9cyclictest0-21swapper/207:17:052
13948996746,15cyclictest26106-21sh10:13:272
13948996746,15cyclictest26106-21sh10:13:272
13948996745,11cyclictest0-21swapper/208:01:432
1394899673,42cyclictest20788-21timerwakeupswit09:02:232
1394899673,42cyclictest20788-21timerwakeupswit09:02:232
1393299679,15cyclictest23313-21sed07:32:180
1393299678,22cyclictest29020-21modprobe07:51:430
1393299678,22cyclictest29020-21modprobe07:51:430
13932996725,31cyclictest5352-21latency08:17:130
13948996648,13cyclictest13922-21cyclictest11:22:132
13948996648,12cyclictest0-21swapper/211:32:082
13948996648,12cyclictest0-21swapper/211:32:082
13948996642,10cyclictest0-21swapper/211:21:462
13948996642,10cyclictest0-21swapper/211:21:462
13948996617,36cyclictest25363-21sh11:11:502
13932996636,22cyclictest0-21swapper/010:17:270
13932996636,22cyclictest0-21swapper/010:17:270
13932996622,33cyclictest6816-21fschecks_time08:22:090
13932996617,37cyclictest356-21runrttasks11:34:020
13932996617,37cyclictest356-21runrttasks11:34:020
13932996610,20cyclictest4158-21cat08:12:220
1393299659,19cyclictest0-21swapper/011:24:190
13932996547,11cyclictest20131-21ssh10:03:040
13932996544,12cyclictest0-21swapper/009:32:160
13932996533,12cyclictest25596-21http07:41:440
1272226216,22sleep20-21swapper/207:02:132
13948996148,9cyclictest58-21kcompactd007:28:212
13932996145,8cyclictest0-21swapper/008:57:190
13932996121,28cyclictest26077-21apt-get07:41:570
1395299606,41cyclictest14604-21chrt10:52:323
1395299604,41cyclictest12757-21chrt08:41:433
1395299603,42cyclictest4893-21taskset09:36:133
1395299599,31cyclictest0-21swapper/310:02:213
1395299573,38cyclictest18358-21taskset10:00:133
1395299552,39cyclictest20144-21chrt09:02:093
1395299552,39cyclictest20144-21chrt09:02:093
1395299544,37cyclictest5149-21chrt08:17:093
1395299544,34cyclictest0-21swapper/307:42:163
1395299543,34cyclictest20891-21taskset07:26:463
1395299533,37cyclictest24311-21chrt11:09:173
1395299533,37cyclictest24311-21chrt11:09:173
1395299533,37cyclictest1188-21taskset12:21:423
1395299523,37cyclictest31809-21chrt10:24:033
1395299523,36cyclictest24185-21chrt10:11:073
1395299523,34cyclictest29349-21chrt09:21:093
1395299522,37cyclictest10707-21chrt10:45:223
1395299514,35cyclictest17650-21taskset10:57:143
1395299514,33cyclictest15446-21chrt07:11:433
1395299513,35cyclictest8633-21chrt11:38:233
1395299513,35cyclictest25246-21taskset10:12:163
1395299513,35cyclictest25246-21taskset10:12:163
1395299513,35cyclictest2212-21taskset08:07:123
1395299513,35cyclictest11732-21chrt10:47:093
1395299513,35cyclictest11732-21chrt10:47:093
1395299513,34cyclictest26687-21chrt11:12:583
1395299513,33cyclictest32358-21chrt08:01:463
1395299512,39cyclictest2351-21chrt09:31:433
1395299512,39cyclictest2351-21chrt09:31:433
1395299512,38cyclictest8776-21chrt09:42:213
1395299507,22cyclictest0-21swapper/308:47:073
1395299503,35cyclictest8236-21chrt08:27:013
1395299503,35cyclictest30463-21chrt11:21:293
1395299503,35cyclictest30463-21chrt11:21:293
1395299503,34cyclictest29529-21taskset07:52:103
1395299503,34cyclictest29111-21chrt07:51:453
1395299503,34cyclictest29111-21chrt07:51:453
1395299503,34cyclictest21654-21chrt12:01:453
1395299502,35cyclictest21927-21taskset11:05:283
1395299502,34cyclictest17682-21taskset11:54:033
1395299493,37cyclictest6799-21chrt11:36:403
1395299493,37cyclictest6799-21chrt11:36:403
1395299493,36cyclictest12541-21taskset09:49:313
1395299493,34cyclictest7169-21chrt08:22:193
1395299493,34cyclictest6944-21taskset12:31:233
1395299493,33cyclictest24661-21chrt07:37:113
1395299493,32cyclictest21845-21chrt07:27:233
1395299493,32cyclictest2029-21chrt12:22:113
1395299493,32cyclictest2029-21chrt12:22:113
1395299493,30cyclictest3847-21chrt10:32:223
1395299492,34cyclictest22493-21chrt12:02:173
1395299483,32cyclictest3748-21taskset11:31:213
1395299483,32cyclictest15274-21chrt09:54:253
1395299483,32cyclictest12367-21chrt11:45:403
1395299482,34cyclictest29924-21taskset09:22:033
1395299482,32cyclictest24855-21chrt09:12:143
1395299482,32cyclictest24855-21chrt09:12:143
1395299482,27cyclictest0-21swapper/308:41:573
1395299474,34cyclictest3378-21chrt08:12:023
1395299473,35cyclictest1741-21taskset10:27:483
1395299473,33cyclictest25202-21chrt12:07:083
1395299473,32cyclictest6184-21chrt10:37:083
1395299473,31cyclictest31395-21chrt11:22:143
1395299473,30cyclictest29244-21chrt10:19:423
1395299473,30cyclictest29244-21chrt10:19:423
1395299472,34cyclictest16051-21chrt07:11:563
1395299472,34cyclictest16051-21chrt07:11:563
1395299472,33cyclictest22761-21chrt07:32:053
1395299472,32cyclictest5554-21chrt09:37:063
1395299472,32cyclictest5554-21chrt09:37:063
1395299472,32cyclictest13908-21taskset11:47:223
1395299463,32cyclictest602-21chrt08:02:143
13952994526,14cyclictest180202sleep308:56:433
1395299452,32cyclictest8664-21chrt12:32:533
1395299452,32cyclictest8664-21chrt12:32:533
1395299422,28cyclictest10299-21chrt08:32:153
1395299422,28cyclictest10299-21chrt08:32:153
1395299393,25cyclictest18926-21chrt07:19:173
1395299393,25cyclictest18926-21chrt07:19:173
1395299382,26cyclictest19420-21chrt08:58:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional