You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-15 - 21:57
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Mon Sep 15, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16399910581,18cyclictest24816-21apt-key08:11:352
16399910318,38cyclictest1-21systemd11:11:312
16309910385,9cyclictest0-21swapper/007:01:360
16309910385,9cyclictest0-21swapper/007:01:360
16309910179,15cyclictest1-21systemd08:51:320
16309910179,15cyclictest1-21systemd08:51:320
1635991004,67cyclictest25473-21ntp_offset10:32:281
163999973,34cyclictest27254-21/usr/sbin/munin11:37:062
163999973,34cyclictest27254-21/usr/sbin/munin11:37:062
163999962,46cyclictest19187-21ssh11:21:472
1639999552,38cyclictest1621-21cyclictest09:36:322
1639999547,41cyclictest19954-21latency_hist07:56:352
1639999547,41cyclictest19954-21latency_hist07:56:352
1639999541,47cyclictest0-21swapper/212:11:352
163999953,50cyclictest25474-21ntp_offset10:32:282
163599953,37cyclictest17045-21fschecks_time07:46:561
1639999475,12cyclictest378-21rs:main2
1639999475,12cyclictest378-21rs:main2
1639999443,44cyclictest0-21swapper/210:51:512
1639999442,42cyclictest0-21swapper/209:01:362
163999943,32cyclictest10965-21latency_hist11:06:362
163999942,35cyclictest14851-21ssh12:14:452
163999942,35cyclictest14851-21ssh12:14:452
1635999472,13cyclictest0-21swapper/108:42:331
1635999444,43cyclictest24237-21apt-key10:31:341
1635999443,45cyclictest0-21swapper/112:27:041
1635999443,44cyclictest8425-21cpu11:01:501
1635999443,44cyclictest8425-21cpu11:01:501
1635999437,8cyclictest0-21swapper/111:47:051
149829413,54sleep11521-21sh07:01:321
149829413,54sleep11521-21sh07:01:321
163599934,35cyclictest6460-21fschecks_time09:57:191
1635999341,45cyclictest19263-21/usr/sbin/munin10:22:131
163599933,35cyclictest20001-21/usr/sbin/munin09:21:501
1635999331,20cyclictest5093-21sort08:47:081
1635999331,20cyclictest5093-21sort08:47:081
1635999329,11cyclictest0-21swapper/109:32:091
1639999241,43cyclictest8065-21unixbench_singl07:17:122
163599922,67cyclictest29674-21apt-get11:41:461
163599922,67cyclictest29674-21apt-get11:41:461
163599922,34cyclictest25189-21acpi08:11:441
1630999272,13cyclictest8886-21diskmemload09:02:340
1630999272,13cyclictest8886-21diskmemload09:02:340
1630999256,27cyclictest18976-21idleruntime-cro10:21:320
1630999256,27cyclictest18976-21idleruntime-cro10:21:320
1639999170,16cyclictest1621-21cyclictest09:27:112
1639999135,46cyclictest0-21swapper/208:31:542
163999913,35cyclictest18848-21sh11:21:312
163999913,35cyclictest18848-21sh11:21:312
1635999134,10cyclictest0-21swapper/111:12:051
163599913,32cyclictest14763-21taskset09:11:501
163599913,31cyclictest30138-21apt-key11:41:361
163599913,31cyclictest24272-21grep08:07:121
163599913,24cyclictest356-21runrttasks10:46:441
163599913,24cyclictest356-21runrttasks10:46:441
163599912,44cyclictest13912-21latency_hist10:11:351
163599912,35cyclictest21394-21idleruntime-cro08:01:321
163599912,34cyclictest12656-21uname09:07:071
1635999122,13cyclictest0-21swapper/111:56:371
1630999165,19cyclictest27209-21latency_hist11:36:360
163999904,42cyclictest17875-21memory09:17:002
163999904,32cyclictest2780-21cron11:51:312
163999903,36cyclictest11416-21/usr/sbin/munin10:07:062
163999903,29cyclictest22396-21ntpq11:26:592
1639999029,54cyclictest356-21runrttasks10:19:372
1639999029,54cyclictest356-21runrttasks10:19:372
163999902,29cyclictest15406-21fschecks_count07:41:552
163599903,32cyclictest7802-21fschecks_time08:56:561
163599902,37cyclictest27677-21fschecks_time10:36:571
163599902,32cyclictest30719-21cstates09:41:481
163599902,32cyclictest1615-21timerandwakeup08:37:051
1630999032,34cyclictest15-21ksoftirqd/010:07:490
1630999010,55cyclictest15-21ksoftirqd/008:42:340
14582906,65sleep20-21swapper/207:01:002
14582906,65sleep20-21swapper/207:01:002
1639998964,9cyclictest181rcu_preempt10:36:522
163999894,37cyclictest17502-21seq12:20:362
1639998937,46cyclictest0-21swapper/209:46:342
163999893,32cyclictest21200-21fschecks_time12:26:562
1635998936,9cyclictest0-21swapper/111:57:031
163599893,36cyclictest30462-21latency_hist09:41:361
163599893,36cyclictest30462-21latency_hist09:41:361
163599893,34cyclictest4237-21kernelversion07:06:591
163599893,33cyclictest13481-21grep07:36:351
163599893,30cyclictest30067-21cpu08:26:491
1635998930,8cyclictest0-21swapper/112:12:031
1635998930,8cyclictest0-21swapper/112:12:031
163599892,32cyclictest556-21cron08:36:321
1630998963,18cyclictest10179-21latency_hist12:06:340
163099895,56cyclictest15-21ksoftirqd/010:41:080
1639998864,16cyclictest1621-21cyclictest10:21:372
163999884,72cyclictest30214-21apt-get11:41:502
163999884,72cyclictest30214-21apt-get11:41:502
163999884,31cyclictest18106-21/usr/sbin/munin12:21:532
1639998837,44cyclictest8886-21diskmemload09:52:492
163999883,32cyclictest24516-21latency_hist10:31:352
163999883,32cyclictest17657-21grep07:47:102
163599884,29cyclictest24476-21latency_hist11:31:351
1635998840,42cyclictest8886-21diskmemload09:16:541
163599883,33cyclictest7517-21grep07:17:001
163599883,33cyclictest28906-21sed08:22:001
163599883,32cyclictest29841-21latency_hist11:41:341
163599883,31cyclictest3168-21modprobe07:06:321
1635998832,7cyclictest0-21swapper/112:02:051
1635998831,13cyclictest0-21swapper/109:26:561
1630998869,12cyclictest27966-21systemd-journal10:42:540
1630998816,43cyclictest15-21ksoftirqd/011:06:480
1639998733,46cyclictest6322-21sh09:57:142
163999873,33cyclictest21587-21latency_hist08:01:352
163999873,32cyclictest6863-21munin-plugin-st07:16:322
163999873,32cyclictest2016-21cut07:01:552
1639998726,11cyclictest0-21swapper/210:05:172
163999872,37cyclictest14367-21latency_hist09:11:342
163999872,35cyclictest9363-21grep11:02:102
1635998738,14cyclictest0-21swapper/108:21:341
1635998730,9cyclictest0-21swapper/107:27:101
1635998728,9cyclictest0-21swapper/108:52:061
1635998725,13cyclictest0-21swapper/111:21:551
163599872,28cyclictest8794-21sort10:01:501
1630998761,19cyclictest15591-21latency_hist12:16:340
1630998761,19cyclictest15591-21latency_hist12:16:340
164599862,37cyclictest6151-21taskset08:51:543
163999863,33cyclictest4681-21cat08:46:582
163999863,33cyclictest4681-21cat08:46:582
163999863,33cyclictest4257-21latency_hist08:46:362
163999863,32cyclictest26670-21latency_hist08:16:352
163999863,30cyclictest356-21runrttasks11:00:542
163999863,29cyclictest996-21wc10:47:052
163999863,29cyclictest996-21wc10:47:052
163999863,29cyclictest30300-21fschecks_time08:26:562
163999862,33cyclictest356-21runrttasks11:55:492
163599863,37cyclictest356-21runrttasks07:11:451
163599863,34cyclictest22507-21timerwakeupswit08:02:071
163599863,32cyclictest7724-21rm11:00:151
163599863,32cyclictest17199-21ntp_offset10:17:031
163599863,32cyclictest17199-21ntp_offset10:17:031
163599863,31cyclictest5418-21latency_hist10:56:351
163599863,31cyclictest19021-21latency_hist11:21:341
163599863,31cyclictest19021-21latency_hist11:21:341
163599863,31cyclictest1658-21timerandwakeup09:47:101
163599863,31cyclictest1658-21timerandwakeup09:47:101
1635998629,9cyclictest0-21swapper/112:22:001
163599862,37cyclictest9731-21vmstat07:22:121
163599862,37cyclictest9731-21vmstat07:22:121
163599862,29cyclictest26862-21sh11:36:151
163599862,29cyclictest26862-21sh11:36:151
163999854,24cyclictest7829-21/usr/sbin/munin12:01:522
1639998535,44cyclictest365-21systemd-logind12:01:322
163999853,33cyclictest29762-21apt08:26:322
163999853,32cyclictest7399-21latency_hist08:56:342
163999853,31cyclictest20106-21ssh09:21:502
1639998530,47cyclictest0-21swapper/211:32:062
1639998530,47cyclictest0-21swapper/211:32:062
163599854,31cyclictest32530-21latency_hist10:46:341
163599853,36cyclictest356-21runrttasks10:14:321
163599853,31cyclictest18687-21fschecks_time07:51:551
163599853,31cyclictest18687-21fschecks_time07:51:551
1635998530,11cyclictest0-21swapper/107:42:101
1635998511,34cyclictest8966-21kworker/u17:3+flush-179:011:09:501
163099853,49cyclictest15-21ksoftirqd/011:31:580
1630998524,40cyclictest15-21ksoftirqd/010:23:550
1630998512,32cyclictest15-21ksoftirqd/011:02:080
1630998512,32cyclictest15-21ksoftirqd/011:02:080
163999845,37cyclictest4315-21latency07:07:002
1639998436,41cyclictest0-21swapper/209:12:062
163999843,34cyclictest22585-21unixbench_multi08:02:082
163999843,33cyclictest13853-21fschecks_time07:36:532
163999843,32cyclictest8936-21fschecks_count07:21:562
163999843,32cyclictest8936-21fschecks_count07:21:562
163999842,34cyclictest12883-21unixbench_multi07:32:082
163599843,34cyclictest5617-21ssh09:56:341
163599843,34cyclictest15689-21/usr/sbin/munin12:17:081
163599843,32cyclictest10651-21fschecks_time12:06:541
1630998463,14cyclictest19301-21runrttasks11:21:500
163099843,46cyclictest15-21ksoftirqd/009:12:230
1630998433,25cyclictest15-21ksoftirqd/009:47:110
1630998433,25cyclictest15-21ksoftirqd/009:47:110
1630998423,30cyclictest15-21ksoftirqd/012:27:470
1630998416,46cyclictest15-21ksoftirqd/011:18:380
1630998416,46cyclictest15-21ksoftirqd/011:18:380
163999833,36cyclictest356-21runrttasks11:14:012
163999833,34cyclictest908-21cstates09:46:512
163999833,34cyclictest908-21cstates09:46:512
1639998327,9cyclictest0-21swapper/210:41:572
1639998323,8cyclictest0-21swapper/207:26:452
163099836,54cyclictest15-21ksoftirqd/007:41:500
1630998359,18cyclictest25045-21latency_hist08:11:360
1630998359,15cyclictest30915-21ssh09:41:540
163099835,46cyclictest15-21ksoftirqd/008:31:260
163099834,45cyclictest15-21ksoftirqd/011:01:230
1630998342,9cyclictest181rcu_preempt09:00:200
163999823,35cyclictest16390-21latency_hist10:16:342
1639998233,43cyclictest0-21swapper/208:36:462
163599823,32cyclictest13759-21fschecks_count07:36:511
1630998256,19cyclictest5556-21latency_hist09:56:340
1630998254,18cyclictest27685-21latency_hist09:36:350
163099823,57cyclictest15-21ksoftirqd/010:49:430
163099823,57cyclictest15-21ksoftirqd/010:49:430
163599812,26cyclictest10651-21grep09:03:011
163599812,26cyclictest10651-21grep09:03:011
1630998159,15cyclictest1621-21cyclictest11:44:230
1630998159,15cyclictest1621-21cyclictest11:44:230
163099815,53cyclictest15-21ksoftirqd/008:17:370
163099815,43cyclictest15-21ksoftirqd/007:36:480
1630998152,21cyclictest16414-21latency_hist10:16:340
163099812,55cyclictest15-21ksoftirqd/010:34:230
163999802,26cyclictest30293-21cron09:41:322
163999802,26cyclictest30293-21cron09:41:322
1630998055,11cyclictest181rcu_preempt07:02:080
163099804,53cyclictest15-21ksoftirqd/009:57:440
163999792,22cyclictest27480-21mailstats08:17:222
163099799,45cyclictest15-21ksoftirqd/011:57:120
1630997958,14cyclictest23163-21kworker/u17:0+rpciod11:29:100
1630997948,22cyclictest5530-21latency_hist10:56:340
163099794,43cyclictest15-21ksoftirqd/007:12:020
163099794,36cyclictest15-21ksoftirqd/007:27:070
163099792,45cyclictest15-21ksoftirqd/008:01:490
163099787,33cyclictest20033-21/usr/sbin/munin07:57:070
1630997857,14cyclictest23165-21ssh09:27:020
1630997853,19cyclictest12251-21cut09:06:560
163099784,46cyclictest15-21ksoftirqd/011:52:030
163099783,51cyclictest15-21ksoftirqd/008:14:160
1630997833,21cyclictest181rcu_preempt12:25:290
163099782,45cyclictest15-21ksoftirqd/009:36:580
163099782,45cyclictest15-21ksoftirqd/009:36:580
1630997822,29cyclictest181rcu_preempt09:18:390
1630997816,36cyclictest15-21ksoftirqd/007:31:590
163099776,29cyclictest15-21ksoftirqd/008:31:450
163099773,62cyclictest14115-21fschecks_time11:11:550
163099765,37cyclictest16404-21sendmail_mailqu12:17:080
1630997653,16cyclictest376-21in:imuxsock10:02:470
1630997649,18cyclictest29964-21latency_hist08:26:350
163099763,34cyclictest1-21systemd08:41:330
133472760,3sleep30-21swapper/312:11:563
133472760,3sleep30-21swapper/312:11:563
163099753,50cyclictest15-21ksoftirqd/011:46:570
1630997525,38cyclictest22118-21ssh10:26:540
163099744,48cyclictest15-21ksoftirqd/007:22:270
163099744,48cyclictest15-21ksoftirqd/007:22:270
163099743,46cyclictest15-21ksoftirqd/007:21:340
163099743,42cyclictest15-21ksoftirqd/007:52:020
163099743,42cyclictest15-21ksoftirqd/007:52:020
1630997422,28cyclictest15-21ksoftirqd/007:47:390
164599734,38cyclictest0-21swapper/311:41:473
164599734,38cyclictest0-21swapper/311:41:473
164599734,33cyclictest14362-21chrt09:11:323
163099736,43cyclictest15-21ksoftirqd/012:08:290
163099734,35cyclictest27355-21cpuspeed11:36:480
163099734,35cyclictest27355-21cpuspeed11:36:480
163099723,37cyclictest15-21ksoftirqd/008:56:320
163099722,49cyclictest15-21ksoftirqd/009:24:150
163099714,35cyclictest15-21ksoftirqd/007:09:360
1645997020,35cyclictest0-21swapper/310:32:223
3263926713,32sleep30-21swapper/306:56:533
3263926713,32sleep30-21swapper/306:56:533
164599627,32cyclictest0-21swapper/309:02:253
164599627,32cyclictest0-21swapper/309:02:253
164599594,44cyclictest26040-21chrt09:32:073
164599592,43cyclictest2733-21chrt08:42:183
1645995911,19cyclictest0-21swapper/308:12:033
164599583,38cyclictest11825-21taskset07:31:343
164599583,37cyclictest17586-21taskset09:16:553
164599564,36cyclictest8768-21chrt07:21:503
164599564,36cyclictest8768-21chrt07:21:503
164599563,40cyclictest28311-21chrt10:37:113
164599557,35cyclictest28909-21chrt11:39:063
164599557,35cyclictest28909-21chrt11:39:063
164599554,38cyclictest24681-21chrt08:11:323
164599554,37cyclictest6085-21chrt09:56:463
164599554,32cyclictest8616-21taskset11:01:553
164599554,32cyclictest8616-21taskset11:01:553
164599553,40cyclictest22366-21taskset11:26:593
164599553,37cyclictest10144-21taskset12:06:333
164599553,36cyclictest7758-21chrt11:00:213
164599544,31cyclictest11676-21taskset10:06:553
164599543,39cyclictest19512-21taskset12:23:023
164599543,38cyclictest28563-21chrt09:37:013
164599543,38cyclictest28563-21chrt09:37:013
164599543,34cyclictest24526-21chrt10:31:363
164599535,35cyclictest7300-21chrt07:16:563
164599533,37cyclictest17386-21chrt10:17:093
164599533,37cyclictest17386-21chrt10:17:093
164599533,36cyclictest15633-21chrt07:41:593
164599533,35cyclictest555-21taskset11:46:483
164599523,38cyclictest2246-21chrt08:41:333
164599523,34cyclictest4010-21chrt07:06:523
164599523,33cyclictest7482-21chrt12:01:353
164599523,33cyclictest22383-21chrt12:28:213
164599523,33cyclictest14606-21chrt11:12:083
164599523,32cyclictest16604-21chrt11:16:543
164599523,32cyclictest16604-21chrt11:16:543
164599522,39cyclictest24548-21chrt11:31:363
164599522,39cyclictest24548-21chrt11:31:363
164599522,31cyclictest2347-21chrt07:02:023
164599515,33cyclictest22770-21chrt08:03:383
164599513,36cyclictest12687-21taskset12:11:313
164599513,34cyclictest32678-21taskset10:46:353
164599513,33cyclictest25125-21chrt09:31:343
164599513,32cyclictest19410-21chrt10:21:513
164599513,31cyclictest13706-21chrt11:11:353
164599512,33cyclictest5759-21chrt07:11:543
164599502,35cyclictest3573-21taskset09:51:523
164599502,30cyclictest12833-21taskset07:32:073
164599494,33cyclictest27829-21chrt08:17:283
164599494,32cyclictest8564-21chrt08:57:343
164599494,32cyclictest20323-21taskset07:56:543
164599494,32cyclictest17855-21chrt12:21:323
164599493,34cyclictest32350-21chrt09:44:223
164599493,33cyclictest18702-21chrt07:51:563
164599493,33cyclictest18702-21chrt07:51:563
164599493,31cyclictest15844-21chrt09:12:253
1645994931,12cyclictest3022sleep310:46:443
1645994931,12cyclictest3022sleep310:46:443
164599483,34cyclictest3538-21taskset10:52:023
164599483,34cyclictest17367-21chrt07:47:063
164599474,34cyclictest5127-21chrt08:47:093
164599474,34cyclictest5127-21chrt08:47:093
164599463,30cyclictest812-21taskset09:46:493
164599463,30cyclictest812-21taskset09:46:493
164599463,30cyclictest19974-21taskset09:21:373
164599454,29cyclictest21181-21chrt11:25:203
164599453,29cyclictest30290-21taskset08:26:553
164599444,29cyclictest10393-21taskset10:04:083
164599442,30cyclictest13589-21taskset07:36:453
164599433,30cyclictest3942-21chrt11:54:383
164599433,28cyclictest28461-21taskset08:21:493
164599433,26cyclictest31672-21taskset08:31:453
164599432,31cyclictest15077-21taskset10:12:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional