You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 22:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sat Jun 28, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12465212315,74sleep012183-21/usr/sbin/munin07:02:200
136879911963,15cyclictest15321-21kworker/u17:2-rpciod10:57:361
2540921130,6sleep20-21swapper/211:15:082
2540921130,6sleep20-21swapper/211:15:082
136949910838,33cyclictest0-21swapper/210:32:132
136949910838,33cyclictest0-21swapper/210:32:132
136779910617,59cyclictest6803-21sh12:36:140
13677991054,70cyclictest12070-21sh10:50:510
13677991054,58cyclictest0-21swapper/010:12:170
1347721046,16sleep10-21swapper/107:06:161
13677991032,83cyclictest31771-21apt-get11:27:080
13677991007,54cyclictest0-21swapper/011:58:500
1367799997,53cyclictest0-21swapper/010:37:000
1367799995,53cyclictest0-21swapper/007:57:100
1367799993,88cyclictest25823-21apt-get07:42:190
13677999928,60cyclictest0-21swapper/010:51:580
1367799992,61cyclictest26956-21/usr/sbin/munin11:17:280
1367799992,23cyclictest11670-21sh09:48:040
1367799985,80cyclictest19224-21apt-get09:02:080
1367799984,56cyclictest2854-21sed11:32:290
1367799983,90cyclictest0-21swapper/011:52:050
13677999830,57cyclictest0-21swapper/012:17:300
13677999735,57cyclictest0-21swapper/010:22:180
13694999671,18cyclictest13664-21cyclictest07:16:552
1367799964,81cyclictest0-21swapper/009:12:090
1367799962,86cyclictest26714-21fschecks_time10:17:160
13694999571,13cyclictest0-21swapper/210:17:312
1367799953,77cyclictest4982-21apt-get09:37:070
1367799953,6cyclictest0-21swapper/007:27:080
1367799953,56cyclictest0-21swapper/007:47:150
1367799952,6cyclictest0-21swapper/007:52:080
1367799952,6cyclictest0-21swapper/007:52:080
13694999451,37cyclictest20568-21diskmemload10:37:292
13677999469,13cyclictest0-21swapper/011:12:200
13677999469,13cyclictest0-21swapper/011:12:200
1367799943,83cyclictest7757-21sh11:42:060
13677999437,52cyclictest13664-21cyclictest10:35:070
13677999437,52cyclictest13664-21cyclictest10:35:070
1367799943,64cyclictest22333-21munin-node07:32:050
1367799943,58cyclictest14650-21seq09:53:540
1367799942,81cyclictest0-21swapper/011:22:080
1367799942,6cyclictest0-21swapper/010:27:060
1367799942,59cyclictest0-21swapper/011:47:110
13694999367,16cyclictest0-21swapper/207:52:182
13694999367,16cyclictest0-21swapper/207:52:182
1369499933,81cyclictest0-21swapper/208:57:072
1369499933,81cyclictest0-21swapper/208:57:072
1369499933,81cyclictest0-21swapper/208:52:042
13694999328,30cyclictest0-21swapper/211:24:322
1368799934,79cyclictest4471-21apt-get11:37:061
1368799934,79cyclictest4471-21apt-get11:37:061
13677999330,55cyclictest12033-21grep08:37:300
13677999327,59cyclictest0-21swapper/012:22:110
13677999326,59cyclictest0-21swapper/009:30:510
13677999325,58cyclictest15974-21cpuspeed07:12:110
1369499922,81cyclictest0-21swapper/211:02:102
1369499922,81cyclictest0-21swapper/211:02:102
1368799923,77cyclictest2197-21apt-get09:32:091
1368799923,77cyclictest2197-21apt-get09:32:091
1367799923,81cyclictest22431-21/usr/sbin/munin12:07:170
13677999236,49cyclictest0-21swapper/011:38:010
13677999236,49cyclictest0-21swapper/011:38:010
1367799923,61cyclictest16874-21ssh10:58:490
1367799922,78cyclictest0-21swapper/012:12:100
1367799922,78cyclictest0-21swapper/012:12:100
1367799922,52cyclictest18172-21latency_hist10:01:530
1369499913,81cyclictest0-21swapper/208:12:072
13687999147,13cyclictest181rcu_preempt10:12:431
1367799914,76cyclictest0-21swapper/012:02:070
1367799914,76cyclictest0-21swapper/012:02:070
1367799913,77cyclictest18495-21apt-get10:02:050
1367799913,56cyclictest18647-21seq08:58:060
1367799913,56cyclictest18647-21seq08:58:060
13694999067,12cyclictest0-21swapper/211:09:382
13694999062,15cyclictest0-21swapper/208:26:542
13694999062,15cyclictest0-21swapper/208:26:542
13687999069,13cyclictest26436-21acpi10:17:041
13677999064,16cyclictest331-21dbus-daemon09:46:520
1367799904,73cyclictest0-21swapper/008:12:070
13677999035,46cyclictest0-21swapper/009:17:270
1367799902,79cyclictest14349-21/usr/sbin/munin08:47:270
1367799902,76cyclictest0-21swapper/007:22:060
1367799902,76cyclictest0-21swapper/007:22:060
1367799902,23cyclictest2509-21cat09:32:430
1367799902,23cyclictest2509-21cat09:32:430
1369499892,74cyclictest0-21swapper/207:07:132
13687998944,23cyclictest19883-21ssh11:04:441
13687998944,23cyclictest19883-21ssh11:04:441
1367799895,48cyclictest18299-21expr11:02:150
1367799895,48cyclictest18299-21expr11:02:150
13677998933,48cyclictest0-21swapper/007:41:560
1367799892,79cyclictest7624-21cron08:26:510
1367799892,79cyclictest7624-21cron08:26:510
1367799892,76cyclictest0-21swapper/008:52:040
13677998925,55cyclictest0-21swapper/011:07:330
1369499882,74cyclictest0-21swapper/209:02:082
1369499882,73cyclictest0-21swapper/209:42:082
13677998831,51cyclictest0-21swapper/008:27:270
1367799882,72cyclictest1444-21apt-get08:07:040
13694998765,12cyclictest0-21swapper/212:27:172
1367799873,73cyclictest3720-21ssh12:30:510
1367799872,74cyclictest0-21swapper/007:37:080
1367799872,55cyclictest31301-21runrttasks09:26:380
13694998665,10cyclictest0-21swapper/207:36:542
1369499863,69cyclictest0-21swapper/208:17:082
1369499863,69cyclictest0-21swapper/208:17:082
1369499862,78cyclictest0-21swapper/209:12:092
1369499862,78cyclictest0-21swapper/207:27:092
13687998658,20cyclictest23103-21threads07:32:291
13677998626,54cyclictest0-21swapper/007:17:170
13677998626,54cyclictest0-21swapper/007:17:170
13694998561,14cyclictest0-21swapper/208:31:542
1369499854,70cyclictest0-21swapper/211:47:122
1369499853,70cyclictest0-21swapper/207:22:062
1369499853,70cyclictest0-21swapper/207:22:062
13677998531,49cyclictest0-21swapper/010:07:300
13677998519,59cyclictest8891-21sh10:44:100
13694998462,12cyclictest0-21swapper/208:06:542
13694998448,31cyclictest0-21swapper/207:42:412
13694998437,39cyclictest0-21swapper/212:32:282
1369499842,69cyclictest0-21swapper/208:32:112
1367799843,76cyclictest0-21swapper/009:07:050
13694998362,14cyclictest13664-21cyclictest12:22:162
13694998359,12cyclictest0-21swapper/208:11:542
1369499834,68cyclictest0-21swapper/207:47:092
1369499833,71cyclictest0-21swapper/208:47:092
1369499832,47cyclictest356-21runrttasks07:39:032
1369499832,47cyclictest356-21runrttasks07:39:032
13687998354,10cyclictest181rcu_preempt12:02:071
13687998354,10cyclictest181rcu_preempt12:02:071
1367799834,71cyclictest13844-21/usr/sbin/munin07:07:290
1367799832,72cyclictest9509-21sort08:32:070
13694998258,13cyclictest0-21swapper/210:27:252
1369499823,64cyclictest25050-21apt-get12:12:072
1369499823,64cyclictest25050-21apt-get12:12:072
1369499822,69cyclictest0-21swapper/210:02:052
13687998254,19cyclictest20568-21diskmemload11:32:221
1368799823,48cyclictest10598-21diskstats11:47:171
1367799823,70cyclictest0-21swapper/008:17:060
1367799823,70cyclictest0-21swapper/008:17:060
1367799823,68cyclictest0-21swapper/008:02:070
1367799822,51cyclictest13027-21grep08:42:160
13694998140,32cyclictest0-21swapper/210:07:252
1369499813,66cyclictest0-21swapper/210:57:102
13694998136,38cyclictest0-21swapper/212:02:312
13694998136,38cyclictest0-21swapper/212:02:312
1369499812,67cyclictest0-21swapper/210:47:122
13687998146,9cyclictest181rcu_preempt12:24:281
13694998059,12cyclictest0-21swapper/208:42:272
13694998058,12cyclictest0-21swapper/207:21:532
13694998058,12cyclictest0-21swapper/207:21:532
13694998034,41cyclictest0-21swapper/210:52:302
1368799806,49cyclictest6162-21latency_hist08:21:541
1368799806,49cyclictest6162-21latency_hist08:21:541
1368799803,45cyclictest6974-21idleruntime-cro10:41:501
13687998033,19cyclictest181rcu_preempt08:42:291
13694997942,31cyclictest0-21swapper/209:22:032
13694997930,44cyclictest0-21swapper/211:41:512
13694997930,44cyclictest0-21swapper/211:41:512
13687997957,14cyclictest17291-21systemd-run07:16:531
13687997926,13cyclictest181rcu_preempt11:46:221
13694997856,13cyclictest0-21swapper/209:56:542
13694997854,13cyclictest0-21swapper/212:21:532
1369499783,70cyclictest0-21swapper/209:07:092
1369499783,58cyclictest2024-21apt-get11:32:062
1369499782,65cyclictest0-21swapper/211:27:042
13687997860,13cyclictest4389-21/usr/sbin/munin12:32:121
13687997835,9cyclictest181rcu_preempt08:03:011
13694997761,9cyclictest0-21swapper/209:32:522
13694997761,9cyclictest0-21swapper/209:32:522
1369499774,47cyclictest0-21swapper/209:17:302
1369499773,68cyclictest0-21swapper/210:12:102
1369499772,70cyclictest0-21swapper/208:37:092
13687997750,18cyclictest20725-21/usr/sbin/munin07:27:211
1368799774,64cyclictest12366-21cron10:51:511
13687997741,26cyclictest29175-21kworker/u18:2+nfsiod12:01:061
13687997740,14cyclictest29175-21kworker/u18:2+rpciod10:22:251
1357927712,40sleep213592-21munin-run07:06:542
13694997653,13cyclictest0-21swapper/209:47:172
13694997630,38cyclictest0-21swapper/211:56:552
13687997657,14cyclictest18869-21sh10:02:181
13687997654,15cyclictest22399-21rm09:08:481
13687997640,12cyclictest181rcu_preempt09:37:211
13694997551,15cyclictest20456-21kworker/u17:2-rpciod12:07:292
13694997531,37cyclictest0-21swapper/207:57:112
13694997531,33cyclictest0-21swapper/211:21:542
1368799758,41cyclictest25317-21apt-get07:42:061
13687997551,15cyclictest11481-21unixbench_singl09:47:321
13687997539,14cyclictest29472-21ssh09:22:211
1369499744,64cyclictest0-21swapper/211:42:092
13687997455,11cyclictest58-21kcompactd011:08:541
13687997450,18cyclictest26567-21sh09:17:151
13687997448,8cyclictest181rcu_preempt11:13:551
13687997448,8cyclictest181rcu_preempt11:13:551
13687997444,9cyclictest181rcu_preempt09:29:061
1368799742,46cyclictest30529-21latency_hist07:56:541
1368799742,46cyclictest30529-21latency_hist07:56:541
1368799742,40cyclictest3038-21head10:32:321
1368799742,40cyclictest3038-21head10:32:321
13694997317,49cyclictest7359-21latency_hist09:41:542
13687997339,14cyclictest10092-21sh09:46:511
13687997338,12cyclictest181rcu_preempt08:22:221
13687997338,12cyclictest181rcu_preempt08:22:221
1368799732,26cyclictest31396-21wc07:57:291
13687997321,9cyclictest0-21swapper/109:57:121
1370199722,44cyclictest0-21swapper/311:37:063
1370199722,44cyclictest0-21swapper/311:37:063
13694997229,37cyclictest0-21swapper/210:22:092
13694997228,34cyclictest0-21swapper/209:57:022
13687997248,15cyclictest12773-21cpuspeed10:52:101
13687997244,17cyclictest0-21swapper/109:56:541
1368799724,31cyclictest28711-21ssh12:17:271
13687997236,12cyclictest181rcu_preempt11:29:491
13687997233,16cyclictest16224-21cut08:52:151
1368799722,36cyclictest2389-21unixbench_singl08:07:291
1368799722,24cyclictest144842/usr/sbin/munin07:07:301
13694997157,8cyclictest0-21swapper/209:31:542
1369499713,48cyclictest0-21swapper/210:42:112
1368799719,46cyclictest18863-21modprobe07:21:511
1368799719,46cyclictest18863-21modprobe07:21:511
13687997154,11cyclictest331-21dbus-daemon12:16:521
13687997154,11cyclictest331-21dbus-daemon12:16:521
13687997150,15cyclictest13664-21cyclictest10:27:491
1368799713,49cyclictest356-21runrttasks11:22:411
1368799713,26cyclictest21235-21ssh10:07:051
13687997131,9cyclictest181rcu_preempt08:47:091
13687997129,12cyclictest181rcu_preempt11:54:461
1368799712,42cyclictest10860-21munin-run08:36:511
13687997054,10cyclictest15447-21munin-node10:46:181
13687997047,14cyclictest0-21swapper/111:17:191
1368799703,46cyclictest356-21runrttasks07:49:551
13687997026,26cyclictest8088-21fschecks_count08:27:151
13687997016,11cyclictest181rcu_preempt08:37:361
13694996928,34cyclictest0-21swapper/211:57:072
13687996942,9cyclictest181rcu_preempt12:29:511
13687996939,9cyclictest181rcu_preempt07:23:321
13687996939,9cyclictest181rcu_preempt07:23:321
1368799693,45cyclictest24169-21fschecks_count07:37:151
1368799693,45cyclictest24169-21fschecks_count07:37:151
1368799692,31cyclictest22092-21apt-get12:07:081
132492699,33sleep30-21swapper/307:03:173
1370199683,34cyclictest0-21swapper/309:32:103
1370199683,34cyclictest0-21swapper/309:32:103
13687996749,11cyclictest202-21systemd-journal09:16:511
13687996745,17cyclictest17590-21/usr/sbin/munin08:57:121
13687996745,17cyclictest17590-21/usr/sbin/munin08:57:121
1368799672,45cyclictest19481-21fschecks_time09:02:171
13687996721,9cyclictest181rcu_preempt08:13:341
1370199604,38cyclictest900-21kworker/3:1+events07:12:303
1370199603,39cyclictest27200-21chrt10:17:283
1370199602,41cyclictest900-21kworker/3:1+events10:56:513
1370199595,30cyclictest0-21swapper/310:36:513
1370199595,30cyclictest0-21swapper/310:36:513
1370199594,39cyclictest2722-21taskset11:32:273
1370199593,42cyclictest1702-21chrt09:31:523
1370199583,41cyclictest14546-21chrt07:07:313
1370199583,39cyclictest22671-21taskset11:10:113
1370199583,38cyclictest30993-21chrt12:22:113
1370199567,37cyclictest3737-21taskset12:30:543
1370199564,37cyclictest9114-21chrt11:43:253
1370199563,37cyclictest27693-21chrt09:18:433
1370199563,33cyclictest28865-21chrt07:51:543
1370199562,36cyclictest21062-21taskset09:07:123
1370199554,34cyclictest21479-21chrt10:07:153
1370199553,38cyclictest19862-21chrt11:04:403
1370199553,38cyclictest19862-21chrt11:04:403
1370199553,38cyclictest17960-21chrt11:59:223
1370199553,37cyclictest18036-21chrt07:17:233
1370199553,37cyclictest18036-21chrt07:17:233
1370199553,34cyclictest9838-21chrt08:32:203
1370199553,34cyclictest900-21kworker/3:1+events09:06:523
13701995511,34cyclictest0-21swapper/307:42:063
1370199545,27cyclictest0-21swapper/310:41:503
1370199543,38cyclictest20840-21taskset10:06:423
1370199543,34cyclictest13293-21chrt08:42:253
1370199538,33cyclictest0-21swapper/311:01:523
1370199534,34cyclictest13915-21chrt11:52:253
1370199533,37cyclictest729-21chrt11:28:343
1370199533,37cyclictest22903-21taskset07:32:263
1370199533,35cyclictest7710-21chrt09:42:103
1370199533,34cyclictest10666-21chrt11:47:183
1370199532,37cyclictest7416-21chrt10:42:113
1370199532,36cyclictest2066-21chrt08:07:263
1370199532,35cyclictest17824-21taskset10:00:343
1370199523,38cyclictest15990-21taskset08:52:093
1370199523,36cyclictest11197-21chrt09:47:293
1370199523,35cyclictest23973-21chrt07:37:043
1370199523,35cyclictest23973-21chrt07:37:043
1370199523,34cyclictest8424-21chrt08:27:223
1370199523,34cyclictest30930-21chrt09:25:243
1370199513,35cyclictest20775-21taskset12:04:003
1370199513,35cyclictest20775-21taskset12:04:003
1370199513,34cyclictest24911-21chrt12:11:543
1370199512,36cyclictest7241-21chrt12:36:543
1370199512,34cyclictest27755-21chrt11:18:383
1370199512,34cyclictest19029-21chrt09:01:533
1370199512,34cyclictest19029-21chrt09:01:533
1370199503,35cyclictest29275-21chrt07:52:153
1370199503,35cyclictest29275-21chrt07:52:153
1370199503,34cyclictest12394-21chrt10:51:513
1370199503,33cyclictest24517-21chrt11:12:343
1370199503,33cyclictest24517-21chrt11:12:343
1370199503,32cyclictest20492-21chrt07:26:513
1370199503,32cyclictest20492-21chrt07:26:513
1370199502,33cyclictest32212-21chrt08:01:553
1370199502,32cyclictest26325-21chrt10:16:553
1370199502,31cyclictest28524-21chrt12:17:213
1370199492,34cyclictest32384-21taskset10:27:203
1370199492,32cyclictest3542-21chrt08:12:243
1370199492,32cyclictest25159-21taskset09:14:173
1370199483,34cyclictest15515-21chrt08:49:343
1370199483,33cyclictest7486-21chrt09:41:543
1370199483,31cyclictest14582-21chrt09:53:413
1370199482,33cyclictest26279-21chrt12:12:553
1370199482,33cyclictest26279-21chrt12:12:553
1370199473,32cyclictest30441-21chrt10:23:113
1370199464,30cyclictest29913-21chrt11:22:313
1370199413,27cyclictest4977-21taskset08:17:183
1370199413,27cyclictest4977-21taskset08:17:183
1370199412,26cyclictest20890-21taskset07:27:123
1370199403,28cyclictest7281-21taskset08:22:323
1370199403,28cyclictest7281-21taskset08:22:323
1370199393,25cyclictest1016-21taskset08:04:593
1370199383,24cyclictest11152-21taskset08:37:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional