You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 11:51
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Sep 16, 2025 00:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31060213512,91sleep131131-21awk18:57:171
318149910623,42cyclictest27406-21seq20:26:091
3149921065,14sleep00-21swapper/018:59:070
318209910116,48cyclictest39-21ksoftirqd/219:47:262
318209910116,48cyclictest39-21ksoftirqd/219:47:262
3182099977,61cyclictest24669-21df21:32:372
3182099977,61cyclictest24669-21df21:32:372
31814999641,43cyclictest28191-21apt-get23:31:421
31814999641,43cyclictest28191-21apt-get23:31:421
31814999619,42cyclictest26029-21seq21:34:181
31814999619,42cyclictest26029-21seq21:34:181
3182099952,62cyclictest1860-21taskset19:06:492
31814999579,10cyclictest25278-21ssh23:26:421
73592940,6sleep30-21swapper/323:51:483
31814999274,10cyclictest0-21swapper/121:56:551
31811999273,12cyclictest23904-21apt-key21:31:330
31811999273,12cyclictest23904-21apt-key21:31:330
31811999174,10cyclictest10017-21sh22:02:090
31811999165,19cyclictest27227-21sh22:32:510
31811999165,19cyclictest27227-21sh22:32:510
31814999044,39cyclictest0-21swapper/121:01:451
3181499902,48cyclictest21728-21seq00:17:441
31814999016,41cyclictest9489-21seq19:29:281
31814999016,41cyclictest9489-21seq19:29:281
31811999052,27cyclictest0-21swapper/000:07:120
3182099893,56cyclictest21751-21cpu00:17:482
31814998971,9cyclictest0-21swapper/122:46:461
31814998971,9cyclictest0-21swapper/122:46:461
31814998942,35cyclictest22982-21cron22:26:291
31814998923,32cyclictest356-21runrttasks21:19:381
31814998923,32cyclictest356-21runrttasks21:19:381
31814998916,34cyclictest1830-21/usr/sbin/munin19:08:021
31814998866,9cyclictest0-21swapper/120:16:331
31814998847,35cyclictest7931-21kworker/u19:1-rpciod23:56:321
3181499883,49cyclictest28725-21seq20:27:221
3181499883,49cyclictest28725-21seq20:27:221
31814998772,8cyclictest0-21swapper/119:06:331
31811998724,30cyclictest0-21swapper/021:31:360
31811998724,30cyclictest0-21swapper/021:31:360
3182099862,44cyclictest39-21ksoftirqd/221:37:102
31814998647,33cyclictest0-21swapper/100:16:331
31814998647,33cyclictest0-21swapper/100:16:331
31811998664,15cyclictest384-21sshd20:09:340
31811998654,23cyclictest29049-21modprobe20:31:290
31811998654,23cyclictest29049-21modprobe20:31:290
31811998632,23cyclictest31802-21cyclictest23:41:570
31820998570,9cyclictest16608-21tune2fs19:51:532
31820998514,39cyclictest39-21ksoftirqd/220:07:182
31814998562,11cyclictest0-21swapper/120:01:561
31814998560,11cyclictest0-21swapper/122:11:311
31811998564,11cyclictest0-21swapper/020:56:510
31811998564,11cyclictest0-21swapper/020:56:510
3182699843,42cyclictest0-21swapper/321:32:373
3182699843,42cyclictest0-21swapper/321:32:373
31820998455,21cyclictest951-21apt-key19:06:332
31811998470,10cyclictest31802-21cyclictest22:47:080
31811998470,10cyclictest31802-21cyclictest22:47:080
31811998469,8cyclictest0-21swapper/000:02:500
31811998465,9cyclictest0-21swapper/022:16:320
31811998465,9cyclictest0-21swapper/021:41:520
31811998459,17cyclictest0-21swapper/022:42:030
31811998459,17cyclictest0-21swapper/022:42:030
31811998425,41cyclictest0-21swapper/022:22:150
31814998371,7cyclictest0-21swapper/119:31:541
31814998371,7cyclictest0-21swapper/119:31:541
31811998365,9cyclictest0-21swapper/023:12:040
31811998365,9cyclictest0-21swapper/023:12:040
31811998359,16cyclictest6348-21diskmemload23:07:050
31811998244,29cyclictest0-21swapper/000:21:530
31820998156,18cyclictest31802-21cyclictest23:07:062
31820998155,17cyclictest356-21runrttasks00:06:572
3182099814,29cyclictest39-21ksoftirqd/219:33:502
3182099814,29cyclictest39-21ksoftirqd/219:33:502
31820998118,34cyclictest39-21ksoftirqd/200:02:322
31814998137,36cyclictest0-21swapper/121:47:091
3181499813,37cyclictest356-21runrttasks22:28:531
31814998130,12cyclictest181rcu_preempt21:31:291
31814998130,12cyclictest181rcu_preempt21:31:291
31811998163,7cyclictest0-21swapper/019:11:480
31811998139,31cyclictest0-21swapper/000:16:330
31811998139,31cyclictest0-21swapper/000:16:330
31811998124,26cyclictest0-21swapper/021:17:210
31811998124,26cyclictest0-21swapper/021:17:210
31811998123,21cyclictest3040-21sh21:51:310
31820998053,21cyclictest21198-21apt-key00:16:342
31820998053,21cyclictest21198-21apt-key00:16:342
31814998042,32cyclictest13357-21kworker/u17:0-rpciod22:46:211
31814998042,32cyclictest13357-21kworker/u17:0-rpciod22:46:211
31814998040,33cyclictest0-21swapper/119:56:551
31814998036,38cyclictest0-21swapper/119:26:321
31814998020,31cyclictest356-21runrttasks20:56:051
31820997963,10cyclictest28966-21latency_hist22:36:322
31820997963,10cyclictest28966-21latency_hist22:36:322
3182099792,34cyclictest39-21ksoftirqd/220:44:522
3182099792,34cyclictest39-21ksoftirqd/220:44:522
31814997961,12cyclictest0-21swapper/121:51:561
31814997959,10cyclictest0-21swapper/123:06:591
31814997955,14cyclictest0-21swapper/123:02:051
3181499793,42cyclictest356-21runrttasks22:21:271
31811997958,14cyclictest24726-21fschecks_time20:16:530
31811997953,22cyclictest31802-21cyclictest21:07:010
31811997953,22cyclictest31802-21cyclictest21:07:010
3182699784,38cyclictest0-21swapper/300:01:563
31820997861,10cyclictest4843-21cut19:16:322
31820997857,14cyclictest27966-21systemd-journal19:30:452
31820997857,14cyclictest27966-21systemd-journal19:30:452
31820997822,33cyclictest39-21ksoftirqd/220:55:532
31814997862,8cyclictest0-21swapper/123:12:001
31814997862,8cyclictest0-21swapper/123:12:001
31814997861,9cyclictest0-21swapper/100:07:101
3181499783,51cyclictest1380-21seq20:45:281
3181499783,51cyclictest1380-21seq20:45:281
31820997754,15cyclictest27966-21systemd-journal19:19:242
3182099772,22cyclictest19852-21fschecks_count20:01:552
31814997757,10cyclictest0-21swapper/100:22:031
31814997743,28cyclictest13357-21kworker/u17:0-rpciod22:12:001
3181499773,48cyclictest32195-21seq20:40:101
31814997732,37cyclictest0-21swapper/120:31:531
31814997732,37cyclictest0-21swapper/120:31:531
31814997731,38cyclictest0-21swapper/122:01:481
31811997759,9cyclictest0-21swapper/023:56:530
31811997759,9cyclictest0-21swapper/023:56:530
31811997759,9cyclictest0-21swapper/020:36:540
31820997660,9cyclictest0-21swapper/219:01:352
31820997655,14cyclictest365-21systemd-logind19:41:312
31820997655,14cyclictest365-21systemd-logind19:41:312
31820997652,8cyclictest181rcu_preempt22:42:312
31820997652,8cyclictest181rcu_preempt22:42:312
31820997642,18cyclictest29636-21perl20:31:532
31820997642,18cyclictest29636-21perl20:31:532
31820997640,20cyclictest19809-21diskstats23:16:512
3182099763,33cyclictest39-21ksoftirqd/223:00:492
31814997659,9cyclictest0-21swapper/100:26:501
31814997659,9cyclictest0-21swapper/100:26:501
31814997654,15cyclictest14295-21unixbench_singl00:02:591
31814997630,36cyclictest0-21swapper/119:46:461
31814997630,36cyclictest0-21swapper/119:46:461
31811997661,9cyclictest29847-21ssh23:33:490
31811997661,9cyclictest29847-21ssh23:33:490
31811997658,9cyclictest0-21swapper/021:52:110
31811997657,9cyclictest0-21swapper/019:26:320
31811997630,17cyclictest19009-21kworker/u17:1+xprtiod23:18:550
31811997616,21cyclictest7366-21sh22:54:580
31820997551,18cyclictest6348-21diskmemload21:13:032
31820997551,18cyclictest6348-21diskmemload21:13:032
31820997533,11cyclictest6348-21diskmemload23:49:402
31820997533,11cyclictest6348-21diskmemload23:49:402
31814997535,33cyclictest0-21swapper/122:32:561
31814997535,33cyclictest0-21swapper/122:32:561
31811997561,7cyclictest0-21swapper/021:13:030
31811997561,7cyclictest0-21swapper/021:13:030
31811997558,11cyclictest31802-21cyclictest20:47:270
31811997558,11cyclictest31802-21cyclictest20:47:270
31811997552,11cyclictest0-21swapper/023:21:520
31811997551,18cyclictest1129-21latency_hist19:06:330
31811997550,18cyclictest26045-21apt-key22:31:330
31820997447,8cyclictest181rcu_preempt22:22:052
31820997439,19cyclictest6348-21diskmemload21:59:022
31820997434,29cyclictest6313-21timerandwakeup22:52:202
31820997419,29cyclictest39-21ksoftirqd/222:02:022
31814997456,9cyclictest0-21swapper/122:36:501
31814997453,11cyclictest0-21swapper/123:41:051
31814997437,30cyclictest0-21swapper/123:26:331
31814997425,39cyclictest27369-21tune2fs21:36:531
31811997456,8cyclictest0-21swapper/022:16:520
31811997450,18cyclictest27999-21latency_hist23:31:330
31811997448,8cyclictest0-21swapper/023:47:000
31811997448,8cyclictest0-21swapper/023:47:000
31811997439,24cyclictest336-21tune2fs20:41:560
31811997439,24cyclictest336-21tune2fs20:41:560
3182099734,25cyclictest39-21ksoftirqd/221:05:122
31814997355,9cyclictest0-21swapper/120:47:021
31814997355,9cyclictest0-21swapper/120:47:021
3181499733,60cyclictest0-21swapper/120:06:431
31814997333,32cyclictest0-21swapper/122:56:291
31814997332,32cyclictest0-21swapper/121:21:591
31814997330,35cyclictest0-21swapper/123:20:331
31820997248,13cyclictest0-21swapper/220:26:332
31820997221,39cyclictest22927-21cat23:21:542
31814997253,12cyclictest0-21swapper/121:12:211
31814997253,12cyclictest0-21swapper/121:12:211
3181499724,39cyclictest356-21runrttasks19:41:021
3181499724,39cyclictest356-21runrttasks19:41:021
31814997233,33cyclictest0-21swapper/123:47:001
31814997233,33cyclictest0-21swapper/123:47:001
31814997231,33cyclictest0-21swapper/122:56:521
31811997259,9cyclictest17007-21kworker/u17:1+rpciod00:26:530
31811997259,9cyclictest17007-21kworker/u17:1+rpciod00:26:530
31811997256,10cyclictest8041-21/usr/sbin/munin23:52:070
31811997250,13cyclictest1385-21latency_hist23:41:330
3182099715,26cyclictest39-21ksoftirqd/220:59:462
3182099715,26cyclictest39-21ksoftirqd/220:59:462
31820997150,8cyclictest181rcu_preempt22:27:002
3182099713,31cyclictest12929-21latency_hist00:01:342
3182099713,31cyclictest12929-21latency_hist00:01:342
31820997118,30cyclictest39-21ksoftirqd/221:41:362
3181499714,57cyclictest9940-21ssh21:07:011
3181499714,57cyclictest9940-21ssh21:07:011
31814997139,23cyclictest5424-21if_eth019:16:551
31811997153,12cyclictest0-21swapper/019:17:070
31811997149,14cyclictest0-21swapper/019:41:470
31811997149,14cyclictest0-21swapper/019:41:470
3182099702,36cyclictest13210-21cpu19:41:472
3182099702,36cyclictest13210-21cpu19:41:472
3182099702,24cyclictest39-21ksoftirqd/220:40:262
31820997019,29cyclictest39-21ksoftirqd/200:28:312
31820997019,29cyclictest39-21ksoftirqd/200:28:312
31820997013,21cyclictest381ktimers/221:46:532
31820997011,35cyclictest39-21ksoftirqd/220:13:342
31811997055,8cyclictest0-21swapper/019:26:590
31811997055,8cyclictest0-21swapper/019:26:590
31811997055,10cyclictest14750irq/292-2188000.ethernet21:56:420
31811997051,9cyclictest0-21swapper/022:41:330
31811997048,16cyclictest22459-21sh00:18:120
31811997043,20cyclictest0-21swapper/019:51:300
31811997043,20cyclictest0-21swapper/019:51:300
31820996943,8cyclictest181rcu_preempt23:37:182
3182099694,26cyclictest39-21ksoftirqd/220:48:162
3182099694,26cyclictest39-21ksoftirqd/220:48:162
31820996938,7cyclictest181rcu_preempt22:39:262
31820996925,31cyclictest331-21dbus-daemon00:22:042
3182099692,25cyclictest39-21ksoftirqd/223:29:232
31820996917,20cyclictest371rcuc/223:32:022
31820996917,20cyclictest371rcuc/223:32:022
31820996916,30cyclictest39-21ksoftirqd/222:48:332
31820996916,30cyclictest39-21ksoftirqd/222:48:332
31820996913,34cyclictest39-21ksoftirqd/222:19:122
3181499694,54cyclictest30193-21cut21:41:521
3181499693,53cyclictest3495-21apt-get19:11:431
3181499693,52cyclictest13178-21sshd19:41:461
3181499693,52cyclictest13178-21sshd19:41:461
31811996951,9cyclictest0-21swapper/023:02:000
31811996948,15cyclictest14568-21sh22:11:320
3182099683,26cyclictest39-21ksoftirqd/220:17:142
31820996820,35cyclictest1-21systemd21:11:312
31820996820,35cyclictest1-21systemd21:11:312
31820996819,20cyclictest39-21ksoftirqd/219:57:012
31820996811,24cyclictest381ktimers/222:06:472
3181499683,52cyclictest16637-21/usr/sbin/munin19:51:531
31814996828,32cyclictest0-21swapper/123:56:491
31814996828,32cyclictest0-21swapper/123:56:491
31811996854,10cyclictest31802-21cyclictest20:33:500
31811996854,10cyclictest31802-21cyclictest20:33:500
3181199682,25cyclictest8238-21runrttasks22:56:480
31820996712,24cyclictest381ktimers/223:46:302
3181499672,52cyclictest2896-21seq23:43:061
31811996752,8cyclictest0-21swapper/019:36:460
31811996752,8cyclictest0-21swapper/019:36:460
31811996751,8cyclictest0-21swapper/021:02:080
31811996751,8cyclictest0-21swapper/019:57:020
31811996749,11cyclictest6348-21diskmemload21:38:200
31811996740,20cyclictest4387-21kworker/u17:1+nfsiod21:22:470
3182099669,40cyclictest21485-21uname21:26:552
3182099669,40cyclictest21485-21uname21:26:552
31820996646,8cyclictest181rcu_preempt21:51:422
31820996645,9cyclictest181rcu_preempt21:18:062
31820996645,9cyclictest181rcu_preempt21:18:062
3182099663,27cyclictest7005-21idleruntime19:21:542
31820996614,21cyclictest381ktimers/223:16:312
31820996614,21cyclictest381ktimers/223:16:312
31820996611,19cyclictest381ktimers/223:01:452
31814996629,30cyclictest0-21swapper/120:17:011
31814996622,36cyclictest6328-21modprobe21:01:301
31814996622,36cyclictest6328-21modprobe21:01:301
31811996651,8cyclictest0-21swapper/019:51:480
31820996546,7cyclictest181rcu_preempt23:55:202
31820996534,7cyclictest181rcu_preempt21:24:382
31820996514,20cyclictest381ktimers/222:12:182
31811996538,19cyclictest0-21swapper/019:07:570
3181199647,48cyclictest20219-21wc20:02:050
31811996428,29cyclictest0-21swapper/020:54:420
3182699633,42cyclictest0-21swapper/322:32:193
3182699633,42cyclictest0-21swapper/322:32:193
31811996341,15cyclictest11753-21fschecks_count19:36:560
31811996341,15cyclictest11753-21fschecks_count19:36:560
31811996331,23cyclictest26926-21cat20:22:030
31826996212,28cyclictest0-21swapper/300:17:453
3182099624,25cyclictest39-21ksoftirqd/220:30:022
3182099624,25cyclictest39-21ksoftirqd/220:30:022
3181199603,31cyclictest23303-21latency20:11:550
3182699594,33cyclictest0-21swapper/323:51:343
3182699594,33cyclictest0-21swapper/323:51:343
3182699589,40cyclictest1887-21taskset19:07:093
3168125812,28sleep30-21swapper/319:01:303
3182699573,28cyclictest0-21swapper/321:12:183
3182699573,28cyclictest0-21swapper/321:12:183
3182699554,39cyclictest17731-21chrt00:09:493
3182699553,40cyclictest24697-21taskset23:25:413
3182699553,36cyclictest8162-21chrt22:56:393
3182699543,38cyclictest14096-21chrt23:06:523
3182699533,39cyclictest837-21taskset22:42:323
3182699533,39cyclictest837-21taskset22:42:323
3182699533,39cyclictest32614-21chrt23:38:303
3182699533,35cyclictest24294-21taskset20:16:323
3182699532,37cyclictest26157-21taskset00:25:583
3182699532,36cyclictest16733-21chrt19:51:553
3182699524,36cyclictest13812-21chrt22:09:393
31826995236,9cyclictest451rcuc/320:08:183
3182699523,36cyclictest23658-21taskset21:31:303
3182699523,36cyclictest23658-21taskset21:31:303
3182699522,39cyclictest21434-21chrt23:19:213
3182699522,37cyclictest8568-21chrt22:01:043
3182699514,36cyclictest27726-21chrt00:27:123
3182699514,36cyclictest27726-21chrt00:27:123
31826995134,9cyclictest451rcuc/321:25:143
31826995134,9cyclictest451rcuc/320:32:193
31826995134,9cyclictest451rcuc/320:32:193
3182699513,37cyclictest3263-21chrt22:47:063
3182699513,37cyclictest3263-21chrt22:47:063
3182699513,35cyclictest11807-21chrt19:36:563
3182699513,35cyclictest11807-21chrt19:36:563
3182699513,33cyclictest19486-21chrt23:16:333
3182699513,33cyclictest19486-21chrt23:16:333
3182699503,38cyclictest20144-21taskset22:21:303
3182699503,36cyclictest22160-21chrt22:24:023
3182699503,35cyclictest21366-21chrt00:16:343
3182699503,35cyclictest21366-21chrt00:16:343
3182699503,34cyclictest5120-21chrt19:16:483
3182699503,32cyclictest9043-21taskset19:27:053
3182699503,32cyclictest9043-21taskset19:27:053
3182699503,32cyclictest26953-21chrt20:22:033
3182699502,38cyclictest6600-21chrt22:52:373
31826994933,8cyclictest451rcuc/319:22:313
3182699493,37cyclictest9279-21taskset22:01:533
3182699493,35cyclictest28699-21chrt20:27:093
3182699493,35cyclictest28699-21chrt20:27:093
3182699493,34cyclictest31136-21chrt20:36:533
3182699493,34cyclictest18000-21chrt21:21:333
3182699493,34cyclictest18000-21chrt21:21:333
3182699493,34cyclictest15348-21chrt22:11:583
3182699493,33cyclictest3943-21chrt19:11:583
3182699493,32cyclictest28808-21chrt23:32:023
3182699493,32cyclictest28808-21chrt23:32:023
3182699492,36cyclictest25857-21taskset20:21:303
3182699492,33cyclictest3678-21chrt23:45:283
3182699483,35cyclictest10138-21chrt23:56:473
3182699483,35cyclictest10138-21chrt23:56:473
3182699483,34cyclictest16166-21taskset19:51:303
3182699483,34cyclictest16166-21taskset19:51:303
3182699483,32cyclictest12051-21chrt21:11:323
3182699483,32cyclictest12051-21chrt21:11:323
3182699482,30cyclictest5509-21taskset20:56:573
3182699482,30cyclictest5509-21taskset20:56:573
3182699473,36cyclictest14604-21taskset19:46:323
3182699473,36cyclictest14604-21taskset19:46:323
3182699473,34cyclictest29402-21chrt22:36:523
3182699473,31cyclictest26169-21chrt23:27:073
3182699473,31cyclictest20019-21chrt20:01:593
3182699473,31cyclictest13492-21chrt23:06:193
3182699473,30cyclictest31035-21chrt21:42:073
31826994732,9cyclictest451rcuc/320:54:483
3182699472,34cyclictest2301-21chrt20:47:013
3182699472,34cyclictest2301-21chrt20:47:013
3182699464,31cyclictest25057-21chrt22:29:043
3182699462,33cyclictest2387-21chrt21:49:453
3182699453,32cyclictest3658-21chrt21:51:563
3182699453,30cyclictest11093-21chrt19:34:053
3182699453,30cyclictest11093-21chrt19:34:053
3182699452,30cyclictest32755-21chrt19:02:063
3182699446,26cyclictest28934-21chrt21:39:243
31826994329,8cyclictest451rcuc/320:01:313
3182699412,27cyclictest8024-21taskset21:02:573
3182699392,24cyclictest1591-21chrt20:46:333
3182699392,24cyclictest1591-21chrt20:46:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional