You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 00:24
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Jul 18, 2025 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2056121330,11sleep2381ktimers/207:00:512
21086212912,33sleep30-21swapper/307:01:143
219809910535,28cyclictest29311-21latency_hist09:05:330
219809910535,28cyclictest29311-21latency_hist09:05:330
219809910344,22cyclictest21873-21ssh09:48:130
219809910344,22cyclictest21873-21ssh09:48:130
219809910341,28cyclictest8221-21ssh12:10:030
219809910341,24cyclictest15841-21ssh12:21:230
219809910341,24cyclictest15841-21ssh12:21:230
219809910139,28cyclictest356-21runrttasks10:05:590
219809910139,26cyclictest9169-21taskset10:21:400
219809910139,26cyclictest9169-21taskset10:21:400
21980991007,78cyclictest5567-21apt-get11:10:470
219809910040,24cyclictest16992-21ssh11:30:110
219809910039,24cyclictest19882-21ssh11:35:000
219809910039,24cyclictest19882-21ssh11:35:000
219809910033,26cyclictest2544-21latency_hist12:00:320
21980999942,22cyclictest27052-21ssh10:51:150
21980999941,24cyclictest27918-21ssh09:58:540
21980999941,24cyclictest27918-21ssh09:58:540
21980999940,39cyclictest25204-21ssh10:49:250
21980999940,24cyclictest24589-21ssh09:52:410
21980999940,24cyclictest10558-21ssh09:28:490
21980999939,24cyclictest3041-21taskset07:42:380
21980999939,23cyclictest21847-21ssh10:43:580
21980999938,25cyclictest10310-21ssh12:12:320
21980999937,25cyclictest5071-21sh09:19:130
137262990,5sleep30-21swapper/309:34:563
137262990,5sleep30-21swapper/309:34:563
21980999842,21cyclictest7070-21fschecks_time07:55:560
21980999840,24cyclictest2998-21sh12:00:510
21980999840,23cyclictest18831-21ssh09:42:470
21980999840,20cyclictest30136-21sed07:26:060
21980999839,27cyclictest6379-21ssh10:16:570
21980999839,24cyclictest19527-21ssh12:28:280
21980999839,23cyclictest27533-21ssh11:46:250
21980999839,22cyclictest29184-21diskmemload10:02:190
21980999837,28cyclictest0-21swapper/007:46:040
21980999837,28cyclictest0-21swapper/007:46:040
21980999830,38cyclictest13227-21latency_hist08:15:330
21980999743,24cyclictest30814-21taskset11:52:200
21980999739,23cyclictest12128-21ssh10:26:510
21980999739,23cyclictest12128-21ssh10:26:510
21980999735,29cyclictest22575-21sh11:39:090
21992999650,10cyclictest181rcu_preempt07:35:451
21980999640,24cyclictest22569-21ssh12:33:410
21980999638,23cyclictest356-21runrttasks09:13:300
21980999551,22cyclictest25769-21kworker/u18:1-writeback10:56:070
21980999540,25cyclictest13974-21ssh12:19:310
21980999540,19cyclictest15903-21seq08:21:100
21980999539,27cyclictest17833-21taskset08:30:260
21980999539,26cyclictest3821-21ssh11:07:080
21980999538,23cyclictest1333-21ssh11:03:290
21980999538,18cyclictest28931-21taskset07:24:230
21980999537,24cyclictest3506-21taskset10:12:070
21980999534,24cyclictest25436-21seq07:14:130
21980999532,29cyclictest21343-21/usr/sbin/munin08:40:570
21980999441,24cyclictest30770-21ssh09:06:410
21980999440,19cyclictest1309-21taskset07:39:080
21980999341,21cyclictest11076-21seq08:06:330
21980999341,21cyclictest11076-21seq08:06:330
21980999340,23cyclictest27328-21taskset07:19:240
21980999340,22cyclictest14300-21taskset08:16:130
21980999339,25cyclictest19231-21seq08:32:440
21980999338,24cyclictest23258-21taskset07:07:270
21992999266,18cyclictest23699-21systemctl07:10:341
21980999240,20cyclictest19882-21cstates08:35:500
21980999238,21cyclictest32145-21taskset07:34:270
21980999238,21cyclictest32145-21taskset07:34:270
21980999238,20cyclictest16696-21seq09:40:080
21980999233,22cyclictest0-21swapper/010:35:570
21980999233,22cyclictest0-21swapper/010:35:570
22000999171,14cyclictest31302-21cron10:05:292
21980999141,19cyclictest7112-21ssh09:21:520
21980999137,19cyclictest13061-21sh09:32:490
21980999137,19cyclictest13061-21sh09:32:490
21980999136,19cyclictest13127-21sh11:22:360
21980999136,19cyclictest13127-21sh11:22:360
2200099906,40cyclictest39-21ksoftirqd/208:45:432
2199299902,45cyclictest25556-21apt-get07:15:591
21980999040,41cyclictest10018-21latency_hist08:05:330
21980999034,26cyclictest15495-21ssh10:32:340
21980999033,26cyclictest23165-21fschecks_count08:45:490
22000998956,10cyclictest181rcu_preempt09:18:112
21980998938,19cyclictest25853-21seq08:54:500
21980998938,19cyclictest25853-21seq08:54:500
21980998935,22cyclictest27523-21taskset09:00:110
21980998934,19cyclictest0-21swapper/008:01:060
2199299881,42cyclictest13959-21ssh10:30:501
21980998839,21cyclictest29184-21diskmemload11:42:280
206662887,56sleep120672-21df_inode07:00:551
22000998755,8cyclictest181rcu_preempt09:31:002
22000998755,8cyclictest181rcu_preempt09:31:002
22000998753,7cyclictest181rcu_preempt09:09:352
21992998764,11cyclictest0-21swapper/109:16:261
21992998737,12cyclictest23000-21kworker/u19:0-flush-0:3010:48:341
22000998645,9cyclictest181rcu_preempt10:16:072
21980998640,16cyclictest10976-21ssh11:19:550
22000998262,14cyclictest8236-21latency_hist08:00:332
22000998258,15cyclictest23750-21munin-node08:17:092
22000998220,49cyclictest25556-21apt-get07:15:442
2199299822,33cyclictest30558-21unixbench_singl09:06:101
2200099817,57cyclictest22957-21apt-get10:45:522
2200099815,40cyclictest39-21ksoftirqd/210:10:472
22000998151,8cyclictest181rcu_preempt07:05:562
22000998130,11cyclictest0-21swapper/211:01:192
21992998160,14cyclictest23912-21unixbench_singl08:46:041
21992998159,15cyclictest28977-21latency_hist11:50:331
22000998041,25cyclictest20130-21systemctl11:35:312
22000998041,25cyclictest20130-21systemctl11:35:312
21992998060,13cyclictest21971-21cyclictest12:26:071
21992998051,12cyclictest13817-21ssh12:19:041
21992998043,12cyclictest31-21ksoftirqd/111:33:101
21992998043,12cyclictest31-21ksoftirqd/111:33:101
21980998029,42cyclictest5244-21cstates07:50:480
21980998029,42cyclictest5244-21cstates07:50:480
22000997959,14cyclictest378-21rs:main2
22000997957,14cyclictest12545-21ssh12:16:052
22000997952,8cyclictest181rcu_preempt07:39:022
22000997944,13cyclictest181rcu_preempt11:59:022
22000997937,12cyclictest181rcu_preempt07:30:502
22000997937,12cyclictest181rcu_preempt07:30:502
21992997941,15cyclictest31-21ksoftirqd/107:46:071
21992997941,15cyclictest31-21ksoftirqd/107:46:071
22000997863,10cyclictest10577-21ntp_offset08:05:582
22000997863,10cyclictest10577-21ntp_offset08:05:582
22000997855,14cyclictest17522-21latency_hist12:25:332
22000997855,14cyclictest17522-21latency_hist12:25:332
22000997854,9cyclictest181rcu_preempt09:10:592
22000997845,7cyclictest181rcu_preempt07:20:492
21992997841,9cyclictest181rcu_preempt11:45:301
2199299782,65cyclictest1932-21apt-get07:40:441
2200099777,40cyclictest39-21ksoftirqd/210:55:582
22000997761,10cyclictest21971-21cyclictest11:42:112
22000997755,8cyclictest181rcu_preempt11:26:062
22000997747,8cyclictest12907-21ssh10:29:112
22000997747,8cyclictest12907-21ssh10:29:112
22000997710,53cyclictest1932-21apt-get07:40:542
21992997744,12cyclictest181rcu_preempt09:21:361
21992997718,13cyclictest356-21runrttasks08:25:491
22000997653,15cyclictest1-21systemd08:42:022
22000997642,12cyclictest181rcu_preempt08:30:572
21992997657,12cyclictest21971-21cyclictest09:32:071
21992997657,12cyclictest21971-21cyclictest09:32:071
21992997650,17cyclictest29184-21diskmemload09:46:051
21992997650,17cyclictest29184-21diskmemload09:46:051
21992997640,12cyclictest181rcu_preempt11:19:141
2199299763,44cyclictest23674-21sendmail_mailst09:51:021
21992997617,31cyclictest0-21swapper/111:21:031
21992997617,31cyclictest0-21swapper/111:21:031
22000997556,12cyclictest19638-21latency_hist10:40:342
22000997556,12cyclictest19638-21latency_hist10:40:342
22000997555,14cyclictest5010-21idleruntime-cro11:10:302
2200099753,44cyclictest3792-21tune2fs07:45:532
2200099753,44cyclictest3792-21tune2fs07:45:532
21992997550,17cyclictest8217-21sh11:15:301
22000997458,10cyclictest378-21rs:main2
22000997438,8cyclictest0-21swapper/210:34:592
21992997449,12cyclictest31-21ksoftirqd/110:58:131
21992997445,16cyclictest21971-21cyclictest10:05:091
2199299744,36cyclictest5337-21apt-config11:10:341
21992997416,43cyclictest15310-21switchtime11:26:051
22000997351,8cyclictest181rcu_preempt09:25:542
22000997350,9cyclictest0-21swapper/208:55:302
22000997350,9cyclictest0-21swapper/208:55:302
21992997348,9cyclictest181rcu_preempt09:36:221
21992997345,8cyclictest181rcu_preempt12:05:471
21992997344,20cyclictest16245-21latency_hist08:25:301
22000997254,13cyclictest58-21kcompactd009:05:272
22000997254,13cyclictest58-21kcompactd009:05:272
22000997253,13cyclictest29184-21diskmemload12:13:502
22000997250,8cyclictest181rcu_preempt09:25:332
22000997240,8cyclictest181rcu_preempt07:30:342
22000997233,14cyclictest181rcu_preempt10:21:032
22000997233,14cyclictest181rcu_preempt10:21:032
22000997231,15cyclictest181rcu_preempt10:10:002
21992997249,16cyclictest31937-21munin-plugin-st11:55:311
21992997248,16cyclictest331-21dbus-daemon10:30:301
21992997248,16cyclictest331-21dbus-daemon10:30:301
21992997224,36cyclictest13092-21latency_hist08:15:321
21980997240,25cyclictest22024-21wc07:05:340
22000997155,11cyclictest5501-21irqstats07:50:592
22000997155,11cyclictest5501-21irqstats07:50:592
22000997154,12cyclictest20511-21threads08:36:082
22000997145,8cyclictest181rcu_preempt11:45:402
22000997145,7cyclictest181rcu_preempt11:38:272
2200099713,46cyclictest356-21runrttasks09:42:282
21992997144,9cyclictest181rcu_preempt12:01:471
21992997142,10cyclictest181rcu_preempt08:35:191
21992997129,17cyclictest7705-21tail07:56:101
2199299712,42cyclictest31-21ksoftirqd/112:13:551
2199299711,35cyclictest6944-21kworker/u18:0+flush-179:008:05:491
2199299711,35cyclictest6944-21kworker/u18:0+flush-179:008:05:491
22000997044,8cyclictest181rcu_preempt09:50:512
22000997035,8cyclictest181rcu_preempt11:52:562
2200099703,45cyclictest22401-21latency_hist10:45:342
22000997034,14cyclictest181rcu_preempt10:53:572
22000997029,9cyclictest0-21swapper/209:35:472
21992997054,9cyclictest26736-21sh10:51:061
21992997050,13cyclictest23750-21munin-node10:15:331
21992997048,12cyclictest0-21swapper/108:20:301
2199299703,44cyclictest356-21runrttasks09:03:511
2199299703,44cyclictest356-21runrttasks09:03:511
2199299702,39cyclictest2469-21ssh12:00:301
22000996952,12cyclictest29184-21diskmemload09:56:062
22000996952,12cyclictest29184-21diskmemload09:56:062
22000996946,8cyclictest181rcu_preempt08:25:322
22000996937,10cyclictest181rcu_preempt12:28:492
22000996928,11cyclictest181rcu_preempt12:34:032
21992996946,15cyclictest0-21swapper/107:30:531
21992996946,15cyclictest0-21swapper/107:30:531
21992996943,9cyclictest181rcu_preempt09:43:031
21992996943,8cyclictest181rcu_preempt10:40:521
22000996847,8cyclictest181rcu_preempt08:55:592
22000996843,11cyclictest181rcu_preempt12:03:112
22000996841,10cyclictest181rcu_preempt11:20:542
22000996841,10cyclictest181rcu_preempt11:20:542
22000996831,14cyclictest181rcu_preempt09:49:552
22000996831,14cyclictest181rcu_preempt09:49:552
22000996812,21cyclictest39-21ksoftirqd/211:20:322
21992996848,11cyclictest0-21swapper/109:10:471
21992996845,14cyclictest58-21kcompactd007:55:481
2199299683,29cyclictest21136-21grep08:40:321
21992996831,11cyclictest0-21swapper/110:21:031
21992996831,11cyclictest0-21swapper/110:21:031
21992996811,23cyclictest0-21swapper/107:10:401
22000996745,8cyclictest181rcu_preempt08:26:042
22000996745,8cyclictest181rcu_preempt08:00:492
22000996745,8cyclictest181rcu_preempt08:00:492
22000996741,11cyclictest181rcu_preempt12:05:492
21992996743,15cyclictest1541-21ssh10:09:521
21992996734,15cyclictest181rcu_preempt11:05:021
21992996732,13cyclictest181rcu_preempt10:35:541
21992996732,13cyclictest181rcu_preempt10:35:541
21992996730,12cyclictest181rcu_preempt12:34:571
22000996650,10cyclictest12180-21ntp_offset08:10:592
21992996643,14cyclictest0-21swapper/107:30:331
21992996643,12cyclictest0-21swapper/109:30:331
21992996630,14cyclictest5134-21chrt10:15:511
21992996620,9cyclictest181rcu_preempt11:35:591
2199299653,35cyclictest0-21swapper/108:01:051
2199299653,35cyclictest0-21swapper/108:01:051
21992996531,7cyclictest181rcu_preempt07:51:091
21992996521,32cyclictest24819-21cut08:50:531
21992996521,32cyclictest24819-21cut08:50:531
2199299643,41cyclictest28335-21mailstats07:21:051
21992996326,28cyclictest22427-21sshd08:42:021
21992996320,16cyclictest181rcu_preempt08:58:591
21992996230,12cyclictest181rcu_preempt12:25:081
21992996230,12cyclictest181rcu_preempt12:25:081
21992996226,12cyclictest181rcu_preempt09:55:591
21992996226,12cyclictest181rcu_preempt09:55:591
274862600,6sleep30-21swapper/311:46:173
2200699564,37cyclictest704-21taskset09:11:053
2200699563,38cyclictest14558-21chrt10:31:333
2200699555,33cyclictest0-21swapper/307:16:023
2200699554,37cyclictest28027-21taskset07:20:563
2200699553,39cyclictest6664-21chrt12:06:063
2200699553,37cyclictest30204-21chrt11:51:033
321382540,6sleep30-21swapper/309:10:333
2200699544,34cyclictest14509-21chrt11:25:473
2200699543,36cyclictest3910-21chrt10:13:173
2200699543,34cyclictest31533-21chrt07:31:053
2200699543,34cyclictest31533-21chrt07:31:053
2200699542,39cyclictest5506-21chrt09:20:303
2200699542,37cyclictest16514-21taskset08:25:423
22006995412,27cyclictest0-21swapper/311:10:513
2200699534,35cyclictest13451-21chrt10:30:313
2200699534,35cyclictest13451-21chrt10:30:313
2200699533,38cyclictest10975-21taskset09:30:063
2200699533,36cyclictest8370-21taskset08:00:363
2200699533,36cyclictest8370-21taskset08:00:363
2200699533,35cyclictest19742-21chrt10:40:403
2200699533,35cyclictest18027-21chrt10:36:273
2200699533,35cyclictest18027-21chrt10:36:273
2200699533,34cyclictest23654-21chrt11:40:543
2200699533,34cyclictest20066-21chrt09:45:423
2200699533,34cyclictest20066-21chrt09:45:423
2200699533,33cyclictest19637-21chrt08:35:343
2200699523,35cyclictest32158-21chrt11:00:553
2200699523,34cyclictest29859-21chrt07:26:023
2200699522,39cyclictest20626-21chrt11:35:523
2200699513,36cyclictest6295-21chrt10:16:423
2200699513,34cyclictest4835-21chrt11:10:023
2200699513,34cyclictest28586-21chrt09:01:053
2200699513,34cyclictest28586-21chrt09:01:053
2200699513,33cyclictest22954-21chrt08:45:383
2200699513,31cyclictest11778-21chrt11:20:523
2200699513,31cyclictest11778-21chrt11:20:523
2200699512,36cyclictest26484-21chrt09:55:593
2200699512,36cyclictest26484-21chrt09:55:593
2200699512,35cyclictest4503-21chrt07:46:073
2200699512,35cyclictest4503-21chrt07:46:073
2200699512,33cyclictest24124-21chrt09:51:173
2200699504,28cyclictest0-21swapper/310:46:113
2200699503,35cyclictest10443-21chrt12:12:553
2200699503,34cyclictest30457-21chrt10:02:563
2200699503,34cyclictest22048-21chrt08:41:053
2200699503,34cyclictest15097-21taskset09:36:063
2200699503,32cyclictest17503-21chrt09:40:563
2200699503,30cyclictest31820-21chrt10:05:523
2200699502,36cyclictest26233-21chrt10:50:543
2200699502,36cyclictest13501-21chrt12:18:103
2200699502,35cyclictest8448-21chrt10:20:593
2200699502,35cyclictest8448-21chrt10:20:593
2200699502,33cyclictest19069-21chrt11:32:403
2200699502,33cyclictest19069-21chrt11:32:403
2200699502,32cyclictest23991-21chrt07:10:393
22006995013,26cyclictest30814-21taskset10:58:273
2200699497,20cyclictest0-21swapper/307:40:513
2200699493,32cyclictest21213-21chrt08:40:323
2200699492,35cyclictest22425-21chrt07:05:533
2200699492,33cyclictest18396-21chrt12:26:043
2200699483,32cyclictest7468-21chrt07:56:073
2200699482,34cyclictest6856-21taskset07:55:473
2200699482,34cyclictest6856-21taskset07:55:473
2200699482,30cyclictest9728-21chrt11:16:193
22006994813,23cyclictest13645-21chrt08:15:553
22006994810,26cyclictest14693-21chrt12:20:493
22006994810,26cyclictest14693-21chrt12:20:493
22006994712,23cyclictest4773-21chrt12:03:393
2200699463,31cyclictest6110-21chrt09:20:543
2200699463,29cyclictest2087-21chrt11:59:263
2200699453,31cyclictest24817-21chrt08:50:533
2200699453,31cyclictest24817-21chrt08:50:533
22006994412,22cyclictest10141-21chrt08:05:473
22006994412,22cyclictest10141-21chrt08:05:473
22006994412,20cyclictest12709-21chrt08:11:543
22006994411,21cyclictest0-21swapper/312:31:023
22006994411,20cyclictest32545-21chrt07:35:463
22006994311,21cyclictest26219-21chrt08:55:423
2200699423,27cyclictest15166-21taskset08:20:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional