You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-16 - 15:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Mon Jun 16, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1899921350,6sleep118998-21systemd-run09:17:421
62639912055,59cyclictest3734-21kernelversion09:48:100
6263991152,24cyclictest3732-21idleruntime-cro08:37:410
6263991133,41cyclictest11771-21ssh12:02:570
62639911237,46cyclictest0-21swapper/011:27:570
6263991093,27cyclictest7244-21latency_hist08:47:420
62639910916,63cyclictest2450-21seq09:46:020
6263991084,27cyclictest18147-21sh09:15:150
6263991083,29cyclictest7767-21sh10:55:200
6263991083,25cyclictest356-21runrttasks09:36:400
6263991072,25cyclictest25031-21taskset09:27:590
6263991063,26cyclictest29400-21runrttasks10:35:270
6263991063,23cyclictest15141-21ssh11:09:360
739421050,3sleep1291rcuc/110:54:011
6263991053,25cyclictest356-21runrttasks10:30:390
6263991053,25cyclictest356-21runrttasks10:30:390
62789910447,16cyclictest181rcu_preempt09:55:371
6263991043,25cyclictest12680-21sh10:04:220
6263991043,25cyclictest12680-21sh10:04:220
6263991043,22cyclictest5958-21latency_hist10:52:440
6263991042,27cyclictest7784-21sh11:55:260
6263991042,26cyclictest356-21runrttasks08:11:000
6263991042,25cyclictest22917-21seq07:56:570
6263991042,24cyclictest10126-21ssh09:59:370
6263991042,24cyclictest10126-21ssh09:59:370
6263991033,25cyclictest24543-21seq08:01:430
6263991033,23cyclictest32083-21sh10:40:210
6263991033,23cyclictest32083-21sh10:40:210
6263991032,22cyclictest31487-21taskset09:38:330
6263991023,28cyclictest19164-21latency_hist10:17:440
6263991023,25cyclictest356-21runrttasks08:22:220
6263991023,23cyclictest4809-21ssh11:50:090
6263991022,23cyclictest23554-21ssh09:23:480
6263991022,23cyclictest23344-21ssh11:25:480
6117210228,48sleep30-21swapper/307:07:403
6263991014,24cyclictest7569-21taskset07:09:560
6263991013,8cyclictest0-21swapper/012:08:000
6263991013,8cyclictest0-21swapper/012:08:000
6263991013,28cyclictest13872-21sh09:07:440
6263991013,25cyclictest21986-21sh09:22:360
6263991013,25cyclictest21986-21sh09:22:360
6263991013,24cyclictest21426-21sh10:21:180
6263991013,21cyclictest12842-21ssh11:05:220
62639910125,43cyclictest0-21swapper/011:38:010
62639910125,43cyclictest0-21swapper/011:38:010
6263991012,26cyclictest356-21runrttasks12:19:160
6263991012,26cyclictest2133-21sh11:45:430
6263991012,25cyclictest8561-21taskset08:49:340
6263991012,24cyclictest27842-21ssh12:29:530
6263991012,24cyclictest27842-21ssh12:29:530
6263991012,23cyclictest10518-21ssh11:01:180
6263991012,22cyclictest23700-21ssh10:25:020
6263991012,22cyclictest11869-21seq09:00:230
62789910051,10cyclictest181rcu_preempt11:37:441
6263991003,27cyclictest14960-21cat10:08:210
6263991003,21cyclictest19787-21/usr/sbin/munin11:18:200
6263991002,24cyclictest28263-21ssh11:33:520
6263991002,22cyclictest24993-21ssh12:25:130
6263991002,22cyclictest24993-21ssh12:25:130
6263991002,21cyclictest2677-21ssh10:46:410
6263991002,21cyclictest2677-21ssh10:46:410
6263991002,21cyclictest10044-21seq07:17:240
6263991002,21cyclictest10044-21seq07:17:240
626399992,23cyclictest29579-21ssh12:33:270
626399982,27cyclictest356-21runrttasks09:11:390
626399982,23cyclictest10196-21ssh11:58:560
626399982,23cyclictest10196-21ssh11:58:560
6263999742,48cyclictest17304-21/usr/sbin/munin12:13:030
626399973,29cyclictest356-21runrttasks07:28:520
626399972,24cyclictest1864-21seq08:29:450
626399972,23cyclictest17953-21taskset07:40:020
626399972,22cyclictest7463-21ssh09:54:530
6278999669,19cyclictest6845-21threads11:53:201
626399963,23cyclictest5528-21taskset08:42:060
626399962,22cyclictest13174-21taskset07:25:570
6278999547,8cyclictest181rcu_preempt12:03:191
627899952,62cyclictest25220-21cpuspeed10:28:181
627899952,62cyclictest25220-21cpuspeed10:28:181
626399952,29cyclictest20220-21cut07:48:030
627899944,70cyclictest19485-21/usr/sbin/munin09:18:161
627899944,70cyclictest19485-21/usr/sbin/munin09:18:161
626399943,27cyclictest17148-21unixbench_multi11:13:190
626399943,27cyclictest17148-21unixbench_multi11:13:190
626399942,26cyclictest356-21runrttasks08:22:470
626399935,24cyclictest28387-21apt-get08:13:180
626399933,24cyclictest10228-21runrttasks08:55:220
626399932,27cyclictest356-21runrttasks08:05:010
626399932,27cyclictest356-21runrttasks08:05:010
515229328,37sleep015-21ksoftirqd/007:03:130
6283999216,38cyclictest0-21swapper/211:38:042
6283999216,38cyclictest0-21swapper/211:38:042
6278999251,24cyclictest11602-21ssh12:02:441
6278999251,24cyclictest11602-21ssh12:02:441
628399912,69cyclictest19724-21expr09:18:152
628399912,69cyclictest19724-21expr09:18:152
6283999120,30cyclictest0-21swapper/212:34:122
6278999156,12cyclictest181rcu_preempt07:18:011
626399912,23cyclictest19649-21taskset07:46:140
627899906,66cyclictest28343-21apt-get08:12:531
6278999061,9cyclictest181rcu_preempt12:35:011
6278999050,17cyclictest181rcu_preempt10:38:421
6278999050,17cyclictest181rcu_preempt10:38:421
626399903,25cyclictest15207-21/usr/sbin/munin07:32:560
628399893,65cyclictest4240-21apt-get08:37:572
6278998961,12cyclictest181rcu_preempt11:46:391
6278998953,14cyclictest181rcu_preempt11:00:481
6278998951,12cyclictest181rcu_preempt10:44:551
6278998951,12cyclictest181rcu_preempt10:44:551
628399883,68cyclictest25216-21cpuspeed10:28:182
628399883,68cyclictest25216-21cpuspeed10:28:182
6283998819,34cyclictest8340-21idleruntime-cro09:57:402
6283998811,37cyclictest0-21swapper/209:27:442
6278998831,12cyclictest181rcu_preempt09:23:041
626399883,26cyclictest356-21runrttasks07:17:460
6283998716,30cyclictest27953-21timerwakeupswit11:33:192
6283998714,26cyclictest0-21swapper/209:33:252
6278998747,13cyclictest181rcu_preempt09:37:531
6278998743,10cyclictest0-21swapper/108:32:591
6283998633,22cyclictest30501-21ntp_kernel_pll_08:18:112
6283998611,34cyclictest0-21swapper/208:53:052
6278998646,12cyclictest181rcu_preempt10:14:361
6278998632,16cyclictest181rcu_preempt10:02:411
6278998632,16cyclictest181rcu_preempt10:02:411
6283998514,40cyclictest0-21swapper/207:47:442
6278998562,15cyclictest365-21systemd-logind08:47:421
6278998551,11cyclictest181rcu_preempt12:23:581
6278998551,11cyclictest181rcu_preempt12:23:581
6278998551,11cyclictest181rcu_preempt08:40:141
6278998542,15cyclictest6257-21cyclictest07:23:171
6278998540,30cyclictest19326-21grep07:43:181
6278998454,14cyclictest181rcu_preempt08:57:401
6278998434,9cyclictest181rcu_preempt11:23:051
6283998322,29cyclictest14651-21cstates12:08:012
6283998322,29cyclictest14651-21cstates12:08:012
6283998321,26cyclictest356-21runrttasks10:39:332
6283998321,26cyclictest356-21runrttasks10:39:332
6283998265,11cyclictest14328-21fschecks_count10:08:052
6283998234,36cyclictest26507-21apt-get08:07:552
6283998220,26cyclictest0-21swapper/208:28:082
6283998219,31cyclictest4017-21latency_hist08:37:442
6283998215,37cyclictest0-21swapper/210:18:012
6283998214,29cyclictest0-21swapper/209:42:422
6283998212,41cyclictest0-21swapper/211:13:162
6283998212,41cyclictest0-21swapper/211:13:162
6278998260,15cyclictest6257-21cyclictest10:18:221
6278998258,9cyclictest181rcu_preempt12:10:071
6278998258,9cyclictest181rcu_preempt12:10:071
6278998245,14cyclictest181rcu_preempt08:23:251
6278998243,13cyclictest181rcu_preempt08:18:051
6278998237,16cyclictest181rcu_preempt09:45:561
6278998235,12cyclictest181rcu_preempt09:03:551
60492826,62sleep20-21swapper/207:06:452
508628214,25sleep10-21swapper/107:03:101
6283998161,15cyclictest26342-21latency_hist08:07:422
6283998161,15cyclictest26342-21latency_hist08:07:422
6283998122,24cyclictest13279-21seq07:27:202
6283998116,26cyclictest24003-21mailstats12:23:172
6283998116,26cyclictest24003-21mailstats12:23:172
6278998150,16cyclictest181rcu_preempt11:22:221
6278998147,13cyclictest181rcu_preempt07:32:571
6278998146,13cyclictest181rcu_preempt11:06:511
6278998145,13cyclictest181rcu_preempt07:08:021
6278998136,14cyclictest181rcu_preempt07:17:431
6278998136,14cyclictest181rcu_preempt07:17:431
6283998025,22cyclictest10174-21seq09:59:482
6283998025,22cyclictest10174-21seq09:59:482
6283998024,25cyclictest24339-21seq07:59:042
6283998023,21cyclictest10844-21seq11:02:312
6283998022,25cyclictest29551-21seq08:15:562
6283998021,24cyclictest6880-21seq08:43:422
6283998020,30cyclictest0-21swapper/212:20:012
6283998015,25cyclictest25152-21grep11:28:202
6283998014,27cyclictest28104-21ssh10:33:112
6278998045,11cyclictest181rcu_preempt09:36:501
6278998045,11cyclictest181rcu_preempt08:51:561
6278998034,9cyclictest181rcu_preempt11:47:571
6283997963,10cyclictest18959-21kworker/u17:3-rpciod11:06:062
6283997924,25cyclictest14443-21wc11:08:182
6283997924,23cyclictest22534-21ssh10:23:082
6283997921,22cyclictest15693-21taskset09:10:332
6283997919,31cyclictest0-21swapper/210:48:502
6283997917,27cyclictest0-21swapper/207:28:162
6283997914,29cyclictest0-21swapper/212:03:042
6283997914,26cyclictest0-21swapper/211:23:052
6278997952,12cyclictest181rcu_preempt09:10:561
6278997936,10cyclictest181rcu_preempt12:13:051
6283997842,11cyclictest0-21swapper/207:37:432
6283997821,27cyclictest18641-21seq10:15:412
6283997817,28cyclictest1300-21cut10:43:162
6283997817,28cyclictest1300-21cut10:43:162
6283997814,28cyclictest10566-21fschecks_count07:18:042
6283997814,28cyclictest0-21swapper/207:13:352
6283997814,28cyclictest0-21swapper/207:13:352
6278997845,12cyclictest181rcu_preempt09:30:381
6278997845,10cyclictest181rcu_preempt10:08:071
6278997843,13cyclictest181rcu_preempt07:58:161
6278997842,14cyclictest181rcu_preempt07:53:011
6278997838,11cyclictest181rcu_preempt10:24:151
6283997722,24cyclictest26575-21seq09:29:572
6283997720,25cyclictest2134-21seq11:45:432
6283997714,28cyclictest0-21swapper/209:13:012
6283997712,27cyclictest0-21swapper/211:18:212
6278997746,8cyclictest181rcu_preempt11:41:241
6278997746,8cyclictest181rcu_preempt11:41:241
6278997739,13cyclictest181rcu_preempt12:29:281
6278997739,13cyclictest181rcu_preempt12:29:281
628399769,31cyclictest0-21swapper/208:27:442
6283997620,27cyclictest2716-21chrt09:46:532
6283997616,28cyclictest4151-21cat09:48:202
6278997661,10cyclictest14768-21unixbench_multi11:08:221
6278997653,8cyclictest181rcu_preempt09:52:411
6278997645,12cyclictest181rcu_preempt08:03:191
6278997645,12cyclictest181rcu_preempt08:03:191
6278997619,21cyclictest0-21swapper/111:12:591
6278997619,21cyclictest0-21swapper/111:12:591
629099755,33cyclictest30254-21chrt12:34:123
6283997521,23cyclictest8108-21seq10:56:272
6283997520,24cyclictest8196-21taskset11:56:402
6283997514,30cyclictest29393-21acpi12:32:582
6283997514,30cyclictest29393-21acpi12:32:582
6283997514,27cyclictest11787-21runrttasks08:59:172
6283997512,24cyclictest7198-21cat07:08:142
6283997511,34cyclictest0-21swapper/209:07:432
627899759,19cyclictest31-21ksoftirqd/111:29:251
6278997559,10cyclictest1-21systemd10:52:401
6278997541,12cyclictest181rcu_preempt12:18:481
6278997537,13cyclictest181rcu_preempt10:05:271
6278997537,13cyclictest181rcu_preempt10:05:271
6283997455,13cyclictest17090-21fschecks_time07:38:032
6283997422,24cyclictest12099-21taskset10:03:222
6283997422,24cyclictest12099-21taskset10:03:222
6283997415,23cyclictest17718-21ssh12:13:102
6283997414,29cyclictest20790-21cat07:48:162
6278997442,13cyclictest181rcu_preempt07:40:411
6278997435,15cyclictest181rcu_preempt07:52:401
6278997435,10cyclictest181rcu_preempt08:58:181
6283997320,23cyclictest11163-21seq12:01:532
6283997320,23cyclictest11163-21seq12:01:532
6283997316,23cyclictest4155-21grep11:48:242
6278997344,8cyclictest181rcu_preempt08:07:501
6278997338,8cyclictest181rcu_preempt08:28:161
6283997255,12cyclictest23121-21latency_hist07:57:432
6283997229,31cyclictest0-21swapper/208:47:582
627899722,40cyclictest31-21ksoftirqd/107:29:471
629099712,28cyclictest21639-21chrt11:23:003
6278997145,8cyclictest181rcu_preempt10:35:381
629099703,31cyclictest11324-21taskset10:02:593
629099703,31cyclictest11324-21taskset10:02:593
629099702,31cyclictest6970-21chrt10:53:303
629099693,29cyclictest21366-21chrt10:20:553
629099692,28cyclictest10561-21taskset07:18:043
629099684,28cyclictest7373-21chrt07:08:163
629099683,26cyclictest22157-21chrt10:22:583
629099672,25cyclictest27613-21chrt11:33:143
629099663,24cyclictest3778-21chrt08:37:413
629099662,27cyclictest25095-21chrt08:03:043
629099662,27cyclictest25095-21chrt08:03:043
629099662,24cyclictest20164-21chrt11:18:553
629099653,26cyclictest8771-21chrt08:52:203
629099653,26cyclictest12066-21chrt09:02:403
629099653,25cyclictest27855-21chrt09:33:003
629099653,25cyclictest18166-21chrt10:14:103
629099653,24cyclictest26514-21chrt08:07:583
629099652,27cyclictest3931-21taskset09:48:163
629099652,27cyclictest26872-21chrt09:31:063
629099652,26cyclictest30360-21chrt11:38:123
629099652,26cyclictest30360-21chrt11:38:123
629099652,26cyclictest10114-21chrt09:59:353
629099652,26cyclictest10114-21chrt09:59:353
629099652,24cyclictest24711-21chrt09:27:413
629099644,28cyclictest950-21chrt09:43:093
629099643,24cyclictest25143-21chrt10:28:173
629099643,24cyclictest25143-21chrt10:28:173
629099643,24cyclictest17098-21taskset09:13:143
629099643,23cyclictest28118-21chrt10:33:113
629099642,27cyclictest5710-21taskset10:52:293
629099642,26cyclictest1987-21chrt10:44:263
629099642,26cyclictest1987-21chrt10:44:263
629099642,25cyclictest12065-21chrt12:03:053
629099633,28cyclictest31493-21taskset08:22:443
629099633,27cyclictest6890-21taskset08:43:493
629099633,24cyclictest21523-21chrt09:21:043
629099633,24cyclictest21523-21chrt09:21:043
629099633,22cyclictest16098-21chrt10:11:263
629099633,22cyclictest15402-21taskset11:11:053
629099633,22cyclictest13739-21chrt07:28:003
629099632,26cyclictest17227-21chrt11:13:203
629099632,26cyclictest17227-21chrt11:13:203
629099632,25cyclictest2289-21chrt11:46:113
629099632,24cyclictest32447-21taskset08:23:163
629099632,24cyclictest1762-21chrt08:28:273
629099632,23cyclictest20273-21chrt12:17:533
6290996312,41cyclictest0-21swapper/307:38:033
629099624,23cyclictest13619-21chrt09:07:143
629099623,24cyclictest31818-21chrt10:39:273
629099623,24cyclictest31818-21chrt10:39:273
629099623,22cyclictest6504-21chrt09:53:153
629099623,22cyclictest27525-21chrt12:29:003
629099623,22cyclictest27525-21chrt12:29:003
629099623,21cyclictest28345-21chrt08:12:523
629099622,23cyclictest9735-21chrt07:13:393
629099622,23cyclictest9735-21chrt07:13:393
629099622,23cyclictest20021-21chrt07:47:493
6290996211,38cyclictest0-21swapper/311:48:153
629099613,43cyclictest6899-21taskset11:53:203
629099613,24cyclictest9934-21chrt10:58:283
629099613,24cyclictest16395-21chrt12:10:473
629099613,24cyclictest16395-21chrt12:10:473
629099613,23cyclictest18418-21chrt07:42:513
629099613,23cyclictest13113-21chrt07:25:093
629099613,21cyclictest9764-21chrt08:53:173
629099613,21cyclictest24328-21chrt07:58:543
629099612,24cyclictest5393-21chrt08:40:213
6290996111,40cyclictest0-21swapper/312:23:173
6290996111,40cyclictest0-21swapper/312:23:173
629099607,13cyclictest0-21swapper/311:58:063
629099607,13cyclictest0-21swapper/311:58:063
629099603,21cyclictest21646-21chrt07:52:513
6290996010,40cyclictest0-21swapper/307:33:023
629099593,25cyclictest32394-21taskset09:41:313
629099593,22cyclictest24426-21chrt11:28:033
629099593,20cyclictest14031-21chrt09:07:593
629099592,23cyclictest17752-21taskset12:13:113
629099583,21cyclictest11237-21chrt11:02:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional