You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-19 - 22:43
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Sun Oct 19, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
225279911585,24cyclictest15419-21latency_hist11:39:562
2226121066,17sleep20-21swapper/207:08:102
225179910180,15cyclictest5160-21latency_hist07:54:571
225139910167,25cyclictest15459-21latency_hist11:39:570
22517999974,14cyclictest0-21swapper/109:00:181
53982960,6sleep00-21swapper/012:16:350
53982960,6sleep00-21swapper/012:16:350
22527999675,15cyclictest18892-21kworker/u17:0-nfsiod10:10:432
22527999675,15cyclictest18892-21kworker/u17:0-nfsiod10:10:432
22527999673,12cyclictest0-21swapper/208:45:172
22527999576,12cyclictest6382-21kworker/u17:0-rpciod09:25:322
22527999572,13cyclictest0-21swapper/208:40:292
22517999579,8cyclictest0-21swapper/108:25:121
22513999518,32cyclictest15-21ksoftirqd/007:57:570
22517999378,8cyclictest0-21swapper/110:16:361
2244329313,50sleep11-21systemd07:09:561
22527999276,9cyclictest0-21swapper/210:55:182
22527999273,9cyclictest0-21swapper/211:25:222
2251399923,40cyclictest24379-21apt-get11:55:110
219082929,53sleep021987-21grep07:05:430
22527999177,10cyclictest22499-21cyclictest11:45:222
22527999176,8cyclictest0-21swapper/207:15:182
2251399914,35cyclictest15-21ksoftirqd/007:49:230
22527999070,9cyclictest0-21swapper/208:00:162
22527999069,13cyclictest446-21cron07:29:532
22527999069,13cyclictest446-21cron07:29:532
22527998965,16cyclictest1541-21latency_hist11:14:552
22517998966,17cyclictest0-21swapper/110:25:211
22527998871,7cyclictest0-21swapper/208:35:232
22527998870,9cyclictest0-21swapper/207:30:182
22513998854,14cyclictest181rcu_preempt10:56:270
22513998829,40cyclictest0-21swapper/012:05:010
22527998649,10cyclictest181rcu_preempt09:35:202
22517998664,13cyclictest0-21swapper/109:40:131
22527998565,14cyclictest11654-21cron08:14:532
22527998565,14cyclictest11654-21cron08:14:532
22527998548,7cyclictest181rcu_preempt09:31:042
22517998569,9cyclictest0-21swapper/109:10:261
22517998569,9cyclictest0-21swapper/109:10:261
22517998568,9cyclictest0-21swapper/108:45:171
22517998563,11cyclictest0-21swapper/112:24:571
22517998560,16cyclictest26139-21awk07:19:541
22513998556,10cyclictest181rcu_preempt12:35:310
22513998556,10cyclictest181rcu_preempt12:35:310
22513998543,8cyclictest181rcu_preempt10:10:520
22513998543,8cyclictest181rcu_preempt10:10:520
22513998510,44cyclictest15-21ksoftirqd/009:19:510
22527998470,8cyclictest0-21swapper/208:20:142
22527998463,10cyclictest0-21swapper/211:40:322
22527998463,10cyclictest0-21swapper/211:40:322
22517998463,12cyclictest0-21swapper/110:09:561
22527998365,12cyclictest23734-21fschecks_count08:50:332
22527998365,12cyclictest23734-21fschecks_count08:50:332
22517998363,11cyclictest0-21swapper/107:14:561
22517998362,11cyclictest0-21swapper/111:20:201
22517998362,10cyclictest0-21swapper/112:15:301
22517998362,10cyclictest0-21swapper/112:15:301
22513998339,12cyclictest181rcu_preempt12:33:180
22527998263,12cyclictest60-21kcompactd008:05:462
22527998263,12cyclictest60-21kcompactd008:05:462
22527998253,10cyclictest181rcu_preempt07:48:312
22527998252,7cyclictest181rcu_preempt12:27:532
22527998246,9cyclictest181rcu_preempt10:36:322
22527998246,9cyclictest181rcu_preempt10:36:322
22517998267,8cyclictest0-21swapper/112:00:261
22517998267,8cyclictest0-21swapper/112:00:261
22517998266,9cyclictest0-21swapper/108:35:351
22517998261,12cyclictest0-21swapper/111:39:561
22513998266,10cyclictest18449-21/usr/sbin/munin11:45:290
22527998163,9cyclictest22244-21kworker/u18:0-rpciod10:04:572
22527998159,13cyclictest0-21swapper/209:20:282
22527998159,13cyclictest0-21swapper/209:20:282
22527998154,15cyclictest181rcu_preempt11:03:092
22527998149,7cyclictest181rcu_preempt09:00:322
22517998163,11cyclictest0-21swapper/111:25:131
22513998152,23cyclictest9943-21latency_hist11:29:550
22513998128,26cyclictest24782-21seq09:59:550
22527998060,11cyclictest0-21swapper/211:05:142
22527998060,11cyclictest0-21swapper/211:05:142
22527998059,11cyclictest0-21swapper/210:40:282
22527998053,7cyclictest181rcu_preempt11:59:312
22527998049,8cyclictest181rcu_preempt10:47:422
22527998047,7cyclictest181rcu_preempt07:20:312
22517998061,9cyclictest0-21swapper/108:20:181
22513998060,7cyclictest181rcu_preempt11:54:540
22513998048,12cyclictest181rcu_preempt10:20:300
2251399803,45cyclictest15-21ksoftirqd/007:40:340
22527997959,9cyclictest0-21swapper/210:32:352
22527997959,11cyclictest0-21swapper/211:30:152
22527997958,14cyclictest23855-21apt-key11:54:542
22527997958,11cyclictest0-21swapper/212:30:142
22527997949,7cyclictest181rcu_preempt10:22:442
22527997948,8cyclictest181rcu_preempt12:22:302
22527997947,7cyclictest181rcu_preempt09:09:552
22517997963,9cyclictest0-21swapper/111:50:261
22517997958,10cyclictest0-21swapper/111:00:281
22513997964,10cyclictest19184-21kworker/1:109:54:560
22513997956,11cyclictest181rcu_preempt10:40:120
22513997938,7cyclictest181rcu_preempt12:20:440
22527997856,15cyclictest22499-21cyclictest09:59:542
22527997853,7cyclictest181rcu_preempt07:58:272
22527997850,9cyclictest181rcu_preempt07:43:122
22527997849,14cyclictest181rcu_preempt12:15:062
22527997849,14cyclictest181rcu_preempt12:15:062
22517997866,7cyclictest0-21swapper/110:45:171
2251399789,37cyclictest15-21ksoftirqd/009:39:490
22527997761,9cyclictest0-21swapper/210:15:172
22527997754,8cyclictest181rcu_preempt10:09:222
22527997754,7cyclictest181rcu_preempt09:13:032
22527997754,7cyclictest181rcu_preempt09:13:032
22527997752,7cyclictest181rcu_preempt12:03:392
22527997752,7cyclictest181rcu_preempt12:03:392
22527997750,8cyclictest181rcu_preempt09:41:112
22527997745,9cyclictest181rcu_preempt12:08:352
22517997762,8cyclictest0-21swapper/109:39:541
22513997756,7cyclictest181rcu_preempt11:13:370
22513997753,7cyclictest181rcu_preempt09:26:510
22513997751,8cyclictest181rcu_preempt11:21:120
22513997748,7cyclictest181rcu_preempt08:25:230
22513997741,7cyclictest181rcu_preempt11:30:220
22513997739,20cyclictest18324-21sed11:44:560
22513997739,20cyclictest18324-21sed11:44:560
2252799765,29cyclictest39-21ksoftirqd/211:20:022
22527997641,8cyclictest181rcu_preempt10:52:122
22517997660,9cyclictest0-21swapper/111:45:191
2251399769,39cyclictest15-21ksoftirqd/012:11:410
22513997655,8cyclictest181rcu_preempt08:33:340
22513997653,8cyclictest181rcu_preempt10:35:250
22513997653,8cyclictest181rcu_preempt10:35:250
22513997651,8cyclictest181rcu_preempt07:30:300
22513997647,21cyclictest12397-21netstat08:15:230
22513997643,14cyclictest181rcu_preempt07:50:220
22527997549,8cyclictest181rcu_preempt12:39:002
22527997549,8cyclictest181rcu_preempt12:39:002
22527997548,8cyclictest181rcu_preempt07:39:402
22527997544,10cyclictest1634-21ssh09:15:592
22527997539,7cyclictest181rcu_preempt11:19:512
22527997539,7cyclictest181rcu_preempt11:19:512
22527997518,24cyclictest371rcuc/208:55:272
2251399759,39cyclictest15-21ksoftirqd/008:22:150
22513997552,17cyclictest14304-21fschecks_count09:40:150
22527997446,7cyclictest181rcu_preempt09:50:582
22527997442,7cyclictest181rcu_preempt08:30:222
22517997454,11cyclictest0-21swapper/110:35:171
22517997454,11cyclictest0-21swapper/110:35:171
22517997420,30cyclictest31-21ksoftirqd/109:22:251
22517997420,30cyclictest31-21ksoftirqd/109:22:251
22517997416,20cyclictest0-21swapper/107:49:551
22513997450,10cyclictest181rcu_preempt07:21:500
22513997445,23cyclictest28316-21latency_hist09:04:570
22513997443,22cyclictest23131-21apt-key08:49:560
22513997433,32cyclictest0-21swapper/010:00:160
22513997422,32cyclictest0-21swapper/010:48:050
22527997354,7cyclictest181rcu_preempt12:14:452
22527997346,7cyclictest181rcu_preempt07:12:422
22527997345,8cyclictest181rcu_preempt10:25:472
22517997359,10cyclictest22499-21cyclictest11:31:041
22517997358,8cyclictest0-21swapper/109:15:241
22517997358,11cyclictest22499-21cyclictest10:10:381
22517997358,11cyclictest22499-21cyclictest10:10:381
22517997342,22cyclictest23933-21cat08:50:371
22517997342,22cyclictest23933-21cat08:50:371
22513997353,14cyclictest0-21swapper/011:19:540
22513997353,14cyclictest0-21swapper/011:19:540
22513997351,8cyclictest181rcu_preempt07:15:260
22513997348,15cyclictest0-21swapper/012:00:150
22513997348,15cyclictest0-21swapper/012:00:150
22513997346,19cyclictest28483-21netstat07:25:220
22513997346,19cyclictest28483-21netstat07:25:220
22513997345,20cyclictest11818-21latency_hist08:14:560
22513997345,20cyclictest11818-21latency_hist08:14:560
22513997340,10cyclictest181rcu_preempt11:00:150
22527997248,8cyclictest181rcu_preempt08:15:292
22527997248,7cyclictest181rcu_preempt08:26:422
22513997251,15cyclictest8310-21latency_hist08:04:560
22513997250,7cyclictest181rcu_preempt10:30:180
22513997241,10cyclictest181rcu_preempt10:16:270
22513997234,8cyclictest181rcu_preempt09:48:150
22513997234,8cyclictest181rcu_preempt09:48:150
22513997211,29cyclictest15-21ksoftirqd/008:39:040
22513997210,34cyclictest15-21ksoftirqd/008:51:400
22513997210,34cyclictest15-21ksoftirqd/008:51:400
22517997156,9cyclictest14206-21unixbench_multi12:30:331
22517997155,10cyclictest25011-21kworker/u17:3-rpciod11:40:321
22517997155,10cyclictest25011-21kworker/u17:3-rpciod11:40:321
22517997153,8cyclictest0-21swapper/112:14:281
22517997152,14cyclictest1463-21apt-key11:14:551
22517997138,23cyclictest20291-21sort09:50:301
22517997129,18cyclictest31-21ksoftirqd/111:08:091
22517997129,18cyclictest31-21ksoftirqd/111:08:091
22513997155,9cyclictest178-21jbd2/mmcblk2p2-809:30:230
22513997151,13cyclictest3467-21fschecks_count09:20:160
22513997151,13cyclictest3467-21fschecks_count09:20:160
22513997147,10cyclictest181rcu_preempt09:10:560
22513997147,10cyclictest181rcu_preempt09:10:560
22513997133,9cyclictest181rcu_preempt10:09:020
22513997133,11cyclictest181rcu_preempt07:10:480
22513997126,18cyclictest181rcu_preempt08:40:200
22527997047,7cyclictest181rcu_preempt07:54:342
22527997041,7cyclictest181rcu_preempt09:46:252
22527997041,7cyclictest181rcu_preempt09:46:252
22517997056,10cyclictest22499-21cyclictest10:44:551
22517997055,9cyclictest0-21swapper/107:24:571
22517997055,8cyclictest6470-21kworker/u17:2-xprtiod09:27:051
22517997052,8cyclictest0-21swapper/112:20:301
22517997051,8cyclictest0-21swapper/107:34:571
22517997049,14cyclictest623-21sh12:09:561
22517997048,15cyclictest19323-21ssh12:39:571
22513997048,8cyclictest181rcu_preempt09:09:560
2251399702,61cyclictest23583-21kworker/u18:0+ext4-rsv-conversion08:05:130
2251399702,61cyclictest23583-21kworker/u18:0+ext4-rsv-conversion08:05:130
22517996953,11cyclictest383-21/usr/sbin/munin07:40:241
22517996952,8cyclictest0-21swapper/109:58:121
22513996919,19cyclictest211rcuc/012:26:450
22517996852,10cyclictest25072-21ntp_offset11:55:411
22517996851,9cyclictest0-21swapper/112:35:211
22517996850,12cyclictest352-21runrttasks11:15:311
22517996850,12cyclictest352-21runrttasks11:15:311
2251799684,31cyclictest25149-21cat10:00:101
22513996848,14cyclictest29722-21diskmemload10:25:200
22513996834,8cyclictest181rcu_preempt08:55:150
22513996828,8cyclictest0-21swapper/010:50:120
22517996753,10cyclictest22499-21cyclictest08:05:551
22517996753,10cyclictest22499-21cyclictest08:05:551
22517996749,11cyclictest0-21swapper/108:14:531
22517996749,11cyclictest0-21swapper/108:14:531
2251799673,24cyclictest31-21ksoftirqd/108:03:351
22513996728,8cyclictest0-21swapper/011:05:090
22513996728,8cyclictest0-21swapper/011:05:090
2251799662,27cyclictest31-21ksoftirqd/110:58:401
2251799662,24cyclictest31-21ksoftirqd/107:29:291
2251799662,24cyclictest31-21ksoftirqd/107:29:291
2251799661,25cyclictest31-21ksoftirqd/109:32:231
151292660,7sleep30-21swapper/308:24:573
22517996549,10cyclictest31721-21ntp_kernel_pll_07:35:241
22517996535,21cyclictest13258-21cron08:19:531
2251799653,24cyclictest31-21ksoftirqd/110:20:501
22513996540,19cyclictest32081-21timerwakeupswit07:35:320
2251799642,26cyclictest31-21ksoftirqd/110:50:281
2251799640,26cyclictest31-21ksoftirqd/109:45:111
2251799640,26cyclictest31-21ksoftirqd/109:45:111
2251799632,26cyclictest31-21ksoftirqd/109:07:211
2251799631,26cyclictest31-21ksoftirqd/110:31:481
2251799621,24cyclictest31-21ksoftirqd/107:57:561
2251799621,23cyclictest31-21ksoftirqd/108:58:381
2251799611,24cyclictest31-21ksoftirqd/108:29:581
2251799591,22cyclictest31-21ksoftirqd/108:41:581
218422569,24sleep30-21swapper/307:05:423
92162524,11sleep30-21swapper/311:27:523
22535995136,8cyclictest451rcuc/308:43:243
22535994934,9cyclictest451rcuc/307:33:463
22535994934,10cyclictest451rcuc/307:57:573
22535994833,9cyclictest451rcuc/310:37:223
22535994833,9cyclictest451rcuc/310:37:223
2253599473,28cyclictest24278-21taskset11:54:573
22535994633,9cyclictest451rcuc/309:42:083
2253599463,32cyclictest14754-21chrt10:40:063
22535994630,10cyclictest451rcuc/310:33:423
2253599453,31cyclictest17511-21taskset10:45:003
2253599453,28cyclictest29903-21taskset09:09:573
2253599453,24cyclictest30201-21taskset12:04:593
22535994530,11cyclictest451rcuc/309:55:013
2253599452,30cyclictest18702-21chrt12:38:583
2253599452,30cyclictest18702-21chrt12:38:583
2253599443,30cyclictest20354-21taskset10:50:073
2253599443,30cyclictest17424-21taskset09:45:273
2253599443,30cyclictest17424-21taskset09:45:273
2253599443,27cyclictest30124-21taskset09:10:133
2253599443,27cyclictest30124-21taskset09:10:133
22535994431,9cyclictest451rcuc/309:20:203
22535994431,9cyclictest451rcuc/309:20:203
2253599442,29cyclictest23543-21taskset08:50:213
2253599442,29cyclictest23543-21taskset08:50:213
2253599434,29cyclictest782-21taskset12:10:013
2253599433,31cyclictest19579-21taskset11:45:383
2253599433,30cyclictest7887-21taskset10:26:283
2253599433,30cyclictest31510-21taskset10:10:483
2253599433,30cyclictest31510-21taskset10:10:483
2253599433,30cyclictest30046-21taskset10:09:143
2253599433,30cyclictest2529-21taskset10:17:083
2253599433,29cyclictest23137-21taskset10:55:013
2253599433,28cyclictest27739-21taskset10:04:573
2253599433,27cyclictest21671-21taskset08:45:063
2253599432,31cyclictest12358-21taskset08:15:223
19372430,7sleep30-21swapper/311:15:033
19372430,7sleep30-21swapper/311:15:033
2253599424,28cyclictest28955-21taskset11:05:153
2253599424,28cyclictest28955-21taskset11:05:153
2253599423,30cyclictest27291-21chrt12:00:083
2253599423,30cyclictest27291-21chrt12:00:083
2253599423,30cyclictest18665-21taskset08:35:183
2253599423,29cyclictest855-21taskset07:40:223
2253599423,29cyclictest4594-21taskset07:50:313
2253599423,29cyclictest11612-21taskset11:32:053
2253599423,28cyclictest6442-21taskset09:25:213
2253599423,28cyclictest31804-21taskset07:35:283
2253599423,28cyclictest27810-21taskset11:03:043
2253599423,28cyclictest19784-21taskset09:50:183
2253599423,26cyclictest22705-21taskset07:10:013
2253599422,30cyclictest4618-21taskset12:15:283
2253599422,30cyclictest4618-21taskset12:15:283
2253599422,30cyclictest3758-21chrt10:20:123
253222415,7sleep30-21swapper/307:15:293
2253599413,29cyclictest592-21taskset09:15:163
2253599413,29cyclictest16464-21taskset08:29:313
2253599413,29cyclictest13172-21taskset12:30:073
2253599413,28cyclictest9594-21chrt12:24:323
2253599413,28cyclictest26164-21taskset11:57:333
2253599413,28cyclictest25225-21taskset08:55:113
2253599413,28cyclictest10422-21taskset09:32:553
2253599413,27cyclictest2145-21taskset07:45:123
2253599413,27cyclictest12466-21taskset12:28:573
2253599413,27cyclictest11394-21taskset09:35:103
2253599413,24cyclictest26636-21chrt07:20:183
2253599412,30cyclictest28360-21taskset07:25:203
2253599412,30cyclictest28360-21taskset07:25:203
2253599412,30cyclictest15637-21taskset11:40:113
2253599412,30cyclictest15637-21taskset11:40:113
2253599412,30cyclictest10329-21taskset08:10:113
2253599412,30cyclictest10329-21taskset08:10:113
2253599412,29cyclictest10087-21chrt08:09:553
2253599412,29cyclictest10087-21chrt08:09:553
2253599412,26cyclictest12814-21taskset11:35:003
2253599403,28cyclictest16781-21chrt08:30:083
2253599403,27cyclictest7390-21taskset08:00:233
2253599403,27cyclictest27294-21taskset09:00:253
2253599403,26cyclictest5131-21taskset11:20:183
2253599402,30cyclictest31514-21taskset11:10:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional