You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 04:48
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Nov 21, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1733121038,70sleep20-21swapper/219:05:052
18283999967,16cyclictest0-21swapper/122:50:071
18283999876,13cyclictest10085-21latency_hist00:29:291
18274999874,14cyclictest9324-21cpu20:20:010
18274999874,14cyclictest9324-21cpu20:20:010
18274999777,11cyclictest0-21swapper/019:29:500
18283999638,35cyclictest0-21swapper/122:43:221
18283999638,35cyclictest0-21swapper/122:43:221
1728029611,59sleep017313-21ntp_kernel_pll_19:05:030
1828399953,60cyclictest15980-21apt-get20:39:421
18274999572,13cyclictest0-21swapper/023:14:490
18274999572,13cyclictest0-21swapper/023:14:490
18274999572,12cyclictest0-21swapper/019:49:310
18283999444,16cyclictest181rcu_preempt20:05:281
18274999473,12cyclictest0-21swapper/020:04:490
18274999472,13cyclictest0-21swapper/022:20:040
18283999254,16cyclictest181rcu_preempt19:14:541
18283999254,16cyclictest181rcu_preempt19:14:541
18283999049,13cyclictest181rcu_preempt19:34:541
18283999038,10cyclictest181rcu_preempt23:54:271
18283999038,10cyclictest181rcu_preempt23:54:271
18274999076,8cyclictest0-21swapper/023:10:040
18274999073,7cyclictest0-21swapper/019:25:010
18283998954,8cyclictest181rcu_preempt21:13:011
18283998935,11cyclictest8662-21fschecks_count21:35:491
18274998952,8cyclictest181rcu_preempt22:18:200
18274998952,8cyclictest181rcu_preempt22:18:200
18274998875,7cyclictest0-21swapper/000:14:290
18283998738,28cyclictest20200-21timerandwakeup20:50:031
18283998730,32cyclictest0-21swapper/120:14:551
18274998620,36cyclictest15-21ksoftirqd/000:30:460
1685528610,36sleep10-21swapper/119:04:481
18283998561,17cyclictest26599-21latency_hist23:04:301
18283998561,16cyclictest0-21swapper/100:09:291
18283998561,16cyclictest0-21swapper/100:09:291
18283998540,16cyclictest1188-21fschecks_time19:54:501
18283998533,23cyclictest18260-21cyclictest20:45:051
18274998572,7cyclictest0-21swapper/019:09:510
18274998536,8cyclictest181rcu_preempt00:21:040
18283998459,10cyclictest181rcu_preempt22:49:301
18283998459,10cyclictest181rcu_preempt22:49:301
18283998432,26cyclictest0-21swapper/119:19:481
18274998454,8cyclictest181rcu_preempt23:26:340
18283998355,13cyclictest181rcu_preempt23:24:551
18283998353,9cyclictest181rcu_preempt00:17:361
18283998353,13cyclictest181rcu_preempt00:22:191
18283998349,8cyclictest181rcu_preempt21:24:301
18283998349,8cyclictest181rcu_preempt20:34:531
18283998349,10cyclictest181rcu_preempt23:30:071
18283998349,10cyclictest181rcu_preempt23:30:071
18283998348,17cyclictest181rcu_preempt00:10:031
18283998348,13cyclictest181rcu_preempt00:30:041
18274998363,10cyclictest0-21swapper/023:45:030
18283998238,11cyclictest1599-21ssh22:20:001
18283998232,24cyclictest0-21swapper/121:55:161
18283998232,24cyclictest0-21swapper/121:55:161
18283998228,8cyclictest0-21swapper/119:10:041
18274998244,32cyclictest7559-21latency_hist21:34:300
18274998244,32cyclictest7559-21latency_hist21:34:300
18274998239,13cyclictest207-21systemd-journal21:18:190
18274998237,10cyclictest181rcu_preempt22:50:170
18283998158,8cyclictest181rcu_preempt22:31:161
18283998158,8cyclictest181rcu_preempt22:31:161
18283998143,8cyclictest181rcu_preempt22:01:301
18283998142,10cyclictest181rcu_preempt19:34:301
18283998126,11cyclictest0-21swapper/123:15:001
18283998126,11cyclictest0-21swapper/123:15:001
18283998122,25cyclictest0-21swapper/120:59:391
18283998122,25cyclictest0-21swapper/120:59:391
18274998161,14cyclictest691-21cron20:44:270
18274998156,19cyclictest17165-21ssh21:50:000
18291998064,9cyclictest0-21swapper/200:14:262
18291998062,12cyclictest12947-21mailstats22:40:012
18291998062,12cyclictest12947-21mailstats22:40:012
18291998057,14cyclictest25616-21sh23:02:072
18283998064,8cyclictest181rcu_preempt00:03:471
18283998055,10cyclictest181rcu_preempt19:44:381
18283998048,10cyclictest181rcu_preempt21:45:001
18283998032,9cyclictest0-21swapper/119:59:501
18283998030,11cyclictest0-21swapper/119:39:511
18274998058,14cyclictest22061-21latency_hist19:19:300
18274998058,14cyclictest22061-21latency_hist19:19:300
18274998051,20cyclictest11710-21sh23:34:490
18274998046,28cyclictest25293-21ssh22:04:500
18291997959,14cyclictest31444-21ssh00:09:292
18291997959,14cyclictest31444-21ssh00:09:292
18283997960,11cyclictest10593-21sh22:35:231
18283997947,12cyclictest181rcu_preempt22:54:401
18283997947,12cyclictest181rcu_preempt22:54:401
18283997946,12cyclictest181rcu_preempt23:10:021
18283997933,22cyclictest0-21swapper/123:43:471
18283997931,11cyclictest0-21swapper/123:38:171
18283997930,14cyclictest0-21swapper/122:18:131
18283997930,14cyclictest0-21swapper/122:18:131
18283997927,25cyclictest0-21swapper/123:54:541
18283997927,25cyclictest0-21swapper/123:54:541
18283997926,31cyclictest0-21swapper/121:34:291
18283997926,31cyclictest0-21swapper/121:34:291
18291997863,10cyclictest18260-21cyclictest22:11:562
18291997859,13cyclictest25647-21diskmemload23:19:522
1829199785,37cyclictest39-21ksoftirqd/220:45:042
18283997844,12cyclictest181rcu_preempt20:29:551
18283997838,19cyclictest181rcu_preempt22:06:221
18283997827,13cyclictest0-21swapper/122:26:231
18283997822,23cyclictest0-21swapper/120:55:101
18274997864,8cyclictest0-21swapper/023:04:530
18274997861,10cyclictest0-21swapper/023:04:290
18274997858,14cyclictest14641-21timerandwakeup21:45:040
18274997849,8cyclictest181rcu_preempt22:42:000
18274997849,8cyclictest181rcu_preempt22:42:000
18274997843,9cyclictest181rcu_preempt20:31:320
18274997836,20cyclictest181rcu_preempt21:10:090
18291997755,13cyclictest8959-21fschecks_time23:29:492
18291997755,13cyclictest8959-21fschecks_time23:29:492
1829199775,28cyclictest39-21ksoftirqd/221:42:492
18283997753,8cyclictest181rcu_preempt23:46:341
18283997746,7cyclictest181rcu_preempt20:25:021
18283997742,12cyclictest181rcu_preempt21:44:291
18283997734,10cyclictest0-21swapper/100:34:461
18283997734,10cyclictest0-21swapper/100:34:461
18283997730,11cyclictest0-21swapper/119:25:011
1827499779,35cyclictest15-21ksoftirqd/022:39:140
1827499779,35cyclictest15-21ksoftirqd/020:58:060
1827499778,34cyclictest15-21ksoftirqd/020:48:380
18274997763,8cyclictest0-21swapper/021:35:420
1827499775,35cyclictest15-21ksoftirqd/020:50:530
18274997752,17cyclictest1900-21ssh21:24:170
18274997713,27cyclictest15-21ksoftirqd/021:08:220
18291997654,15cyclictest16733-21sh21:49:502
18291997646,21cyclictest15020-21grep20:35:042
18291997612,35cyclictest39-21ksoftirqd/222:50:092
18283997653,8cyclictest181rcu_preempt20:20:201
18283997653,8cyclictest181rcu_preempt20:20:201
18283997644,14cyclictest181rcu_preempt19:49:481
18283997644,14cyclictest181rcu_preempt19:49:481
18283997637,8cyclictest181rcu_preempt22:10:281
18283997628,10cyclictest0-21swapper/121:09:301
18283997626,8cyclictest0-21swapper/123:24:261
1827499765,28cyclictest15-21ksoftirqd/023:43:440
18274997644,8cyclictest181rcu_preempt22:09:420
18274997612,33cyclictest15-21ksoftirqd/020:02:360
1829199757,35cyclictest39-21ksoftirqd/220:23:102
1829199757,35cyclictest39-21ksoftirqd/220:23:102
18291997559,11cyclictest18260-21cyclictest23:19:272
18291997559,11cyclictest18260-21cyclictest23:19:272
18291997554,15cyclictest314-21in:imuxsock00:28:302
18283997543,9cyclictest181rcu_preempt21:14:581
18283997526,8cyclictest0-21swapper/120:09:481
18283997522,26cyclictest0-21swapper/121:25:021
18283997522,26cyclictest0-21swapper/121:25:021
18274997548,7cyclictest181rcu_preempt22:01:210
18274997542,13cyclictest181rcu_preempt23:57:390
18274997542,13cyclictest181rcu_preempt23:57:390
1827499752,40cyclictest15-21ksoftirqd/020:59:410
1827499752,40cyclictest15-21ksoftirqd/020:59:410
18274997518,21cyclictest15-21ksoftirqd/019:38:310
1829199744,25cyclictest39-21ksoftirqd/219:09:592
1829199743,26cyclictest39-21ksoftirqd/219:19:452
1829199742,28cyclictest39-21ksoftirqd/219:15:072
1829199742,28cyclictest39-21ksoftirqd/219:15:072
1829199742,27cyclictest39-21ksoftirqd/222:04:522
18283997444,22cyclictest26953-21fschecks_count23:04:511
1827499746,36cyclictest15-21ksoftirqd/022:26:480
18274997448,7cyclictest181rcu_preempt00:08:170
18274997448,7cyclictest181rcu_preempt00:08:170
18274997442,18cyclictest181rcu_preempt00:25:190
18274997434,8cyclictest181rcu_preempt19:49:560
18274997434,8cyclictest181rcu_preempt19:49:560
18274997433,10cyclictest181rcu_preempt22:56:460
18274997433,10cyclictest181rcu_preempt22:56:460
18274997410,26cyclictest15-21ksoftirqd/022:29:510
18274997410,26cyclictest15-21ksoftirqd/022:29:510
18291997359,10cyclictest18260-21cyclictest00:32:072
1829199733,25cyclictest39-21ksoftirqd/223:59:482
1827499733,27cyclictest15-21ksoftirqd/021:24:340
1827499733,27cyclictest15-21ksoftirqd/021:24:340
1827499732,38cyclictest15-21ksoftirqd/023:54:220
1827499732,38cyclictest15-21ksoftirqd/023:54:220
18274997316,21cyclictest15-21ksoftirqd/020:28:040
18291997257,10cyclictest14674-21fschecks_time23:39:502
18291997250,15cyclictest27809-21unixbench_singl23:05:082
1829199724,28cyclictest39-21ksoftirqd/221:59:412
18291997242,21cyclictest20364-21cat23:49:452
18291997242,21cyclictest20364-21cat23:49:452
1829199723,28cyclictest39-21ksoftirqd/221:06:282
1829199723,26cyclictest39-21ksoftirqd/223:36:232
1829199722,30cyclictest39-21ksoftirqd/222:36:332
1829199722,27cyclictest39-21ksoftirqd/222:29:572
1829199722,27cyclictest39-21ksoftirqd/222:29:572
18291997213,29cyclictest39-21ksoftirqd/223:44:492
18274997238,10cyclictest181rcu_preempt20:14:570
1827499722,32cyclictest15-21ksoftirqd/019:19:550
1827499722,28cyclictest15-21ksoftirqd/023:59:480
18291997151,14cyclictest27999-21kworker/u17:1-rpciod23:12:052
18291997117,9cyclictest0-21swapper/221:14:402
18283997126,9cyclictest0-21swapper/121:50:011
18274997148,17cyclictest2035-21/usr/sbin/munin00:14:470
1827499714,38cyclictest15-21ksoftirqd/019:55:110
1827499712,37cyclictest15-21ksoftirqd/023:30:530
1827499712,37cyclictest15-21ksoftirqd/023:30:530
1829899702,40cyclictest0-21swapper/300:19:573
1829199705,24cyclictest39-21ksoftirqd/222:58:032
1829199705,24cyclictest39-21ksoftirqd/222:58:032
18291997052,13cyclictest18260-21cyclictest21:30:052
18291997052,13cyclictest18260-21cyclictest21:30:052
1829199704,31cyclictest39-21ksoftirqd/221:13:182
1829199702,27cyclictest31529-21awk19:49:272
1829199701,35cyclictest39-21ksoftirqd/220:49:522
18291997013,30cyclictest39-21ksoftirqd/200:21:182
1829199701,29cyclictest39-21ksoftirqd/222:45:582
1829199701,29cyclictest39-21ksoftirqd/222:45:582
18274997054,10cyclictest18260-21cyclictest21:39:550
18274997053,10cyclictest0-21swapper/020:09:520
18274997046,8cyclictest181rcu_preempt23:19:370
18274997046,7cyclictest181rcu_preempt19:42:150
18274997027,8cyclictest181rcu_preempt22:45:520
18274997027,8cyclictest181rcu_preempt22:45:520
18274997010,29cyclictest15-21ksoftirqd/020:35:340
1829199697,21cyclictest39-21ksoftirqd/222:15:062
1829199697,21cyclictest39-21ksoftirqd/222:15:062
18291996949,14cyclictest25043-21kworker/u17:3+events_unbound20:19:262
1829199693,32cyclictest39-21ksoftirqd/222:24:522
18274996914,28cyclictest15-21ksoftirqd/021:55:160
18274996914,28cyclictest15-21ksoftirqd/021:55:160
1829199685,33cyclictest39-21ksoftirqd/220:59:472
1829199685,33cyclictest39-21ksoftirqd/220:59:472
18291996850,13cyclictest32563-21timerandwakeup19:50:022
18291996850,13cyclictest32563-21timerandwakeup19:50:022
18291996846,16cyclictest14902-21sh21:45:172
18291996839,19cyclictest8388-21kworker/u19:3+nfsiod23:55:392
18291996839,19cyclictest8388-21kworker/u19:3+nfsiod23:55:392
18291996834,11cyclictest3754-21ssh00:17:032
18274996855,9cyclictest18260-21cyclictest00:36:050
18274996855,9cyclictest18260-21cyclictest00:36:050
18291996752,9cyclictest27999-21kworker/u17:1-rpciod23:24:532
18291996747,14cyclictest21637-21processes20:55:062
18291996747,14cyclictest13210-21/usr/sbin/munin00:35:022
18291996747,14cyclictest13210-21/usr/sbin/munin00:35:022
1829199672,27cyclictest39-21ksoftirqd/220:40:162
1829199671,27cyclictest39-21ksoftirqd/219:59:542
18291996652,10cyclictest18260-21cyclictest20:12:292
18291996635,11cyclictest1431-21ssh21:22:542
1829199662,25cyclictest39-21ksoftirqd/219:34:582
18291996610,17cyclictest25915-21ps19:30:002
18291996512,13cyclictest4759-21ssh21:29:272
18291996512,13cyclictest4759-21ssh21:29:272
18291996429,11cyclictest181rcu_preempt21:38:202
1829199642,26cyclictest39-21ksoftirqd/222:19:532
1829199642,23cyclictest39-21ksoftirqd/221:59:002
1829199642,23cyclictest39-21ksoftirqd/221:59:002
1829199633,28cyclictest39-21ksoftirqd/219:54:572
18291996332,21cyclictest12750-21tune2fs20:29:552
1829199625,26cyclictest39-21ksoftirqd/220:24:512
1829199613,26cyclictest39-21ksoftirqd/219:24:402
18291996113,14cyclictest5537-21sh20:09:272
18291995936,9cyclictest181rcu_preempt19:40:202
1829899553,36cyclictest19358-21taskset20:49:303
1694025412,26sleep30-21swapper/319:04:503
231382520,6sleep30-21swapper/320:59:553
231382520,6sleep30-21swapper/320:59:553
1829899523,29cyclictest182642sleep322:50:073
1829899522,30cyclictest6339-21chrt21:30:393
1829899522,30cyclictest6339-21chrt21:30:393
1829899513,33cyclictest10608-21chrt21:39:263
18298995120,23cyclictest18621-21chrt19:09:503
1829899503,35cyclictest23088-21chrt22:58:143
1829899503,35cyclictest23088-21chrt22:58:143
1829899493,33cyclictest23377-21taskset22:00:263
1829899492,32cyclictest11355-21chrt23:34:303
1829899492,32cyclictest11355-21chrt23:34:303
1829899484,33cyclictest5562-21chrt23:24:283
18298994833,9cyclictest91102sleep322:34:273
18298994833,9cyclictest91102sleep322:34:273
1829899483,32cyclictest16156-21chrt22:45:053
1829899483,32cyclictest16156-21chrt22:45:053
1829899473,32cyclictest31704-21taskset00:09:403
1829899473,30cyclictest30941-21taskset19:45:023
1829899473,28cyclictest29874-21chrt23:09:493
1829899463,30cyclictest4193-21taskset20:04:383
1829899463,30cyclictest27653-21chrt22:09:303
1829899463,28cyclictest2524-21chrt00:14:553
1829899462,31cyclictest20949-21taskset19:14:563
1829899462,31cyclictest20949-21taskset19:14:563
1829899453,29cyclictest9573-21chrt00:27:563
1829899453,28cyclictest17371-21taskset23:44:433
1829899453,28cyclictest14072-21chrt00:35:053
1829899453,28cyclictest14072-21chrt00:35:053
1829899453,26cyclictest4054-21chrt20:04:303
18298994531,8cyclictest130322chrt20:30:023
1829899443,31cyclictest4027-21taskset22:24:503
1829899443,30cyclictest3674-21chrt22:24:303
1829899443,29cyclictest11288-21chrt21:39:513
1829899443,28cyclictest6023-21chrt20:09:523
1829899443,27cyclictest26807-21chrt23:04:463
1829899443,25cyclictest0-21swapper/320:39:443
1829899442,29cyclictest28047-21taskset22:09:553
1829899442,29cyclictest12248-21chrt00:32:263
1829899442,28cyclictest4850-21taskset21:29:293
1829899442,28cyclictest4850-21taskset21:29:293
18298994421,13cyclictest0-21swapper/320:20:303
18298994421,13cyclictest0-21swapper/320:20:303
1829899433,31cyclictest29359-21taskset19:39:593
1829899433,29cyclictest29455-21taskset00:04:543
1829899433,29cyclictest29455-21taskset00:04:543
1829899433,27cyclictest11161-21chrt20:24:513
1829899433,25cyclictest13929-21chrt22:41:573
1829899433,25cyclictest13929-21chrt22:41:573
18298994330,7cyclictest0-21swapper/323:35:073
1829899432,28cyclictest27896-21taskset19:35:053
1829899423,27cyclictest30571-21taskset22:14:373
1829899423,27cyclictest30571-21taskset22:14:373
1829899423,27cyclictest1448-21taskset23:16:053
1829899423,27cyclictest1448-21taskset23:16:053
1829899423,26cyclictest28381-21taskset21:13:443
1829899423,26cyclictest25493-21taskset23:01:423
1829899422,31cyclictest29777-21taskset21:15:023
1829899422,27cyclictest14347-21taskset21:44:583
1829899422,26cyclictest7682-21chrt23:27:283
1829899422,26cyclictest18159-21chrt21:51:473
1829899413,28cyclictest14494-21chrt23:39:453
1829899413,27cyclictest9370-21taskset22:34:353
1829899413,25cyclictest22208-21chrt20:57:203
1829899412,29cyclictest28371-21taskset00:03:443
1829899412,29cyclictest23182-21chrt23:54:403
1829899412,29cyclictest23182-21chrt23:54:403
1829899403,28cyclictest8411-21taskset20:15:193
1829899403,28cyclictest19586-21chrt20:49:463
1829899403,28cyclictest15299-21taskset20:37:433
1829899403,27cyclictest25622-21taskset19:29:513
1829899403,27cyclictest22698-21chrt23:54:043
1829899403,27cyclictest22698-21chrt23:54:043
1829899403,25cyclictest31833-21chrt21:19:323
1829899393,27cyclictest31948-21taskset19:49:483
1829899393,27cyclictest31948-21taskset19:49:483
1829899393,27cyclictest22135-21chrt19:19:423
1829899393,26cyclictest24417-21taskset21:04:453
1829899393,26cyclictest23893-21taskset19:24:493
1829899393,23cyclictest19591-21taskset21:54:513
1829899393,23cyclictest19591-21taskset21:54:513
1829899383,26cyclictest1943-21taskset19:55:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional