You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 09:02
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackfslot4.osadl.org (updated Mon Nov 24, 2025 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
136722780,1sleep128494-21kworker/u8:619:05:281
223522750,2sleep328490-21kworker/u8:219:10:003
145122580,8sleep225-21ksoftirqd/219:05:322
226398159152,4cyclictest1050ksoftirqd/021:11:540
226498145140,3cyclictest14568-21kworker/u8:3021:11:531
920721390,2sleep10-21swapper/119:25:391
226398129118,8cyclictest10-21ksoftirqd/020:26:490
1558421250,2sleep10-21swapper/119:45:011
1558421250,2sleep10-21swapper/119:45:011
226498112106,4cyclictest24458-21kworker/u8:020:26:491
22639810498,3cyclictest1050ksoftirqd/022:57:210
142192880,8sleep225-21ksoftirqd/222:15:122
315162850,2sleep10-21swapper/122:50:181
282432850,2chrt0-21swapper/323:50:263
2263988277,3cyclictest1050ksoftirqd/020:42:110
2263988170,8cyclictest10-21ksoftirqd/022:42:000
2269987569,4cyclictest1481-21kworker/u8:2621:11:533
2264987465,5cyclictest1490-21kworker/u8:3420:42:111
2263986758,6cyclictest10-21ksoftirqd/023:27:030
2263986655,8cyclictest10-21ksoftirqd/019:17:120
2264986458,3cyclictest18342-21kworker/u8:022:41:591
2263986247,12cyclictest10-21ksoftirqd/020:37:040
2266986052,5cyclictest12693-21kworker/u8:621:11:532
2264985245,5cyclictest12684-21kworker/u8:2723:27:021
2264984943,3cyclictest1944-21kworker/u8:2219:17:121
136082460,5sleep313605-21munin-run19:40:003
284582450,2sleep20-21swapper/223:50:332
95022440,18sleep224860-1kworker/u9:200:18:002
13782430,14sleep10-21swapper/120:31:091
2264984135,3cyclictest1515-21kworker/u8:5620:37:041
2263984130,8cyclictest10-21ksoftirqd/000:31:460
149402410,4sleep119-21ksoftirqd/121:10:011
282482400,4chrt28204-21apt-get23:50:271
2269984035,3cyclictest28172-21kworker/u8:420:26:493
129322360,11sleep225-21ksoftirqd/221:04:432
2263983425,6cyclictest10-21ksoftirqd/021:27:140
226498332,5cyclictest121rcu_preempt23:35:121
2263983328,2cyclictest1050ksoftirqd/023:02:290
150942320,19sleep115092-21ntpq22:15:321
72512310,2sleep30-21swapper/322:00:223
324642310,2sleep20-21swapper/220:30:012
226498311,3cyclictest121rcu_preempt23:00:121
194942310,9sleep30-21swapper/300:39:403
168212310,2sleep20-21swapper/219:45:422
292752300,8sleep32258-21cyclictest21:38:543
255832300,13sleep3121rcu_preempt23:45:103
204432300,2sleep30-21swapper/321:20:223
17302300,6sleep119-21ksoftirqd/121:48:531
114132300,2sleep30-21swapper/323:15:193
10922300,7sleep331-21ksoftirqd/320:30:433
95502290,7sleep331-21ksoftirqd/300:18:173
291212290,3sleep129119-21cut20:20:271
291212290,3sleep129119-21cut20:20:271
241102290,1sleep221790-21kworker/u8:1021:26:512
2264982910,3cyclictest19-21ksoftirqd/121:40:161
2263982916,11cyclictest10-21ksoftirqd/022:17:250
126472290,16sleep11983-21bootmenu22:10:301
126472290,16sleep11983-21bootmenu22:10:291
66592280,1sleep129689-21kworker/u8:600:10:471
244092280,3sleep11983-21bootmenu21:28:231
226498282,3cyclictest121rcu_preempt20:10:151
226498282,3cyclictest121rcu_preempt00:20:121
2264982819,3cyclictest19-21ksoftirqd/123:30:141
226498281,3cyclictest121rcu_preempt22:45:371
226498281,3cyclictest121rcu_preempt00:35:131
2263982817,8cyclictest10-21ksoftirqd/022:02:040
168112280,1sleep312681-21kworker/u8:2423:25:283
132082280,16sleep225-21ksoftirqd/221:05:112
115772280,9sleep331-21ksoftirqd/300:21:063
79432270,5sleep1191ksoftirqd/123:06:591
78212270,6sleep119-21ksoftirqd/119:22:171
78212270,6sleep119-21ksoftirqd/119:22:171
73452270,16sleep331-21ksoftirqd/320:49:403
302752270,7sleep331-21ksoftirqd/320:23:473
302752270,7sleep331-21ksoftirqd/320:23:473
268542270,16sleep30-21swapper/321:33:443
268542270,16sleep30-21swapper/321:33:433
252442270,3sleep325243-21df20:10:203
24762270,15sleep2489-21kworker/u8:5322:55:422
226698276,9cyclictest121rcu_preempt20:20:272
226698276,9cyclictest121rcu_preempt20:20:272
226498271,3cyclictest121rcu_preempt22:30:161
203012270,8sleep331-21ksoftirqd/323:33:523
202792270,7sleep1191ksoftirqd/119:55:341
133972270,1sleep111638-21kworker/u8:000:25:311
83002260,3sleep38291-21apt-get19:25:173
75342260,3sleep30-21swapper/323:05:583
32582260,3sleep11980-21bootmenu22:58:511
226698263,5cyclictest25-21ksoftirqd/222:05:262
226498264,6cyclictest19-21ksoftirqd/100:30:271
226498261,3cyclictest121rcu_preempt23:10:301
226498261,14cyclictest0-21swapper/123:19:191
122802260,14sleep211638-21kworker/u8:000:24:472
78452250,2sleep30-21swapper/319:22:353
78452250,2sleep30-21swapper/319:22:343
71872250,4sleep11983-21bootmenu20:47:371
39782250,4sleep331-21ksoftirqd/321:53:073
226998255,3cyclictest121rcu_preempt22:30:273
226998255,3cyclictest121rcu_preempt22:05:113
226998255,3cyclictest121rcu_preempt21:00:463
226998254,5cyclictest31-21ksoftirqd/322:40:213
226498254,3cyclictest121rcu_preempt21:35:141
2264982517,3cyclictest19-21ksoftirqd/121:30:161
2264982517,3cyclictest19-21ksoftirqd/121:30:151
2263982518,4cyclictest10-21ksoftirqd/020:15:180
2263982513,9cyclictest10-21ksoftirqd/021:17:000
208832250,2sleep30-21swapper/319:58:143
207602250,2sleep30-21swapper/323:35:043
188342250,2sleep30-21swapper/319:50:503
185962250,5sleep119-21ksoftirqd/121:15:361
16112250,5sleep2251ksoftirqd/220:33:152
134182250,2sleep10-21swapper/119:38:011
109762250,2sleep20-21swapper/220:58:452
73062240,3sleep21491-21kworker/u8:3520:49:112
62812240,2sleep10-21swapper/121:57:351
322552240,5sleep2251ksoftirqd/222:50:362
226998244,5cyclictest31-21ksoftirqd/322:35:303
226998243,5cyclictest31-21ksoftirqd/319:20:013
226498241,3cyclictest121rcu_preempt23:45:191
226498241,3cyclictest121rcu_preempt22:40:011
226498241,3cyclictest121rcu_preempt00:05:211
189592240,2sleep20-21swapper/200:36:492
170022240,2sleep30-21swapper/319:46:083
138772240,2sleep30-21swapper/322:14:413
138772240,2sleep30-21swapper/322:14:413
73882230,2sleep30-21swapper/300:14:313
62672230,1sleep26168-21kworker/u8:321:57:272
6062230,1sleep3490-21kworker/u8:5422:52:573
37492230,2sleep20-21swapper/221:51:482
324692230,2sleep10-21swapper/123:59:011
283882230,12sleep20-21swapper/220:19:082
247012230,11sleep119-21ksoftirqd/120:09:301
227122230,1sleep321795-21kworker/u8:1521:25:193
226998232,4cyclictest31-21ksoftirqd/323:21:213
226998232,4cyclictest31-21ksoftirqd/323:21:203
226698231,3cyclictest121rcu_preempt21:40:282
226698231,3cyclictest0-21swapper/222:21:552
226698231,3cyclictest0-21swapper/222:21:542
226498234,4cyclictest19-21ksoftirqd/122:08:151
226498233,6cyclictest121rcu_preempt20:00:351
226498233,6cyclictest121rcu_preempt20:00:351
226498233,3cyclictest121rcu_preempt20:50:381
226498232,3cyclictest121rcu_preempt21:20:371
226498232,3cyclictest121rcu_preempt19:55:011
226498231,8cyclictest121rcu_preempt23:20:351
226498231,8cyclictest121rcu_preempt23:20:351
226498231,5cyclictest0-21swapper/122:24:221
226498231,5cyclictest0-21swapper/122:24:211
226498231,2cyclictest121rcu_preempt21:00:181
226498231,2cyclictest121rcu_preempt00:15:371
2263982317,4cyclictest118550irq/29-4f00000.19:50:160
21022230,2sleep30-21swapper/300:01:523
21022230,2sleep30-21swapper/300:01:523
193552230,11sleep214569-1kworker/u9:321:17:072
94942220,2sleep19486-21apt-get20:55:131
55312220,3sleep25530-21ssh23:03:012
52082220,2sleep30-21swapper/320:41:123
33282220,2sleep30-21swapper/320:36:223
283692220,2sleep30-21swapper/320:18:523
264302220,2sleep20-21swapper/220:13:002
226998224,3cyclictest121rcu_preempt21:15:383
2269982216,4cyclictest32738-21kworker/u8:2122:57:213
2269982213,6cyclictest4144-21xargs00:05:463
226998221,2cyclictest0-21swapper/321:40:303
226698226,3cyclictest121rcu_preempt19:30:002
226698223,3cyclictest121rcu_preempt23:25:242
226698223,3cyclictest121rcu_preempt19:10:192
226698222,2cyclictest0-21swapper/222:41:092
226498222,7cyclictest121rcu_preempt20:15:191
226498222,3cyclictest121rcu_preempt19:45:351
226498221,8cyclictest121rcu_preempt19:30:391
226498221,3cyclictest121rcu_preempt22:25:441
226498221,3cyclictest121rcu_preempt22:00:161
226498221,3cyclictest121rcu_preempt19:10:251
226398222,5cyclictest6613-21ntpq20:45:330
2263982216,3cyclictest118550irq/29-4f00000.19:55:150
22252220,12sleep119-21ksoftirqd/121:50:101
188482220,11sleep215215-21kworker/u8:1619:51:012
106112220,12sleep331-21ksoftirqd/323:13:383
80632210,1sleep27884-21kworker/u8:1123:07:412
54042210,2sleep20-21swapper/220:43:372
321452210,11sleep225-21ksoftirqd/223:57:172
268962210,11sleep225-21ksoftirqd/221:33:592
268962210,11sleep225-21ksoftirqd/221:33:592
226998215,4cyclictest121rcu_preempt22:25:223
226998214,3cyclictest121rcu_preempt22:20:333
226998214,3cyclictest121rcu_preempt22:20:323
226998213,3cyclictest121rcu_preempt20:55:223
226998211,6cyclictest2258-21cyclictest19:10:313
226698211,7cyclictest26443-21ntp_kernel_pll_23:45:292
226698211,3cyclictest121rcu_preempt22:13:192
226698211,3cyclictest121rcu_preempt22:13:182
226698211,10cyclictest0-21swapper/223:22:332
226698211,10cyclictest0-21swapper/223:22:322
226498211,3cyclictest121rcu_preempt23:40:501
226498211,3cyclictest121rcu_preempt00:00:301
226498211,3cyclictest121rcu_preempt00:00:301
2263982116,3cyclictest118550irq/29-4f00000.23:50:110
2263982115,4cyclictest118550irq/29-4f00000.22:25:190
2263982112,7cyclictest10-21ksoftirqd/021:20:310
2263982111,8cyclictest10-21ksoftirqd/022:20:460
2263982111,8cyclictest10-21ksoftirqd/022:20:450
152732210,11sleep331-21ksoftirqd/319:42:343
152732210,11sleep331-21ksoftirqd/319:42:343
145112210,10sleep331-21ksoftirqd/322:15:223
29292202,4sleep2251ksoftirqd/220:35:412
226998201,3cyclictest121rcu_preempt23:05:023
2269982013,4cyclictest1938-21runrttasks21:55:173
2269982011,6cyclictest30352-21sshd22:46:203
226698203,8cyclictest296882kworker/u8:500:08:432
226698203,4cyclictest121rcu_preempt22:30:112
226698201,5cyclictest19375-21aten2.4-expect22:25:232
226698201,5cyclictest121rcu_preempt00:02:562
226698201,5cyclictest121rcu_preempt00:02:562
226698201,3cyclictest121rcu_preempt00:10:462
226698201,3cyclictest0-21swapper/221:36:362
2266982013,4cyclictest24948-21latency_hist20:10:012
2263982016,2cyclictest1050ksoftirqd/021:37:290
2263982015,3cyclictest10-21ksoftirqd/023:40:170
2263982015,3cyclictest10-21ksoftirqd/021:50:350
2263982015,3cyclictest10-21ksoftirqd/000:20:150
2263982014,4cyclictest10-21ksoftirqd/022:30:160
2263982014,4cyclictest10-21ksoftirqd/021:45:370
2263982014,4cyclictest10-21ksoftirqd/021:30:150
2263982014,4cyclictest10-21ksoftirqd/021:30:140
2263982013,5cyclictest10-21ksoftirqd/000:05:200
2263982012,6cyclictest9441-21sh22:05:120
2263982011,7cyclictest10-21ksoftirqd/000:17:130
209082200,10sleep225-21ksoftirqd/219:58:332
18832200,10sleep331-21ksoftirqd/321:49:453
139862200,10sleep331-21ksoftirqd/300:26:093
132042200,10sleep331-21ksoftirqd/321:05:083
129142200,10sleep225-21ksoftirqd/223:18:022
11032200,10sleep225-21ksoftirqd/221:45:572
90972190,9sleep331-21ksoftirqd/320:53:193
45442190,9sleep225-21ksoftirqd/219:15:092
249542190,9sleep225-21ksoftirqd/223:43:032
227412190,9sleep331-21ksoftirqd/320:03:043
227412190,9sleep331-21ksoftirqd/320:03:043
226998196,3cyclictest31-21ksoftirqd/323:44:373
2269981912,4cyclictest15731-21load00:30:263
226998191,10cyclictest0-21swapper/323:55:153
226698191,4cyclictest0-21swapper/222:00:232
226698191,2cyclictest0-21swapper/221:20:292
226698190,4cyclictest121rcu_preempt23:12:042
226398193,8cyclictest0-21swapper/019:26:330
2263981915,2cyclictest10-21ksoftirqd/019:30:150
2263981914,3cyclictest28754-21/usr/sbin/munin20:20:190
2263981914,3cyclictest28754-21/usr/sbin/munin20:20:190
2263981914,3cyclictest118550irq/29-4f00000.23:10:190
2263981914,3cyclictest118550irq/29-4f00000.21:05:150
2263981914,3cyclictest118550irq/29-4f00000.20:10:160
2263981914,3cyclictest111rcuc/000:10:310
2263981914,3cyclictest10-21ksoftirqd/023:15:150
2263981914,3cyclictest10-21ksoftirqd/019:35:170
2263981914,3cyclictest10-21ksoftirqd/000:00:160
2263981914,3cyclictest10-21ksoftirqd/000:00:160
2263981913,4cyclictest118550irq/29-4f00000.23:30:160
2263981913,4cyclictest118550irq/29-4f00000.20:55:140
2263981913,4cyclictest10-21ksoftirqd/023:55:400
2263981913,4cyclictest10-21ksoftirqd/019:40:420
2263981913,4cyclictest10-21ksoftirqd/019:40:420
2263981913,3cyclictest10-21ksoftirqd/023:10:010
2263981912,5cyclictest10-21ksoftirqd/023:20:380
2263981912,5cyclictest10-21ksoftirqd/023:20:380
2263981912,4cyclictest10-21ksoftirqd/022:45:240
2263981912,4cyclictest10-21ksoftirqd/000:35:160
2263981911,6cyclictest10-21ksoftirqd/021:04:590
226322190,9sleep225-21ksoftirqd/220:01:422
226322190,9sleep225-21ksoftirqd/220:01:422
79172180,8sleep225-21ksoftirqd/219:23:312
79172180,8sleep225-21ksoftirqd/219:23:302
246142180,8sleep32258-21cyclictest20:08:233
226998181,3cyclictest0-21swapper/319:30:193
226698189,6cyclictest24647-21sed22:35:252
226698188,7cyclictest14299-21sshd00:27:442
226698181,2cyclictest121rcu_preempt22:47:002
2266981812,4cyclictest1938-21runrttasks19:35:312
2266981811,4cyclictest8468-21ntp_offset20:50:312
226398189,7cyclictest10-21ksoftirqd/023:45:140
2263981813,3cyclictest10-21ksoftirqd/023:35:110
2263981813,3cyclictest10-21ksoftirqd/022:50:160
2263981813,3cyclictest10-21ksoftirqd/022:35:180
2263981813,3cyclictest10-21ksoftirqd/021:55:170
2263981813,3cyclictest10-21ksoftirqd/020:05:340
2263981813,3cyclictest10-21ksoftirqd/020:00:310
2263981813,3cyclictest10-21ksoftirqd/020:00:310
2263981813,3cyclictest10-21ksoftirqd/019:45:160
2263981813,3cyclictest10-21ksoftirqd/019:10:400
2263981812,4cyclictest111rcuc/022:10:160
2263981812,4cyclictest111rcuc/022:10:150
2263981812,4cyclictest10-21ksoftirqd/020:50:150
2263981812,4cyclictest10-21ksoftirqd/020:30:150
2263981810,6cyclictest10-21ksoftirqd/021:40:140
152202180,8sleep225-21ksoftirqd/219:41:522
152202180,8sleep225-21ksoftirqd/219:41:522
226698171,5cyclictest20841-21sh23:35:162
2266981710,4cyclictest1938-21runrttasks00:33:142
2266981710,4cyclictest18814-21irqrtprio23:30:232
226398171,7cyclictest2271-21latency_hist19:10:010
2263981712,3cyclictest118550irq/29-4f00000.19:20:150
2263981712,3cyclictest118550irq/29-4f00000.19:20:150
2263981711,3cyclictest10-21ksoftirqd/000:25:400
110452170,7sleep225-21ksoftirqd/219:30:392
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional