You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 15:40
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot6.osadl.org (updated Fri Nov 28, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3081121450,5sleep0253199cyclictest00:45:560
1773911040,4ptp4l401ktimersoftd/320:16:353
2531999075,6cyclictest9-21ksoftirqd/000:15:520
262932880,2sleep10-21swapper/122:53:231
2542998575,6cyclictest33-21ksoftirqd/201:15:522
2542997666,5cyclictest33-21ksoftirqd/201:40:572
2542997660,10cyclictest33-21ksoftirqd/220:20:472
2542997260,6cyclictest33-21ksoftirqd/221:15:492
177391710,1ptp4l401ktimersoftd/301:16:393
2531997057,7cyclictest9-21ksoftirqd/023:55:520
324412690,2sleep30-21swapper/321:20:513
324412690,2sleep30-21swapper/321:20:513
2531996857,6cyclictest9-21ksoftirqd/001:10:570
177391680,9ptp4l401ktimersoftd/300:23:553
177391680,1ptp4l401ktimersoftd/301:26:503
101612670,2sleep10-21swapper/101:36:031
2542996654,6cyclictest33-21ksoftirqd/221:45:482
2531996554,5cyclictest9-21ksoftirqd/023:40:510
2531996552,6cyclictest9-21ksoftirqd/023:15:530
2531996552,5cyclictest9-21ksoftirqd/001:35:560
2542996452,6cyclictest33-21ksoftirqd/223:50:512
2531996452,6cyclictest9-21ksoftirqd/023:30:500
177391630,1ptp4l401ktimersoftd/322:42:283
177032630,4sleep017705-21proc_pri00:31:020
2531996251,5cyclictest9-21ksoftirqd/001:00:550
177391620,1ptp4l401ktimersoftd/321:51:063
101802620,3sleep010184-21unixbench_multi23:46:100
2531996149,6cyclictest9-21ksoftirqd/023:20:500
78192600,1sleep00-21swapper/023:09:000
78192600,1sleep00-21swapper/023:09:000
76982600,1sleep10-21swapper/100:20:551
2542996046,8cyclictest33-21ksoftirqd/200:25:522
2531996048,5cyclictest9-21ksoftirqd/022:30:490
2542995949,5cyclictest33-21ksoftirqd/201:45:542
177391590,1ptp4l401ktimersoftd/323:36:163
2531995846,6cyclictest9-21ksoftirqd/000:35:530
177391580,2ptp4l12137-21kworker/3:223:41:293
177391580,1ptp4l401ktimersoftd/321:31:423
123572580,1sleep20-21swapper/223:50:212
2531995444,5cyclictest9-21ksoftirqd/023:25:500
177391540,1ptp4l401ktimersoftd/323:27:513
32752530,2sleep30-21swapper/322:28:123
2531995343,5cyclictest9-21ksoftirqd/021:40:480
2542995240,7cyclictest33-21ksoftirqd/201:05:522
2531995242,5cyclictest9-21ksoftirqd/020:45:470
2542994938,5cyclictest33-21ksoftirqd/222:50:502
2531994837,5cyclictest9-21ksoftirqd/021:35:490
2531994837,5cyclictest9-21ksoftirqd/020:25:460
177391480,2ptp4l32235-21kworker/3:021:04:163
2531994737,4cyclictest9-21ksoftirqd/000:55:540
2531994737,4cyclictest9-21ksoftirqd/000:55:540
2531994519,3cyclictest9-21ksoftirqd/020:40:470
177391450,3ptp4l29250-21kworker/3:122:58:043
177391440,2ptp4l12137-21kworker/3:223:55:133
2542994336,4cyclictest33-21ksoftirqd/222:35:422
2531994332,5cyclictest9-21ksoftirqd/020:35:470
177391430,0ptp4l401ktimersoftd/322:33:233
33424219,12sleep125-21ksoftirqd/120:15:481
2531994217,3cyclictest9-21ksoftirqd/020:20:580
177391420,17ptp4l401ktimersoftd/300:10:563
2531994134,3cyclictest9-21ksoftirqd/000:40:440
2542994034,3cyclictest33-21ksoftirqd/200:40:522
2542994027,11cyclictest33-21ksoftirqd/223:01:032
2542994015,6cyclictest33-21ksoftirqd/200:10:392
2531994033,4cyclictest9-21ksoftirqd/023:35:510
2531994018,14cyclictest9-21ksoftirqd/021:10:580
2531994017,14cyclictest9-21ksoftirqd/000:21:000
2531994016,3cyclictest9-21ksoftirqd/020:30:530
2542993922,6cyclictest33-21ksoftirqd/223:05:562
2542993922,6cyclictest33-21ksoftirqd/223:05:562
2531993930,5cyclictest9-21ksoftirqd/023:10:500
2531993930,5cyclictest9-21ksoftirqd/000:05:520
2531993926,10cyclictest9-21ksoftirqd/000:50:540
2531993926,10cyclictest9-21ksoftirqd/000:50:540
2531993921,4cyclictest9-21ksoftirqd/023:51:080
177391390,1ptp4l401ktimersoftd/323:58:403
177391390,1ptp4l401ktimersoftd/320:40:463
177391390,1ptp4l401ktimersoftd/320:20:583
177391390,1ptp4l401ktimersoftd/300:45:373
177391390,1ptp4l401ktimersoftd/300:37:313
177391390,16ptp4l401ktimersoftd/320:30:563
177391390,0ptp4l401ktimersoftd/323:34:253
2542993825,10cyclictest33-21ksoftirqd/223:30:512
2531993832,3cyclictest9-21ksoftirqd/021:50:360
2531993831,3cyclictest9-21ksoftirqd/023:00:520
2531993816,7cyclictest9-21ksoftirqd/022:28:230
177391380,1ptp4l401ktimersoftd/323:18:043
177391380,1ptp4l401ktimersoftd/323:01:403
177391380,1ptp4l401ktimersoftd/322:48:573
177391380,1ptp4l401ktimersoftd/322:37:243
177391380,1ptp4l401ktimersoftd/322:24:583
177391380,1ptp4l401ktimersoftd/300:29:533
2531993732,3cyclictest9-21ksoftirqd/001:50:420
2531993730,4cyclictest9-21ksoftirqd/001:15:520
177391370,1ptp4l401ktimersoftd/323:50:293
177391370,1ptp4l401ktimersoftd/323:20:513
177391370,1ptp4l401ktimersoftd/322:51:593
177391370,1ptp4l401ktimersoftd/320:55:523
177391370,1ptp4l401ktimersoftd/301:40:333
177391370,1ptp4l401ktimersoftd/301:11:443
177391370,0ptp4l401ktimersoftd/323:14:023
177391370,0ptp4l401ktimersoftd/322:11:083
177391370,0ptp4l401ktimersoftd/322:01:433
177391370,0ptp4l401ktimersoftd/301:04:583
177391370,0ptp4l401ktimersoftd/300:04:343
2542993630,3cyclictest33-21ksoftirqd/200:55:402
2542993630,3cyclictest33-21ksoftirqd/200:55:402
2542993630,3cyclictest33-21ksoftirqd/200:40:402
2531993630,3cyclictest9-21ksoftirqd/001:10:400
2531993623,10cyclictest9-21ksoftirqd/001:26:020
253199361,4cyclictest10006-21diskstats22:35:560
226123618,8sleep00-21swapper/020:18:220
177391360,1ptp4l401ktimersoftd/323:05:473
177391360,1ptp4l401ktimersoftd/323:05:473
177391360,1ptp4l401ktimersoftd/300:47:143
177391360,1ptp4l401ktimersoftd/300:32:143
177391360,0ptp4l401ktimersoftd/300:58:143
177391360,0ptp4l401ktimersoftd/300:58:143
177391360,0ptp4l401ktimersoftd/300:52:503
177391360,0ptp4l401ktimersoftd/300:52:503
177391360,0ptp4l401ktimersoftd/300:18:133
177391360,0ptp4l401ktimersoftd/300:07:123
2542993530,3cyclictest33-21ksoftirqd/200:15:392
2542993529,3cyclictest33-21ksoftirqd/201:30:552
2542993518,5cyclictest33-21ksoftirqd/201:26:162
254299351,5cyclictest121rcu_preempt23:19:392
254299350,3cyclictest121rcu_preempt00:21:092
2531993529,3cyclictest9-21ksoftirqd/022:15:370
2531993529,3cyclictest9-21ksoftirqd/000:25:590
2531993523,9cyclictest9-21ksoftirqd/001:20:550
2542993429,3cyclictest33-21ksoftirqd/223:15:372
2542993429,3cyclictest33-21ksoftirqd/222:20:372
2542993428,3cyclictest33-21ksoftirqd/223:40:562
2542993428,3cyclictest33-21ksoftirqd/223:30:382
2542993428,3cyclictest33-21ksoftirqd/221:30:532
2542993428,3cyclictest33-21ksoftirqd/220:31:062
2542993418,13cyclictest33-21ksoftirqd/201:11:142
254299341,5cyclictest121rcu_preempt22:28:532
254299341,3cyclictest121rcu_preempt01:00:552
2531993428,3cyclictest9-21ksoftirqd/022:05:360
2531993422,9cyclictest9-21ksoftirqd/021:00:490
2531993420,11cyclictest9-21ksoftirqd/000:15:390
2531993418,5cyclictest9-21ksoftirqd/022:55:370
2531993416,8cyclictest9-21ksoftirqd/023:00:370
177391340,1ptp4l401ktimersoftd/321:45:533
177391340,0ptp4l401ktimersoftd/301:31:273
2542993328,3cyclictest33-21ksoftirqd/222:41:082
2542993328,3cyclictest33-21ksoftirqd/222:30:542
2542993328,3cyclictest33-21ksoftirqd/222:20:562
2542993328,3cyclictest33-21ksoftirqd/221:45:362
2542993328,3cyclictest33-21ksoftirqd/200:55:562
2542993328,3cyclictest33-21ksoftirqd/200:55:562
2542993327,3cyclictest33-21ksoftirqd/220:20:342
2542993327,3cyclictest33-21ksoftirqd/200:15:562
2542993316,5cyclictest33-21ksoftirqd/223:36:262
254299330,3cyclictest121rcu_preempt00:00:472
2531993321,2cyclictest9-21ksoftirqd/001:31:020
253199330,3cyclictest121rcu_preempt21:05:540
177391330,1ptp4l401ktimersoftd/301:45:513
177391330,1ptp4l401ktimersoftd/301:07:023
177391330,0ptp4l401ktimersoftd/320:27:493
2542993227,3cyclictest33-21ksoftirqd/221:10:352
2542993226,3cyclictest33-21ksoftirqd/220:50:502
2542993226,3cyclictest33-21ksoftirqd/220:35:462
2542993219,10cyclictest33-21ksoftirqd/222:45:592
254299320,5cyclictest121rcu_preempt01:23:252
2531993228,2cyclictest9-21ksoftirqd/001:45:410
2531993227,3cyclictest9-21ksoftirqd/022:40:520
2531993227,3cyclictest9-21ksoftirqd/022:20:500
2531993226,3cyclictest9-21ksoftirqd/022:46:000
2531993224,4cyclictest9-21ksoftirqd/022:05:490
2531993219,10cyclictest9-21ksoftirqd/021:20:520
2531993219,10cyclictest9-21ksoftirqd/021:20:520
177391320,1ptp4l401ktimersoftd/321:28:063
254299316,5cyclictest33-21ksoftirqd/223:23:002
254299315,5cyclictest33-21ksoftirqd/221:21:052
254299315,5cyclictest33-21ksoftirqd/221:21:052
2542993126,3cyclictest33-21ksoftirqd/221:35:512
2542993126,3cyclictest33-21ksoftirqd/200:46:092
2542993125,3cyclictest33-21ksoftirqd/223:55:392
2542993125,3cyclictest33-21ksoftirqd/222:55:542
2542993125,3cyclictest33-21ksoftirqd/220:45:522
2542993125,3cyclictest33-21ksoftirqd/200:30:572
2538993125,3cyclictest25-21ksoftirqd/122:36:011
2538993124,4cyclictest25-21ksoftirqd/123:00:301
253199315,3cyclictest9-21ksoftirqd/000:01:490
2531993126,3cyclictest9-21ksoftirqd/021:50:530
2531993125,3cyclictest9-21ksoftirqd/022:20:370
2531993125,3cyclictest9-21ksoftirqd/021:20:350
2531993125,3cyclictest9-21ksoftirqd/020:55:340
177391310,1ptp4l401ktimersoftd/321:10:573
177391310,0ptp4l401ktimersoftd/321:17:213
177391310,0ptp4l401ktimersoftd/321:07:043
2542993025,3cyclictest33-21ksoftirqd/221:55:562
2542993025,3cyclictest33-21ksoftirqd/201:36:032
2542993024,3cyclictest33-21ksoftirqd/222:11:102
2542993024,3cyclictest33-21ksoftirqd/221:55:362
2542993024,3cyclictest33-21ksoftirqd/220:40:492
2531993025,3cyclictest9-21ksoftirqd/021:35:360
2531993024,3cyclictest9-21ksoftirqd/021:55:540
2531993024,3cyclictest9-21ksoftirqd/021:25:490
184092300,2sleep10-21swapper/123:20:531
177391300,0ptp4l401ktimersoftd/320:52:553
177391300,0ptp4l401ktimersoftd/320:38:053
2542992924,3cyclictest33-21ksoftirqd/221:05:352
2542992924,3cyclictest33-21ksoftirqd/220:25:532
2542992923,3cyclictest33-21ksoftirqd/222:01:012
2542992923,3cyclictest33-21ksoftirqd/221:25:472
2542992923,3cyclictest33-21ksoftirqd/221:15:352
2542992916,10cyclictest33-21ksoftirqd/222:06:002
253899290,3cyclictest121rcu_preempt23:39:591
2531992923,3cyclictest9-21ksoftirqd/020:55:490
177391290,0ptp4l401ktimersoftd/321:41:373
177391290,0ptp4l401ktimersoftd/320:49:383
2542992823,3cyclictest33-21ksoftirqd/220:55:452
2538992821,4cyclictest25-21ksoftirqd/123:26:431
2538992821,4cyclictest25-21ksoftirqd/120:38:461
177391280,1ptp4l401ktimersoftd/301:44:133
2538992621,3cyclictest25-21ksoftirqd/122:42:471
2538992620,3cyclictest25-21ksoftirqd/121:06:191
177391260,1ptp4l401ktimersoftd/321:55:413
2538992520,3cyclictest25-21ksoftirqd/122:11:201
2538992520,3cyclictest25-21ksoftirqd/100:03:181
2538992519,4cyclictest25-21ksoftirqd/121:47:511
2538992518,4cyclictest25-21ksoftirqd/122:06:341
2538992518,4cyclictest25-21ksoftirqd/122:04:211
2538992518,4cyclictest25-21ksoftirqd/121:28:521
2538992518,4cyclictest25-21ksoftirqd/120:57:031
2538992518,4cyclictest25-21ksoftirqd/101:03:591
2538992517,5cyclictest25-21ksoftirqd/121:41:491
2538992512,10cyclictest5118-21cat22:30:511
2538992511,11cyclictest12520-21sendmail23:50:391
177391250,1ptp4l401ktimersoftd/321:39:533
177391250,0ptp4l401ktimersoftd/322:10:343
253899249,12cyclictest25-21ksoftirqd/121:22:301
253899249,12cyclictest25-21ksoftirqd/121:22:301
2538992418,4cyclictest25-21ksoftirqd/101:32:061
2538992417,4cyclictest25-21ksoftirqd/121:57:391
2538992417,4cyclictest25-21ksoftirqd/121:32:301
2538992417,4cyclictest25-21ksoftirqd/120:32:481
177391240,1ptp4l401ktimersoftd/322:15:543
177391240,1ptp4l401ktimersoftd/301:23:263
48412231,18sleep14904-21munin-node20:25:341
2538992317,3cyclictest25-21ksoftirqd/123:43:511
2538992317,3cyclictest25-21ksoftirqd/101:21:501
2538992317,3cyclictest25-21ksoftirqd/101:16:521
2538992310,4cyclictest27875-21diskmemload00:45:551
2538992216,4cyclictest25-21ksoftirqd/121:37:561
2538992216,4cyclictest25-21ksoftirqd/101:09:171
2538992216,3cyclictest25-21ksoftirqd/123:51:371
2538992216,3cyclictest25-21ksoftirqd/120:43:171
2538992216,3cyclictest25-21ksoftirqd/120:28:301
2538992216,3cyclictest25-21ksoftirqd/100:38:581
253899214,3cyclictest121rcu_preempt01:00:331
253899214,3cyclictest121rcu_preempt01:00:331
2538992115,4cyclictest25-21ksoftirqd/120:46:531
2538992115,3cyclictest25-21ksoftirqd/121:14:101
2538992115,3cyclictest20594-21turbostat.cron00:35:391
2538992114,4cyclictest999-21if_err_ens322:25:541
2538992114,4cyclictest617-21grep23:00:591
2538992114,4cyclictest27318-21timerandwakeup00:41:081
2538992114,4cyclictest27306-21cpuspeed_turbos23:30:511
2538992114,4cyclictest26749-21grep00:05:561
2538992114,4cyclictest25-21ksoftirqd/121:04:031
2538992114,4cyclictest17926-21if_enp1s023:55:571
2538992114,4cyclictest17166-21apt01:45:531
2538992114,4cyclictest14962-21sendmail_mailqu23:16:041
2538992114,4cyclictest13275-21latency_hist23:15:381
2538992015,3cyclictest27143-21kworker/1:100:51:181
2538992015,3cyclictest27143-21kworker/1:100:51:181
2538992014,3cyclictest25-21ksoftirqd/121:52:141
2538992014,3cyclictest25-21ksoftirqd/101:25:471
2538992014,3cyclictest19560-21sed20:51:031
2538992013,4cyclictest3726-21sed00:15:591
2538992013,4cyclictest21816-21sh01:11:141
2538991914,3cyclictest28272-21dpkg22:20:491
2538991913,4cyclictest13486-21sh00:26:081
2538991913,3cyclictest26736-21cat22:15:581
2538991912,4cyclictest4635-21expr23:05:521
2538991912,4cyclictest4635-21expr23:05:521
2538991912,4cyclictest30412-21irqrtprio21:15:551
2538991814,2cyclictest0-21swapper/122:45:511
2538991813,3cyclictest31160-21kworker/1:300:10:431
2538991812,3cyclictest14755-21seq01:41:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional