You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 08:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot3.osadl.org (updated Fri Nov 28, 2025 00:45:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
120382230163,22sleep20-21swapper/219:08:272
137191206172,22phc2sys0-21swapper/319:06:033
120462202168,23sleep00-21swapper/019:08:330
119042201168,22sleep10-21swapper/119:06:531
1210021700,5sleep11250699cyclictest20:15:201
739821370,2sleep20-21swapper/220:05:222
12508997011,57cyclictest16359-21kworker/3:220:33:473
136991570,1ptp4l391rcuc/323:58:063
1250899576,26cyclictest19526-21taskset23:52:283
12505995649,3cyclictest11157-21ls22:30:000
144732530,2sleep0101ktimersoftd/022:35:200
59298520,12rtkit-daemon591-21rtkit-daemon21:55:461
281932500,3sleep20-21swapper/219:40:232
1250899493,22cyclictest0-21swapper/320:38:173
1250899483,24cyclictest0-21swapper/321:58:243
1250899454,22cyclictest0-21swapper/300:38:543
59298440,2rtkit-daemon591-21rtkit-daemon23:41:350
136991430,0ptp4l401ktimersoftd/323:02:273
1250899436,26cyclictest12363-21taskset23:37:073
1250899436,23cyclictest8385-21seq20:09:013
1250599432,2cyclictest121rcu_preempt21:32:340
136991420,0ptp4l401ktimersoftd/322:45:023
136991420,0ptp4l401ktimersoftd/320:14:353
136991410,0ptp4l401ktimersoftd/323:45:003
1250899410,2cyclictest136991ptp4l22:10:153
136991400,0ptp4l401ktimersoftd/320:25:223
1250899391,3cyclictest0-21swapper/300:25:313
1250899381,36cyclictest16359-21kworker/3:222:32:353
12507993832,4cyclictest33-21ksoftirqd/200:35:192
12506993833,3cyclictest8477-21latency_hist20:10:001
1250899370,36cyclictest35592irq/126-enp1s0-19:57:473
1250899362,12cyclictest16359-21kworker/3:222:17:513
12507993631,3cyclictest33-21ksoftirqd/219:57:302
12507993631,3cyclictest33-21ksoftirqd/200:23:292
12507993630,4cyclictest33-21ksoftirqd/221:50:592
12507993630,4cyclictest33-21ksoftirqd/221:44:232
12506993631,3cyclictest25-21ksoftirqd/122:02:551
12506993631,3cyclictest25-21ksoftirqd/121:40:001
1250899353,19cyclictest16359-21kworker/3:222:29:193
12508993529,4cyclictest41-21ksoftirqd/319:12:283
12507993530,3cyclictest33-21ksoftirqd/222:48:032
12507993530,3cyclictest33-21ksoftirqd/200:11:542
12507993529,4cyclictest33-21ksoftirqd/222:22:182
12507993529,4cyclictest33-21ksoftirqd/221:30:302
12507993529,4cyclictest33-21ksoftirqd/200:16:472
12507993529,4cyclictest33-21ksoftirqd/200:04:162
12507993528,5cyclictest33-21ksoftirqd/220:14:132
12506993529,3cyclictest25-21ksoftirqd/120:45:001
12505993530,3cyclictest9-21ksoftirqd/019:45:110
12505993529,4cyclictest9-21ksoftirqd/021:36:390
1250899343,20cyclictest16359-21kworker/3:221:53:393
12508993429,3cyclictest41-21ksoftirqd/323:26:143
12508993428,4cyclictest41-21ksoftirqd/323:48:293
12508993428,4cyclictest41-21ksoftirqd/322:56:333
12508993428,4cyclictest41-21ksoftirqd/300:24:273
12508993428,4cyclictest41-21ksoftirqd/300:07:533
12508993427,4cyclictest41-21ksoftirqd/300:17:053
1250899341,32cyclictest16359-21kworker/3:222:24:193
12508993413,18cyclictest1723-21sed23:15:193
1250899341,2cyclictest0-21swapper/322:36:253
1250899340,4cyclictest27313-21/usr/sbin/munin19:40:163
1250899340,4cyclictest19093-21chrt19:20:583
12507993428,4cyclictest33-21ksoftirqd/223:31:502
12507993428,4cyclictest33-21ksoftirqd/223:15:102
12507993428,4cyclictest33-21ksoftirqd/220:59:022
12507993428,4cyclictest33-21ksoftirqd/220:24:442
12507993428,3cyclictest33-21ksoftirqd/222:31:452
12507993428,3cyclictest33-21ksoftirqd/222:05:172
12506993429,3cyclictest25-21ksoftirqd/123:55:001
12506993429,3cyclictest25-21ksoftirqd/123:26:351
12506993429,3cyclictest25-21ksoftirqd/120:50:491
12506993429,3cyclictest25-21ksoftirqd/100:30:001
12506993428,4cyclictest25-21ksoftirqd/122:18:351
12506993428,4cyclictest25-21ksoftirqd/121:06:311
12506993428,4cyclictest25-21ksoftirqd/121:04:021
12506993428,4cyclictest25-21ksoftirqd/119:47:401
12505993429,3cyclictest9-21ksoftirqd/022:25:000
12505993429,3cyclictest9-21ksoftirqd/020:35:370
12505993428,4cyclictest9-21ksoftirqd/020:02:070
59298330,1rtkit-daemon591-21rtkit-daemon22:27:461
12508993328,3cyclictest41-21ksoftirqd/320:05:003
12508993327,4cyclictest41-21ksoftirqd/323:09:013
12508993327,4cyclictest41-21ksoftirqd/319:37:553
12508993327,4cyclictest41-21ksoftirqd/319:25:113
12508993326,5cyclictest41-21ksoftirqd/300:31:243
1250899332,30cyclictest16359-21kworker/3:222:43:033
1250899332,30cyclictest13220-21kworker/3:100:03:513
1250899331,3cyclictest0-21swapper/323:32:463
1250899330,4cyclictest12891-21sed20:15:283
12507993328,3cyclictest33-21ksoftirqd/223:23:072
12507993328,3cyclictest33-21ksoftirqd/222:57:032
12507993328,3cyclictest33-21ksoftirqd/219:50:012
12507993328,3cyclictest33-21ksoftirqd/200:33:282
12507993327,4cyclictest33-21ksoftirqd/223:29:452
12507993327,4cyclictest33-21ksoftirqd/223:09:202
12507993327,4cyclictest33-21ksoftirqd/221:26:222
12507993327,4cyclictest33-21ksoftirqd/219:50:382
12506993328,3cyclictest25-21ksoftirqd/123:15:011
12506993327,4cyclictest25-21ksoftirqd/123:32:301
12506993327,4cyclictest25-21ksoftirqd/122:52:421
12506993327,4cyclictest25-21ksoftirqd/122:36:451
12506993327,4cyclictest25-21ksoftirqd/121:27:171
12506993327,4cyclictest25-21ksoftirqd/120:25:471
12505993328,3cyclictest9-21ksoftirqd/023:05:000
12505993328,3cyclictest9-21ksoftirqd/019:18:590
12505993327,4cyclictest9-21ksoftirqd/023:35:450
12505993327,4cyclictest9-21ksoftirqd/019:40:050
12505993327,4cyclictest9-21ksoftirqd/000:11:520
12505993326,5cyclictest9-21ksoftirqd/021:17:500
12505993326,5cyclictest9-21ksoftirqd/021:08:510
12508993227,3cyclictest41-21ksoftirqd/319:45:473
12508993226,4cyclictest41-21ksoftirqd/322:54:453
12508993226,4cyclictest41-21ksoftirqd/322:01:533
12508993226,4cyclictest41-21ksoftirqd/320:56:183
12508993214,16cyclictest6083-21sed21:10:193
12507993227,3cyclictest33-21ksoftirqd/220:45:002
12507993226,4cyclictest33-21ksoftirqd/223:54:402
12507993226,4cyclictest33-21ksoftirqd/222:52:422
12507993226,4cyclictest33-21ksoftirqd/221:48:042
12507993226,4cyclictest33-21ksoftirqd/221:23:132
12507993226,4cyclictest33-21ksoftirqd/221:16:112
12507993226,4cyclictest33-21ksoftirqd/221:14:182
12507993226,4cyclictest33-21ksoftirqd/220:25:522
12507993226,4cyclictest33-21ksoftirqd/219:12:352
12507993226,4cyclictest33-21ksoftirqd/200:05:322
12506993227,3cyclictest25-21ksoftirqd/123:06:581
12506993227,3cyclictest25-21ksoftirqd/122:45:131
12506993227,3cyclictest25-21ksoftirqd/122:43:351
12506993226,4cyclictest25-21ksoftirqd/122:08:081
12506993226,4cyclictest25-21ksoftirqd/121:46:071
12506993226,4cyclictest25-21ksoftirqd/121:31:311
12506993226,4cyclictest25-21ksoftirqd/120:55:401
12506993226,4cyclictest25-21ksoftirqd/100:02:191
12506993226,3cyclictest25-21ksoftirqd/121:19:591
12506993225,5cyclictest25-21ksoftirqd/121:23:401
12506993225,5cyclictest25-21ksoftirqd/119:12:551
12505993227,3cyclictest9-21ksoftirqd/022:41:190
12505993227,3cyclictest9-21ksoftirqd/020:10:340
12505993227,3cyclictest9-21ksoftirqd/019:58:100
12505993227,3cyclictest9-21ksoftirqd/019:22:440
12505993226,4cyclictest9-21ksoftirqd/022:31:350
12505993226,4cyclictest9-21ksoftirqd/021:10:470
12505993226,4cyclictest9-21ksoftirqd/020:15:510
12505993226,4cyclictest9-21ksoftirqd/000:01:490
12505993225,5cyclictest9-21ksoftirqd/023:27:510
136991310,6ptp4l16123-21netstat21:30:243
12508993126,3cyclictest41-21ksoftirqd/320:49:163
12508993126,3cyclictest41-21ksoftirqd/300:10:363
12508993125,4cyclictest41-21ksoftirqd/321:29:403
12508993125,4cyclictest41-21ksoftirqd/321:08:343
12508993114,15cyclictest30050-21latency_hist20:55:013
1250899310,3cyclictest10185-21/usr/sbin/munin21:20:123
1250899310,2cyclictest0-21swapper/321:45:193
12507993125,4cyclictest33-21ksoftirqd/221:56:112
12507993125,4cyclictest33-21ksoftirqd/220:52:522
12507993125,4cyclictest33-21ksoftirqd/219:25:312
12506993126,3cyclictest25-21ksoftirqd/123:48:151
12506993126,3cyclictest25-21ksoftirqd/121:52:471
12506993126,3cyclictest25-21ksoftirqd/120:31:161
12506993125,4cyclictest25-21ksoftirqd/123:37:311
12506993125,4cyclictest25-21ksoftirqd/123:23:551
12506993125,4cyclictest25-21ksoftirqd/121:14:501
12506993125,4cyclictest25-21ksoftirqd/120:36:101
12506993125,4cyclictest25-21ksoftirqd/119:24:091
1250699312,3cyclictest121rcu_preempt23:55:111
12505993126,3cyclictest9-21ksoftirqd/023:45:360
12505993126,3cyclictest9-21ksoftirqd/022:51:480
12505993126,3cyclictest9-21ksoftirqd/020:08:170
12505993126,3cyclictest9-21ksoftirqd/019:30:100
12505993125,4cyclictest9-21ksoftirqd/022:57:260
12505993125,4cyclictest9-21ksoftirqd/022:49:080
12505993125,4cyclictest9-21ksoftirqd/021:48:590
12505993125,4cyclictest9-21ksoftirqd/021:43:010
12505993125,4cyclictest9-21ksoftirqd/020:56:510
12505993125,4cyclictest9-21ksoftirqd/020:52:100
12505993125,4cyclictest9-21ksoftirqd/000:26:070
1250899308,19cyclictest31011-21cut23:10:143
1250899306,21cyclictest684-21ls21:00:183
12508993014,13cyclictest8414-21if_err_enp4s021:15:163
12508993010,17cyclictest903-21seq19:50:443
1250899300,2cyclictest0-21swapper/323:20:283
1250899300,29cyclictest16359-21kworker/3:221:40:513
12507993025,3cyclictest33-21ksoftirqd/222:02:102
12507993025,3cyclictest33-21ksoftirqd/200:27:362
12507993024,4cyclictest33-21ksoftirqd/223:35:512
12507993024,4cyclictest33-21ksoftirqd/223:12:202
12507993024,4cyclictest33-21ksoftirqd/222:16:092
12507993024,4cyclictest33-21ksoftirqd/221:08:172
12507993024,4cyclictest33-21ksoftirqd/219:15:382
12506993026,3cyclictest25-21ksoftirqd/119:27:241
12506993025,3cyclictest25-21ksoftirqd/123:05:001
12506993025,3cyclictest25-21ksoftirqd/120:49:171
12506993025,3cyclictest25-21ksoftirqd/120:24:551
12506993025,3cyclictest25-21ksoftirqd/120:10:191
12506993024,3cyclictest25-21ksoftirqd/119:52:291
12506993023,5cyclictest25-21ksoftirqd/100:33:271
12506993020,9cyclictest25-21ksoftirqd/123:15:261
12505993025,3cyclictest9-21ksoftirqd/023:11:190
12505993025,3cyclictest9-21ksoftirqd/022:10:370
12505993025,3cyclictest9-21ksoftirqd/021:57:500
12505993025,3cyclictest9-21ksoftirqd/021:54:500
12505993025,3cyclictest9-21ksoftirqd/019:27:490
12505993025,3cyclictest9-21ksoftirqd/000:18:400
12505993024,4cyclictest9-21ksoftirqd/023:59:320
12505993024,4cyclictest9-21ksoftirqd/023:33:480
12505993024,4cyclictest9-21ksoftirqd/022:17:510
12505993024,4cyclictest9-21ksoftirqd/021:29:010
12505993024,4cyclictest9-21ksoftirqd/020:44:120
12505993024,4cyclictest9-21ksoftirqd/020:26:310
1250899293,3cyclictest16359-21kworker/3:222:08:313
1250899293,3cyclictest16359-21kworker/3:221:37:113
12508992923,4cyclictest41-21ksoftirqd/320:24:173
1250899290,3cyclictest22027-21/usr/sbin/munin19:30:203
12507992924,3cyclictest33-21ksoftirqd/223:04:512
12507992924,3cyclictest33-21ksoftirqd/222:45:002
12507992924,3cyclictest33-21ksoftirqd/219:34:392
12507992924,3cyclictest33-21ksoftirqd/219:22:242
12507992923,4cyclictest33-21ksoftirqd/223:45:502
12506992924,3cyclictest25-21ksoftirqd/122:20:491
12506992924,3cyclictest25-21ksoftirqd/121:42:411
12506992924,3cyclictest25-21ksoftirqd/100:16:231
12506992923,4cyclictest25-21ksoftirqd/119:31:071
12505992923,4cyclictest9-21ksoftirqd/023:52:540
12505992923,4cyclictest9-21ksoftirqd/019:51:320
1250899289,15cyclictest16292-21run-parts19:15:253
1250899281,3cyclictest23076-21/usr/sbin/munin20:40:273
12507992823,4cyclictest33-21ksoftirqd/220:05:002
12507992823,3cyclictest33-21ksoftirqd/223:59:382
12507992823,3cyclictest33-21ksoftirqd/222:10:472
12507992823,3cyclictest33-21ksoftirqd/221:01:202
12507992822,4cyclictest33-21ksoftirqd/220:18:342
12507992822,4cyclictest33-21ksoftirqd/219:37:342
12507992821,4cyclictest33-21ksoftirqd/220:40:002
12506992823,3cyclictest25-21ksoftirqd/122:56:271
12506992823,3cyclictest25-21ksoftirqd/122:30:281
12506992822,4cyclictest25-21ksoftirqd/120:02:121
12506992822,3cyclictest25-21ksoftirqd/119:55:181
12506992821,4cyclictest25-21ksoftirqd/119:35:101
12505992823,3cyclictest9-21ksoftirqd/023:23:070
12505992823,3cyclictest9-21ksoftirqd/019:10:320
12505992823,3cyclictest9-21ksoftirqd/000:32:060
12505992822,4cyclictest9-21ksoftirqd/019:38:170
12507992722,2cyclictest1600-1kworker/2:1H22:40:012
12506992722,3cyclictest8205-21cut00:35:181
12506992722,3cyclictest25-21ksoftirqd/122:14:131
12506992722,3cyclictest25-21ksoftirqd/119:20:011
12506992722,3cyclictest25-21ksoftirqd/100:23:361
12506992721,4cyclictest25-21ksoftirqd/123:41:281
12505992722,3cyclictest9-21ksoftirqd/000:40:010
12505992721,4cyclictest9-21ksoftirqd/022:09:090
12505992721,4cyclictest9-21ksoftirqd/020:24:340
12507992621,3cyclictest33-21ksoftirqd/223:40:222
12507992621,3cyclictest33-21ksoftirqd/220:48:572
12506992619,5cyclictest25-21ksoftirqd/119:42:431
12506992619,5cyclictest25-21ksoftirqd/100:09:401
12505992621,3cyclictest9-21ksoftirqd/023:19:040
12505992621,3cyclictest9-21ksoftirqd/021:21:020
12505992620,4cyclictest9-21ksoftirqd/021:01:130
12507992519,4cyclictest33-21ksoftirqd/222:27:112
12506992520,3cyclictest25-21ksoftirqd/100:11:081
12505992520,3cyclictest9-21ksoftirqd/022:01:060
12505992520,3cyclictest9-21ksoftirqd/000:05:420
12505992519,4cyclictest9-21ksoftirqd/020:30:520
12505992519,4cyclictest9-21ksoftirqd/000:23:190
12507992418,4cyclictest33-21ksoftirqd/220:32:302
12505992318,3cyclictest9-21ksoftirqd/023:09:220
12505992318,3cyclictest9-21ksoftirqd/020:46:480
12507992219,2cyclictest33-21ksoftirqd/221:35:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional