You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 04:54
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackdslot4.osadl.org (updated Wed Dec 03, 2025 00:45:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
112432203167,24sleep30-21swapper/319:09:313
110062203170,22sleep00-21swapper/019:06:340
111792201167,22sleep10-21swapper/119:08:401
109252201167,23sleep20-21swapper/219:05:322
2491721380,2sleep30-21swapper/320:40:013
11573997011,16cyclictest19341-21sed23:40:271
427291680,1ptp4l401ktimersoftd/323:07:513
1157399669,18cyclictest5478-21df_abs21:05:151
11573996411,11cyclictest14477-21sed19:15:151
1157399609,10cyclictest4907-21/usr/sbin/munin22:10:121
1157399599,11cyclictest246952sleep122:49:571
1128125839,10sleep00-21swapper/019:10:000
11573995710,8cyclictest26245-21phc2sys22:50:231
1157399569,15cyclictest17292-21latency_hist22:35:001
1157399559,15cyclictest27297-21/usr/sbin/munin21:50:181
11573995410,9cyclictest13054-21diskstats22:25:141
11573995410,16cyclictest30936-21ls21:55:191
78962530,2sleep00-21swapper/021:10:140
1157399539,15cyclictest32383-21fschecks_count19:50:191
11573995311,8cyclictest1016-21irqstats23:05:201
11573995311,7cyclictest3989-21memory00:15:211
11573995310,9cyclictest28015-21ps19:40:241
11573995310,16cyclictest8475-21basename00:25:181
1157399528,8cyclictest3106-21ls19:55:211
11573995211,8cyclictest11502-21users20:10:291
11573995210,7cyclictest30490-21phc2sys19:45:231
1157399519,9cyclictest13417-21latency20:15:211
1157399519,8cyclictest23108-21iostat_ios21:40:201
11573995110,9cyclictest11086-21meminfo21:15:201
11573995110,8cyclictest8534-21irqstats23:20:181
11573995110,8cyclictest8524-21ps20:05:231
11573995110,8cyclictest6153-21memory23:15:201
11573995110,8cyclictest13064-21fw_packets21:20:171
182072500,3sleep318211-21irqrtprio21:30:193
1157399509,8cyclictest9636-21apt-get22:20:041
1157399509,8cyclictest13116-21python19:10:251
1157399506,11cyclictest225722sleep123:50:061
11573995010,8cyclictest8906-21python22:15:261
97902490,1sleep29794-21wc22:20:012
53492490,2sleep20-21swapper/223:15:132
1157399499,8cyclictest20063-21cpu22:40:131
11573994910,8cyclictest325-21df_inode22:00:151
11573994910,8cyclictest15896-21iostat23:35:191
11575994832,13cyclictest41-21ksoftirqd/320:30:583
1157399488,9cyclictest16063-21memory21:25:221
317442470,3sleep00-21swapper/020:50:290
317442470,3sleep00-21swapper/020:50:290
1157399478,8cyclictest8599-21ls21:10:201
1157399468,8cyclictest29280-21ls00:00:261
1157399459,15cyclictest1621-21gnome-shell19:35:581
1157399458,8cyclictest3181-21iostat_ios22:05:181
11573994510,13cyclictest2928-21df_abs23:10:141
427291440,0ptp4l401ktimersoftd/323:38:223
11575994428,13cyclictest41-21ksoftirqd/320:45:593
11574994438,3cyclictest32199-21dump-pmu-power23:05:002
1157399448,8cyclictest30676-21hddtemp_smartct00:05:181
1157399447,9cyclictest18530-21meminfo21:30:211
11572994438,4cyclictest6974-21sshd22:12:540
11572994437,4cyclictest24809-21cron23:55:000
427291430,0ptp4l401ktimersoftd/323:45:223
427291430,0ptp4l401ktimersoftd/322:53:173
11574994324,10cyclictest33-21ksoftirqd/223:40:192
1157399439,8cyclictest20565-21if_enp4s021:35:171
1157399437,8cyclictest14287-21ls23:30:271
1157399434,16cyclictest0-21swapper/120:31:081
427291420,34ptp4l0-21swapper/319:10:003
427291420,0ptp4l401ktimersoftd/319:36:193
11574994237,2cyclictest33-21ksoftirqd/200:15:002
427291410,0ptp4l401ktimersoftd/321:10:323
11574994136,3cyclictest33-21ksoftirqd/220:39:592
11575994034,4cyclictest41-21ksoftirqd/322:22:593
11575994034,4cyclictest41-21ksoftirqd/320:01:463
11574994030,9cyclictest33-21ksoftirqd/219:35:232
11575993933,4cyclictest41-21ksoftirqd/323:42:343
11575993933,4cyclictest41-21ksoftirqd/323:30:433
11575993933,4cyclictest41-21ksoftirqd/323:18:513
11575993933,4cyclictest41-21ksoftirqd/321:56:113
11575993933,4cyclictest41-21ksoftirqd/300:32:453
11575993932,5cyclictest41-21ksoftirqd/321:15:313
11575993932,4cyclictest41-21ksoftirqd/322:44:493
11574993934,3cyclictest33-21ksoftirqd/220:15:002
11574993934,3cyclictest33-21ksoftirqd/200:40:002
11574993933,3cyclictest33-21ksoftirqd/221:05:002
11572993932,4cyclictest9-21ksoftirqd/023:49:010
11572993932,4cyclictest9-21ksoftirqd/021:31:090
36998380,2rtkit-daemon0-21swapper/222:35:292
11575993832,4cyclictest41-21ksoftirqd/323:00:103
11575993832,4cyclictest41-21ksoftirqd/320:38:443
11575993832,4cyclictest41-21ksoftirqd/320:20:393
11575993832,4cyclictest41-21ksoftirqd/320:20:393
11575993832,4cyclictest41-21ksoftirqd/319:34:533
11575993832,4cyclictest41-21ksoftirqd/319:26:233
11575993831,4cyclictest41-21ksoftirqd/322:12:563
11575993830,5cyclictest41-21ksoftirqd/320:07:433
11575993823,12cyclictest41-21ksoftirqd/322:07:583
11574993833,3cyclictest33-21ksoftirqd/223:00:002
1157399388,12cyclictest18824-21python20:25:261
11573993831,4cyclictest25-21ksoftirqd/119:24:441
11573993831,4cyclictest25-21ksoftirqd/100:36:001
1157299388,27cyclictest28592-21sshd19:41:180
11572993832,4cyclictest9-21ksoftirqd/023:30:560
11572993831,4cyclictest9-21ksoftirqd/022:58:220
11572993831,4cyclictest9-21ksoftirqd/020:00:540
11572993831,4cyclictest9-21ksoftirqd/000:13:000
11575993731,4cyclictest41-21ksoftirqd/321:09:013
11575993731,4cyclictest41-21ksoftirqd/319:46:343
11575993730,4cyclictest41-21ksoftirqd/319:57:093
11575993713,4cyclictest15190-21grep22:30:123
11574993732,3cyclictest33-21ksoftirqd/220:30:192
11574993732,2cyclictest33-21ksoftirqd/220:40:182
11574993731,4cyclictest33-21ksoftirqd/222:11:422
11574993731,4cyclictest33-21ksoftirqd/219:51:372
11574993726,9cyclictest33-21ksoftirqd/223:10:162
11574993724,10cyclictest33-21ksoftirqd/222:40:252
11574993724,10cyclictest33-21ksoftirqd/222:05:242
1157399377,18cyclictest0-21swapper/100:10:201
11573993732,3cyclictest25-21ksoftirqd/123:02:591
11573993732,3cyclictest25-21ksoftirqd/120:49:571
11572993731,4cyclictest9-21ksoftirqd/019:31:100
11572993731,4cyclictest9-21ksoftirqd/019:27:170
11572993731,4cyclictest9-21ksoftirqd/019:16:260
11572993731,4cyclictest9-21ksoftirqd/000:31:580
11572993731,3cyclictest9-21ksoftirqd/019:47:580
11572993730,4cyclictest9-21ksoftirqd/022:09:520
11572993730,4cyclictest9-21ksoftirqd/019:52:270
11575993631,3cyclictest41-21ksoftirqd/321:46:003
11575993631,3cyclictest41-21ksoftirqd/321:02:323
11575993630,4cyclictest41-21ksoftirqd/323:57:143
11575993630,4cyclictest41-21ksoftirqd/320:59:463
11575993630,4cyclictest41-21ksoftirqd/320:10:563
11575993630,4cyclictest41-21ksoftirqd/319:16:123
11575993630,4cyclictest41-21ksoftirqd/300:16:123
11575993630,4cyclictest41-21ksoftirqd/300:14:193
11575993629,4cyclictest41-21ksoftirqd/321:28:393
11574993632,3cyclictest33-21ksoftirqd/220:05:192
11574993631,3cyclictest33-21ksoftirqd/222:20:192
11574993631,3cyclictest33-21ksoftirqd/220:57:402
11574993631,3cyclictest33-21ksoftirqd/219:25:012
11574993630,4cyclictest33-21ksoftirqd/222:01:502
11574993630,4cyclictest33-21ksoftirqd/221:08:162
11574993630,4cyclictest33-21ksoftirqd/220:53:072
11574993630,4cyclictest33-21ksoftirqd/220:53:072
11574993630,4cyclictest33-21ksoftirqd/219:59:422
11574993630,4cyclictest33-21ksoftirqd/200:08:252
11574993628,7cyclictest33-21ksoftirqd/221:35:222
11574993625,9cyclictest33-21ksoftirqd/221:20:222
1157399366,15cyclictest1264-21snmpd20:20:381
1157399366,15cyclictest1264-21snmpd20:20:381
11573993630,4cyclictest25-21ksoftirqd/122:35:041
11573993629,5cyclictest25-21ksoftirqd/121:45:341
11573993611,16cyclictest30607-21sshd20:50:201
11573993611,16cyclictest30607-21sshd20:50:201
11572993630,4cyclictest9-21ksoftirqd/023:42:510
11572993630,4cyclictest9-21ksoftirqd/022:20:350
11572993630,4cyclictest9-21ksoftirqd/021:37:260
11572993630,4cyclictest9-21ksoftirqd/000:25:560
11572993629,4cyclictest9-21ksoftirqd/022:51:200
11572993629,4cyclictest9-21ksoftirqd/022:45:500
11575993529,4cyclictest41-21ksoftirqd/322:27:443
11575993529,4cyclictest41-21ksoftirqd/322:17:093
11575993529,4cyclictest41-21ksoftirqd/321:36:213
11575993528,4cyclictest41-21ksoftirqd/322:57:053
11574993531,3cyclictest33-21ksoftirqd/223:45:192
11574993531,3cyclictest33-21ksoftirqd/222:25:132
11574993530,3cyclictest33-21ksoftirqd/223:25:002
11574993530,3cyclictest33-21ksoftirqd/219:16:382
11574993529,4cyclictest33-21ksoftirqd/221:59:272
11574993529,4cyclictest33-21ksoftirqd/220:25:432
11574993529,4cyclictest33-21ksoftirqd/219:49:012
11574993529,4cyclictest33-21ksoftirqd/219:27:592
11574993529,4cyclictest33-21ksoftirqd/200:34:402
11574993529,4cyclictest33-21ksoftirqd/200:25:302
11574993528,4cyclictest33-21ksoftirqd/200:00:342
11573993529,4cyclictest25-21ksoftirqd/119:27:251
11572993529,4cyclictest9-21ksoftirqd/021:01:160
11572993529,4cyclictest9-21ksoftirqd/020:12:330
11572993529,3cyclictest9-21ksoftirqd/023:27:270
11572993528,4cyclictest9-21ksoftirqd/023:20:570
11572993528,4cyclictest9-21ksoftirqd/023:19:160
11575993429,3cyclictest41-21ksoftirqd/319:40:513
11575993428,4cyclictest41-21ksoftirqd/323:11:293
11575993428,4cyclictest41-21ksoftirqd/320:27:253
11575993428,4cyclictest41-21ksoftirqd/300:39:223
11575993428,3cyclictest41-21ksoftirqd/323:26:593
11575993427,4cyclictest41-21ksoftirqd/320:54:543
11575993427,4cyclictest41-21ksoftirqd/320:54:543
11575993427,4cyclictest41-21ksoftirqd/320:16:183
11575993427,4cyclictest41-21ksoftirqd/300:09:063
11574993430,3cyclictest33-21ksoftirqd/222:50:192
11574993430,3cyclictest33-21ksoftirqd/221:45:132
11574993430,3cyclictest33-21ksoftirqd/221:25:202
11574993429,3cyclictest33-21ksoftirqd/220:50:002
11574993429,3cyclictest33-21ksoftirqd/220:18:442
11574993429,3cyclictest33-21ksoftirqd/220:05:002
11574993428,4cyclictest33-21ksoftirqd/222:50:002
11574993428,4cyclictest33-21ksoftirqd/222:31:132
11574993428,4cyclictest33-21ksoftirqd/221:34:492
1157399347,8cyclictest28525-21ls22:55:201
11573993429,3cyclictest25-21ksoftirqd/120:58:481
11573993427,4cyclictest25-21ksoftirqd/100:30:421
11572993429,3cyclictest9-21ksoftirqd/021:22:260
11572993428,4cyclictest9-21ksoftirqd/023:37:540
11572993428,4cyclictest9-21ksoftirqd/020:28:420
11572993428,4cyclictest9-21ksoftirqd/019:59:380
11572993428,3cyclictest9-21ksoftirqd/019:36:230
11575993328,3cyclictest41-21ksoftirqd/323:53:193
11575993328,3cyclictest41-21ksoftirqd/322:00:373
11575993314,3cyclictest7829-21/usr/sbin/munin00:25:213
11574993328,3cyclictest33-21ksoftirqd/223:58:052
11574993327,4cyclictest33-21ksoftirqd/200:24:402
1157499330,2cyclictest121rcu_preempt23:55:002
1157399339,16cyclictest21157-21kernelversion23:45:191
1157399339,14cyclictest6105-21iostat_ios00:20:181
1157399335,16cyclictest0-21swapper/123:55:191
11572993328,3cyclictest9-21ksoftirqd/023:04:160
11572993328,3cyclictest9-21ksoftirqd/022:40:370
11572993328,3cyclictest9-21ksoftirqd/022:18:380
11572993328,3cyclictest9-21ksoftirqd/000:23:140
11572993327,4cyclictest9-21ksoftirqd/021:58:290
11572993327,4cyclictest9-21ksoftirqd/020:36:530
11572993327,4cyclictest9-21ksoftirqd/020:33:520
11575993227,3cyclictest41-21ksoftirqd/322:45:083
11575993227,3cyclictest41-21ksoftirqd/321:21:343
11575993226,4cyclictest41-21ksoftirqd/321:53:223
11575993225,5cyclictest41-21ksoftirqd/300:23:503
11574993228,3cyclictest33-21ksoftirqd/221:15:002
11574993227,3cyclictest33-21ksoftirqd/223:35:332
11574993227,3cyclictest33-21ksoftirqd/219:31:202
11574993227,3cyclictest33-21ksoftirqd/200:15:182
11574993226,4cyclictest33-21ksoftirqd/219:42:192
1157499322,28cyclictest20852-21kworker/2:023:05:142
11573993227,3cyclictest25-21ksoftirqd/120:02:291
11573993226,4cyclictest25-21ksoftirqd/120:39:181
11573993226,4cyclictest25-21ksoftirqd/119:30:111
11572993226,4cyclictest9-21ksoftirqd/022:25:530
11572993226,4cyclictest9-21ksoftirqd/020:08:040
11572993226,4cyclictest9-21ksoftirqd/000:36:270
11572993226,4cyclictest9-21ksoftirqd/000:16:080
11572993226,3cyclictest9-21ksoftirqd/023:06:000
11572993226,3cyclictest9-21ksoftirqd/021:19:450
11572993225,4cyclictest9-21ksoftirqd/000:09:310
11575993125,3cyclictest41-21ksoftirqd/319:21:513
11574993127,3cyclictest33-21ksoftirqd/219:10:202
11574993126,3cyclictest33-21ksoftirqd/220:22:472
11574993126,3cyclictest33-21ksoftirqd/220:22:472
11574993125,4cyclictest33-21ksoftirqd/221:17:142
11572993126,3cyclictest9-21ksoftirqd/023:56:310
11572993126,3cyclictest9-21ksoftirqd/021:40:480
11572993125,4cyclictest9-21ksoftirqd/022:36:520
11572993125,4cyclictest9-21ksoftirqd/020:17:560
11572993124,4cyclictest9-21ksoftirqd/021:50:540
11575993024,4cyclictest41-21ksoftirqd/319:53:473
11575993024,3cyclictest41-21ksoftirqd/322:36:533
11574993025,3cyclictest33-21ksoftirqd/223:25:192
11574993025,3cyclictest33-21ksoftirqd/221:44:562
11574993024,4cyclictest33-21ksoftirqd/221:54:412
11572993024,4cyclictest9-21ksoftirqd/022:32:270
11572993024,4cyclictest9-21ksoftirqd/021:05:090
11575992924,3cyclictest41-21ksoftirqd/323:21:113
11575992924,3cyclictest41-21ksoftirqd/300:01:283
11572992924,3cyclictest9-21ksoftirqd/020:59:470
11572992922,4cyclictest9-21ksoftirqd/019:24:570
11575992823,3cyclictest41-21ksoftirqd/321:42:383
11574992823,2cyclictest33-21ksoftirqd/223:35:002
11573992810,7cyclictest12287-21sshd23:28:031
11572992823,3cyclictest9-21ksoftirqd/023:11:100
11572992721,4cyclictest9-21ksoftirqd/022:02:100
11572992720,4cyclictest9-21ksoftirqd/020:21:450
11572992720,4cyclictest9-21ksoftirqd/020:21:450
11573992615,8cyclictest25155-21/usr/sbin/munin20:40:131
11573992615,7cyclictest4390-21/usr/sbin/munin21:00:291
11572992520,3cyclictest9-21ksoftirqd/020:42:060
11572992519,4cyclictest9-21ksoftirqd/000:02:160
11572992419,3cyclictest9-21ksoftirqd/021:46:570
11572992418,4cyclictest9-21ksoftirqd/021:26:390
11572992418,4cyclictest9-21ksoftirqd/020:45:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional