You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-09 - 12:33
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot2.osadl.org (updated Wed Jul 09, 2025 00:45:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
281992227164,52sleep00-21swapper/019:08:110
137391205172,22phc2sys0-21swapper/319:06:123
262022204172,22sleep20-21swapper/219:05:032
262012202169,22sleep10-21swapper/119:05:011
28633997012,17cyclictest3203-21latency_hist19:25:002
28633996810,18cyclictest310-21fschecks_count22:40:172
92962540,3sleep39254-21sendmail23:00:003
298842500,1sleep30-21swapper/322:35:013
100232500,2sleep30-21swapper/321:50:293
314632490,1sleep131465-21awk22:35:271
28633994718,27cyclictest19400-21kworker/2:121:40:192
2863499461,23cyclictest2494-21gnome-shell00:19:593
137291440,0ptp4l401ktimersoftd/323:57:463
28632994325,3cyclictest25-21ksoftirqd/121:15:041
137291430,0ptp4l401ktimersoftd/323:19:253
137291430,0ptp4l401ktimersoftd/322:05:003
137291430,0ptp4l401ktimersoftd/319:53:243
137291430,0ptp4l401ktimersoftd/300:31:243
2863399420,22cyclictest16393-21sshd20:56:242
137291420,0ptp4l401ktimersoftd/321:36:253
137291420,0ptp4l401ktimersoftd/319:56:243
137291420,0ptp4l401ktimersoftd/300:20:483
2863399410,28cyclictest447-21gmain00:07:232
2863399410,1cyclictest0-21swapper/222:00:232
28631994137,2cyclictest9-21ksoftirqd/000:35:010
137291410,0ptp4l401ktimersoftd/320:29:173
137291410,0ptp4l401ktimersoftd/300:25:253
137291410,0ptp4l401ktimersoftd/300:04:523
28633994025,12cyclictest33-21ksoftirqd/223:25:122
2863399400,21cyclictest6947-21irqstats19:30:182
2863399400,21cyclictest24718-21kernelversion23:30:202
28631994036,2cyclictest9-21ksoftirqd/023:40:000
28631994035,3cyclictest9-21ksoftirqd/022:40:010
2863399380,23cyclictest24405-21ls00:35:282
2863299389,27cyclictest0-21swapper/119:25:501
28631993833,3cyclictest9-21ksoftirqd/023:35:000
2863399370,23cyclictest31544-21if_err_enp4s023:45:172
2863399370,20cyclictest1314-21snmpd19:55:272
2863299371,33cyclictest10737-21ls21:55:001
28631993733,2cyclictest9-21ksoftirqd/021:40:000
28631993732,3cyclictest9-21ksoftirqd/022:30:000
28631993732,3cyclictest9-21ksoftirqd/019:15:000
28631993732,3cyclictest9-21ksoftirqd/000:30:010
2863399361,21cyclictest3276-21sshd21:39:242
2863399360,19cyclictest11128-21chrt23:00:272
2863399360,19cyclictest11128-21chrt23:00:262
28632993630,4cyclictest25-21ksoftirqd/123:08:331
28632993630,4cyclictest25-21ksoftirqd/121:45:041
28631993632,3cyclictest9-21ksoftirqd/021:00:210
28631993632,2cyclictest9-21ksoftirqd/020:50:000
28631993632,2cyclictest9-21ksoftirqd/019:40:500
28631993632,2cyclictest9-21ksoftirqd/000:25:010
42098350,13rtkit-daemon1574-21apache219:13:471
2863499354,27cyclictest12622-21grep21:55:273
28634993529,4cyclictest41-21ksoftirqd/323:21:243
28634993529,4cyclictest41-21ksoftirqd/322:51:013
28634993529,4cyclictest41-21ksoftirqd/322:24:223
28634993529,4cyclictest41-21ksoftirqd/319:27:433
28634993529,3cyclictest41-21ksoftirqd/321:30:113
2863499352,29cyclictest6003-21taskset19:30:053
28633993529,4cyclictest33-21ksoftirqd/219:28:412
2863399350,19cyclictest8953-21sshd22:56:292
2863399350,19cyclictest32106-21irqstats21:30:192
28632993529,4cyclictest25-21ksoftirqd/122:22:191
28632993529,4cyclictest25-21ksoftirqd/121:35:531
28632993529,4cyclictest25-21ksoftirqd/120:33:351
28632993527,5cyclictest25-21ksoftirqd/120:40:331
2863299350,2cyclictest121rcu_preempt22:56:561
28631993530,3cyclictest9-21ksoftirqd/020:35:000
28631993529,4cyclictest9-21ksoftirqd/023:03:140
28631993529,4cyclictest9-21ksoftirqd/023:03:130
28631993529,4cyclictest9-21ksoftirqd/021:10:290
125202350,4sleep00-21swapper/023:05:190
28634993429,3cyclictest41-21ksoftirqd/321:18:333
28634993429,3cyclictest41-21ksoftirqd/320:20:123
28634993429,3cyclictest41-21ksoftirqd/319:23:253
28634993428,4cyclictest41-21ksoftirqd/323:14:473
28634993428,4cyclictest41-21ksoftirqd/319:41:183
2863399345,27cyclictest0-21swapper/222:35:012
28633993428,4cyclictest33-21ksoftirqd/223:16:202
28633993428,4cyclictest33-21ksoftirqd/222:14:302
28633993428,4cyclictest33-21ksoftirqd/222:14:302
28633993428,4cyclictest33-21ksoftirqd/221:49:462
28633993428,4cyclictest33-21ksoftirqd/220:39:542
28633993428,4cyclictest33-21ksoftirqd/220:06:162
28632993428,4cyclictest25-21ksoftirqd/123:11:581
28632993428,4cyclictest25-21ksoftirqd/121:05:421
28632993428,4cyclictest25-21ksoftirqd/120:56:061
28632993428,4cyclictest25-21ksoftirqd/120:48:481
28632993428,4cyclictest25-21ksoftirqd/100:31:331
28632993428,4cyclictest25-21ksoftirqd/100:19:251
28632993428,3cyclictest25-21ksoftirqd/123:55:121
28632993427,5cyclictest25-21ksoftirqd/123:28:591
28632993427,5cyclictest25-21ksoftirqd/122:42:291
28632993427,5cyclictest25-21ksoftirqd/120:21:451
28632993427,5cyclictest25-21ksoftirqd/119:59:371
28632993427,4cyclictest25-21ksoftirqd/123:20:001
28632993427,4cyclictest25-21ksoftirqd/122:09:311
28632993427,4cyclictest25-21ksoftirqd/122:09:311
28632993427,4cyclictest25-21ksoftirqd/121:11:021
28631993430,3cyclictest9-21ksoftirqd/020:40:010
28631993430,3cyclictest9-21ksoftirqd/020:25:000
28631993429,3cyclictest9-21ksoftirqd/023:20:010
28631993429,3cyclictest9-21ksoftirqd/020:15:170
28631993429,3cyclictest9-21ksoftirqd/020:10:190
28631993428,4cyclictest9-21ksoftirqd/023:45:020
28631993428,4cyclictest9-21ksoftirqd/022:18:240
28631993428,4cyclictest9-21ksoftirqd/020:53:000
28631993427,5cyclictest9-21ksoftirqd/021:31:120
28634993327,4cyclictest41-21ksoftirqd/323:27:323
28634993327,4cyclictest41-21ksoftirqd/323:02:573
28634993327,4cyclictest41-21ksoftirqd/323:02:573
28634993327,4cyclictest41-21ksoftirqd/321:24:003
28634993327,4cyclictest41-21ksoftirqd/320:50:023
28634993327,4cyclictest41-21ksoftirqd/320:39:073
28633993327,4cyclictest33-21ksoftirqd/223:59:362
28633993327,4cyclictest33-21ksoftirqd/222:45:072
28633993327,4cyclictest33-21ksoftirqd/222:37:372
28633993327,4cyclictest33-21ksoftirqd/221:18:172
28633993327,4cyclictest33-21ksoftirqd/219:52:452
28633993312,18cyclictest1285-21runrttasks22:25:192
2863399330,21cyclictest15390-21taskset19:48:112
2863399330,21cyclictest12903-21latency23:05:212
28632993328,3cyclictest25-21ksoftirqd/122:02:391
28632993327,4cyclictest25-21ksoftirqd/123:43:391
28632993327,4cyclictest25-21ksoftirqd/122:31:301
28632993327,4cyclictest25-21ksoftirqd/122:27:291
28632993326,5cyclictest25-21ksoftirqd/122:47:341
28632993326,5cyclictest25-21ksoftirqd/100:38:511
28632993326,5cyclictest25-21ksoftirqd/100:03:451
28631993330,2cyclictest9-21ksoftirqd/020:40:210
28631993329,3cyclictest9-21ksoftirqd/021:15:210
28631993329,3cyclictest9-21ksoftirqd/019:15:200
28631993329,2cyclictest9-21ksoftirqd/023:30:010
28631993328,3cyclictest9-21ksoftirqd/022:40:200
28631993328,3cyclictest9-21ksoftirqd/021:10:000
28631993328,3cyclictest9-21ksoftirqd/019:50:200
28631993328,3cyclictest9-21ksoftirqd/019:25:110
28631993327,4cyclictest9-21ksoftirqd/020:59:020
28631993327,4cyclictest9-21ksoftirqd/019:47:450
28631993327,4cyclictest9-21ksoftirqd/000:38:580
28631993327,4cyclictest9-21ksoftirqd/000:19:080
28631993327,4cyclictest9-21ksoftirqd/000:08:000
28631993327,3cyclictest9-21ksoftirqd/020:10:000
28634993227,3cyclictest41-21ksoftirqd/323:53:013
28634993227,3cyclictest41-21ksoftirqd/322:26:463
28634993226,4cyclictest41-21ksoftirqd/320:59:373
28634993226,4cyclictest41-21ksoftirqd/320:11:163
28634993226,4cyclictest41-21ksoftirqd/300:12:413
2863399323,17cyclictest0-21swapper/221:20:202
28633993226,4cyclictest33-21ksoftirqd/220:18:162
28632993227,3cyclictest25-21ksoftirqd/120:05:001
28632993227,3cyclictest25-21ksoftirqd/119:49:251
28632993226,4cyclictest25-21ksoftirqd/123:22:121
28632993226,4cyclictest25-21ksoftirqd/123:00:101
28632993226,4cyclictest25-21ksoftirqd/123:00:101
28632993226,4cyclictest25-21ksoftirqd/122:14:251
28632993226,4cyclictest25-21ksoftirqd/122:14:251
28632993226,4cyclictest25-21ksoftirqd/121:42:271
28632993226,4cyclictest25-21ksoftirqd/119:41:081
28632993226,4cyclictest25-21ksoftirqd/119:16:171
28632993226,3cyclictest25-21ksoftirqd/123:36:121
28631993228,2cyclictest860-21sed20:25:210
28631993227,3cyclictest9-21ksoftirqd/021:40:060
28631993226,4cyclictest9-21ksoftirqd/021:49:200
28631993226,4cyclictest9-21ksoftirqd/019:55:390
28631993226,4cyclictest9-21ksoftirqd/000:14:260
28634993126,3cyclictest41-21ksoftirqd/321:47:563
28634993125,4cyclictest41-21ksoftirqd/323:41:523
28634993125,4cyclictest41-21ksoftirqd/323:34:303
28634993125,4cyclictest41-21ksoftirqd/320:19:223
28634993125,4cyclictest41-21ksoftirqd/320:09:303
28634993125,4cyclictest41-21ksoftirqd/319:48:323
28633993126,3cyclictest33-21ksoftirqd/221:57:242
28633993125,4cyclictest33-21ksoftirqd/223:43:232
28633993125,4cyclictest33-21ksoftirqd/221:12:402
28632993126,3cyclictest25-21ksoftirqd/121:03:521
28632993125,4cyclictest25-21ksoftirqd/123:52:251
28632993125,4cyclictest25-21ksoftirqd/122:17:381
28632993125,4cyclictest25-21ksoftirqd/120:37:121
28632993125,4cyclictest25-21ksoftirqd/119:31:501
28631993126,3cyclictest9-21ksoftirqd/022:54:030
28631993126,3cyclictest9-21ksoftirqd/022:07:580
28631993126,3cyclictest9-21ksoftirqd/022:07:580
28631993125,4cyclictest9-21ksoftirqd/022:48:100
28631993125,4cyclictest9-21ksoftirqd/021:54:060
28631993125,3cyclictest9-21ksoftirqd/019:30:180
28634993025,3cyclictest41-21ksoftirqd/323:40:003
28634993025,3cyclictest41-21ksoftirqd/322:18:233
28634993025,3cyclictest41-21ksoftirqd/320:42:163
28634993024,4cyclictest41-21ksoftirqd/322:47:363
28633993025,3cyclictest33-21ksoftirqd/222:05:472
28633993025,3cyclictest33-21ksoftirqd/222:05:472
28633993025,3cyclictest33-21ksoftirqd/221:07:592
28633993025,3cyclictest33-21ksoftirqd/220:13:342
28633993024,4cyclictest33-21ksoftirqd/223:52:382
28633993024,4cyclictest33-21ksoftirqd/221:01:402
28633993024,4cyclictest33-21ksoftirqd/220:25:382
28633993024,4cyclictest33-21ksoftirqd/219:43:342
28633993024,4cyclictest33-21ksoftirqd/219:39:002
28633993024,4cyclictest33-21ksoftirqd/200:03:252
2863399302,25cyclictest9105-21sed20:40:292
2863399301,26cyclictest218-21systemd-journal00:15:492
2863399301,25cyclictest14296-21apt-get23:10:142
28633993012,15cyclictest20585-21grep00:30:152
2863399300,27cyclictest8849-21df_abs21:50:172
2863399300,18cyclictest13113-21irqstats20:50:232
28632993024,4cyclictest25-21ksoftirqd/123:46:441
28632993024,4cyclictest25-21ksoftirqd/121:33:171
28632993024,4cyclictest25-21ksoftirqd/121:24:261
28632993024,4cyclictest25-21ksoftirqd/119:20:461
28631993026,3cyclictest9-21ksoftirqd/020:00:180
28631993026,3cyclictest9-21ksoftirqd/019:35:210
28631993025,3cyclictest9-21ksoftirqd/023:24:320
28631993025,3cyclictest9-21ksoftirqd/022:35:010
28631993024,4cyclictest9-21ksoftirqd/023:13:170
28631993024,4cyclictest9-21ksoftirqd/000:00:460
28634992925,3cyclictest41-21ksoftirqd/321:11:083
28634992924,3cyclictest41-21ksoftirqd/321:42:103
28634992923,4cyclictest41-21ksoftirqd/319:17:593
2863499292,22cyclictest19315-21apt-get21:05:153
28634992911,16cyclictest32597-21cat22:40:153
28634992911,15cyclictest8624-21df_abs00:05:163
28634992910,16cyclictest3349-21irqrtprio20:30:223
2863499290,3cyclictest30001-21cpu22:35:143
28633992924,3cyclictest33-21ksoftirqd/220:24:342
28633992924,3cyclictest33-21ksoftirqd/200:27:352
28633992923,4cyclictest33-21ksoftirqd/200:24:042
2863399291,25cyclictest436-21rs:main2
28632992924,3cyclictest25-21ksoftirqd/120:16:061
2863499286,19cyclictest18347-21mailstats21:00:273
28634992822,4cyclictest41-21ksoftirqd/319:35:483
2863499282,22cyclictest22766-21apt-get00:35:153
2863499282,22cyclictest0-21swapper/321:25:133
2863499281,24cyclictest13384-21cat23:05:263
2863499280,1cyclictest0-21swapper/323:49:233
28633992823,3cyclictest33-21ksoftirqd/222:21:442
28633992822,4cyclictest33-21ksoftirqd/223:35:012
2863399281,24cyclictest1314-21snmpd22:15:192
28632992823,3cyclictest25-21ksoftirqd/100:10:421
28632992823,3cyclictest25-21ksoftirqd/100:09:311
28632992822,4cyclictest25-21ksoftirqd/119:53:401
28631992823,3cyclictest9-21ksoftirqd/023:55:130
28631992823,3cyclictest9-21ksoftirqd/022:55:180
28631992823,3cyclictest9-21ksoftirqd/021:55:130
2863499274,20cyclictest29087-21fschecks_count19:10:173
28634992722,3cyclictest41-21ksoftirqd/320:48:473
2863499272,20cyclictest18099-21apt-get22:10:133
2863499272,20cyclictest18099-21apt-get22:10:133
2863499271,23cyclictest1314-21snmpd20:00:123
2863499271,22cyclictest17963-21ls22:10:003
2863499271,22cyclictest17963-21ls22:10:003
2863399279,15cyclictest20779-21expr20:00:132
2863399279,15cyclictest11612-21irqrtprio00:10:212
28633992723,3cyclictest33-21ksoftirqd/222:51:242
28633992722,3cyclictest33-21ksoftirqd/219:15:012
2863399271,23cyclictest218-21systemd-journal23:20:342
28633992710,15cyclictest293082chrt19:10:202
2863399270,24cyclictest218-21systemd-journal20:45:012
28632992722,3cyclictest25-21ksoftirqd/100:21:251
28632992720,5cyclictest25-21ksoftirqd/119:36:271
28631992724,2cyclictest9-21ksoftirqd/022:20:250
28631992722,3cyclictest9-21ksoftirqd/023:54:160
28631992721,4cyclictest9-21ksoftirqd/023:42:460
28631992721,4cyclictest9-21ksoftirqd/021:27:420
28631992721,4cyclictest9-21ksoftirqd/019:24:220
28633992621,3cyclictest33-21ksoftirqd/220:34:492
28632992621,3cyclictest25-21ksoftirqd/120:06:551
28631992621,3cyclictest9-21ksoftirqd/022:10:140
28631992621,3cyclictest9-21ksoftirqd/022:10:140
28631992621,3cyclictest9-21ksoftirqd/021:25:010
28632992520,3cyclictest25-21ksoftirqd/100:28:271
28632992519,4cyclictest25-21ksoftirqd/123:33:321
28632992519,4cyclictest25-21ksoftirqd/120:28:021
28631992420,3cyclictest9-21ksoftirqd/022:00:250
28632992318,3cyclictest25-21ksoftirqd/121:59:121
28632992318,3cyclictest25-21ksoftirqd/120:14:501
28632992116,3cyclictest25-21ksoftirqd/122:51:201
28632992015,3cyclictest25-21ksoftirqd/120:50:381
28632991813,3cyclictest31107-21cat21:30:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional