You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-22 - 17:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot3.osadl.org (updated Sat Mar 22, 2025 12:45:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
262952232168,52sleep30-21swapper/307:09:363
261612207173,23sleep10-21swapper/107:08:221
260802204170,23sleep20-21swapper/207:07:192
262922201167,23sleep00-21swapper/007:09:340
1371918363,11phc2sys0-21swapper/307:10:013
293552740,2sleep10-21swapper/108:25:201
133002630,4sleep013301-21latency_hist07:50:010
2662699613,56cyclictest30694-21kworker/1:107:40:461
2662699599,8cyclictest13469-21mailstats10:20:271
2662699599,17cyclictest7721-21cut08:50:181
70822580,3sleep07086-21irqrtprio08:50:140
139282570,5sleep22662799cyclictest11:50:002
2662699559,11cyclictest22940-21cat12:10:121
2662699488,8cyclictest16777-21users10:30:231
2662699469,8cyclictest11560-21diskstats09:00:161
2662699458,8cyclictest2154-21df09:55:151
136991430,0ptp4l401ktimersoftd/311:45:413
136991420,0ptp4l401ktimersoftd/311:15:153
2662699418,8cyclictest9814-21meminfo08:55:221
136991410,1ptp4l401ktimersoftd/311:44:293
136991410,0ptp4l401ktimersoftd/309:41:003
59298400,9rtkit-daemon0-21swapper/308:57:563
59298400,13rtkit-daemon566-21irqbalance09:25:060
2662699408,8cyclictest4592-21df10:00:151
2662699407,8cyclictest4246-21memory11:20:221
2662699407,8cyclictest4246-21memory11:20:221
2662699398,8cyclictest24843-21irqstats09:30:151
2662699395,19cyclictest0-21swapper/109:05:131
2662599392,35cyclictest17164-21cron10:35:000
26627993832,4cyclictest33-21ksoftirqd/209:53:352
26627993832,4cyclictest33-21ksoftirqd/209:30:472
26626993810,16cyclictest18197-21cut10:35:191
26625993832,4cyclictest9-21ksoftirqd/008:20:000
59298370,11rtkit-daemon566-21irqbalance09:24:060
26625993712,22cyclictest11481-21sed11:40:150
59298360,11rtkit-daemon566-21irqbalance09:35:050
26628993630,4cyclictest41-21ksoftirqd/312:01:083
26627993630,4cyclictest33-21ksoftirqd/210:26:512
26627993630,4cyclictest33-21ksoftirqd/209:35:452
26627993630,4cyclictest33-21ksoftirqd/209:23:222
26627993630,4cyclictest33-21ksoftirqd/208:09:532
2662699367,11cyclictest0-21swapper/107:25:201
26625993631,3cyclictest4034-21cron10:00:010
26625993613,20cyclictest29664-21iostat07:15:180
136991360,0ptp4l401ktimersoftd/309:15:243
26628993530,3cyclictest41-21ksoftirqd/311:09:333
26628993530,3cyclictest41-21ksoftirqd/310:28:373
26628993530,3cyclictest41-21ksoftirqd/309:50:003
26628993529,4cyclictest41-21ksoftirqd/310:45:183
26628993529,4cyclictest41-21ksoftirqd/310:41:263
26628993529,4cyclictest41-21ksoftirqd/308:24:143
26628993529,4cyclictest41-21ksoftirqd/308:01:513
26628993529,4cyclictest41-21ksoftirqd/307:27:213
26627993530,3cyclictest33-21ksoftirqd/210:08:092
26627993530,3cyclictest33-21ksoftirqd/208:58:142
26627993529,4cyclictest33-21ksoftirqd/211:27:572
26627993529,4cyclictest33-21ksoftirqd/209:26:432
26627993529,4cyclictest33-21ksoftirqd/208:48:522
26627993529,4cyclictest33-21ksoftirqd/208:29:092
26626993530,3cyclictest25-21ksoftirqd/109:42:591
26626993529,4cyclictest25-21ksoftirqd/108:03:221
26625993529,4cyclictest9-21ksoftirqd/011:30:290
26625993529,4cyclictest9-21ksoftirqd/010:53:220
26625993511,3cyclictest20163-21sed12:00:180
26628993427,5cyclictest41-21ksoftirqd/310:16:173
26627993429,3cyclictest33-21ksoftirqd/209:02:512
26627993428,4cyclictest33-21ksoftirqd/212:02:162
26627993428,4cyclictest33-21ksoftirqd/211:30:512
26627993428,4cyclictest33-21ksoftirqd/210:19:062
26627993428,4cyclictest33-21ksoftirqd/210:01:042
26627993428,4cyclictest33-21ksoftirqd/209:47:452
26627993428,4cyclictest33-21ksoftirqd/208:14:542
2662699344,9cyclictest0-21swapper/107:35:151
26626993427,4cyclictest25-21ksoftirqd/109:23:521
2662599349,3cyclictest18498-21sed10:35:200
26625993428,4cyclictest9-21ksoftirqd/011:52:020
26625993428,4cyclictest9-21ksoftirqd/011:36:270
26625993428,4cyclictest9-21ksoftirqd/009:54:410
26625993428,4cyclictest9-21ksoftirqd/008:41:280
26625993427,5cyclictest9-21ksoftirqd/010:15:510
26625993414,17cyclictest13545-21apt-get09:05:100
136991340,0ptp4l401ktimersoftd/307:35:123
26628993328,3cyclictest41-21ksoftirqd/312:25:003
26628993327,4cyclictest41-21ksoftirqd/310:09:243
26628993327,4cyclictest41-21ksoftirqd/310:02:183
26628993327,4cyclictest41-21ksoftirqd/309:31:133
26628993327,4cyclictest41-21ksoftirqd/308:54:063
26628993327,4cyclictest41-21ksoftirqd/307:41:323
26628993327,4cyclictest41-21ksoftirqd/307:21:193
26628993326,5cyclictest41-21ksoftirqd/312:15:003
26627993327,4cyclictest33-21ksoftirqd/212:38:462
26627993327,4cyclictest33-21ksoftirqd/212:30:162
26627993327,4cyclictest33-21ksoftirqd/212:13:222
26627993327,4cyclictest33-21ksoftirqd/211:58:182
26627993327,4cyclictest33-21ksoftirqd/211:52:322
26627993327,4cyclictest33-21ksoftirqd/211:38:402
26627993327,4cyclictest33-21ksoftirqd/211:11:252
26627993327,4cyclictest33-21ksoftirqd/210:48:272
26627993327,4cyclictest33-21ksoftirqd/210:21:512
26627993316,15cyclictest33-21ksoftirqd/208:35:002
2662699339,7cyclictest16526-21iostat_ios07:55:201
2662699335,17cyclictest0-21swapper/112:20:191
26626993327,4cyclictest25-21ksoftirqd/111:43:501
26626993327,4cyclictest25-21ksoftirqd/110:06:171
26626993327,4cyclictest25-21ksoftirqd/109:28:131
26626993326,4cyclictest25-21ksoftirqd/112:26:011
2662599339,4cyclictest16002-21grep07:55:140
2662599333,10cyclictest25262-21sort10:55:140
26625993327,4cyclictest9-21ksoftirqd/012:18:580
26625993327,4cyclictest9-21ksoftirqd/009:32:220
26625993327,4cyclictest9-21ksoftirqd/007:29:340
26625993312,18cyclictest8864-21timerwakeupswit10:10:220
26625993311,3cyclictest19474-21phc2sys-jitter08:00:230
26625993310,3cyclictest20311-21seq09:18:100
136991330,12ptp4l401ktimersoftd/307:55:153
136991330,0ptp4l401ktimersoftd/311:10:013
26628993227,3cyclictest41-21ksoftirqd/311:39:593
26628993227,3cyclictest41-21ksoftirqd/309:13:323
26628993227,3cyclictest41-21ksoftirqd/308:47:573
26628993226,4cyclictest41-21ksoftirqd/312:25:433
26628993226,4cyclictest41-21ksoftirqd/311:25:053
26628993226,4cyclictest41-21ksoftirqd/309:07:583
26627993227,3cyclictest33-21ksoftirqd/208:37:542
26627993227,3cyclictest33-21ksoftirqd/207:14:502
26627993226,4cyclictest33-21ksoftirqd/211:40:092
26627993226,4cyclictest33-21ksoftirqd/211:04:102
26627993226,4cyclictest33-21ksoftirqd/210:34:342
26627993226,4cyclictest33-21ksoftirqd/209:57:232
26627993226,4cyclictest33-21ksoftirqd/209:07:362
26627993226,4cyclictest33-21ksoftirqd/208:51:412
26627993226,4cyclictest33-21ksoftirqd/207:54:482
26627993226,4cyclictest33-21ksoftirqd/207:29:002
2662699329,7cyclictest19477-21phc2sys09:15:231
2662699326,16cyclictest0-21swapper/108:40:251
2662699326,16cyclictest0-21swapper/108:10:221
2662699324,9cyclictest0-21swapper/110:15:161
26626993227,3cyclictest25-21ksoftirqd/111:35:101
26626993226,4cyclictest25-21ksoftirqd/107:30:121
2662599327,4cyclictest10931-21sendmail09:00:010
26625993227,3cyclictest9-21ksoftirqd/007:10:410
26625993226,4cyclictest9-21ksoftirqd/010:27:230
26625993226,4cyclictest9-21ksoftirqd/010:23:050
26625993226,4cyclictest9-21ksoftirqd/008:23:350
26625993224,5cyclictest9-21ksoftirqd/011:25:000
26625993224,5cyclictest9-21ksoftirqd/011:25:000
136991320,0ptp4l401ktimersoftd/307:50:193
26628993127,3cyclictest41-21ksoftirqd/311:55:593
26628993127,3cyclictest41-21ksoftirqd/309:37:143
26628993126,3cyclictest41-21ksoftirqd/312:19:003
26628993126,3cyclictest41-21ksoftirqd/308:11:303
26628993125,4cyclictest41-21ksoftirqd/310:50:593
26628993125,4cyclictest41-21ksoftirqd/310:33:273
26628993124,5cyclictest41-21ksoftirqd/308:30:243
26627993125,4cyclictest33-21ksoftirqd/212:25:382
26627993125,4cyclictest33-21ksoftirqd/211:05:282
26627993125,4cyclictest33-21ksoftirqd/209:42:072
26627993125,4cyclictest33-21ksoftirqd/208:42:422
2662799310,3cyclictest121rcu_preempt10:40:162
2662699318,11cyclictest25018-21date10:55:001
2662699315,16cyclictest0-21swapper/110:10:141
26626993125,4cyclictest25-21ksoftirqd/112:36:441
26626993125,4cyclictest25-21ksoftirqd/112:08:441
26626993125,4cyclictest25-21ksoftirqd/111:05:261
2662599318,3cyclictest5415-21timerwakeupswit07:30:290
2662599318,3cyclictest19857-21cpuspeed10:40:120
2662599317,4cyclictest296072chrt12:26:240
2662599312,6cyclictest16750-21apt-get11:55:150
26625993126,3cyclictest9-21ksoftirqd/012:37:530
26625993126,3cyclictest9-21ksoftirqd/009:02:210
26625993125,4cyclictest9-21ksoftirqd/011:03:000
26625993125,4cyclictest9-21ksoftirqd/010:06:320
26625993125,4cyclictest9-21ksoftirqd/010:03:590
26625993125,4cyclictest9-21ksoftirqd/009:47:430
26625993124,5cyclictest13917-21sh11:50:000
136991310,0ptp4l401ktimersoftd/312:32:123
136991310,0ptp4l401ktimersoftd/308:25:263
26628993025,3cyclictest41-21ksoftirqd/309:53:313
26628993024,4cyclictest41-21ksoftirqd/312:37:023
26628993024,4cyclictest41-21ksoftirqd/310:21:113
26628993024,4cyclictest41-21ksoftirqd/309:28:163
26628993024,4cyclictest41-21ksoftirqd/307:31:463
26628993023,4cyclictest41-21ksoftirqd/310:10:143
26627993024,4cyclictest33-21ksoftirqd/212:05:292
26627993024,4cyclictest33-21ksoftirqd/210:55:442
26627993024,4cyclictest33-21ksoftirqd/210:35:422
26627993024,4cyclictest33-21ksoftirqd/208:17:232
2662699309,7cyclictest25403-21iostat_ios12:15:181
2662699308,8cyclictest12657-21ssh07:45:321
2662699307,9cyclictest0-21swapper/109:50:191
2662699305,15cyclictest0-21swapper/111:50:101
2662699304,9cyclictest0-21swapper/107:20:211
26626993025,3cyclictest25-21ksoftirqd/110:45:001
26626993025,3cyclictest25-21ksoftirqd/110:26:061
26626993024,4cyclictest25-21ksoftirqd/111:32:011
26626993024,4cyclictest25-21ksoftirqd/110:48:451
2662599308,3cyclictest16849-21kernelversion09:10:230
2662599306,4cyclictest30718-21sed08:30:000
2662599302,6cyclictest22383-21apt-get12:10:080
26625993025,3cyclictest9-21ksoftirqd/011:05:160
26625993025,3cyclictest9-21ksoftirqd/007:44:590
26625993024,4cyclictest9-21ksoftirqd/012:24:240
26625993024,4cyclictest9-21ksoftirqd/011:10:070
26625993024,4cyclictest9-21ksoftirqd/009:44:530
26625993024,4cyclictest9-21ksoftirqd/008:48:220
26625993024,4cyclictest9-21ksoftirqd/008:05:210
26628992924,3cyclictest41-21ksoftirqd/311:04:533
26628992924,3cyclictest41-21ksoftirqd/310:36:453
26628992924,3cyclictest41-21ksoftirqd/309:25:003
26628992923,4cyclictest41-21ksoftirqd/308:17:363
26627992924,3cyclictest33-21ksoftirqd/211:17:112
26627992924,3cyclictest33-21ksoftirqd/208:20:212
26627992924,3cyclictest33-21ksoftirqd/207:22:562
26627992924,3cyclictest33-21ksoftirqd/207:19:162
26627992923,4cyclictest33-21ksoftirqd/207:38:032
2662699295,10cyclictest0-21swapper/111:55:201
26626992924,3cyclictest25-21ksoftirqd/108:38:001
26626992923,4cyclictest25-21ksoftirqd/111:16:291
26626992923,4cyclictest25-21ksoftirqd/111:12:121
2662599298,3cyclictest32367-21latency07:20:200
26625992924,3cyclictest9-21ksoftirqd/012:07:320
26628992822,4cyclictest41-21ksoftirqd/308:37:583
26628992822,4cyclictest41-21ksoftirqd/307:19:473
26627992823,3cyclictest33-21ksoftirqd/207:41:472
26626992823,3cyclictest25-21ksoftirqd/108:48:451
26626992822,4cyclictest25-21ksoftirqd/112:02:421
26626992822,4cyclictest25-21ksoftirqd/111:45:261
2662599288,2cyclictest101ktimersoftd/008:35:150
26625992823,3cyclictest9-21ksoftirqd/007:39:100
2662599281,5cyclictest5965-21latency_hist11:30:000
136991280,0ptp4l401ktimersoftd/311:52:103
136991280,0ptp4l401ktimersoftd/309:02:403
136991280,0ptp4l401ktimersoftd/308:43:193
26628992722,3cyclictest41-21ksoftirqd/307:45:433
26628992721,4cyclictest41-21ksoftirqd/312:05:333
26627992722,3cyclictest33-21ksoftirqd/212:21:372
26627992722,3cyclictest33-21ksoftirqd/212:16:352
26627992722,3cyclictest33-21ksoftirqd/211:24:302
26627992722,3cyclictest33-21ksoftirqd/211:24:302
26627992722,3cyclictest33-21ksoftirqd/207:45:132
26627992721,4cyclictest33-21ksoftirqd/210:12:552
26627992721,4cyclictest33-21ksoftirqd/209:13:592
26626992722,3cyclictest25-21ksoftirqd/107:15:111
26626992721,4cyclictest25-21ksoftirqd/108:15:041
26626992720,4cyclictest25610-21grep10:55:161
2662599273,4cyclictest30134-21apt-get12:30:080
26625992722,3cyclictest9-21ksoftirqd/008:34:050
26625992722,3cyclictest9-21ksoftirqd/007:50:490
26628992621,3cyclictest41-21ksoftirqd/311:31:283
26628992621,3cyclictest41-21ksoftirqd/310:59:553
26627992620,4cyclictest33-21ksoftirqd/207:55:552
26627992620,4cyclictest33-21ksoftirqd/207:30:162
26626992621,3cyclictest25-21ksoftirqd/111:27:001
26626992620,4cyclictest25-21ksoftirqd/107:13:301
26625992620,3cyclictest9-21ksoftirqd/010:45:010
26628992517,5cyclictest21075-21cat08:10:003
26627992520,3cyclictest33-21ksoftirqd/210:50:412
26627992520,3cyclictest33-21ksoftirqd/208:00:442
2662699253,9cyclictest0-21swapper/112:30:131
2662699253,10cyclictest0-21swapper/109:10:191
26626992519,4cyclictest25-21ksoftirqd/109:36:291
26626992519,4cyclictest25-21ksoftirqd/107:51:381
26628992419,3cyclictest41-21ksoftirqd/309:56:163
26628992418,4cyclictest41-21ksoftirqd/311:24:313
26628992418,4cyclictest41-21ksoftirqd/311:24:303
26627992420,3cyclictest33-21ksoftirqd/209:15:212
26626992419,3cyclictest25-21ksoftirqd/108:34:391
26626992419,3cyclictest25-21ksoftirqd/108:24:401
26626992412,8cyclictest31114-21chrt09:45:591
26625992418,4cyclictest9-21ksoftirqd/008:11:470
26626992313,7cyclictest28002-21/usr/sbin/munin11:00:181
26626992112,6cyclictest181-21jbd2/mmcblk1p1-08:08:001
26625992115,4cyclictest9-21ksoftirqd/011:15:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional