You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 04:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot4.osadl.org (updated Tue May 13, 2025 00:45:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
226422206173,22sleep10-21swapper/119:09:431
225762204170,22sleep20-21swapper/219:08:512
223782204169,24sleep00-21swapper/019:06:150
223242202168,23sleep30-21swapper/319:05:343
2266728160,11sleep00-21swapper/019:10:010
22958997411,11cyclictest24883-21awk21:25:000
22958997313,12cyclictest17489-21ls21:10:010
22958997111,11cyclictest3274-21fschecks_time20:40:170
22958997016,10cyclictest14444-21dump-pmu-power23:15:000
22958997010,11cyclictest4193-21latency_hist00:00:000
2295899699,11cyclictest0-21swapper/000:05:010
22958996911,17cyclictest22204-21cpuspeed_turbos23:30:140
22958996910,10cyclictest26722-21latency_hist23:40:000
22958996811,11cyclictest6199-21sed19:40:190
22958996811,11cyclictest19156-21cron00:30:000
22958996810,11cyclictest0-21swapper/021:45:000
22962996732,4cyclictest41-21ksoftirqd/320:40:003
2295899678,17cyclictest0-21swapper/022:55:010
22958996715,10cyclictest30243-21cstates21:35:160
2295899668,17cyclictest0-21swapper/019:30:140
2295899667,16cyclictest0-21swapper/022:25:170
22958996614,9cyclictest16406-21cat23:15:310
22958996612,16cyclictest32228-21ls20:35:000
22958996612,16cyclictest17699-21/usr/sbin/munin21:10:140
22958996612,11cyclictest8027-21cpuspeed_turbos19:45:140
22958996611,16cyclictest4325-21grep19:35:240
22958996611,12cyclictest15055-21grep22:10:160
22958996611,11cyclictest11135-21cat22:00:210
22962996538,3cyclictest41-21ksoftirqd/322:50:183
2295899659,19cyclictest0-21swapper/000:05:240
2295899658,17cyclictest0-21swapper/019:25:250
2295899649,18cyclictest0-21swapper/020:00:000
22958996415,9cyclictest22201-21expr00:35:150
22958996413,11cyclictest22986-21df_abs20:15:120
22958996411,10cyclictest5422-21df_abs21:50:130
22958996410,10cyclictest1779-21latency_hist23:55:000
2295899639,16cyclictest0-21swapper/023:40:140
22958996312,10cyclictest18063-21df_abs20:05:160
22958996311,10cyclictest9154-21sed23:00:290
22958996310,17cyclictest13193-21df_abs21:00:140
22958996310,11cyclictest10899-21fschecks_time20:55:150
22962996230,4cyclictest41-21ksoftirqd/320:25:223
22962996230,4cyclictest41-21ksoftirqd/320:25:213
22958996210,16cyclictest28617-21mailstats22:35:270
22958996210,14cyclictest0-21swapper/022:15:200
22958996210,11cyclictest27718-21cstates20:25:160
22958996210,11cyclictest27718-21cstates20:25:160
22958996210,11cyclictest26257-21needreboot20:20:230
22962996130,7cyclictest41-21ksoftirqd/321:05:163
22962996130,4cyclictest41-21ksoftirqd/319:50:003
22962996129,7cyclictest41-21ksoftirqd/319:10:163
22962996127,8cyclictest41-21ksoftirqd/300:20:003
2295899618,16cyclictest25071-21/usr/sbin/munin21:25:250
2295899618,16cyclictest0-21swapper/000:30:210
2295899618,15cyclictest0-21swapper/019:20:250
22958996114,10cyclictest27804-21latency_hist19:20:000
22958996112,16cyclictest29714-21cstates23:45:140
22958996111,10cyclictest11809-21latency_hist00:15:000
22958996110,9cyclictest20345-21cstates20:10:160
22958996110,11cyclictest6714-21mailstats20:45:280
22962996029,4cyclictest41-21ksoftirqd/320:10:223
22962996027,7cyclictest41-21ksoftirqd/300:30:013
2295899609,11cyclictest0-21swapper/000:20:010
22958996011,16cyclictest4122-21sed21:45:270
22958996011,11cyclictest15160-21if_enp1s000:20:180
22958996011,10cyclictest20500-21cstates21:15:160
22958996011,10cyclictest12194-21latency_hist23:10:000
22962995930,6cyclictest41-21ksoftirqd/323:50:003
22962995928,7cyclictest41-21ksoftirqd/323:35:243
22962995928,4cyclictest41-21ksoftirqd/319:25:253
22962995926,8cyclictest41-21ksoftirqd/319:35:003
22962995925,7cyclictest41-21ksoftirqd/321:25:003
22962995925,7cyclictest41-21ksoftirqd/320:35:003
22962995924,7cyclictest41-21ksoftirqd/323:15:283
22962995924,5cyclictest41-21ksoftirqd/321:50:153
22962995924,4cyclictest41-21ksoftirqd/321:25:213
22962995921,9cyclictest41-21ksoftirqd/322:10:013
22958995912,15cyclictest13887-21grep22:05:260
22958995911,11cyclictest26965-21latency_hist22:35:010
22958995910,11cyclictest21302-21switchtime22:20:260
22962995831,5cyclictest41-21ksoftirqd/319:40:013
22962995830,7cyclictest41-21ksoftirqd/321:40:143
22962995828,8cyclictest41-21ksoftirqd/321:10:143
22962995827,8cyclictest41-21ksoftirqd/323:20:193
22962995826,7cyclictest41-21ksoftirqd/322:30:143
22962995826,4cyclictest41-21ksoftirqd/323:10:013
22962995825,9cyclictest41-21ksoftirqd/300:35:303
22962995825,4cyclictest41-21ksoftirqd/320:45:163
22962995821,8cyclictest41-21ksoftirqd/320:40:013
2295899587,10cyclictest0-21swapper/022:50:000
22958995812,17cyclictest32708-21expr20:35:160
22958995812,15cyclictest18841-21sed23:20:290
22962995730,7cyclictest41-21ksoftirqd/322:55:013
22962995729,7cyclictest41-21ksoftirqd/300:10:013
22962995729,4cyclictest41-21ksoftirqd/319:55:193
22962995728,8cyclictest41-21ksoftirqd/323:00:173
22962995726,4cyclictest41-21ksoftirqd/322:10:193
22962995725,7cyclictest41-21ksoftirqd/322:45:153
22962995722,8cyclictest41-21ksoftirqd/300:15:003
22962995720,7cyclictest41-21ksoftirqd/320:55:233
22958995713,11cyclictest20748-21wc23:25:220
22958995712,16cyclictest30233-21basename22:40:180
22958995711,10cyclictest27934-21df_abs21:30:160
22958995710,11cyclictest2411-21cstates22:50:150
22958995710,10cyclictest7709-21cstates21:55:160
22962995629,4cyclictest41-21ksoftirqd/300:30:283
22962995628,6cyclictest41-21ksoftirqd/322:40:153
22962995628,3cyclictest41-21ksoftirqd/323:10:233
22962995627,3cyclictest41-21ksoftirqd/323:28:173
22962995625,8cyclictest41-21ksoftirqd/323:50:203
22962995625,3cyclictest41-21ksoftirqd/322:05:013
22962995622,7cyclictest41-21ksoftirqd/319:20:273
22962995619,8cyclictest41-21ksoftirqd/321:30:243
22962995619,7cyclictest41-21ksoftirqd/319:15:153
22962995529,8cyclictest41-21ksoftirqd/321:00:263
22962995524,10cyclictest41-21ksoftirqd/321:35:213
22962995518,8cyclictest41-21ksoftirqd/323:30:143
22962995518,8cyclictest41-21ksoftirqd/321:15:273
22958995512,17cyclictest8747-21sed20:50:210
110992550,4sleep10-21swapper/123:05:261
22962995424,7cyclictest41-21ksoftirqd/320:05:013
22962995423,3cyclictest41-21ksoftirqd/321:50:003
22962995419,7cyclictest41-21ksoftirqd/300:20:273
22962995326,7cyclictest41-21ksoftirqd/320:50:183
22962995324,9cyclictest41-21ksoftirqd/319:50:263
22962995324,8cyclictest41-21ksoftirqd/320:15:203
22962995320,4cyclictest41-21ksoftirqd/323:55:163
22962995319,9cyclictest41-21ksoftirqd/319:40:233
22959995318,7cyclictest25-21ksoftirqd/123:40:011
22962995224,6cyclictest41-21ksoftirqd/323:40:153
22959995220,8cyclictest25-21ksoftirqd/119:30:001
22958995214,15cyclictest15451-21wc20:00:160
22962995127,7cyclictest41-21ksoftirqd/322:20:203
22962995125,7cyclictest41-21ksoftirqd/320:20:153
22962995117,8cyclictest41-21ksoftirqd/300:00:153
22962995017,8cyclictest41-21ksoftirqd/320:00:183
22962995017,7cyclictest41-21ksoftirqd/322:15:243
22959995017,7cyclictest25-21ksoftirqd/121:45:001
22959995012,4cyclictest25-21ksoftirqd/120:30:241
22959994920,7cyclictest25-21ksoftirqd/119:30:171
22959994917,4cyclictest25-21ksoftirqd/123:55:251
22962994821,8cyclictest41-21ksoftirqd/322:35:253
22959994820,8cyclictest25-21ksoftirqd/121:45:291
22959994819,8cyclictest25-21ksoftirqd/121:20:211
22959994718,3cyclictest25-21ksoftirqd/123:15:001
22959994716,4cyclictest25-21ksoftirqd/121:10:021
22959994715,7cyclictest25-21ksoftirqd/123:45:201
22959994711,4cyclictest25-21ksoftirqd/121:50:221
22958994712,18cyclictest12687-21latency_hist19:55:000
2296299466,6cyclictest41-21ksoftirqd/321:55:293
22962994612,7cyclictest41-21ksoftirqd/322:29:393
2295999466,9cyclictest25-21ksoftirqd/100:35:011
22959994617,8cyclictest25-21ksoftirqd/100:10:161
22959994616,8cyclictest25-21ksoftirqd/120:50:161
22959994615,5cyclictest25-21ksoftirqd/100:05:271
22959994615,3cyclictest25-21ksoftirqd/120:50:001
22959994612,4cyclictest25-21ksoftirqd/100:35:261
2295999454,2cyclictest22020-21awk22:25:001
2295999454,2cyclictest121rcu_preempt21:55:201
22959994516,8cyclictest25-21ksoftirqd/122:40:221
22959994513,4cyclictest25-21ksoftirqd/119:45:161
22959994510,8cyclictest25-21ksoftirqd/100:15:251
22959994427,3cyclictest25-21ksoftirqd/120:00:281
22959994416,7cyclictest25-21ksoftirqd/121:35:211
22959994415,4cyclictest25-21ksoftirqd/100:25:011
22959994413,4cyclictest25-21ksoftirqd/122:15:171
22959994411,4cyclictest25-21ksoftirqd/123:00:011
2295999438,9cyclictest25-21ksoftirqd/123:35:011
2295999438,9cyclictest25-21ksoftirqd/122:45:201
2295999436,8cyclictest25-21ksoftirqd/120:40:151
22959994330,3cyclictest25-21ksoftirqd/122:35:261
22959994315,7cyclictest25-21ksoftirqd/120:10:011
22959994229,4cyclictest25-21ksoftirqd/122:15:001
2295999422,2cyclictest121rcu_preempt20:20:001
22959994215,7cyclictest25-21ksoftirqd/123:25:261
22959994214,7cyclictest25-21ksoftirqd/119:40:161
2295999421,3cyclictest121rcu_preempt22:00:131
22959994211,5cyclictest25-21ksoftirqd/121:30:141
22959994210,4cyclictest25-21ksoftirqd/119:20:171
2295999416,11cyclictest25-21ksoftirqd/122:25:161
22959994113,8cyclictest25-21ksoftirqd/123:00:171
2295999406,9cyclictest25-21ksoftirqd/123:50:141
2295999403,9cyclictest25-21ksoftirqd/120:55:121
22959994014,7cyclictest25-21ksoftirqd/122:50:201
22959994013,8cyclictest25-21ksoftirqd/120:35:221
22959994012,7cyclictest25-21ksoftirqd/121:25:241
2295999400,2cyclictest121rcu_preempt21:00:191
2295999397,4cyclictest25-21ksoftirqd/123:40:211
2295999395,4cyclictest25-21ksoftirqd/121:20:001
22959993925,4cyclictest25-21ksoftirqd/120:00:001
2296199381,35cyclictest8218-21kworker/2:020:05:272
2296199381,35cyclictest537-21kworker/2:122:29:072
2295999388,3cyclictest121rcu_preempt00:25:191
22959993833,3cyclictest25-21ksoftirqd/123:15:171
22959993832,3cyclictest25-21ksoftirqd/122:30:131
22959993832,3cyclictest25-21ksoftirqd/120:10:221
22959993831,4cyclictest25-21ksoftirqd/120:29:121
22959993831,4cyclictest25-21ksoftirqd/120:29:121
22959993812,8cyclictest25-21ksoftirqd/119:35:261
22961993731,4cyclictest33-21ksoftirqd/219:43:042
22961993731,4cyclictest33-21ksoftirqd/200:11:022
22961993730,5cyclictest33-21ksoftirqd/223:30:582
2296199371,35cyclictest537-21kworker/2:121:24:262
22959993732,3cyclictest25-21ksoftirqd/100:05:011
22959993731,3cyclictest25-21ksoftirqd/119:15:241
22959993725,3cyclictest25-21ksoftirqd/121:10:131
2295999372,2cyclictest121rcu_preempt19:15:001
2295999371,2cyclictest121rcu_preempt23:20:161
22961993630,4cyclictest33-21ksoftirqd/221:47:202
22961993630,4cyclictest33-21ksoftirqd/200:01:232
22961993629,4cyclictest33-21ksoftirqd/200:37:222
2296199360,34cyclictest0-21swapper/221:28:472
22959993631,3cyclictest25-21ksoftirqd/120:20:141
22959993630,3cyclictest25-21ksoftirqd/119:50:161
22961993530,3cyclictest33-21ksoftirqd/219:27:562
22961993529,4cyclictest33-21ksoftirqd/223:49:242
22961993529,4cyclictest33-21ksoftirqd/221:52:102
22961993529,4cyclictest33-21ksoftirqd/220:37:362
22959993529,3cyclictest25-21ksoftirqd/122:05:181
22961993428,4cyclictest33-21ksoftirqd/223:35:592
22961993428,4cyclictest33-21ksoftirqd/222:00:082
22961993428,4cyclictest33-21ksoftirqd/220:20:032
22961993427,4cyclictest33-21ksoftirqd/221:15:032
22961993328,3cyclictest33-21ksoftirqd/219:47:332
22961993327,4cyclictest33-21ksoftirqd/222:47:182
22961993327,4cyclictest33-21ksoftirqd/222:09:552
22961993327,4cyclictest33-21ksoftirqd/221:43:352
22961993327,4cyclictest33-21ksoftirqd/220:49:112
22961993327,4cyclictest33-21ksoftirqd/219:24:432
22961993227,3cyclictest33-21ksoftirqd/223:54:122
22961993227,3cyclictest33-21ksoftirqd/221:09:212
22961993227,3cyclictest33-21ksoftirqd/220:41:172
22961993226,4cyclictest33-21ksoftirqd/223:10:372
22961993226,4cyclictest33-21ksoftirqd/222:37:192
22961993226,4cyclictest33-21ksoftirqd/222:13:472
22961993226,4cyclictest33-21ksoftirqd/221:38:262
22961993226,3cyclictest33-21ksoftirqd/200:32:522
22961993225,4cyclictest33-21ksoftirqd/220:52:442
22961993126,3cyclictest33-21ksoftirqd/223:44:572
22961993126,3cyclictest33-21ksoftirqd/220:17:322
22961993126,3cyclictest33-21ksoftirqd/219:37:232
22961993125,4cyclictest33-21ksoftirqd/222:50:032
22961993125,4cyclictest33-21ksoftirqd/220:27:332
22961993125,4cyclictest33-21ksoftirqd/220:27:332
22961993125,4cyclictest33-21ksoftirqd/219:16:202
22961993125,4cyclictest33-21ksoftirqd/219:13:392
22961993124,5cyclictest33-21ksoftirqd/220:56:452
22961993026,3cyclictest33-21ksoftirqd/219:56:392
22961993025,3cyclictest33-21ksoftirqd/223:03:172
22961993025,3cyclictest33-21ksoftirqd/222:15:042
22961993025,3cyclictest33-21ksoftirqd/221:56:572
22961993025,3cyclictest33-21ksoftirqd/221:34:182
22961993025,3cyclictest33-21ksoftirqd/220:11:092
22961993024,4cyclictest33-21ksoftirqd/223:19:432
22961993024,4cyclictest33-21ksoftirqd/222:55:082
22961993024,4cyclictest33-21ksoftirqd/221:02:192
22961993024,4cyclictest33-21ksoftirqd/219:54:312
22961992924,3cyclictest33-21ksoftirqd/222:22:182
22961992924,3cyclictest33-21ksoftirqd/220:02:432
22961992923,4cyclictest33-21ksoftirqd/221:13:062
22961992923,4cyclictest33-21ksoftirqd/200:19:402
22961992823,3cyclictest33-21ksoftirqd/223:23:482
22961992722,3cyclictest33-21ksoftirqd/222:42:472
22961992721,4cyclictest33-21ksoftirqd/222:31:092
22961992621,3cyclictest33-21ksoftirqd/200:27:222
22961992620,3cyclictest33-21ksoftirqd/200:09:582
22961992520,3cyclictest33-21ksoftirqd/223:29:142
22961992520,3cyclictest33-21ksoftirqd/200:23:412
22961992519,4cyclictest33-21ksoftirqd/223:07:582
22961992318,3cyclictest33-21ksoftirqd/223:55:142
22961992317,2cyclictest33-21ksoftirqd/220:30:242
22961992217,3cyclictest33-21ksoftirqd/219:31:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional