You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 10:08
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot4.osadl.org (updated Sun Apr 28, 2024 00:46:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
129891218195,10phc2sys0-21swapper/319:07:513
357721740,3sleep2321999cyclictest21:30:272
1774721440,3sleep0321799cyclictest19:35:300
26702137114,10sleep20-21swapper/219:06:412
2854221120,8sleep1321899cyclictest20:05:311
2627210280,9sleep10-21swapper/119:06:061
28082815,19sleep00-21swapper/019:08:250
3219997865,8cyclictest33-21ksoftirqd/219:55:072
321899726,13cyclictest582-21polkitd19:14:361
3220996861,4cyclictest41-21ksoftirqd/322:45:153
321899684,18cyclictest11823-21/usr/sbin/munin00:05:251
321899684,18cyclictest11823-21/usr/sbin/munin00:05:241
3220996760,4cyclictest41-21ksoftirqd/321:45:093
321899672,19cyclictest9817-21fschecks_count21:45:151
3220996633,7cyclictest41-21ksoftirqd/323:25:273
321899666,12cyclictest19722-21fschecks_count19:45:131
316222660,4sleep1241ktimersoftd/123:40:141
321899636,14cyclictest582-21polkitd20:18:261
321899636,11cyclictest19834-21cat23:15:181
321899632,16cyclictest4051-21df_inode23:50:131
315042630,3sleep031507-21grep21:20:290
3220996255,4cyclictest41-21ksoftirqd/322:40:133
3220996254,5cyclictest41-21ksoftirqd/321:55:113
321899624,12cyclictest17599-21fgrep19:35:291
227882620,2sleep20-21swapper/223:20:282
3220996154,4cyclictest41-21ksoftirqd/323:35:103
3220996154,4cyclictest41-21ksoftirqd/319:30:123
3220996154,4cyclictest41-21ksoftirqd/319:10:073
3220996153,5cyclictest41-21ksoftirqd/323:50:083
3220996153,5cyclictest41-21ksoftirqd/322:25:103
321899613,9cyclictest22344-21python22:10:291
3220996052,5cyclictest41-21ksoftirqd/323:05:123
3220996052,5cyclictest41-21ksoftirqd/300:05:123
3220996052,5cyclictest41-21ksoftirqd/300:05:113
3220995952,4cyclictest41-21ksoftirqd/322:55:113
3220995952,4cyclictest41-21ksoftirqd/321:40:093
3220995936,11cyclictest41-21ksoftirqd/319:25:183
3220995851,4cyclictest41-21ksoftirqd/323:40:113
3220995851,4cyclictest41-21ksoftirqd/322:35:103
3220995851,4cyclictest41-21ksoftirqd/320:10:143
321899581,12cyclictest0-21swapper/120:30:101
321899573,18cyclictest6110-21gdbus23:55:011
129991570,1ptp4l401ktimersoftd/320:06:233
3220995649,4cyclictest41-21ksoftirqd/320:55:203
3220995648,4cyclictest41-21ksoftirqd/323:30:133
321899563,9cyclictest25569-21ssh23:25:291
321899562,9cyclictest23235-21ls19:55:321
3220995549,3cyclictest41-21ksoftirqd/321:30:103
321899556,18cyclictest31120-21chrt20:12:411
321899553,15cyclictest26702-21mailstats21:10:331
321899553,10cyclictest12367-21iostat_ios19:25:201
321899552,18cyclictest18296-21sshd22:03:451
3218995510,42cyclictest27898-21latency_hist22:25:001
3220995447,4cyclictest41-21ksoftirqd/300:20:013
23872540,2sleep30-21swapper/320:20:223
3220995346,4cyclictest41-21ksoftirqd/321:40:003
3220995346,4cyclictest41-21ksoftirqd/320:40:133
3220995345,4cyclictest41-21ksoftirqd/320:00:013
3220995345,4cyclictest41-21ksoftirqd/300:35:013
321899532,9cyclictest20795-21diskstats00:25:171
321899532,10cyclictest15391-21irqstats23:05:231
321899531,8cyclictest10676-21ls22:55:241
3220995245,3cyclictest41-21ksoftirqd/323:15:323
196962520,2sleep30-21swapper/319:45:123
3220995143,5cyclictest41-21ksoftirqd/321:20:083
3220995143,5cyclictest20101-21cron00:25:003
3220995142,5cyclictest41-21ksoftirqd/321:25:123
321799511,3cyclictest131rcu_sched19:14:390
171102510,2sleep30-21swapper/323:10:183
3220995043,3cyclictest41-21ksoftirqd/322:20:013
321899503,9cyclictest7545-21perl21:40:161
321899501,10cyclictest5433-21diskstats22:45:181
3217995042,4cyclictest9-21ksoftirqd/022:35:000
3217995042,4cyclictest9-21ksoftirqd/022:35:000
3220994943,3cyclictest41-21ksoftirqd/323:50:003
3220994943,3cyclictest41-21ksoftirqd/319:20:003
3220994942,3cyclictest41-21ksoftirqd/320:50:363
3220994941,4cyclictest41-21ksoftirqd/323:25:003
3220994941,4cyclictest41-21ksoftirqd/321:00:163
322099493,11cyclictest41-21ksoftirqd/300:05:003
3219994942,4cyclictest33-21ksoftirqd/220:55:002
321899492,9cyclictest10491-21users20:35:341
3220994843,3cyclictest41-21ksoftirqd/320:20:003
3220994843,3cyclictest41-21ksoftirqd/319:39:593
3220994841,3cyclictest41-21ksoftirqd/323:59:593
3220994742,3cyclictest41-21ksoftirqd/323:00:113
3220994741,3cyclictest41-21ksoftirqd/321:10:153
3220994739,4cyclictest41-21ksoftirqd/322:05:003
3220994737,6cyclictest41-21ksoftirqd/322:21:503
322099470,3cyclictest121rcu_preempt20:30:273
3219994741,3cyclictest33-21ksoftirqd/221:06:002
3219994741,3cyclictest33-21ksoftirqd/221:06:002
321899472,14cyclictest14423-21diskstats21:55:161
3217994742,3cyclictest9-21ksoftirqd/023:59:590
3220994641,3cyclictest41-21ksoftirqd/321:15:203
3220994640,4cyclictest41-21ksoftirqd/321:50:273
3220994639,4cyclictest41-21ksoftirqd/322:09:593
322099460,3cyclictest121rcu_preempt20:00:333
3219994640,4cyclictest33-21ksoftirqd/219:14:202
3217994639,4cyclictest9-21ksoftirqd/019:55:010
3220994538,4cyclictest41-21ksoftirqd/320:45:143
3220994538,4cyclictest41-21ksoftirqd/319:25:013
3217994540,3cyclictest9-21ksoftirqd/021:50:010
3217994539,3cyclictest1-21systemd20:18:250
3217994538,4cyclictest9-21ksoftirqd/022:13:450
129991450,1ptp4l401ktimersoftd/300:10:163
3220994438,4cyclictest41-21ksoftirqd/319:45:003
3220994438,4cyclictest41-21ksoftirqd/319:45:003
3220994437,4cyclictest41-21ksoftirqd/320:28:343
3220994437,4cyclictest41-21ksoftirqd/300:25:143
3217994438,3cyclictest9-21ksoftirqd/019:30:000
3217994437,4cyclictest9-21ksoftirqd/023:44:590
3217994437,4cyclictest9-21ksoftirqd/020:49:590
3220994337,3cyclictest41-21ksoftirqd/322:50:323
3220994330,10cyclictest41-21ksoftirqd/320:40:003
322099430,3cyclictest121rcu_preempt22:30:153
322099430,3cyclictest121rcu_preempt22:30:153
3219994336,4cyclictest5834-21kworker/2:322:15:112
3219994336,4cyclictest33-21ksoftirqd/220:26:342
3217994337,3cyclictest9-21ksoftirqd/023:00:010
3217994337,3cyclictest9-21ksoftirqd/000:05:000
3217994336,4cyclictest9-21ksoftirqd/000:10:000
3217994336,4cyclictest9-21ksoftirqd/000:09:590
3217994336,3cyclictest9-21ksoftirqd/021:20:010
194024336,3sleep09-21ksoftirqd/023:45:200
3220994237,3cyclictest41-21ksoftirqd/322:10:143
3219994235,4cyclictest33-21ksoftirqd/219:21:212
3217994236,3cyclictest9-21ksoftirqd/000:15:010
322099419,4cyclictest41-21ksoftirqd/321:05:263
322099419,4cyclictest41-21ksoftirqd/321:05:263
3219994134,4cyclictest33-21ksoftirqd/200:30:142
3219994133,5cyclictest33-21ksoftirqd/220:00:062
3218994134,4cyclictest25-21ksoftirqd/121:10:001
3218994134,4cyclictest25-21ksoftirqd/121:09:591
3217994135,4cyclictest9-21ksoftirqd/023:11:000
3217994134,4cyclictest9-21ksoftirqd/021:30:000
321799411,3cyclictest121rcu_preempt22:25:270
3220994033,3cyclictest41-21ksoftirqd/300:35:143
3217994035,3cyclictest9-21ksoftirqd/021:00:010
3217994034,4cyclictest9-21ksoftirqd/022:45:150
3217994034,3cyclictest9-21ksoftirqd/020:05:000
3217994033,4cyclictest9-21ksoftirqd/023:16:330
3217994032,5cyclictest9-21ksoftirqd/022:01:410
321799400,3cyclictest121rcu_preempt19:15:200
3220993933,4cyclictest41-21ksoftirqd/319:52:113
3219993933,3cyclictest33-21ksoftirqd/222:29:002
3219993933,3cyclictest33-21ksoftirqd/219:54:502
3219993932,4cyclictest33-21ksoftirqd/223:20:002
3219993932,4cyclictest33-21ksoftirqd/222:36:242
3219993932,4cyclictest33-21ksoftirqd/221:00:002
3218993934,3cyclictest25-21ksoftirqd/123:04:591
3218993933,4cyclictest25-21ksoftirqd/123:10:591
3218993932,4cyclictest25-21ksoftirqd/123:47:091
3218993932,4cyclictest25-21ksoftirqd/122:05:401
3218993932,4cyclictest25-21ksoftirqd/120:55:591
3218993932,4cyclictest25-21ksoftirqd/119:24:191
3217993932,4cyclictest9-21ksoftirqd/022:15:120
3217993932,4cyclictest9-21ksoftirqd/022:07:500
3217993932,4cyclictest9-21ksoftirqd/021:34:470
3219993832,4cyclictest33-21ksoftirqd/200:15:232
3219993832,3cyclictest33-21ksoftirqd/222:51:332
3219993831,4cyclictest33-21ksoftirqd/223:52:322
3219993831,4cyclictest33-21ksoftirqd/223:44:582
3219993831,4cyclictest33-21ksoftirqd/223:06:102
3219993831,4cyclictest33-21ksoftirqd/221:47:502
3219993831,4cyclictest33-21ksoftirqd/221:11:432
3219993831,4cyclictest33-21ksoftirqd/221:02:182
3218993832,3cyclictest25-21ksoftirqd/119:17:061
3218993831,4cyclictest25-21ksoftirqd/119:40:371
3218993831,4cyclictest25-21ksoftirqd/119:40:361
3217993833,3cyclictest9-21ksoftirqd/023:05:220
3217993832,3cyclictest9-21ksoftirqd/022:45:000
3217993832,3cyclictest9-21ksoftirqd/000:20:010
3217993831,4cyclictest9-21ksoftirqd/021:59:360
3217993831,4cyclictest9-21ksoftirqd/021:50:550
3217993826,4cyclictest9-21ksoftirqd/020:28:340
123522380,5sleep125-21ksoftirqd/121:50:181
90823729,3sleep125-21ksoftirqd/122:35:191
3219993732,3cyclictest33-21ksoftirqd/222:01:532
3219993731,4cyclictest33-21ksoftirqd/221:24:312
3219993731,3cyclictest33-21ksoftirqd/222:21:572
3219993730,4cyclictest33-21ksoftirqd/222:49:242
3219993730,4cyclictest33-21ksoftirqd/220:21:012
3219993730,4cyclictest33-21ksoftirqd/219:48:582
3219993730,4cyclictest33-21ksoftirqd/200:02:362
3218993730,4cyclictest25-21ksoftirqd/123:35:581
3218993730,4cyclictest25-21ksoftirqd/122:31:371
3218993730,4cyclictest25-21ksoftirqd/122:31:371
3218993730,4cyclictest25-21ksoftirqd/121:34:031
3218993730,4cyclictest25-21ksoftirqd/119:34:511
3217993732,3cyclictest9-21ksoftirqd/020:30:210
3217993731,4cyclictest9-21ksoftirqd/021:08:530
3217993731,4cyclictest9-21ksoftirqd/021:08:520
3217993731,3cyclictest9-21ksoftirqd/021:00:300
3217993731,3cyclictest9-21ksoftirqd/020:25:010
3217993730,4cyclictest9-21ksoftirqd/021:12:060
3217993730,4cyclictest9-21ksoftirqd/020:38:260
3217993730,4cyclictest9-21ksoftirqd/019:24:270
3217993730,4cyclictest9-21ksoftirqd/000:25:000
3219993630,3cyclictest33-21ksoftirqd/221:52:532
3219993630,3cyclictest33-21ksoftirqd/220:42:412
3219993630,3cyclictest33-21ksoftirqd/219:16:382
3219993629,4cyclictest33-21ksoftirqd/223:39:592
3219993629,4cyclictest33-21ksoftirqd/220:36:222
3219993628,6cyclictest33-21ksoftirqd/221:25:182
3219993628,5cyclictest33-21ksoftirqd/220:11:222
321999360,3cyclictest121rcu_preempt00:22:202
3218993630,3cyclictest25-21ksoftirqd/120:20:401
3218993629,4cyclictest25-21ksoftirqd/122:53:131
3218993629,4cyclictest25-21ksoftirqd/121:29:021
3218993629,4cyclictest25-21ksoftirqd/121:01:351
3218993629,4cyclictest25-21ksoftirqd/120:01:331
3218993629,4cyclictest25-21ksoftirqd/100:11:101
3218993628,5cyclictest25-21ksoftirqd/120:47:421
3217993630,4cyclictest9-21ksoftirqd/019:55:330
3217993630,3cyclictest9-21ksoftirqd/020:07:130
3217993629,4cyclictest9-21ksoftirqd/021:37:320
3217993629,4cyclictest9-21ksoftirqd/019:41:330
3217993629,4cyclictest9-21ksoftirqd/019:41:330
3217993629,4cyclictest9-21ksoftirqd/000:30:040
3219993530,3cyclictest33-21ksoftirqd/222:00:002
3219993529,3cyclictest33-21ksoftirqd/222:57:002
3219993529,3cyclictest33-21ksoftirqd/222:11:152
3219993528,4cyclictest33-21ksoftirqd/223:32:002
3219993528,4cyclictest33-21ksoftirqd/220:32:302
3219993528,4cyclictest33-21ksoftirqd/219:37:052
3219993528,4cyclictest33-21ksoftirqd/200:10:542
3219993518,4cyclictest33-21ksoftirqd/221:15:322
3218993528,4cyclictest25-21ksoftirqd/100:03:511
3217993529,4cyclictest9-21ksoftirqd/022:35:210
3217993529,4cyclictest9-21ksoftirqd/020:14:050
3217993528,4cyclictest9-21ksoftirqd/023:38:250
3217993528,4cyclictest9-21ksoftirqd/020:43:240
3217993528,4cyclictest9-21ksoftirqd/000:35:140
3219993429,3cyclictest33-21ksoftirqd/221:40:312
3219993429,3cyclictest33-21ksoftirqd/220:20:012
3219993429,3cyclictest33-21ksoftirqd/219:30:242
3219993429,3cyclictest33-21ksoftirqd/219:26:592
3219993428,4cyclictest33-21ksoftirqd/222:43:492
3219993428,3cyclictest33-21ksoftirqd/223:55:132
3219993428,3cyclictest33-21ksoftirqd/221:38:502
3219993427,4cyclictest33-21ksoftirqd/200:05:572
3219993427,4cyclictest33-21ksoftirqd/200:05:572
3219993426,5cyclictest33-21ksoftirqd/220:08:402
3219993421,4cyclictest33-21ksoftirqd/223:25:282
3218993427,4cyclictest25-21ksoftirqd/121:16:481
3218993427,4cyclictest25-21ksoftirqd/120:51:041
3218993427,4cyclictest25-21ksoftirqd/120:42:081
3218993427,4cyclictest25-21ksoftirqd/100:35:001
3218993427,4cyclictest25-21ksoftirqd/100:17:561
3217993428,4cyclictest9-21ksoftirqd/023:26:400
3217993428,4cyclictest9-21ksoftirqd/023:00:310
3217993428,3cyclictest9-21ksoftirqd/021:42:220
3217993427,4cyclictest9-21ksoftirqd/023:52:300
3217993427,4cyclictest9-21ksoftirqd/023:33:180
3217993427,4cyclictest9-21ksoftirqd/023:23:150
3217993427,4cyclictest9-21ksoftirqd/022:20:200
3217993427,4cyclictest9-21ksoftirqd/020:51:060
3219993328,3cyclictest33-21ksoftirqd/219:43:462
3219993328,3cyclictest33-21ksoftirqd/219:43:452
3219993327,3cyclictest33-21ksoftirqd/220:47:382
3217993326,4cyclictest9-21ksoftirqd/022:54:000
3219993227,3cyclictest33-21ksoftirqd/223:10:262
3219993227,3cyclictest33-21ksoftirqd/223:01:002
3219993225,4cyclictest33-21ksoftirqd/222:30:542
3219993225,4cyclictest33-21ksoftirqd/222:30:542
321999320,3cyclictest121rcu_preempt00:27:542
321899328,21cyclictest28063-21unixbench_multi23:30:351
3218993225,4cyclictest25-21ksoftirqd/119:52:021
3217993227,3cyclictest9-21ksoftirqd/000:26:070
3219993125,4cyclictest33-21ksoftirqd/222:07:472
3219993124,4cyclictest33-21ksoftirqd/200:40:012
3218993124,4cyclictest25-21ksoftirqd/121:37:411
3218993124,4cyclictest25-21ksoftirqd/120:27:151
321899301,3cyclictest121rcu_preempt00:37:341
3217993024,3cyclictest9-21ksoftirqd/019:45:060
3219992922,4cyclictest2740-21/usr/sbin/munin23:45:292
3218992922,4cyclictest25-21ksoftirqd/123:24:561
3218992922,4cyclictest25-21ksoftirqd/121:22:111
321899282,6cyclictest241ktimersoftd/122:28:391
3218992821,4cyclictest25-21ksoftirqd/100:20:371
321899272,5cyclictest241ktimersoftd/122:15:171
3218992720,4cyclictest25-21ksoftirqd/122:43:091
3217992619,4cyclictest9-21ksoftirqd/019:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional