You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-14 - 00:42
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot4.osadl.org (updated Sat Jul 13, 2024 12:46:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1307421730,7sleep02706799cyclictest10:00:360
3218721610,6sleep12706899cyclictest09:35:151
1299911220,5ptp4l401ktimersoftd/307:09:533
1092521070,6sleep02706799cyclictest08:50:130
564981060,16rtkit-daemon0-21swapper/007:08:130
1034521000,9sleep22706999cyclictest07:40:382
27067998634,10cyclictest9-21ksoftirqd/008:02:550
2706799810,4cyclictest121rcu_preempt09:20:120
27067997940,5cyclictest9-21ksoftirqd/012:20:010
27067997866,6cyclictest9-21ksoftirqd/012:30:120
27067997740,4cyclictest9-21ksoftirqd/009:00:220
27067997740,4cyclictest9-21ksoftirqd/009:00:220
27067997737,4cyclictest9-21ksoftirqd/011:40:000
2657727656,9sleep20-21swapper/207:07:292
2706899742,17cyclictest5251-21grep10:55:141
27067997437,5cyclictest9-21ksoftirqd/009:40:380
27067997428,5cyclictest9-21ksoftirqd/009:25:150
2706899730,17cyclictest16435-21sendmail11:20:001
27067997361,7cyclictest9-21ksoftirqd/012:05:110
27067997337,4cyclictest9-21ksoftirqd/007:20:180
27067997336,5cyclictest9-21ksoftirqd/010:25:390
27067997336,5cyclictest9-21ksoftirqd/010:25:390
27067997334,5cyclictest9-21ksoftirqd/008:30:130
27067997260,7cyclictest9-21ksoftirqd/011:50:110
27067997236,6cyclictest9-21ksoftirqd/011:45:170
27067997236,6cyclictest9-21ksoftirqd/011:45:170
27067997236,5cyclictest9-21ksoftirqd/011:05:190
27067997236,4cyclictest9-21ksoftirqd/007:10:370
27067997160,6cyclictest9-21ksoftirqd/009:10:140
27067997136,4cyclictest9-21ksoftirqd/011:00:120
27067997135,4cyclictest9-21ksoftirqd/012:25:170
27067997134,4cyclictest9-21ksoftirqd/009:50:360
27067997133,4cyclictest9-21ksoftirqd/008:45:000
27067997131,5cyclictest9-21ksoftirqd/007:25:190
27067997130,5cyclictest9-21ksoftirqd/008:40:010
2706899706,14cyclictest0-21swapper/107:40:261
27068997012,12cyclictest30884-21grep11:50:131
27067997034,4cyclictest9-21ksoftirqd/008:25:350
27067997033,4cyclictest9-21ksoftirqd/010:35:340
27067996932,7cyclictest9-21ksoftirqd/009:45:150
27067996931,5cyclictest9-21ksoftirqd/012:40:000
27067996930,7cyclictest9-21ksoftirqd/010:50:320
27067996926,7cyclictest9-21ksoftirqd/009:15:310
27067996835,7cyclictest9-21ksoftirqd/010:45:320
27067996830,6cyclictest9-21ksoftirqd/007:15:250
27067996829,5cyclictest9-21ksoftirqd/010:00:000
27067996825,7cyclictest9-21ksoftirqd/011:55:140
2706899670,20cyclictest18725-21sed09:05:291
27067996733,4cyclictest9-21ksoftirqd/012:00:170
27067996732,5cyclictest9-21ksoftirqd/007:40:250
27067996731,5cyclictest9-21ksoftirqd/011:20:170
27067996731,5cyclictest9-21ksoftirqd/010:15:160
27067996730,7cyclictest9-21ksoftirqd/009:35:000
27067996723,7cyclictest9-21ksoftirqd/008:20:170
27067996631,4cyclictest9-21ksoftirqd/011:35:010
27067996630,4cyclictest9-21ksoftirqd/011:00:020
27067996626,6cyclictest9-21ksoftirqd/007:55:200
27067996626,5cyclictest9-21ksoftirqd/011:40:350
124891660,8getstats2707099cyclictest12:39:133
2706899650,13cyclictest22912-21df_abs09:15:141
27067996531,4cyclictest9-21ksoftirqd/007:50:140
27067996530,4cyclictest9-21ksoftirqd/010:05:210
2706799652,9cyclictest554-21dbus-daemon10:24:520
27067996523,7cyclictest9-21ksoftirqd/011:10:400
2706899640,14cyclictest21105-21systemd08:03:191
27067996424,7cyclictest9-21ksoftirqd/009:35:250
27067996424,4cyclictest9-21ksoftirqd/012:10:160
2652626440,13sleep125-21ksoftirqd/107:06:461
2706899630,21cyclictest1442-21latency_hist07:25:001
2706899630,18cyclictest10285-21chrt12:15:101
27067996328,4cyclictest9-21ksoftirqd/008:10:010
27067996327,5cyclictest9-21ksoftirqd/010:30:110
27067996323,6cyclictest9-21ksoftirqd/010:40:000
129991630,1ptp4l401ktimersoftd/310:55:353
27069996253,5cyclictest33-21ksoftirqd/211:40:112
27069996252,6cyclictest33-21ksoftirqd/209:20:132
27069996241,5cyclictest33-21ksoftirqd/208:03:102
2706899626,18cyclictest24205-21sed08:10:011
2706899625,15cyclictest7951-21cat07:35:371
2706899620,14cyclictest5352-21cat12:05:101
2706899620,13cyclictest27210-21/usr/sbin/munin09:25:161
2706899620,13cyclictest24410-21bc08:10:101
2706899620,13cyclictest15773-21latency_hist07:55:001
2706899620,12cyclictest8846-21fschecks_time08:45:191
27067996228,4cyclictest9-21ksoftirqd/007:35:010
27067996228,4cyclictest9-21ksoftirqd/007:35:010
27067996226,5cyclictest9-21ksoftirqd/011:25:390
27067996222,6cyclictest9-21ksoftirqd/007:35:380
27069996152,6cyclictest33-21ksoftirqd/207:25:152
27067996123,4cyclictest9-21ksoftirqd/008:15:390
27067996122,4cyclictest9-21ksoftirqd/008:40:190
2706799611,6cyclictest17170-21ls10:10:310
2706899600,14cyclictest1858-21apt-config09:40:011
2706899600,14cyclictest18170-21latency12:30:271
2706899600,14cyclictest16984-21latency_hist12:30:011
2706899600,13cyclictest5077-21missed_timers07:30:271
2706899600,13cyclictest5077-21missed_timers07:30:271
2706899600,13cyclictest3959-21latency12:00:231
27067996023,5cyclictest9-21ksoftirqd/011:19:590
129991600,1ptp4l401ktimersoftd/310:23:513
2706899590,19cyclictest15353-21cat09:00:091
2706899590,19cyclictest15353-21cat09:00:091
2706899590,13cyclictest3588-21latency_hist08:35:011
2706899590,12cyclictest0-21swapper/111:30:311
2706899590,11cyclictest20154-21awk10:15:371
27067995924,4cyclictest9-21ksoftirqd/008:55:380
2706899580,20cyclictest10308-21/usr/sbin/munin11:05:211
2706899580,14cyclictest28137-21sed08:15:361
2706899580,13cyclictest1670-21mailstats10:45:321
27067995823,4cyclictest9-21ksoftirqd/007:45:150
2706899570,14cyclictest27160-21grep11:40:301
2706899570,12cyclictest15470-21sed10:05:351
2706899570,11cyclictest16889-21ls07:55:251
27067995724,4cyclictest9-21ksoftirqd/012:20:140
27067995721,5cyclictest9-21ksoftirqd/008:15:010
27067995518,5cyclictest9-21ksoftirqd/009:05:120
129991550,1ptp4l391rcuc/311:10:483
27069995447,4cyclictest33-21ksoftirqd/212:20:002
27069995445,6cyclictest33-21ksoftirqd/210:30:092
2706899540,13cyclictest30779-21cat07:15:331
2706899540,13cyclictest24673-21date10:25:371
2706899540,13cyclictest24673-21date10:25:371
27069995343,6cyclictest33-21ksoftirqd/211:30:122
2706999516,4cyclictest131rcu_sched09:49:192
27069995141,6cyclictest33-21ksoftirqd/207:20:122
219182510,2sleep10-21swapper/110:20:281
129991500,1ptp4l401ktimersoftd/311:09:023
27069994944,3cyclictest33-21ksoftirqd/211:19:592
27069994943,3cyclictest33-21ksoftirqd/210:40:002
2706899490,11cyclictest16769-21cpu11:20:111
27069994838,7cyclictest33-21ksoftirqd/208:50:112
27070994739,4cyclictest41-21ksoftirqd/311:40:003
27069994741,3cyclictest33-21ksoftirqd/210:20:002
27069994739,4cyclictest33-21ksoftirqd/212:30:262
1298914739,3phc2sys41-21ksoftirqd/309:10:003
27069994640,3cyclictest33-21ksoftirqd/211:40:012
27069994633,9cyclictest33-21ksoftirqd/207:45:122
27069994632,8cyclictest33-21ksoftirqd/211:20:102
27069994539,3cyclictest33-21ksoftirqd/209:10:012
27069994532,8cyclictest33-21ksoftirqd/208:20:102
27069994532,7cyclictest33-21ksoftirqd/212:20:122
27069994530,9cyclictest33-21ksoftirqd/208:35:132
27068994538,4cyclictest1012-21latency_hist08:30:001
27070994437,4cyclictest41-21ksoftirqd/308:03:543
27069994439,3cyclictest33-21ksoftirqd/210:10:012
27069994431,7cyclictest33-21ksoftirqd/212:10:102
27069994430,8cyclictest33-21ksoftirqd/208:45:132
27068994437,3cyclictest25-21ksoftirqd/111:55:211
27069994337,4cyclictest33-21ksoftirqd/207:40:002
27069994336,4cyclictest33-21ksoftirqd/209:00:002
27069994329,8cyclictest33-21ksoftirqd/210:10:112
2706999431,4cyclictest121rcu_preempt09:15:382
27070994237,3cyclictest41-21ksoftirqd/309:15:013
27069994236,3cyclictest33-21ksoftirqd/209:15:002
27069994235,4cyclictest33-21ksoftirqd/209:00:222
27069994235,4cyclictest33-21ksoftirqd/209:00:222
27069994235,4cyclictest33-21ksoftirqd/207:20:002
27069994229,7cyclictest33-21ksoftirqd/209:35:122
2706899420,13cyclictest0-21swapper/107:10:291
27069994136,3cyclictest33-21ksoftirqd/210:25:252
27069994136,3cyclictest33-21ksoftirqd/210:25:252
27069994135,3cyclictest33-21ksoftirqd/210:50:352
27069994135,3cyclictest33-21ksoftirqd/208:30:002
27070994034,4cyclictest41-21ksoftirqd/310:45:373
27070994033,4cyclictest41-21ksoftirqd/312:19:463
27070994029,8cyclictest41-21ksoftirqd/307:30:513
27070994029,8cyclictest41-21ksoftirqd/307:30:503
27069994035,3cyclictest33-21ksoftirqd/207:13:452
27069994034,4cyclictest33-21ksoftirqd/208:34:592
27069994034,3cyclictest33-21ksoftirqd/212:40:002
27069994034,3cyclictest33-21ksoftirqd/209:33:012
27069994033,4cyclictest33-21ksoftirqd/212:06:132
27069994033,4cyclictest33-21ksoftirqd/210:40:042
27069994033,4cyclictest33-21ksoftirqd/209:27:292
27069994033,4cyclictest33-21ksoftirqd/208:09:102
27068994033,4cyclictest25-21ksoftirqd/109:33:461
27068994033,4cyclictest25-21ksoftirqd/108:22:431
30362390,4sleep1241ktimersoftd/107:25:401
27070993933,3cyclictest41-21ksoftirqd/311:42:023
27070993932,4cyclictest41-21ksoftirqd/309:57:343
27070993932,4cyclictest41-21ksoftirqd/309:25:293
27070993932,4cyclictest41-21ksoftirqd/308:29:453
27070993931,5cyclictest41-21ksoftirqd/310:27:053
27070993931,5cyclictest41-21ksoftirqd/310:27:043
27070993931,5cyclictest41-21ksoftirqd/309:23:323
27069993933,4cyclictest33-21ksoftirqd/209:55:132
27069993933,3cyclictest33-21ksoftirqd/211:13:122
27069993932,4cyclictest33-21ksoftirqd/209:51:182
27068993932,4cyclictest25-21ksoftirqd/108:51:191
27070993832,3cyclictest41-21ksoftirqd/309:01:143
27070993832,3cyclictest41-21ksoftirqd/309:01:143
27070993832,3cyclictest41-21ksoftirqd/308:33:253
27070993831,4cyclictest41-21ksoftirqd/312:08:303
27070993831,4cyclictest41-21ksoftirqd/311:48:393
27070993831,4cyclictest41-21ksoftirqd/311:48:393
27070993831,4cyclictest41-21ksoftirqd/310:19:263
27070993831,4cyclictest41-21ksoftirqd/310:07:213
27070993831,4cyclictest41-21ksoftirqd/309:49:193
27070993830,5cyclictest41-21ksoftirqd/309:51:313
27069993832,4cyclictest33-21ksoftirqd/207:31:402
27069993832,4cyclictest33-21ksoftirqd/207:31:402
27069993832,3cyclictest33-21ksoftirqd/212:00:002
27069993832,3cyclictest33-21ksoftirqd/210:24:522
27069993831,4cyclictest33-21ksoftirqd/211:29:302
27069993831,4cyclictest33-21ksoftirqd/210:45:492
27069993831,4cyclictest33-21ksoftirqd/207:51:382
27068993831,4cyclictest25-21ksoftirqd/110:02:011
27068993831,4cyclictest25-21ksoftirqd/109:51:041
27068993830,5cyclictest25-21ksoftirqd/110:54:331
27070993731,4cyclictest41-21ksoftirqd/312:03:463
27070993731,3cyclictest41-21ksoftirqd/308:40:593
27070993730,4cyclictest41-21ksoftirqd/308:58:283
27070993730,4cyclictest41-21ksoftirqd/308:38:133
27070993730,4cyclictest41-21ksoftirqd/308:13:153
27070993730,4cyclictest41-21ksoftirqd/307:49:103
27070993730,4cyclictest41-21ksoftirqd/307:27:283
27070993730,4cyclictest41-21ksoftirqd/307:24:083
27070993729,5cyclictest41-21ksoftirqd/310:03:163
27070993729,5cyclictest41-21ksoftirqd/308:52:333
27069993732,3cyclictest33-21ksoftirqd/211:45:232
27069993732,3cyclictest33-21ksoftirqd/211:45:232
27069993731,4cyclictest33-21ksoftirqd/211:00:352
27069993731,4cyclictest33-21ksoftirqd/208:40:252
27069993731,4cyclictest33-21ksoftirqd/208:15:372
27069993731,3cyclictest33-21ksoftirqd/212:00:382
27069993730,4cyclictest33-21ksoftirqd/210:57:232
27069993730,4cyclictest33-21ksoftirqd/209:43:352
27069993730,4cyclictest33-21ksoftirqd/208:11:392
27068993731,4cyclictest25-21ksoftirqd/111:02:401
27068993731,4cyclictest25-21ksoftirqd/108:35:091
27068993730,4cyclictest25-21ksoftirqd/111:11:461
27070993630,4cyclictest41-21ksoftirqd/307:55:133
27070993630,3cyclictest41-21ksoftirqd/311:20:513
27070993630,3cyclictest41-21ksoftirqd/310:33:443
27070993630,3cyclictest41-21ksoftirqd/308:06:053
27070993629,4cyclictest41-21ksoftirqd/311:02:053
27070993629,4cyclictest41-21ksoftirqd/309:43:323
27070993629,4cyclictest41-21ksoftirqd/308:17:493
27069993630,3cyclictest33-21ksoftirqd/211:07:232
27069993629,4cyclictest33-21ksoftirqd/210:01:132
27068993629,4cyclictest25-21ksoftirqd/111:28:381
27068993629,4cyclictest25-21ksoftirqd/110:12:211
27068993629,4cyclictest25-21ksoftirqd/109:22:461
129991360,1ptp4l401ktimersoftd/310:40:203
129991360,13ptp4l401ktimersoftd/311:26:443
27070993529,3cyclictest41-21ksoftirqd/311:55:463
27070993529,3cyclictest41-21ksoftirqd/310:14:583
27070993528,4cyclictest41-21ksoftirqd/312:29:343
27070993528,4cyclictest41-21ksoftirqd/311:19:483
27070993528,4cyclictest41-21ksoftirqd/309:37:063
27070993528,4cyclictest41-21ksoftirqd/309:32:203
27070993528,4cyclictest41-21ksoftirqd/308:21:343
27070993528,4cyclictest41-21ksoftirqd/307:35:103
27068993529,4cyclictest25-21ksoftirqd/110:37:181
27068993528,4cyclictest25-21ksoftirqd/109:14:471
27068993527,5cyclictest25-21ksoftirqd/108:58:131
1628023528,3sleep233-21ksoftirqd/212:25:322
27070993428,4cyclictest41-21ksoftirqd/310:37:303
27070993428,3cyclictest41-21ksoftirqd/311:31:073
27070993428,3cyclictest41-21ksoftirqd/310:53:473
27070993428,3cyclictest41-21ksoftirqd/309:17:563
27070993428,3cyclictest41-21ksoftirqd/307:18:443
27070993427,4cyclictest41-21ksoftirqd/312:12:263
27070993427,4cyclictest41-21ksoftirqd/307:12:013
27069993427,4cyclictest33-21ksoftirqd/211:52:272
27068993427,4cyclictest25-21ksoftirqd/112:37:021
27068993427,4cyclictest25-21ksoftirqd/109:46:191
129991340,1ptp4l401ktimersoftd/307:54:193
129991340,1ptp4l401ktimersoftd/307:42:233
27070993326,4cyclictest41-21ksoftirqd/312:32:113
27069993326,4cyclictest33-21ksoftirqd/207:55:232
2706899334,13cyclictest6822-21irqrtprio08:40:251
27068993327,3cyclictest25-21ksoftirqd/107:45:131
27068993226,4cyclictest25-21ksoftirqd/111:46:521
27068993226,4cyclictest25-21ksoftirqd/111:46:521
27068993225,4cyclictest25-21ksoftirqd/109:56:561
129991320,1ptp4l401ktimersoftd/311:52:593
27070993124,4cyclictest41-21ksoftirqd/312:20:523
27068993126,3cyclictest25-21ksoftirqd/112:14:561
27068993125,3cyclictest25-21ksoftirqd/110:44:481
27068993124,4cyclictest25-21ksoftirqd/112:22:011
27070993024,3cyclictest41-21ksoftirqd/308:49:213
27068993023,4cyclictest25-21ksoftirqd/110:34:461
2706899290,13cyclictest24881-21phc2sys11:35:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional