You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-22 - 03:28

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot2.osadl.org (updated Fri Oct 15, 2021 12:46:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1268911720,6ptp4l401ktimersoftd/307:07:513
867721390,2sleep20-21swapper/212:00:312
55598930,15rtkit-daemon0-21swapper/007:09:420
308262860,2sleep20-21swapper/207:10:032
30452998071,4cyclictest391rcuc/312:25:143
161152750,3sleep20-21swapper/209:30:192
161152750,3sleep20-21swapper/209:30:192
30452997464,5cyclictest41-21ksoftirqd/311:05:143
3045199741,28cyclictest18070-21sed12:15:012
3045199741,28cyclictest18070-21sed12:15:002
3045199740,31cyclictest30101-21sed09:50:012
3045199740,29cyclictest10930-21/usr/sbin/munin11:25:102
3045199740,28cyclictest18443-21/usr/sbin/munin10:55:042
2976027453,10sleep10-21swapper/107:07:331
3045199731,31cyclictest9832-21/usr/sbin/munin09:25:022
3045199731,29cyclictest31886-21grep07:50:192
3045199721,27cyclictest9842-21cat07:25:002
3045199720,29cyclictest6798-21sed08:00:082
3045199711,28cyclictest2033-21irqrtprio11:55:012
3045199711,27cyclictest16247-21fschecks_time10:50:312
3045199710,28cyclictest25212-21switchtime07:40:492
3045199701,34cyclictest6535-21df_abs10:00:072
3045199701,28cyclictest25048-21snmpd09:50:252
3045199701,25cyclictest25910-21/usr/sbin/munin09:05:152
3045199700,33cyclictest908-21sed10:30:302
3045199700,29cyclictest24831-21date09:40:342
3045199700,29cyclictest24831-21date09:40:332
3045199700,28cyclictest5816-21latency_hist10:40:012
3045199700,28cyclictest26962-21cat12:25:072
3045199700,28cyclictest17272-21sed08:50:322
3045199700,27cyclictest4639-21sed07:15:302
3045199691,31cyclictest16944-21grep10:10:432
3045199691,29cyclictest12331-21if_enp4s010:45:172
3045199691,26cyclictest2527-21df_abs09:55:072
3045199690,29cyclictest27970-21cat08:25:162
280142690,2sleep30-21swapper/310:25:173
126891690,2ptp4l1589-21sshd11:54:193
3045199681,27cyclictest24152-21cat11:40:252
3045199681,26cyclictest12763-21awk10:05:302
30452996733,5cyclictest41-21ksoftirqd/309:40:333
30452996733,5cyclictest41-21ksoftirqd/309:40:323
3045199670,29cyclictest20068-21cpuspeed_turbos09:35:162
3045199670,27cyclictest28429-21irqrtprio10:25:222
3045199670,25cyclictest5928-21irqrtprio10:40:012
30452996655,6cyclictest41-21ksoftirqd/310:40:133
3045199660,26cyclictest1987-21mailstats07:55:012
196712660,4sleep10-21swapper/108:15:151
196712660,4sleep10-21swapper/108:15:151
42782650,4sleep1241ktimersoftd/111:15:261
3045199651,24cyclictest31136-21perl09:10:082
30452996423,8cyclictest41-21ksoftirqd/310:00:003
30452996232,4cyclictest41-21ksoftirqd/308:05:073
30452996230,4cyclictest41-21ksoftirqd/311:20:293
30452996150,7cyclictest41-21ksoftirqd/308:35:133
30452996031,4cyclictest41-21ksoftirqd/312:00:233
30452996030,4cyclictest41-21ksoftirqd/308:45:013
30452996025,5cyclictest41-21ksoftirqd/309:25:313
126891600,1ptp4l401ktimersoftd/308:31:393
30452995930,8cyclictest41-21ksoftirqd/309:05:143
30452995928,4cyclictest41-21ksoftirqd/310:05:023
30452995928,4cyclictest41-21ksoftirqd/309:35:093
30452995924,5cyclictest41-21ksoftirqd/307:20:383
30451995911,30cyclictest1728-21/usr/sbin/munin08:35:032
275652592,18sleep227613-21grep07:05:172
30452995851,4cyclictest41-21ksoftirqd/308:45:053
30452995828,4cyclictest41-21ksoftirqd/311:00:173
30452995828,4cyclictest41-21ksoftirqd/309:20:183
30452995827,5cyclictest41-21ksoftirqd/312:30:173
30452995826,8cyclictest41-21ksoftirqd/309:00:073
30452995823,4cyclictest41-21ksoftirqd/311:30:003
30452995822,5cyclictest41-21ksoftirqd/307:15:143
202582580,2sleep20-21swapper/208:55:192
30452995728,7cyclictest41-21ksoftirqd/310:10:123
30452995727,8cyclictest41-21ksoftirqd/307:55:143
30452995725,8cyclictest41-21ksoftirqd/307:40:313
30452995649,4cyclictest41-21ksoftirqd/310:30:143
30452995648,5cyclictest41-21ksoftirqd/310:00:143
30452995647,6cyclictest41-21ksoftirqd/312:10:053
30452995647,6cyclictest41-21ksoftirqd/312:10:043
30452995627,4cyclictest41-21ksoftirqd/311:45:073
30452995626,4cyclictest41-21ksoftirqd/308:10:323
30452995625,5cyclictest41-21ksoftirqd/310:35:063
30452995625,4cyclictest41-21ksoftirqd/311:40:073
30452995621,5cyclictest41-21ksoftirqd/310:20:313
30452995620,4cyclictest41-21ksoftirqd/309:45:203
3045199561,21cyclictest0-21swapper/211:20:302
30452995526,4cyclictest41-21ksoftirqd/310:15:103
30452995525,4cyclictest41-21ksoftirqd/312:10:003
30452995525,4cyclictest41-21ksoftirqd/312:10:003
30452995518,5cyclictest41-21ksoftirqd/307:35:013
30449995547,5cyclictest9-21ksoftirqd/011:20:120
123191550,8getstats3045299cyclictest10:52:383
30452995445,4cyclictest41-21ksoftirqd/311:30:193
30452995425,4cyclictest41-21ksoftirqd/307:35:073
30452995425,4cyclictest41-21ksoftirqd/307:25:033
30452995423,8cyclictest41-21ksoftirqd/308:00:043
30452995420,5cyclictest41-21ksoftirqd/311:15:083
3045199540,24cyclictest20645-21missed_timers10:15:252
30452995318,4cyclictest41-21ksoftirqd/308:15:163
30452995318,4cyclictest41-21ksoftirqd/308:15:153
30452995315,8cyclictest41-21ksoftirqd/308:50:293
30452995220,4cyclictest41-21ksoftirqd/310:55:143
30452995216,5cyclictest41-21ksoftirqd/308:20:173
30452995215,5cyclictest41-21ksoftirqd/311:35:353
3045199521,32cyclictest23480-21grep10:20:152
30452995140,6cyclictest41-21ksoftirqd/307:50:193
30452995139,4cyclictest41-21ksoftirqd/312:15:303
30452995120,5cyclictest41-21ksoftirqd/309:30:163
30452995120,5cyclictest41-21ksoftirqd/309:30:163
30452995115,5cyclictest41-21ksoftirqd/309:10:093
3045199514,28cyclictest18966-21df_abs12:15:072
3045199512,40cyclictest27454-21apt-get11:05:132
30452995038,4cyclictest41-21ksoftirqd/308:25:233
30452995019,4cyclictest41-21ksoftirqd/311:10:013
30452995014,5cyclictest41-21ksoftirqd/310:45:013
55598490,2rtkit-daemon0-21swapper/010:56:130
30452994918,4cyclictest41-21ksoftirqd/307:45:133
30452994916,5cyclictest41-21ksoftirqd/312:35:073
30450994923,20cyclictest19495-21cpuspeed_turbos10:15:131
30452994842,4cyclictest41-21ksoftirqd/309:50:133
3045299483,8cyclictest41-21ksoftirqd/309:15:153
3045299481,4cyclictest41-21ksoftirqd/312:00:003
30452994740,4cyclictest41-21ksoftirqd/308:55:133
30452994537,5cyclictest41-21ksoftirqd/312:20:363
30452994531,5cyclictest41-21ksoftirqd/307:15:013
3045199451,26cyclictest4475-21irqrtprio12:35:252
3045199442,33cyclictest31588-21apt-get11:50:112
3045199434,31cyclictest23361-21apt-get11:00:132
3045199431,24cyclictest0-21swapper/212:05:082
3045199431,24cyclictest0-21swapper/212:05:072
30450994325,15cyclictest25-21ksoftirqd/108:11:421
3045199422,33cyclictest7145-21apt-get09:20:142
3045199422,31cyclictest3053-21apt-get11:15:142
3045199421,26cyclictest31627-21grep08:30:152
3045199421,26cyclictest0-21swapper/212:30:092
3045199420,24cyclictest0-21swapper/211:35:332
3045199411,22cyclictest0-21swapper/211:35:012
3045199410,25cyclictest0-21swapper/209:00:022
3045199403,30cyclictest11443-21apt-get08:45:142
3045199402,30cyclictest19406-21apt-get07:35:132
3045199400,25cyclictest0-21swapper/211:10:002
3045199400,24cyclictest0-21swapper/212:20:252
30450994023,14cyclictest25-21ksoftirqd/111:14:261
30450994015,4cyclictest5329-21unixbench_singl07:55:351
30450994015,4cyclictest12982-21mailstats10:45:251
30450994014,4cyclictest11597-21expr07:25:141
3045099401,36cyclictest2941-21grep09:55:091
3045199398,27cyclictest25048-21snmpd08:43:152
3045199393,30cyclictest0-21swapper/207:25:202
3045199393,29cyclictest19430-21apt-get08:15:132
3045199393,29cyclictest19430-21apt-get08:15:132
3045199392,29cyclictest15424-21apt-get08:10:142
30451993914,21cyclictest13236-21sed08:05:332
3045199390,4cyclictest29506-21ls11:45:312
3045199390,2cyclictest0-21swapper/209:15:102
3045199390,26cyclictest22988-21latency08:20:082
30450993921,15cyclictest25-21ksoftirqd/112:27:361
30450993914,4cyclictest15714-21tr12:10:151
30450993914,4cyclictest15714-21tr12:10:141
30450993913,3cyclictest11790-21expr11:25:221
30451993812,22cyclictest26373-21switchtime07:45:022
30451993811,23cyclictest17173-21cat07:30:332
30450993831,4cyclictest25-21ksoftirqd/110:20:141
30450993813,4cyclictest27894-21expr11:05:141
30450993812,4cyclictest7552-21expr12:00:151
3044999383,4cyclictest121rcu_preempt10:20:010
30449993832,4cyclictest9-21ksoftirqd/008:10:080
30450993715,4cyclictest24069-21acpi12:20:231
30450993712,4cyclictest7409-21expr08:40:151
30450993712,4cyclictest13791-21seq12:09:401
30450993712,4cyclictest13791-21seq12:09:401
30450993711,5cyclictest15043-21grep09:30:091
30450993711,5cyclictest15043-21grep09:30:091
30450993711,4cyclictest15358-21cat11:30:151
3044999378,9cyclictest121rcu_preempt12:25:140
30449993730,4cyclictest9-21ksoftirqd/009:04:580
30449993728,5cyclictest9-21ksoftirqd/010:50:230
30450993628,5cyclictest25-21ksoftirqd/109:00:441
30450993620,13cyclictest25-21ksoftirqd/107:24:401
30450993613,4cyclictest28849-21memory09:45:301
30450993612,4cyclictest1562-21latency_hist07:55:001
30450993611,5cyclictest8107-21seq11:20:181
30450993611,4cyclictest6196-21sendmail_mailqu08:00:021
30450993611,4cyclictest31087-21cut08:30:091
30449993631,3cyclictest9-21ksoftirqd/009:45:080
30449993631,3cyclictest9-21ksoftirqd/008:00:150
30449993630,4cyclictest9-21ksoftirqd/010:25:230
30449993630,4cyclictest9-21ksoftirqd/010:10:100
30450993529,4cyclictest25-21ksoftirqd/111:55:121
30450993529,4cyclictest25-21ksoftirqd/109:25:031
30450993529,4cyclictest25-21ksoftirqd/108:25:081
30450993524,6cyclictest25-21ksoftirqd/111:45:141
30450993511,4cyclictest14085-21sed07:30:011
30450993510,4cyclictest19805-21grep12:15:141
30449993529,4cyclictest9-21ksoftirqd/012:39:190
30449993529,4cyclictest9-21ksoftirqd/011:05:140
30449993529,4cyclictest9-21ksoftirqd/009:25:080
30449993529,3cyclictest9-21ksoftirqd/007:15:000
3045099348,6cyclictest5588-21cron09:19:591
3045099347,4cyclictest0-21swapper/110:25:171
3045099343,8cyclictest31646-21apt-get10:30:141
30450993429,3cyclictest25-21ksoftirqd/110:50:041
30450993429,3cyclictest25-21ksoftirqd/110:36:341
30450993427,4cyclictest25-21ksoftirqd/107:19:021
30449993429,3cyclictest9-21ksoftirqd/012:15:160
30449993429,3cyclictest9-21ksoftirqd/010:45:130
30449993428,4cyclictest9-21ksoftirqd/008:25:010
30449993428,3cyclictest9-21ksoftirqd/012:10:450
30449993428,3cyclictest9-21ksoftirqd/012:10:450
30449993428,3cyclictest9-21ksoftirqd/007:44:320
30449993427,4cyclictest9-21ksoftirqd/011:29:130
30449993427,4cyclictest9-21ksoftirqd/009:23:400
30449993427,4cyclictest9-21ksoftirqd/008:54:580
30450993327,4cyclictest25-21ksoftirqd/110:59:251
30450993327,4cyclictest25-21ksoftirqd/108:45:141
30450993326,4cyclictest25-21ksoftirqd/109:53:361
30450993326,4cyclictest25-21ksoftirqd/108:56:371
30450993313,5cyclictest25-21ksoftirqd/109:43:321
30450993313,5cyclictest25-21ksoftirqd/109:43:321
30450993311,5cyclictest3688-21basename12:35:171
30449993327,4cyclictest9-21ksoftirqd/011:40:030
30449993327,4cyclictest9-21ksoftirqd/011:31:380
30449993326,4cyclictest9-21ksoftirqd/011:36:280
30449993326,4cyclictest9-21ksoftirqd/010:05:500
30449993326,4cyclictest9-21ksoftirqd/009:37:200
30449993326,4cyclictest9-21ksoftirqd/009:08:340
30449993326,4cyclictest9-21ksoftirqd/008:49:020
30449993321,7cyclictest9-21ksoftirqd/011:50:140
30450993227,3cyclictest25-21ksoftirqd/109:20:141
30450993226,4cyclictest25-21ksoftirqd/111:35:481
30450993226,4cyclictest25-21ksoftirqd/109:06:421
30450993225,4cyclictest25-21ksoftirqd/111:54:081
30450993225,4cyclictest25-21ksoftirqd/111:03:291
30450993225,4cyclictest25-21ksoftirqd/110:02:531
30449993227,3cyclictest9-21ksoftirqd/008:15:030
30449993227,3cyclictest9-21ksoftirqd/008:15:030
30449993226,4cyclictest9-21ksoftirqd/011:10:030
30449993226,4cyclictest9-21ksoftirqd/010:44:280
30449993226,3cyclictest9-21ksoftirqd/010:39:200
30449993226,3cyclictest9-21ksoftirqd/010:20:130
30449993226,3cyclictest9-21ksoftirqd/009:55:030
30449993226,3cyclictest9-21ksoftirqd/008:08:430
30449993225,4cyclictest9-21ksoftirqd/012:02:470
30449993225,4cyclictest9-21ksoftirqd/011:19:340
30449993225,4cyclictest9-21ksoftirqd/009:32:340
30449993225,4cyclictest9-21ksoftirqd/009:32:340
30449993225,4cyclictest9-21ksoftirqd/008:57:380
30449993225,4cyclictest9-21ksoftirqd/008:23:400
30449993225,4cyclictest9-21ksoftirqd/007:29:050
30449993225,4cyclictest9-21ksoftirqd/007:18:210
30450993125,4cyclictest25-21ksoftirqd/107:45:121
30450993124,4cyclictest25-21ksoftirqd/108:53:091
30449993126,3cyclictest9-21ksoftirqd/012:09:360
30449993126,3cyclictest9-21ksoftirqd/012:09:350
30449993126,3cyclictest9-21ksoftirqd/008:38:540
30449993125,4cyclictest9-21ksoftirqd/012:30:130
30449993125,4cyclictest9-21ksoftirqd/009:54:020
30449993125,4cyclictest9-21ksoftirqd/007:24:160
30449993124,4cyclictest9-21ksoftirqd/009:18:290
30450993024,4cyclictest25-21ksoftirqd/108:22:581
30450993024,3cyclictest25-21ksoftirqd/112:32:411
30450993024,3cyclictest25-21ksoftirqd/107:41:341
30450993023,4cyclictest25-21ksoftirqd/111:42:521
30450993023,4cyclictest25-21ksoftirqd/110:06:121
30449993025,3cyclictest9-21ksoftirqd/011:57:010
30449993024,4cyclictest9-21ksoftirqd/007:55:040
30449993024,3cyclictest9-21ksoftirqd/012:22:010
30449993024,3cyclictest9-21ksoftirqd/010:32:190
30449993024,3cyclictest9-21ksoftirqd/009:40:070
30449993024,3cyclictest9-21ksoftirqd/009:40:070
30450992923,4cyclictest25-21ksoftirqd/107:39:181
30450992922,4cyclictest25-21ksoftirqd/110:41:001
30450992922,4cyclictest25-21ksoftirqd/109:36:361
30449992923,4cyclictest9-21ksoftirqd/008:40:080
30449992922,4cyclictest9-21ksoftirqd/010:01:490
30450992821,4cyclictest25-21ksoftirqd/108:08:021
30449992823,3cyclictest9-21ksoftirqd/011:47:490
30449992822,4cyclictest9-21ksoftirqd/008:33:560
30449992822,4cyclictest9-21ksoftirqd/007:33:540
30450992721,4cyclictest25-21ksoftirqd/108:39:401
30449992721,4cyclictest9-21ksoftirqd/011:00:250
30449992721,4cyclictest9-21ksoftirqd/007:37:480
30449992716,6cyclictest101ktimersoftd/009:10:250
30449992619,4cyclictest9-21ksoftirqd/007:48:110
30450992518,4cyclictest25-21ksoftirqd/110:13:021
30450992518,4cyclictest25-21ksoftirqd/109:11:241
30449992419,3cyclictest9-21ksoftirqd/007:51:410
30450992114,4cyclictest796-21ptp4l07:10:341
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional