You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-07-29 - 16:10

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot3.osadl.org (updated Thu Jul 29, 2021 00:46:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1283911810,6ptp4l401ktimersoftd/319:08:203
566981000,9rtkit-daemon0-21swapper/019:05:040
84442820,2sleep10-21swapper/123:50:111
67052754,58sleep10-21swapper/119:05:441
261492690,6sleep026151-21proc_pri20:50:270
231912660,3sleep2311rcuc/223:30:012
111042660,5sleep011103-21readlink23:55:000
124462640,9sleep3736899cyclictest23:15:133
250802620,2sleep20-21swapper/220:10:142
93882610,3sleep30-21swapper/319:50:163
54002580,2sleep30-21swapper/320:25:213
128391530,2ptp4l22081-21/usr/sbin/munin21:25:273
7368995028,19cyclictest41-21ksoftirqd/321:51:583
7368995028,18cyclictest41-21ksoftirqd/320:32:473
7368994823,5cyclictest41-21ksoftirqd/300:18:373
7368994723,4cyclictest41-21ksoftirqd/323:28:213
7368994723,4cyclictest41-21ksoftirqd/321:56:313
128391470,1ptp4l401ktimersoftd/300:20:013
128391470,1ptp4l401ktimersoftd/300:20:013
7365994628,12cyclictest9-21ksoftirqd/020:18:590
670124613,20sleep20-21swapper/219:05:402
128391460,3ptp4l736899cyclictest21:02:543
7366994527,14cyclictest25-21ksoftirqd/119:20:581
128391450,1ptp4l401ktimersoftd/323:00:063
128391450,1ptp4l401ktimersoftd/321:30:203
7366994417,4cyclictest23024-21gpgv23:30:001
7365994427,14cyclictest9-21ksoftirqd/021:23:070
7365994427,14cyclictest9-21ksoftirqd/019:43:580
128391440,1ptp4l401ktimersoftd/323:07:513
128391440,1ptp4l401ktimersoftd/323:07:513
128391440,1ptp4l401ktimersoftd/320:18:363
128391440,1ptp4l401ktimersoftd/320:07:073
736799431,7cyclictest16574-21apt-get22:00:152
736799431,7cyclictest16574-21apt-get22:00:142
7366994325,15cyclictest25-21ksoftirqd/121:44:481
7365994319,21cyclictest17280-21grep00:00:200
128391430,1ptp4l401ktimersoftd/323:35:493
7368994234,5cyclictest41-21ksoftirqd/323:40:133
736799422,9cyclictest570-21apt-get20:20:122
7367994226,12cyclictest33-21ksoftirqd/221:28:582
128391420,1ptp4l401ktimersoftd/321:13:313
7367994117,21cyclictest27274-21/usr/sbin/munin23:35:142
7367994115,22cyclictest13311-21cut22:35:252
7367994114,23cyclictest10994-21wc21:55:012
736699412,10cyclictest29196-21sh23:35:161
736799406,3cyclictest0-21swapper/220:40:012
7367994025,12cyclictest33-21ksoftirqd/200:10:392
7367994022,14cyclictest33-21ksoftirqd/200:09:592
736699403,9cyclictest16734-21bc21:20:141
7365994022,15cyclictest9-21ksoftirqd/021:36:370
736799399,27cyclictest41850irq/96-enp4s0-T21:35:152
736799392,7cyclictest8534-21apt-get23:10:122
736799392,7cyclictest8534-21apt-get23:10:122
736799392,6cyclictest13002-21apt-get21:15:212
736799391,6cyclictest8867-21unixbench_multi00:30:132
7367993913,23cyclictest432-21unixbench_multi21:40:142
7367993910,4cyclictest17070-21cpuspeed_turbos23:20:142
736599394,28cyclictest12816-21apt-get00:35:170
7365993913,22cyclictest1725-21gnome-shell19:26:590
7365993911,24cyclictest3787-21cpuspeed_turbos21:45:070
7365993911,24cyclictest12021-21mailstats21:55:090
736599390,4cyclictest0-21swapper/021:10:010
7367993831,4cyclictest33-21ksoftirqd/220:03:082
736799382,32cyclictest32066-21cut22:20:072
7367993814,21cyclictest4874-21acpi20:25:152
7367993812,23cyclictest9950-21sed21:10:252
7367993812,22cyclictest23245-21cpu19:30:002
736699383,8cyclictest24459-21cat21:30:171
7366993831,4cyclictest25-21ksoftirqd/100:20:111
7366993831,4cyclictest25-21ksoftirqd/100:20:111
736699382,8cyclictest4698-21acpi23:45:151
7366993816,3cyclictest22141-21phc2sys-jitter21:25:281
736599383,28cyclictest688-21apt-get00:20:130
736599383,28cyclictest688-21apt-get00:20:130
7365993810,25cyclictest17859-21needreboot22:00:260
7365993810,25cyclictest17859-21needreboot22:00:250
736599380,5cyclictest19207-21dpkg22:45:010
736599380,4cyclictest9566-21if_enp1s021:10:200
736599380,4cyclictest7114-21latency_hist23:10:010
736599380,4cyclictest7114-21latency_hist23:10:000
736599380,4cyclictest6313-21ptp4l19:45:330
7368993731,3cyclictest41-21ksoftirqd/300:09:383
736799378,8cyclictest0-21swapper/200:00:202
736799377,5cyclictest2847-21latency_hist23:05:002
736799373,31cyclictest23232-21cron20:10:002
736799372,9cyclictest29085-21apt-get00:15:122
7367993714,20cyclictest16780-21/usr/sbin/munin19:20:242
7367993713,21cyclictest9560-21cut19:10:172
7367993712,21cyclictest31172-21wc21:00:012
7367993710,4cyclictest30260-21needreboot20:15:242
736699371,8cyclictest8936-21gzip19:50:141
7365993722,12cyclictest9-21ksoftirqd/022:25:390
7365993713,21cyclictest794-21cstates23:40:150
7365993713,21cyclictest25608-21ls00:10:150
7365993712,22cyclictest8137-21missed_timers21:50:110
128391370,1ptp4l401ktimersoftd/323:57:033
7368993630,4cyclictest41-21ksoftirqd/322:55:223
7368993630,3cyclictest41-21ksoftirqd/322:46:553
7368993629,4cyclictest41-21ksoftirqd/322:02:513
7368993629,4cyclictest41-21ksoftirqd/322:02:503
7368993628,5cyclictest41-21ksoftirqd/323:23:083
736799369,4cyclictest28782-21/usr/sbin/munin22:15:142
736799369,4cyclictest20882-21basename22:45:152
736799368,5cyclictest6103-21wc23:45:282
736799363,10cyclictest16521-21apt-get22:40:112
736799362,6cyclictest32754-21apt-get23:40:132
7367993616,17cyclictest3113-21switchtime23:05:012
7367993616,17cyclictest3113-21switchtime23:05:002
7367993612,21cyclictest610-21/usr/sbin/munin21:00:322
7367993612,20cyclictest11663-21cat23:55:042
7367993611,22cyclictest11356-21/usr/sbin/munin19:55:122
7367993611,21cyclictest11408-21cstates23:15:062
736599369,24cyclictest11220-21cut20:35:010
7365993612,21cyclictest32529-21fschecks_count21:00:100
7365993612,21cyclictest32445-21sed22:20:110
7365993612,21cyclictest23800-21expr20:10:050
7365993610,22cyclictest25565-21fschecks_count23:30:310
7365993610,22cyclictest13732-21irqrtprio20:35:240
736599360,5cyclictest26422-21date21:30:320
128391360,1ptp4l401ktimersoftd/322:26:153
128391360,1ptp4l401ktimersoftd/321:44:493
128391360,13ptp4l401ktimersoftd/320:02:253
1282913629,3phc2sys41-21ksoftirqd/323:12:583
1282913629,3phc2sys41-21ksoftirqd/323:12:583
7368993529,3cyclictest41-21ksoftirqd/300:10:583
7368993528,4cyclictest41-21ksoftirqd/322:15:513
7368993528,4cyclictest41-21ksoftirqd/322:05:383
7368993528,4cyclictest41-21ksoftirqd/320:59:543
7368993527,5cyclictest41-21ksoftirqd/321:15:593
736799359,4cyclictest24716-21phc2sys22:50:132
736799358,5cyclictest818-21unixbench_multi19:40:132
736799358,4cyclictest1429-21fschecks_count00:20:162
736799358,4cyclictest1429-21fschecks_count00:20:162
7367993529,3cyclictest33-21ksoftirqd/221:20:482
736799352,7cyclictest24852-21apt-get21:30:182
736799352,6cyclictest4441-21apt-get21:45:122
736799351,6cyclictest8717-21basename22:30:152
736799351,6cyclictest4522-21apt-get21:05:142
736799351,10cyclictest28660-21apache219:30:172
736799350,4cyclictest15183-21latency_hist20:40:002
7366993529,4cyclictest25-21ksoftirqd/122:45:041
7366993529,3cyclictest25-21ksoftirqd/122:25:241
7366993527,5cyclictest25-21ksoftirqd/119:46:161
7366993517,4cyclictest11128-21awk19:15:001
736599359,22cyclictest15440-21switchtime20:00:010
736599353,26cyclictest20556-21apt-get23:25:120
736599353,10cyclictest0-21swapper/023:45:150
7365993528,4cyclictest9-21ksoftirqd/000:15:150
7365993512,20cyclictest4393-21cut20:25:110
7365993511,20cyclictest13390-21sed23:55:190
2471923528,3sleep125-21ksoftirqd/100:10:111
128391350,1ptp4l401ktimersoftd/323:45:193
128391350,1ptp4l401ktimersoftd/322:43:033
128391350,1ptp4l401ktimersoftd/320:11:193
128391350,1ptp4l401ktimersoftd/319:30:523
128391350,12ptp4l401ktimersoftd/319:18:383
7368993427,5cyclictest41-21ksoftirqd/319:55:123
7368993426,5cyclictest41-21ksoftirqd/323:34:153
7368993426,5cyclictest41-21ksoftirqd/321:05:563
7368993426,5cyclictest41-21ksoftirqd/319:21:013
736799348,4cyclictest8313-21sed20:30:112
736799342,8cyclictest8894-21apt-get23:50:202
736799342,8cyclictest12764-21apt-get19:15:122
736799342,7cyclictest20655-21apt-get20:45:122
736799342,7cyclictest20655-21apt-get20:45:122
7367993426,5cyclictest33-21ksoftirqd/219:37:342
7367993413,18cyclictest5363-21hwlatdetect22:25:222
736799341,11cyclictest23479-21latency20:50:042
736699344,9cyclictest481-21apt-get23:40:141
7366993428,4cyclictest25-21ksoftirqd/122:51:041
7366993427,4cyclictest25-21ksoftirqd/121:55:031
7366993427,4cyclictest25-21ksoftirqd/100:39:021
736599346,3cyclictest15135-21dump-pmu-power19:20:000
736599343,25cyclictest8487-21apt-get22:30:130
736599342,6cyclictest8863-21apt-get19:50:130
7365993414,17cyclictest28578-21/usr/sbin/munin22:55:280
7365993414,17cyclictest19202-21/usr/sbin/munin20:45:130
7365993414,17cyclictest19202-21/usr/sbin/munin20:45:120
7365993414,17cyclictest1189-21runrttasks21:40:070
7365993413,18cyclictest29514-21grep20:55:180
7365993412,19cyclictest21307-21fschecks_count22:05:180
7365993412,18cyclictest19571-21cstates21:25:060
7365993411,20cyclictest16169-21df_abs23:20:090
7365993410,21cyclictest17556-21irqrtprio22:40:230
7365993410,20cyclictest29706-21if_enp1s022:15:190
736599340,5cyclictest15111-21chrt19:59:360
7368993327,4cyclictest41-21ksoftirqd/322:23:503
7368993326,4cyclictest41-21ksoftirqd/322:35:583
7368993326,4cyclictest41-21ksoftirqd/321:36:373
7368993326,4cyclictest41-21ksoftirqd/319:35:133
736799338,3cyclictest28800-21acpi22:55:122
736799337,4cyclictest24655-21acpi22:10:142
7367993313,17cyclictest10179-21awk19:50:252
7367993312,18cyclictest1189-21runrttasks22:09:432
7367993311,19cyclictest13927-21latency00:35:242
7367993310,4cyclictest11031-21/usr/sbin/munin21:55:112
736799330,5cyclictest4975-21/usr/sbin/munin00:25:362
7366993327,4cyclictest25-21ksoftirqd/123:20:121
7366993327,4cyclictest25-21ksoftirqd/121:10:141
7366993326,4cyclictest25-21ksoftirqd/122:41:181
7366993326,4cyclictest25-21ksoftirqd/120:36:431
7366993326,4cyclictest25-21ksoftirqd/119:19:261
7366993326,4cyclictest25-21ksoftirqd/100:26:491
7366993325,5cyclictest25-21ksoftirqd/121:09:351
736599335,5cyclictest23372-21ls20:10:010
736599335,4cyclictest31199-21sh23:40:000
7365993326,4cyclictest9-21ksoftirqd/021:15:170
7365993325,5cyclictest9-21ksoftirqd/000:28:200
736599332,4cyclictest0-21swapper/020:40:130
7365993311,19cyclictest8298-21cat19:10:100
7365993310,20cyclictest24467-21irqrtprio22:10:110
736599330,4cyclictest27296-21apt-config19:35:000
128391330,1ptp4l401ktimersoftd/319:46:353
128391330,11ptp4l401ktimersoftd/322:11:293
1282913326,3phc2sys41-21ksoftirqd/300:00:163
7368993225,4cyclictest41-21ksoftirqd/321:23:033
7368993225,4cyclictest41-21ksoftirqd/320:37:023
7368993224,5cyclictest41-21ksoftirqd/319:14:143
7368993224,5cyclictest41-21ksoftirqd/300:26:033
736799328,4cyclictest3778-21open_files19:45:062
736799328,4cyclictest22926-21apt-get23:30:012
7366993226,4cyclictest25-21ksoftirqd/123:03:091
7366993226,4cyclictest25-21ksoftirqd/121:35:121
7366993225,4cyclictest25-21ksoftirqd/123:17:481
7366993225,4cyclictest25-21ksoftirqd/122:56:551
7366993225,4cyclictest25-21ksoftirqd/122:20:021
7366993225,4cyclictest25-21ksoftirqd/122:00:391
7366993225,4cyclictest25-21ksoftirqd/122:00:381
7366993224,5cyclictest25-21ksoftirqd/120:02:461
736599328,21cyclictest2856-21latency_hist23:05:000
736599328,20cyclictest13631-21irqrtprio22:35:290
736599328,20cyclictest11833-21cat23:15:080
736599325,4cyclictest19319-21cut19:25:000
736599324,5cyclictest31959-21/usr/sbin/munin20:20:050
736599322,4cyclictest0-21swapper/022:50:130
736599320,4cyclictest29853-21fschecks_count19:35:210
128391320,14ptp4l401ktimersoftd/300:30:113
7366993125,4cyclictest25-21ksoftirqd/119:37:371
7366993125,3cyclictest25-21ksoftirqd/122:12:261
7366993125,3cyclictest25-21ksoftirqd/121:02:201
7366993124,5cyclictest25-21ksoftirqd/120:17:561
7366993124,5cyclictest25-21ksoftirqd/119:31:551
7366993124,4cyclictest25-21ksoftirqd/123:09:441
7366993124,4cyclictest25-21ksoftirqd/123:09:441
7366993124,4cyclictest25-21ksoftirqd/121:54:481
7366993124,4cyclictest25-21ksoftirqd/120:12:291
7366993124,4cyclictest25-21ksoftirqd/119:29:131
7366993123,5cyclictest25-21ksoftirqd/120:32:331
736599317,21cyclictest23331-21cron00:09:590
736599316,4cyclictest8955-21cat00:30:140
736599314,4cyclictest7333-21latency23:10:030
736599314,4cyclictest7333-21latency23:10:030
128391310,1ptp4l401ktimersoftd/323:54:563
7368993024,3cyclictest41-21ksoftirqd/319:26:003
7368993023,4cyclictest41-21ksoftirqd/320:51:153
7368993022,5cyclictest41-21ksoftirqd/320:46:093
7368993022,5cyclictest41-21ksoftirqd/320:46:093
7368993022,5cyclictest41-21ksoftirqd/320:44:363
7366993024,4cyclictest25-21ksoftirqd/100:16:371
7366993023,4cyclictest25-21ksoftirqd/123:14:181
7366993023,4cyclictest25-21ksoftirqd/123:14:171
128391300,1ptp4l401ktimersoftd/300:36:053
7368992923,4cyclictest41-21ksoftirqd/322:52:023
7368992922,4cyclictest41-21ksoftirqd/319:41:593
7366992924,3cyclictest25-21ksoftirqd/100:05:391
7366992923,4cyclictest25-21ksoftirqd/121:17:541
7366992923,4cyclictest25-21ksoftirqd/120:50:321
7366992923,4cyclictest25-21ksoftirqd/120:05:031
7366992922,4cyclictest25-21ksoftirqd/122:36:541
7366992922,4cyclictest25-21ksoftirqd/120:48:311
7366992922,4cyclictest25-21ksoftirqd/120:48:311
7366992922,4cyclictest25-21ksoftirqd/100:03:401
128391290,1ptp4l401ktimersoftd/321:46:593
7366992820,5cyclictest25-21ksoftirqd/122:17:011
7366992721,4cyclictest25-21ksoftirqd/120:27:561
7366992720,4cyclictest25-21ksoftirqd/119:42:281
3238522720,3sleep125-21ksoftirqd/120:20:101
7366992620,3cyclictest25-21ksoftirqd/100:34:431
7366992619,4cyclictest25-21ksoftirqd/119:58:311
7366992518,4cyclictest25-21ksoftirqd/120:57:211
128391250,1ptp4l401ktimersoftd/322:30:163
128391250,1ptp4l401ktimersoftd/320:20:123
7366992417,4cyclictest25-21ksoftirqd/122:07:291
7366992316,4cyclictest26368-21find23:30:461
7366992316,4cyclictest25-21ksoftirqd/122:31:091
7366992316,4cyclictest25-21ksoftirqd/121:47:501
7366992215,4cyclictest25-21ksoftirqd/123:55:031
7366992215,4cyclictest18548-21tail20:40:301
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional