You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-05-12 - 14:26

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot1.osadl.org (updated Wed May 12, 2021 00:46:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1415621810,8sleep0293099cyclictest22:00:200
1146911760,23ptp4l1746-21gnome-shell19:08:580
1146911680,8ptp4l0-21swapper/119:05:331
595721640,7sleep1293199cyclictest20:30:241
3112421580,3sleep1293199cyclictest00:20:251
25372135112,10sleep20-21swapper/219:08:592
118791131108,10phc2sys0-21swapper/319:09:043
2932999585,6cyclictest33-21ksoftirqd/221:05:122
2932999081,5cyclictest33-21ksoftirqd/200:20:152
2932998776,6cyclictest33-21ksoftirqd/220:50:142
2932998676,6cyclictest33-21ksoftirqd/220:20:142
2932998573,7cyclictest33-21ksoftirqd/220:40:132
2932998573,7cyclictest33-21ksoftirqd/220:40:132
87702770,2sleep30-21swapper/319:55:123
2932997762,8cyclictest33-21ksoftirqd/223:05:112
2932997661,7cyclictest33-21ksoftirqd/221:20:142
2932997565,5cyclictest33-21ksoftirqd/219:50:162
2932997460,8cyclictest33-21ksoftirqd/220:45:142
2932997159,6cyclictest33-21ksoftirqd/223:25:142
2932997159,6cyclictest33-21ksoftirqd/222:45:122
2932997158,8cyclictest33-21ksoftirqd/200:10:132
2932997058,6cyclictest33-21ksoftirqd/200:00:152
114691700,23ptp4l0-21swapper/223:55:102
2930996860,4cyclictest9-21ksoftirqd/019:25:120
293399665,4cyclictest114691ptp4l20:15:373
2932996651,8cyclictest33-21ksoftirqd/222:10:162
2932996553,6cyclictest33-21ksoftirqd/223:35:132
2932996246,8cyclictest33-21ksoftirqd/223:40:162
114691620,5ptp4l20637-21apt-get20:10:113
183362610,4sleep1293199cyclictest22:05:231
2932996045,9cyclictest33-21ksoftirqd/220:10:122
114691600,2ptp4l0-21swapper/221:15:022
114691600,1ptp4l0-21swapper/023:50:000
2932995944,7cyclictest33-21ksoftirqd/220:35:162
2932995944,7cyclictest33-21ksoftirqd/220:35:162
114691590,2ptp4l0-21swapper/320:05:073
114691590,2ptp4l0-21swapper/121:25:131
114691590,2ptp4l0-21swapper/023:25:010
114691590,2ptp4l0-21swapper/021:05:130
114691590,2ptp4l0-21swapper/020:51:590
114691590,15ptp4l0-21swapper/100:30:181
114691590,12ptp4l8564-21ls00:35:022
2932995850,5cyclictest33-21ksoftirqd/219:45:052
2930995749,4cyclictest9-21ksoftirqd/020:00:130
171002570,2sleep30-21swapper/300:05:093
114691570,2ptp4l0-21swapper/222:21:392
114691570,2ptp4l0-21swapper/220:33:162
114691570,2ptp4l0-21swapper/200:18:462
114691570,2ptp4l0-21swapper/019:17:500
114691570,2ptp4l0-21swapper/000:06:170
114691570,1ptp4l0-21swapper/023:03:520
114691560,3ptp4l25799-21df_inode22:55:160
114691560,2ptp4l0-21swapper/222:04:102
114691560,2ptp4l0-21swapper/220:04:222
114691560,2ptp4l0-21swapper/021:40:160
114691560,2ptp4l0-21swapper/021:38:060
114691560,2ptp4l0-21swapper/021:10:320
114691560,12ptp4l0-21swapper/022:23:270
55098550,5rtkit-daemon0-21swapper/221:59:542
2932995543,7cyclictest33-21ksoftirqd/223:00:152
2932995542,8cyclictest33-21ksoftirqd/219:20:142
2932995542,7cyclictest33-21ksoftirqd/221:50:142
2932995537,9cyclictest33-21ksoftirqd/221:00:122
114791550,8getstats0-21swapper/222:18:272
114691550,8ptp4l25-21ksoftirqd/122:20:051
114691550,2ptp4l0-21swapper/222:56:552
114691550,2ptp4l0-21swapper/121:16:231
114691550,2ptp4l0-21swapper/022:46:200
114691550,1ptp4l0-21swapper/121:45:281
114691550,1ptp4l0-21swapper/023:55:320
114691550,16ptp4l0-21swapper/021:34:390
114691540,5ptp4l28140-21awk00:20:000
114691540,3ptp4l0-21swapper/222:52:462
114691540,2ptp4l0-21swapper/019:55:080
114691540,2ptp4l0-21swapper/000:25:330
114691540,1ptp4l0-21swapper/123:06:081
114691540,1ptp4l0-21swapper/019:53:350
114691540,11ptp4l0-21swapper/023:30:030
114691530,9ptp4l0-21swapper/123:27:311
114691530,2ptp4l0-21swapper/200:32:392
114691530,2ptp4l0-21swapper/022:05:060
114691530,2ptp4l0-21swapper/020:55:140
114691530,2ptp4l0-21swapper/020:20:260
114691530,1ptp4l0-21swapper/019:10:020
114691520,2ptp4l0-21swapper/222:27:092
114691520,2ptp4l0-21swapper/023:25:170
114691520,2ptp4l0-21swapper/022:41:060
114691520,2ptp4l0-21swapper/020:30:110
114691520,2ptp4l0-21swapper/020:29:410
114691520,10ptp4l0-21swapper/019:20:200
2932995138,7cyclictest33-21ksoftirqd/223:45:142
114691510,2ptp4l0-21swapper/021:19:050
114691510,2ptp4l0-21swapper/019:33:590
114691510,2ptp4l0-21swapper/019:33:580
114691510,2ptp4l0-21swapper/000:00:090
114691510,10ptp4l0-21swapper/100:11:351
2932995042,4cyclictest33-21ksoftirqd/223:15:042
2932995037,7cyclictest33-21ksoftirqd/222:35:162
114691500,2ptp4l0-21swapper/220:57:172
114691500,2ptp4l0-21swapper/023:50:260
114691500,2ptp4l0-21swapper/021:25:130
114691500,2ptp4l0-21swapper/000:33:050
2932994942,4cyclictest33-21ksoftirqd/220:05:132
2931994931,15cyclictest25-21ksoftirqd/120:29:311
2930994943,3cyclictest9-21ksoftirqd/019:45:050
114691490,9ptp4l0-21swapper/120:35:341
114691490,9ptp4l0-21swapper/120:35:341
114691490,4ptp4l231rcuc/120:50:141
114691490,2ptp4l0-21swapper/219:41:142
114691490,2ptp4l0-21swapper/021:51:380
114691490,2ptp4l0-21swapper/020:45:110
114691490,1ptp4l0-21swapper/000:20:430
114691490,11ptp4l32368-21fschecks_time22:25:053
2933994831,14cyclictest41-21ksoftirqd/322:14:373
2933994831,14cyclictest41-21ksoftirqd/300:15:003
2932994837,6cyclictest33-21ksoftirqd/221:35:152
2932994835,8cyclictest33-21ksoftirqd/219:15:222
114791480,9getstats0-21swapper/021:55:350
114791480,9getstats0-21swapper/020:17:550
114691480,9ptp4l0-21swapper/119:55:251
114691480,2ptp4l0-21swapper/219:33:132
114691480,2ptp4l0-21swapper/219:33:122
114691480,2ptp4l0-21swapper/023:16:420
114691480,2ptp4l0-21swapper/000:11:460
114691480,1ptp4l0-21swapper/021:03:430
114691480,12ptp4l20409-21irqrtprio23:30:052
114791470,9getstats0-21swapper/023:40:410
114791470,9getstats0-21swapper/022:33:320
114791470,9getstats0-21swapper/022:13:240
114791470,8getstats0-21swapper/021:23:370
114691470,8ptp4l0-21swapper/200:29:512
114691470,2ptp4l0-21swapper/220:18:062
114691470,1ptp4l0-21swapper/322:35:333
114691470,1ptp4l0-21swapper/023:38:590
114691460,8ptp4l0-21swapper/219:27:592
114691460,8ptp4l0-21swapper/020:39:490
114691460,8ptp4l0-21swapper/020:39:490
114691460,6ptp4l101ktimersoftd/020:14:090
114691460,2ptp4l0-21swapper/321:26:273
114691460,2ptp4l0-21swapper/321:01:173
114691460,2ptp4l0-21swapper/220:27:522
114691460,2ptp4l0-21swapper/020:42:190
114691460,2ptp4l0-21swapper/020:42:190
114691460,1ptp4l0-21swapper/321:08:013
114691460,1ptp4l0-21swapper/320:42:293
114691460,1ptp4l0-21swapper/320:42:293
2932994539,4cyclictest33-21ksoftirqd/221:40:062
2932994532,9cyclictest33-21ksoftirqd/223:10:142
114691450,8ptp4l0-21swapper/223:22:122
114691450,7ptp4l0-21swapper/222:07:172
114691450,7ptp4l0-21swapper/219:59:532
114691450,2ptp4l0-21swapper/221:19:112
114691450,1ptp4l0-21swapper/219:38:492
114791440,8getstats0-21swapper/023:11:290
114691440,7ptp4l0-21swapper/020:06:180
114691440,6ptp4l12713-21iostat_ios22:00:101
114691440,5ptp4l15343-21/usr/sbin/munin20:45:083
114691440,2ptp4l0-21swapper/322:47:253
114691440,2ptp4l0-21swapper/320:51:333
114691440,2ptp4l0-21swapper/121:02:121
114691440,1ptp4l0-21swapper/221:32:112
114691440,10ptp4l0-21swapper/121:55:261
2931994325,15cyclictest25-21ksoftirqd/120:08:401
114691430,9ptp4l0-21swapper/321:17:003
114691430,9ptp4l0-21swapper/221:46:442
114691430,7ptp4l0-21swapper/100:00:221
114691430,5ptp4l5376-21df_inode21:50:151
114691430,4ptp4l29139-21apt-get20:20:143
114691430,4ptp4l2650-21snmpd22:32:003
114691430,4ptp4l12438-21apt-get19:20:133
114691430,2ptp4l0-21swapper/321:34:043
114691430,2ptp4l0-21swapper/321:24:173
114691430,2ptp4l0-21swapper/320:00:523
114691430,2ptp4l0-21swapper/319:54:573
114691430,2ptp4l0-21swapper/222:42:292
114691430,2ptp4l0-21swapper/222:32:352
114691430,1ptp4l0-21swapper/323:07:403
114691430,1ptp4l0-21swapper/320:28:483
114691430,1ptp4l0-21swapper/019:42:040
55098420,5rtkit-daemon0-21swapper/322:41:493
293399425,4cyclictest114791getstats23:00:133
293199421,11cyclictest25099-21timerandwakeup22:55:121
114691420,6ptp4l5662-21apt-get23:50:151
114691420,6ptp4l29283-21acpi23:40:123
114691420,5ptp4l21788-21diskstats22:10:161
114691420,4ptp4l7889-21http00:35:000
114691420,4ptp4l25077-21fw_packets19:35:140
114691420,4ptp4l1569-21apt-get00:25:153
114691420,4ptp4l13379-21apt-get23:20:141
114691420,3ptp4l8904-21apt-get20:35:153
114691420,3ptp4l8904-21apt-get20:35:153
114691420,3ptp4l25-21ksoftirqd/119:44:251
114691420,2ptp4l0-21swapper/322:55:323
114691420,2ptp4l0-21swapper/321:45:513
114691420,2ptp4l0-21swapper/123:50:001
114691420,1ptp4l0-21swapper/119:34:151
114691420,1ptp4l0-21swapper/119:34:141
114691420,1ptp4l0-21swapper/023:06:450
293399410,5cyclictest21477-21gunzip23:30:133
2932994133,4cyclictest33-21ksoftirqd/221:25:052
114691410,6ptp4l4481-21/usr/sbin/munin20:30:133
114691410,6ptp4l28141-21timerwakeupswit20:20:081
114691410,5ptp4l9711-21memory19:15:331
114691410,5ptp4l9193-21tr22:35:140
114691410,5ptp4l6146-21ntp_states19:50:241
114691410,5ptp4l32535-21meminfo19:45:111
114691410,5ptp4l30365-21meminfo21:40:311
114691410,5ptp4l28716-21diskstats00:20:053
114691410,5ptp4l25968-21ntp_states19:35:263
114691410,5ptp4l25460-21processes22:15:151
114691410,5ptp4l25355-21phc2sys21:35:141
114691410,5ptp4l25-21ksoftirqd/120:00:171
114691410,5ptp4l23749-21fschecks_count20:55:011
114691410,5ptp4l22475-21ntp_states22:50:253
114691410,5ptp4l22193-21irqstats21:30:221
114691410,5ptp4l20285-21ntp_states22:50:071
114691410,5ptp4l1746-21gnome-shell19:39:581
114691410,5ptp4l16709-21ntp_states00:05:071
114691410,4ptp4l32177-21diskstats22:25:040
114691410,4ptp4l25426-21apt-get22:15:153
114691410,4ptp4l14998-21users22:40:311
114691410,3ptp4l4592-21apt-get21:50:113
114691410,2ptp4l0-21swapper/219:11:102
114691410,2ptp4l0-21swapper/200:06:472
114691410,1ptp4l0-21swapper/323:13:083
293199401,10cyclictest17665-21df_abs22:45:151
114691400,5ptp4l6040-21irqrtprio21:10:283
114691400,5ptp4l3821-21irqstats19:10:081
114691400,5ptp4l32732-21diskstats00:25:051
114691400,5ptp4l25465-21/usr/sbin/munin00:15:171
114691400,5ptp4l23472-21df_inode20:15:011
114691400,5ptp4l21100-21meminfo23:30:111
114691400,5ptp4l13275-21grep21:20:141
114691400,5ptp4l12221-21meminfo19:20:111
114691400,5ptp4l10231-21df00:35:151
114691400,4ptp4l30458-21iostat23:40:201
114691400,4ptp4l19692-21df22:50:010
114691400,4ptp4l18456-21ls22:05:253
114691400,4ptp4l16527-21apt-get19:25:141
114691400,3ptp4l15311-21ls19:25:033
114691400,2ptp4l0-21swapper/223:51:182
114691400,2ptp4l0-21swapper/123:55:361
114691400,2ptp4l0-21swapper/123:38:271
114691400,2ptp4l0-21swapper/021:45:040
2933993931,5cyclictest41-21ksoftirqd/321:40:583
2933993912,24cyclictest29163-21expr19:40:153
2933993912,24cyclictest24991-21expr23:35:103
293399390,5cyclictest27023-21sed20:55:413
293399390,4cyclictest9978-21sort23:15:173
293199393,7cyclictest5106-21apt-get21:10:151
2931993911,4cyclictest9200-21cpuspeed_turbos23:15:131
114691390,5ptp4l12829-21/usr/sbin/munin20:40:131
114691390,5ptp4l12829-21/usr/sbin/munin20:40:131
114691390,4ptp4l9164-21meminfo23:55:103
114691390,4ptp4l5960-21iostat_ios22:30:221
114691390,4ptp4l31099-21/usr/sbin/munin23:00:351
114691390,3ptp4l25668-21cpuspeed_turbos22:15:170
114691390,3ptp4l21433-21if_enp1s019:30:193
114691390,3ptp4l21433-21if_enp1s019:30:183
114691390,2ptp4l0-21swapper/122:25:151
114691390,2ptp4l0-21swapper/121:05:061
293399382,30cyclictest0-21swapper/323:50:153
293399380,6cyclictest24920-21apt-get21:35:143
114691380,4ptp4l7582-21fschecks_count22:35:011
114691380,3ptp4l0-21swapper/300:15:263
114691380,2ptp4l0-21swapper/322:00:153
114691380,2ptp4l0-21swapper/319:45:073
114691380,2ptp4l0-21swapper/120:45:291
293399373,30cyclictest0-21swapper/322:20:153
2933993731,3cyclictest41-21ksoftirqd/323:24:153
293399370,4cyclictest8225-21sort00:35:023
293399370,4cyclictest8047-21fschecks_time21:55:053
114691370,2ptp4l0-21swapper/323:45:073
114691370,2ptp4l0-21swapper/319:15:373
114691370,2ptp4l0-21swapper/300:30:353
114691370,2ptp4l0-21swapper/300:00:263
114691370,2ptp4l0-21swapper/120:14:201
114691370,1ptp4l0-21swapper/323:29:313
114691360,2ptp4l0-21swapper/319:14:583
114691360,2ptp4l0-21swapper/123:14:411
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional