You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-21 - 21:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Atom(TM) Processor E3950 @ 1.60GHz, Linux 5.2.21-rt13 (Profile)

Latency plot of shadow in rack #b, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackbslot4s.osadl.org (updated Mon Sep 30, 2019 00:45:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
683999584057584046,9cyclictest950ksoftirqd/023:23:400
68429953265316,8cyclictest24511-21kworker/3:223:24:093
68399952375218,17cyclictest16459-21kworker/0:122:51:090
2959421850,5sleep0683999cyclictest23:15:070
973621520,4sleep0683999cyclictest20:25:030
6659211268,16sleep30-21swapper/319:09:363
6667211066,16sleep10-21swapper/119:09:421
6663210867,16sleep20-21swapper/219:09:382
638029571,16sleep00-21swapper/019:06:070
1721989880,1phc2sys0-21swapper/100:35:291
1721690710,1ptp4l21-21ksoftirqd/123:20:141
177102700,1sleep00-21swapper/020:40:120
1721989690,1phc2sys0-21swapper/123:01:391
1721989660,1phc2sys0-21swapper/123:58:581
1721989650,1phc2sys0-21swapper/123:19:561
1721989640,1phc2sys0-21swapper/100:16:251
1721989640,1phc2sys0-21swapper/100:16:251
1721989630,2phc2sys684099cyclictest23:09:371
37002600,1sleep10-21swapper/123:30:011
1721989600,3phc2sys684099cyclictest23:53:241
193102590,2sleep30-21swapper/320:45:043
158922590,3sleep00-21swapper/020:35:180
1721690570,0ptp4l21-21ksoftirqd/123:35:191
1721690560,1ptp4l21-21ksoftirqd/100:05:121
1721989550,1phc2sys0-21swapper/100:10:041
116187550,2custom_interrup0-21swapper/223:23:402
296542540,4sleep3684299cyclictest22:15:043
1721989540,1phc2sys0-21swapper/100:32:311
1721690540,0ptp4l21-21ksoftirqd/100:27:581
1721690530,0ptp4l82050irq/133-enp2s0-00:00:081
1721690520,0ptp4l21-21ksoftirqd/122:51:191
1721989500,1phc2sys0-21swapper/122:57:271
1721989490,1phc2sys0-21swapper/123:30:281
1721690480,1ptp4l0-21swapper/123:40:401
143082480,1sleep314303-21strings23:50:133
1721989460,1phc2sys0-21swapper/123:45:471
1721989460,1phc2sys0-21swapper/123:45:471
1721690440,1ptp4l21-21ksoftirqd/123:11:421
6839994311,15cyclictest13713-21munin-run19:25:010
683999429,15cyclictest28667-21munin-run23:15:010
1721690410,1ptp4l0-21swapper/100:21:011
683999389,13cyclictest0-21swapper/019:55:010
683999386,15cyclictest380-21munin-run21:15:010
684199373,11cyclictest20656-21munin-run19:40:012
683999362,11cyclictest8319-21munin-run23:40:010
79122350,4sleep228-21ksoftirqd/223:35:192
6840993520,12cyclictest724-21NetworkManager22:29:571
683999350,32cyclictest444-21systemd-journal22:30:380
683999340,32cyclictest796-21systemd-logind22:17:270
683999340,31cyclictest796-21systemd-logind23:47:580
683999340,31cyclictest796-21systemd-logind23:47:570
6842993328,3cyclictest707-21avahi-daemon22:28:273
6841993328,3cyclictest707-21avahi-daemon22:15:252
6839993327,4cyclictest724-21NetworkManager22:47:280
683999330,4cyclictest0-21swapper/022:39:500
683999330,30cyclictest796-21systemd-logind19:44:160
6842993227,3cyclictest103-21printk22:51:083
683999320,5cyclictest11236-21cc122:43:260
683999320,5cyclictest11236-21cc122:43:260
684099311,27cyclictest444-21systemd-journal21:45:021
684099311,27cyclictest444-21systemd-journal21:10:001
6839993114,15cyclictest5474-21/usr/sbin/munin21:25:140
6839993112,16cyclictest23174-21/usr/sbin/munin19:45:220
6841993025,3cyclictest17166-21ifconfig22:51:092
6840993012,16cyclictest29336-21/usr/sbin/munin22:15:111
684099300,27cyclictest444-21systemd-journal21:24:321
6839993028,1cyclictest724-21NetworkManager21:37:430
6839993028,1cyclictest724-21NetworkManager21:37:430
684099292,23cyclictest11530-21cc122:43:351
684099292,23cyclictest11530-21cc122:43:351
684099291,25cyclictest444-21systemd-journal21:52:241
6840992912,15cyclictest14141-21ls19:25:031
6840992911,16cyclictest26983-21sh21:00:101
6840992911,15cyclictest22982-21latency_hist19:45:001
684099290,3cyclictest24446-21munin-run22:05:021
684099290,26cyclictest444-21systemd-journal22:49:541
683999299,16cyclictest28019-21fschecks_time00:20:060
683999291,25cyclictest677-21systemd-machine22:07:000
6839992912,14cyclictest3992-21if_enp1s021:20:090
683999290,14cyclictest0-21swapper/019:40:010
6842992825,1cyclictest724-21NetworkManager22:05:293
6841992824,2cyclictest15644-21kworker/2:222:13:282
6841992823,2cyclictest724-21NetworkManager21:56:432
684099289,16cyclictest7600-21sadc21:30:001
684099288,17cyclictest24181-21cut19:45:081
684099288,17cyclictest12659-21/usr/sbin/munin19:20:111
684099282,23cyclictest20071-21cut21:55:061
684099281,22cyclictest8614-21dnf22:39:491
6840992811,14cyclictest7862-21interrupts19:10:101
684099280,3cyclictest12432-21munin-run21:40:011
684099280,3cyclictest12432-21munin-run21:40:011
684099280,2cyclictest10791-21ntp_kernel_pll_20:25:141
684099280,25cyclictest796-21systemd-logind20:05:451
683999283,13cyclictest17644-21munin-run00:00:010
683999282,6cyclictest25373-21cc123:05:530
6839992812,13cyclictest22610-21fschecks_time22:00:070
6839992812,13cyclictest1546-21cut22:25:010
6839992811,13cyclictest23712-21cat20:55:040
6839992810,15cyclictest13142-21cut21:40:070
6839992810,14cyclictest1155-21switchtime00:30:170
683999280,25cyclictest677-21systemd-machine22:30:010
6842992723,2cyclictest3862-21kworker/3:222:38:423
6840992723,2cyclictest724-21NetworkManager21:46:481
6840992722,3cyclictest724-21NetworkManager22:14:441
684099271,23cyclictest444-21systemd-journal21:30:281
6840992710,15cyclictest18896-21/usr/sbin/munin20:45:071
6840992710,15cyclictest1862-21nfs4_client21:15:121
684099270,3cyclictest25445-21iostat22:05:101
6839992711,13cyclictest18125-21sed22:55:040
6839992710,14cyclictest580-21/usr/sbin/munin21:15:180
6839992710,14cyclictest30270-21fschecks_count00:25:060
683999270,12cyclictest796-21systemd-logind21:01:360
6842992621,2cyclictest724-21NetworkManager21:38:273
6842992621,2cyclictest724-21NetworkManager21:38:273
684099269,14cyclictest9413-21latency_hist20:25:021
684099260,3cyclictest17060-21cut20:40:051
684099260,24cyclictest444-21systemd-journal22:23:321
684099260,23cyclictest28520-21cut19:55:051
683999263,7cyclictest22415-21ld23:03:200
683999263,5cyclictest31710-21smart_sda20:00:170
683999262,5cyclictest0-21swapper/000:07:120
683999261,11cyclictest3378-21fw_conntrack20:10:070
6839992610,13cyclictest16458-21cat21:45:180
6842992521,2cyclictest724-21NetworkManager21:53:363
6841992521,2cyclictest8339-21kworker/2:022:37:582
684099258,14cyclictest363-21latency_hist21:15:011
684099257,15cyclictest25705-21/usr/sbin/munin19:50:131
684099257,15cyclictest16807-21sort19:30:051
684099256,16cyclictest21432-21expr20:50:031
684099254,18cyclictest6861-21sshd20:17:101
684099251,4cyclictest3699-21memory20:10:121
684099251,21cyclictest444-21systemd-journal22:30:121
684099250,3cyclictest14145-21munin-run20:35:021
684099250,2cyclictest0-21swapper/120:00:171
684099250,2cyclictest0-21swapper/119:15:101
684099250,22cyclictest796-21systemd-logind19:39:401
683999259,3cyclictest19184-21processes00:00:180
683999252,2cyclictest299562sleep019:57:510
683999250,13cyclictest796-21systemd-logind20:09:160
684299241,20cyclictest796-21systemd-logind20:53:553
684199241,20cyclictest444-21systemd-journal20:45:012
684099245,16cyclictest24178-21/usr/sbin/munin20:55:041
684099241,19cyclictest15624-21smart_sda20:35:171
683999249,2cyclictest25011-21/usr/sbin/munin00:15:190
683999249,2cyclictest25011-21/usr/sbin/munin00:15:190
683999248,3cyclictest8041-21expr20:20:080
683999248,3cyclictest19587-21fschecks_time20:45:070
683999248,3cyclictest14550-21smart_sda23:50:180
683999248,3cyclictest10109-21expr23:40:170
683999243,5cyclictest30230-40dnf21:06:530
683999242,3cyclictest24136-21smart_sda00:10:170
683999240,21cyclictest444-21systemd-journal21:33:510
683999240,11cyclictest444-21systemd-journal19:27:330
683999240,10cyclictest69212sleep020:17:530
6842992319,2cyclictest0-21swapper/320:30:013
684299230,20cyclictest796-21systemd-logind00:03:283
6841992319,2cyclictest707-21avahi-daemon22:44:282
6841992319,2cyclictest707-21avahi-daemon22:44:282
6841992319,2cyclictest0-21swapper/223:00:002
6841992318,3cyclictest724-21NetworkManager21:30:292
684199231,19cyclictest479-21systemd-udevd22:07:012
684199230,20cyclictest796-21systemd-logind00:13:502
684199230,20cyclictest479-21systemd-udevd23:05:012
684199230,20cyclictest444-21systemd-journal20:05:012
683999237,3cyclictest139552chrt20:33:530
683999235,4cyclictest28792-21cargo22:11:480
683999231,5cyclictest5257-21sort23:30:170
683999230,9cyclictest3306-21runrttasks23:25:510
6842992211,2cyclictest17611-21memory20:40:103
684299220,19cyclictest796-21systemd-logind22:12:173
684199221,19cyclictest444-21systemd-journal23:07:362
684199220,19cyclictest444-21systemd-journal20:10:022
683999227,3cyclictest7915-21cut19:10:110
683999225,3cyclictest2499-21df_abs00:35:040
683999224,5cyclictest19357-21nm-dhcp-helper21:54:440
683999222,5cyclictest17593-21smart_sda19:30:160
683999221,6cyclictest10721-21chrt19:15:170
683999220,19cyclictest796-21systemd-logind21:56:360
37642220,2sleep30-21swapper/300:35:203
684299213,12cyclictest10604-21dnf22:40:293
684299213,12cyclictest10604-21dnf22:40:293
6842992115,3cyclictest724-21NetworkManager21:33:233
684299210,19cyclictest444-21systemd-journal22:45:173
684299210,18cyclictest21747-21turbostat22:00:013
6841992110,6cyclictest81050irq/127-enp1s0-23:11:132
684199210,19cyclictest677-21systemd-machine19:55:362
684199210,19cyclictest444-21systemd-journal20:55:342
684199210,18cyclictest796-21systemd-logind22:31:072
684199210,18cyclictest479-21systemd-udevd19:45:022
683999216,3cyclictest21387-21cstates20:50:030
284552210,1sleep30-21swapper/321:05:023
684299204,6cyclictest30491-21dnf23:15:263
684299201,9cyclictest0-21swapper/321:12:153
6842992015,3cyclictest724-21NetworkManager21:24:363
6842992014,4cyclictest20192-21/usr/sbin/munin23:00:143
684299200,4cyclictest25389-21lto1-ltrans23:05:553
684299200,18cyclictest796-21systemd-logind22:32:263
684299200,18cyclictest444-21systemd-journal21:47:363
684299200,17cyclictest444-21systemd-journal22:03:143
684199209,8cyclictest5829-21idleruntime20:15:072
684199204,5cyclictest30230-40dnf21:06:522
6841992014,4cyclictest12544-21/usr/sbin/munin21:40:102
6841992013,4cyclictest766-21sssd_nss21:50:172
684199201,16cyclictest444-21systemd-journal20:15:002
684199200,17cyclictest479-21systemd-udevd00:05:022
684299193,6cyclictest28108-21yum23:11:143
6842991914,3cyclictest724-21NetworkManager21:44:583
6842991914,3cyclictest32096-21acpi00:30:033
6842991914,3cyclictest18949-21ntpq22:55:143
684299191,3cyclictest17261-21memory19:30:113
6842991913,4cyclictest7658-21munin-plugin-st21:30:003
6842991913,4cyclictest6956-21/usr/sbin/munin19:10:173
6842991913,3cyclictest27826-21sed00:20:053
6842991911,5cyclictest6370-21cat23:35:043
684299191,15cyclictest27096-21memory21:00:123
684299190,17cyclictest796-21systemd-logind00:07:193
6841991915,2cyclictest10201-21fschecks_time20:25:062
6841991914,3cyclictest724-21NetworkManager22:23:282
6841991914,3cyclictest724-21NetworkManager21:47:322
6841991914,3cyclictest3112-21/usr/sbin/munin21:20:192
6841991914,3cyclictest13158-21open_files20:30:142
6841991913,4cyclictest30894-21lto1-ltrans23:16:162
6841991913,3cyclictest4219-21sed23:30:052
6841991913,3cyclictest31643-21iostat_ios21:10:112
684199190,16cyclictest444-21systemd-journal22:47:262
684299185,5cyclictest0-21swapper/300:25:183
684299182,13cyclictest26496-21grep19:50:083
6842991814,2cyclictest31710-21smart_sda20:00:163
6842991813,3cyclictest9273-21/usr/sbin/munin19:15:133
6842991813,3cyclictest5366-21sed23:30:183
6842991813,3cyclictest32668-21/usr/sbin/munin20:05:093
6842991813,3cyclictest28733-21fw_conntrack19:55:063
6842991813,3cyclictest25498-21sed00:15:053
6842991813,3cyclictest25498-21sed00:15:043
6842991813,3cyclictest24599-21basename20:55:093
6842991813,3cyclictest20859-21/usr/sbin/munin19:40:053
6842991813,3cyclictest13922-21cat19:25:023
6842991812,3cyclictest22799-21expr00:10:043
684299181,15cyclictest444-21systemd-journal20:13:333
6842991810,6cyclictest1772450kworker/3:1H00:00:013
6841991813,3cyclictest724-21NetworkManager22:03:132
6841991813,3cyclictest2083-21/usr/sbin/munin00:35:152
6841991813,3cyclictest14833-21iostat_ios19:25:122
6841991813,3cyclictest10154-21/usr/sbin/munin21:35:162
6841991813,3cyclictest10154-21/usr/sbin/munin21:35:152
6841991812,3cyclictest26873-21expr21:00:092
6841991812,3cyclictest22544-21date00:10:032
6841991812,3cyclictest19242-21/usr/sbin/munin20:45:052
6841991811,4cyclictest11631-21sed23:45:092
6841991811,4cyclictest11631-21sed23:45:092
684199180,3cyclictest8502-21ntp_states20:20:142
684199180,16cyclictest677-21systemd-machine22:26:572
684299172,12cyclictest12343-21wc23:45:183
684299172,12cyclictest12343-21wc23:45:183
6842991713,2cyclictest724-21NetworkManager22:21:573
6842991712,3cyclictest14186-21/usr/sbin/munin20:35:123
684299171,13cyclictest444-21systemd-journal20:25:013
684199176,9cyclictest1-21systemd23:30:002
684199175,9cyclictest23319-21crond20:55:012
6841991712,3cyclictest8588-21cpuspeed_turbos23:40:032
6841991712,3cyclictest6498-21cat21:25:122
6841991712,3cyclictest27153-21ls19:50:192
6841991711,3cyclictest7029-21ls19:10:032
684199171,13cyclictest26687-21unixbench-2d00:15:192
684199171,13cyclictest26687-21unixbench-2d00:15:192
684199171,13cyclictest13645-21grep23:50:072
6841991710,4cyclictest9711-21sh19:15:052
6842991612,2cyclictest18999-21forks19:35:053
6842991611,3cyclictest6012-21cut20:15:113
6842991611,3cyclictest2213-21fw_conntrack23:25:083
6842991611,3cyclictest12861-21iostat20:30:103
6842991611,3cyclictest12172-21fschecks_time19:20:063
684299160,14cyclictest0-21swapper/321:17:003
6841991612,2cyclictest747-21runrttasks00:25:112
6841991611,3cyclictest32526-21sed00:30:042
6841991611,3cyclictest27509-21cat00:20:032
6841991611,3cyclictest23246-21cpuspeed_turbos19:45:032
6841991611,3cyclictest12751-21nfs4_client19:20:132
6841991610,3cyclictest17124-21/usr/sbin/munin19:30:092
684199160,9cyclictest17644-21munin-run00:00:012
684199160,14cyclictest0-21swapper/220:35:132
6842991511,2cyclictest9564-21missed_timers23:40:113
6842991511,2cyclictest24534-21netstat19:45:133
6841991510,3cyclictest929-21ls21:15:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional