You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-24 - 13:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 24, 2024 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
223392241162,25sleep00-21swapper/019:05:050
242982237161,24sleep30-21swapper/319:09:553
239642225194,21sleep20-21swapper/219:05:382
242742224193,20sleep10-21swapper/119:09:361
2440999106102,3cyclictest0-21swapper/200:38:422
130452980,4sleep232-21ksoftirqd/222:25:182
24409996662,3cyclictest0-21swapper/222:04:552
165652660,5sleep02439999cyclictest23:45:280
303732590,4sleep02439999cyclictest00:20:000
24409995954,3cyclictest0-21swapper/220:05:092
9552580,4sleep02439999cyclictest20:45:120
68102580,5sleep12440499cyclictest22:10:201
41132550,1sleep20-21swapper/200:30:312
24409995451,2cyclictest30-21rcuc/223:05:162
83092450,1sleep30-21swapper/322:15:113
2439999356,13cyclictest31953-21awk23:10:010
2440499291,26cyclictest0-21swapper/120:43:311
2439999296,11cyclictest24728-21awk21:40:010
2440499289,2cyclictest5125-21missed_timers23:20:211
2440499280,26cyclictest0-21swapper/123:05:021
2440499272,16cyclictest0-21swapper/123:41:131
2440499271,17cyclictest0-21swapper/100:07:101
2440499270,25cyclictest0-21swapper/122:47:021
2440499270,16cyclictest0-21swapper/100:05:021
211262270,4sleep339-21rcuc/322:45:013
198842270,3sleep019888-21missed_timers22:40:220
2440499262,15cyclictest0-21swapper/100:15:031
24404992614,3cyclictest0-21swapper/123:53:281
24404992613,9cyclictest3449-21irqstats00:30:191
2440499261,22cyclictest2634-21sensors_temp22:00:261
2441699255,7cyclictest11516-21diskmemload00:20:183
2440999251,5cyclictest0-21swapper/223:31:092
2440499258,3cyclictest28498-21munin-run20:35:011
2440499256,16cyclictest0-21swapper/119:55:221
2440499251,2cyclictest0-21swapper/123:05:361
2440499251,2cyclictest0-21swapper/120:05:351
2440499251,22cyclictest0-21swapper/122:29:281
2440499251,22cyclictest0-21swapper/121:58:031
2440499251,12cyclictest29677-21df21:50:141
257642240,1sleep10-21swapper/121:40:211
2440999242,4cyclictest0-21swapper/221:19:262
24409992417,4cyclictest6519-21tr23:25:112
2440499242,15cyclictest25071-21sshd22:51:481
24404992414,2cyclictest0-21swapper/123:55:281
2440499241,11cyclictest30073-21latency_hist19:25:021
2440499241,11cyclictest18109-21latency_hist21:25:021
24404992410,12cyclictest10415-21kworker/u8:123:35:521
2440499240,17cyclictest0-21swapper/121:35:571
2441699233,8cyclictest11528-40dnf21:10:043
2440999232,6cyclictest20311-21munin-run21:30:022
2440499235,10cyclictest15868-21hwlatdetect23:45:171
2440499232,19cyclictest0-21swapper/120:20:111
2440499231,3cyclictest5760-21latency_hist22:10:021
2440499231,3cyclictest27528-21latency_hist23:00:021
2440499231,3cyclictest21263-21if_enp1s021:30:191
2440499231,3cyclictest19119-21sed20:10:251
2440499231,3cyclictest14387-21ntp_states20:00:221
2440499231,3cyclictest14387-21ntp_states20:00:221
2440499230,5cyclictest0-21swapper/100:11:121
2440499230,15cyclictest0-21swapper/119:32:481
2440999225,9cyclictest86550irq/126-enp1s0-23:25:022
2440999225,4cyclictest23938-21gltestperf22:50:142
2440999224,3cyclictest3761-21kworker/u8:023:40:262
2440999223,9cyclictest4833-21/usr/sbin/munin22:05:272
2440499227,13cyclictest29724-21kworker/u8:123:12:541
2440499227,13cyclictest29724-21kworker/u8:123:12:541
2440499222,3cyclictest0-21swapper/121:02:251
2440499222,3cyclictest0-21swapper/121:02:251
2440499222,18cyclictest0-21swapper/121:17:461
2440499221,2cyclictest2816-21irqstats23:15:201
2440499221,19cyclictest0-21swapper/120:51:511
2440499221,19cyclictest0-21swapper/119:45:171
2440499221,11cyclictest26287-21sh20:30:011
2440499220,11cyclictest0-21swapper/121:28:421
24399992215,4cyclictest5900-21ntp_states00:35:250
2439999221,18cyclictest8370-21munin-run23:30:010
183042220,4sleep232-21ksoftirqd/220:10:142
325672211,2sleep20-21swapper/219:30:132
277792210,18sleep30-21swapper/320:30:233
2441699212,5cyclictest5182-21diskstats00:35:163
24416992116,3cyclictest12845-21cut22:25:163
24416992115,3cyclictest677-21tail19:30:173
24416992114,4cyclictest20237-21sed23:55:183
24416992110,3cyclictest5563-21awk22:10:003
2440999215,5cyclictest11516-21diskmemload22:34:562
2440999213,11cyclictest792-21avahi-daemon21:10:042
2440999212,5cyclictest21763-21gltestperf22:45:172
2440999212,15cyclictest0-21swapper/200:11:332
2440999211,6cyclictest17487-21grep23:50:122
2440999211,6cyclictest11516-21diskmemload00:25:012
2440999211,5cyclictest0-21swapper/200:17:352
2440999211,11cyclictest3514-21unixbench_multi23:15:302
2440999211,11cyclictest27196-21hddtemp_smartct20:30:142
2440999211,10cyclictest0-21swapper/223:14:242
2440999211,10cyclictest0-21swapper/223:14:242
2440499212,18cyclictest0-21swapper/122:44:141
2440499212,17cyclictest0-21swapper/122:37:491
2440499212,17cyclictest0-21swapper/100:25:231
24404992117,3cyclictest0-21swapper/119:16:471
2440499211,3cyclictest0-21swapper/120:48:191
24404992113,5cyclictest4717-21cstates19:40:121
2440499211,18cyclictest0-21swapper/100:36:481
2440499211,18cyclictest0-21swapper/100:21:591
2439999219,4cyclictest26204-21/usr/sbin/munin00:10:260
2439999217,5cyclictest0-21swapper/000:27:450
2439999215,2cyclictest86550irq/126-enp1s0-23:52:500
2439999211,11cyclictest18338-21/usr/sbin/munin21:25:190
2439999211,11cyclictest10169-21/usr/sbin/munin22:20:250
2441699202,11cyclictest22737-21/usr/sbin/munin21:35:263
2441699201,7cyclictest0-21swapper/323:20:193
24416992014,3cyclictest26711-21latency_hist19:15:013
2441699201,11cyclictest3247-21sshd22:03:483
2441699201,10cyclictest2977-21df_inode00:30:133
24416992010,2cyclictest31534-21/usr/sbin/munin21:55:213
2440999206,3cyclictest86550irq/126-enp1s0-20:45:192
2440999206,3cyclictest12050irq/123-ahci[0021:25:022
2440999204,8cyclictest86550irq/126-enp1s0-22:15:222
2440999204,3cyclictest25092-21cut21:40:132
2440999201,6cyclictest792-21avahi-daemon21:55:212
2440999201,6cyclictest10311-21mii-tool21:05:182
2440999201,12cyclictest0-21swapper/222:42:562
2440999201,11cyclictest0-21swapper/222:12:282
2440999201,11cyclictest0-21swapper/200:00:112
2440999200,7cyclictest7719-21needreboot19:45:222
2440499202,3cyclictest27951-21irqstats21:45:181
2440499202,17cyclictest0-21swapper/119:36:521
2440499201,8cyclictest0-21swapper/122:21:351
24404992016,3cyclictest0-21swapper/122:34:171
24404992016,3cyclictest0-21swapper/122:16:271
2440499201,3cyclictest0-21swapper/120:37:221
2440499201,18cyclictest0-21swapper/119:11:041
2440499201,17cyclictest0-21swapper/123:31:141
2440499201,11cyclictest12398-21cut21:10:151
2440499200,19cyclictest0-21swapper/120:59:571
2440499200,15cyclictest0-21swapper/123:25:221
2439999208,3cyclictest11521-21munin-run21:10:000
2439999203,14cyclictest15083-21sed21:15:240
2439999202,10cyclictest11751-21/usr/sbin/munin21:10:140
2439999201,13cyclictest0-21swapper/020:15:480
2439999201,11cyclictest2636-21/usr/sbin/munin00:30:190
2439999201,11cyclictest0-21swapper/022:26:350
2439999201,10cyclictest23656-21sshd00:03:470
266432191,3sleep30-21swapper/321:41:453
2441699198,4cyclictest842-21lldpd22:14:123
2441699198,3cyclictest27759-21/usr/sbin/munin23:00:243
2441699192,6cyclictest0-21swapper/323:10:483
2441699192,6cyclictest0-21swapper/323:10:483
2441699191,7cyclictest4464-21cat19:40:023
2441699191,7cyclictest1826-21ntp_states20:45:253
24416991913,3cyclictest25976-21cut22:55:133
24416991913,3cyclictest15040-21grep22:30:163
24416991912,4cyclictest22116-21kworker/u8:220:50:323
2441699191,10cyclictest28405-21/usr/sbin/munin00:15:223
2441699191,10cyclictest22249-21sshd21:31:493
2441699191,10cyclictest16765-21/usr/sbin/munin22:35:153
2441699191,10cyclictest11044-21cstates23:35:153
2440999198,3cyclictest4315-21awk19:40:022
2440999195,5cyclictest24007-21chrt20:24:072
2440999194,9cyclictest0-21swapper/200:26:422
2440999194,3cyclictest13277-21sed20:00:022
2440999193,9cyclictest0-21swapper/222:55:252
2440999193,8cyclictest86550irq/126-enp1s0-23:55:152
2440999193,10cyclictest0-21swapper/221:31:172
2440999192,9cyclictest0-21swapper/221:39:592
2440999191,6cyclictest3694-21if_enp2s020:50:192
2440999191,6cyclictest28529-21if_err_enp1s023:00:192
2440999191,6cyclictest0-21swapper/220:25:112
2440999191,5cyclictest792-21avahi-daemon19:50:232
2440999191,5cyclictest17102-21df22:35:162
24409991914,3cyclictest5145-21gltestperf19:40:172
2440999191,11cyclictest28216-21ntpq21:45:242
2440999191,10cyclictest0-21swapper/222:20:202
2440499191,2cyclictest0-21swapper/121:05:591
24404991912,4cyclictest20910-21tr20:15:211
2440499191,17cyclictest0-21swapper/119:50:561
2440499191,16cyclictest0-21swapper/119:26:371
2439999199,2cyclictest25954-21sshd20:25:480
2439999198,3cyclictest4121-21sshd19:37:500
2439999197,2cyclictest11516-21diskmemload22:54:330
2439999197,2cyclictest10710-21kworker/0:219:58:160
2439999196,2cyclictest22155-21kworker/u8:221:31:050
2439999191,8cyclictest24942-21/usr/sbin/munin21:40:260
24399991914,3cyclictest7252-21grep22:10:260
2439999191,3cyclictest1780-21munin-run23:15:010
2439999191,3cyclictest1780-21munin-run23:15:010
24399991913,3cyclictest30683-21turbostat.cron20:40:010
24399991912,4cyclictest836-21runrttasks19:30:280
24399991912,4cyclictest22772-21grep20:20:150
2439999191,16cyclictest29752-21awk23:05:010
2439999191,10cyclictest3579-21/usr/sbin/munin22:05:300
2439999191,10cyclictest14566-21/usr/sbin/munin22:30:270
2439999190,9cyclictest0-21swapper/022:35:360
2439999190,6cyclictest0-21swapper/022:59:470
168312191,2sleep30-21swapper/323:46:053
62832181,3sleep00-21swapper/019:42:220
38892181,2sleep00-21swapper/023:19:080
325572181,2sleep00-21swapper/020:40:550
322592180,2sleep00-21swapper/019:29:590
289432181,2sleep30-21swapper/321:48:033
273152181,2sleep00-21swapper/020:30:150
2441699187,3cyclictest29339-21/usr/sbin/munin21:50:163
2441699185,4cyclictest22468-21diskstats20:20:143
2441699183,5cyclictest11516-21diskmemload21:20:133
2441699182,5cyclictest21886-21latency_hist20:20:023
2441699181,9cyclictest30918-21/usr/sbin/munin20:40:223
2441699181,7cyclictest21500-21df_inode22:45:133
2441699181,7cyclictest10558-21latency_hist23:35:013
24416991813,3cyclictest29011-21expr20:35:123
24416991812,3cyclictest31045-21sh23:05:253
24416991811,4cyclictest9965-21grep21:05:153
24416991811,4cyclictest842-21lldpd22:22:123
2441699181,10cyclictest14634-21snmpd00:27:133
2441699181,10cyclictest0-21swapper/323:40:273
2440999184,7cyclictest86550irq/126-enp1s0-23:45:442
2440999184,3cyclictest29779-21chrt20:35:232
2440999181,9cyclictest25593-21needreboot19:10:212
2440999181,9cyclictest0-21swapper/221:50:542
2440999181,9cyclictest0-21swapper/219:25:122
2440999181,6cyclictest6136-21needreboot20:55:222
24409991814,2cyclictest15308-21awk20:05:012
24409991814,2cyclictest15308-21awk20:05:012
2440999181,10cyclictest0-21swapper/200:06:172
2439999186,3cyclictest7973-21/usr/sbin/munin22:15:120
2439999181,8cyclictest4193-21/usr/sbin/munin23:20:190
2439999181,8cyclictest13316-21/usr/sbin/munin20:00:200
2439999181,8cyclictest13316-21/usr/sbin/munin20:00:200
2439999181,8cyclictest10789-21/usr/sbin/munin23:35:240
2439999181,4cyclictest12988-21/usr/sbin/munin23:40:210
24399991814,2cyclictest8712-21awk19:50:010
24399991812,3cyclictest25142-21grep00:05:270
2439999181,14cyclictest29944-21head21:50:160
2439999181,11cyclictest0-21swapper/021:24:280
236482181,2sleep30-21swapper/300:03:433
124792180,3sleep20-21swapper/223:36:372
2441699174,6cyclictest27448-1kworker/u9:121:20:013
2441699172,12cyclictest26204-21/usr/sbin/munin00:10:233
2441699172,12cyclictest24314-21/usr/sbin/munin20:25:143
2441699172,12cyclictest18338-21/usr/sbin/munin21:25:273
2441699171,7cyclictest16317-21if_err_enp2s020:05:183
24416991712,3cyclictest8194-21unixbench-2d19:45:303
24416991712,3cyclictest24814-21kernelversion00:05:223
24416991712,3cyclictest23461-21grep22:50:093
24416991712,3cyclictest17792-21wc20:10:103
2441699171,14cyclictest0-21swapper/323:50:053
24416991711,3cyclictest6211-21uname20:55:243
24416991711,3cyclictest4784-21expr19:40:143
24416991711,3cyclictest27174-21cat19:15:043
2441699171,13cyclictest2001-21/usr/sbin/munin23:15:253
24416991710,4cyclictest7698-21/usr/sbin/munin23:25:253
24416991710,4cyclictest31151-21latency19:25:143
24416991710,4cyclictest14580-21smart_sda20:00:263
24416991710,4cyclictest14580-21smart_sda20:00:263
2440999171,2cyclictest30918-21/usr/sbin/munin20:40:122
24409991712,3cyclictest27684-21expr19:15:122
2439999177,3cyclictest86750irq/128-enp1s0-20:58:420
2439999176,9cyclictest86550irq/126-enp1s0-21:00:150
2439999176,9cyclictest86550irq/126-enp1s0-21:00:150
2439999171,9cyclictest0-21swapper/023:59:000
2439999171,4cyclictest0-21swapper/023:32:530
24399991711,4cyclictest0-21swapper/022:49:200
24399991711,3cyclictest32247-21cat21:55:170
24399991711,3cyclictest15523-21/usr/sbin/munin20:05:160
2439999171,10cyclictest0-21swapper/000:23:470
2441699161,6cyclictest11186-21cpuspeed_turbos19:55:133
2441699161,5cyclictest0-21swapper/319:50:253
2441699161,14cyclictest0-21swapper/319:24:553
2441699161,12cyclictest7998-21gltestperf21:00:203
2441699161,12cyclictest7998-21gltestperf21:00:203
2440999161,9cyclictest0-21swapper/219:24:592
2440999161,3cyclictest86550irq/126-enp1s0-20:15:172
24409991611,3cyclictest7657-21expr21:00:152
24409991611,3cyclictest7657-21expr21:00:152
2439999161,9cyclictest0-21swapper/022:02:410
24399991611,3cyclictest3825-21sed20:50:200
24399991611,3cyclictest27702-21cut21:45:140
24399991611,3cyclictest24598-21acpi19:10:110
2439999161,12cyclictest8927-21/usr/sbin/munin19:50:150
24399991610,3cyclictest27376-21if_enp1s019:15:050
2439999160,8cyclictest0-21swapper/019:22:490
195742161,2sleep00-21swapper/020:13:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional