You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-25 - 11:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Jul 25, 2024 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
161472239163,24sleep20-21swapper/219:08:022
160522237161,24sleep00-21swapper/019:06:470
161742217164,40sleep10-21swapper/119:08:211
160902202166,24sleep30-21swapper/319:07:173
16395999894,3cyclictest0-21swapper/219:19:502
16395998885,2cyclictest0-21swapper/221:31:342
16395998277,4cyclictest0-21swapper/223:44:532
247732600,2sleep20-21swapper/223:05:262
279682590,1sleep027963-21cpuspeed_turbos23:15:110
237082480,4sleep123-21ksoftirqd/120:35:271
16389993710,6cyclictest23-21ksoftirqd/122:20:211
16380993610,12cyclictest24855-21basename21:55:020
1638999341,16cyclictest31763-21cut22:10:091
152112340,4sleep339-21rcuc/300:00:023
1638999331,24cyclictest0-21swapper/120:49:281
1638099336,12cyclictest7277-21perf21:15:010
1638099334,13cyclictest21937-21munin-run00:30:000
1638999322,14cyclictest0-21swapper/123:21:341
1638999321,12cyclictest0-21swapper/119:10:161
1638999321,12cyclictest0-21swapper/119:10:151
1638999309,18cyclictest29838-21latency_hist23:20:001
1638999301,12cyclictest10679-21strings21:20:221
1638999291,14cyclictest0-21swapper/123:29:051
1638999291,12cyclictest0-21swapper/121:10:561
1638999290,11cyclictest0-21swapper/123:03:211
1638099290,13cyclictest27049-21awk22:00:020
1638999271,7cyclictest0-21swapper/123:56:071
1638999271,17cyclictest16402-21tr21:35:131
1638999271,11cyclictest0-21swapper/122:40:171
1640099261,6cyclictest39-21rcuc/323:20:203
1638999269,4cyclictest221ktimersoftd/100:11:271
1638099261,13cyclictest25851-21idleruntime-cro19:30:010
50462250,5sleep09-21rcu_preempt23:35:170
50462250,5sleep09-21rcu_preempt23:35:170
269102250,4sleep230-21rcuc/220:45:112
1639599251,6cyclictest0-21swapper/222:08:122
1638999259,3cyclictest11367-21sh22:35:251
1638999251,8cyclictest0-21swapper/122:55:151
1638999250,14cyclictest0-21swapper/122:15:291
1638099251,13cyclictest4086-21awk23:35:000
10092250,3sleep121-21rcuc/121:00:121
192952241,3sleep019388-21/usr/sbin/munin21:40:180
16395992411,3cyclictest9343-21taskset23:45:162
1638999241,3cyclictest26268-21cstates19:30:111
1638999241,16cyclictest0-21swapper/119:55:121
1638099241,12cyclictest1264-21awk22:15:010
1638099241,11cyclictest18870-21idleruntime-cro22:55:020
1640099239,3cyclictest0-21swapper/320:35:193
1639599231,5cyclictest0-21swapper/221:49:302
1638999237,2cyclictest0-21swapper/121:44:221
1638999234,5cyclictest23636-40dnf19:25:031
1638999233,3cyclictest0-21swapper/121:31:201
1638999233,16cyclictest16644-21grep00:00:261
1638999232,4cyclictest30829-21munin-run20:55:021
1638999231,8cyclictest0-21swapper/121:15:041
16389992317,3cyclictest24757-21cstates20:40:131
16389992313,7cyclictest0-21swapper/122:26:231
1638999231,16cyclictest0-21swapper/121:50:391
1638999231,16cyclictest0-21swapper/120:10:171
16389992311,4cyclictest1-21systemd22:50:091
1638999230,17cyclictest0-21swapper/120:27:151
1638999230,16cyclictest0-21swapper/121:05:061
1638999230,10cyclictest0-21swapper/123:44:251
1638999230,10cyclictest0-21swapper/123:39:211
1638999230,10cyclictest0-21swapper/123:39:211
1638099236,5cyclictest16834-21grep22:50:020
225352221,4sleep222646-21timerandwakeup23:00:252
1640099221,11cyclictest0-21swapper/321:37:503
1639599224,15cyclictest30489-21cut23:20:132
16395992216,3cyclictest2862-21idleruntime-cro21:05:002
1639599221,10cyclictest0-21swapper/200:16:042
1638999225,2cyclictest0-21swapper/122:00:191
1638999221,8cyclictest0-21swapper/120:17:401
16389992215,1cyclictest0-21swapper/120:00:281
16389992213,3cyclictest0-21swapper/119:51:231
1638999221,16cyclictest0-21swapper/119:15:461
1638999221,15cyclictest0-21swapper/121:49:101
16389992211,4cyclictest0-21swapper/121:55:271
1638999221,11cyclictest2522-21sort23:30:131
1638999221,10cyclictest0-21swapper/120:06:481
1638999220,9cyclictest0-21swapper/100:07:151
1638099227,6cyclictest0-21swapper/022:18:050
1638099225,15cyclictest14455-21munin-run22:45:010
250142210,5sleep230-21rcuc/219:25:222
1640099219,3cyclictest0-21swapper/322:57:163
1640099212,16cyclictest9656-21head22:30:283
1640099212,16cyclictest9656-21head22:30:283
1640099212,12cyclictest842-21lldpd19:43:383
1640099211,6cyclictest0-21swapper/322:51:173
1640099211,13cyclictest0-21swapper/321:59:153
1640099211,11cyclictest14634-21snmpd23:00:223
1640099211,11cyclictest0-21swapper/319:48:103
1640099210,5cyclictest0-21swapper/322:40:173
16389992117,3cyclictest0-21swapper/123:45:231
1638999211,15cyclictest0-21swapper/123:05:491
1638999211,15cyclictest0-21swapper/122:07:461
1638999211,15cyclictest0-21swapper/120:34:241
1638999211,15cyclictest0-21swapper/119:42:061
1638999211,15cyclictest0-21swapper/100:31:161
1638999211,15cyclictest0-21swapper/100:29:101
1638999211,11cyclictest17646-21latency_hist20:25:021
1638999211,11cyclictest17646-21latency_hist20:25:021
1638999211,10cyclictest0-21swapper/123:13:011
1638999210,15cyclictest0-21swapper/119:36:201
1638099218,7cyclictest86550irq/126-enp1s0-21:17:500
1638099211,11cyclictest16740-21ntp_offset20:20:220
1638099211,11cyclictest16740-21ntp_offset20:20:220
1640099201,8cyclictest0-21swapper/323:33:553
1640099201,13cyclictest0-21swapper/321:40:133
1640099200,9cyclictest0-21swapper/320:45:213
1640099200,15cyclictest0-21swapper/321:25:103
1640099200,15cyclictest0-21swapper/321:25:093
1639599201,6cyclictest0-21swapper/222:58:342
1639599201,5cyclictest0-21swapper/221:40:352
1638999206,4cyclictest610-21grep19:45:161
1638999206,3cyclictest0-21swapper/120:55:181
1638999201,8cyclictest0-21swapper/122:32:031
1638999201,8cyclictest0-21swapper/122:32:031
16389992012,2cyclictest0-21swapper/122:45:061
1638999201,18cyclictest0-21swapper/123:53:541
1638999201,18cyclictest0-21swapper/100:36:431
1638999201,18cyclictest0-21swapper/100:23:321
1638999200,15cyclictest0-21swapper/121:28:281
1638999200,15cyclictest0-21swapper/121:28:281
1638099209,8cyclictest15429-21tr00:00:120
1638099209,3cyclictest22464-21sshd23:00:220
1638099206,5cyclictest0-21swapper/022:20:510
1638099200,8cyclictest0-21swapper/023:57:530
283492190,2sleep2311ktimersoftd/219:35:102
1640099197,4cyclictest39-21rcuc/323:35:153
1640099197,4cyclictest39-21rcuc/323:35:153
1640099196,3cyclictest0-21swapper/319:40:003
1640099195,5cyclictest0-21swapper/300:07:443
1640099192,15cyclictest0-21swapper/323:12:403
1640099191,7cyclictest2979-1kworker/u9:000:22:263
1640099191,4cyclictest0-21swapper/323:19:503
1640099191,4cyclictest0-21swapper/300:10:123
16400991911,5cyclictest32043-21strings20:55:213
1640099191,12cyclictest0-21swapper/321:15:123
1640099191,12cyclictest0-21swapper/300:38:363
1639599192,6cyclictest5065-21diskmemload22:25:002
1639599191,7cyclictest0-21swapper/221:23:042
16395991914,3cyclictest27474-21grep19:30:252
16395991914,3cyclictest26540-21grep21:55:312
16395991914,3cyclictest1839-21grep19:45:282
16395991913,3cyclictest11652-21grep20:10:132
1638999191,3cyclictest0-21swapper/100:18:491
1638999191,10cyclictest22493-21kernelversion19:20:161
1638099198,3cyclictest9687-21/usr/sbin/munin21:20:230
1638099198,2cyclictest0-21swapper/019:31:220
1638099195,4cyclictest10894-21grep22:35:190
1638099192,6cyclictest28999-21kworker/0:122:07:060
1638099191,9cyclictest0-21swapper/000:36:080
1638099191,8cyclictest20762-21cpuspeed_turbos21:45:130
1638099191,3cyclictest0-21swapper/021:25:320
1638099191,3cyclictest0-21swapper/021:25:320
1638099191,12cyclictest0-21swapper/020:50:220
1638099191,11cyclictest0-21swapper/023:42:070
1638099191,10cyclictest17875-21/usr/sbin/munin20:25:100
1638099191,10cyclictest0-21swapper/000:30:540
1638099190,5cyclictest2979-1kworker/u9:000:09:400
183412181,2sleep00-21swapper/000:12:560
1640099185,6cyclictest39-21rcuc/321:08:153
1640099185,3cyclictest0-21swapper/319:28:043
1640099185,3cyclictest0-21swapper/319:21:303
1640099183,2cyclictest10573-21sort21:20:193
1640099183,12cyclictest24991-21tail23:05:273
1640099182,4cyclictest0-21swapper/320:34:013
1640099182,15cyclictest0-21swapper/319:33:383
1640099181,8cyclictest0-21swapper/322:45:243
1640099181,3cyclictest0-21swapper/322:15:423
16400991813,3cyclictest450-21mailstats22:10:223
16400991812,3cyclictest27266-21/usr/sbin/munin22:00:253
16400991812,3cyclictest17954-21cat20:25:103
1640099181,15cyclictest0-21swapper/319:50:023
16400991811,4cyclictest848-21sssd_nss21:10:373
16400991811,4cyclictest20786-21tr21:45:133
1640099181,13cyclictest0-21swapper/322:20:083
1640099181,10cyclictest842-21lldpd20:43:073
1640099181,10cyclictest0-21swapper/323:43:293
1639599186,4cyclictest39512cat19:50:262
1639599185,4cyclictest29246-21perf22:05:012
1639599183,7cyclictest86550irq/126-enp1s0-00:00:282
1639599181,7cyclictest27631-21cat23:14:572
1639599181,7cyclictest0-21swapper/223:57:372
1639599181,7cyclictest0-21swapper/200:29:052
1639599181,6cyclictest0-21swapper/222:35:222
1639599181,5cyclictest0-21swapper/222:13:312
16395991813,3cyclictest5037-21if_enp1s023:35:182
16395991813,3cyclictest5037-21if_enp1s023:35:182
1639599181,11cyclictest0-21swapper/200:39:122
1638099186,4cyclictest0-21swapper/000:21:440
1638099186,3cyclictest0-21swapper/022:56:110
1638099181,7cyclictest0-21swapper/020:15:170
1638099181,3cyclictest16921-21df19:10:100
1638099181,3cyclictest16921-21df19:10:100
16380991812,3cyclictest26372-21sshd20:44:220
16380991812,3cyclictest19445-21sort19:15:050
1638099181,15cyclictest5061-21turbostat.cron21:10:010
1638099181,11cyclictest0-21swapper/021:34:310
1638099181,10cyclictest0-21swapper/022:33:030
1638099181,10cyclictest0-21swapper/022:33:030
1638099180,9cyclictest0-21swapper/019:51:220
141622181,2sleep20-21swapper/222:41:252
97172170,3sleep20-21swapper/222:30:412
97172170,3sleep20-21swapper/222:30:412
8322171,2sleep00-21swapper/019:45:170
75332171,3sleep20-21swapper/222:25:562
27592171,2sleep30-21swapper/321:03:483
1640099173,5cyclictest0-21swapper/323:45:123
1640099172,13cyclictest29998-21grep22:05:153
1640099172,12cyclictest12347-21ntp_states20:10:213
1640099171,9cyclictest15676-21/usr/sbin/munin20:20:143
1640099171,9cyclictest15676-21/usr/sbin/munin20:20:143
1640099171,7cyclictest0-21swapper/323:25:253
1640099171,4cyclictest0-21swapper/300:26:243
1640099171,3cyclictest0-21swapper/320:52:363
16400991713,3cyclictest0-21swapper/320:10:003
16400991713,3cyclictest0-21swapper/319:15:183
16400991712,3cyclictest5390-21sort19:55:153
16400991712,3cyclictest14762-21hwlatdetect21:30:173
1640099171,15cyclictest0-21swapper/321:51:193
1640099171,10cyclictest0-21swapper/323:52:523
1640099171,10cyclictest0-21swapper/320:15:183
1640099170,10cyclictest0-21swapper/300:00:153
1639599175,3cyclictest2979-1kworker/u9:021:35:132
1639599173,5cyclictest30002-21tr20:50:232
1639599172,3cyclictest0-21swapper/220:00:552
1639599171,7cyclictest0-21swapper/220:55:292
1639599171,6cyclictest25526-21sshd20:40:212
1639599171,6cyclictest18756-21cat19:15:022
1639599171,6cyclictest18756-21cat19:15:022
1639599171,6cyclictest0-21swapper/200:34:352
1639599171,6cyclictest0-21swapper/200:20:572
1639599171,5cyclictest0-21swapper/223:32:282
1639599171,3cyclictest0-21swapper/223:26:562
16395991712,3cyclictest4166-21ntp_offset21:05:322
16395991712,3cyclictest17570-21hwlatdetect22:50:172
16395991711,4cyclictest24336-21cat21:50:292
1639599171,11cyclictest0-21swapper/200:12:062
16395991710,4cyclictest18453-21cut20:25:142
1638099176,3cyclictest86750irq/128-enp1s0-23:20:250
1638099176,3cyclictest86750irq/128-enp1s0-20:45:260
1638099174,8cyclictest7570-40dnf22:26:250
1638099173,7cyclictest23051-21kworker/u8:023:08:100
1638099172,8cyclictest32610-21diskstats23:25:110
1638099171,6cyclictest0-21swapper/020:30:190
1638099171,6cyclictest0-21swapper/000:18:070
16380991712,3cyclictest23590-21seq19:24:330
16380991712,3cyclictest10261-21grep23:45:260
1638099171,15cyclictest0-21swapper/019:41:230
16380991711,3cyclictest28320-21grep22:00:250
1638099171,13cyclictest28259-21/usr/sbin/munin19:35:170
16380991711,3cyclictest22635-21awk20:35:150
16380991711,3cyclictest16955-21cut21:35:180
1638099171,11cyclictest0-21swapper/020:55:160
16380991710,4cyclictest11533-21cut23:50:130
1638099170,5cyclictest0-21swapper/023:11:570
126642171,2sleep20-21swapper/223:52:432
108832171,2sleep20-21swapper/220:07:592
1640099169,4cyclictest7001-21cpuspeed_turbos20:00:113
1640099164,10cyclictest11850-21head22:35:313
1640099161,6cyclictest0-21swapper/300:33:493
1640099161,14cyclictest0-21swapper/300:18:553
16400991611,3cyclictest6514-21gltestperf22:25:133
1640099161,12cyclictest16587-21/usr/sbin/munin19:10:113
1640099161,12cyclictest16587-21/usr/sbin/munin19:10:113
1639599164,4cyclictest6233-21memory21:10:372
1639599161,6cyclictest0-21swapper/222:47:122
1639599161,6cyclictest0-21swapper/200:05:242
1639599161,5cyclictest0-21swapper/221:27:322
1639599161,5cyclictest0-21swapper/221:27:322
1639599161,5cyclictest0-21swapper/220:30:132
16395991611,3cyclictest7787-21ls21:15:132
16395991611,3cyclictest5262-21fw_packets19:55:142
16395991611,3cyclictest22726-21ntp_offset19:20:182
16395991611,3cyclictest14425-21missed_timers20:15:202
16395991610,4cyclictest31133-21/usr/sbin/munin19:40:172
16395991610,3cyclictest1758-21expr22:15:122
1639599160,6cyclictest0-21swapper/223:15:162
16380991611,3cyclictest5782-21open_inodes19:55:220
16380991611,3cyclictest2740-21seq21:03:320
16380991611,3cyclictest11300-21cpu20:10:110
1638099161,12cyclictest6879-21/usr/sbin/munin20:00:170
16380991610,3cyclictest9286-21cat20:05:120
1639599158,4cyclictest23612-21timerandwakeup20:35:262
16395991510,3cyclictest15723-21cpu20:20:112
16395991510,3cyclictest15723-21cpu20:20:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional