You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-10-04 - 00:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Oct 03, 2024 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
262832236163,24sleep10-21swapper/107:09:231
259952235162,22sleep30-21swapper/307:05:383
259992231163,55sleep20-21swapper/207:05:412
263042203164,25sleep00-21swapper/007:09:370
1294521550,3sleep02641299cyclictest09:05:010
1166621500,4sleep22642699cyclictest10:15:122
26426999986,2cyclictest0-21swapper/209:27:012
26426996461,2cyclictest0-21swapper/211:38:432
73932600,4sleep011-21rcuc/007:35:260
303742570,8sleep123-21ksoftirqd/107:15:191
80202560,2sleep20-21swapper/207:40:012
174612560,1sleep30-21swapper/310:25:253
283202540,4sleep32643399cyclictest09:40:013
213242470,1sleep10-21swapper/111:50:141
213242470,1sleep10-21swapper/111:50:141
200872470,1sleep220088-21df09:20:132
26433993913,3cyclictest3424-21smart_sda08:40:263
2643399352,8cyclictest86550irq/126-enp1s0-10:05:173
2643399348,4cyclictest2409-21fw_packets12:20:153
2643399348,4cyclictest22142-21irqstats08:10:173
2643399341,2cyclictest86750irq/128-enp1s0-09:43:373
2643399341,2cyclictest86750irq/128-enp1s0-09:43:373
26433993412,3cyclictest32322-21/usr/sbin/munin08:35:233
26433993412,3cyclictest32322-21/usr/sbin/munin08:35:233
26433993411,3cyclictest23901-21threads10:40:273
26433993410,3cyclictest20524-21cpuspeed_turbos10:35:113
2641299348,12cyclictest22958-21munin-run11:55:010
2641299348,12cyclictest22958-21munin-run11:55:010
296372330,3sleep12642099cyclictest09:40:231
296372330,3sleep12642099cyclictest09:40:231
26433993311,3cyclictest9995-21/usr/sbin/munin11:25:163
26433993311,3cyclictest26756-21grep07:10:123
26433993310,3cyclictest836-21runrttasks10:02:453
2641299335,13cyclictest31497-21perf07:20:000
28522320,3sleep12642099cyclictest12:20:211
2643399328,7cyclictest7693-21sshd07:37:273
2643399327,3cyclictest10097-21sh07:45:013
2643399323,6cyclictest31077-40dnf10:59:283
2643399323,5cyclictest0-21swapper/309:54:583
2643399322,4cyclictest0-21swapper/310:25:013
2643399321,7cyclictest7798-21/usr/sbin/munin11:20:253
2643399321,6cyclictest31659-21missed_timers09:45:203
26433993212,3cyclictest2577-21ntpq07:25:193
2643399321,15cyclictest0-21swapper/312:06:573
2643399321,14cyclictest0-21swapper/311:40:183
2643399321,11cyclictest29515-21/usr/sbin/munin07:15:143
2643399321,11cyclictest16920-21/usr/sbin/munin08:00:193
26433993210,3cyclictest28254-21expr08:25:123
26433993210,3cyclictest26985-21sendmail_mailtr08:20:243
26433993210,3cyclictest21567-21gltestperf11:50:153
26433993210,3cyclictest21567-21gltestperf11:50:153
26433993210,3cyclictest12641-21tr07:50:123
2643399319,4cyclictest0-21swapper/311:30:103
2643399313,3cyclictest12050irq/123-ahci[0009:30:143
2643399311,9cyclictest0-21swapper/310:11:403
2643399311,4cyclictest2278-21ntp_states11:05:223
26433993111,3cyclictest836-21runrttasks12:16:563
26433993111,3cyclictest13153-21/usr/sbin/munin09:05:153
26433993110,3cyclictest20128-21crond10:35:003
26420993121,3cyclictest10574-21perl11:25:161
190822310,4sleep230-21rcuc/211:45:122
2643399304,3cyclictest86550irq/126-enp1s0-12:02:053
2643399303,9cyclictest0-21swapper/312:36:193
2643399303,4cyclictest5090-21sshd11:11:273
2643399303,3cyclictest0-21swapper/310:16:533
2643399301,4cyclictest0-21swapper/311:02:003
2643399301,3cyclictest28552-21sshd10:50:263
26433993013,10cyclictest12113-21proc_pri09:00:233
2641299302,13cyclictest17954-21awk10:30:000
2643399298,4cyclictest18367-21kernelversion09:15:183
2643399292,3cyclictest25150-21sh12:00:013
26433992912,3cyclictest19633-21wc11:45:173
26433992911,3cyclictest3848-21expr07:30:123
26433992911,3cyclictest3848-21expr07:30:123
2643399291,11cyclictest0-21swapper/312:11:273
2643399291,11cyclictest0-21swapper/309:58:463
26433992910,3cyclictest5437-21open_files08:45:213
26433992910,3cyclictest21212-21cut08:10:023
26433992910,3cyclictest19749-21/usr/sbin/munin09:20:113
2641299291,13cyclictest4747-21munin-run10:00:010
2643399289,4cyclictest0-21swapper/310:45:173
2643399289,4cyclictest0-21swapper/310:45:173
2643399284,15cyclictest780-21rngd07:48:513
2643399282,3cyclictest0-21swapper/311:37:433
2643399281,9cyclictest0-21swapper/312:27:433
2643399281,9cyclictest0-21swapper/309:12:073
2643399281,4cyclictest7824-21sshd12:30:273
26433992812,3cyclictest7108-21sed11:15:313
26433992811,3cyclictest22313-21fschecks_count09:25:133
26433992810,3cyclictest31822-21expr07:20:113
26433992810,3cyclictest15049-21expr07:55:123
26426992825,2cyclictest0-21swapper/209:16:382
2641299282,11cyclictest2563-21perf09:55:020
11802280,4sleep232-21ksoftirqd/209:50:152
325322270,4sleep1221ktimersoftd/112:15:151
2643399272,10cyclictest25491-21latency_hist08:20:013
2641299261,12cyclictest6335-21awk08:50:010
2643399251,8cyclictest14634-21snmpd08:30:253
2642099252,17cyclictest0-21swapper/110:07:161
2642099252,12cyclictest29898-21munin-run08:30:001
26420992520,2cyclictest56942sleep107:34:581
26420992520,2cyclictest56942sleep107:34:581
2642099251,17cyclictest14799-21nfs4_client10:20:211
26420992511,12cyclictest9596-21kworker/u8:211:35:451
87052241,2sleep10-21swapper/111:20:191
26426992418,3cyclictest28322-21ls09:40:012
2642099243,16cyclictest31077-40dnf10:59:311
2642099242,16cyclictest0-21swapper/108:54:221
2642099241,7cyclictest0-21swapper/111:55:151
2642099241,17cyclictest0-21swapper/111:16:171
2642099241,17cyclictest0-21swapper/108:47:461
2642099240,17cyclictest0-21swapper/107:37:271
2641299241,11cyclictest23309-21perf08:15:000
2642099232,15cyclictest0-21swapper/110:35:061
2642099232,15cyclictest0-21swapper/110:18:111
2642099232,13cyclictest20558-21irqstats09:20:171
2642099232,12cyclictest23012-21ntp_states09:25:201
26420992314,2cyclictest25751-21strings10:45:191
26420992314,2cyclictest25751-21strings10:45:191
2642099231,19cyclictest12186-21sa111:30:101
2642099231,17cyclictest0-21swapper/108:22:041
2642099231,11cyclictest3518-21perf07:30:011
92202220,9sleep081ktimersoftd/011:20:260
271702220,8sleep011-21rcuc/008:20:260
2643399221,5cyclictest9407-21grep08:55:163
2642099226,4cyclictest0-21swapper/109:10:051
2642099222,15cyclictest0-21swapper/111:13:391
2642099222,15cyclictest0-21swapper/109:51:261
2642099222,15cyclictest0-21swapper/108:15:411
2642099222,15cyclictest0-21swapper/107:46:071
26420992217,3cyclictest16591-21/usr/sbin/munin11:40:261
26420992214,5cyclictest1-21systemd07:15:001
26420992214,3cyclictest0-21swapper/109:18:131
2642099221,3cyclictest0-21swapper/112:00:181
2642099221,15cyclictest0-21swapper/108:04:291
2642099221,15cyclictest0-21swapper/107:55:151
2642099221,11cyclictest0-21swapper/112:36:461
2642099221,11cyclictest0-21swapper/112:10:471
2641299229,4cyclictest22637-21hwlatdetect09:25:160
26426992115,3cyclictest29096-21wc07:15:022
2642099219,3cyclictest8807-21hddtemp_smartct07:40:161
2642099219,3cyclictest12305-21timerandwakeup09:00:261
2642099216,6cyclictest0-21swapper/109:37:551
2642099212,18cyclictest0-21swapper/110:03:221
2642099211,7cyclictest0-21swapper/109:49:051
2642099211,15cyclictest0-21swapper/111:01:311
2642099211,15cyclictest0-21swapper/110:50:281
2642099211,15cyclictest0-21swapper/110:32:541
2642099211,15cyclictest0-21swapper/110:27:431
2642099211,15cyclictest0-21swapper/110:13:241
2642099211,15cyclictest0-21swapper/109:30:511
2642099211,15cyclictest0-21swapper/108:42:581
2642099211,11cyclictest29180-21users12:05:261
2642099211,10cyclictest0-21swapper/109:55:371
2642099210,7cyclictest0-21swapper/111:48:081
2642099210,14cyclictest0-21swapper/111:05:071
2641299213,10cyclictest8749-21/usr/sbin/munin08:55:240
2641299211,6cyclictest0-21swapper/011:31:370
2641299211,6cyclictest0-21swapper/011:01:550
2641299211,6cyclictest0-21swapper/010:33:000
2641299211,5cyclictest0-21swapper/011:18:190
2641299211,12cyclictest31990-21/usr/sbin/munin12:15:170
2643399209,3cyclictest6551-21/usr/sbin/munin08:50:243
2642699201,5cyclictest0-21swapper/210:29:292
2642099202,17cyclictest0-21swapper/112:29:311
26420992013,2cyclictest0-21swapper/107:50:411
2642099201,2cyclictest0-21swapper/108:37:231
2642099201,2cyclictest0-21swapper/108:37:231
2642099201,17cyclictest0-21swapper/112:31:271
2642099201,14cyclictest0-21swapper/110:41:191
2642099201,14cyclictest0-21swapper/109:05:561
2642099201,14cyclictest0-21swapper/108:55:561
2642099201,14cyclictest0-21swapper/108:05:581
2642099200,15cyclictest0-21swapper/108:31:401
2641299201,7cyclictest0-21swapper/010:47:420
2641299201,7cyclictest0-21swapper/010:47:420
2641299201,12cyclictest0-21swapper/011:12:510
2641299201,10cyclictest14634-21snmpd09:48:020
26412992010,3cyclictest86750irq/128-enp1s0-12:12:510
185462201,2sleep218691-21latency_hist11:45:012
15242201,15sleep01771-21expr07:25:120
70032191,3sleep20-21swapper/212:30:172
2642699191,6cyclictest16485-40dnf07:57:332
26426991913,3cyclictest10978-21vmstat10:10:262
26426991910,2cyclictest4757-21awk10:00:012
26420991915,3cyclictest0-21swapper/108:12:321
2642099191,17cyclictest0-21swapper/107:21:031
2641299198,3cyclictest14634-21snmpd09:33:000
2641299196,3cyclictest29909-21awk08:30:010
2641299192,8cyclictest22565-21/usr/sbin/munin10:40:210
2641299192,7cyclictest6229-21sensors_temp10:00:240
2641299192,14cyclictest13166-21/usr/sbin/munin10:15:280
2641299191,8cyclictest0-21swapper/010:08:530
2641299191,7cyclictest0-21swapper/010:52:210
2641299191,5cyclictest0-21swapper/009:12:150
2641299191,4cyclictest0-21swapper/011:09:220
26412991913,3cyclictest17650-21cut09:15:120
2641299191,11cyclictest0-21swapper/010:10:260
2641299191,10cyclictest14634-21snmpd09:22:590
2641299191,10cyclictest13774-21/usr/sbin/munin10:20:150
2641299191,10cyclictest12525-21/usr/sbin/munin07:50:270
320962181,2sleep20-21swapper/212:15:122
285202181,3sleep20-21swapper/207:14:182
2642699188,2cyclictest86850irq/129-enp1s0-12:00:192
2642699186,3cyclictest12050irq/123-ahci[0012:10:012
2642699185,4cyclictest31453-21cat09:45:172
2642699184,4cyclictest30936-21pmu-power12:10:222
2642699182,5cyclictest0-21swapper/210:30:202
2642699181,7cyclictest31913-21fschecks_time11:00:142
2642699181,7cyclictest14114-21df_inode10:20:142
2642699181,7cyclictest0-21swapper/211:08:412
2642699181,7cyclictest0-21swapper/210:50:562
2642699181,7cyclictest0-21swapper/210:35:242
2642699181,4cyclictest0-21swapper/211:21:562
2642699181,4cyclictest0-21swapper/209:10:162
26426991812,3cyclictest4037-21sed11:10:142
26426991812,3cyclictest25287-21fschecks_time10:45:142
26426991812,3cyclictest25287-21fschecks_time10:45:142
26426991812,3cyclictest23294-21latency_hist08:15:012
26426991812,3cyclictest11375-21ntpq07:45:202
2642699181,15cyclictest0-21swapper/210:05:052
2642699181,14cyclictest5989-21latency_hist12:30:012
2641299181,9cyclictest25385-21/usr/sbin/munin12:00:120
2641299181,8cyclictest4022-21/usr/sbin/munin12:25:190
2641299181,8cyclictest0-21swapper/011:27:140
2641299181,7cyclictest0-21swapper/012:09:420
26412991812,3cyclictest8514-21crond08:55:010
26412991811,4cyclictest21467-21pluginstate10:35:210
26412991810,7cyclictest7368-21kworker/u8:011:40:340
226062180,3sleep20-21swapper/211:50:432
226062180,3sleep20-21swapper/211:50:432
2642699175,5cyclictest24142-21/usr/sbin/munin09:30:132
2642699171,9cyclictest0-21swapper/210:59:562
2642699171,7cyclictest27927-21/usr/sbin/munin08:25:252
2642699171,7cyclictest0-21swapper/211:27:402
2642699171,5cyclictest0-21swapper/210:42:462
26426991712,3cyclictest32583-21expr08:35:122
26426991712,3cyclictest32583-21expr08:35:122
26426991712,3cyclictest28807-21ls09:40:132
26426991712,3cyclictest28807-21ls09:40:132
26426991712,3cyclictest12967-21fschecks_count07:50:152
2642699171,15cyclictest0-21swapper/212:38:092
2642699171,15cyclictest0-21swapper/210:01:412
2642699171,15cyclictest0-21swapper/209:03:342
2642699171,14cyclictest0-21swapper/207:20:022
26426991711,3cyclictest5055-21sort07:30:252
26426991711,3cyclictest5055-21sort07:30:252
26426991711,3cyclictest30229-21grep08:30:122
2641299174,4cyclictest19121-21df11:45:120
2641299171,9cyclictest14634-21snmpd12:36:270
2641299171,7cyclictest0-21swapper/012:24:060
2641299171,6cyclictest24241-21ntp_states11:55:210
2641299171,3cyclictest16478-21sshd07:57:280
26412991712,3cyclictest11418-21open_files07:45:220
2641299171,14cyclictest0-21swapper/008:16:080
231692171,2sleep20-21swapper/211:55:052
280062160,2sleep00-21swapper/009:36:100
2642699169,4cyclictest5459-21pmu-power08:45:222
2642699161,9cyclictest0-21swapper/212:20:232
2642699161,6cyclictest4214-21cat08:45:012
2642699161,6cyclictest25726-21/usr/sbin/munin08:20:172
2642699161,6cyclictest23870-21diskstats08:15:122
26426991612,3cyclictest0-21swapper/211:15:472
26426991612,3cyclictest0-21swapper/208:59:172
2642699161,14cyclictest0-21swapper/208:51:442
26426991611,3cyclictest17987-21sh08:00:212
26426991611,3cyclictest12191-21/usr/sbin/munin11:30:162
26426991610,4cyclictest0-21swapper/208:05:192
26426991610,3cyclictest8357-21expr07:40:122
26426991610,3cyclictest13834-21hwlatdetect09:05:172
2641299165,3cyclictest20212-21ntp_states08:05:300
2641299162,3cyclictest86550irq/126-enp1s0-07:10:150
2641299162,2cyclictest86550irq/126-enp1s0-09:05:560
2641299162,12cyclictest869-21unixbench_singl07:20:260
2641299162,11cyclictest2163-21/usr/sbin/munin08:40:250
2641299161,9cyclictest0-21swapper/010:55:190
2641299161,7cyclictest0-21swapper/009:40:160
2641299161,7cyclictest0-21swapper/009:40:150
2641299161,3cyclictest5574-21sshd07:33:280
2641299161,3cyclictest5574-21sshd07:33:280
2641299161,2cyclictest0-21swapper/008:36:590
2641299161,2cyclictest0-21swapper/008:36:590
2641299161,14cyclictest0-21swapper/008:03:420
26412991611,3cyclictest31012-21sort08:30:190
2641299161,12cyclictest6221-21/usr/sbin/munin12:30:140
26412991610,3cyclictest8920-21if_err_enp2s007:40:190
2642699151,5cyclictest1540-21acpi07:25:102
26412991510,3cyclictest14717-21expr11:35:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional