You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-23 - 20:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Apr 23, 2024 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
52572254192,20sleep30-21swapper/307:07:153
54202242164,24sleep00-21swapper/007:09:200
53822238163,24sleep20-21swapper/207:08:512
54632219145,61sleep10-21swapper/107:09:541
2104021660,5sleep3557999cyclictest10:10:113
1622121610,5sleep0556399cyclictest07:30:200
5575998076,3cyclictest0-21swapper/207:22:232
69622630,4sleep2557599cyclictest08:25:012
136632550,3sleep2557599cyclictest12:20:132
5575994340,2cyclictest0-21swapper/207:17:422
557999411,15cyclictest21825-21df11:25:143
300052410,3sleep230008-21missed_timers09:15:192
5563993910,13cyclictest1449-21munin-run10:40:000
557999385,8cyclictest8354-40dnf12:05:513
5579993817,18cyclictest3304-21cut11:55:213
557999362,10cyclictest22791-21diskstats09:00:163
557999362,10cyclictest13044-21/usr/sbin/munin07:25:133
557999362,10cyclictest13044-21/usr/sbin/munin07:25:133
5579993511,3cyclictest12442-21sshd11:04:493
557999341,7cyclictest4271-21fschecks_count10:45:163
557999341,6cyclictest0-21swapper/310:15:113
557999341,4cyclictest26311-21ntpq10:20:243
5579993314,3cyclictest24129-21tr07:50:113
557999331,3cyclictest28200-21munin-run08:00:013
557999331,2cyclictest86750irq/128-enp1s0-09:28:023
5579993311,3cyclictest28313-21cat11:40:123
5579993311,2cyclictest18047-21acpi08:50:103
5579993310,4cyclictest0-21swapper/308:05:293
557599332,20cyclictest5516-21kworker/2:209:36:222
557599332,20cyclictest5516-21kworker/2:209:36:222
5579993217,4cyclictest0-21swapper/310:05:263
557999321,6cyclictest15205-21cat09:55:253
557999321,22cyclictest0-21swapper/307:20:193
557999321,11cyclictest6808-21sshd08:24:483
557999321,10cyclictest11461-21/usr/sbin/munin08:35:143
557999321,10cyclictest11461-21/usr/sbin/munin08:35:143
557999320,9cyclictest0-21swapper/312:29:483
557999311,6cyclictest18872-21chrt11:17:443
557999311,5cyclictest9663-21awk09:45:013
557999311,4cyclictest21828-21/usr/sbin/munin07:45:213
557999311,1cyclictest0-21swapper/309:45:323
557999311,10cyclictest29047-21/usr/sbin/munin09:15:163
5579993110,4cyclictest28972-21unixbench_multi10:25:323
5579993110,3cyclictest5868-21fschecks_count09:35:153
5579993110,3cyclictest5868-21fschecks_count09:35:153
5579993110,3cyclictest26634-21perf09:10:013
5579993110,3cyclictest20105-21grep07:40:153
5579993110,3cyclictest17969-21fschecks_time07:35:183
5563993112,16cyclictest23465-21munin-run11:30:010
557999301,5cyclictest14039-21latency_hist09:55:023
5579993013,2cyclictest19904-21/usr/sbin/munin12:35:113
557999301,12cyclictest14634-21snmpd11:13:033
557999301,11cyclictest32474-21/usr/sbin/munin11:50:273
557999300,9cyclictest0-21swapper/311:35:273
557999297,6cyclictest0-21swapper/309:20:143
557999292,9cyclictest0-21swapper/309:30:143
557999292,3cyclictest29313-21latency08:00:193
557999291,5cyclictest24951-21smart_sda11:30:263
557999291,4cyclictest15931-21sshd08:45:123
557999291,4cyclictest0-21swapper/311:21:143
557999291,2cyclictest0-21swapper/310:37:283
5579992911,4cyclictest4859-21dnf12:00:143
557999291,11cyclictest0-21swapper/307:14:543
556399291,13cyclictest32250-21munin-run11:50:010
556399291,13cyclictest32250-21munin-run11:50:010
556399291,13cyclictest29438-21munin-run10:30:000
5563992910,16cyclictest9656-21munin-run09:45:000
205912290,4sleep020578-21missed_timers07:40:190
557999282,5cyclictest8658-21users08:25:293
557999281,9cyclictest0-21swapper/310:01:533
5579992811,3cyclictest21463-21rt-features08:55:233
557999281,12cyclictest14634-21snmpd08:12:263
5579992810,3cyclictest31199-21wc11:45:203
5579992810,3cyclictest31199-21wc11:45:203
557999280,9cyclictest0-21swapper/310:33:303
557999279,4cyclictest13111-21tune2fs11:05:163
5579992711,3cyclictest9035-21cpuspeed_turbos12:10:143
5579992710,3cyclictest1912-21expr10:40:143
5579992710,3cyclictest1912-21expr10:40:143
557999270,12cyclictest0-21swapper/310:51:423
557599272,17cyclictest4200-21kworker/2:112:09:022
557599272,16cyclictest20663-21kworker/2:010:34:222
557599271,10cyclictest5516-21kworker/2:207:31:342
557999261,7cyclictest0-21swapper/310:58:453
557599262,12cyclictest20663-21kworker/2:011:58:542
557599261,12cyclictest5516-21kworker/2:209:57:182
5563992610,14cyclictest28210-21perf08:00:010
557999253,3cyclictest26622-21diskmemload12:20:183
557599252,11cyclictest20663-21kworker/2:011:49:342
557599252,11cyclictest20663-21kworker/2:011:49:342
557599252,10cyclictest20663-21kworker/2:010:10:062
5575992516,2cyclictest20663-21kworker/2:011:22:222
557599251,11cyclictest20663-21kworker/2:011:39:182
556899252,20cyclictest30408-21munin-run08:05:011
5568992512,6cyclictest25091-21kworker/u8:307:30:191
556399251,12cyclictest26641-21perf09:10:010
557999244,3cyclictest0-21swapper/308:30:113
557999243,2cyclictest86750irq/128-enp1s0-09:14:563
557999241,13cyclictest0-21swapper/307:16:183
557599242,11cyclictest20663-21kworker/2:011:13:262
5575992418,3cyclictest27109-21latency07:55:172
556899246,10cyclictest0-21swapper/112:30:311
5568992415,2cyclictest9036-21/usr/sbin/munin08:30:011
556899241,17cyclictest86850irq/129-enp1s0-11:50:361
556899241,17cyclictest0-21swapper/112:11:321
556899240,17cyclictest0-21swapper/110:03:111
557999234,3cyclictest0-21swapper/312:19:573
557999231,10cyclictest0-21swapper/307:32:143
557599232,11cyclictest5516-21kworker/2:208:05:342
557599231,14cyclictest5516-21kworker/2:209:05:022
557599231,13cyclictest5516-21kworker/2:210:03:422
5575992311,3cyclictest20663-21kworker/2:011:29:022
557599231,10cyclictest5516-21kworker/2:209:41:262
556899239,4cyclictest16874-21munin-run11:15:011
556899234,12cyclictest86750irq/128-enp1s0-10:32:541
556899231,9cyclictest6422-21unixbench_singl08:20:251
5568992315,3cyclictest0-21swapper/111:31:141
556899231,19cyclictest7459-21munin-run09:40:011
556899231,19cyclictest7459-21munin-run09:40:011
556899231,18cyclictest22899-21ntp_states07:45:231
556899231,16cyclictest0-21swapper/110:15:161
556899231,16cyclictest0-21swapper/109:09:441
556899231,16cyclictest0-21swapper/109:00:251
556899230,17cyclictest86850irq/129-enp1s0-09:43:201
556899230,16cyclictest0-21swapper/110:42:581
556899230,16cyclictest0-21swapper/110:42:581
557999220,10cyclictest0-21swapper/308:40:183
557599222,9cyclictest5516-21kworker/2:209:23:342
557599222,9cyclictest20663-21kworker/2:010:20:382
557599221,8cyclictest4200-21kworker/2:112:28:302
557599221,4cyclictest0-21swapper/211:00:172
557599221,15cyclictest20663-21kworker/2:010:50:242
557599221,10cyclictest20663-21kworker/2:010:39:142
556899222,16cyclictest0-21swapper/108:41:121
556899222,15cyclictest0-21swapper/109:23:291
5568992215,2cyclictest0-21swapper/110:55:481
5568992215,2cyclictest0-21swapper/109:55:041
556899221,4cyclictest0-21swapper/109:47:391
556899221,16cyclictest0-21swapper/111:35:581
556899221,16cyclictest0-21swapper/110:21:011
556899221,15cyclictest0-21swapper/111:05:071
556899221,10cyclictest28313-21cat08:00:021
556899220,20cyclictest0-21swapper/111:18:101
556899220,15cyclictest0-21swapper/110:55:021
556399223,16cyclictest6508-21awk12:05:000
556399221,11cyclictest29844-21cstates10:30:100
557999211,11cyclictest0-21swapper/308:15:203
557999211,10cyclictest0-21swapper/312:31:503
557599218,4cyclictest26002-21awk07:55:012
557599212,9cyclictest5516-21kworker/2:209:48:062
557599212,7cyclictest5516-21kworker/2:208:19:262
557599212,10cyclictest20663-21kworker/2:010:59:262
5575992115,3cyclictest8536-21grep08:25:292
557599211,10cyclictest4200-21kworker/2:112:00:062
557599211,10cyclictest20663-21kworker/2:011:18:542
556899217,4cyclictest15103-21sshd12:22:481
556899212,18cyclictest0-21swapper/107:28:591
556899212,18cyclictest0-21swapper/107:28:591
5568992116,3cyclictest21023-21if_err_enp1s008:55:161
5568992113,3cyclictest0-21swapper/109:18:111
5568992113,3cyclictest0-21swapper/108:38:421
5568992113,3cyclictest0-21swapper/108:38:421
556899211,15cyclictest0-21swapper/112:35:421
556899211,15cyclictest0-21swapper/110:26:331
556899211,15cyclictest0-21swapper/110:10:051
556899211,15cyclictest0-21swapper/110:08:171
556899211,14cyclictest0-21swapper/112:25:091
556899211,11cyclictest4546-21smart_sda09:30:251
556899210,15cyclictest0-21swapper/112:05:151
5563992114,4cyclictest18029-21grep10:00:310
556399211,11cyclictest3281-21/usr/sbin/munin09:30:110
5563992110,3cyclictest24027-21df_inode11:30:130
557599202,8cyclictest5516-21kworker/2:210:07:102
557599202,5cyclictest27448-1kworker/u9:108:39:332
557599202,5cyclictest27448-1kworker/u9:108:39:332
5575992013,4cyclictest4535-21gltestperf10:45:172
5575992013,4cyclictest18613-21proc_pri07:35:272
557599201,18cyclictest20663-21kworker/2:010:16:462
557599201,16cyclictest11934-21/usr/sbin/munin12:15:202
557599201,10cyclictest4200-21kworker/2:112:35:102
556899207,3cyclictest780-21rngd07:40:561
556899202,4cyclictest7716-21sshd07:14:551
556899202,3cyclictest0-21swapper/109:12:411
556899201,9cyclictest29154-21ntp_states11:40:231
5568992016,3cyclictest0-21swapper/108:48:191
5568992013,3cyclictest0-21swapper/108:53:111
5568992012,3cyclictest0-21swapper/111:02:391
556899201,18cyclictest0-21swapper/107:20:371
556899201,17cyclictest0-21swapper/108:06:571
556899201,17cyclictest0-21swapper/107:50:151
556899201,14cyclictest0-21swapper/111:21:271
556899201,14cyclictest0-21swapper/109:26:281
556899201,14cyclictest0-21swapper/108:31:131
556899200,15cyclictest0-21swapper/112:18:121
556399203,6cyclictest24391-40dnf09:04:510
556399202,3cyclictest20646-21munin-run10:10:010
556399201,5cyclictest0-21swapper/009:47:050
556399201,12cyclictest0-21swapper/011:40:070
556399201,11cyclictest18581-21unixbench_multi11:15:270
556399201,11cyclictest18045-21/usr/sbin/munin08:50:180
556399201,11cyclictest13659-21/usr/sbin/munin08:40:160
556399201,11cyclictest12067-21/usr/sbin/munin09:50:120
556399201,11cyclictest0-21swapper/011:55:250
556399201,10cyclictest21828-21/usr/sbin/munin07:45:120
556399201,10cyclictest0-21swapper/011:50:260
557599192,8cyclictest5516-21kworker/2:209:11:262
557599192,7cyclictest5516-21kworker/2:209:28:222
557599191,6cyclictest9995-21ntp_states12:10:262
5575991914,3cyclictest18077-21fschecks_count12:30:142
5575991913,3cyclictest7729-21latency_hist07:15:012
557599191,11cyclictest5516-21kworker/2:207:28:382
557599191,11cyclictest5516-21kworker/2:207:28:382
556899192,16cyclictest0-21swapper/112:00:471
556899191,8cyclictest0-21swapper/111:46:221
556899191,8cyclictest0-21swapper/111:46:221
5568991916,2cyclictest0-21swapper/110:36:241
5568991915,3cyclictest0-21swapper/107:17:101
5568991912,4cyclictest18321-21latency07:35:221
5568991912,3cyclictest0-21swapper/111:28:191
556899191,17cyclictest0-21swapper/108:10:371
5568991911,3cyclictest0-21swapper/111:55:121
556899191,10cyclictest13696-21chrt09:50:431
556399199,3cyclictest15511-21/usr/sbin/munin12:25:250
556399198,3cyclictest10845-21/usr/sbin/munin07:20:190
556399197,2cyclictest11643-21cstates08:35:140
556399197,2cyclictest11643-21cstates08:35:140
556399195,7cyclictest3317-21kworker/u8:110:46:380
556399191,4cyclictest8707-21munin-run12:10:000
556399191,2cyclictest86550irq/126-enp1s0-10:14:450
556399191,12cyclictest0-21swapper/010:20:210
556399191,10cyclictest14634-21snmpd09:17:400
556399191,10cyclictest0-21swapper/011:20:460
556399190,9cyclictest0-21swapper/010:17:470
556399190,6cyclictest0-21swapper/011:06:290
557599182,10cyclictest5516-21kworker/2:209:05:182
557599181,6cyclictest0-21swapper/209:53:152
557599181,6cyclictest0-21swapper/207:45:242
5575991813,3cyclictest19755-21grep07:40:122
5575991813,3cyclictest18682-21tail08:50:152
5575991812,3cyclictest9341-21cat08:30:112
557599181,15cyclictest0-21swapper/211:30:402
556899181,9cyclictest4597-21idleruntime10:45:181
556899181,16cyclictest0-21swapper/108:17:001
556399189,2cyclictest26622-21diskmemload09:25:300
556399185,3cyclictest7898-21kworker/u8:211:13:180
556399182,13cyclictest30629-21/usr/sbin/munin08:05:150
556399181,6cyclictest10118-21chrt10:58:170
5563991812,4cyclictest20897-21needreboot12:35:200
556399181,15cyclictest26012-21perf07:55:010
556399181,11cyclictest0-21swapper/010:50:400
556399181,10cyclictest13664-21/usr/sbin/munin12:20:120
556399180,9cyclictest0-21swapper/011:36:480
556399180,12cyclictest0-21swapper/007:15:280
50882180,2sleep20-21swapper/209:33:062
19352181,2sleep20-21swapper/211:53:162
122982181,2sleep00-21swapper/011:02:560
557599172,12cyclictest2365-21sed10:40:172
557599172,12cyclictest2365-21sed10:40:172
557599172,10cyclictest20663-21kworker/2:011:40:222
5575991713,3cyclictest0-21swapper/208:01:062
5575991712,3cyclictest14795-21sed11:10:022
557599171,15cyclictest20663-21kworker/2:010:25:342
556399174,7cyclictest15180-21kworker/u8:209:58:330
556399173,9cyclictest0-21swapper/008:25:230
556399172,12cyclictest20240-21/usr/sbin/munin08:55:240
556399171,9cyclictest26622-21diskmemload12:19:080
556399171,7cyclictest15850-21/usr/sbin/munin08:45:280
556399171,7cyclictest0-21swapper/007:10:320
556399171,6cyclictest0-21swapper/012:11:320
556399171,4cyclictest0-21swapper/012:32:070
5563991712,3cyclictest29887-21unixbench-2d08:00:280
556399170,6cyclictest0-21swapper/009:14:490
556399170,11cyclictest0-21swapper/010:41:110
556399170,11cyclictest0-21swapper/010:41:110
22872171,2sleep20-21swapper/208:13:142
557599161,6cyclictest0-21swapper/208:55:142
557599161,5cyclictest0-21swapper/208:40:142
5575991611,3cyclictest15837-21date08:45:022
556399165,3cyclictest5474-21/usr/sbin/munin09:35:270
556399165,3cyclictest5474-21/usr/sbin/munin09:35:270
556399161,6cyclictest32323-21ntp_states09:20:270
556399161,6cyclictest0-21swapper/008:30:140
5563991612,3cyclictest0-21swapper/007:29:160
5563991612,3cyclictest0-21swapper/007:29:160
5563991611,3cyclictest947-21cut08:10:150
5563991611,3cyclictest2674-21basename08:15:130
556399152,10cyclictest18297-21ls07:35:210
5563991510,3cyclictest4858-21/usr/sbin/munin08:20:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional