You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-19 - 23:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Fri Apr 19, 2024 12:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1976888994643,2cyclictest2386917-21kworker/u8:3+events_unbound12:40:011
1976888994643,2cyclictest2196624-21kworker/u8:0+events_unbound10:25:281
1976891994541,3cyclictest74-21kswapd011:10:002
1976888994135,4cyclictest2319123-21latency_hist11:40:011
1976888994135,4cyclictest2319123-21latency_hist11:40:011
1976881994136,3cyclictest74-21kswapd012:20:000
1976888993935,2cyclictest2252427-21kworker/u8:2+events_unbound11:15:101
1976888993632,3cyclictest2123153-21kworker/u8:2+events_unbound09:50:261
1976888993531,3cyclictest2325532-21kworker/u8:1+flush-259:012:10:271
1976895993429,3cyclictest1976646-21latency_hist07:10:023
1976891993430,3cyclictest74-21kswapd012:10:002
1976888993427,5cyclictest2023129-21latency_hist07:55:001
1976895993230,1cyclictest2143179-21fwupd09:35:503
1976895993228,3cyclictest2297104-21kworker/u8:4+events_unbound11:45:013
1976888993228,3cyclictest2182484-21kworker/u8:4+events_unbound10:55:241
1976895993126,3cyclictest2034070-21apt-get08:05:103
1976891993125,4cyclictest253-21systemd-journal09:30:012
1976895993026,3cyclictest2145434-21kworker/u8:1+flush-259:010:05:243
1976891993026,3cyclictest2123153-21kworker/u8:2+flush-259:009:55:272
1976891992926,2cyclictest74-21kswapd011:05:002
1976891992823,3cyclictest2039534-21base6408:10:002
1976888992821,5cyclictest253-21systemd-journal08:55:001
1976888992821,5cyclictest2394076-21latency_hist12:30:011
1976888992722,4cyclictest1982476-21apt-get07:15:131
1976888992722,3cyclictest2267578-21apt-get11:05:111
1976891992621,3cyclictest1997855-21apt-get07:30:112
1976888992623,2cyclictest2196624-21kworker/u8:0+flush-259:011:10:121
1976888992622,3cyclictest2280523-21kworker/u8:1+events_unbound11:25:231
1976888992622,3cyclictest2085236-21kworker/u8:3+events_unbound09:25:101
1976888992622,2cyclictest2085236-21kworker/u8:3+flush-259:009:45:011
1976895992521,3cyclictest1922655-21kworker/u8:4+flush-259:007:20:283
1976891992518,5cyclictest2190260-21ssh10:10:122
1976888992522,2cyclictest2033852-21kworker/u8:0+flush-259:008:20:001
1976895992421,2cyclictest74-21kswapd011:00:003
1976895992421,2cyclictest2252427-21kworker/u8:2+events_unbound11:00:263
1976891992420,3cyclictest2282383-21apt-get11:15:092
1976891992419,3cyclictest2028808-21base6408:00:012
1976888992421,2cyclictest2145434-21kworker/u8:1+flush-259:009:45:211
1976888992420,3cyclictest1992451-21kworker/u8:1+events_unbound07:45:001
1976888992418,4cyclictest253-21systemd-journal11:50:001
1976895992320,2cyclictest1966271-21kworker/u8:0+events_unbound07:25:163
1976891992319,3cyclictest74-21kswapd010:35:142
1976891992319,3cyclictest2188854-21ssh10:08:022
1976891992318,3cyclictest2252793-21/usr/sbin/munin10:55:202
197688899238,8cyclictest1982214-21munin-run07:15:001
1976888992320,2cyclictest2349245-21kworker/u8:0+events_unbound12:05:241
1976888992320,2cyclictest2145434-21kworker/u8:1+flush-259:010:25:011
1976888992319,3cyclictest253-21systemd-journal07:29:501
1976888992319,3cyclictest2252427-21kworker/u8:2+flush-259:011:01:341
1976895992219,2cyclictest2297103-21kworker/u8:3+flush-259:011:35:273
1976895992219,2cyclictest2297103-21kworker/u8:3+flush-259:011:35:263
1976895992218,3cyclictest2113986-21ssh09:16:503
1976891992218,3cyclictest2135880-21ssh09:31:042
1976888992219,2cyclictest2033852-21kworker/u8:0+flush-259:008:25:161
1976888992218,3cyclictest2196624-21kworker/u8:0+flush-259:010:45:101
1976888992218,3cyclictest2113578-21kworker/u8:0+events_unbound09:16:431
1976888992218,2cyclictest2325532-21kworker/u8:1+events_unbound12:22:041
1976888992217,3cyclictest2173006-21ssh09:56:141
1976888992216,4cyclictest2304641-21/usr/sbin/munin11:30:151
1976888992216,4cyclictest2304641-21/usr/sbin/munin11:30:141
1976881992218,3cyclictest74-21kswapd011:35:000
1976881992218,3cyclictest74-21kswapd011:35:000
1976881992216,4cyclictest2200580-21latency_hist10:20:010
1976895992118,2cyclictest2101118-21ssh09:10:083
1976891992118,2cyclictest2356125-21sh12:05:002
1976891992118,2cyclictest2297104-21kworker/u8:4+flush-259:011:35:532
1976891992118,2cyclictest2297104-21kworker/u8:4+flush-259:011:35:532
1976888992117,3cyclictest2235495-21ssh10:40:591
1976888992117,3cyclictest2002951-21apt-get07:35:121
1976888992116,3cyclictest2362958-21kworker/u8:2+flush-259:012:31:281
1976895992016,3cyclictest2172867-21ssh09:55:593
1976895992016,3cyclictest2085236-21kworker/u8:3+flush-259:009:30:063
1976895992016,2cyclictest2297104-21kworker/u8:4+flush-259:011:45:073
1976888992017,2cyclictest2123153-21kworker/u8:2+events_unbound09:35:531
1976888992016,2cyclictest516-21rs:main1
1976888992014,4cyclictest253-21systemd-journal08:05:501
1976888992014,4cyclictest253-21systemd-journal07:57:521
1976881992017,2cyclictest498-21dbus-daemon12:25:000
1976895991915,3cyclictest2406596-21ssh12:35:393
1976895991915,3cyclictest2309816-21ssh11:31:283
1976895991915,3cyclictest2309816-21ssh11:31:283
1976895991914,3cyclictest2007800-21latency_hist07:40:013
1976895991913,4cyclictest2252481-21latency_hist10:55:003
1976888991916,2cyclictest2191337-21kthreadcore10:10:181
1976888991916,2cyclictest2123153-21kworker/u8:2+flush-259:009:31:521
1976888991915,3cyclictest2212533-21kworker/u8:2+events_unbound10:34:591
1976888991915,2cyclictest2033852-21kworker/u8:0+events_unbound08:30:191
1976881991916,2cyclictest2108275-21idleruntime-cro09:15:000
1976881991914,3cyclictest2018000-21sh07:49:590
1976881991913,4cyclictest1977355-21apt-get07:10:090
1976891991813,3cyclictest2023392-21apt-get07:55:112
1976888991815,2cyclictest2325532-21kworker/u8:1+events_unbound11:55:281
1976888991815,2cyclictest2297104-21kworker/u8:4+flush-259:011:53:481
1976888991815,2cyclictest2145434-21kworker/u8:1+flush-259:010:17:391
1976888991815,2cyclictest2033852-21kworker/u8:0+flush-259:008:20:171
1976888991815,1cyclictest0-21swapper/112:20:001
197688199187,9cyclictest253-21systemd-journal11:25:000
1976881991816,2cyclictest74-21kswapd011:40:000
1976881991816,2cyclictest74-21kswapd011:40:000
1976881991814,2cyclictest61150irq/132-enp0s29f1:rx-011:10:170
1976895991714,1cyclictest0-21swapper/310:40:003
1976891991713,3cyclictest74-21kswapd012:14:592
1976891991713,3cyclictest2340995-21ssh11:54:252
1976888991713,3cyclictest2101196-21ssh09:10:131
1976881991714,2cyclictest61150irq/132-enp0s29f1:rx-012:06:350
1976881991714,2cyclictest61150irq/132-enp0s29f1:rx-011:50:140
1976881991714,2cyclictest61150irq/132-enp0s29f1:rx-011:44:050
1976881991714,2cyclictest2085236-21kworker/u8:3+flush-259:009:00:250
1976881991713,3cyclictest253-21systemd-journal09:08:520
1976881991713,3cyclictest2123468-21apt-get09:25:090
1976881991713,2cyclictest61150irq/132-enp0s29f1:rx-008:00:010
1976881991712,2cyclictest61150irq/132-enp0s29f1:rx-012:00:410
1976881991711,4cyclictest2245082-21latency_hist10:50:000
197689599168,5cyclictest253-21systemd-journal08:39:593
197689599165,5cyclictest21522561ssh09:43:593
1976895991614,1cyclictest0-21swapper/309:50:173
1976895991612,3cyclictest878-21ntpd10:20:153
1976895991612,3cyclictest2362047-21ssh12:07:143
1976891991612,3cyclictest2235253-21ssh10:40:332
1976891991612,3cyclictest2167411-21ssh09:54:382
1976888991613,2cyclictest2297104-21kworker/u8:4+events_unbound12:01:511
1976888991613,2cyclictest2252427-21kworker/u8:2+events_unbound11:20:361
1976888991613,2cyclictest2113578-21kworker/u8:0+events_unbound09:25:001
1976888991612,3cyclictest2309552-21kworker/u8:0+flush-259:011:42:061
1976888991612,3cyclictest2054771-21kworker/u8:1+events_unbound08:45:141
1976888991611,4cyclictest2175376-21apt-get10:00:101
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-012:35:260
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-012:32:120
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-012:11:140
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-011:55:240
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-010:32:170
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-010:10:590
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-010:01:540
1976881991614,1cyclictest61150irq/132-enp0s29f1:rx-009:55:210
1976881991613,2cyclictest61150irq/132-enp0s29f1:rx-011:15:150
1976881991613,2cyclictest61150irq/132-enp0s29f1:rx-011:08:380
1976881991613,2cyclictest61150irq/132-enp0s29f1:rx-010:56:170
1976881991613,2cyclictest61150irq/132-enp0s29f1:rx-010:39:500
1976881991613,2cyclictest61150irq/132-enp0s29f1:rx-009:49:290
1976881991613,2cyclictest61150irq/132-enp0s29f1:rx-009:21:210
1976881991613,1cyclictest61150irq/132-enp0s29f1:rx-010:40:280
1976881991612,3cyclictest2023116-21sh07:54:590
1976881991612,2cyclictest61150irq/132-enp0s29f1:rx-009:50:110
1976881991612,2cyclictest2064982-21turbostat08:34:590
1976895991513,1cyclictest0-21swapper/312:20:013
1976895991512,2cyclictest491ktimers/311:52:423
1976895991510,4cyclictest2182345-21ssh10:04:513
197689599150,2cyclictest2130530-21grep09:30:013
1976891991513,1cyclictest401ktimers/211:40:222
1976891991513,1cyclictest401ktimers/210:50:102
1976891991512,2cyclictest401ktimers/210:32:092
1976891991512,2cyclictest401ktimers/209:19:492
1976891991512,2cyclictest2145434-21kworker/u8:1+flush-259:010:15:112
1976891991511,3cyclictest2401069-21ssh12:34:152
1976891991511,3cyclictest2060125-21/usr/sbin/munin08:30:222
1976891991511,3cyclictest2002916-21kworker/u8:2+flush-259:007:50:282
1976891991510,3cyclictest2049630-21latency_hist08:20:012
197688899158,5cyclictest1997830-21/usr/sbin/munin07:30:201
1976888991512,2cyclictest2212533-21kworker/u8:2+events_unbound10:36:541
1976888991512,2cyclictest2182484-21kworker/u8:4+events_unbound10:53:181
1976888991512,2cyclictest2075190-21turbostat.cron08:45:001
1976888991511,3cyclictest2002916-21kworker/u8:2+events_unbound07:45:231
1976881991513,2cyclictest61950irq/133-enp0s29f1:tx-007:35:180
1976881991513,1cyclictest61150irq/132-enp0s29f1:rx-012:26:320
1976881991513,1cyclictest61150irq/132-enp0s29f1:rx-011:45:380
1976881991513,1cyclictest61150irq/132-enp0s29f1:rx-011:00:400
1976881991513,1cyclictest61150irq/132-enp0s29f1:rx-009:36:060
1976881991513,1cyclictest61150irq/132-enp0s29f1:rx-009:16:450
1976881991513,1cyclictest61150irq/132-enp0s29f1:rx-008:49:020
1976881991512,2cyclictest61150irq/132-enp0s29f1:rx-010:54:060
1976881991512,2cyclictest61150irq/132-enp0s29f1:rx-010:26:360
1976881991512,2cyclictest61150irq/132-enp0s29f1:rx-010:21:370
1976881991512,2cyclictest61150irq/132-enp0s29f1:rx-010:05:590
1976881991512,2cyclictest61150irq/132-enp0s29f1:rx-007:19:170
1976881991511,3cyclictest2133238-21kthreadcore09:30:190
1976895991412,1cyclictest0-21swapper/311:25:003
1976895991411,2cyclictest508-21systemd-logind11:20:003
1976895991411,2cyclictest2234709-21ntp_kernel_pll_10:40:243
1976895991411,2cyclictest2222784-21cron10:34:593
1976895991411,2cyclictest2090520-21kworker/u8:2+flush-259:009:05:243
1976895991411,2cyclictest0-21swapper/308:00:003
1976895991410,3cyclictest2392710-21ssh12:27:273
1976895991410,3cyclictest2349053-21/usr/sbin/munin12:00:263
197689199149,3cyclictest1977307-21cp07:10:012
1976891991412,1cyclictest401ktimers/210:24:172
1976891991410,3cyclictest2294706-21ssh11:21:012
1976891991410,3cyclictest2145839-21ssh09:40:142
1976881991411,2cyclictest61950irq/133-enp0s29f1:tx-007:20:200
1976881991411,2cyclictest61150irq/132-enp0s29f1:rx-008:55:170
1976881991410,2cyclictest2059888-21turbostat08:30:000
1976881991410,2cyclictest2012896-21turbostat07:45:000
197689599139,3cyclictest2272716-21ssh11:06:263
197689599138,4cyclictest22227621kworker/u8:1+rpciod10:49:073
197689599138,3cyclictest0-21swapper/311:29:593
197689599137,4cyclictest2215373-21latency_hist10:30:013
1976895991310,1cyclictest2380229-21kthreadcore12:20:213
197689199139,3cyclictest2333946-21kworker/u8:2+flush-259:011:55:172
197689199139,3cyclictest2303811-21ssh11:29:052
197689199139,3cyclictest2297104-21kworker/u8:4+flush-259:012:20:292
197689199138,3cyclictest253-21systemd-journal12:26:512
1976891991313,0cyclictest2150784-21kworker/2:0+events_power_efficient10:02:562
1976891991310,2cyclictest2095266-21unixbench_singl09:00:282
1976891991310,2cyclictest0-21swapper/209:20:172
1976891991310,2cyclictest0-21swapper/208:24:592
1976888991310,2cyclictest2090535-21cron09:00:011
197688199139,3cyclictest1966271-21kworker/u8:0+flush-259:007:25:150
197688199137,4cyclictest2054771-21kworker/u8:1+flush-259:008:50:210
1976881991310,2cyclictest61150irq/132-enp0s29f1:rx-008:15:160
197689599129,2cyclictest2116339-21fschecks_time09:20:133
197689599129,2cyclictest0-21swapper/312:35:003
197689599129,1cyclictest0-21swapper/310:12:083
197689599128,3cyclictest2297104-21kworker/u8:4+flush-259:012:10:173
197689599128,3cyclictest2038917-21apt-get08:10:013
197689599128,2cyclictest2002684-21sh07:34:593
197689199129,2cyclictest2099019-21kthreadcore09:05:182
197689199129,2cyclictest2013569-21sed07:45:162
197689199128,3cyclictest2106080-21ssh09:10:552
197689199127,4cyclictest2244857-21ssh10:49:372
197689199127,3cyclictest498-21dbus-daemon08:50:012
197689199126,4cyclictest498-21dbus-daemon07:24:592
197688899129,2cyclictest2090520-21kworker/u8:2+events_unbound09:05:141
197688899129,2cyclictest2012920-21kworker/u8:3+flush-259:008:00:191
197688899129,2cyclictest1922655-21kworker/u8:4+flush-259:007:20:241
197688199129,2cyclictest2033404-21unixbench_singl08:00:310
1976881991210,2cyclictest61150irq/132-enp0s29f1:rx-008:10:240
197689599119,1cyclictest0-21swapper/307:10:233
197689599118,1cyclictest0-21swapper/308:55:173
197689599117,3cyclictest253-21systemd-journal08:29:313
197689599117,3cyclictest2145434-21kworker/u8:1+flush-259:009:45:273
197689599117,3cyclictest2033852-21kworker/u8:0+flush-259:008:20:143
197689599117,3cyclictest2002916-21kworker/u8:2+flush-259:007:50:003
197689599116,4cyclictest2064978-21latency_hist08:35:013
197689599115,4cyclictest253-21systemd-journal08:05:003
197689599111,3cyclictest2075189-21latency_hist08:45:003
1976895991110,1cyclictest4950ktimers/311:11:073
1976895991110,1cyclictest491ktimers/310:17:193
197689199119,1cyclictest2276050-21kthreadcore11:10:162
197689199119,1cyclictest0-21swapper/209:45:242
197689199118,2cyclictest2325532-21kworker/u8:1+xprtiod12:35:312
197689199118,2cyclictest2085756-21cpuspeed_turbos08:55:142
197689199118,2cyclictest2055084-21cpuspeed_turbos08:25:172
197689199118,2cyclictest0-21swapper/211:35:002
197689199118,2cyclictest0-21swapper/211:35:002
197689199118,2cyclictest0-21swapper/207:44:592
197689199118,1cyclictest2065659-21tr08:35:162
197689199118,1cyclictest0-21swapper/207:35:222
1976891991110,1cyclictest401ktimers/210:27:502
197688899118,2cyclictest2085236-21kworker/u8:3+events_unbound09:00:221
197688199118,2cyclictest61150irq/132-enp0s29f1:rx-008:05:170
197688199118,2cyclictest61150irq/132-enp0s29f1:rx-007:30:130
197688199117,3cyclictest2054771-21kworker/u8:1+flush-259:008:35:190
197688199111,8cyclictest2070383-21apt-get08:40:110
197689599107,2cyclictest2346208-21unixbench_multi11:55:283
197689599107,2cyclictest2033852-21kworker/u8:0+flush-259:008:15:263
197689599107,2cyclictest0-21swapper/308:50:243
197689599107,1cyclictest0-21swapper/307:15:153
197689599106,2cyclictest0-21swapper/308:50:013
197689599105,3cyclictest2009002-21sed07:40:153
197689199108,1cyclictest401ktimers/209:39:042
197689199108,1cyclictest1976467-21kworker/2:2+events_power_efficient08:50:122
197689199108,1cyclictest1976467-21kworker/2:2+events_power_efficient07:15:122
197689199107,2cyclictest1992891-21fschecks_count07:25:122
197689199107,2cyclictest0-21swapper/208:40:172
197689199105,3cyclictest2034290-21tune2fs08:05:142
19768959996,2cyclictest2022320-21nvmesmart_nvme007:50:263
19768959995,2cyclictest0-21swapper/309:05:013
19768919996,2cyclictest0-21swapper/207:10:012
19768889998,1cyclictest311ktimers/108:10:171
19768889991,6cyclictest1976646-21latency_hist07:10:011
19768819994,3cyclictest514-21in:imuxsock08:24:310
19768819990,6cyclictest1976882-21sort07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional