You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-13 - 02:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Aug 12, 2022 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
79172235162,25sleep20-21swapper/207:08:592
76902222192,20sleep30-21swapper/307:06:023
78292204168,23sleep10-21swapper/107:07:501
78082204167,24sleep00-21swapper/007:07:320
1435821560,2sleep2809999cyclictest07:20:232
809999107103,3cyclictest0-21swapper/209:26:052
8099998884,3cyclictest0-21swapper/210:39:502
79092610,5sleep3810499cyclictest10:45:283
87872600,4sleep341-21ksoftirqd/310:50:143
200492590,4sleep230-21rcuc/207:35:122
154022520,3sleep339-21rcuc/307:25:013
294392420,2sleep10-21swapper/109:10:131
294392420,2sleep10-21swapper/109:10:131
150042340,1sleep10-21swapper/109:50:231
809299311,27cyclictest18515-21ntpq11:10:281
809299301,28cyclictest0-21swapper/109:27:451
809299301,27cyclictest0-21swapper/109:01:481
46782300,4sleep0808799cyclictest10:40:150
809299292,25cyclictest0-21swapper/107:53:031
809299291,26cyclictest0-21swapper/111:27:321
808799291,22cyclictest0-21swapper/009:51:460
808799291,20cyclictest0-21swapper/011:53:410
808799291,18cyclictest86550irq/126-enp1s0-11:34:160
809299283,20cyclictest9218-40dnf08:23:161
809299281,26cyclictest0-21swapper/112:33:171
809299281,26cyclictest0-21swapper/112:33:171
809299281,25cyclictest0-21swapper/111:55:171
809299281,25cyclictest0-21swapper/108:25:181
809299281,25cyclictest0-21swapper/107:28:181
808799281,19cyclictest0-21swapper/012:20:040
8087992810,15cyclictest27409-21sshd10:18:120
8087992810,15cyclictest27409-21sshd10:18:120
810499273,11cyclictest9280-21/usr/sbin/munin12:05:253
809299271,25cyclictest0-21swapper/109:22:381
809299271,23cyclictest0-21swapper/108:50:161
809299270,25cyclictest0-21swapper/112:23:301
8087992712,10cyclictest0-21swapper/011:47:310
808799271,19cyclictest0-21swapper/012:08:570
808799271,19cyclictest0-21swapper/009:41:020
808799271,19cyclictest0-21swapper/009:36:110
808799271,19cyclictest0-21swapper/009:17:090
808799271,18cyclictest0-21swapper/012:38:520
110342270,4sleep339-21rcuc/309:40:213
810499261,22cyclictest30647-21sshd07:59:133
809999262,14cyclictest23754-40dnf11:24:332
809299262,4cyclictest9357-21timerandwakeup09:35:291
809299262,11cyclictest20072-21ntp_states10:00:281
809299261,23cyclictest0-21swapper/109:58:321
809299261,23cyclictest0-21swapper/109:07:081
809299261,22cyclictest29769-21pluginstate10:25:011
809299261,11cyclictest19745-21munin-run07:35:011
8087992612,8cyclictest0-21swapper/012:03:000
808799261,18cyclictest0-21swapper/011:38:310
808799261,18cyclictest0-21swapper/011:28:550
808799261,18cyclictest0-21swapper/011:18:430
808799261,18cyclictest0-21swapper/010:24:500
808799261,17cyclictest0-21swapper/012:30:150
808799261,17cyclictest0-21swapper/012:30:150
808799261,17cyclictest0-21swapper/011:57:420
808799261,17cyclictest0-21swapper/010:53:190
808799261,17cyclictest0-21swapper/010:03:420
8087992611,3cyclictest0-21swapper/012:16:390
810499252,11cyclictest21206-21/usr/sbin/munin10:05:163
810499252,10cyclictest28433-21/usr/sbin/munin11:35:153
810499251,7cyclictest0-21swapper/309:46:543
810499251,21cyclictest784-21NetworkManager10:00:303
810499251,10cyclictest784-21NetworkManager12:10:303
809299251,3cyclictest0-21swapper/110:55:231
809299251,11cyclictest8012-21users10:45:291
809299251,11cyclictest27061-21irqstats11:30:261
809299251,11cyclictest15344-21latency_hist07:25:011
809299250,5cyclictest0-21swapper/112:04:561
808799255,12cyclictest86850irq/129-enp1s0-12:10:230
8087992512,2cyclictest86850irq/129-enp1s0-10:05:190
808799251,17cyclictest0-21swapper/009:21:210
808799251,10cyclictest0-21swapper/010:35:310
808799250,16cyclictest10534-21kworker/u8:011:03:380
19402250,5sleep341-21ksoftirqd/308:05:243
19402250,5sleep341-21ksoftirqd/308:05:243
93722240,11sleep30-21swapper/307:10:243
810499249,3cyclictest5821-21/usr/sbin/munin09:30:203
810499248,3cyclictest17776-21/usr/sbin/munin07:30:263
810499243,6cyclictest86850irq/129-enp1s0-11:24:333
810499242,9cyclictest27797-21/usr/sbin/munin10:20:313
810499241,5cyclictest0-21swapper/310:39:453
810499241,20cyclictest20265-21/usr/sbin/munin12:30:273
810499241,20cyclictest20265-21/usr/sbin/munin12:30:273
810499241,11cyclictest24969-21/usr/sbin/munin09:00:193
810499241,11cyclictest24701-21gltestperf11:25:183
810499241,10cyclictest823-21snmpd09:28:543
810499241,10cyclictest15866-21/usr/sbin/munin12:20:193
810499241,10cyclictest13056-21/usr/sbin/munin11:00:213
809999245,3cyclictest0-21swapper/209:15:352
809299244,5cyclictest0-21swapper/112:05:311
809299241,12cyclictest2428-21crond11:50:011
809299241,11cyclictest32758-21proc_pri09:15:351
809299241,11cyclictest30016-21smart_sda07:55:251
809299241,11cyclictest20482-21cat08:50:021
809299241,10cyclictest22470-21cpu12:35:131
809299241,10cyclictest15630-21latency_hist12:20:011
809299241,10cyclictest10584-21latency_hist10:55:021
809299240,1cyclictest0-21swapper/107:39:051
808799245,10cyclictest26235-21taskset09:00:250
808799244,12cyclictest31743-21kworker/u8:210:28:030
808799241,16cyclictest0-21swapper/011:44:180
808799241,16cyclictest0-21swapper/009:58:180
808799241,16cyclictest0-21swapper/009:47:530
808799241,15cyclictest11383-21kworker/u8:209:11:450
808799241,15cyclictest11383-21kworker/u8:209:11:450
810499239,3cyclictest86850irq/129-enp1s0-12:29:153
810499231,7cyclictest0-21swapper/312:15:413
810499231,4cyclictest6753-21head08:15:273
810499231,4cyclictest17149-21/usr/sbin/munin09:55:183
810499231,11cyclictest16841-21users11:05:293
810499231,10cyclictest17456-21/usr/sbin/munin11:10:283
809299239,2cyclictest19125-21ntpq12:25:261
809299238,2cyclictest11236-21dump-pmu-power12:14:561
809299232,2cyclictest0-21swapper/108:12:381
809299232,19cyclictest0-21swapper/110:33:401
809299231,3cyclictest0-21swapper/109:30:191
809299231,13cyclictest2778-21awk08:10:011
809299231,13cyclictest2778-21awk08:10:011
808799234,4cyclictest0-21swapper/011:05:270
808799233,12cyclictest86550irq/126-enp1s0-11:15:010
808799233,10cyclictest31743-21kworker/u8:210:45:310
808799232,15cyclictest86550irq/126-enp1s0-10:56:050
8087992311,4cyclictest0-21swapper/010:30:330
808799231,14cyclictest0-21swapper/009:25:150
808799230,17cyclictest0-21swapper/012:26:060
808799230,15cyclictest29169-21kworker/u8:109:34:150
810499229,3cyclictest28688-21grep09:05:293
810499226,3cyclictest7066-21/usr/sbin/munin12:00:223
810499226,3cyclictest29964-21/usr/sbin/munin09:10:183
810499226,3cyclictest29964-21/usr/sbin/munin09:10:183
810499226,3cyclictest25597-21/usr/sbin/munin10:15:263
810499226,3cyclictest25597-21/usr/sbin/munin10:15:263
810499226,3cyclictest12050irq/123-ahci[0011:00:023
810499224,4cyclictest16877-21hostname08:40:183
810499223,9cyclictest23581-21cut12:35:253
810499221,9cyclictest0-21swapper/309:55:003
810499221,8cyclictest0-21swapper/310:12:563
810499221,7cyclictest11772-21cat08:30:023
810499221,11cyclictest32146-21grep11:40:283
810499221,10cyclictest22162-21/usr/sbin/munin07:40:233
810499221,10cyclictest11963-21sshd08:30:133
809999226,7cyclictest3057-21taskset09:20:592
809999224,9cyclictest2430-21sh11:50:012
809999224,6cyclictest86550irq/126-enp1s0-11:08:492
809999221,12cyclictest20033-21munin-run12:30:012
809999221,12cyclictest0-21swapper/209:38:562
809299229,3cyclictest28776-21df_inode11:35:151
809299224,9cyclictest0-21swapper/110:06:161
809299222,18cyclictest0-21swapper/108:36:131
809299222,18cyclictest0-21swapper/108:02:351
8092992216,3cyclictest823-21snmpd11:24:561
809299221,2cyclictest0-21swapper/111:43:001
809299221,2cyclictest0-21swapper/110:12:221
809299221,19cyclictest0-21swapper/111:00:191
809299221,14cyclictest0-21swapper/111:53:181
809299221,10cyclictest15521-21cstates11:05:151
808799221,10cyclictest0-21swapper/010:10:250
810499215,3cyclictest29997-21/usr/sbin/munin10:25:153
810499214,8cyclictest86550irq/126-enp1s0-11:30:303
810499211,9cyclictest0-21swapper/309:23:173
810499211,8cyclictest842-21lldpd07:15:233
810499211,8cyclictest0-21swapper/311:49:283
810499211,7cyclictest20028-21taskset08:46:023
8104992113,5cyclictest1088-21/usr/sbin/munin10:30:283
810499211,10cyclictest0-21swapper/310:41:093
810499211,10cyclictest0-21swapper/308:00:133
809999216,8cyclictest86850irq/129-enp1s0-12:00:272
809999216,3cyclictest10304-21idleruntime08:25:172
809999215,7cyclictest0-21swapper/211:40:262
809999214,9cyclictest32-21ksoftirqd/211:30:232
809999212,10cyclictest4963-21if_enp1s010:40:172
809999212,10cyclictest30195-21irqstats09:10:212
809999212,10cyclictest30195-21irqstats09:10:212
809999211,6cyclictest22454-21/usr/sbin/munin12:35:282
809999211,11cyclictest477-21munin-run08:05:022
809999211,11cyclictest25935-21df10:15:152
809999211,11cyclictest25935-21df10:15:152
809299211,8cyclictest0-21swapper/110:15:131
809299211,8cyclictest0-21swapper/110:15:131
8092992117,3cyclictest0-21swapper/110:37:161
8092992117,3cyclictest0-21swapper/107:46:181
809299211,3cyclictest0-21swapper/110:40:231
809299211,3cyclictest0-21swapper/107:40:371
809299211,2cyclictest0-21swapper/108:59:401
809299211,18cyclictest0-21swapper/110:26:071
809299211,18cyclictest0-21swapper/108:41:071
809299211,18cyclictest0-21swapper/108:17:591
809299210,5cyclictest0-21swapper/109:41:581
808799215,3cyclictest86750irq/128-enp1s0-08:50:260
808799214,8cyclictest25918-21grep07:45:270
8087992111,2cyclictest86550irq/126-enp1s0-11:21:020
810499205,3cyclictest86850irq/129-enp1s0-11:55:213
810499204,10cyclictest9218-40dnf08:23:153
810499203,9cyclictest0-21swapper/309:38:563
810499201,5cyclictest32643-21awk09:15:333
810499201,5cyclictest15684-21cpuspeed_turbos07:25:133
810499201,10cyclictest0-21swapper/311:50:093
810499200,10cyclictest0-21swapper/311:15:183
809999207,4cyclictest2784-21awk08:10:022
809999207,4cyclictest2784-21awk08:10:012
809999207,3cyclictest86850irq/129-enp1s0-10:09:242
809999206,3cyclictest9649-21sed07:10:272
809999204,4cyclictest27790-21sshd10:20:122
809999202,9cyclictest11476-21/usr/sbin/munin12:10:302
809999202,10cyclictest13766-21munin-run08:35:012
809999201,9cyclictest17792-21needreboot09:55:242
809999201,9cyclictest0-21swapper/210:11:022
809999201,10cyclictest5501-21grep11:55:222
809299201,3cyclictest20014-21awk11:15:171
8092992012,2cyclictest0-21swapper/107:16:231
809299201,18cyclictest0-21swapper/109:46:511
808799206,3cyclictest86750irq/128-enp1s0-07:40:160
808799202,5cyclictest24746-21munin-run09:00:010
808799201,9cyclictest19745-21munin-run07:35:010
808799201,7cyclictest13402-21NetworkManager08:30:290
808799201,5cyclictest2315-21grep08:05:270
808799201,5cyclictest2315-21grep08:05:270
808799201,13cyclictest0-21swapper/007:50:170
808799201,10cyclictest29820-21ntp_states07:55:220
808799201,10cyclictest18381-21/usr/sbin/munin08:45:250
810499192,14cyclictest24357-21/usr/sbin/munin07:45:143
810499191,5cyclictest3269-21cat08:10:123
810499191,5cyclictest23620-21/usr/sbin/munin08:55:193
810499191,3cyclictest0-21swapper/308:35:213
809999196,3cyclictest25214-21ps11:25:252
809999193,8cyclictest86850irq/129-enp1s0-09:45:142
809999192,9cyclictest581-21systemd-journal12:17:112
809999191,9cyclictest23107-21df08:55:132
809999191,8cyclictest0-21swapper/211:50:052
809999191,6cyclictest9644-21sshd12:05:212
809999191,6cyclictest86850irq/129-enp1s0-11:18:162
8099991913,4cyclictest10606-21cut07:15:022
809999191,13cyclictest0-21swapper/208:15:122
809999191,12cyclictest0-21swapper/209:52:462
809299191,17cyclictest0-21swapper/108:31:401
809299191,17cyclictest0-21swapper/107:12:531
808799196,3cyclictest12407-21gunzip07:15:240
808799193,2cyclictest86550irq/126-enp1s0-08:45:010
808799192,9cyclictest32019-21ntp_states08:00:230
808799191,4cyclictest8292-21/usr/sbin/munin07:10:280
8087991913,3cyclictest823-21snmpd07:28:310
810499181,5cyclictest21492-21load08:50:193
8104991812,3cyclictest27076-21cut07:50:153
809999184,4cyclictest31957-21munin-plugin-st10:30:012
809999182,5cyclictest86850irq/129-enp1s0-12:33:192
809999182,5cyclictest86850irq/129-enp1s0-12:33:192
809999182,5cyclictest0-21swapper/208:23:152
809999181,9cyclictest28523-21sh07:55:012
809999181,8cyclictest86850irq/129-enp1s0-10:51:382
809999181,8cyclictest26329-21timerandwakeup09:00:262
809999181,8cyclictest0-21swapper/209:41:302
809999181,7cyclictest16690-21fschecks_time08:40:172
809999181,5cyclictest20044-21grep10:00:272
809999181,5cyclictest17773-21sshd07:30:142
809999181,3cyclictest0-21swapper/211:12:312
8099991812,4cyclictest30720-21dump-pmu-power08:00:022
8099991812,3cyclictest836-21runrttasks07:25:062
8099991812,3cyclictest28433-21/usr/sbin/munin11:35:152
809999181,15cyclictest0-21swapper/210:30:462
809999181,11cyclictest0-21swapper/207:40:222
809999181,10cyclictest0-21swapper/210:55:142
808799184,3cyclictest5582-21grep08:15:160
808799183,4cyclictest13372-21sadc07:20:090
808799181,9cyclictest4063-21ntp_states08:10:210
808799181,11cyclictest0-21swapper/008:20:190
218112181,2sleep00-21swapper/007:38:360
810499170,9cyclictest0-21swapper/307:35:153
809999176,3cyclictest86550irq/126-enp1s0-08:50:122
809999174,6cyclictest86750irq/128-enp1s0-10:45:222
809999173,7cyclictest3645-21grep08:10:172
809999171,9cyclictest0-21swapper/212:21:042
809999171,7cyclictest30-21rcuc/211:00:282
809999171,4cyclictest6169-21diskstats09:30:222
809999171,3cyclictest28168-21needreboot09:05:212
809999171,3cyclictest0-21swapper/207:45:112
8099991713,2cyclictest20344-21idleruntime-cro08:50:012
8099991712,3cyclictest13992-21acpi08:35:112
808799172,4cyclictest27510-21df09:05:130
808799171,13cyclictest9598-21/usr/sbin/munin08:25:230
8087991711,3cyclictest14860-21cat08:35:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional