You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-14 - 20:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack5slot3s.osadl.org (updated Sun Aug 14, 2022 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
274092241163,25sleep00-21swapper/007:08:080
274692239162,23sleep10-21swapper/107:08:521
274912238162,24sleep30-21swapper/307:09:103
255942222192,20sleep20-21swapper/207:05:072
1870921770,3sleep12764999cyclictest11:45:131
3061721630,4sleep32765999cyclictest10:58:163
27653999086,3cyclictest0-21swapper/212:09:432
27653997167,3cyclictest0-21swapper/209:50:152
240622510,3sleep124060-21mailstats11:55:251
221922490,2sleep20-21swapper/210:35:272
85062480,2sleep30-21swapper/311:20:283
190022480,2sleep00-21swapper/008:00:210
253182470,2sleep00-21swapper/012:00:180
62292390,5sleep02764199cyclictest12:30:210
2764199371,11cyclictest27227-21/usr/sbin/munin12:05:230
2764199361,11cyclictest3051-21kworker/0:111:25:290
171882360,9sleep121-21rcuc/109:10:181
2764199355,14cyclictest25752-21munin-run10:45:010
2764199341,11cyclictest18762-21df_inode11:45:140
2764199334,14cyclictest1569-21munin-run09:50:010
237252330,1sleep323726-21proc_pri08:10:243
237252330,1sleep323726-21proc_pri08:10:243
2764199320,18cyclictest7887-21kworker/0:110:13:470
2764999312,4cyclictest6549-21diskstats10:00:221
2764199307,5cyclictest3686-21gltestperf11:10:220
27641993010,3cyclictest772-21kworker/0:011:05:210
2764199296,19cyclictest2867-21kworker/0:108:38:470
2764199291,13cyclictest29136-21kworker/0:112:12:190
27641992911,15cyclictest3430-21munin-run12:25:000
27649992825,2cyclictest0-21swapper/109:41:081
27649992824,3cyclictest0-21swapper/109:24:551
27649992824,3cyclictest0-21swapper/108:29:241
2764999281,24cyclictest29810-21latency_hist07:15:021
2764999281,21cyclictest0-21swapper/110:40:291
2764999280,22cyclictest0-21swapper/111:18:171
2764199289,2cyclictest18398-21kworker/0:009:15:310
2764199289,2cyclictest18398-21kworker/0:009:15:310
27641992812,3cyclictest30707-21kworker/0:111:00:290
27649992724,2cyclictest0-21swapper/109:32:071
27649992723,3cyclictest0-21swapper/107:36:281
2764999272,12cyclictest19124-21df_inode09:15:191
2764999272,12cyclictest19124-21df_inode09:15:191
2764199279,4cyclictest0-21swapper/011:40:290
2764199272,13cyclictest3770-21munin-run09:55:010
2765399260,6cyclictest29195-21kworker/u8:010:00:232
2764999261,23cyclictest0-21swapper/110:05:191
2764999261,22cyclictest23557-21perf10:40:011
2764999261,22cyclictest12908-21ntp_states11:30:321
2764999261,22cyclictest12908-21ntp_states11:30:321
2764999261,19cyclictest0-21swapper/111:07:281
2764199267,16cyclictest30770-21munin-run11:00:010
2764199261,14cyclictest27253-21kworker/0:109:39:050
2764199260,12cyclictest17930-21perf08:00:000
27649992523,1cyclictest0-21swapper/108:10:481
27649992523,1cyclictest0-21swapper/108:10:481
2764999251,22cyclictest0-21swapper/109:55:271
2764999251,21cyclictest25100-21diskstats08:15:161
2764999251,12cyclictest26089-21ntp_states12:00:271
2764199259,3cyclictest27597-21kworker/0:207:13:280
2764199251,12cyclictest27343-21basename09:35:010
2764199251,10cyclictest9800-21kworker/0:112:39:160
2764199251,10cyclictest9800-21kworker/0:112:39:160
2765399247,9cyclictest0-21swapper/209:40:182
2764999249,3cyclictest22035-21timerwakeupswit11:50:261
2764999247,4cyclictest2019-21uname08:35:261
2764999242,3cyclictest10043-21hddtemp_smartct07:40:181
2764999242,20cyclictest0-21swapper/108:55:251
2764999241,7cyclictest21214-21kworker/u8:210:33:511
2764999241,21cyclictest0-21swapper/109:50:241
2764999241,21cyclictest0-21swapper/108:49:421
2764999241,20cyclictest8136-21entropy08:50:141
2764999241,20cyclictest4380-21needreboot08:40:211
2764999241,20cyclictest29148-21diskstats10:55:151
2764999241,20cyclictest15022-21swap07:50:251
2764999241,20cyclictest13720-21memory10:15:271
2764999241,11cyclictest519-21awk11:05:011
2764999241,10cyclictest10015-21latency_hist12:40:011
2764999241,10cyclictest10015-21latency_hist12:40:011
2764199246,2cyclictest3081-21kworker/0:312:25:140
2764199242,9cyclictest13747-21kworker/0:011:39:330
2765399236,9cyclictest12050irq/123-ahci[0011:25:202
2765399232,7cyclictest10020-21munin-run12:40:012
2765399232,7cyclictest10020-21munin-run12:40:012
2765399232,6cyclictest29954-21munin-node07:15:012
2765399231,7cyclictest5359-21munin-run08:45:022
2765399231,12cyclictest0-21swapper/210:58:002
2764999238,3cyclictest13246-21ntp_states09:00:251
27649992312,3cyclictest0-21swapper/111:36:451
2764999231,20cyclictest0-21swapper/111:28:211
2764999231,20cyclictest0-21swapper/110:13:541
2764999231,20cyclictest0-21swapper/107:45:171
2764999231,19cyclictest29770-21sh12:10:191
2764999231,16cyclictest0-21swapper/111:24:201
2764999231,16cyclictest0-21swapper/111:12:191
2764999231,10cyclictest29530-21latency_hist09:40:011
27649992310,4cyclictest0-21swapper/110:45:221
2764999230,21cyclictest0-21swapper/107:34:341
2764199239,3cyclictest20700-21kworker/0:209:25:490
2764199237,3cyclictest8865-21kworker/0:107:40:230
2764199236,4cyclictest2867-21kworker/0:108:55:230
2764199236,4cyclictest0-21swapper/007:20:210
2764199232,3cyclictest16914-21kworker/0:010:27:070
2764199232,10cyclictest12381-21kworker/0:210:22:130
2764199231,9cyclictest13484-21kworker/0:007:51:250
2764199231,13cyclictest6938-21kworker/0:011:20:310
27641992310,3cyclictest22596-21kworker/0:309:24:210
2765999227,6cyclictest12875-21switchtime07:45:303
2765999221,12cyclictest0-21swapper/307:15:293
2765999221,11cyclictest0-21swapper/311:16:463
2765399225,9cyclictest23750-21missed_timers11:55:202
2765399225,4cyclictest10059-21cpuspeed_turbos08:55:132
2765399224,6cyclictest1442-21gltestperf11:05:222
2765399222,5cyclictest21356-21munin-run10:35:022
2765399221,7cyclictest6562-21users07:30:302
2765399221,13cyclictest0-21swapper/211:11:062
2765399221,12cyclictest0-21swapper/211:41:242
2765399221,11cyclictest0-21swapper/212:29:512
2764999222,8cyclictest0-21swapper/112:15:171
27649992218,3cyclictest0-21swapper/107:55:421
2764999221,2cyclictest2958-21cstates07:25:151
2764999221,20cyclictest0-21swapper/107:15:291
2764999221,19cyclictest0-21swapper/110:27:201
2764999221,19cyclictest0-21swapper/109:47:531
2764999221,19cyclictest0-21swapper/108:34:211
2764999221,19cyclictest0-21swapper/108:07:461
2764999221,18cyclictest0-21swapper/107:20:211
27649992211,3cyclictest0-21swapper/110:50:161
2764999221,10cyclictest768-21in:imjournal11:40:141
2764999220,20cyclictest0-21swapper/108:00:251
2764199228,3cyclictest2265-21kworker/0:307:35:490
2764199228,2cyclictest2265-21kworker/0:307:27:010
2764199222,9cyclictest15696-21kworker/0:108:21:410
2764199222,10cyclictest12381-21kworker/0:210:15:290
2764199222,10cyclictest11644-21kworker/0:009:03:410
2764199221,12cyclictest14381-21/usr/sbin/munin09:05:150
2764199220,3cyclictest0-21swapper/012:15:170
2765999213,7cyclictest86850irq/129-enp1s0-12:22:593
2765999212,16cyclictest32271-21cut12:15:233
2765999211,12cyclictest823-21snmpd07:51:553
2765399215,4cyclictest21298-21expr09:20:202
2765399214,9cyclictest86850irq/129-enp1s0-09:55:262
2765399214,9cyclictest30039-21tail12:10:222
2765399212,11cyclictest16353-21diskmemload12:00:262
2765399211,7cyclictest21862-21mailstats11:50:242
2765399211,7cyclictest0-21swapper/210:24:382
2765399211,6cyclictest823-21snmpd07:45:282
2765399211,6cyclictest20025-40dnf11:45:342
2765399211,6cyclictest13687-21unixbench_multi09:00:292
2764999212,8cyclictest0-21swapper/112:30:351
27649992117,3cyclictest0-21swapper/109:28:011
27649992115,3cyclictest2530-21ntp_states12:20:241
2764999211,4cyclictest4008-21diskstats12:25:151
2764999211,19cyclictest0-21swapper/110:21:351
2764999211,19cyclictest0-21swapper/109:06:141
2764999211,19cyclictest0-21swapper/108:20:201
2764199218,4cyclictest16577-21sshd09:10:130
2764199212,12cyclictest15968-21kworker/0:211:50:190
2764199212,11cyclictest24750-21/usr/sbin/munin08:15:180
2764199212,10cyclictest21622-21smart_sda08:05:300
2764199211,9cyclictest3376-21kworker/0:009:59:450
2764199211,9cyclictest21255-21kworker/0:110:45:250
27641992110,3cyclictest12513-21hddtemp_smartct11:30:250
27641992110,3cyclictest12513-21hddtemp_smartct11:30:250
27641992110,3cyclictest12381-21kworker/0:210:36:370
2764199210,13cyclictest20024-21kworker/0:111:55:190
244012210,7sleep339-21rcuc/309:25:313
2765999203,7cyclictest0-21swapper/309:33:423
2765999201,8cyclictest0-21swapper/311:02:583
2765999201,7cyclictest0-21swapper/310:11:313
2765999201,11cyclictest0-21swapper/311:39:233
2765399205,3cyclictest15021-21needreboot11:35:262
2765399204,8cyclictest86850irq/129-enp1s0-08:05:332
2765399204,4cyclictest27731-21expr09:35:182
2765399201,7cyclictest16353-21diskmemload09:45:222
2765399201,7cyclictest0-21swapper/210:49:202
2765399201,7cyclictest0-21swapper/208:15:182
2765399201,6cyclictest0-21swapper/210:19:472
27653992016,3cyclictest533-21awk11:05:022
2765399201,11cyclictest16581-21/usr/sbin/munin09:10:272
2765399201,10cyclictest10605-21/usr/sbin/munin10:10:222
2765399200,6cyclictest18186-21needreboot10:25:242
2764999201,3cyclictest27644-21fschecks_count12:05:221
2764199209,3cyclictest29153-21/usr/sbin/munin08:25:260
2764199207,3cyclictest2265-21kworker/0:307:48:530
2764199207,2cyclictest12381-21kworker/0:210:30:050
2764199206,2cyclictest32356-21kworker/0:107:16:290
2764199205,3cyclictest86750irq/128-enp1s0-08:46:070
2764199204,3cyclictest86650irq/127-enp1s0-08:40:510
16322201,2sleep20-21swapper/207:20:272
80312191,2sleep30-21swapper/310:03:243
315422191,2sleep30-21swapper/309:42:453
2765999191,9cyclictest0-21swapper/311:44:143
2765999191,7cyclictest0-21swapper/312:25:063
2765999191,2cyclictest0-21swapper/310:31:263
27659991912,4cyclictest10395-21grep08:55:183
2765999191,10cyclictest27845-21/usr/sbin/munin07:10:293
2765999191,10cyclictest25016-21/usr/sbin/munin12:00:323
2765999191,10cyclictest14991-21/usr/sbin/munin10:20:233
2765399196,3cyclictest0-21swapper/207:58:032
2765399195,4cyclictest30-21rcuc/207:35:182
2765399195,3cyclictest5918-21which11:15:222
2765399192,15cyclictest15859-21grep09:05:312
2765399191,8cyclictest0-21swapper/210:53:132
2765399191,7cyclictest0-21swapper/212:18:352
2765399191,7cyclictest0-21swapper/209:26:572
2765399191,6cyclictest9511-21taskset10:05:322
2765399191,6cyclictest25903-21sort09:30:242
2765399191,6cyclictest12187-21diskstats11:30:232
2765399191,6cyclictest12187-21diskstats11:30:232
2765399191,5cyclictest0-21swapper/211:20:292
2765399191,11cyclictest0-21swapper/212:30:552
2765399191,10cyclictest823-21snmpd07:53:242
2765399191,10cyclictest6579-21needreboot08:45:272
2765399191,10cyclictest23782-21/usr/sbin/munin10:40:172
2764199196,3cyclictest27253-21kworker/0:109:41:170
2764199195,3cyclictest6938-21kworker/0:011:19:570
2764199191,7cyclictest15696-21kworker/0:108:13:450
2764199191,7cyclictest15696-21kworker/0:108:13:450
2764199191,4cyclictest0-21swapper/010:51:270
27641991913,3cyclictest6047-21ntpq07:30:250
2764199191,10cyclictest8405-21/usr/sbin/munin10:05:340
2764199191,10cyclictest7777-21/usr/sbin/munin08:50:240
2764199190,9cyclictest30841-21kworker/0:308:32:330
75962180,2sleep30-21swapper/308:50:023
5912181,2sleep20-21swapper/208:31:442
35252181,3sleep30-21swapper/309:52:043
2765999187,3cyclictest823-21snmpd11:05:283
2765999183,8cyclictest86550irq/126-enp1s0-10:51:033
2765999183,7cyclictest20051-21sshd08:04:143
2765999182,7cyclictest3048-21/usr/sbin/munin11:10:253
2765999181,9cyclictest823-21snmpd08:32:223
2765999181,9cyclictest823-21snmpd08:22:203
2765999181,8cyclictest1162-21/usr/sbin/munin08:35:233
2765999181,7cyclictest9714-21diskstats07:40:173
2765999181,7cyclictest0-21swapper/312:10:193
2765999181,7cyclictest0-21swapper/310:36:163
27659991813,3cyclictest9398-21latency_hist11:25:013
27659991813,3cyclictest10373-21munin-plugin-st10:10:003
27659991812,3cyclictest20360-21/usr/sbin/munin08:05:223
2765999181,11cyclictest0-21swapper/309:35:393
2765999181,10cyclictest13796-21strings10:15:283
2765999180,17cyclictest0-21swapper/307:39:073
2765399184,7cyclictest86550irq/126-enp1s0-08:20:292
2765399181,6cyclictest29628-21uniq08:25:182
2765399181,6cyclictest0-21swapper/207:15:032
2765399181,5cyclictest22735-21cstates08:10:142
2765399181,5cyclictest22735-21cstates08:10:142
2765399181,4cyclictest0-21swapper/212:20:262
27653991813,3cyclictest9743-21latency_hist08:55:022
27653991812,4cyclictest20128-21/usr/sbin/munin08:05:012
27653991810,5cyclictest848-21sssd_nss09:15:262
27653991810,5cyclictest848-21sssd_nss09:15:262
2764199180,9cyclictest0-21swapper/010:00:210
259802181,2sleep30-21swapper/310:45:153
2765999178,3cyclictest659-21cstates07:20:173
2765999176,3cyclictest86550irq/126-enp1s0-07:30:153
2765999176,3cyclictest5750irq/9-acpi07:30:023
2765999174,3cyclictest86550irq/126-enp1s0-09:14:383
2765999173,3cyclictest86550irq/126-enp1s0-12:33:583
2765999171,9cyclictest0-21swapper/311:56:143
2765999171,9cyclictest0-21swapper/311:32:493
2765999171,9cyclictest0-21swapper/311:32:483
2765999171,8cyclictest823-21snmpd08:27:213
2765999171,6cyclictest15947-21kworker/u8:211:45:163
2765999171,6cyclictest0-21swapper/311:54:273
2765999171,5cyclictest0-21swapper/312:08:113
27659991712,3cyclictest8289-21cat12:35:193
27659991712,3cyclictest8289-21cat12:35:193
27659991712,3cyclictest17547-21vmstat07:55:303
27659991712,3cyclictest14381-21/usr/sbin/munin09:05:253
2765999171,15cyclictest0-21swapper/309:24:523
27659991711,4cyclictest842-21lldpd09:01:563
2765999171,14cyclictest0-21swapper/308:51:003
2765999171,10cyclictest0-21swapper/309:15:223
2765999171,10cyclictest0-21swapper/309:15:223
2765999170,15cyclictest0-21swapper/308:15:393
2765399179,5cyclictest4035-21/usr/sbin/munin07:25:262
2765399171,5cyclictest0-21swapper/207:40:192
257022170,2sleep30-21swapper/310:44:353
188462171,2sleep30-21swapper/310:26:123
11532171,3sleep20-21swapper/208:35:092
2765999163,5cyclictest86850irq/129-enp1s0-09:45:213
2765999161,5cyclictest0-21swapper/309:55:033
27659991611,3cyclictest3384-21/usr/sbin/munin08:40:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional