You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-05-31 - 17:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Pentium N4200 @1100 MHz, Linux 5.9.1-rt20 (Profile)

Latency plot of shadow in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot4s.osadl.org (updated Wed May 31, 2023 12:44:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
177242296225,30sleep30-21swapper/307:05:343
177242296225,30sleep30-21swapper/307:05:343
177682285212,61sleep20-21swapper/207:05:582
177682285212,61sleep20-21swapper/207:05:582
174482281215,55sleep10-21swapper/107:05:281
174482281215,55sleep10-21swapper/107:05:281
179622245182,48sleep00-21swapper/007:07:410
179622245182,48sleep00-21swapper/007:07:410
44021310,3sleep0101rcuc/010:05:340
1857121210,3sleep331-21ksoftirqd/307:10:013
2568421120,2sleep30-21swapper/309:50:203
314172430,2sleep20-21swapper/207:35:262
257292400,3sleep318073-21kworker/3:108:35:293
212212400,3sleep09-21ksoftirqd/007:15:000
1846999380,9cyclictest8891-21runrttasks11:50:050
18469993311,5cyclictest8324-21grep07:55:300
1846999331,13cyclictest758-21apt-get08:55:170
1848199321,27cyclictest9154-21python309:15:023
1848199321,27cyclictest9154-21python309:15:013
18469993215,15cyclictest11629-21latency11:55:200
18469993214,16cyclictest5064-21processes11:40:260
18469993212,6cyclictest18810-21sed08:20:320
1846999318,5cyclictest25752-21expr09:50:200
1846999317,6cyclictest9160-21wc09:15:000
1846999317,6cyclictest25043-21awk11:10:370
18469993112,17cyclictest6320-21/usr/sbin/munin11:45:350
18469993111,4cyclictest577-21dpkg07:40:190
1846999309,5cyclictest2971-21entropy07:45:210
1846999307,6cyclictest16475-21cat12:05:530
1846999306,6cyclictest889-21gdbus08:50:260
1846999301,9cyclictest10402-21apt-get10:40:190
18469993011,5cyclictest26149-21grep11:15:210
18469993011,5cyclictest19123-21expr09:35:230
1846999299,2cyclictest11745-21munin-plugin-st09:20:000
1846999299,2cyclictest11745-21munin-plugin-st09:20:000
1846999297,6cyclictest12786-21apt-check10:45:060
1846999297,4cyclictest39-21kauditd10:35:000
1846999292,6cyclictest13916-21apt-get08:10:180
1846999291,7cyclictest11544-21lsb_release08:05:000
18469992912,15cyclictest20074-21missed_timers11:00:210
18469992912,14cyclictest27890-21latency_hist07:30:010
1846999291,12cyclictest23139-21apt-get07:20:050
18469992911,2cyclictest17597-21cstates12:10:190
1846999291,10cyclictest8248-21apt-get10:35:210
1846999288,4cyclictest5425-21cut07:50:230
1846999287,5cyclictest16591-21cut09:30:220
1846999281,8cyclictest1804-21grep11:35:000
1846999281,6cyclictest17268-21apt-get10:55:170
1846999281,11cyclictest23717-21latency_hist12:25:010
1846999281,10cyclictest23354-21sed07:20:000
1846999280,10cyclictest28871-21fschecks_count07:30:220
1846999279,3cyclictest26978-21/usr/sbin/munin08:40:190
1846999279,2cyclictest30805-21latency_hist08:49:590
1846999276,5cyclictest30783-21cat10:00:330
1846999276,5cyclictest22009-21processes09:40:280
1846999273,7cyclictest2968-21apt-get09:00:150
1846999271,9cyclictest29236-21snmp_o5powerctl12:35:360
1846999271,9cyclictest26259-21df_inode12:30:200
1846999271,9cyclictest12095-21cpuspeed_turbos09:20:200
18469992710,4cyclictest14547-21awk12:00:300
1846999270,9cyclictest28519-21snmp_o5powerctl09:55:300
1846999270,10cyclictest5052-21turbostat10:25:000
232392260,11sleep09-21ksoftirqd/008:30:270
1846999268,5cyclictest32173-21awk07:35:370
1846999265,3cyclictest0-21swapper/008:25:220
1846999261,9cyclictest20399-21ntp_states12:15:210
1846999261,9cyclictest1753-21diskstats10:10:210
1848199257,5cyclictest31-21ksoftirqd/308:33:573
1848199253,8cyclictest31-21ksoftirqd/308:19:553
1848199252,15cyclictest29663-21grep10:00:193
1846999258,2cyclictest2480-21idleruntime11:35:240
1846999251,10cyclictest21573-21apt-get11:05:180
1848199245,7cyclictest31-21ksoftirqd/310:27:313
1848199244,8cyclictest31-21ksoftirqd/308:54:403
1848199244,7cyclictest31-21ksoftirqd/311:38:323
1848199243,8cyclictest31-21ksoftirqd/311:32:253
1848199243,8cyclictest31-21ksoftirqd/311:15:273
1848199243,7cyclictest31-21ksoftirqd/309:57:313
1848199243,6cyclictest31-21ksoftirqd/307:30:233
1848199242,8cyclictest31-21ksoftirqd/309:29:193
1848199242,8cyclictest31-21ksoftirqd/309:03:493
1848199242,8cyclictest31-21ksoftirqd/308:24:503
1848199242,8cyclictest31-21ksoftirqd/307:55:423
1848199242,8cyclictest31-21ksoftirqd/307:40:283
1848199242,7cyclictest31-21ksoftirqd/312:40:003
1848199242,7cyclictest31-21ksoftirqd/308:10:003
1848199242,14cyclictest30141-21taskset11:25:183
18481992413,6cyclictest12315-21http10:45:013
1848199241,18cyclictest27938-21python307:30:013
18473992418,4cyclictest19-21ksoftirqd/112:40:011
18473992415,6cyclictest23148-21users12:20:341
1846999249,2cyclictest15561-21hwlatdetect10:50:230
1846999242,7cyclictest5101-21apt-get09:05:160
1846999240,7cyclictest25476-21pmu-power08:35:250
1848199235,3cyclictest12313-21python310:45:013
1848199233,7cyclictest31-21ksoftirqd/307:53:133
1848199233,6cyclictest31-21ksoftirqd/308:05:183
1848199232,8cyclictest31-21ksoftirqd/311:06:013
1848199232,8cyclictest31-21ksoftirqd/310:38:303
1848199232,8cyclictest31-21ksoftirqd/309:22:313
1848199232,8cyclictest31-21ksoftirqd/309:07:193
1848199232,6cyclictest31-21ksoftirqd/311:47:413
1848199232,6cyclictest31-21ksoftirqd/311:22:063
1848199231,7cyclictest31-21ksoftirqd/311:58:353
1848199231,7cyclictest31-21ksoftirqd/309:35:253
1848199231,7cyclictest31-21ksoftirqd/307:20:333
1848199231,18cyclictest9153-21python309:15:013
1848199230,7cyclictest31-21ksoftirqd/310:33:303
1848199223,8cyclictest31-21ksoftirqd/308:26:353
1848199222,7cyclictest31-21ksoftirqd/312:03:153
1848199222,7cyclictest31-21ksoftirqd/310:10:093
1848199222,6cyclictest31-21ksoftirqd/311:50:063
1848199221,7cyclictest31-21ksoftirqd/308:59:383
1848199221,5cyclictest1665-21snmpd07:36:473
1848199220,9cyclictest31-21ksoftirqd/312:26:153
1848199220,8cyclictest31-21ksoftirqd/311:41:063
1848199213,7cyclictest31-21ksoftirqd/311:10:043
1848199212,8cyclictest31-21ksoftirqd/307:46:253
1848199212,7cyclictest31-21ksoftirqd/312:12:483
1848199212,7cyclictest31-21ksoftirqd/310:20:313
1848199212,6cyclictest31-21ksoftirqd/310:07:163
1848199211,8cyclictest31-21ksoftirqd/312:05:233
1848199211,7cyclictest31-21ksoftirqd/312:16:573
1848199211,7cyclictest31-21ksoftirqd/309:40:163
1848199211,6cyclictest31-21ksoftirqd/308:48:233
1848199211,6cyclictest28165-21runrttasks08:40:423
1848199211,13cyclictest23228-21apt-get09:45:183
1848199210,8cyclictest31-21ksoftirqd/308:01:533
1848199210,7cyclictest31-21ksoftirqd/310:54:183
1848199210,6cyclictest31-21ksoftirqd/307:15:583
18475992115,4cyclictest25-21ksoftirqd/207:20:002
18475992114,5cyclictest29935-21snmp_o5powerctl08:45:312
18475992114,5cyclictest25246-21cat09:50:002
18475992114,5cyclictest19880-21ntp_kernel_err07:10:342
18475992114,5cyclictest1170-21sed07:40:262
1847599211,17cyclictest13222-21fschecks_count10:45:212
1847599210,7cyclictest121850lldpd12:14:102
18473992114,5cyclictest17064-21snmp_o5powerctl08:15:281
18473992113,6cyclictest21705-21users08:25:321
18473992113,6cyclictest15463-21perl10:50:221
18473992112,6cyclictest13704-21cut08:10:011
1847399210,5cyclictest0-21swapper/110:10:161
18469992112,6cyclictest12450-21missed_timers08:05:260
18481992017,2cyclictest301rcuc/310:16:373
1848199201,5cyclictest31-21ksoftirqd/309:30:303
1848199200,6cyclictest31-21ksoftirqd/312:31:093
1848199200,6cyclictest31-21ksoftirqd/311:02:593
1847599202,12cyclictest26071-21taskset12:30:172
18475992013,5cyclictest25432-21ls09:50:182
18475992012,6cyclictest2888-21cut09:00:002
18475992012,6cyclictest27520-21ntp_kernel_err08:40:242
18475992012,5cyclictest4241-21systemctl10:20:002
18475992011,7cyclictest3279-21perl09:00:242
1847599201,13cyclictest23711-21apt-get11:10:212
18473992013,5cyclictest6069-21ls11:45:001
18473992013,5cyclictest31798-21dpkg10:05:191
18473992013,5cyclictest28050-21date12:35:011
18473992012,6cyclictest30631-21cat11:25:221
18473992012,6cyclictest10357-21date10:40:011
18473992012,5cyclictest24536-21cat08:35:011
18473992011,7cyclictest25857-21cat09:50:211
18473992010,7cyclictest17880-21ubuntu-distro-i09:31:241
1847399200,5cyclictest0-21swapper/109:16:051
1847399200,5cyclictest0-21swapper/109:16:051
1847399200,4cyclictest0-21swapper/110:41:541
18469992013,5cyclictest24167-21tune2fs12:25:210
18469992012,5cyclictest31402-21grep11:25:300
18481991916,2cyclictest301rcuc/310:59:053
1848199190,7cyclictest31-21ksoftirqd/312:21:033
18475991915,2cyclictest8333-21unixbench_singl07:55:292
18475991915,2cyclictest7980-21sh10:35:002
18475991915,2cyclictest7117-21open_files11:45:292
18475991912,5cyclictest9784-21expr08:00:192
18475991912,5cyclictest8981-21nfs4_client10:35:272
18475991912,5cyclictest7682-21grep09:10:252
18475991912,5cyclictest26745-21awk07:25:272
18475991912,5cyclictest24781-21ls08:35:192
18475991912,5cyclictest16540-21cat12:05:532
18475991911,6cyclictest28652-21wc11:20:262
18475991911,6cyclictest2825-21ls07:45:192
18475991911,6cyclictest11345-21tune2fs11:55:182
1847599191,13cyclictest9801-40apt-check09:15:072
1847599191,13cyclictest9801-40apt-check09:15:072
1847599191,13cyclictest28209-21apt-get12:35:212
1847399192,12cyclictest8924-21apt-get11:50:181
18473991915,2cyclictest5737-21pmu-power07:50:281
18473991915,2cyclictest28443-21sendmail_mailst09:55:271
18473991915,2cyclictest28240-21apt-config07:30:001
18473991915,2cyclictest13613-21missed_timers10:45:241
18473991913,4cyclictest8027-21ntp_states09:10:301
18473991913,3cyclictest5725-21runrttasks10:29:031
18473991912,5cyclictest5081-21wc10:25:001
18473991912,5cyclictest21306-21forks09:40:231
18473991912,5cyclictest19116-21dpkg07:10:141
18473991912,5cyclictest16149-21cut09:30:001
18473991912,5cyclictest13850-21sed12:00:211
18473991912,4cyclictest8200-21latency_hist11:50:001
18473991911,6cyclictest31731-21uname08:50:211
18473991911,6cyclictest2661-21cat11:35:261
18473991911,6cyclictest23373-21apt-config07:20:011
18473991911,5cyclictest9889-21tune2fs08:00:211
1847399191,13cyclictest26784-21apt-get08:40:171
1847399191,13cyclictest25842-21apt-get11:15:201
1847399191,13cyclictest17273-21runrttasks10:55:181
1847399190,4cyclictest0-21swapper/112:26:541
1847399190,4cyclictest0-21swapper/111:23:221
1847399190,4cyclictest0-21swapper/108:45:361
18469991912,5cyclictest28812-21pluginstate11:20:270
18469991912,5cyclictest16109-21wc08:15:190
171112190,4sleep20-21swapper/209:30:282
18475991814,2cyclictest4476-21runrttasks10:20:022
18475991814,2cyclictest22830-21swap12:20:312
18475991814,2cyclictest18938-21/usr/sbin/munin09:35:282
18475991813,3cyclictest21165-21open_inodes08:25:252
18475991812,4cyclictest32516-21tune2fs11:30:222
18475991812,4cyclictest30711-21wc11:25:232
18475991812,4cyclictest22121-21swap09:40:322
18475991812,4cyclictest1739-21smbd08:50:272
18475991811,5cyclictest9511-21cat11:50:252
18475991811,5cyclictest29445-21sed10:00:002
18475991811,5cyclictest2220-21cat10:10:252
18475991811,5cyclictest19999-21expr12:15:162
18475991811,5cyclictest18394-21expr08:20:202
18475991811,5cyclictest16746-21grep08:15:232
18475991811,5cyclictest15294-21expr10:50:202
18475991811,5cyclictest12226-21basename08:05:232
18475991811,5cyclictest12119-21seq09:20:192
1847599181,13cyclictest4185-21apt-get11:40:192
1847599181,12cyclictest29658-21apt-get10:00:202
1847599181,12cyclictest24894-21snmp_o5powerctl07:20:342
18475991810,5cyclictest6050-21which09:05:272
18475991810,5cyclictest26282-21basename11:15:232
18473991812,4cyclictest24241-21snmp_o5powerctl09:45:301
18473991812,4cyclictest20205-21grep12:15:191
18473991811,5cyclictest25373-21ntp_kernel_pll_08:35:241
18473991811,5cyclictest15494-21expr12:05:211
18473991811,5cyclictest12127-21ls11:55:261
18473991811,5cyclictest1045-21idleruntime07:40:241
1847399181,13cyclictest8048-21snmp_o5powerctl07:55:271
18473991810,6cyclictest2937-21expr07:45:201
1847399180,4cyclictest0-21swapper/108:55:131
18469991811,5cyclictest15229-21sed09:25:320
201972170,2sleep2241rcuc/211:00:232
1847599179,6cyclictest24308-21grep12:25:202
18475991713,2cyclictest28621-21/usr/sbin/munin07:30:322
18475991711,4cyclictest21679-21cat11:05:202
18475991711,4cyclictest15095-21run-parts09:25:282
18475991711,4cyclictest10681-21cut10:40:232
1847599171,12cyclictest5920-21snmp_o5powerctl07:50:292
1847599171,12cyclictest3043-21snmp_o5powerctl11:35:322
1847599171,12cyclictest14930-21snmp_o5powerctl08:10:272
18475991710,5cyclictest18326-21grep10:55:352
1847399178,6cyclictest19880-21cat11:00:191
1847399175,9cyclictest3765-21runrttasks09:00:321
18473991713,2cyclictest22304-21netstat11:05:271
18473991711,4cyclictest32709-21cat11:30:221
18473991711,4cyclictest29812-21expr10:00:191
18473991711,4cyclictest17950-21cat12:10:211
18473991711,4cyclictest14399-21if_err_enp2s008:10:211
1847399171,12cyclictest23197-21python307:20:001
18473991710,5cyclictest5204-21expr09:05:161
18473991710,5cyclictest26135-21expr07:25:191
18473991710,5cyclictest25633-21date11:15:001
18473991710,5cyclictest21050-21cut09:40:011
18473991710,5cyclictest12638-21cat09:20:241
1847399170,6cyclictest0-21swapper/107:35:171
1847399170,15cyclictest20294-21wc08:25:011
18469991711,4cyclictest23551-21tune2fs09:45:220
1847599161,11cyclictest22504-21runrttasks08:30:202
1847599161,11cyclictest13216-21apt-get12:00:162
18475991610,4cyclictest32293-21interrupts10:05:252
18473991612,2cyclictest7046-21sendmail_mailst10:30:241
18473991611,3cyclictest3842-21runrttasks10:16:441
18469991612,2cyclictest1215-21runrttasks10:15:430
1847599150,14cyclictest0-21swapper/210:27:032
1846999150,1cyclictest0-21swapper/010:25:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional