You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 16:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Pentium N4200 @1100 MHz, Linux 5.9.1-rt18 (Profile)

Latency plot of shadow in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot4s.osadl.org (updated Mon Mar 20, 2023 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
55992261193,34sleep30-21swapper/307:07:193
54082243192,37sleep10-21swapper/107:05:361
54482241193,35sleep20-21swapper/207:05:572
57542240193,34sleep00-21swapper/007:08:410
1807621180,3sleep0614999cyclictest12:30:250
1228521130,1sleep30-21swapper/308:35:163
1852221080,1sleep30-21swapper/307:35:223
310602480,1sleep20-21swapper/210:30:462
223652380,3sleep0614999cyclictest07:45:160
233022370,4sleep323301-21fschecks_time11:30:243
21662370,1sleep00-21swapper/010:40:210
6164993612,21cyclictest19740-21dpkg12:35:023
6164993513,19cyclictest19280-21readlink12:35:003
6164993211,19cyclictest11616-21sed12:15:273
6164993211,19cyclictest1153-21cat11:55:003
6164993211,18cyclictest23597-21expr10:15:173
6164993111,18cyclictest17010-21expr10:00:193
6164993111,18cyclictest17010-21expr10:00:193
6164993110,18cyclictest8355-21expr10:55:193
616499310,8cyclictest22692-21ntpq07:45:213
616499309,19cyclictest28384-21cat11:40:263
6164993010,18cyclictest29000-21dpkg09:15:153
59052300,2sleep20-21swapper/212:05:012
616499291,25cyclictest31313-21grep08:05:163
6164992912,15cyclictest15221-21open_inodes11:10:283
616499287,4cyclictest5891-21sed08:20:213
616499287,3cyclictest1936-21gltestperf10:40:193
616499286,5cyclictest12876-21cat11:05:193
616499281,8cyclictest6073-21chrt12:05:183
616499281,6cyclictest3825-21gltestperf12:00:213
616499280,9cyclictest30178-21ntp_kernel_pll_11:45:273
616499280,8cyclictest10745-21snmp_o5powerctl08:30:253
614999282,21cyclictest10156-21apt-get11:00:190
616499271,9cyclictest14461-21fschecks_count09:55:163
616499271,8cyclictest1306-21latency_hist10:40:003
616499271,11cyclictest23021-21latency_hist10:15:013
6156992715,6cyclictest24296-40apt-check07:50:062
616499269,3cyclictest30266-21cat10:30:223
616499267,6cyclictest4068-21snmp_o5powerctl09:30:303
616499267,4cyclictest836-21gdbus10:50:263
616499266,2cyclictest0-21swapper/309:37:263
616499265,4cyclictest0-21swapper/308:15:263
616499261,8cyclictest8430-21fschecks_count12:10:193
616499261,8cyclictest31157-21watchprog-snmpd09:20:143
616499261,8cyclictest30700-21munin-run08:05:013
616499261,8cyclictest23341-21awk09:00:203
616499261,10cyclictest18285-21apt-get08:50:123
616499260,4cyclictest7815-21apt-get09:40:173
616499259,14cyclictest740-21cron09:50:003
616499258,6cyclictest13464-21expr07:25:153
616499256,5cyclictest25285-21cat07:50:243
616499256,3cyclictest21679-21turbostat07:45:003
616499256,2cyclictest0-21swapper/308:10:383
616499251,8cyclictest1830-21/usr/sbin/munin09:25:223
616499251,8cyclictest15477-21/usr/sbin/munin07:30:143
616499251,5cyclictest8370-21ntp_states08:25:203
616499250,8cyclictest15849-21ntpq12:25:273
616499250,8cyclictest12543-21apt-config12:20:003
6156992514,7cyclictest277-21systemd-journal07:40:002
615699250,7cyclictest396-21turbostat08:10:012
22872250,3sleep10-21swapper/110:40:221
147922250,2sleep20-21swapper/212:24:492
616499247,2cyclictest27667-21snmp_o5powerctl07:55:263
616499246,5cyclictest19709-21mailstats11:20:233
616499245,4cyclictest28849-21unixbench_singl10:25:273
616499245,4cyclictest26744-21users10:20:293
616499245,2cyclictest0-21swapper/308:42:193
616499243,4cyclictest16160-21apt-get08:45:153
616499241,7cyclictest9882-21mailstats07:15:243
616499241,7cyclictest20204-21sed10:05:283
616499241,6cyclictest7606-21sed07:10:213
616499240,8cyclictest27862-21sensors09:10:233
616499240,8cyclictest25642-21awk09:05:253
616499240,8cyclictest1940-21sed11:55:203
616499240,8cyclictest12452-21grep07:20:313
616499240,8cyclictest11448-21grep11:00:363
6152992412,7cyclictest3461-21grep12:00:181
614999242,16cyclictest27538-21chrt10:25:130
118602240,3sleep225-21ksoftirqd/207:20:212
616499237,2cyclictest4537-21pmu-power10:45:273
616499231,7cyclictest20627-21/usr/sbin/munin08:55:143
616499231,7cyclictest20627-21/usr/sbin/munin08:55:143
616499231,6cyclictest25407-21tune2fs11:35:173
616499231,6cyclictest20849-21cut11:25:143
616499230,7cyclictest17222-21cat11:15:203
616499230,7cyclictest12649-21sed09:50:273
615699238,13cyclictest16108-21turbostat10:00:002
615699230,6cyclictest9496-21munin-plugin-st08:29:592
6149992311,8cyclictest7395-21dpkg09:40:000
615699229,9cyclictest19103-21apt-get12:35:062
6156992217,3cyclictest24022-21latency_hist07:50:012
6156992215,4cyclictest22739-21munin-run11:30:012
615699221,4cyclictest27272-21cron10:25:002
615699221,3cyclictest5235-40apt-check08:20:042
6156992213,7cyclictest5614-21dpkg10:50:002
6156992211,8cyclictest5036-21sed08:19:592
615699221,14cyclictest16836-21apt-get10:00:162
615699221,14cyclictest16836-21apt-get10:00:152
615699220,6cyclictest9936-21/usr/sbin/munin09:45:232
615299220,5cyclictest0-21swapper/108:11:101
615299220,4cyclictest0-21swapper/110:39:571
6149992216,3cyclictest5418-21apt-config12:05:000
262532220,2sleep20-21swapper/211:35:262
6156992115,3cyclictest5349-21latency_hist12:04:592
615699211,3cyclictest19135-21python312:35:012
6156992113,6cyclictest31346-21df_inode09:20:172
6152992114,5cyclictest29415-21apt11:45:181
615299210,5cyclictest0-21swapper/112:33:081
615299210,5cyclictest0-21swapper/110:26:541
615299210,4cyclictest0-21swapper/111:33:231
615299210,4cyclictest0-21swapper/111:21:191
614999212,16cyclictest11777-21grep07:20:200
6149992113,4cyclictest7069-21sh09:39:590
6149992112,6cyclictest11588-21grep12:15:280
615699207,11cyclictest16961-21idleruntime-cro12:30:012
615699202,16cyclictest31355-21cat11:49:592
6156992016,2cyclictest23189-21sh11:30:232
6156992016,2cyclictest15919-21perf08:44:592
6156992015,3cyclictest17530-21updatedb.mlocat07:35:002
6156992015,3cyclictest12273-21python312:20:012
615699201,3cyclictest28754-21munin-run09:15:012
6156992013,5cyclictest1076-21df_inode09:25:162
6156992012,5cyclictest13781-21tr08:39:592
615699201,16cyclictest7432-21apt-key09:40:012
615699201,16cyclictest30710-21lsb_release08:05:012
6156992011,6cyclictest18906-21wc08:50:162
615699200,4cyclictest26615-21munin-run09:10:012
6152992016,2cyclictest5729-21latency09:35:211
6152992016,2cyclictest11077-21processes11:00:311
6152992015,3cyclictest12010-21rt-features07:20:221
6152992013,5cyclictest8950-21run-parts10:55:261
6152992013,5cyclictest25637-21tune2fs10:20:171
6152992013,5cyclictest23249-21awk07:45:271
6152992013,5cyclictest18768-21if_err_enp2s008:50:161
6152992013,5cyclictest12578-21tune2fs11:05:171
6152992012,6cyclictest7406-21load07:10:191
615299201,13cyclictest20429-21taskset08:55:131
615299201,13cyclictest20429-21taskset08:55:131
615299200,4cyclictest0-21swapper/112:11:041
615299200,4cyclictest0-21swapper/111:53:181
615299200,4cyclictest0-21swapper/111:28:231
615299200,4cyclictest0-21swapper/110:54:531
615299200,4cyclictest0-21swapper/110:33:211
615299200,4cyclictest0-21swapper/108:00:491
6149992013,5cyclictest31936-21ls10:35:160
6149992013,5cyclictest23122-21expr11:30:220
6149992013,4cyclictest12748-21update-motd-upd12:20:020
6149992012,6cyclictest21202-21ntpq08:55:210
6149992012,6cyclictest21202-21ntpq08:55:210
614999201,13cyclictest5435-21grep08:20:080
614999200,5cyclictest0-21swapper/010:53:350
615699193,14cyclictest25126-21latency_hist10:20:002
6156991914,3cyclictest1894-21snmpd10:43:142
615699191,3cyclictest277-21systemd-journal08:15:012
6156991913,3cyclictest22313-21idleruntime-cro09:00:002
6156991913,3cyclictest22313-21idleruntime-cro09:00:002
6156991912,5cyclictest27923-21expr10:25:172
6156991911,6cyclictest18458-21dpkg11:20:002
615699191,12cyclictest5182-21apt-get09:35:172
615699190,3cyclictest14176-21dpkg11:10:002
615699190,2cyclictest9385-21idleruntime07:15:192
615699190,17cyclictest18050-21munin-run08:49:592
615699190,17cyclictest1087-21cat11:54:592
615699190,16cyclictest9872-21latency_hist11:00:002
615699190,16cyclictest20608-21munin-run11:25:002
615299192,15cyclictest12411-21cat12:20:011
6152991915,2cyclictest14584-21gltestperf09:55:161
6152991914,3cyclictest30638-21runrttasks09:18:011
6152991914,3cyclictest12655-21open_files08:35:201
6152991913,4cyclictest29310-21ls11:45:001
6152991913,4cyclictest22909-21fw_packets09:00:171
6152991912,5cyclictest1781-21cat11:55:181
6152991912,5cyclictest16240-21cat12:25:341
6152991912,5cyclictest15046-21cat12:25:011
6152991911,6cyclictest14313-21threads07:25:251
6152991911,6cyclictest13980-21lsb_release09:55:001
6152991911,5cyclictest4751-21lsb_release08:19:591
615299191,13cyclictest14017-21apt-get08:40:131
615299191,12cyclictest12552-21kworker/u8:107:50:151
615299190,5cyclictest0-21swapper/109:24:561
614999192,13cyclictest23837-21python307:50:010
614999192,12cyclictest31148-21apt-get09:20:130
6149991915,2cyclictest7920-21timerandwakeup07:10:260
6149991913,4cyclictest16912-21ntp_kernel_pll_08:45:230
6149991912,5cyclictest9057-21sed10:55:300
6149991912,5cyclictest23319-21ntp_kernel_err09:00:200
6149991912,5cyclictest21998-21sed11:25:270
6149991912,5cyclictest13037-21ntp_kernel_pll_11:05:220
6149991912,3cyclictest27310-21latency_hist10:25:000
6149991911,6cyclictest15392-21perl12:25:210
6149991911,6cyclictest14899-21awk11:10:230
614999191,13cyclictest2835-21chrt08:15:140
19612190,2sleep00-21swapper/008:10:310
615699182,12cyclictest31800-21apt-get10:35:142
6156991814,2cyclictest1894-21snmpd10:11:552
6156991814,2cyclictest1894-21snmpd09:51:062
6156991814,2cyclictest14417-21/usr/sbin/munin11:10:342
6156991814,2cyclictest11671-21cat08:34:592
6156991813,3cyclictest23798-21unixbench-2d09:00:272
6156991811,5cyclictest6966-21expr10:50:292
615699181,12cyclictest28998-21apt-get09:15:182
615699181,12cyclictest27271-21apt-get11:40:142
615699181,12cyclictest19783-21apt-get07:40:132
615299182,11cyclictest26850-21apt-get09:10:131
6152991814,2cyclictest18437-21sendmail11:20:001
6152991814,2cyclictest15435-21sensors_temp11:10:301
6152991814,2cyclictest11161-21unixbench-2d09:45:261
6152991813,3cyclictest5816-21sh12:05:021
6152991812,4cyclictest8352-21ntp_kernel_pll_08:25:201
6152991812,4cyclictest26035-21ps11:35:221
6152991812,4cyclictest21686-21sh12:40:001
6152991812,4cyclictest16235-21awk07:30:221
6152991811,5cyclictest21711-21load10:10:201
6152991811,5cyclictest19886-21expr07:40:131
6152991811,5cyclictest19039-21ls10:05:151
6152991811,5cyclictest18569-21basename07:35:231
6152991811,5cyclictest17293-21ls08:45:301
615299181,12cyclictest9728-21apt-get08:30:131
615299181,12cyclictest8932-21apt-get07:15:141
615299181,12cyclictest881-21apt-get09:25:151
615299181,12cyclictest16797-40apt-check10:00:111
615299181,12cyclictest16797-40apt-check10:00:111
6152991810,5cyclictest5069-21users10:45:341
6149991814,2cyclictest4166-21idleruntime10:45:220
6149991814,2cyclictest19540-21pmu-power11:20:220
6149991814,2cyclictest1269-21runrttasks09:50:270
6149991811,5cyclictest31608-21sort11:50:170
6149991811,5cyclictest14735-21cut09:55:180
614999181,13cyclictest1278-21apt-get11:55:150
614999181,12cyclictest26652-21apt-get07:55:150
614999181,12cyclictest25112-21apt-get11:35:140
614999181,12cyclictest17608-21apt-get07:35:140
614999181,12cyclictest16797-40apt-check10:00:080
614999181,12cyclictest16797-40apt-check10:00:080
615699179,6cyclictest19157-21df10:05:162
6156991713,2cyclictest1269-21runrttasks11:55:102
6156991712,3cyclictest26838-21expr07:55:152
6156991711,4cyclictest11030-21proc_pri11:00:302
615699171,11cyclictest6792-21apt-get07:10:132
615699171,11cyclictest4065-21snmp_o5powerctl09:30:292
6156991710,5cyclictest8851-21sed12:10:262
6152991713,2cyclictest31965-21swap08:05:231
6152991710,5cyclictest3710-21cat09:30:231
6152991710,5cyclictest26804-21ls07:55:141
6152991710,5cyclictest23899-21sed10:15:181
614999179,6cyclictest24368-21grep10:15:250
614999172,8cyclictest6032-40apt-check12:05:060
614999172,7cyclictest9-21ksoftirqd/007:40:450
6149991714,2cyclictest30931-21/usr/sbin/munin08:05:170
6149991714,2cyclictest29414-21/usr/sbin/munin11:45:320
6149991714,1cyclictest11857-21/usr/sbin/munin08:35:150
6149991713,2cyclictest29008-21sh08:00:200
6149991713,2cyclictest0-21swapper/008:25:240
6149991712,3cyclictest19535-21unixbench_multi08:50:250
6149991711,4cyclictest29745-21ntp_kernel_pll_09:15:230
6149991711,4cyclictest21261-21expr10:10:150
6149991711,3cyclictest2791-21turbostat.cron09:29:590
614999171,13cyclictest20412-21runrttasks10:06:140
614999171,12cyclictest31161-21runrttasks10:31:400
614999171,11cyclictest3051-21apt-get09:30:180
614999171,11cyclictest16553-21apt-get11:15:150
6149991710,5cyclictest19875-21ls12:35:200
6149991710,5cyclictest16531-21cat07:30:260
6149991710,5cyclictest14357-21cat07:25:250
6149991710,5cyclictest14180-21expr08:40:140
6149991710,5cyclictest10146-21entropy09:45:150
615699161,10cyclictest13207-21runrttasks07:25:062
6152991612,2cyclictest8740-21rt-features09:40:261
6152991611,3cyclictest24719-21/usr/sbin/munin09:05:301
615299161,11cyclictest6489-21snmp_o5powerctl08:20:291
614999164,9cyclictest26214-21runrttasks09:06:410
6149991614,1cyclictest0-21swapper/009:11:490
6149991612,2cyclictest8887-21needreboot12:10:260
6149991612,2cyclictest1269-21runrttasks07:17:090
6149991611,3cyclictest27891-21wc11:40:190
614999161,12cyclictest10742-21snmp_o5powerctl08:30:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional