You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-14 - 15:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot4s.osadl.org (updated Sat Sep 14, 2024 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
100782251186,51sleep20-21swapper/207:08:312
98542250202,34sleep30-21swapper/307:06:313
97592250198,37sleep00-21swapper/007:05:380
83612243191,37sleep10-21swapper/107:05:071
289762620,2sleep20-21swapper/209:05:122
191902430,3sleep2241rcuc/212:25:122
300942400,3sleep11044699cyclictest09:05:231
89242370,2sleep00-21swapper/009:30:240
249002360,2sleep30-21swapper/308:55:153
249002360,2sleep30-21swapper/308:55:153
10442993613,7cyclictest21862-21users10:00:290
141242340,2sleep00-21swapper/011:00:140
141242340,2sleep00-21swapper/011:00:140
278622320,2sleep00-21swapper/009:00:230
169862260,3chrt1044699cyclictest11:05:221
1045499251,4cyclictest20424-21python311:15:003
1045499251,22cyclictest3384-21runrttasks09:20:143
10454992512,11cyclictest5239-21kworker/u8:210:50:003
10454992511,13cyclictest31-21ksoftirqd/308:40:003
1045499242,19cyclictest12475-21python307:15:003
10454992314,6cyclictest0-21swapper/308:05:123
1045499230,5cyclictest22967-21python311:20:003
10451992316,3cyclictest20387-21dump-pmu-power11:15:002
10451992314,7cyclictest4456-21users10:35:272
10446992315,6cyclictest14023-21df_inode11:00:141
10446992315,6cyclictest14023-21df_inode11:00:131
1045499226,3cyclictest18127-21awk09:55:003
1045499223,17cyclictest10048-21cut09:35:163
10451992217,3cyclictest18117-21ls09:55:002
10451992215,5cyclictest2807-21cut11:45:212
10451992214,6cyclictest4180-21users08:05:242
10446992214,6cyclictest16986-21ntpq09:50:181
10446992214,6cyclictest10739-21expr07:10:111
10446992213,7cyclictest16335-21cat07:20:181
10446992213,6cyclictest2122-21tune2fs11:45:151
1044699221,15cyclictest20513-21runrttasks10:00:131
1044699220,5cyclictest0-21swapper/109:58:521
10442992215,5cyclictest29172-21awk11:30:250
10442992214,6cyclictest8011-21memory10:45:170
1045499213,15cyclictest24399-21basename08:55:003
10454992116,3cyclictest22019-21latency_hist07:35:013
10454992116,3cyclictest16543-21runrttasks07:20:193
1045499211,5cyclictest17881-21lsb_release11:10:013
10454992113,6cyclictest13706-21expr08:30:133
10451992114,5cyclictest5610-21sed08:10:182
10451992114,5cyclictest32279-21sed09:10:242
10451992114,5cyclictest2053-21sed10:30:242
10451992114,5cyclictest14852-21cat07:15:272
10451992113,6cyclictest25224-21ls10:10:152
10451992113,6cyclictest18747-21ntp_states07:25:192
10451992113,4cyclictest15491-21latency_hist08:35:002
10451992112,7cyclictest17970-21ls08:40:002
1044699212,14cyclictest16076-21apt-get08:35:121
10446992114,5cyclictest8409-21expr12:00:131
10446992114,5cyclictest5720-21basename10:40:151
10446992114,5cyclictest25537-21snmp_o5powerctl07:40:231
10446992113,6cyclictest17990-21expr12:20:241
10442992114,5cyclictest30303-21fw_packets11:35:150
10442992114,5cyclictest27589-21needreboot10:15:160
10442992114,5cyclictest15912-21ls07:20:150
10442992113,6cyclictest23694-21expr11:20:120
10442992113,6cyclictest2226-21awk10:30:250
10442992113,6cyclictest19611-21awk08:40:500
10442992113,6cyclictest15438-21users09:45:280
10442992112,7cyclictest12784-21apt-config07:15:010
1044299211,14cyclictest18354-40apt-check11:10:060
10454992016,2cyclictest32045-21pmu-power10:25:183
10454992016,2cyclictest28551-21munin-run07:50:003
10454992016,2cyclictest28551-21munin-run07:50:003
10454992015,3cyclictest20924-21awk12:30:003
1045499201,3cyclictest13650-40apt-check07:15:043
10454992013,5cyclictest8218-21sed12:00:003
10454992013,5cyclictest22766-21expr10:05:133
10454992013,5cyclictest12775-21df_inode12:10:143
10454992013,5cyclictest12775-21df_inode12:10:133
10454992012,6cyclictest5938-21sed09:25:153
10454992012,6cyclictest4966-21apt08:10:113
10454992012,6cyclictest14508-21sed09:45:163
1045499201,17cyclictest22967-21python311:20:003
1045499200,3cyclictest313-21systemd-journal09:34:553
1045499200,3cyclictest2882-21lsb_release10:34:593
1045499200,3cyclictest2607-21lsb_release08:05:003
1045199202,13cyclictest30092-21snmp_o5powerctl10:20:212
1045199202,13cyclictest20366-21apt-get08:45:132
1045199202,12cyclictest17893-21python311:10:002
10451992015,3cyclictest20128-21munin-plugin-st08:45:012
10451992013,5cyclictest8285-21sed08:15:232
10451992013,5cyclictest14886-21sed11:00:242
10451992013,5cyclictest14886-21sed11:00:242
10451992012,6cyclictest18634-21cut09:55:142
10451992011,6cyclictest27845-21sshd11:30:132
1045199201,14cyclictest8306-21apt-get12:00:132
10446992016,2cyclictest7544-21idleruntime08:15:141
10446992015,3cyclictest5903-21turbostat.cron11:55:001
10446992015,3cyclictest25148-21interrupts08:55:171
10446992015,3cyclictest25148-21interrupts08:55:171
10446992015,3cyclictest15923-21runrttasks09:49:431
10446992014,4cyclictest6824-21runrttasks08:14:541
10446992013,5cyclictest23076-21sed10:05:151
10446992013,5cyclictest15926-21sed12:15:231
10446992012,6cyclictest3163-21perl08:05:151
10446992012,6cyclictest31105-21df07:55:121
10446992012,6cyclictest2038-21users08:00:241
1044699201,14cyclictest768-21grep09:15:121
1044699200,5cyclictest0-21swapper/112:25:221
1044699200,5cyclictest0-21swapper/110:17:021
1044699200,4cyclictest0-21swapper/110:25:391
1044699200,4cyclictest0-21swapper/109:26:541
1044699200,17cyclictest11730-21sshd10:55:141
10442992015,3cyclictest4428-21sed11:50:150
10442992013,5cyclictest5365-21seq10:40:120
10442992013,5cyclictest12239-21cut09:40:150
10442992012,6cyclictest8463-21cat12:00:140
10442992012,6cyclictest3293-21basename08:05:160
10442992012,6cyclictest16426-21basename11:05:160
1044299201,13cyclictest9532-21apt-get10:50:130
1044299201,13cyclictest24778-21apt-get10:10:110
1044299200,10cyclictest508-21turbostat09:15:000
1045499193,15cyclictest24413-21cat07:40:013
1045499192,15cyclictest20474-21cat10:00:013
1045499192,13cyclictest29280-21diskstats10:20:143
10454991915,2cyclictest11549-21threads12:05:253
10454991914,3cyclictest23062-21lsb_release12:35:003
10454991913,3cyclictest22271-21lsb_release08:50:013
10454991912,5cyclictest909-21expr09:15:133
10454991912,5cyclictest32065-21cat07:55:233
10454991912,5cyclictest29038-21sed10:20:013
10454991912,5cyclictest26254-21expr11:25:173
10454991912,5cyclictest25614-21sed10:10:193
10454991912,5cyclictest16650-21awk12:20:003
10454991912,5cyclictest13210-21grep09:40:263
10454991912,5cyclictest12037-21expr10:55:153
10454991912,5cyclictest10216-21awk12:05:003
10454991911,6cyclictest17656-21ntpq12:20:193
10454991911,5cyclictest31655-21cat09:10:163
1045499190,6cyclictest31870-21perf11:39:593
1045499190,3cyclictest19252-1kworker/3:2H10:05:003
1045199199,8cyclictest22966-21python311:20:002
1045199193,13cyclictest23914-21unixbench_multi10:05:252
1045199192,13cyclictest3374-21apt-get09:20:152
1045199192,12cyclictest22501-21apt-get08:50:142
1045199192,12cyclictest21151-21apt-get12:30:122
10451991915,2cyclictest11694-21latency_hist09:40:002
10451991914,3cyclictest760-21/usr/sbin/munin09:15:242
10451991913,4cyclictest26807-21grep11:25:252
10451991912,5cyclictest867-21sshd08:00:142
10451991912,5cyclictest6396-21cut09:25:202
10451991912,5cyclictest26558-21tr09:00:002
10451991912,5cyclictest26558-21tr09:00:002
10451991912,5cyclictest21578-21sed10:00:282
10451991912,5cyclictest11781-21grep08:25:152
10451991912,5cyclictest10581-21grep10:50:242
10451991911,6cyclictest18891-21sed12:25:002
10451991911,5cyclictest27234-21cut10:15:152
1045199191,13cyclictest6152-21apt-get11:55:122
1045199191,13cyclictest10453-21apt-get12:05:132
1045199190,16cyclictest21357-21grep07:30:232
10446991912,5cyclictest9621-21sed09:35:001
10446991912,5cyclictest4153-21expr10:35:261
10446991912,5cyclictest31106-21cat10:25:001
10446991912,5cyclictest25385-21date12:40:001
10446991912,5cyclictest23243-21ntp_kernel_err08:50:191
10446991912,5cyclictest21818-21sed12:30:171
10446991912,5cyclictest1174-21grep10:30:151
10446991912,4cyclictest28474-21runrttasks07:49:361
10446991912,4cyclictest28474-21runrttasks07:49:361
10446991912,4cyclictest13017-21apt-key07:15:021
10446991911,6cyclictest24834-21grep11:20:251
1044699191,13cyclictest25802-21snmp_o5powerctl10:10:221
1044699191,13cyclictest18229-21apt-get08:40:121
1044699191,12cyclictest3382-21apt-get09:20:141
10442991915,2cyclictest23427-21rt-features08:50:210
10442991915,2cyclictest17106-21processes09:50:220
10442991915,2cyclictest11154-21turbostat08:24:590
10442991915,2cyclictest10821-21fschecks_time07:10:130
10442991914,3cyclictest21061-21sendmail_mailqu07:30:190
10442991912,5cyclictest9931-21ls09:35:140
10442991912,5cyclictest6567-21sed11:55:150
10442991912,5cyclictest30005-21cat09:05:230
10442991912,5cyclictest26223-21cut11:25:160
10442991912,5cyclictest23387-21ntpq10:05:180
10442991912,5cyclictest21178-21pmu-power08:45:200
10442991911,6cyclictest25516-21mailstats07:40:210
10442991911,6cyclictest23954-21cat12:35:180
10442991911,6cyclictest2166-21who09:15:260
1044299191,13cyclictest640-21apt-get08:00:130
1044299191,13cyclictest5506-21apt-get09:25:120
1044299191,13cyclictest3374-21apt-get09:20:160
1044299191,13cyclictest16089-21apt-get08:35:140
1044299191,12cyclictest7096-21apt-get08:15:110
10454991813,3cyclictest5054-21swap11:50:263
10454991813,3cyclictest19446-21runrttasks07:26:273
10454991812,4cyclictest28703-21basename09:05:003
10454991811,5cyclictest9467-21diskstats08:20:163
10454991811,5cyclictest3267-21expr10:35:153
1045499181,13cyclictest452-21ntp_states11:40:193
1045499181,13cyclictest12415-21snmp_o5powerctl08:25:223
10454991810,6cyclictest28907-21expr07:50:133
10454991810,6cyclictest13932-21expr11:00:133
10454991810,6cyclictest13932-21expr11:00:123
1045199182,13cyclictest27350-21strings07:45:182
1045199182,13cyclictest27350-21strings07:45:172
10451991814,2cyclictest12439-21idleruntime-cro07:14:592
10451991813,3cyclictest25211-21awk12:40:012
10451991813,2cyclictest14466-21sh12:15:012
10451991813,2cyclictest14466-21sh12:15:002
10451991811,5cyclictest31565-21wc07:55:162
10451991811,5cyclictest30360-21sed11:35:152
10451991811,5cyclictest27072-21cut09:00:142
10451991811,5cyclictest25528-21wc11:25:012
1045199181,13cyclictest6274-21snmp_o5powerctl10:40:222
1045199181,13cyclictest23372-21snmp_o5powerctl07:35:212
1045199181,13cyclictest10279-21snmp_o5powerctl08:20:232
1045199181,12cyclictest29799-21snmp_o5powerctl07:50:222
1045199181,12cyclictest11673-21apt-get10:55:122
10451991810,5cyclictest9321-21sshd09:33:132
1044699189,6cyclictest22948-21cat07:35:161
10446991816,1cyclictest0-21swapper/109:35:261
10446991814,2cyclictest20124-21/usr/sbin/munin07:30:221
10446991811,5cyclictest7940-21cut10:45:161
10446991811,5cyclictest30077-21cat11:35:131
10446991811,5cyclictest29989-21cat07:50:221
10446991811,5cyclictest12997-21expr12:10:141
10446991811,5cyclictest12997-21expr12:10:141
10446991811,5cyclictest12114-21expr09:40:131
1044699181,13cyclictest20400-21apt-get11:15:061
1044699181,12cyclictest27840-21apt-get11:30:131
1044699181,12cyclictest18480-40apt-check11:10:051
1044699181,12cyclictest13542-21apt-get08:30:131
1044699181,12cyclictest11402-21apt-get08:25:121
10446991810,6cyclictest10991-21cut12:05:181
1044299182,11cyclictest28782-21apt-get07:50:110
1044299182,11cyclictest12579-21apt-get12:10:120
1044299182,11cyclictest12579-21apt-get12:10:120
10442991815,2cyclictest11673-21apt-get10:55:120
10442991814,2cyclictest1625-21idleruntime-cro11:45:000
10442991813,3cyclictest20732-21apt-config11:15:020
10442991813,3cyclictest15295-21cat12:15:150
10442991812,4cyclictest29757-21needreboot10:20:180
10442991811,5cyclictest31558-21expr10:25:150
1044299181,13cyclictest15462-21python308:35:010
1045499179,6cyclictest18563-21fw_packets08:40:153
1045499172,11cyclictest24309-21snmp_o5powerctl12:35:233
10454991712,3cyclictest30818-21sh09:10:003
10454991711,4cyclictest28297-21basename11:30:173
1045499171,12cyclictest10563-21snmp_o5powerctl10:50:243
10454991710,5cyclictest7718-21wc08:15:163
10454991710,4cyclictest1934-21cpuspeed_turbos11:45:133
1045499170,16cyclictest668-17auditd07:45:003
1045199179,6cyclictest25294-21ntp_states07:40:192
1045199179,5cyclictest31291-21tr10:25:122
10451991713,2cyclictest8378-21mailstats10:45:202
10451991712,3cyclictest15103-21cut09:45:252
10451991711,4cyclictest16277-21grep07:20:172
10451991711,3cyclictest5032-21runrttasks11:50:242
1045199171,11cyclictest32127-21apt-get11:40:122
1045199170,13cyclictest14718-21apt-get12:15:112
10446991715,1cyclictest0-21swapper/108:46:521
10446991713,2cyclictest6572-21hwlatdetect11:55:151
10446991713,2cyclictest10519-21sendmail_mailtr10:50:221
1044699171,12cyclictest710-21snmp_o5powerctl11:40:221
1044699171,12cyclictest27813-21snmp_o5powerctl09:00:231
1044699171,12cyclictest17973-21apt-get07:25:131
10446991710,5cyclictest31228-21ls09:10:121
1044299179,6cyclictest1879-21dpkg11:45:110
10442991712,3cyclictest22851-21cat07:35:160
10442991712,3cyclictest12702-21unixbench_singl08:25:250
10442991711,4cyclictest21577-21cat12:30:140
1044299171,11cyclictest24632-21/usr/sbin/munin08:55:270
1044299171,11cyclictest24632-21/usr/sbin/munin08:55:270
1044299171,11cyclictest19455-21runrttasks07:26:300
1044299171,11cyclictest18354-21apt-get09:55:130
1044299171,10cyclictest3111-21grep10:35:120
10442991710,5cyclictest27925-21awk07:45:240
10442991710,5cyclictest27925-21awk07:45:230
10454991610,3cyclictest5614-21gltestperf10:40:163
1045199161,11cyclictest11944-21apt-get09:40:132
10446991612,2cyclictest25698-21/usr/sbin/munin11:25:251
1044699161,11cyclictest9271-21watchprog-snmpd08:20:141
1044299169,5cyclictest6060-21cat08:10:240
1044299169,5cyclictest17855-21grep12:20:210
10442991612,2cyclictest32338-21runrttasks07:55:590
10442991611,3cyclictest19897-21processes12:25:190
1044299161,12cyclictest11157-21cut12:05:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional