You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-22 - 03:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot4s.osadl.org (updated Sun Jul 21, 2024 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32701993711,9cyclictest20318-21lsb_release11:25:000
32701993710,9cyclictest28122-21sed10:10:000
32701993710,11cyclictest5955-21cstates07:25:100
32701993610,8cyclictest15097-21cat09:30:000
3270199359,8cyclictest2628-21sed12:10:000
32701993530,3cyclictest32711-21wc07:10:010
3270199349,9cyclictest5427-21unixbench-2d07:20:280
3270199349,9cyclictest19990-21cat09:45:020
3270199347,17cyclictest0-21swapper/010:40:190
32701993411,8cyclictest26388-21df08:25:120
3270199339,6cyclictest13484-21cat09:25:000
3270199329,7cyclictest30825-21apt-get08:40:020
3270199319,8cyclictest12301-21cut12:40:000
3270199319,6cyclictest2252-21cat10:30:000
3270199292,24cyclictest12515-21tr11:00:180
32701992918,5cyclictest28208-21apt-get10:10:090
32701992917,6cyclictest30230-21apt-get11:55:130
32701992912,14cyclictest5908-21tail09:00:140
3270199287,8cyclictest0-21swapper/011:50:010
3270199287,8cyclictest0-21swapper/011:50:010
3270199287,8cyclictest0-21swapper/007:55:010
32701992816,6cyclictest2307-21apt-get10:30:110
32701992812,13cyclictest30807-21cron08:39:590
32701992719,3cyclictest7436-40apt-check10:45:090
32701992718,4cyclictest7839-21apt-get07:30:120
32701992718,3cyclictest17277-21apt-get11:15:110
32701992713,11cyclictest11723-21cat07:40:170
32701992617,3cyclictest17658-21apt-get08:00:120
32701992616,3cyclictest20882-21apt-get08:10:100
32701992615,7cyclictest10994-21sort07:40:000
32701992614,9cyclictest20324-21/usr/sbin/munin09:45:130
32701992610,13cyclictest11657-21grep09:15:250
32701992610,13cyclictest11657-21grep09:15:240
32712992516,5cyclictest10165-21idleruntime-cro09:15:002
32701992519,2cyclictest26788-21lsb_release11:45:000
32701992516,3cyclictest31865-21apt-get12:00:090
32701992513,9cyclictest7553-21sed09:05:140
32701992513,9cyclictest24141-21sed11:35:210
32701992512,10cyclictest10780-21cat12:35:000
32701992511,11cyclictest24088-21grep09:55:230
32701992417,5cyclictest2813-21cut08:50:160
32701992415,7cyclictest24666-21cpuspeed_turbos08:20:120
32701992415,7cyclictest18482-21cat09:40:010
32701992414,7cyclictest3299-21/usr/sbin/munin07:15:250
32701992414,7cyclictest30561-21/usr/sbin/munin10:15:200
32701992414,7cyclictest19953-21needreboot08:05:200
32701992413,8cyclictest31646-21df10:20:150
32701992413,8cyclictest25249-21fschecks_time10:00:130
32701992412,9cyclictest4917-21/usr/sbin/munin10:35:220
32701992412,10cyclictest17084-21sed07:55:230
32701992314,7cyclictest9103-21/usr/sbin/munin09:10:150
32701992314,6cyclictest14046-21/usr/sbin/munin11:05:250
32701992314,6cyclictest10608-21latency_hist10:55:010
32701992313,8cyclictest21938-21forks09:50:150
32701992313,7cyclictest23699-21ntp_offset08:15:190
32701992313,7cyclictest15996-21expr11:10:160
32701992313,4cyclictest15260-21apt-get09:30:100
32701992312,9cyclictest14194-21munin-run07:50:000
32701992312,8cyclictest29967-21pidof11:52:330
32701992312,7cyclictest7980-21gltestperf12:25:190
3271699224,16cyclictest31-21ksoftirqd/312:05:013
32716992215,5cyclictest302-21systemd-journal12:35:003
32716992215,5cyclictest2079-21latency_hist10:30:003
32716992214,6cyclictest27214-21sed10:05:193
3271699221,18cyclictest6975-21python310:45:013
32712992215,4cyclictest10563-21dump-pmu-power10:55:002
3271299221,14cyclictest13838-21lsb_release11:05:012
32704992215,5cyclictest20148-21awk11:20:271
32704992213,6cyclictest2595-21tune2fs08:50:151
3270199222,14cyclictest5366-21cat09:00:000
32701992216,2cyclictest21927-21lsb_release11:30:000
32701992214,5cyclictest4375-21/usr/sbin/munin12:15:110
32701992213,7cyclictest28098-21tune2fs08:30:150
32701992212,8cyclictest12075-21head10:55:260
32701992212,7cyclictest2073-21sh08:50:000
32716992116,3cyclictest22544-40apt-check08:13:363
32716992115,4cyclictest30807-21cron08:40:003
32716992114,5cyclictest17169-21perl09:35:143
3271699211,17cyclictest12215-21lsb_release12:40:013
3271299217,8cyclictest26010-21latency_hist08:25:012
3271299212,13cyclictest4205-21apt-get07:20:132
32712992116,3cyclictest18660-21sendmail11:19:592
32712992115,4cyclictest22544-40apt-check08:13:302
3271299211,4cyclictest31229-21munin-run10:20:002
32712992113,5cyclictest3850-21awk07:15:332
32712992112,6cyclictest3294-21snmp_o5powerctl08:50:262
32712992112,6cyclictest16663-21cat07:55:182
3271299211,14cyclictest7567-21apt-get10:45:152
32712992110,7cyclictest21909-21cron11:29:592
32704992116,3cyclictest2114-21timerwakeupswit07:15:011
32704992114,5cyclictest23541-21diskstats09:55:181
32704992113,6cyclictest14117-21users07:45:281
32704992112,6cyclictest25729-21ntpq10:00:181
32701992114,5cyclictest904-21sed07:10:190
32701992113,6cyclictest6728-21ntp_states12:20:210
32701992112,7cyclictest2758-21cpu12:10:110
32701992112,6cyclictest22492-21gltestperf11:30:150
3271699202,16cyclictest15451-21awk11:10:003
32716992016,2cyclictest6415-21processes09:00:213
3271699201,3cyclictest21917-21munin-run11:29:593
3271699201,3cyclictest19059-21lsb_release08:05:013
32716992013,5cyclictest469-21ls07:10:123
32716992013,5cyclictest19935-21cut09:45:003
32716992012,6cyclictest6851-21sed07:25:233
32716992012,6cyclictest3203-21cat12:10:153
32716992012,6cyclictest26686-21cat08:25:153
3271699201,17cyclictest24744-21munin-run10:00:003
3271299202,12cyclictest22953-21apt-get08:15:112
32712992014,3cyclictest12195-21sendmail11:00:012
32712992013,5cyclictest7720-21sed09:05:162
32712992013,5cyclictest6923-21wc07:25:242
32712992013,5cyclictest469-21latency_hist08:45:002
32712992013,5cyclictest28053-21expr08:30:152
32712992013,5cyclictest26535-21expr08:25:142
32712992013,5cyclictest23702-21expr09:55:192
32712992013,5cyclictest17412-21wc09:35:162
32712992012,6cyclictest5086-21ntp_states12:15:172
32712992011,6cyclictest25789-21cut10:00:192
3271299201,14cyclictest19909-21snmp_o5powerctl11:20:232
3271299201,13cyclictest2717-21apt-get12:10:092
3271299201,13cyclictest23755-21apt-get11:35:182
32704992015,3cyclictest7491-21gltestperf09:05:141
32704992015,3cyclictest4396-21if_err_enp2s008:55:151
32704992013,5cyclictest27295-21perl11:45:151
32704992013,5cyclictest27295-21perl11:45:141
32704992013,5cyclictest17327-21if_enp2s009:35:151
32704992012,6cyclictest9451-21sed07:34:591
32704992012,6cyclictest32454-21snmp_o5powerctl10:20:261
32704992012,6cyclictest31658-21lsb_release12:00:011
32704992012,6cyclictest17432-21expr11:15:121
32704992012,5cyclictest6288-21fw_packets12:20:161
3271699199,6cyclictest73550irq/132-enp2s008:40:113
32716991915,2cyclictest26778-21perf11:45:003
32716991915,2cyclictest23522-21dump-pmu-power11:34:593
32716991915,2cyclictest18411-21timerwakeupswit11:15:253
3271699191,3cyclictest8567-21munin-run09:10:003
32716991913,3cyclictest31460-21apt-get10:20:133
32716991913,3cyclictest0-21swapper/310:45:063
3271699191,2cyclictest28405-21lsb_release11:50:013
3271699191,2cyclictest28405-21lsb_release11:50:003
32716991912,5cyclictest7726-21expr12:25:173
32716991912,5cyclictest751-21seq08:45:123
32716991912,5cyclictest5523-21cut09:00:003
32716991912,5cyclictest4920-21snmp_o5powerctl10:35:233
32716991912,5cyclictest4911-21cat12:15:153
32716991912,5cyclictest25844-21grep08:20:253
32716991912,5cyclictest24089-21expr11:35:213
32716991912,5cyclictest21888-21expr09:50:143
32716991912,5cyclictest15728-21ls11:10:133
32716991912,5cyclictest14265-21cat09:25:173
32716991912,5cyclictest10567-21expr09:15:113
32716991912,5cyclictest10567-21expr09:15:113
32716991911,5cyclictest7024-21cat12:20:253
32716991911,5cyclictest2469-21expr10:30:123
3271699191,13cyclictest28616-21apt-get11:50:103
3271699191,13cyclictest16022-21apt-get07:55:113
3271699191,13cyclictest14410-21apt-get07:50:113
3271699191,12cyclictest2567-21apt-get07:15:153
3271699190,6cyclictest3704-21awk08:54:593
3271699190,17cyclictest12715-21tune2fs11:00:213
32712991915,2cyclictest12207-21turbostat12:40:002
32712991913,4cyclictest21944-21awk09:50:162
32712991912,5cyclictest4389-21awk10:35:142
32712991912,5cyclictest1800-21sed08:45:262
32712991912,4cyclictest32183-21cut12:00:142
32712991911,6cyclictest6656-21sed10:40:262
32712991911,6cyclictest4908-21snmp_o5powerctl08:55:322
32712991911,6cyclictest25-21ksoftirqd/212:20:242
32712991911,6cyclictest10338-21expr12:30:292
32712991911,5cyclictest29032-21awk10:10:202
3271299191,13cyclictest27000-21apt-get11:45:122
3271299191,13cyclictest27000-21apt-get11:45:122
3271299191,12cyclictest14049-21apt-get11:05:122
32712991910,6cyclictest20802-21grep09:45:172
32704991915,2cyclictest8687-21timerandwakeup12:25:301
32704991915,2cyclictest29151-21vmstat08:30:271
32704991915,2cyclictest21485-21missed_timers08:10:181
32704991915,2cyclictest14573-21sendmail_mailst09:25:221
32704991914,3cyclictest30383-21latency10:15:181
32704991913,4cyclictest6255-21ls07:25:151
32704991913,4cyclictest15151-21ntpq07:50:191
32704991912,5cyclictest31772-21fw_packets08:40:151
32704991912,5cyclictest26729-21expr10:05:141
32704991912,5cyclictest15086-21sed11:05:261
32704991912,5cyclictest12889-21proc_pri09:20:211
32704991911,6cyclictest2018-21vmstat10:25:281
32704991910,6cyclictest15844-21expr11:10:121
32716991815,2cyclictest7571-21apt-config07:30:013
32716991813,3cyclictest28205-21/usr/sbin/munin10:10:113
32716991812,4cyclictest12871-21grep09:20:213
32716991811,5cyclictest9915-21ntpq10:50:163
32716991811,5cyclictest8783-21dpkg09:10:093
32716991811,5cyclictest15877-21sed09:30:193
32716991811,5cyclictest13906-21timerandwakeup07:45:253
32716991811,5cyclictest11334-21expr07:40:113
3271699181,14cyclictest1436-21grep12:05:163
3271699181,12cyclictest29821-21apt-get10:15:123
3271699181,12cyclictest20121-21apt-get09:45:113
3271699180,2cyclictest19266-21date08:05:013
3271699180,16cyclictest5594-21perf07:25:003
3271699180,16cyclictest31629-21munin-run12:00:003
3271299189,7cyclictest8831-21snmp_o5powerctl07:30:232
3271299185,10cyclictest26787-21basename11:45:002
3271299183,9cyclictest649-21apt-get10:25:112
3271299182,12cyclictest32715-21latency_hist07:10:012
32712991812,4cyclictest39-21kauditd07:15:002
32712991811,5cyclictest29258-21cat11:50:162
32712991811,5cyclictest2811-21irqstats10:30:152
32712991811,5cyclictest27541-21basename10:05:232
32712991811,5cyclictest18615-21which08:00:222
32712991811,5cyclictest15713-21awk11:10:122
32712991811,5cyclictest12769-21ntpq09:20:202
3271299181,12cyclictest9549-21apt-get07:35:132
3271299181,12cyclictest19270-21apt-get08:05:112
3271299181,12cyclictest12797-21apt-get07:45:112
32712991810,6cyclictest12351-21grep07:40:252
32712991810,5cyclictest15770-21cut09:30:172
3271299180,4cyclictest15808-21dpkg07:55:002
3270499185,11cyclictest10582-21dpkg10:55:011
32704991815,2cyclictest875-21fschecks_count08:45:151
32704991814,2cyclictest8845-21cpuspeed_turbos09:10:121
32704991814,2cyclictest26313-21sendmail_mailst11:40:211
32704991814,2cyclictest23881-21sendmail_mailst08:15:221
32704991813,3cyclictest2492-21latency_hist12:10:001
32704991813,3cyclictest12194-21sh11:00:011
32704991813,3cyclictest10812-21cpu12:35:121
32704991812,3cyclictest25279-21cut08:20:181
32704991811,5cyclictest5479-21cat12:15:231
32704991811,5cyclictest4781-21sed07:20:181
32704991811,5cyclictest3861-21cut10:35:011
32704991811,5cyclictest28348-21expr10:10:111
32704991811,5cyclictest22867-21ntp_kernel_err11:30:181
32704991811,5cyclictest22139-21grep09:50:161
32704991811,5cyclictest21077-21sed11:25:181
32704991811,5cyclictest20942-21grep09:45:201
32704991811,5cyclictest18622-21expr09:40:101
32704991811,5cyclictest17757-21expr08:00:121
32704991811,5cyclictest17243-21unixbench_multi07:55:251
32704991811,5cyclictest11347-21expr07:40:121
32704991810,6cyclictest32041-21expr12:00:121
32704991810,6cyclictest16209-21grep09:30:211
32704991810,6cyclictest12428-21date11:00:011
3270499180,16cyclictest6281-21/usr/sbin/munin10:40:181
3271699178,6cyclictest27818-21apt-get08:30:123
32716991714,1cyclictest9541-21package-data-do07:35:013
32716991711,4cyclictest19554-21missed_timers11:20:193
32716991711,4cyclictest11817-21expr10:55:243
32716991710,5cyclictest24370-21lsb_release08:20:003
32712991711,4cyclictest30231-21dpkg11:55:112
3271299171,13cyclictest25-21ksoftirqd/209:00:092
3271299171,12cyclictest19501-21snmp_o5powerctl09:40:212
3271299171,11cyclictest30872-21python308:39:592
3271299171,11cyclictest10396-21apt-get09:15:112
3271299171,11cyclictest10396-21apt-get09:15:112
3271299170,6cyclictest1442-21grep12:05:162
3271299170,10cyclictest15042-21perf09:30:012
32704991713,2cyclictest9692-21interrupts12:30:191
32704991713,2cyclictest26953-21ntp_offset08:25:171
32704991712,3cyclictest4247-21gltestperf10:35:141
32704991711,4cyclictest5698-21expr09:00:121
32704991711,4cyclictest29571-21expr08:35:171
3270499171,12cyclictest29608-21snmp_o5powerctl11:50:231
3270499171,11cyclictest7570-21apt-get10:45:121
32704991710,5cyclictest19723-21if_enp2s008:05:161
32704991710,5cyclictest10579-21df09:15:121
32704991710,5cyclictest10579-21df09:15:111
3270499170,15cyclictest1349-21snmp_o5powerctl07:10:251
32716991610,4cyclictest26132-21sed10:00:253
32712991611,2cyclictest7789-21forks12:25:202
3271299161,11cyclictest27376-21snmpd10:20:122
32704991613,2cyclictest9549-21apt-get07:35:111
32704991613,2cyclictest23966-21fschecks_count11:35:201
3270499161,12cyclictest3714-21snmp_o5powerctl12:10:221
32712991513,1cyclictest0-21swapper/211:30:102
327169991,6cyclictest32724-21cat07:10:013
327049991,6cyclictest32706-21latency_hist07:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional