You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 15:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot4s.osadl.org (updated Sat Apr 27, 2024 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7059992612,8cyclictest19-21ksoftirqd/119:45:061
706899257,16cyclictest797-21cron22:05:003
7059992518,4cyclictest10311-21sed20:55:161
7068992416,6cyclictest8908-21df_inode19:15:013
705999242,14cyclictest14677-21apt-get19:30:131
706899231,19cyclictest15448-21munin-run22:50:013
706899221,4cyclictest19152-21python319:45:003
706899221,18cyclictest30690-21ntp_states21:55:203
706899220,20cyclictest31776-21latency_hist23:40:013
706899220,20cyclictest31776-21latency_hist23:40:003
706399222,13cyclictest5940-21apt-get22:20:112
7059992215,5cyclictest17731-21awk21:15:421
705999221,14cyclictest31331-21latency_hist22:00:011
705999221,14cyclictest27544-21apt-get21:50:121
705599222,14cyclictest24795-21kworker/u8:119:30:060
7055992215,5cyclictest32701-21ls20:25:140
7055992215,5cyclictest15220-21sed21:10:180
7055992214,6cyclictest8495-21memory00:05:160
7055992214,6cyclictest23509-21memory21:35:210
7055992214,6cyclictest10112-21memory00:10:180
7055992213,6cyclictest31366-21systemctl22:00:010
706899212,17cyclictest7423-21cat22:25:003
706899211,9cyclictest1812-21apt-get20:30:093
7068992116,3cyclictest7608-21cpu22:25:103
706899211,3cyclictest23567-21lsb_release23:15:013
706899211,3cyclictest21020-21munin-run19:50:003
706899211,17cyclictest14006-21python319:30:013
706899211,14cyclictest17189-21snmpd23:30:113
706899210,5cyclictest5727-21tr22:20:013
706899210,19cyclictest9269-21cron00:09:593
706899210,18cyclictest20007-21tr21:25:153
706399212,13cyclictest7130-21apt-get19:10:112
706399212,13cyclictest28034-40apt-check21:50:082
7063992115,4cyclictest29098-21cron20:15:002
7063992114,4cyclictest27551-21latency_hist21:50:012
7063992114,4cyclictest27551-21latency_hist21:50:002
7063992113,6cyclictest381-21snmp_o5powerctl22:00:252
7063992111,5cyclictest11390-21cut20:59:592
7063992110,7cyclictest31614-21sed22:00:002
705999219,8cyclictest30016-21cron23:35:001
705999219,8cyclictest19066-21cat00:40:001
705999213,14cyclictest27513-21awk20:10:001
705999212,13cyclictest17287-21apt-get22:55:131
705999212,13cyclictest17287-21apt-get22:55:121
7059992111,7cyclictest31738-21sed23:40:001
7059992111,7cyclictest31738-21sed23:40:001
705999211,13cyclictest31641-21apt-get23:40:121
7055992116,3cyclictest6782-21kernelversion00:00:170
7055992114,5cyclictest9652-21cat22:30:160
7055992114,5cyclictest7319-21df_inode19:10:140
7055992114,5cyclictest13848-21needreboot21:05:220
7055992114,5cyclictest11075-21timerwakeupswit20:55:290
7055992113,6cyclictest11727-21memory00:15:160
7055992112,6cyclictest26837-21ntp_kernel_pll_20:05:180
7068992016,2cyclictest1269-21threads20:25:273
7068992013,5cyclictest8214-21expr19:10:273
7068992013,5cyclictest4235-21grep20:35:233
7068992013,5cyclictest31815-21proc_pri20:20:213
7068992013,5cyclictest21969-21ntpq19:50:193
7068992013,5cyclictest14959-21expr21:10:133
7068992013,5cyclictest14506-21expr00:25:133
7068992013,5cyclictest13501-21ntpq00:20:253
7068992012,6cyclictest20775-21tune2fs23:05:143
7068992012,6cyclictest16571-21cat19:35:153
706899200,3cyclictest30759-21lsb_release20:20:003
706899200,17cyclictest18692-21uname23:00:003
706899200,17cyclictest18692-21uname23:00:003
706399202,13cyclictest13185-21apt-get21:05:142
706399202,11cyclictest6692-21apt-get20:45:132
7063992015,3cyclictest25563-21timerandwakeup20:00:252
7063992013,5cyclictest31495-21cut20:20:162
7063992012,6cyclictest6214-21sed00:00:012
7063992012,6cyclictest20701-21expr23:05:132
706399201,14cyclictest14051-21apt-get22:45:112
706399201,13cyclictest9512-21apt-get00:10:102
706399201,13cyclictest5077-21apt-get20:40:122
706399201,13cyclictest32165-21apt-get23:40:172
706399201,13cyclictest31687-21python323:40:002
706399201,13cyclictest31687-21python323:40:002
706399201,13cyclictest12587-21apt-get19:25:122
706399201,13cyclictest12434-21apt-get22:40:122
706399200,16cyclictest19607-40apt-check19:45:072
705999208,8cyclictest3201-21cron20:35:001
705999202,13cyclictest9755-21latency_hist20:54:591
705999202,13cyclictest10798-21apt-get22:35:141
7059992016,2cyclictest8478-21sendmail_mailqu22:25:211
7059992016,2cyclictest31918-21cut22:00:131
7059992015,3cyclictest5644-21switchtime23:55:251
7059992013,5cyclictest7006-21sed22:20:321
7059992013,4cyclictest7675-21turbostat00:05:001
7059992011,6cyclictest4819-21cron20:40:001
705999201,14cyclictest3015-21apt-get23:50:111
705999201,13cyclictest32590-21apt-get20:25:131
705999201,13cyclictest14804-21apt-get21:10:141
705599202,13cyclictest30975-21apt-get20:20:110
7055992014,4cyclictest17772-21cat22:55:200
7055992014,4cyclictest17772-21cat22:55:190
7055992013,5cyclictest19883-21entropy21:25:140
7055992013,5cyclictest16594-21which22:50:250
7055992012,6cyclictest7064-21cat19:10:000
7055992012,6cyclictest5989-21sshd23:57:480
7055992012,6cyclictest558-21awk23:40:270
7055992012,6cyclictest20809-21cat19:45:240
705599201,13cyclictest22158-21apt-get23:10:130
706899192,15cyclictest10898-21latency_hist00:15:013
706899191,8cyclictest27095-21snmp_o5powerctl20:05:223
7068991915,2cyclictest11572-21cpu21:00:113
7068991915,2cyclictest11356-21sendmail21:00:003
7068991914,3cyclictest20060-21timerwakeupswit23:00:283
706899191,2cyclictest24272-21lsb_release20:00:003
7068991912,5cyclictest7257-21grep00:00:263
7068991912,5cyclictest2521-21cat23:45:253
7068991912,5cyclictest12427-21date22:40:003
7068991911,6cyclictest27303-21cut23:25:163
7068991911,6cyclictest18297-21tune2fs21:20:163
7068991911,5cyclictest11920-21grep19:20:213
706899191,14cyclictest26163-21apt-get21:45:143
706899191,14cyclictest26163-21apt-get21:45:143
706899191,13cyclictest4631-21apt-get23:55:113
706899190,2cyclictest10587-21munin-run22:34:593
706399193,9cyclictest28641-21apt-get23:30:102
706399193,9cyclictest17620-21apt-get00:35:112
706399193,10cyclictest22909-21apt-get21:35:132
706399192,9cyclictest8725-21munin-node23:25:132
7063991912,5cyclictest9011-21sed00:05:252
7063991912,5cyclictest19826-21expr21:25:112
7063991912,5cyclictest16059-21sed19:35:002
7063991912,4cyclictest12962-21munin-plugin-st21:05:002
7063991912,3cyclictest24252-21latency_hist20:00:002
7063991911,6cyclictest3878-21basename20:35:182
7063991911,5cyclictest2382-21grep23:45:212
706399191,15cyclictest22844-21head23:10:212
706399191,13cyclictest4302-21apt-get22:15:102
706399191,13cyclictest29336-21apt-get20:15:102
706399191,13cyclictest23778-21apt-get23:15:112
706399191,13cyclictest16418-21apt-get21:15:112
706399191,13cyclictest11126-21apt-get00:15:102
706399191,13cyclictest10971-21apt-get19:20:102
706399191,12cyclictest2685-21apt-get22:10:122
706399191,12cyclictest21296-21apt-get21:30:102
706399191,12cyclictest19153-21python319:45:002
7059991913,3cyclictest15763-21sh00:30:001
7059991912,5cyclictest4204-21cat22:15:001
7059991912,5cyclictest26410-21snmp_o5powerctl23:20:301
7059991912,5cyclictest23517-21uname19:55:201
7059991912,5cyclictest23252-21sed23:10:291
7059991912,5cyclictest12798-21expr22:40:151
7059991912,5cyclictest11100-21ls19:20:121
705999191,13cyclictest7894-21apt-get00:05:101
705999191,13cyclictest24491-21apt-get20:00:111
705999191,13cyclictest18064-21apt-get21:20:131
705999191,13cyclictest1390-21apt-get23:45:111
7055991915,2cyclictest24814-21switchtime23:15:280
7055991915,2cyclictest19261-21unixbench-2d21:20:290
7055991914,3cyclictest8801-21cat20:50:160
7055991914,3cyclictest3016-21unixbench-2d20:30:250
7055991914,3cyclictest18212-21if_err_enp2s019:40:180
7055991912,5cyclictest9706-21expr19:15:160
7055991912,5cyclictest4803-21sshd20:38:510
7055991912,5cyclictest30905-21sed23:35:180
7055991912,5cyclictest30905-21sed23:35:180
7055991912,5cyclictest28918-21grep20:10:250
7055991912,5cyclictest27216-21df_inode23:25:140
7055991912,5cyclictest26825-21needreboot21:45:210
7055991912,5cyclictest26825-21needreboot21:45:210
7055991912,5cyclictest26363-21grep23:20:270
7055991912,5cyclictest16100-21cat00:30:130
7055991912,5cyclictest11116-21expr19:20:110
7055991911,6cyclictest3965-21grep23:50:230
7055991911,5cyclictest21263-21nfs4_client23:05:190
7068991813,3cyclictest13364-21sendmail_mailst22:40:233
7068991813,3cyclictest1251-21sh22:05:163
7068991812,4cyclictest9154-21awk20:50:213
7068991812,4cyclictest27909-21dpkg21:50:013
7068991812,4cyclictest22342-21ls21:30:243
7068991811,5cyclictest24784-21grep21:40:163
7068991811,5cyclictest24726-21grep23:15:243
7068991811,5cyclictest16977-21grep00:30:233
706899181,13cyclictest25412-21apt-get23:20:133
7068991810,4cyclictest25894-21munin-run20:05:003
706399183,9cyclictest16089-21apt-get19:35:092
706399183,8cyclictest18926-21apt-get23:00:132
706399182,11cyclictest26106-21apt-get20:05:102
7063991814,2cyclictest3012-21/usr/sbin/munin23:50:202
7063991813,3cyclictest9178-21/usr/sbin/munin22:30:292
7063991811,5cyclictest15711-21rt-features21:10:242
706399181,13cyclictest25544-21snmp_o5powerctl21:40:282
706399181,13cyclictest15377-21snmp_o5powerctl00:25:272
706399181,13cyclictest11795-21snmp_o5powerctl22:35:302
706399181,12cyclictest8315-21apt-get20:50:112
706399181,12cyclictest21234-21apt-get19:50:102
7063991810,6cyclictest686-21cut20:25:182
7063991810,5cyclictest1053-21apt-get22:05:132
705999183,12cyclictest14661-21wc22:45:181
7059991814,2cyclictest19-21ksoftirqd/121:30:191
7059991813,3cyclictest30683-21sh20:17:001
7059991812,4cyclictest24390-21grep21:40:001
7059991811,5cyclictest8650-21sed19:15:001
7059991811,5cyclictest24963-21cat23:15:281
7059991811,5cyclictest20402-21awk21:25:181
7059991811,5cyclictest16172-21expr00:30:131
705999181,12cyclictest27721-21apt-get20:10:101
705999181,12cyclictest12768-21apt-get00:20:141
705999181,12cyclictest1-21systemd19:30:011
7059991810,6cyclictest18949-21sed19:40:321
7055991814,2cyclictest4301-21apt22:15:080
7055991814,2cyclictest3449-21sh22:10:210
7055991814,2cyclictest24926-21idleruntime20:00:150
7055991812,5cyclictest14537-21cat22:45:160
7055991812,4cyclictest8262-21cut20:49:590
7055991812,3cyclictest13806-21unixbench_multi19:25:270
7055991812,3cyclictest11933-21gltestperf21:00:140
7055991811,5cyclictest32245-21head22:00:160
7055991811,5cyclictest29363-21ntp_kernel_err23:30:200
7055991811,5cyclictest1716-21cat22:05:190
7055991811,5cyclictest15291-21rt-features00:25:240
705599181,12cyclictest28054-21apt-get21:50:130
7055991810,6cyclictest12957-21df_inode00:20:170
7055991810,5cyclictest17713-21latency_hist19:40:000
7068991713,2cyclictest4177-21timerwakeupswit23:50:283
7068991713,2cyclictest29115-21latency_hist20:15:003
7068991713,2cyclictest17836-21dpkg21:20:013
7068991711,4cyclictest15462-21sensors19:30:223
7068991711,4cyclictest13332-21ls21:05:153
706899171,12cyclictest3680-21snmp_o5powerctl22:10:263
7068991710,5cyclictest23038-21ls21:35:133
706899170,15cyclictest19009-21turbostat00:39:593
706899170,15cyclictest17056-21latency_hist22:55:003
706899170,15cyclictest0-21swapper/323:40:053
706899170,15cyclictest0-21swapper/320:40:103
706399174,10cyclictest19473-21lsb_release21:25:012
706399173,8cyclictest9331-21apt-get19:15:092
7063991714,2cyclictest17748-21if_enp2s022:55:192
7063991714,2cyclictest17748-21if_enp2s022:55:192
7063991711,4cyclictest16306-21cut00:30:142
7063991711,4cyclictest13846-21grep00:20:332
706399171,12cyclictest7275-21snmp_o5powerctl00:00:292
7063991710,5cyclictest25555-21cat23:20:142
705999179,5cyclictest771-21dbus-daemon22:20:001
705999179,5cyclictest24961-21sed21:40:181
7059991713,2cyclictest18923-21/usr/sbin/munin23:00:301
7059991711,4cyclictest27498-21cat23:25:171
7059991711,4cyclictest2646-21grep22:10:011
705999171,12cyclictest9181-21apt-get22:30:141
705999171,12cyclictest12566-21snmp_o5powerctl21:00:251
705999171,11cyclictest27579-21python321:49:591
705999171,11cyclictest27579-21python321:49:591
705999171,11cyclictest20544-21apt-get23:05:121
7059991710,5cyclictest9294-21sed19:15:031
705999170,15cyclictest0-21swapper/119:38:521
705599179,6cyclictest23998-21ls19:55:280
7055991713,2cyclictest30261-21sendmail_mailst20:15:220
7055991713,2cyclictest12758-21gltestperf22:40:150
7055991712,3cyclictest5747-21tr20:40:180
7055991712,3cyclictest11310-21kernelversion22:35:190
7055991711,4cyclictest25740-21unixbench-2d21:40:290
7055991710,5cyclictest21782-21cat21:30:160
705599170,3cyclictest21426-21df_inode19:50:120
705599170,15cyclictest0-21swapper/021:15:020
706899165,7cyclictest17189-21snmpd00:15:103
7068991612,2cyclictest8111-21idleruntime-cro20:50:003
706399161,11cyclictest15668-21apt-get22:50:132
705999168,5cyclictest10332-21grep00:10:221
7059991612,2cyclictest15665-21/usr/sbin/munin22:50:241
7059991611,3cyclictest5706-21sed20:40:181
7059991611,3cyclictest12370-21unixbench_multi00:15:271
7059991610,4cyclictest7121-21sed19:10:001
7059991610,4cyclictest6747-21wc20:45:131
7059991610,4cyclictest13475-21cut21:05:181
705599169,5cyclictest6398-21if_enp2s022:20:170
705599169,5cyclictest19470-21ls23:00:190
7055991611,3cyclictest8175-21ls22:25:160
7055991611,2cyclictest17666-21cpu00:35:120
706899159,4cyclictest7122-21cut19:10:013
706399151,11cyclictest2211-21sed20:30:142
7063991510,3cyclictest8519-21which22:25:202
705999159,4cyclictest31809-21awk20:20:201
705599158,5cyclictest1490-21cat23:45:120
705999145,6cyclictest21612-21sed19:50:151
706399131,10cyclictest7047-21cyclictest19:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional