You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-08 - 16:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot4s.osadl.org (updated Wed May 08, 2024 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2201999262,19cyclictest12898-21apt-get23:35:163
22014992611,10cyclictest1267-21apt-get23:00:122
2201499252,13cyclictest3875-21apt-get21:30:102
2201199242,15cyclictest21465-21kworker/u8:022:40:061
2201999231,8cyclictest17912-21lsb_release20:34:593
2201999231,4cyclictest32003-40apt-check22:55:053
2201499232,14cyclictest32410-21apt-get00:35:122
2201499232,14cyclictest32410-21apt-get00:35:122
22014992314,7cyclictest3999-21cat19:50:162
22014992313,7cyclictest31796-21apt-key22:55:002
2201199239,10cyclictest25912-21cron19:20:011
22005992315,6cyclictest27044-21awk21:00:220
2201999221,5cyclictest31552-21python322:54:593
2201999221,5cyclictest26591-21dpkg22:40:013
2201999221,4cyclictest27577-21munin-run19:25:013
22019992214,6cyclictest3376-21cat23:05:193
22019992214,6cyclictest16433-21sed22:05:303
22019992213,6cyclictest8563-21cat21:45:003
2201999220,6cyclictest8176-21perf20:05:013
2201999220,19cyclictest29295-21python321:10:003
2201499223,16cyclictest26706-21cut19:20:182
2201499222,14cyclictest4508-21apt-get23:10:112
2201499222,13cyclictest24624-21apt-get20:55:152
22014992214,4cyclictest21055-21turbostat.cron00:00:012
22014992212,6cyclictest340-21basename21:20:012
22014992211,7cyclictest32429-21readlink19:39:592
2201499221,14cyclictest18035-21apt-get23:50:142
2201499220,8cyclictest12022-21sed20:15:202
2201199229,9cyclictest27613-21cut00:20:001
2201199228,9cyclictest467-21apt-get00:35:151
2201199228,9cyclictest467-21apt-get00:35:141
2201199228,9cyclictest22917-21apt-get00:05:111
22011992212,6cyclictest26022-21systemctl21:00:001
2201199221,15cyclictest6126-21apt-get23:15:101
2201199221,15cyclictest18481-21apt-get22:15:141
2201199221,15cyclictest18481-21apt-get22:15:141
22011992210,8cyclictest17613-21date23:50:011
2200599222,13cyclictest19653-21apt-get23:55:110
2200599221,6cyclictest25931-21idleruntime-cro00:15:000
22019992115,4cyclictest13526-21cat22:00:003
22019992114,5cyclictest32490-21cut19:40:003
22019992114,5cyclictest18508-21cat23:50:183
22019992112,7cyclictest23915-21snmp_o5powerctl00:05:263
2201499218,9cyclictest7768-21apt-get23:20:132
2201499214,14cyclictest5532-21gltestperf19:55:202
2201499214,14cyclictest5532-21gltestperf19:55:202
2201499212,13cyclictest5491-21apt-get21:35:112
2201499212,13cyclictest13618-21apt-get22:00:142
22014992114,5cyclictest29387-21cat19:30:002
22014992113,5cyclictest24771-21idleruntime-cro22:35:002
22014992112,6cyclictest16289-21basename20:29:592
2201199218,9cyclictest2891-21apt-get23:05:131
2201199212,13cyclictest27878-21apt-get21:05:131
2201199212,13cyclictest21294-21apt-get00:00:111
22011992112,6cyclictest4889-21sed23:10:171
2201199211,14cyclictest9386-21apt-get23:25:131
2200599212,13cyclictest23003-21apt-get20:50:120
22005992116,3cyclictest2336-21smbd23:38:030
22005992113,6cyclictest28457-21needreboot00:20:190
22005992113,5cyclictest23319-21sensors_temp19:10:350
22019992013,5cyclictest8843-21grep23:20:323
22019992013,5cyclictest6061-21which19:55:293
22019992013,5cyclictest6061-21which19:55:293
22019992013,5cyclictest30139-21ntpq00:25:223
22019992013,5cyclictest29880-21expr21:10:153
22019992013,5cyclictest24851-21cat22:35:003
22019992013,5cyclictest22139-21expr22:25:203
22019992013,5cyclictest12430-21http23:35:003
22019992012,6cyclictest25622-21/usr/sbin/munin20:55:303
22019992012,5cyclictest16431-21tune2fs23:45:143
22019992012,4cyclictest8746-21apt-get21:45:113
2201999201,17cyclictest15023-21munin-run22:05:003
2201999201,16cyclictest0-21swapper/300:35:063
2201999201,16cyclictest0-21swapper/300:35:053
2201999200,3cyclictest32426-21munin-run00:35:003
2201999200,2cyclictest21838-21python319:10:013
2201499206,9cyclictest10362-21apt-get21:50:132
2201499202,13cyclictest12425-21python323:35:012
2201499202,12cyclictest26892-21apt-get22:40:152
2201499202,12cyclictest22917-21apt-get00:05:112
2201499202,10cyclictest21837-21python319:10:012
22014992016,2cyclictest28928-21timerwakeupswit00:20:292
22014992014,4cyclictest17951-21sed22:10:342
22014992014,3cyclictest27473-21grep21:00:282
22014992013,5cyclictest21884-21awk20:45:162
22014992012,6cyclictest20879-21ntp_states22:20:232
22014992011,6cyclictest4180-21grep23:05:322
22014992011,6cyclictest32660-21apt-get19:40:122
22014992011,6cyclictest2254-21apt-get21:25:102
2201499201,14cyclictest10008-21apt-get20:10:162
2201499201,13cyclictest32447-21python300:35:012
2201499201,13cyclictest26433-21wc22:40:012
2201499201,13cyclictest19764-21apt-get20:40:142
2201499201,12cyclictest6126-21apt-get23:15:112
2201199207,8cyclictest1888-21apt-get19:45:141
2201199202,13cyclictest7130-21apt-get21:40:141
2201199202,13cyclictest27790-21apt-get19:25:121
2201199202,10cyclictest12897-21apt-get23:35:121
22011992013,5cyclictest1270-21http23:00:121
22011992013,5cyclictest11363-21snmp_o5powerctl21:50:291
22011992013,5cyclictest11173-21sed20:10:381
22011992013,4cyclictest26579-21apt-key22:40:021
2201199201,13cyclictest19653-21apt-get23:55:111
2201199201,12cyclictest27793-21apt-get00:20:121
2200599202,12cyclictest28510-21apt-get22:45:140
2200599202,12cyclictest2254-21apt-get21:25:120
22005992015,3cyclictest1-21systemd23:34:590
22005992014,4cyclictest15679-21ls22:05:170
22005992013,5cyclictest7395-21unixbench_singl23:15:280
22005992013,5cyclictest25738-21ls20:55:320
22005992013,5cyclictest23992-21threads00:05:280
22005992013,5cyclictest22489-21ntp_kernel_pll_22:25:240
22005992013,5cyclictest18870-21ntpq20:35:220
22005992013,5cyclictest14770-21expr23:40:170
22005992012,6cyclictest18019-21sed22:10:340
22005992011,6cyclictest5253-21cron21:35:010
2200599201,13cyclictest8746-21apt-get21:45:120
2200599201,13cyclictest32396-21kworker/u8:200:35:060
2200599201,13cyclictest32396-21kworker/u8:200:35:060
2200599201,13cyclictest29411-21apt-get00:25:110
2200599201,13cyclictest26895-21apt-get22:40:130
2200599201,13cyclictest13265-21apt-get20:20:110
2201999192,15cyclictest7525-21latency_hist23:19:593
2201999192,15cyclictest3427-21cat19:50:003
22019991915,2cyclictest22693-21munin-run00:05:003
22019991915,2cyclictest20850-21unixbench-2d23:55:293
22019991912,5cyclictest8711-21fw_packets20:05:173
22019991912,5cyclictest29595-21ls22:45:313
22019991912,5cyclictest28032-21grep00:20:153
22019991912,5cyclictest19767-21http20:40:133
22019991912,5cyclictest19719-21date20:40:003
22019991912,5cyclictest15202-21tune2fs20:25:183
22019991912,5cyclictest13052-21tr20:20:013
2201999191,16cyclictest0-21swapper/321:40:003
2201999191,13cyclictest22330-21apt-get19:10:133
2201999190,9cyclictest24001-21snmp_o5powerctl20:50:283
2201499198,7cyclictest14534-21apt-get23:40:122
2201499193,9cyclictest12894-21apt-get23:35:162
2201499192,12cyclictest22310-40apt-check19:10:092
22014991912,5cyclictest8639-21sed21:45:012
22014991911,6cyclictest1870-21grep21:20:292
2201499191,13cyclictest21739-21apt-get22:25:142
2201499191,13cyclictest13265-21apt-get20:20:112
2201499191,13cyclictest11981-21apt-get21:55:122
2201499191,12cyclictest29768-21apt-get21:10:162
2201499191,12cyclictest16501-21apt-get20:30:142
2201499191,12cyclictest0-21swapper/220:50:122
2201199193,9cyclictest19764-21apt-get20:40:131
22011991913,4cyclictest28513-21http22:45:141
22011991913,3cyclictest32419-21latency_hist00:35:001
22011991912,5cyclictest5272-21expr19:55:171
22011991912,5cyclictest5272-21expr19:55:161
22011991912,5cyclictest17336-21awk20:30:281
22011991912,5cyclictest13816-21diskstats22:00:161
2201199191,13cyclictest8386-21apt-get20:05:121
2201199191,13cyclictest26174-21apt-get19:20:101
2201199191,13cyclictest14883-21apt-get20:25:121
2201199191,12cyclictest31021-21apt-get19:35:111
2201199191,12cyclictest29407-21apt-get19:30:111
2201199191,12cyclictest21381-21apt-get20:45:111
2200599199,7cyclictest21870-21cat20:45:150
2200599192,12cyclictest1267-21apt-get23:00:110
22005991915,2cyclictest18901-21hwlatdetect22:15:190
22005991915,2cyclictest18901-21hwlatdetect22:15:190
22005991914,3cyclictest25584-21missed_timers22:35:180
22005991913,4cyclictest32706-21vmstat21:15:280
22005991912,5cyclictest8478-21grep20:05:130
22005991912,5cyclictest4608-21seq23:10:130
22005991912,5cyclictest19925-21expr20:40:140
22005991912,5cyclictest18138-21cat23:50:140
22005991912,5cyclictest1474-21proc_pri21:20:230
22005991912,5cyclictest12714-21ntp_kernel_pll_21:55:210
22005991912,5cyclictest11834-21forks20:15:180
2200599191,13cyclictest32660-21apt-get19:40:140
2200599191,13cyclictest31821-21sort22:55:000
2201999182,12cyclictest10999-21snmp_o5powerctl20:10:343
22019991814,2cyclictest29404-21/usr/sbin/munin19:30:153
22019991812,4cyclictest31516-21expr21:15:123
22019991812,4cyclictest18592-21seq22:15:143
22019991812,4cyclictest18592-21seq22:15:143
22019991811,5cyclictest4197-21fw_packets21:30:133
22019991811,5cyclictest21685-21cut22:25:013
22019991811,5cyclictest15553-21expr23:40:313
22019991811,5cyclictest1255-21cat21:20:203
22019991811,5cyclictest10445-21cat23:25:293
2201999181,15cyclictest0-21swapper/322:40:133
22019991811,4cyclictest5052-21cat19:55:013
2201999181,13cyclictest27259-21snmp_o5powerctl21:00:253
22019991810,5cyclictest4856-21expr23:10:163
2201999180,1cyclictest0-21swapper/319:20:013
2201999180,17cyclictest0-21swapper/321:30:013
2201999180,16cyclictest27566-21basename00:19:593
2201499182,9cyclictest21294-21apt-get00:00:112
22014991814,2cyclictest664-21timerandwakeup22:55:342
22014991811,5cyclictest16484-21grep22:05:312
22014991811,4cyclictest9573-21head21:45:242
2201499181,13cyclictest30405-21snmp_o5powerctl19:30:272
2201499181,13cyclictest2893-21snmp_o5powerctl19:45:322
2201499181,13cyclictest25532-21snmp_o5powerctl19:15:322
2201499181,13cyclictest18123-21apt-get20:35:122
2201499181,12cyclictest8386-21apt-get20:05:122
2201499181,12cyclictest29411-21apt-get00:25:132
2201499181,12cyclictest16149-21apt-get23:45:112
2201499181,11cyclictest29446-21ls21:10:002
2201499181,11cyclictest26152-21apt-get00:15:112
2201499181,11cyclictest24534-21apt-get00:10:142
2201199183,10cyclictest29751-40apt-check21:10:091
22011991814,2cyclictest4805-21sendmail_mailst21:30:211
22011991814,2cyclictest23936-21mailstats20:50:241
22011991811,5cyclictest4035-21cut19:50:171
22011991811,5cyclictest32053-21grep21:15:171
22011991811,5cyclictest23269-21cut22:30:001
22011991811,5cyclictest21373-21awk22:20:311
2201199181,13cyclictest1629-21snmp_o5powerctl21:20:271
2201199181,12cyclictest31513-21apt-get22:55:101
2201199181,12cyclictest26261-21apt-get21:00:121
2201199181,12cyclictest23362-21apt-get22:30:141
2201199181,12cyclictest11001-21apt-get23:30:131
2201199181,11cyclictest6768-21apt-get20:00:111
22011991810,5cyclictest12053-21cat20:15:211
22005991815,2cyclictest7129-21apt21:40:110
22005991814,2cyclictest6563-21threads21:35:320
22005991814,2cyclictest26217-21cpuspeed_turbos00:15:120
22005991814,2cyclictest16649-21interrupts23:45:170
22005991812,4cyclictest5795-21needreboot19:55:240
22005991812,4cyclictest5795-21needreboot19:55:230
22005991811,5cyclictest8091-21vmstat20:00:340
22005991811,5cyclictest23903-21wc22:30:210
22005991811,5cyclictest21418-21expr00:00:120
22005991811,5cyclictest17103-21load20:30:230
22005991811,5cyclictest16411-21cat20:30:000
2200599181,13cyclictest25529-21snmp_o5powerctl19:15:310
2200599181,13cyclictest13618-21apt-get22:00:120
2200599181,12cyclictest7768-21apt-get23:20:130
2200599181,12cyclictest21838-21python319:10:010
22019991713,2cyclictest25924-21idleruntime-cro00:14:593
22019991711,4cyclictest327-21expr19:40:133
22019991711,4cyclictest16970-21cat22:10:153
22019991711,4cyclictest14442-21cat20:20:293
22019991711,4cyclictest10415-21ls21:50:143
2201999171,12cyclictest28785-21snmp_o5powerctl19:25:293
2201999171,12cyclictest2272-21snmp_o5powerctl23:00:273
2201499173,8cyclictest28510-21apt-get22:45:122
2201499172,8cyclictest6768-21apt-get20:00:092
2201499172,11cyclictest9386-21apt-get23:25:142
2201199172,8cyclictest11981-21apt-get21:55:111
22011991713,2cyclictest18955-21sendmail_mailqu23:50:261
22011991713,2cyclictest17779-21sendmail_mailqu22:10:301
22011991712,3cyclictest3013-21ntp_offset21:25:221
2201199171,12cyclictest9744-21snmp_o5powerctl21:45:261
2201199171,12cyclictest6483-21snmp_o5powerctl21:35:221
2201199171,11cyclictest22310-40apt-check19:10:061
22011991710,5cyclictest26063-21cat00:15:001
22011991710,5cyclictest13316-21awk20:20:111
2201199170,15cyclictest18639-21cut20:35:191
2200599179,6cyclictest11478-21ls21:50:310
22005991713,2cyclictest3870-21tail19:50:140
22005991712,3cyclictest29377-21rm21:10:010
22005991712,3cyclictest20939-21ls22:20:240
22005991711,4cyclictest28146-21grep19:25:160
22005991710,5cyclictest31968-21find22:55:010
2201999160,1cyclictest0-21swapper/320:50:013
2201499161,11cyclictest19226-21ntp_kernel_pll_22:15:242
2201499161,11cyclictest19226-21ntp_kernel_pll_22:15:242
2201199163,8cyclictest30127-21apt-get22:50:131
2201199161,8cyclictest22020-21apt-config19:10:011
22011991614,1cyclictest0-21swapper/119:40:261
22011991612,2cyclictest30806-21latency_hist00:30:001
2201199161,12cyclictest8769-21snmp_o5powerctl23:20:311
2201199161,10cyclictest16677-21latency_hist22:10:001
22011991610,4cyclictest15047-21cut23:40:221
2200599169,5cyclictest32157-21cat00:30:300
2200599169,5cyclictest30426-21expr19:30:270
2200599169,5cyclictest10156-21ls20:10:180
2200599169,4cyclictest31567-21latency19:35:170
2200599161,12cyclictest2895-21snmp_o5powerctl19:45:330
22005991610,4cyclictest9609-21cut23:25:170
2200599159,4cyclictest3642-21ntpq23:05:230
2200599148,4cyclictest30551-21/usr/sbin/munin21:10:260
22005991410,2cyclictest26502-21grep19:20:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional