You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot4s.osadl.org (updated Sat Jul 27, 2024 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28279993411,7cyclictest15324-21fschecks_count21:45:133
2827999310,3cyclictest21360-21turbostat.cron22:05:003
28275992816,8cyclictest24423-21apt-get20:35:102
2826699243,14cyclictest24746-21apt-get20:35:150
28266992410,4cyclictest9-21ksoftirqd/019:35:140
2827999231,3cyclictest29828-21munin-node19:15:013
28279992216,4cyclictest5599-21http22:55:003
28279992216,3cyclictest22178-21python323:45:013
28279992215,4cyclictest32315-21munin-run21:00:013
28275992218,2cyclictest7059-21ls19:40:212
28275992213,7cyclictest4410-21tune2fs22:50:122
28270992214,6cyclictest26161-21df23:55:141
28266992213,6cyclictest5771-21date22:55:010
2827999212,13cyclictest22419-21apt-get20:30:133
28279992116,3cyclictest25116-21ntp_offset23:50:183
28279992113,6cyclictest9879-21awk21:25:333
28279992113,6cyclictest30138-21awk20:50:203
28279992113,6cyclictest16896-21snmp_o5powerctl23:25:233
2827999210,5cyclictest10811-21turbostat23:10:003
2827999210,19cyclictest24160-21apt-config20:35:023
2827599212,13cyclictest32464-21apt-get00:15:112
28275992113,6cyclictest7176-21sed00:35:172
28275992113,6cyclictest32714-21ls21:00:132
28275992112,6cyclictest2949-21grep00:20:272
28275992112,6cyclictest21996-21grep23:40:242
2827599211,14cyclictest20805-21apt-get20:25:122
2827099212,14cyclictest14329-21apt-get20:05:101
28270992115,4cyclictest19717-21cron22:00:001
28270992114,5cyclictest424-21fw_packets21:00:171
28270992113,6cyclictest5747-21sed22:55:011
28270992113,6cyclictest23357-21memory23:45:151
28270992112,7cyclictest1294-21ntpq22:40:201
28266992116,3cyclictest21722-21sendmail_mailqu20:25:210
28266992115,3cyclictest1547-21latency_hist21:05:000
28266992114,5cyclictest27888-21forks20:45:140
28266992114,5cyclictest13930-21awk23:15:260
2826699211,14cyclictest11036-21apt-get23:10:110
2826699210,19cyclictest10916-21sed21:30:270
2827999201,3cyclictest3804-21apt-config22:50:013
28279992013,5cyclictest4315-21wc21:10:153
28279992013,5cyclictest32695-21tune2fs00:15:133
28279992013,5cyclictest31510-21snmp_o5powerctl19:15:243
28279992013,5cyclictest30330-21sed00:05:243
28279992012,6cyclictest7006-21basename00:35:163
28279992012,5cyclictest21526-21nfs4_client20:25:193
2827999200,7cyclictest17344-21lsb_release20:15:003
2827999200,3cyclictest6815-21lsb_release21:20:003
2827999200,16cyclictest1542-21apt-get21:05:053
2827999200,16cyclictest1542-21apt-get21:05:053
2827999200,14cyclictest4543-21cat19:35:003
2827599202,14cyclictest6175-21apt-get22:55:122
28275992013,5cyclictest6654-21awk21:15:282
28275992013,5cyclictest27767-21expr00:00:142
28275992013,5cyclictest20154-21which20:20:222
28275992012,6cyclictest31290-21awk19:15:202
28275992012,6cyclictest23368-21ls23:45:152
28275992012,6cyclictest15689-21cat21:45:172
2827599201,14cyclictest17514-21apt-get23:30:132
2827599201,14cyclictest17514-21apt-get23:30:132
28275992010,7cyclictest14111-21basename20:05:012
2827599200,14cyclictest7028-21apt-get21:20:092
2827099203,9cyclictest23805-21apt-get20:35:101
28270992015,3cyclictest7567-21sendmail23:00:001
28270992013,5cyclictest31017-21vmstat22:30:281
28270992013,5cyclictest21845-21tune2fs22:05:171
28270992013,5cyclictest18398-21proc_pri20:15:221
28270992012,6cyclictest9641-21snmp_o5powerctl21:25:311
28270992012,6cyclictest703-21sensors_temp19:20:231
28270992012,6cyclictest3285-21sed22:45:271
28270992012,6cyclictest20575-21memory22:00:181
28270992011,6cyclictest13060-21hostname23:15:151
2827099201,13cyclictest9470-21apt-get19:50:101
2826699202,12cyclictest16726-21apt-get21:50:120
28266992013,5cyclictest9-21ksoftirqd/000:10:120
28266992013,5cyclictest5223-21cut00:30:120
28266992013,5cyclictest22898-21expr23:45:110
28266992013,5cyclictest22699-21grep22:05:290
28266992013,5cyclictest20655-21grep22:00:190
28266992013,4cyclictest22796-21grep20:30:150
28266992012,5cyclictest32377-21threads22:35:300
28266992012,5cyclictest29972-21ntp_kernel_pll_00:05:180
2827999192,16cyclictest31076-21latency_hist22:35:013
2827999192,15cyclictest6329-21latency_hist00:35:013
28279991915,2cyclictest6844-21missed_timers19:40:193
28279991914,3cyclictest26199-21sh22:20:003
28279991914,3cyclictest26199-21sh22:20:003
28279991913,4cyclictest32551-21dpkg21:00:123
2827999191,2cyclictest29084-21munin-run20:49:593
28279991912,5cyclictest7331-21cat22:55:283
28279991911,6cyclictest22030-21cat22:05:183
28279991911,6cyclictest19174-21awk21:55:213
28279991911,6cyclictest11649-21wc19:55:193
28279991911,6cyclictest11649-21wc19:55:183
2827999191,13cyclictest19124-21apt-get23:35:103
28279991910,7cyclictest3095-21cron00:24:593
2827999190,17cyclictest32439-21cat00:14:593
28275991915,2cyclictest8560-21ntp_offset23:00:212
28275991913,3cyclictest0-21swapper/219:35:102
28275991912,5cyclictest3649-21tune2fs00:25:152
28275991912,5cyclictest18484-21expr21:55:122
28275991912,5cyclictest18387-21awk20:15:222
28275991912,5cyclictest13307-21wc21:40:002
28275991912,4cyclictest2128-21apt-key21:05:012
28275991912,4cyclictest2128-21apt-key21:05:012
28275991911,6cyclictest32468-21sed22:35:302
28275991911,6cyclictest32111-21awk00:10:282
28275991911,6cyclictest29841-21memory00:05:172
28275991911,6cyclictest2157-21cut22:45:012
28275991911,6cyclictest14168-21needreboot21:40:182
2827599191,13cyclictest26061-21apt-get20:40:122
2827599191,13cyclictest25978-21apt-get23:55:142
2827599191,13cyclictest15761-21latency_hist20:10:002
2827599191,13cyclictest14289-21apt-get23:20:132
2827599191,12cyclictest2164-21apt-get22:45:112
2827599191,12cyclictest15900-21apt-get23:25:112
28275991910,6cyclictest24956-21cat23:50:172
2827099192,12cyclictest30915-21apt-get20:55:111
28270991915,2cyclictest7573-21vmstat19:40:281
28270991915,2cyclictest13250-21sendmail21:40:001
28270991915,2cyclictest12719-21cpu20:00:111
28270991914,3cyclictest3426-21idleruntime19:30:131
28270991913,4cyclictest25825-21latency_hist23:55:011
28270991912,5cyclictest8121-21sed21:20:221
28270991912,5cyclictest31151-21ls19:15:171
28270991912,5cyclictest20367-21ls20:20:261
28270991912,5cyclictest19192-21wc23:35:101
28270991912,5cyclictest18967-21wc23:35:011
28270991912,5cyclictest18967-21wc23:35:011
28270991912,5cyclictest16475-21sed23:25:171
28270991912,5cyclictest10750-21vmstat23:05:261
28270991911,6cyclictest13927-21cat21:40:171
28270991911,6cyclictest11938-21proc_pri19:55:241
28270991911,6cyclictest11938-21proc_pri19:55:241
2827099191,13cyclictest4329-21snmp_o5powerctl00:25:251
2827099191,13cyclictest30851-21apt-get00:10:121
2826699199,2cyclictest111rcu_preempt19:32:160
2826699192,13cyclictest26061-21apt-get20:40:110
2826699192,12cyclictest20759-21apt-get23:40:100
28266991915,2cyclictest8415-21latency_hist21:25:000
28266991915,2cyclictest25000-21missed_timers23:50:170
28266991914,3cyclictest621-21rt-features19:20:220
28266991914,3cyclictest3629-21apt-key22:50:010
28266991913,4cyclictest29802-21expr22:30:100
28266991912,5cyclictest4571-21pluginstate21:10:190
28266991912,5cyclictest3624-21grep21:10:010
28266991912,5cyclictest3624-21grep21:10:010
28266991912,5cyclictest31057-21cat20:55:130
28266991912,5cyclictest28799-21awk22:25:240
28266991912,5cyclictest17185-21grep20:10:260
28266991912,5cyclictest14292-21grep20:05:000
28266991912,5cyclictest11464-21expr19:55:170
28266991912,5cyclictest11464-21expr19:55:160
2826699191,13cyclictest5567-21apt-get22:55:060
2826699191,13cyclictest18339-21apt-get21:55:110
2826699191,13cyclictest15112-21apt-get21:45:110
2826699191,12cyclictest8641-21apt-get21:25:120
2826699191,12cyclictest491-21apt-get22:40:110
28266991910,2cyclictest111rcu_preempt19:50:130
28279991814,2cyclictest20590-21perf20:25:003
28279991814,2cyclictest18908-21munin-run23:35:003
28279991814,2cyclictest18908-21munin-run23:35:003
2827999181,15cyclictest9215-21awk23:05:003
28279991811,5cyclictest5557-21mailstats19:35:213
28279991811,5cyclictest26508-21ls22:20:113
28279991811,5cyclictest23320-21wc23:45:143
28279991811,5cyclictest14800-21cat20:05:163
28279991811,5cyclictest11581-21cat23:10:173
2827999181,14cyclictest1945-21perf22:45:013
2827999181,13cyclictest29076-21snmp_o5powerctl22:25:313
2827999181,12cyclictest12648-21apt-get23:15:123
2827999181,12cyclictest11869-21apt-get21:35:113
2827999181,11cyclictest7852-21apt-get19:45:103
28279991810,6cyclictest27607-21cat20:45:003
2827999180,16cyclictest0-21swapper/320:05:013
2827599189,7cyclictest13131-21cat23:15:142
2827599183,8cyclictest15942-21apt-get20:10:092
28275991812,4cyclictest4157-21sed21:10:132
28275991812,4cyclictest27008-21sed22:20:172
28275991812,4cyclictest22587-21expr20:30:122
28275991811,5cyclictest21105-21sed22:00:272
28275991811,5cyclictest17660-21which21:50:232
28275991811,5cyclictest10565-21sed19:50:262
28275991811,4cyclictest30970-21grep22:30:272
2827599181,13cyclictest30912-21/usr/sbin/munin20:55:302
2827599181,12cyclictest4946-21apt-get00:30:102
2827599181,12cyclictest24818-21apt-get22:15:112
2827599181,12cyclictest24818-21apt-get22:15:112
2827599181,12cyclictest1369-21apt-get19:25:112
2827599181,12cyclictest11081-21apt-get19:55:132
2827599181,12cyclictest11081-21apt-get19:55:132
2827599181,12cyclictest10255-21apt-get21:30:142
28275991810,6cyclictest28865-21cat20:45:242
28275991810,6cyclictest20626-21sed23:40:012
28275991810,6cyclictest12170-21cat23:10:252
28270991814,2cyclictest29266-21sendmail_mailtr19:10:221
28270991814,2cyclictest26558-21interrupts20:40:181
28270991813,3cyclictest25141-21gltestperf22:15:141
28270991813,3cyclictest25141-21gltestperf22:15:141
28270991813,3cyclictest10633-21sed21:30:191
28270991812,4cyclictest15310-21entropy21:45:131
28270991811,5cyclictest5268-21fw_packets00:30:121
28270991811,5cyclictest31946-21cat22:35:211
28270991811,5cyclictest29418-21expr00:05:111
28270991811,5cyclictest28066-21apt22:25:121
28270991811,5cyclictest22964-21cat20:30:171
28270991811,5cyclictest21859-21sed20:25:251
2827099181,13cyclictest7556-21snmp_o5powerctl00:35:241
2827099181,13cyclictest27454-21snmp_o5powerctl22:20:241
2827099181,12cyclictest7812-21apt-get23:00:121
2827099181,12cyclictest29299-21apt-get20:50:121
28270991810,6cyclictest20955-21diskstats23:40:121
2826699183,9cyclictest30509-21apt-get19:15:110
28266991815,2cyclictest3569-21fschecks_count00:25:130
28266991814,2cyclictest9308-21cat23:04:590
28266991814,2cyclictest9247-21munin-plugin-st19:50:000
28266991812,4cyclictest14906-21sed23:20:200
28266991811,5cyclictest32735-21cut00:15:140
28266991811,5cyclictest20686-21wc20:25:000
28266991811,5cyclictest18450-21readlink23:30:230
28266991811,5cyclictest18450-21readlink23:30:230
28266991811,4cyclictest26210-21sendmail22:20:000
28266991811,4cyclictest26210-21sendmail22:20:000
2826699181,12cyclictest27618-21apt-get00:00:110
2826699181,12cyclictest13502-21apt-get21:40:110
28266991811,1cyclictest7905-21smbd00:36:360
2826699180,4cyclictest0-21swapper/019:40:200
2827999179,6cyclictest3889-21sed00:25:163
28279991714,2cyclictest27615-21/usr/sbin/munin00:00:173
28279991713,2cyclictest7524-21interrupts21:20:153
28279991713,2cyclictest26906-21sendmail_mailst23:55:223
28279991711,4cyclictest2215-21proc_pri19:25:203
28279991711,4cyclictest10084-21cat19:50:173
28279991710,5cyclictest23203-21dpkg22:10:073
28279991710,5cyclictest18376-21grep20:15:213
28279991710,5cyclictest13931-21idleruntime21:40:163
2827999170,16cyclictest0-21swapper/321:55:003
28275991713,3cyclictest9422-21/usr/sbin/munin23:05:202
28275991713,3cyclictest30387-21ls20:50:262
28275991711,4cyclictest32319-21expr19:20:132
2827599171,12cyclictest21589-21apt-get22:05:142
2827599171,11cyclictest23201-21apt-get22:10:092
28275991710,5cyclictest8879-21grep21:25:142
2827099179,6cyclictest2698-21users19:25:261
28270991713,2cyclictest6257-21processes21:15:231
28270991713,2cyclictest15993-21cpuspeed_turbos20:10:121
2827099171,13cyclictest24195-21snmp_o5powerctl22:10:221
2827099171,12cyclictest8853-21snmp_o5powerctl19:45:241
2827099171,12cyclictest2715-21snmp_o5powerctl00:20:251
2827099171,12cyclictest12033-21snmp_o5powerctl23:10:221
2826699179,6cyclictest9743-21tune2fs23:05:130
28266991714,2cyclictest1921-21fschecks_count00:20:130
28266991713,2cyclictest29167-21wc19:10:210
28266991713,2cyclictest1372-21cpu19:25:110
28266991711,4cyclictest19490-21grep23:35:130
2826699171,12cyclictest30300-21snmp_o5powerctl20:50:240
2826699171,11cyclictest5391-21apt-get21:15:110
28266991710,5cyclictest17854-21grep20:15:160
28266991710,5cyclictest14744-21sed20:05:160
28266991710,4cyclictest9-21ksoftirqd/022:20:100
2827999169,5cyclictest11374-21sed21:30:503
2827999161,2cyclictest15686-21awk23:24:593
28279991610,4cyclictest31635-21grep22:35:173
2827999160,14cyclictest32214-21wc19:20:123
28275991613,2cyclictest29923-21munin-node19:30:092
28275991612,2cyclictest7849-21/usr/sbin/munin19:45:152
2827599161,12cyclictest29301-21snmp_o5powerctl19:10:222
2827599161,12cyclictest29074-21snmp_o5powerctl22:25:302
2827099169,5cyclictest4881-21cut19:35:131
2827099169,4cyclictest27800-21expr20:45:121
2827099168,6cyclictest3947-21expr21:10:121
2827099168,6cyclictest28947-21users00:00:291
28270991611,3cyclictest32242-21latency_hist00:15:011
2827099161,11cyclictest15284-21snmp_o5powerctl23:20:271
2826699168,2cyclictest111rcu_preempt21:38:510
28266991612,2cyclictest16004-21cstates23:25:110
28266991611,3cyclictest23343-21expr22:10:100
2827599151,11cyclictest28081-21latency_hist19:10:012
2827099159,4cyclictest16873-21ls21:50:131
2827099158,5cyclictest1919-21sed21:05:021
2827099158,5cyclictest1919-21sed21:05:011
2826699158,5cyclictest26965-21sensors_temp23:55:220
2827999120,2cyclictest28081-21latency_hist19:10:013
2826699101,7cyclictest28268-21latency_hist19:10:010
282709994,4cyclictest28281-21latency_hist19:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional