You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-10 - 21:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot4s.osadl.org (updated Mon Sep 16, 2024 12:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
106232316247,57sleep20-21swapper/207:05:402
112052305218,70sleep10-21swapper/107:09:391
110662251202,35sleep00-21swapper/007:08:260
108812248196,38sleep30-21swapper/307:06:463
2173321460,4sleep21144299cyclictest11:15:002
27802520,9sleep119-21ksoftirqd/109:15:171
192382510,1sleep00-21swapper/011:10:000
1144499270,24cyclictest1454-21python308:00:003
1144499261,22cyclictest24855-21latency_hist07:40:013
11444992517,6cyclictest29799-21pidof09:02:453
1144299242,15cyclictest23861-21taskset08:50:082
1144499231,19cyclictest23267-21python308:50:003
1144499231,18cyclictest19995-40apt-check12:25:053
1144499230,20cyclictest25361-40apt-check07:40:063
11442992314,6cyclictest24896-21id07:40:002
1144499221,18cyclictest21406-21python311:15:003
1144299222,14cyclictest6344-21apt-get08:10:122
1144299222,13cyclictest24167-21apt-get11:20:132
11442992217,3cyclictest23235-21apt-get08:50:012
11442992216,3cyclictest26578-21sendmail12:40:002
1143799229,7cyclictest19-21ksoftirqd/107:55:551
11437992213,6cyclictest10937-21tune2fs08:20:131
11437992210,3cyclictest19-21ksoftirqd/107:34:231
1143799220,8cyclictest22316-21grep08:45:201
11433992214,5cyclictest14076-21sensors_temp10:55:240
11433992213,6cyclictest17610-21expr12:20:130
11433992213,6cyclictest17610-21expr12:20:130
42102210,10sleep225-21ksoftirqd/209:20:002
11444992114,5cyclictest14409-21ls07:15:133
1144499211,3cyclictest28202-21munin-run11:29:593
11444992113,6cyclictest22560-21cat11:15:173
11444992113,6cyclictest19584-21snmp_o5powerctl07:25:223
11444992113,6cyclictest12516-21tr07:10:193
1144499211,17cyclictest19408-21python312:25:003
1144499211,17cyclictest19408-21python312:24:593
1144499210,19cyclictest3855-21munin-run10:35:003
1144299219,8cyclictest25509-21apt-get07:40:152
1144299212,13cyclictest26299-21apt-get11:25:122
1144299212,13cyclictest14273-21apt-get07:15:132
11442992116,3cyclictest1482-21gpgv08:00:012
11442992114,5cyclictest4430-21awk08:05:142
11442992114,5cyclictest28714-21awk10:15:202
11442992114,5cyclictest24252-21wc10:05:182
11442992114,5cyclictest17233-21sed11:05:012
11442992113,6cyclictest13052-21date12:10:002
11442992110,7cyclictest12581-21date08:25:002
1143799212,14cyclictest8483-21apt-get08:15:131
1143799211,7cyclictest19-21ksoftirqd/110:09:051
1143799211,6cyclictest30455-21latency_hist11:35:001
11437992114,4cyclictest26186-21cut10:10:151
11437992110,3cyclictest19-21ksoftirqd/109:09:371
1143399219,8cyclictest15215-21apt-get11:00:130
1143399216,13cyclictest111rcu_preempt08:50:010
11433992116,3cyclictest5620-21grep11:50:210
11433992115,4cyclictest20802-21ntp_states12:25:220
11433992114,5cyclictest26329-21pmu-power07:40:220
11433992114,5cyclictest23957-21grep07:35:200
11433992114,5cyclictest12686-21wc08:25:000
11433992113,6cyclictest9480-21threads10:45:240
11433992113,6cyclictest17960-21ntpq09:50:190
1143399210,5cyclictest0-21swapper/010:10:270
11444992016,2cyclictest18958-21munin-run08:40:003
11444992016,2cyclictest15464-21latency08:30:163
11444992016,2cyclictest1130-21runrttasks10:55:143
11444992015,3cyclictest29801-21sh10:19:593
11444992014,4cyclictest27923-21idleruntime-cro09:00:013
11444992014,3cyclictest3858-21runrttasks09:18:013
11444992014,3cyclictest31846-21runrttasks09:06:583
1144499201,3cyclictest24079-21python312:35:013
11444992013,5cyclictest24291-21expr11:20:133
11444992013,5cyclictest23761-21cat10:05:143
11444992013,5cyclictest20644-21awk09:55:243
11444992013,5cyclictest16567-21expr07:20:133
11444992012,6cyclictest7343-21cat11:55:163
11444992012,6cyclictest18354-21sed09:50:253
11444992012,5cyclictest5635-21snmp_o5powerctl11:50:223
11444992010,7cyclictest839-21dbus-daemon12:35:013
1144299202,12cyclictest19873-40apt-check12:25:092
1144299202,12cyclictest19485-21apt-get11:10:132
1144299202,11cyclictest13625-21munin-node10:20:132
11442992013,5cyclictest937-21sed10:25:232
11442992013,5cyclictest18498-21sed07:25:002
11442992013,5cyclictest14989-21sed09:45:012
11442992013,5cyclictest14989-21sed09:45:002
11442992012,6cyclictest3568-21sed11:45:232
11442992012,6cyclictest28734-21cat09:00:162
11442992012,6cyclictest22927-21ntpq12:30:212
11442992011,6cyclictest12447-21systemctl10:55:002
1144299201,13cyclictest21499-21apt-get10:00:172
1143799202,3cyclictest19-21ksoftirqd/109:30:001
1143799202,12cyclictest4361-21apt-get09:20:111
1143799201,7cyclictest19-21ksoftirqd/111:45:151
1143799201,7cyclictest19-21ksoftirqd/109:40:131
1143799201,7cyclictest19-21ksoftirqd/109:40:131
1143799201,5cyclictest30570-21apt-get11:35:111
11437992014,3cyclictest19382-21latency_hist12:25:011
11437992014,3cyclictest19382-21latency_hist12:25:011
11437992013,5cyclictest28326-21cat10:15:161
11437992012,6cyclictest26420-21expr11:25:121
11437992012,6cyclictest21894-21grep10:00:201
11437992012,5cyclictest19685-21date12:25:001
11437992012,5cyclictest18195-21grep11:05:191
1143799201,13cyclictest31706-21latency_hist07:55:001
1143799201,13cyclictest19332-21apt-get09:55:121
1143799200,5cyclictest4208-21/usr/sbin/munin08:05:211
1143399209,3cyclictest217342readlink11:15:000
1143399202,15cyclictest19045-21awk07:25:150
11433992016,2cyclictest21100-21ls08:45:010
11433992015,3cyclictest22635-21sh07:35:000
11433992013,5cyclictest8429-21sed08:15:000
11433992013,5cyclictest3415-21uptime08:00:250
11433992013,5cyclictest29280-21awk11:30:200
11433992013,5cyclictest2816-21memory09:15:160
11433992013,5cyclictest24591-21readlink10:05:200
11433992013,5cyclictest12498-21which07:10:190
11433992012,5cyclictest31066-21mailstats10:20:230
11433992012,5cyclictest16061-21nfs4_client12:15:190
11433992011,6cyclictest4413-21tune2fs10:35:150
1144499199,6cyclictest10022-21timerandwakeup12:00:253
11444991914,3cyclictest14674-21ls08:30:003
11444991914,2cyclictest23731-40apt-check08:50:063
11444991913,4cyclictest7239-21ntpq09:25:163
11444991913,4cyclictest4485-21expr09:20:133
11444991912,5cyclictest8524-21expr10:45:123
11444991912,5cyclictest7477-21grep10:40:253
11444991912,5cyclictest7477-21grep10:40:253
11444991912,5cyclictest313-21systemd-journal10:25:003
11444991912,5cyclictest30170-21expr07:50:153
11444991912,5cyclictest28595-21which07:45:203
11444991912,5cyclictest26616-21awk10:10:213
11444991912,5cyclictest1784-21cut10:30:003
11444991912,5cyclictest15496-21cat09:45:153
11444991912,5cyclictest13252-21ls12:10:143
11444991912,5cyclictest10957-21cat10:50:173
1144499191,13cyclictest23693-21snmpd08:20:113
1144499191,13cyclictest21503-21pidof10:00:153
11444991910,6cyclictest8921-21idleruntime08:15:173
1144499190,3cyclictest1456-21python308:00:023
1144499190,17cyclictest6023-21lsb_release10:40:013
1144299193,8cyclictest13064-21apt-get10:55:132
1144299191,4cyclictest0-21swapper/207:49:142
11442991912,5cyclictest9401-21snmp_o5powerctl10:45:222
11442991912,5cyclictest7422-21cat11:55:172
11442991912,5cyclictest7271-21snmp_o5powerctl10:40:242
11442991912,5cyclictest7271-21snmp_o5powerctl10:40:242
11442991912,5cyclictest18692-21expr07:25:122
11442991912,4cyclictest1953-21latency_hist09:15:002
11442991911,6cyclictest26618-21sed08:55:182
11442991911,5cyclictest27120-21who10:10:262
1144299191,13cyclictest8910-21apt-get12:00:122
1144299191,13cyclictest32724-21apt-get11:40:122
1144299191,13cyclictest15327-21apt-get12:15:122
11442991910,6cyclictest30511-21diskstats09:05:132
1143799193,13cyclictest12220-21cat12:05:221
1143799192,5cyclictest19-21ksoftirqd/109:35:121
1143799192,13cyclictest25091-21touch07:40:011
1143799192,13cyclictest21409-21http11:15:001
11437991913,3cyclictest2526-21latency10:30:161
11437991912,5cyclictest7531-21open_files11:55:181
11437991912,5cyclictest4313-21uniq10:35:151
11437991912,5cyclictest17000-21cat07:20:171
11437991912,5cyclictest15475-21cat07:15:231
11437991912,5cyclictest15470-21expr09:45:141
11437991912,5cyclictest15365-21cat11:00:141
11437991912,4cyclictest25151-21apt-key07:40:011
11437991911,6cyclictest29282-21grep09:00:231
11437991911,6cyclictest2206-21expr08:00:121
11437991911,5cyclictest24509-21expr11:20:161
11437991911,5cyclictest17132-21sshd12:19:211
11437991911,5cyclictest14943-21dump-pmu-power11:00:001
1143799191,13cyclictest18074-21snmp_o5powerctl08:35:231
1143799191,13cyclictest1739-21latency_hist10:30:011
1143799191,12cyclictest23731-40apt-check08:50:031
1143799191,12cyclictest20512-21wc07:30:001
1143799190,7cyclictest19-21ksoftirqd/112:37:191
1143399192,12cyclictest28172-21apt-get09:00:110
11433991915,2cyclictest23267-21python308:50:000
11433991914,4cyclictest9-21ksoftirqd/010:00:010
11433991914,3cyclictest31307-21nfs4_client11:35:170
11433991913,4cyclictest30537-21ntpq07:50:190
11433991912,5cyclictest8796-21ls09:30:200
11433991912,5cyclictest655-21cat07:55:240
11433991912,5cyclictest4544-21cat11:50:010
11433991912,5cyclictest2999-21expr10:30:230
11433991912,5cyclictest27322-21swap11:25:230
11433991912,5cyclictest24821-21expr12:35:150
11433991912,5cyclictest10097-21sed12:00:250
11433991911,6cyclictest14067-21unixbench_singl07:15:020
1143399191,14cyclictest26017-21apt-get08:55:130
1143399191,13cyclictest32444-21apt-get09:10:130
11433991910,6cyclictest2291-21sed11:45:000
1144499188,8cyclictest393-21gltestperf09:10:153
11444991814,2cyclictest16048-21pmu-power11:00:203
11444991812,3cyclictest21137-21latency_hist08:45:003
11444991811,5cyclictest8400-21date08:14:593
11444991811,5cyclictest4882-21cat08:05:173
11444991811,5cyclictest407-21expr11:40:133
11444991811,5cyclictest28926-21cat11:30:163
11444991811,5cyclictest17460-21cat12:20:013
11444991811,5cyclictest13066-21expr09:40:133
11444991811,5cyclictest13066-21expr09:40:133
11444991811,5cyclictest12955-21pluginstate12:10:003
1144499181,13cyclictest20727-21apt-get07:30:123
11444991810,5cyclictest19242-21grep11:10:003
1144499180,9cyclictest31582-21snmp_o5powerctl11:35:223
1144299188,7cyclictest22608-21cron07:35:002
11442991812,4cyclictest4970-21ps10:35:212
11442991811,5cyclictest5144-21ntpq09:20:192
11442991811,5cyclictest31250-21cut11:35:182
11442991811,5cyclictest17354-21dpkg11:05:102
1144299181,13cyclictest9505-21snmp_o5powerctl08:15:242
1144299181,13cyclictest13803-21snmp_o5powerctl08:25:252
1144299181,12cyclictest29779-21apt-get07:50:122
1144299181,12cyclictest17057-21apt-get08:35:132
1144299181,12cyclictest15067-21apt-get09:45:122
1144299181,12cyclictest14920-21apt-get08:30:122
11442991810,6cyclictest8475-21sed09:30:002
11442991810,6cyclictest642-21cat07:55:252
11442991810,6cyclictest20109-21ntp_states09:55:172
11442991810,5cyclictest9034-21sed09:30:212
1143799182,5cyclictest0-21swapper/111:50:331
11437991812,4cyclictest26225-21entropy08:55:151
11437991811,5cyclictest8747-21expr09:30:201
11437991811,5cyclictest6689-21grep08:10:151
11437991811,5cyclictest537-21sed09:10:171
11437991811,5cyclictest30444-21expr10:20:151
11437991811,5cyclictest28044-21head07:45:161
11437991811,5cyclictest10730-21entropy10:50:161
11437991811,4cyclictest22420-21sed11:15:161
1143799181,13cyclictest6248-21apt-get10:40:131
1143799181,13cyclictest6248-21apt-get10:40:131
1143799181,12cyclictest23994-21runrttasks12:34:381
1143799181,12cyclictest2116-21runrttasks11:44:021
11437991810,6cyclictest9607-21grep12:00:191
1143799180,6cyclictest19-21ksoftirqd/108:30:011
11433991815,2cyclictest32294-21cstates10:25:120
11433991814,2cyclictest9434-21sendmail_mailst08:15:220
11433991814,2cyclictest6858-21cstates11:55:120
11433991814,2cyclictest28844-21sendmail_mailqu10:15:210
11433991814,2cyclictest11659-21processes09:35:190
11433991811,5cyclictest20472-21systemctl11:10:210
11433991811,5cyclictest13054-21tune2fs08:25:180
1143399181,13cyclictest25183-21snmp_o5powerctl11:20:220
1144499172,11cyclictest10766-21apt-get09:35:103
11444991711,4cyclictest9457-21pluginstate09:30:263
11444991711,4cyclictest3181-21strings11:45:183
11442991711,4cyclictest19590-21cut08:40:152
11442991711,4cyclictest10883-21cat09:35:122
1144299171,12cyclictest5633-21snmp_o5powerctl11:50:222
1144299171,12cyclictest18218-21snmp_o5powerctl09:50:222
1144299171,12cyclictest14211-21snmp_o5powerctl12:10:222
11442991710,5cyclictest2377-21sed10:30:162
1143799173,7cyclictest17205-21apt-get09:50:121
1143799171,12cyclictest19207-21apt-get08:40:121
1143799171,12cyclictest13739-21runrttasks12:10:171
1143799171,11cyclictest12791-21apt-get08:25:151
11437991710,5cyclictest8791-21expr10:45:151
11437991710,5cyclictest11899-21grep07:10:141
1143399179,6cyclictest30311-21dpkg09:05:110
1143399178,6cyclictest21590-21grep10:00:160
11433991713,2cyclictest7185-21sendmail_mailqu10:40:220
11433991713,2cyclictest7185-21sendmail_mailqu10:40:210
11433991713,2cyclictest15280-21latency_hist12:15:000
11433991712,3cyclictest16928-21if_err_enp2s007:20:160
11433991711,4cyclictest4368-21expr08:05:130
11433991711,4cyclictest13692-21ntpq09:40:180
11433991711,4cyclictest13692-21ntpq09:40:180
1143399171,12cyclictest28665-21snmp_o5powerctl07:45:230
11433991710,5cyclictest9-21ksoftirqd/009:45:150
11433991710,5cyclictest22689-21cut12:30:190
11433991710,5cyclictest16252-21awk08:30:250
11433991710,5cyclictest11545-21snmp_o5powerctl10:50:230
11433991710,5cyclictest11498-21ls12:05:140
1144299161,12cyclictest29446-21snmp_o5powerctl11:30:222
1144299161,11cyclictest17487-21apt-get12:20:122
1144299161,11cyclictest17487-21apt-get12:20:122
11442991610,4cyclictest11815-21cut07:10:132
11433991610,4cyclictest9-21ksoftirqd/009:25:190
1143399160,15cyclictest0-21swapper/009:23:550
1143399159,3cyclictest17473-21sed08:35:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional