You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-29 - 19:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Pentium N4200 @1100 MHz, Linux 5.9.1-rt18 (Profile)

Latency plot of shadow in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackcslot4s.osadl.org (updated Sun Jan 29, 2023 12:44:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1301099332,27cyclictest21539-21ls07:34:593
13010993211,19cyclictest13608-21ls08:45:203
1301099310,10cyclictest22138-21fschecks_time07:35:213
1301099301,16cyclictest14526-21sed10:25:233
13010993011,17cyclictest13737-21expr07:10:163
1301099297,6cyclictest31336-21grep11:15:293
1301099290,7cyclictest24930-21python307:45:013
1301099290,5cyclictest24916-21lsb_release07:45:003
1301099288,4cyclictest29745-21switchtime11:10:283
1301099282,7cyclictest30447-21apt-get08:00:173
1301099281,13cyclictest9091-21python311:50:003
1301099279,5cyclictest19374-21grep12:20:003
1301099272,8cyclictest29626-21munin-run09:35:013
1301099272,8cyclictest11148-21/usr/sbin/munin11:55:003
1301099271,8cyclictest592-21latency_hist09:45:003
1301099271,8cyclictest24524-21apt-config09:20:013
1301099271,7cyclictest9210-21apt-config08:33:313
1301099271,5cyclictest1141-21apt-get09:45:163
13010992711,13cyclictest2227-21mailstats08:10:263
1301099270,10cyclictest4301-21sh08:20:013
13007992715,7cyclictest11384-21sed08:40:001
1301099268,5cyclictest3963-21cat09:50:283
1301099267,4cyclictest19864-21tail12:20:203
1301099266,4cyclictest27885-21unixbench_singl09:25:383
1301099266,4cyclictest0-21swapper/310:15:003
1301099263,8cyclictest9913-21apt-get08:35:163
1301099262,6cyclictest16605-21munin-run07:20:013
1301099262,4cyclictest0-21swapper/312:30:163
1301099262,4cyclictest0-21swapper/312:05:183
1301099262,4cyclictest0-21swapper/312:00:173
1301099261,9cyclictest7552-21latency_hist08:30:003
1301099261,9cyclictest30224-21basename08:00:013
1301099261,9cyclictest25544-21sed12:35:303
1301099261,9cyclictest20043-21grep09:05:203
1301099261,9cyclictest20043-21grep09:05:203
1301099261,8cyclictest26769-21unixbench_singl11:00:323
1301099261,7cyclictest21584-21snmpd11:20:283
1301099261,7cyclictest18766-21ntpq09:00:243
1301099261,6cyclictest21584-21snmpd12:30:003
1301099261,6cyclictest21584-21snmpd12:30:003
1301099261,5cyclictest16183-21munin-run08:55:003
13010992612,12cyclictest19652-21unixbench-2d07:25:323
1301099261,11cyclictest5793-21turbostat.cron10:00:003
1301099261,11cyclictest12366-21snmp_o5powerctl11:55:323
1300899260,4cyclictest0-21swapper/209:20:482
1301099254,18cyclictest0-21swapper/311:25:333
1301099251,9cyclictest5279-21wc11:35:283
1301099251,8cyclictest13484-21awk10:20:253
1301099251,7cyclictest18593-21unixbench_multi10:35:303
1301099251,5cyclictest27820-21ntpq11:05:263
1301099251,5cyclictest0-21swapper/310:40:283
1301099251,10cyclictest7884-21fschecks_count10:05:193
1301099250,9cyclictest6705-21sed10:00:213
1301099250,8cyclictest4846-21fschecks_time08:20:193
1301099250,11cyclictest17160-21idleruntime-cro10:35:003
1300899250,8cyclictest25573-40apt-check07:45:062
1301099248,4cyclictest16633-21fschecks_count08:55:233
1301099247,4cyclictest6401-21tail11:40:213
1301099243,7cyclictest29837-21apt-get09:35:173
1301099243,7cyclictest16237-21apt-get12:10:153
1301099243,5cyclictest0-21swapper/310:15:253
1301099242,6cyclictest24851-40apt-check09:20:073
1301099242,6cyclictest12840-21head08:40:293
1301099241,9cyclictest25178-21awk10:55:393
1301099241,4cyclictest992-21awk08:05:303
1301099241,4cyclictest27198-21apt-get07:50:143
13010992414,6cyclictest2806-21apt-get11:30:173
13007992417,3cyclictest9255-21apt-key11:50:001
13007992414,7cyclictest29598-21cron09:35:011
13007992413,7cyclictest24934-21gpgv07:45:011
13007992413,6cyclictest21535-21update-cracklib07:35:001
13007992413,5cyclictest31289-21latency_hist09:40:001
1301099231,8cyclictest18234-21munin-run07:24:593
1301099231,5cyclictest21338-21cpuspeed_turbos09:10:173
1301099231,3cyclictest0-21swapper/310:50:163
1301099230,7cyclictest20668-21sort10:45:153
1300899234,17cyclictest24110-21latency_hist12:35:002
13008992319,3cyclictest241rcuc/208:35:502
1300899231,4cyclictest0-21swapper/212:15:002
1300799232,15cyclictest28499-21lsb_release11:09:591
13007992314,6cyclictest28617-21grep07:55:001
13006992314,6cyclictest1343-21df_inode09:45:190
13006992313,7cyclictest1095-21sed08:10:000
1300899221,19cyclictest21878-21missed_timers09:10:232
1300799222,13cyclictest7644-21apt-get10:05:171
13007992213,5cyclictest9179-21gpgv08:33:311
1300799221,15cyclictest25734-21/usr/sbin/munin12:40:001
13006992214,6cyclictest20317-21users10:40:300
13006992214,6cyclictest18451-21apt07:25:190
1300899212,17cyclictest16009-21latency_hist12:10:002
13008992117,3cyclictest241rcuc/211:00:332
13008992113,6cyclictest6791-21ntp_states10:00:222
13008992113,6cyclictest30631-21df08:00:182
13008992113,6cyclictest25125-21awk10:55:382
13008992113,6cyclictest24337-21dpkg12:35:162
1300899211,17cyclictest31251-21lsb_release09:40:002
1300899210,18cyclictest19449-21latency_hist12:20:002
1300899210,18cyclictest1-21systemd09:20:002
1300899210,17cyclictest30110-21lsb_release11:15:002
1300799219,8cyclictest17125-21cron10:35:011
1300799212,14cyclictest16840-21apt-get07:20:151
1300799212,13cyclictest22880-21apt-get09:15:161
1300799212,12cyclictest24902-21apt-get07:45:131
1300799212,12cyclictest17850-21apt-get12:15:161
13007992114,4cyclictest632-21latency_hist09:45:001
1300799211,14cyclictest6160-21apt-get08:25:211
13006992114,5cyclictest23320-21sed07:39:590
13006992113,6cyclictest24808-21cat12:35:210
13006992112,6cyclictest17187-21cat10:35:010
1300899208,10cyclictest13191-21tr10:20:222
1300899201,7cyclictest25-21ksoftirqd/209:32:062
13008992017,2cyclictest241rcuc/209:02:212
1300899201,6cyclictest25-21ksoftirqd/211:25:312
1300899201,6cyclictest25-21ksoftirqd/211:25:002
13008992014,4cyclictest674-21apt-config09:44:592
1300899201,3cyclictest2665-1kworker/2:0H12:30:012
1300899201,3cyclictest2665-1kworker/2:0H12:30:012
13008992013,5cyclictest19662-21expr12:20:182
13008992013,5cyclictest19421-21grep10:40:202
13008992013,4cyclictest11158-21tune2fs10:15:182
13008992012,6cyclictest21050-21basename09:10:002
13008992012,6cyclictest21050-21basename09:10:002
13008992011,7cyclictest13494-21cat12:00:222
13008992011,7cyclictest11151-21lsb_release11:55:002
1300899200,4cyclictest9331-21dpkg11:50:002
1300899200,17cyclictest1132-40dpkg09:45:052
1300799209,7cyclictest5775-21cron10:00:001
1300799203,13cyclictest23125-21tail12:30:201
1300799203,10cyclictest28013-21lsb_release09:30:011
1300799202,13cyclictest21876-21apt-get07:35:191
1300799202,12cyclictest9258-21apt-get10:10:161
1300799202,12cyclictest15199-21apt-get07:15:141
1300799202,12cyclictest11362-21apt-get11:55:191
13007992012,6cyclictest24881-21sed10:55:371
13007992012,5cyclictest4092-21grep11:30:301
1300799201,17cyclictest30223-21lsb_release08:00:001
1300799201,13cyclictest4354-21latency_hist08:20:001
1300799201,13cyclictest11550-21apt-get08:40:171
13006992016,2cyclictest15831-21missed_timers07:15:200
13006992014,4cyclictest14354-21smbd12:02:080
13006992013,5cyclictest9640-21lsb_release08:33:460
13006992013,5cyclictest6353-21perl11:40:200
13006992013,5cyclictest4555-21grep08:20:160
13006992013,5cyclictest29996-21expr09:35:170
13006992013,5cyclictest2093-21grep08:10:240
13006992012,6cyclictest29538-21ntp_kernel_err07:55:250
13006992012,6cyclictest20492-21awk09:05:250
13006992012,6cyclictest20492-21awk09:05:250
13006992012,6cyclictest10470-21nfs4_client11:50:230
13006992012,5cyclictest14573-21sensors_temp07:10:240
13006992011,7cyclictest19416-21cron09:05:000
1300899192,15cyclictest15154-21cat07:15:012
1300899191,7cyclictest25-21ksoftirqd/207:20:242
13008991915,3cyclictest241rcuc/209:50:192
13008991915,2cyclictest0-21swapper/207:35:012
13008991912,5cyclictest7746-21cat10:05:162
13008991912,5cyclictest28034-21awk07:50:242
13008991912,5cyclictest26867-21uniq09:25:252
13008991911,6cyclictest6198-21expr11:40:182
13008991911,6cyclictest3596-21awk11:30:242
13008991911,6cyclictest18602-21ls07:25:202
13008991911,6cyclictest15966-21awk07:15:212
1300899191,16cyclictest0-21swapper/209:00:002
13008991911,3cyclictest25-21ksoftirqd/210:25:192
13008991910,6cyclictest6345-21df08:25:222
1300899190,7cyclictest25-21ksoftirqd/210:10:342
1300899190,2cyclictest24960-21apt-config07:45:012
1300799199,6cyclictest288-21systemd-journal12:30:001
1300799199,6cyclictest288-21systemd-journal12:30:001
1300799197,8cyclictest17363-21apt-get10:35:171
1300799192,10cyclictest5833-21lsb_release11:40:001
1300799192,10cyclictest4189-21lsb_release09:55:001
13007991912,5cyclictest20221-21grep09:05:221
13007991912,5cyclictest20221-21grep09:05:221
13007991911,6cyclictest4972-21grep08:20:191
1300799191,13cyclictest24869-21apt-key09:20:021
1300799191,13cyclictest22053-21sed10:50:011
1300799191,12cyclictest32064-21apt-get08:05:161
13006991915,2cyclictest28270-21cpu09:30:190
13006991915,2cyclictest13267-21ntp_offset10:20:230
13006991914,3cyclictest18346-21interrupts12:15:200
13006991914,3cyclictest15130-21snmp_o5powerctl10:25:310
13006991913,4cyclictest3050-21expr08:15:170
13006991913,4cyclictest15052-21ls12:05:210
13006991912,5cyclictest9550-21tune2fs10:10:190
13006991912,5cyclictest7016-21sensors_temp10:00:250
13006991912,5cyclictest6966-21cut08:25:280
13006991912,5cyclictest5718-21users09:55:330
13006991912,5cyclictest22234-21grep10:50:160
13006991912,5cyclictest18317-21grep10:35:260
13006991912,5cyclictest11203-21expr10:15:200
13006991911,6cyclictest6023-21ls11:40:010
13008991814,3cyclictest241rcuc/208:50:172
13008991814,2cyclictest21584-21snmpd11:57:562
13008991813,3cyclictest241rcuc/210:38:012
13008991812,4cyclictest30749-21ls11:15:212
13008991812,4cyclictest13611-21basename08:45:202
1300899181,15cyclictest5817-21latency_hist11:40:002
13008991811,5cyclictest5704-21sed08:20:282
13008991811,5cyclictest5581-21cat09:55:322
13008991811,5cyclictest32216-21cat08:05:172
13008991811,5cyclictest21005-21sendmail_mailqu07:30:272
13008991810,6cyclictest29248-21basename07:55:232
1300899180,7cyclictest25-21ksoftirqd/211:08:072
1300899180,7cyclictest25-21ksoftirqd/208:32:502
1300899180,6cyclictest25-21ksoftirqd/210:48:032
1300899180,3cyclictest4345-21sendmail-msp08:20:002
1300899180,13cyclictest22226-21apt-get10:50:172
1300799189,6cyclictest19107-21sshd09:00:291
1300799183,8cyclictest14983-21munin-run07:15:001
1300799182,13cyclictest30851-21tail08:00:201
1300799182,12cyclictest959-40apt-check09:45:091
13007991811,5cyclictest19542-21cat07:25:321
13007991811,5cyclictest1615-21gltestperf08:10:191
1300799181,13cyclictest23647-21sort10:55:011
1300799181,13cyclictest17637-21munin-run12:15:001
1300799181,12cyclictest7488-21latency_hist11:45:001
1300799181,12cyclictest6029-21apt-get10:00:161
1300799181,12cyclictest20916-21cut12:25:001
1300799181,11cyclictest31960-21apt-get11:20:191
1300799181,11cyclictest16401-21apt-get08:55:211
13007991810,6cyclictest20601-21sed10:45:011
1300699189,6cyclictest28886-21df11:10:170
13006991814,2cyclictest27510-21hwlatdetect11:05:220
13006991814,2cyclictest24591-21ntp_offset10:55:300
13006991813,3cyclictest3340-21latency11:30:220
13006991813,3cyclictest27013-21hwlatdetect09:25:260
13006991813,3cyclictest25072-21latency_hist07:45:000
13006991812,4cyclictest17058-21grep12:10:230
13006991812,4cyclictest13587-21cat08:45:210
13006991811,5cyclictest32430-21mailstats09:40:250
13006991811,5cyclictest25923-21cat11:00:210
13006991811,5cyclictest20036-21cat12:20:220
13006991811,5cyclictest12487-21readlink08:40:260
13006991811,5cyclictest12108-21ntpq11:55:280
13006991811,5cyclictest11155-21grep08:35:300
1300699181,13cyclictest24987-21apt-get09:20:160
1300699181,12cyclictest30318-21apt-get11:15:170
13006991810,6cyclictest26415-21awk07:45:250
13006991810,5cyclictest21076-21sensors_temp07:30:270
13008991713,2cyclictest15743-21/usr/sbin/munin10:30:212
13008991712,3cyclictest25-21ksoftirqd/208:10:202
1300799175,9cyclictest14388-21cron12:05:011
13007991712,3cyclictest15298-21timerwakeupswit10:25:331
1300799171,13cyclictest21955-21head09:10:231
1300799171,12cyclictest288-21systemd-journal10:25:001
1300799171,12cyclictest10870-21apt-get10:15:161
1300799171,11cyclictest26223-21ntp_kernel_pll_11:00:251
13007991710,5cyclictest10829-21wc11:50:291
1300799170,15cyclictest0-21swapper/112:05:291
1300699172,11cyclictest24331-21python309:20:010
13006991714,2cyclictest15250-21if_enp2s008:50:230
13006991713,2cyclictest21467-21gltestperf12:25:230
13006991713,2cyclictest21467-21gltestperf12:25:230
13006991712,3cyclictest2174-21mailstats11:25:320
13006991711,4cyclictest30623-21expr08:00:180
1300699171,12cyclictest522-21snmp_o5powerctl11:20:290
13006991710,5cyclictest8581-21readlink10:05:260
13006991710,5cyclictest21752-21cat09:10:210
13008991610,4cyclictest11791-21uniq08:40:192
1300799163,10cyclictest1270-21cpuspeed_turbos11:25:211
1300799162,11cyclictest30204-21wc11:15:001
13007991612,2cyclictest30315-21/usr/sbin/munin11:15:281
1300799161,11cyclictest13859-21head08:45:221
1300799161,10cyclictest15516-21ntpq08:50:241
13006991613,2cyclictest9140-21apt-config11:50:010
13006991613,2cyclictest27701-21irqstats07:50:190
13006991612,2cyclictest22763-21cpu12:30:180
13006991612,2cyclictest21325-21netstat10:45:210
13006991611,3cyclictest16605-21entropy08:55:230
13006991610,4cyclictest17460-21cat07:20:200
1300699159,4cyclictest3622-21proc_pri09:50:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional