You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-06-16 - 23:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems

Intel(R) Atom(TM) CPU E3825 @ 1.33GHz, Linux 4.19.37-rt20 (Profile)

Latency plot of shadow in rack #b, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackbslot6s.osadl.org (updated Thu Jun 10, 2021 12:43:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
85891930,4ptp4l241ktimersoftd/110:49:291
85891890,1ptp4l241ktimersoftd/113:58:101
324182890,3sleep10-21swapper/114:17:551
85891860,2ptp4l241ktimersoftd/113:15:221
85891860,1ptp4l241ktimersoftd/112:56:321
85891850,1ptp4l241ktimersoftd/113:50:181
85891850,1ptp4l241ktimersoftd/113:10:391
60399832,46cyclictest21240-21ssh15:19:170
60399822,49cyclictest21118-21ssh14:06:090
60399801,45cyclictest6221-21ssh15:38:540
60399801,45cyclictest410-21sh15:31:590
241292800,3sleep10-21swapper/114:46:221
241292800,3sleep10-21swapper/114:46:221
60399782,42cyclictest32041-21df13:41:390
85891770,1ptp4l241ktimersoftd/113:28:381
60399771,38cyclictest21224-21/usr/sbin/munin15:57:060
603997614,38cyclictest26151-21fschecks_time16:01:550
60399758,39cyclictest3414-21ssh16:11:590
241412750,3sleep00-21swapper/014:08:370
85891740,1ptp4l241ktimersoftd/113:40:431
60399744,43cyclictest26852-21apt-get13:36:220
60399744,43cyclictest26016-21apt-get14:11:390
60399742,39cyclictest21807-21snmp_o5powerctl12:51:580
60399742,39cyclictest12589-21snmp_o5powerctl14:32:060
60399733,43cyclictest10104-21hddtemp_smartct15:06:520
60399732,44cyclictest9412-21snmp_o5powerctl15:42:000
603997316,37cyclictest32244-21sh16:07:550
603997316,37cyclictest15754-21sh15:49:590
60399731,39cyclictest26139-21ssh12:56:580
60399723,38cyclictest2431-21apt-get14:21:410
60399701,41cyclictest30343-21seq14:16:450
60399698,36cyclictest29080-21ssh15:27:330
603996913,34cyclictest5365-21latency_hist11:01:110
603996912,36cyclictest12198-21sh13:18:260
60399682,42cyclictest4780-21memory13:46:570
60399682,40cyclictest9148-21memory13:51:530
60399681,52cyclictest5592-21ssh15:01:500
85891670,1ptp4l241ktimersoftd/116:04:411
60399677,35cyclictest23173-21ssh13:31:460
60399674,53cyclictest29219-21apt-get13:01:380
60399674,37cyclictest6566-21apt-get16:16:450
60399671,43cyclictest27967-21ssh14:50:380
60399671,43cyclictest27967-21ssh14:50:380
60399671,41cyclictest14317-21ssh13:21:440
603996712,35cyclictest4562-21sh15:01:110
60399662,41cyclictest720-21apt-get10:51:350
60399662,39cyclictest5523-21apt-get13:11:410
60399662,39cyclictest20197-21diskstats14:41:460
60399661,42cyclictest20282-21ssh13:27:200
60399661,40cyclictest2606-21ssh13:07:040
60399661,38cyclictest19596-21ssh15:53:580
603996611,41cyclictest13751-21runrttasks11:19:380
60399652,42cyclictest15818-21seq11:23:440
60399652,38cyclictest20058-21diskmemload14:28:360
60399649,36cyclictest30396-21runrttasks11:56:560
60399646,34cyclictest845-21runrttasks11:44:010
60399642,39cyclictest12636-21ssh13:56:310
60399642,38cyclictest15639-21/usr/sbin/munin14:36:410
60399642,37cyclictest19336-21python12:46:570
60399641,39cyclictest30880-21ssh14:52:590
60399641,39cyclictest13916-21/usr/sbin/munin15:11:470
60399641,39cyclictest13821-21/usr/sbin/munin12:36:370
60399641,39cyclictest12933-21snmp_o5powerctl12:31:550
60399632,37cyclictest2257-21python12:06:560
60399631,42cyclictest18085-21seq11:30:150
60399631,42cyclictest18085-21seq11:30:150
60399631,39cyclictest23346-21/usr/sbin/munin15:21:500
60399631,39cyclictest11366-21sed11:12:010
60399631,38cyclictest197-21systemd-journal12:45:160
85891620,1ptp4l241ktimersoftd/113:20:391
60399622,38cyclictest32555-21snmp_o5powerctl12:01:590
60399621,41cyclictest9037-21taskset12:22:260
60399611,41cyclictest197-21systemd-journal12:28:000
60399611,40cyclictest20094-21taskset11:32:270
60399602,35cyclictest21870-21snmp_o5powerctl11:36:580
60399601,40cyclictest7393-21seq11:04:580
60399601,40cyclictest22211-21snmpd11:08:240
60399601,37cyclictest3154-21/usr/sbin/munin12:11:540
60399601,37cyclictest24905-21/usr/sbin/munin11:46:490
60399601,36cyclictest5283-21/usr/sbin/munin12:16:400
60399601,36cyclictest5283-21/usr/sbin/munin12:16:400
60399575,42cyclictest27112-21apt-get11:51:390
85891560,3ptp4l31682-21kworker/1:112:55:411
604995619,14cyclictest15431-21runrttasks12:37:321
85891540,1ptp4l241ktimersoftd/116:15:221
604995413,16cyclictest15598-21ssh15:49:411
85891530,1ptp4l241ktimersoftd/114:27:441
60499538,19cyclictest11549-21ssh13:55:021
60499535,23cyclictest35692irq/88-enp1s0-r14:36:251
604995315,15cyclictest7182-21ssh16:16:551
60499528,20cyclictest19208-21ssh14:02:431
604995216,15cyclictest13344-21ssh15:10:531
604995212,16cyclictest8668-21ssh15:05:211
60499517,17cyclictest12589-21snmp_o5powerctl14:32:051
85891500,1ptp4l241ktimersoftd/115:33:231
60499507,18cyclictest2353-21users14:57:071
60499506,22cyclictest19744-21ssh15:17:061
60499506,21cyclictest24375-21ssh14:09:031
604995025,15cyclictest14001-21apt-get15:11:421
85891490,1ptp4l241ktimersoftd/115:52:591
85891490,1ptp4l241ktimersoftd/115:30:001
85891490,1ptp4l241ktimersoftd/115:21:421
85891490,1ptp4l241ktimersoftd/114:45:211
85891490,1ptp4l241ktimersoftd/114:15:421
60499495,21cyclictest25775-21ssh13:34:191
60499493,20cyclictest4214-21ssh14:22:181
85891480,1ptp4l241ktimersoftd/115:41:331
85891480,1ptp4l241ktimersoftd/114:52:341
85891480,1ptp4l241ktimersoftd/113:42:161
85891480,1ptp4l241ktimersoftd/113:22:121
85891480,1ptp4l241ktimersoftd/113:03:391
60499484,21cyclictest31443-21ssh16:07:021
604994819,21cyclictest3787-21diskstats15:36:471
604994813,13cyclictest19747-21snmp_o5powerctl11:32:031
85891470,1ptp4l241ktimersoftd/112:21:131
85891470,1ptp4l241ktimersoftd/111:06:331
85891470,1ptp4l241ktimersoftd/110:57:421
60499479,17cyclictest24857-21ssh16:00:321
604994711,14cyclictest19597-21awk12:47:061
60499468,17cyclictest4400-21snmp_o5powerctl12:12:001
604994610,15cyclictest6802-21snmp_o5powerctl11:01:581
60499458,15cyclictest11962-21/usr/sbin/munin11:17:021
60499446,17cyclictest16228-21/usr/sbin/munin11:27:051
60499446,17cyclictest16228-21/usr/sbin/munin11:27:051
60499437,16cyclictest28673-21sshd11:52:541
604994324,12cyclictest17219-21snmp_o5powerctl12:41:591
85891420,3ptp4l31682-21kworker/1:112:33:361
60499426,16cyclictest26544-21seq11:47:541
85891390,1ptp4l241ktimersoftd/112:02:071
604993922,9cyclictest9822-21/usr/sbin/munin11:12:031
604993912,21cyclictest21782-21find11:36:511
604993821,10cyclictest2260-21snmp_o5powerctl12:06:571
604993720,10cyclictest24020-21snmp_o5powerctl11:41:581
604993710,22cyclictest29162-21/usr/sbin/munin11:56:561
604993620,9cyclictest6527-21python12:17:021
604993620,9cyclictest6527-21python12:17:021
604993617,12cyclictest10053-21hddtemp_smartct12:26:441
156302350,5sleep1599-21cyclictest11:22:061
604993315,12cyclictest720-21apt-get10:51:351
3217123314,8sleep09-21ksoftirqd/010:46:580
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional