You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-22 - 07:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot6s.osadl.org (updated Fri Sep 22, 2023 00:43:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32725911350,3ptp4l241ktimersoftd/123:02:511
106732940,5sleep010668-21cron00:24:230
3272591850,1ptp4l241ktimersoftd/101:30:331
8208997118,15cyclictest4377-21ssh02:41:371
8208997110,16cyclictest12078-21diskstats02:09:471
8208997015,17cyclictest1931-21sh03:21:361
8208997014,19cyclictest27138-21apt-get02:29:451
8208996811,20cyclictest13338-21apt-get04:19:481
8208996715,16cyclictest18738-21sh02:18:221
820899668,17cyclictest18533-21ssh03:43:461
820899668,17cyclictest18533-21ssh03:43:461
8208996615,17cyclictest26370-21seq03:10:301
8208996612,16cyclictest11884-21/usr/sbin/munin23:09:531
8208996611,18cyclictest9600-21apt-get02:49:451
8208996610,19cyclictest12717-21snmp_o5powerctl03:35:091
820899658,21cyclictest28292-21snmp_o5powerctl03:55:121
8208996517,15cyclictest23386-21apt-get02:24:461
8208996514,18cyclictest28556-21awk23:49:481
8208996514,17cyclictest13323-21sh01:27:231
8208996512,16cyclictest18525-21ptp4l-jitter00:39:571
8208996511,17cyclictest18097-21cut23:24:501
8199996529,4cyclictest9-21ksoftirqd/001:58:580
820899648,16cyclictest20953-21apt-get04:29:461
820899646,17cyclictest15292-21cut23:19:221
8208996415,14cyclictest30744-21/usr/sbin/munin02:35:011
8208996414,16cyclictest13243-21sh02:54:371
8208996414,15cyclictest10167-21sh03:32:251
8208996412,18cyclictest23711-21cat03:49:571
8208996412,16cyclictest25784-21cat23:44:231
8208996412,15cyclictest11300-21rm02:08:481
8208996411,17cyclictest30621-21/usr/sbin/munin04:00:161
8208996411,16cyclictest9921-21rm01:23:041
8208996411,15cyclictest8649-21fschecks_count23:04:431
8208996411,15cyclictest6089-21sh02:44:471
8208996411,15cyclictest24130-21ssh03:09:001
8208996410,16cyclictest5221-21rm01:15:481
820899639,16cyclictest27126-21wc01:00:041
820899639,15cyclictest17260-21apt-get04:24:491
820899638,16cyclictest29662-21snmp_o5powerctl03:15:071
820899637,17cyclictest2355-21diskstats04:04:531
8208996315,15cyclictest12098-21sh04:17:221
8208996314,12cyclictest16582-21sed00:35:091
8208996312,16cyclictest5235-21taskset03:25:261
8208996312,16cyclictest5235-21taskset03:25:261
8208996311,16cyclictest21922-21cat01:39:441
8208996311,15cyclictest7243-21wc04:10:161
8208996310,16cyclictest6786-21taskset02:01:581
8208996310,16cyclictest31627-21switchtime23:55:061
8208996310,16cyclictest27679-21taskset23:47:451
8208996310,15cyclictest19373-21rm03:01:341
8199996330,8cyclictest9-21ksoftirqd/001:25:290
8199996329,6cyclictest9-21ksoftirqd/004:14:110
8199996327,4cyclictest9-21ksoftirqd/003:21:050
8199996326,5cyclictest9-21ksoftirqd/004:16:560
8199996326,5cyclictest9-21ksoftirqd/003:08:450
820899629,15cyclictest10679-21latency_hist00:24:241
820899626,16cyclictest19338-21apt-config02:19:261
8208996214,15cyclictest1745-21taskset00:01:551
8208996210,16cyclictest19371-21cat03:44:481
8199996230,5cyclictest9-21ksoftirqd/004:22:520
8199996230,13cyclictest9-21ksoftirqd/003:39:050
8199996227,4cyclictest9-21ksoftirqd/004:25:330
8199996227,4cyclictest9-21ksoftirqd/003:16:300
8199996227,4cyclictest9-21ksoftirqd/002:41:560
8199996227,4cyclictest9-21ksoftirqd/002:15:420
820899619,16cyclictest6612-21acpi00:14:381
820899618,16cyclictest30732-21snmp_o5powerctl01:50:071
820899615,17cyclictest13083-21ls00:29:411
820899615,16cyclictest21569-21cut23:34:221
8208996113,17cyclictest28121-21unattended-upgr01:04:401
8208996111,16cyclictest28073-21sh01:46:521
8199996130,9cyclictest9-21ksoftirqd/001:23:380
8199996129,9cyclictest9-21ksoftirqd/001:50:130
8199996127,5cyclictest9-21ksoftirqd/003:39:410
8199996127,5cyclictest9-21ksoftirqd/003:39:410
8199996127,4cyclictest9-21ksoftirqd/002:05:280
8199996127,4cyclictest9-21ksoftirqd/001:07:550
8199996125,4cyclictest9-21ksoftirqd/002:53:370
820899609,17cyclictest2532-21runrttasks01:13:221
820899608,17cyclictest21374-21ssh01:38:501
8199996031,8cyclictest9-21ksoftirqd/002:57:200
8199996030,5cyclictest9-21ksoftirqd/001:15:190
8199996024,4cyclictest9-21ksoftirqd/004:30:570
8199996024,4cyclictest9-21ksoftirqd/003:29:530
8199996023,4cyclictest9-21ksoftirqd/002:09:280
820899599,16cyclictest23736-21taskset00:54:261
820899598,12cyclictest20684-21systemctl00:44:581
8199995928,10cyclictest9-21ksoftirqd/001:30:290
8199995927,4cyclictest9-21ksoftirqd/002:31:550
8199995926,10cyclictest9-21ksoftirqd/002:03:150
8199995925,4cyclictest9-21ksoftirqd/002:21:000
8199995924,5cyclictest9-21ksoftirqd/001:46:270
8199995921,6cyclictest9-21ksoftirqd/000:54:240
8199995920,4cyclictest9-21ksoftirqd/004:05:100
820899589,12cyclictest1515-21irqstats01:54:551
820899587,13cyclictest15473-21ls23:19:451
820899586,13cyclictest14520-21timedrift23:15:051
8208995812,11cyclictest22246-21/usr/sbin/munin23:34:501
8199995825,10cyclictest9-21ksoftirqd/003:02:110
8199995824,5cyclictest9-21ksoftirqd/003:45:380
8199995820,6cyclictest9-21ksoftirqd/002:24:590
8199995817,6cyclictest9-21ksoftirqd/003:54:490
8208995710,15cyclictest12643-21runrttasks00:27:541
8199995730,5cyclictest9-21ksoftirqd/003:12:010
8199995721,5cyclictest9-21ksoftirqd/000:04:560
8199995720,6cyclictest9-21ksoftirqd/001:39:580
8199995719,6cyclictest9-21ksoftirqd/000:11:490
8199995622,4cyclictest9-21ksoftirqd/002:38:030
8199995619,5cyclictest9-21ksoftirqd/000:35:090
8199995619,5cyclictest9-21ksoftirqd/000:29:530
8199995618,6cyclictest9-21ksoftirqd/023:19:520
8199995618,6cyclictest9-21ksoftirqd/000:54:550
8199995618,5cyclictest9-21ksoftirqd/003:59:570
8199995617,6cyclictest9-21ksoftirqd/023:49:500
8199995522,10cyclictest9-21ksoftirqd/002:48:450
8199995520,5cyclictest9-21ksoftirqd/000:14:450
8199995519,6cyclictest9-21ksoftirqd/023:39:450
8199995519,6cyclictest9-21ksoftirqd/023:29:500
8199995517,6cyclictest9-21ksoftirqd/003:24:470
8199995517,6cyclictest9-21ksoftirqd/003:24:470
8199995429,4cyclictest9-21ksoftirqd/023:45:360
8199995420,4cyclictest9-21ksoftirqd/003:50:550
8199995417,6cyclictest9-21ksoftirqd/023:54:260
8199995417,6cyclictest9-21ksoftirqd/023:14:490
8199995417,6cyclictest9-21ksoftirqd/023:14:490
8199995417,6cyclictest9-21ksoftirqd/023:06:040
8199995415,6cyclictest9-21ksoftirqd/001:09:460
8199995415,6cyclictest9-21ksoftirqd/000:39:580
820899538,15cyclictest6219-21apt-get00:14:231
8199995317,6cyclictest9-21ksoftirqd/000:45:510
8199995317,5cyclictest9-21ksoftirqd/023:09:410
8208995212,12cyclictest23431-21chrt00:52:571
8199995117,10cyclictest9-21ksoftirqd/000:59:510
8199995115,6cyclictest9-21ksoftirqd/001:35:280
8199995030,5cyclictest9-21ksoftirqd/023:59:510
8199995010,6cyclictest9-21ksoftirqd/023:24:430
8199994930,5cyclictest9-21ksoftirqd/000:24:560
8208994814,24cyclictest2224-21apt-get00:04:421
8199994719,5cyclictest9-21ksoftirqd/023:34:430
819999435,4cyclictest9-21ksoftirqd/023:04:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional