You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-31 - 05:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Jan 31, 2023 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
313522165151,8sleep30-21swapper/319:05:203
324782157105,28sleep20-21swapper/219:07:362
323802156101,30sleep00-21swapper/019:06:220
324602154101,28sleep10-21swapper/119:07:221
131472750,3sleep30-21swapper/300:16:473
90912560,14sleep30-21swapper/323:40:233
90912560,14sleep30-21swapper/323:40:223
325782490,1sleep20-21swapper/200:03:262
238492480,2sleep00-21swapper/000:29:010
311642470,1sleep00-21swapper/021:20:250
509994413,17cyclictest20523-21sh22:47:162
121482430,4sleep334-21ksoftirqd/321:34:243
50999421,6cyclictest17211-21ssh23:48:032
509994212,17cyclictest27866-21sh23:26:012
506994223,17cyclictest19745-21diskmemload21:31:170
50699421,3cyclictest19745-21diskmemload23:57:200
50699411,2cyclictest0-21swapper/023:11:280
50999390,37cyclictest0-21swapper/221:15:202
50999390,37cyclictest0-21swapper/221:15:192
50999371,1cyclictest0-21swapper/222:53:422
50999370,1cyclictest0-21swapper/222:42:322
509993620,14cyclictest0-21swapper/223:32:162
50699362,9cyclictest10039-21snmpd23:45:170
506993619,14cyclictest28562-21ssh00:33:410
506993619,14cyclictest28562-21ssh00:33:400
506993613,4cyclictest6162-21sh21:27:550
50699361,32cyclictest0-21swapper/022:14:590
50699361,1cyclictest0-21swapper/021:19:560
50699361,1cyclictest0-21swapper/021:19:550
50699361,19cyclictest0-21swapper/023:41:080
50699361,19cyclictest0-21swapper/023:41:070
506993611,6cyclictest20719-21expr22:15:200
50699360,26cyclictest0-21swapper/000:06:160
509993513,19cyclictest9351-21systemd-journal00:20:002
50999350,1cyclictest0-21swapper/222:35:362
50699357,11cyclictest20599-21unixbench_multi23:50:340
50699351,1cyclictest0-21swapper/022:51:160
50699351,1cyclictest0-21swapper/000:01:380
50699348,6cyclictest3430-21kernelversion20:30:230
506993420,12cyclictest0-21swapper/022:46:550
50699341,31cyclictest0-21swapper/021:10:330
50699341,30cyclictest19745-21diskmemload21:51:180
506993412,3cyclictest2170-21sh23:00:520
50699341,1cyclictest0-21swapper/023:25:060
506993410,5cyclictest10317-21seq22:05:040
506993410,5cyclictest10317-21seq22:05:030
50699340,1cyclictest0-21swapper/023:34:340
509993325,6cyclictest1914-1kworker/u9:121:24:472
50999331,30cyclictest0-21swapper/221:26:432
50999331,1cyclictest0-21swapper/221:51:462
50799330,30cyclictest5182-21ssh21:26:211
50699331,30cyclictest0-21swapper/022:55:410
50699331,1cyclictest0-21swapper/022:23:560
509993213,16cyclictest4613-21sh23:04:332
50999321,28cyclictest20528-21sh23:20:022
50799321,28cyclictest18553-21runrttasks23:17:311
50799320,29cyclictest5627-21ssh00:09:511
506993223,7cyclictest13802-21cstates20:55:190
506993223,7cyclictest0-21swapper/022:35:370
326412320,1chrt0-21swapper/022:26:390
51099313,25cyclictest2652-21apt-config20:30:013
50999319,19cyclictest10764-21awk20:45:312
509993122,7cyclictest0-21swapper/223:39:072
50999311,23cyclictest0-21swapper/222:26:142
50999311,1cyclictest0-21swapper/222:58:102
50999311,1cyclictest0-21swapper/222:10:142
509993110,18cyclictest15936-21expr21:00:182
50799311,28cyclictest0-21swapper/122:29:231
50799310,28cyclictest1929-21sh23:33:101
50799310,28cyclictest10437-21taskset22:36:581
50699312,9cyclictest55150irq/140-enp3s0-00:10:180
510993022,5cyclictest27067-21ssh00:31:283
510993022,5cyclictest27067-21ssh00:31:273
510993022,5cyclictest19745-21diskmemload23:32:053
51099301,27cyclictest19745-21diskmemload21:28:563
51099300,28cyclictest1531-21awk23:00:293
51099300,27cyclictest9791-21ssh00:14:043
51099300,27cyclictest8838-21chrt23:08:203
50999308,5cyclictest24072-21seq23:55:122
50999308,19cyclictest13058-21cat21:35:192
50999306,6cyclictest28691-21grep20:15:192
50999305,5cyclictest13617-21grep19:40:182
509993023,4cyclictest22014-21ssh23:52:342
509993019,9cyclictest55450irq/143-enp3s0-00:06:152
509993012,15cyclictest4647-21sh21:59:212
509993010,17cyclictest14212-21sh22:08:452
509993010,17cyclictest14212-21sh22:08:442
50999300,28cyclictest0-21swapper/221:33:382
50999300,1cyclictest23059-21ssh22:17:072
507993023,5cyclictest11433-21sh23:42:031
507993023,5cyclictest11433-21sh23:42:021
50799301,26cyclictest26353-21ssh22:53:251
50699302,7cyclictest11336-21apt-get19:35:170
50699302,4cyclictest22605-21apt-get21:45:040
506993023,5cyclictest19745-21diskmemload23:20:110
50699301,9cyclictest18024-21apt-get21:40:190
50699301,11cyclictest7035-21apt-get19:25:170
51099290,26cyclictest19745-21diskmemload22:22:363
51099290,26cyclictest15477-21ssh00:20:163
50999299,18cyclictest8215-21cron23:44:552
50999299,18cyclictest8215-21cron23:44:552
50999299,15cyclictest55450irq/143-enp3s0-21:10:042
50999297,5cyclictest9082-21cut19:30:012
50999297,4cyclictest1293-21strings20:25:242
509992922,5cyclictest0-21swapper/200:27:042
509992922,5cyclictest0-21swapper/200:24:442
509992920,7cyclictest0-21swapper/200:11:032
509992919,8cyclictest55450irq/143-enp3s0-23:11:282
509992917,10cyclictest0-21swapper/221:49:402
50999291,26cyclictest0-21swapper/222:21:402
50799291,3cyclictest31947-21ssh21:54:031
50799291,3cyclictest19745-21diskmemload22:33:201
50799291,26cyclictest27808-21ssh22:21:511
50799291,26cyclictest0-21swapper/123:54:061
50799291,26cyclictest0-21swapper/120:15:311
50799291,25cyclictest3077-21ssh21:56:501
506992921,6cyclictest0-21swapper/000:36:090
506992920,7cyclictest0-21swapper/000:17:520
506992918,9cyclictest0-21swapper/000:22:140
50699291,3cyclictest5656-21expr22:00:190
50699291,3cyclictest15615-21sh21:37:030
50699291,25cyclictest0-21swapper/023:35:170
50699291,25cyclictest0-21swapper/023:17:310
510992822,4cyclictest9592-21ssh22:35:413
510992822,3cyclictest18024-21apt-get21:40:213
51099281,4cyclictest19745-21diskmemload21:52:353
51099281,24cyclictest25175-21ssh22:51:353
51099281,24cyclictest19131-21ssh22:13:443
51099281,1cyclictest0-21swapper/323:57:523
50999288,6cyclictest19623-21expr21:40:402
50999285,6cyclictest6644-21cat23:05:302
509992823,3cyclictest19300-21ssh23:18:362
50999282,2cyclictest0-21swapper/222:00:172
50799281,3cyclictest7959-21ssh23:39:341
50799281,3cyclictest667-21sh21:21:561
50799281,3cyclictest27326-21ntp_states20:10:301
50799281,2cyclictest25801-21/usr/sbin/munin21:15:201
50799281,2cyclictest25801-21/usr/sbin/munin21:15:191
50799281,2cyclictest0-21swapper/100:11:011
50799281,26cyclictest0-21swapper/123:48:381
50799281,25cyclictest1310-21ssh00:39:521
50799281,25cyclictest0-21swapper/123:27:081
50799281,25cyclictest0-21swapper/123:03:061
50799281,25cyclictest0-21swapper/122:00:191
50799281,25cyclictest0-21swapper/121:31:481
50799281,25cyclictest0-21swapper/100:21:321
50799280,3cyclictest30402-21sh22:57:021
506992823,3cyclictest2345-21ssh21:55:430
50699281,25cyclictest0-21swapper/022:42:000
510992723,2cyclictest24730-21apt-get21:15:013
510992723,2cyclictest24730-21apt-get21:15:013
510992721,3cyclictest20858-21ssh23:50:483
51099271,25cyclictest0-21swapper/322:17:523
51099271,24cyclictest734-21ssh22:27:283
51099271,24cyclictest5064-21python323:35:333
51099271,24cyclictest0-21swapper/321:56:413
51099271,24cyclictest0-21swapper/321:12:413
51099270,4cyclictest24954-21kworker/u8:222:59:083
50999279,15cyclictest5783-21ntp_kernel_pll_19:20:242
50999272,3cyclictest0-21swapper/200:32:542
50999272,3cyclictest0-21swapper/200:32:542
507992725,1cyclictest0-21swapper/122:13:401
50799271,2cyclictest20917-21seq22:47:541
50799271,2cyclictest0-21swapper/123:06:261
50799271,25cyclictest0-21swapper/122:09:291
50799271,25cyclictest0-21swapper/122:09:291
50799271,25cyclictest0-21swapper/121:49:411
50799271,24cyclictest0-21swapper/123:55:131
50799271,24cyclictest0-21swapper/123:22:461
50799271,24cyclictest0-21swapper/123:10:161
50799271,24cyclictest0-21swapper/122:40:511
50799271,24cyclictest0-21swapper/121:36:071
50799271,24cyclictest0-21swapper/121:06:081
50799271,24cyclictest0-21swapper/120:40:271
50799271,24cyclictest0-21swapper/100:15:231
50799271,1cyclictest0-21swapper/122:19:061
50799271,1cyclictest0-21swapper/121:40:561
50799271,1cyclictest0-21swapper/121:11:401
50799270,25cyclictest0-21swapper/100:01:541
50699271,24cyclictest0-21swapper/023:08:530
510992622,2cyclictest26197-21ssh21:47:403
510992621,3cyclictest13751-21ssh22:08:013
510992621,3cyclictest13751-21ssh22:08:003
51099261,24cyclictest0-21swapper/323:17:233
51099261,24cyclictest0-21swapper/321:23:363
51099261,24cyclictest0-21swapper/300:38:043
51099261,23cyclictest0-21swapper/323:45:063
51099261,23cyclictest0-21swapper/323:25:123
51099261,23cyclictest0-21swapper/323:10:243
51099261,23cyclictest0-21swapper/322:40:013
51099261,23cyclictest0-21swapper/322:32:403
51099261,23cyclictest0-21swapper/322:01:183
51099261,23cyclictest0-21swapper/300:05:243
51099261,23cyclictest0-21swapper/300:01:043
51099261,1cyclictest0-21swapper/323:20:453
51099261,1cyclictest0-21swapper/321:39:403
51099261,1cyclictest0-21swapper/320:22:543
50999269,14cyclictest19740-21latency_hist21:10:002
50999267,5cyclictest3905-21sendmail_mailtr20:30:282
50999267,3cyclictest42942chrt19:16:102
50999267,3cyclictest4142-21grep22:30:312
50999266,5cyclictest13318-21cat20:55:012
509992623,2cyclictest0-21swapper/200:37:192
50999261,6cyclictest30841-21grep20:20:242
50999260,7cyclictest18552-21/usr/sbin/munin19:50:222
507992624,1cyclictest0-21swapper/100:30:161
507992624,1cyclictest0-21swapper/100:30:151
507992623,2cyclictest0-21swapper/100:28:341
50799261,1cyclictest0-21swapper/120:39:061
50699261,8cyclictest16215-21tail21:00:200
50699261,24cyclictest0-21swapper/022:30:590
50699261,24cyclictest0-21swapper/020:22:300
51099251,23cyclictest0-21swapper/322:45:433
51099251,22cyclictest0-21swapper/300:28:013
50999251,6cyclictest7685-21ls20:40:262
50999251,6cyclictest20051-21grep19:55:172
50999251,5cyclictest13547-21/usr/sbin/munin20:55:162
506992523,1cyclictest0-21swapper/021:06:090
506992523,1cyclictest0-21swapper/020:25:280
510992422,1cyclictest0-21swapper/321:00:283
510992422,1cyclictest0-21swapper/320:40:353
50999243,5cyclictest23176-21processes20:00:262
50999241,7cyclictest9561-21fschecks_count19:30:212
50999241,7cyclictest9561-21fschecks_count19:30:202
50999241,6cyclictest5050-21grep20:35:182
50999241,6cyclictest26552-21cpuspeed_turbos20:10:212
50999241,5cyclictest15777-21grep19:45:182
50999240,8cyclictest24742-21tail20:05:212
50999240,7cyclictest1081-21head19:10:172
507992422,1cyclictest0-21swapper/120:45:321
506992422,1cyclictest0-21swapper/020:45:310
50999231,6cyclictest11580-21expr19:35:182
507992119,1cyclictest0-21swapper/120:25:291
507992110,8cyclictest9510-21df_inode19:30:191
507992110,8cyclictest9510-21df_inode19:30:191
506992113,6cyclictest17867-21ls19:50:160
506992112,6cyclictest9351-21systemd-journal19:45:000
507992012,6cyclictest4209-21awk19:15:291
507992011,6cyclictest9351-21systemd-journal19:39:591
507992011,6cyclictest3112-21diskstats20:30:181
50699209,7cyclictest15650-21apt-get19:45:170
506992012,6cyclictest10015-21needreboot19:30:250
506992012,6cyclictest10015-21needreboot19:30:240
510991917,1cyclictest0-21swapper/321:05:333
510991917,1cyclictest0-21swapper/320:45:213
510991911,6cyclictest18542-21load19:50:233
510991911,6cyclictest10993-21seq19:34:063
510991911,6cyclictest10993-21seq19:34:053
510991910,6cyclictest2773-21awk19:15:163
507991911,6cyclictest31922-21cat20:20:421
507991911,6cyclictest19933-21apt19:55:141
506991917,1cyclictest0-21swapper/020:15:320
506991912,5cyclictest4024-21cat19:15:280
506991911,6cyclictest9113-21cut20:45:010
51099189,6cyclictest11996-21cat20:50:223
510991816,1cyclictest0-21swapper/320:25:383
510991813,3cyclictest1484-21ntp_states19:10:233
510991812,3cyclictest21067-21sendmail_mailtr19:55:263
510991811,5cyclictest30016-21grep20:15:313
510991811,5cyclictest16184-21sed19:45:223
510991811,5cyclictest12153-21cat19:35:243
510991810,6cyclictest22445-21diskstats20:00:203
510991810,5cyclictest26993-21basename20:10:253
50799189,7cyclictest1541-21cut19:10:241
507991813,3cyclictest12067-21kernelversion20:50:241
507991811,5cyclictest23023-21cat20:00:261
507991810,6cyclictest15908-21cat21:00:181
507991810,6cyclictest13800-21cat20:55:191
50699188,7cyclictest5470-21cut20:35:210
506991816,1cyclictest0-21swapper/020:10:300
510991712,3cyclictest24225-21/usr/sbin/munin20:05:263
510991710,5cyclictest15311-21seq19:44:093
510991710,5cyclictest15056-21uptime20:55:323
50799179,6cyclictest5542-21uname19:20:231
50799175,10cyclictest19692-21awk19:55:011
507991711,4cyclictest24962-21latency20:05:221
50699179,5cyclictest20101-21cat19:55:170
506991712,3cyclictest5888-21awk19:20:270
506991710,5cyclictest25595-21sed20:05:300
506991710,5cyclictest22273-21cat20:00:180
51099169,5cyclictest5089-21cut20:35:193
51099161,6cyclictest6937-21cat19:25:013
510991611,3cyclictest641-21runrttasks19:25:173
50799169,5cyclictest16954-21cat19:45:291
50799162,10cyclictest13509-21apt-get19:40:191
507991610,4cyclictest7559-21expr19:25:191
50699163,8cyclictest11397-21apt-get20:50:180
50699162,10cyclictest555-21apt-get19:10:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional