You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 17:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of shadow in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Mon Mar 20, 2023 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2544023624,0sleep10-21swapper/106:56:081
2526623624,0sleep20-21swapper/206:54:092
295750350,0irq/25-eth10-21swapper/306:51:413
2519422723,0sleep00-21swapper/006:53:190
255369983,0cyclictest28022-21awk07:46:471
255369980,0cyclictest0-21swapper/109:23:111
255369973,0cyclictest32690-21grep11:11:441
255369973,0cyclictest21498-21awk11:26:401
255369973,0cyclictest14812-21awk07:36:401
339960,0migration/38710-21sh10:58:513
339960,0migration/36367-21sh10:38:073
339960,0migration/34157-21sh11:34:203
339960,0migration/33872-21sh10:17:033
339960,0migration/332507-21sh09:37:433
339960,0migration/322883-21taskset10:30:573
339960,0migration/322278-21sh09:13:073
339960,0migration/31687-21munin-node09:56:513
28855060,0irq/27-eth0-tx-601-21munin-node07:26:490
28855060,0irq/27-eth0-tx-27166-21munin-node07:46:530
28855060,0irq/27-eth0-tx-19860-21munin-node07:16:400
28845060,0irq/26-eth0-rx-4931-21sed11:51:573
28845060,0irq/26-eth0-rx-28302-21pmu-power08:56:473
28845060,0irq/26-eth0-rx-16768-21ssh11:02:483
28845060,0irq/26-eth0-rx-14760-21ssh09:10:143
28845060,0irq/26-eth0-rx-12373-21sh10:06:093
255369963,0cyclictest3469-21lldpd11:06:271
255369962,0cyclictest5451-21expr10:36:511
255369962,0cyclictest3469-21lldpd07:22:261
255369962,0cyclictest24390-21sh09:33:371
255369962,0cyclictest22267-21awk10:11:411
255369962,0cyclictest15308-21date08:26:241
255369961,0cyclictest907-21munin-node12:11:581
255369961,0cyclictest632-21kworker/1:211:41:391
255369961,0cyclictest31780-21sh10:52:301
255369961,0cyclictest3079-21needreboot10:16:491
255369961,0cyclictest18915-21gltestperf12:21:451
255369961,0cyclictest11200-21awk11:01:421
255369961,0cyclictest0-21swapper/107:01:411
255369960,0cyclictest9617-21ssh11:00:471
255369960,0cyclictest9548-21aten2_r1power_v08:46:361
255369960,0cyclictest9541-21munin-node08:21:391
255369960,0cyclictest8780-21tune2fs10:41:411
255369960,0cyclictest8755-21grep09:06:441
255369960,0cyclictest8337-21df07:56:361
255369960,0cyclictest7186-21sed07:31:241
255369960,0cyclictest3427-21snmpd07:44:211
255369960,0cyclictest3298-21cut07:51:411
255369960,0cyclictest25452-21idleruntime12:06:411
255369960,0cyclictest24876-21sh09:16:441
255369960,0cyclictest2456-21munin-node11:31:531
255369960,0cyclictest24263-21kthreadcore09:51:441
255369960,0cyclictest23892-21latency_hist09:16:241
255369960,0cyclictest22441-21diskmemload10:48:101
255369960,0cyclictest2210-21ssh09:41:361
255369960,0cyclictest21541-21sh10:28:051
255369960,0cyclictest20807-21sh07:16:411
255369960,0cyclictest17424-21sh09:47:541
255369960,0cyclictest13582-21diskstats07:11:381
255369960,0cyclictest13541-21sh10:06:401
255369960,0cyclictest13199-21ssh11:40:161
255369960,0cyclictest12477-21kthreadcore11:56:511
255369960,0cyclictest0-21swapper/106:56:441
255349963,0cyclictest0-21swapper/011:56:240
149960,0migration/04339-21sh11:34:430
149960,0migration/030982-21sh12:07:110
149960,0migration/016151-21sh09:29:140
8225050,0irq/24-0000:00:9926-21latency_hist11:01:242
8225050,0irq/24-0000:00:7826-21crond07:56:242
8225050,0irq/24-0000:00:30723-21munin-run09:01:242
339950,0migration/39902-21fschecks_count08:46:383
339950,0migration/39385-21smart_sda11:16:583
339950,0migration/38581-21taskset09:42:233
339950,0migration/34569-21kthreadcore07:26:423
339950,0migration/34216-21kthreadcore07:51:413
339950,0migration/328923-21kthreadcore07:46:493
339950,0migration/326541-21taskset07:42:023
339950,0migration/326511-21load08:06:483
339950,0migration/320966-21kthreadcore09:51:433
339950,0migration/320727-21aten2_r1power_v11:26:363
339950,0migration/318917-21kthreadcore08:26:423
339950,0migration/317902-21fschecks_count09:31:393
339950,0migration/314980-21taskset08:25:053
339950,0migration/314742-21runrttasks11:59:113
339950,0migration/313557-21munin-node10:06:413
339950,0migration/313544-21kthreadcore09:26:483
28855050,0irq/27-eth0-tx-21961-21munin-node08:31:530
28855050,0irq/27-eth0-tx-21285-21ssh10:11:272
28855050,0irq/27-eth0-tx-18588-21irqcore09:31:422
28855050,0irq/27-eth0-tx-0-21swapper/211:11:172
28855050,0irq/27-eth0-tx-0-21swapper/210:38:422
28855050,0irq/27-eth0-tx-0-21swapper/210:03:052
28855050,0irq/27-eth0-tx-0-21swapper/209:31:122
28855050,0irq/27-eth0-tx-0-21swapper/008:49:320
28845050,0irq/26-eth0-rx-9233-21kthreadcore11:36:453
28845050,0irq/26-eth0-rx-28095-21hostname11:11:403
28845050,0irq/26-eth0-rx-24255-21egrep09:16:413
28845050,0irq/26-eth0-rx-23406-21hostname11:46:433
28845050,0irq/26-eth0-rx-16004-21cut08:01:423
28845050,0irq/26-eth0-rx-15778-21cut11:41:433
28845050,0irq/26-eth0-rx-0-21swapper/312:22:263
28845050,0irq/26-eth0-rx-0-21swapper/312:12:353
28845050,0irq/26-eth0-rx-0-21swapper/312:01:443
28845050,0irq/26-eth0-rx-0-21swapper/311:25:443
28845050,0irq/26-eth0-rx-0-21swapper/311:07:503
28845050,0irq/26-eth0-rx-0-21swapper/310:51:343
28845050,0irq/26-eth0-rx-0-21swapper/310:46:513
28845050,0irq/26-eth0-rx-0-21swapper/310:43:073
28845050,0irq/26-eth0-rx-0-21swapper/310:31:573
28845050,0irq/26-eth0-rx-0-21swapper/310:21:423
28845050,0irq/26-eth0-rx-0-21swapper/310:12:523
28845050,0irq/26-eth0-rx-0-21swapper/309:24:483
28845050,0irq/26-eth0-rx-0-21swapper/309:02:313
255379955,0cyclictest0-21swapper/211:32:332
255379955,0cyclictest0-21swapper/209:13:582
255379954,0cyclictest3427-21snmpd08:26:152
255379950,0cyclictest0-21swapper/211:18:222
255379950,0cyclictest0-21swapper/209:22:582
255379950,0cyclictest0-21swapper/209:06:242
255369951,0cyclictest28305-21latency_hist08:36:241
255369950,0cyclictest9433-21cat12:16:241
255369950,0cyclictest8784-21latency_hist09:26:241
255369950,0cyclictest7325-21taskset07:31:331
255369950,0cyclictest6548-21cat07:06:241
255369950,0cyclictest6085-21latency_hist10:21:241
255369950,0cyclictest4083-21cat10:01:241
255369950,0cyclictest3407-21latency_hist11:16:241
255369950,0cyclictest31187-21ssh11:51:281
255369950,0cyclictest30950-21seq09:01:271
255369950,0cyclictest27843-21cat09:56:241
255369950,0cyclictest27808-21cat08:11:241
255369950,0cyclictest2682-21latency_hist08:41:241
255369950,0cyclictest25894-21sh09:36:241
255369950,0cyclictest23311-21sed10:31:241
255369950,0cyclictest22708-21ssh08:56:331
255369950,0cyclictest22602-21ssh11:46:321
255369950,0cyclictest21877-21sed08:31:241
255369950,0cyclictest21285-21munin-node08:06:381
255369950,0cyclictest2031-21taskset08:16:301
255369950,0cyclictest16037-21cat08:51:241
255369950,0cyclictest15935-21ls12:01:241
255369950,0cyclictest14603-21sed08:01:241
255369950,0cyclictest11957-21latency_hist11:21:241
255349955,0cyclictest0-21swapper/012:19:100
255349950,0cyclictest13713-21basename07:36:240
255349950,0cyclictest0-21swapper/011:22:340
255349950,0cyclictest0-21swapper/010:29:450
249950,0watchdog/23330-21processes10:16:512
249950,0watchdog/230927-21ssh10:34:512
249950,0watchdog/228938-21iostat09:56:432
149950,0migration/08019-21kthreadcore08:41:430
149950,0migration/07238-21munin-node09:06:380
149950,0migration/06637-21sh09:05:580
149950,0migration/05813-21timerwakeupswit07:01:510
149950,0migration/05709-21kthreadcore11:16:510
149950,0migration/05526-21timerandwakeup10:36:520
149950,0migration/05266-21kthreadcore09:21:470
149950,0migration/031508-21sh09:19:340
149950,0migration/030495-21fschecks_count10:16:410
149950,0migration/030252-21sh09:00:230
149950,0migration/029916-21kthreadcore08:36:410
149950,0migration/027582-21sh09:56:190
149950,0migration/026225-21sh11:09:410
149950,0migration/025678-21latency_hist06:56:240
149950,0migration/024687-21taskset09:34:150
149950,0migration/023844-21kthreadcore11:26:430
149950,0migration/021550-21cpuspeed_turbos10:11:370
149950,0migration/020393-21timerandwakeup08:01:480
149950,0migration/019018-21sh10:07:030
149950,0migration/018559-21kthreadcore12:01:430
149950,0migration/018499-21kthreadcore08:26:420
149950,0migration/018231-21kthreadcore10:46:500
149950,0migration/017687-21taskset11:04:470
149950,0migration/016610-21fschecks_time08:51:410
149950,0migration/01325-21sendmail_mailst09:56:490
149950,0migration/012859-21kthreadcore07:56:420
339940,0migration/33689-21kthreadcore08:16:413
339940,0migration/327119-21taskset12:06:443
339940,0migration/322053-21taskset08:52:033
339940,0migration/314500-21kthreadcore07:11:423
339940,0migration/310803-21kthreadcore07:06:443
28855040,0irq/27-eth0-tx-31794-21ssh10:52:322
28855040,0irq/27-eth0-tx-24583-21ssh12:06:252
28855040,0irq/27-eth0-tx-15832-21proc_pri11:01:472
28855040,0irq/27-eth0-tx-0-21swapper/212:16:272
28855040,0irq/27-eth0-tx-0-21swapper/211:56:452
28855040,0irq/27-eth0-tx-0-21swapper/211:51:432
28855040,0irq/27-eth0-tx-0-21swapper/211:46:422
28855040,0irq/27-eth0-tx-0-21swapper/211:26:352
28855040,0irq/27-eth0-tx-0-21swapper/211:22:162
28855040,0irq/27-eth0-tx-0-21swapper/211:12:262
28855040,0irq/27-eth0-tx-0-21swapper/210:27:082
28855040,0irq/27-eth0-tx-0-21swapper/210:22:002
28855040,0irq/27-eth0-tx-0-21swapper/210:06:262
28855040,0irq/27-eth0-tx-0-21swapper/209:52:452
28855040,0irq/27-eth0-tx-0-21swapper/209:42:322
28855040,0irq/27-eth0-tx-0-21swapper/008:06:390
28855040,0irq/27-eth0-tx-0-21swapper/007:51:460
28855040,0irq/27-eth0-tx-0-21swapper/007:06:400
28845040,0irq/26-eth0-rx-0-21swapper/312:16:373
28845040,0irq/26-eth0-rx-0-21swapper/309:47:183
28845040,0irq/26-eth0-rx-0-21swapper/308:41:423
28845040,0irq/26-eth0-rx-0-21swapper/306:56:433
255379944,0cyclictest0-21swapper/212:13:422
255379943,0cyclictest9219-21sadc08:46:242
255379943,0cyclictest819-21awk07:01:432
255379943,0cyclictest8145-21awk07:31:402
255379943,0cyclictest3469-21lldpd11:36:282
255379943,0cyclictest2911-21awk08:16:392
255379943,0cyclictest27967-21awk07:46:462
255379943,0cyclictest27272-21awk07:21:412
255379943,0cyclictest19043-21awk12:21:462
255379943,0cyclictest17801-21awk10:46:482
255379943,0cyclictest16963-21awk12:01:412
255379943,0cyclictest1556-21awk07:26:402
255379943,0cyclictest11731-21awk09:46:442
255349944,0cyclictest0-21swapper/010:32:170
255349944,0cyclictest0-21swapper/009:51:080
255349943,0cyclictest7195-21awk10:21:420
255349943,0cyclictest3154-21irqbalance11:11:530
255349943,0cyclictest3154-21irqbalance09:41:540
255349943,0cyclictest3154-21irqbalance08:21:440
255349943,0cyclictest3154-21irqbalance08:17:040
255349943,0cyclictest3154-21irqbalance07:41:340
255349943,0cyclictest3154-21irqbalance07:21:340
255349943,0cyclictest28654-21awk08:11:390
255349943,0cyclictest17788-21cut09:11:430
259940,0migration/231921-21timerandwakeup09:37:002
259940,0migration/230635-21sh09:17:422
259940,0migration/225538-21timerandwakeup07:16:492
259940,0migration/225147-21kthreadcore08:06:472
259940,0migration/222100-21runrttasks08:52:382
259940,0migration/220029-21kthreadcore08:26:422
259940,0migration/212811-21runrttasks07:09:152
259940,0migration/210836-21kthreadcore10:41:452
249940,0watchdog/22017-21kthreadcore09:01:472
249940,0watchdog/220151-21users11:41:502
249940,0watchdog/214505-21kthreadcore08:46:432
249940,0watchdog/20-21swapper/208:31:432
249940,0watchdog/20-21swapper/207:41:432
149940,0migration/04197-21kthreadcore10:56:500
149940,0migration/024518-21sh12:22:250
149940,0migration/014800-21kthreadcore07:11:420
149940,0migration/014529-21fschecks_time07:36:380
149940,0migration/01239-21sh09:40:090
149940,0migration/011238-21kthreadcore11:56:500
339930,0migration/331837-21taskset07:21:463
339930,0migration/328098-21runrttasks08:34:073
339930,0migration/315153-21munin-node07:36:413
339930,0migration/314020-21runrttasks07:56:563
28845030,0irq/26-eth0-rx-0-21swapper/307:32:193
255379932,0cyclictest29419-21awk08:36:392
255379932,0cyclictest28571-21grep08:11:392
255379932,0cyclictest15547-21awk08:01:402
255379931,0cyclictest29035-21kthreadcore06:56:432
255379930,0cyclictest0-21swapper/207:56:242
255349933,0cyclictest0-21swapper/010:05:000
255349932,0cyclictest7176-21ssh12:12:030
255349932,0cyclictest6329-21awk11:36:430
255349932,0cyclictest3427-21snmpd10:41:450
255349932,0cyclictest3154-21irqbalance11:41:430
255349932,0cyclictest26613-21uname10:51:430
255349932,0cyclictest23503-21awk11:46:440
259930,0migration/213784-21fschecks_time07:11:382
28845020,0irq/26-eth0-rx-0-21swapper/308:36:533
28845020,0irq/26-eth0-rx-0-21swapper/308:13:463
28845020,0irq/26-eth0-rx-0-21swapper/307:16:453
28845020,0irq/26-eth0-rx-0-21swapper/307:01:253
255379920,0cyclictest0-21swapper/207:36:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional