You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 14:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Apr 27, 2024 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350570,0irq/25-eth10-21swapper/306:58:273
794325038,0sleep10-21swapper/106:57:261
807323627,0sleep20-21swapper/206:58:592
815723423,0sleep00-21swapper/006:59:570
82929960,0cyclictest0-21swapper/109:44:221
82919965,0cyclictest41ktimersoftd/009:43:590
82919964,0cyclictest9433-21awk07:00:530
82919963,0cyclictest21586-21munin-run07:10:410
82919963,0cyclictest14910-21kthreadcore09:46:010
82919962,0cyclictest30267-21df_inode09:35:570
82919962,0cyclictest2625-21idleruntime-cro12:00:410
82919961,0cyclictest9461-21munin-node07:25:540
82919960,0cyclictest0-21swapper/009:33:260
339960,0migration/33367-21sh09:56:443
339960,0migration/330899-21munin-node09:15:553
339960,0migration/325681-21sh12:13:003
339960,0migration/318694-21sh11:07:083
339960,0migration/311709-21munin-node12:05:553
339960,0migration/310396-21sh12:04:553
28415060,0irq/27-eth0-tx-25118-21ssh12:11:232
28415060,0irq/27-eth0-tx-0-21swapper/211:48:242
28415060,0irq/27-eth0-tx-0-21swapper/211:35:572
28415060,0irq/27-eth0-tx-0-21swapper/211:16:022
28415060,0irq/27-eth0-tx-0-21swapper/210:58:032
28415060,0irq/27-eth0-tx-0-21swapper/210:41:022
28415060,0irq/27-eth0-tx-0-21swapper/210:39:142
28415060,0irq/27-eth0-tx-0-21swapper/210:30:572
28415060,0irq/27-eth0-tx-0-21swapper/210:15:522
28415060,0irq/27-eth0-tx-0-21swapper/209:37:472
28415060,0irq/27-eth0-tx-0-21swapper/209:34:292
28405060,0irq/26-eth0-rx-9838-21nfs4_client11:01:093
28405060,0irq/26-eth0-rx-8676-21vmstat12:21:083
27160,0ktimersoftd/20-21swapper/211:56:152
249960,0watchdog/20-21swapper/211:51:232
249960,0watchdog/20-21swapper/211:28:352
249960,0watchdog/20-21swapper/210:51:442
249960,0watchdog/20-21swapper/210:46:402
249960,0watchdog/20-21swapper/209:56:162
249960,0watchdog/20-21swapper/209:55:242
82929955,0cyclictest0-21swapper/112:01:031
82929955,0cyclictest0-21swapper/110:33:051
82929955,0cyclictest0-21swapper/110:15:541
82929954,0cyclictest0-21swapper/111:41:021
82919954,0cyclictest6286-21perl08:36:000
82919953,0cyclictest31458-21irqcore09:16:010
82919953,0cyclictest18960-21irqcore08:20:580
82919952,0cyclictest825-21interrupts08:55:580
82919952,0cyclictest7148-21ssh10:01:010
82919952,0cyclictest6971-21sed09:00:520
82919952,0cyclictest4915-21expr08:10:550
82919952,0cyclictest4479-21tr11:00:540
82919952,0cyclictest4127-21grep07:46:010
82919952,0cyclictest3636-21grep12:00:540
82919952,0cyclictest3636-21cpuspeed_turbos11:40:540
82919952,0cyclictest3411-21lldpd10:30:560
82919952,0cyclictest29754-21idleruntime10:35:540
82919952,0cyclictest29660-21cpuspeed_turbos07:40:520
82919952,0cyclictest29651-21grep07:15:590
82919952,0cyclictest28749-21perl11:15:540
82919952,0cyclictest27386-21grep12:15:550
82919952,0cyclictest26384-21cut08:50:560
82919952,0cyclictest26236-21grep11:11:100
82919952,0cyclictest24677-21cut09:11:050
82919952,0cyclictest23760-21cut07:35:590
82919952,0cyclictest22672-21sed09:51:010
82919952,0cyclictest22518-21tr07:56:050
82919952,0cyclictest22086-21cut08:46:000
82919952,0cyclictest20567-21hostname10:50:520
82919952,0cyclictest20473-21tune2fs11:30:590
82919952,0cyclictest19619-21grep12:10:540
82919952,0cyclictest1804-21kthreadcore10:16:000
82919952,0cyclictest16460-21kthreadcore09:26:020
82919952,0cyclictest1622-21sh11:36:290
82919952,0cyclictest13333-21fschecks_count10:05:580
82919952,0cyclictest12623-21sed11:05:550
82919952,0cyclictest12157-21rm10:45:170
82919952,0cyclictest12058-21gltestperf08:15:590
82919952,0cyclictest11881-21if_eth1.212:05:570
82919952,0cyclictest113452sleep011:45:440
82919952,0cyclictest10067-21memory10:21:000
82919951,0cyclictest4959-21irqcore11:21:030
82919951,0cyclictest30632-21irqrtprio09:56:040
82919951,0cyclictest29668-21sort10:56:030
82919951,0cyclictest13122-21irqrtprio11:26:000
82919951,0cyclictest0-21swapper/012:25:570
82919951,0cyclictest0-21swapper/012:20:550
82919951,0cyclictest0-21swapper/011:50:540
82919951,0cyclictest0-21swapper/010:25:550
82919951,0cyclictest0-21swapper/010:14:250
82919951,0cyclictest0-21swapper/009:20:560
82919951,0cyclictest0-21swapper/009:06:010
82919951,0cyclictest0-21swapper/008:40:570
82919951,0cyclictest0-21swapper/008:25:570
82919951,0cyclictest0-21swapper/008:01:010
82919951,0cyclictest0-21swapper/007:30:570
82919951,0cyclictest0-21swapper/007:21:060
82919950,0cyclictest0-21swapper/010:45:530
82919950,0cyclictest0-21swapper/008:05:560
82919950,0cyclictest0-21swapper/007:51:060
35150,0ktimersoftd/30-21swapper/309:43:593
339950,0migration/3955-21kthreadcore08:31:033
339950,0migration/36370-21kthreadcore11:41:023
339950,0migration/33235-21munin-node08:06:073
339950,0migration/323644-21kthreadcore10:51:023
339950,0migration/322594-21diskmemload09:51:013
339950,0migration/322559-21munin-node09:30:533
339950,0migration/318131-21runrttasks11:26:343
339950,0migration/317369-21munin-node07:31:013
339950,0migration/317262-21kthreadcore09:06:013
339950,0migration/31723-21taskset11:36:473
339950,0migration/313832-21munin-node10:06:023
339950,0migration/310500-21aten_r1power_vo07:50:533
339950,0migration/310481-21munin-node10:21:043
28415050,0irq/27-eth0-tx-5985-21munin-node09:21:082
28415050,0irq/27-eth0-tx-20246-21ssh09:06:062
28415050,0irq/27-eth0-tx-19597-21ssh10:27:532
28415050,0irq/27-eth0-tx-19325-21ssh10:07:012
28415050,0irq/27-eth0-tx-14391-21ssh09:25:552
28415050,0irq/27-eth0-tx-13023-21ssh09:02:022
28415050,0irq/27-eth0-tx-12249-21kernelversion12:05:592
28415050,0irq/27-eth0-tx-12232-21munin-node11:06:032
28415050,0irq/27-eth0-tx-11352-21ssh10:01:502
28415050,0irq/27-eth0-tx-0-21swapper/212:28:562
28415050,0irq/27-eth0-tx-0-21swapper/212:18:102
28415050,0irq/27-eth0-tx-0-21swapper/212:03:422
28415050,0irq/27-eth0-tx-0-21swapper/211:21:272
28415050,0irq/27-eth0-tx-0-21swapper/211:02:492
28415050,0irq/27-eth0-tx-0-21swapper/209:11:492
28405050,0irq/26-eth0-rx-5278-21grep10:40:573
28405050,0irq/26-eth0-rx-28155-21ssh10:14:203
28405050,0irq/26-eth0-rx-23635-40yum-updatesd-he08:21:063
28405050,0irq/26-eth0-rx-19592-21ssh10:48:443
28405050,0irq/26-eth0-rx-12066-21hddtemp_smartct08:15:593
28405050,0irq/26-eth0-rx-1005-21ssh12:17:303
28405050,0irq/26-eth0-rx-0-21swapper/312:30:293
28405050,0irq/26-eth0-rx-0-21swapper/311:55:433
28405050,0irq/26-eth0-rx-0-21swapper/311:21:353
28405050,0irq/26-eth0-rx-0-21swapper/311:10:493
28405050,0irq/26-eth0-rx-0-21swapper/310:36:533
28405050,0irq/26-eth0-rx-0-21swapper/310:17:203
28405050,0irq/26-eth0-rx-0-21swapper/309:39:193
28405050,0irq/26-eth0-rx-0-21swapper/309:26:003
28405050,0irq/26-eth0-rx-0-21swapper/309:22:323
28405050,0irq/26-eth0-rx-0-21swapper/307:10:383
259950,0migration/27514-21kthreadcore10:20:582
259950,0migration/24147-21kthreadcore12:20:592
259950,0migration/23458-21fschecks_time07:20:572
259950,0migration/222359-21missed_timers07:56:032
259950,0migration/220963-21munin-node10:10:512
259950,0migration/219089-21kthreadcore07:31:022
259950,0migration/21310-21kthreadcore08:56:002
259950,0migration/211006-21needreboot09:41:042
249950,0watchdog/221029-21ssh09:50:282
249950,0watchdog/220151-21df11:30:552
249950,0watchdog/20-21swapper/211:42:512
249950,0watchdog/20-21swapper/211:12:072
249950,0watchdog/20-21swapper/209:16:002
249950,0watchdog/20-21swapper/208:51:042
249950,0watchdog/20-21swapper/208:30:042
249950,0watchdog/20-21swapper/208:11:162
249950,0watchdog/20-21swapper/207:55:402
249950,0watchdog/20-21swapper/207:35:482
249950,0watchdog/20-21swapper/207:17:482
249950,0watchdog/20-21swapper/207:10:362
249950,0watchdog/20-21swapper/207:04:002
179950,0migration/128806-21taskset07:36:381
179950,0migration/120281-21if_eth1.211:50:561
179950,0migration/119300-21taskset11:30:011
179950,0migration/114875-21munin-node07:26:021
169950,0watchdog/130692-21basename09:36:001
83019943,0cyclictest6445-21snmpd07:25:462
83019943,0cyclictest30307-21awk07:40:572
83019943,0cyclictest13183-21awk08:40:552
82929944,0cyclictest0-21swapper/112:20:461
82929944,0cyclictest0-21swapper/110:52:291
82929944,0cyclictest0-21swapper/107:50:421
82929943,0cyclictest9353-21awk07:00:521
82929943,0cyclictest6445-21snmpd11:09:471
82929943,0cyclictest6445-21snmpd10:06:171
82929943,0cyclictest5530-21awk10:20:551
82929943,0cyclictest4528-21awk07:46:031
82929943,0cyclictest3425-21lldpd10:47:261
82929943,0cyclictest25495-21kworker/1:012:07:011
82929943,0cyclictest25495-21kworker/1:009:12:031
82929943,0cyclictest21639-21sadc09:30:411
82929943,0cyclictest21181-21awk11:10:591
82929943,0cyclictest12302-21awk11:46:011
82919942,0cyclictest31614-21df_abs08:30:560
82919942,0cyclictest21716-21cat07:10:410
339940,0migration/39105-21kthreadcore08:36:043
339940,0migration/37042-21kthreadcore07:46:063
339940,0migration/34858-21latency_hist10:00:413
339940,0migration/331888-21kthreadcore07:41:013
339940,0migration/331462-21kthreadcore10:56:053
339940,0migration/326108-21diskmemload11:31:373
339940,0migration/314852-21kthreadcore10:26:013
28405040,0irq/26-eth0-rx-0-21swapper/311:50:523
28405040,0irq/26-eth0-rx-0-21swapper/311:45:463
28405040,0irq/26-eth0-rx-0-21swapper/311:16:093
28405040,0irq/26-eth0-rx-0-21swapper/310:30:563
28405040,0irq/26-eth0-rx-0-21swapper/309:45:483
28405040,0irq/26-eth0-rx-0-21swapper/309:10:453
28405040,0irq/26-eth0-rx-0-21swapper/309:00:563
28405040,0irq/26-eth0-rx-0-21swapper/308:58:543
28405040,0irq/26-eth0-rx-0-21swapper/308:49:593
28405040,0irq/26-eth0-rx-0-21swapper/308:25:413
259940,0migration/29364-21munin-node07:46:132
259940,0migration/24393-21netstat08:31:062
259940,0migration/226590-21kthreadcore08:01:012
249940,0watchdog/221932-21kthreadcore08:21:002
249940,0watchdog/220775-21cat08:46:002
249940,0watchdog/20-21swapper/208:35:522
249940,0watchdog/20-21swapper/208:20:122
249940,0watchdog/20-21swapper/208:06:202
249940,0watchdog/20-21swapper/207:10:562
179940,0migration/19590-21kthreadcore09:00:581
179940,0migration/18251-21nfs_client07:21:071
179940,0migration/17357-21kthreadcore08:36:041
179940,0migration/13327-21taskset07:44:431
179940,0migration/128853-21kthreadcore12:16:001
179940,0migration/12791-21runrttasks09:56:121
179940,0migration/126045-21kthreadcore08:26:021
179940,0migration/125123-21taskset08:48:071
179940,0migration/123058-21kthreadcore09:51:021
179940,0migration/118498-21kthreadcore07:05:591
179940,0migration/117422-21sh12:28:141
179940,0migration/115115-21fschecks_time09:05:541
179940,0migration/114218-21kthreadcore08:16:051
179940,0migration/110822-21taskset11:02:221
169940,0watchdog/16711-21fschecks_time09:20:561
169940,0watchdog/15588-21kthreadcore11:21:031
169940,0watchdog/14654-21ssh09:17:281
169940,0watchdog/131616-21kthreadcore08:06:041
169940,0watchdog/131614-21munin-node08:30:561
169940,0watchdog/127381-21ssh10:12:121
169940,0watchdog/116496-21entropy07:30:561
169940,0watchdog/111276-21ssh10:42:401
169940,0watchdog/110716-21processes10:01:081
169940,0watchdog/10-21swapper/111:57:351
169940,0watchdog/10-21swapper/111:34:271
169940,0watchdog/10-21swapper/111:15:591
169940,0watchdog/10-21swapper/110:36:001
169940,0watchdog/10-21swapper/109:33:401
169940,0watchdog/10-21swapper/108:51:001
169940,0watchdog/10-21swapper/107:15:521
82929933,0cyclictest25495-21kworker/1:012:10:471
82929933,0cyclictest25495-21kworker/1:011:35:431
82929933,0cyclictest25495-21kworker/1:010:55:471
82929933,0cyclictest25495-21kworker/1:010:25:551
82929933,0cyclictest25495-21kworker/1:008:55:431
82929933,0cyclictest25495-21kworker/1:008:40:471
82929933,0cyclictest25495-21kworker/1:008:20:471
82929933,0cyclictest25495-21kworker/1:008:10:431
82929933,0cyclictest25495-21kworker/1:008:00:471
82929933,0cyclictest25495-21kworker/1:007:55:431
82929933,0cyclictest25495-21kworker/1:007:10:431
339930,0migration/329270-21processes08:01:073
28405030,0irq/26-eth0-rx-0-21swapper/308:43:413
28405030,0irq/26-eth0-rx-0-21swapper/307:56:053
28405030,0irq/26-eth0-rx-0-21swapper/307:25:513
28405030,0irq/26-eth0-rx-0-21swapper/307:20:543
28405030,0irq/26-eth0-rx-0-21swapper/307:15:523
28405030,0irq/26-eth0-rx-0-21swapper/307:00:473
169930,0watchdog/10-21swapper/109:45:441
339920,0migration/323748-21iostat_ios07:35:583
28405020,0irq/26-eth0-rx-0-21swapper/308:50:423
28405020,0irq/26-eth0-rx-0-21swapper/308:11:333
28405020,0irq/26-eth0-rx-0-21swapper/307:10:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional