You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-17 - 16:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Mon Jun 17, 2024 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350530,0irq/25-eth10-21swapper/306:53:533
3043823424,0sleep10-21swapper/106:54:041
335523228,0sleep20-21swapper/206:55:212
360322723,0sleep00-21swapper/006:58:110
28415070,0irq/27-eth0-tx-0-21swapper/210:46:382
28415070,0irq/27-eth0-tx-0-21swapper/210:27:112
37419960,0cyclictest0-21swapper/209:54:052
37419960,0cyclictest0-21swapper/207:54:062
37379965,0cyclictest31060-21ls07:23:491
37359966,0cyclictest0-21swapper/010:17:470
37359966,0cyclictest0-21swapper/009:20:240
339960,0migration/34724-21sh10:52:063
339960,0migration/3327-21sh10:05:213
339960,0migration/323616-21sh11:45:113
339960,0migration/320688-21kthreadcore07:14:143
339960,0migration/320203-21sh12:24:133
339960,0migration/319256-21sh09:36:033
339960,0migration/310763-21sh09:53:103
28415060,0irq/27-eth0-tx-0-21swapper/212:00:062
28415060,0irq/27-eth0-tx-0-21swapper/211:53:282
28415060,0irq/27-eth0-tx-0-21swapper/211:45:132
28415060,0irq/27-eth0-tx-0-21swapper/211:27:262
28415060,0irq/27-eth0-tx-0-21swapper/211:09:292
28415060,0irq/27-eth0-tx-0-21swapper/210:23:332
28415060,0irq/27-eth0-tx-0-21swapper/209:49:052
28415060,0irq/27-eth0-tx-0-21swapper/209:41:232
28415060,0irq/27-eth0-tx-0-21swapper/209:33:562
28415060,0irq/27-eth0-tx-0-21swapper/209:32:422
28415060,0irq/27-eth0-tx-0-21swapper/007:54:250
28415060,0irq/27-eth0-tx-0-21swapper/007:46:330
28415060,0irq/27-eth0-tx-0-21swapper/007:29:220
28405060,0irq/26-eth0-rx-31928-21kthreadcore09:04:133
28405060,0irq/26-eth0-rx-31726-21munin-node12:14:013
28405060,0irq/26-eth0-rx-17586-21ssh09:55:143
249960,0watchdog/2270512sleep209:19:192
149960,0migration/013838-21sh10:35:530
8225050,0irq/24-0000:00:27448-21cat08:13:482
7167250,0sleep38809-21kthreadcore09:09:083
4150,0ktimersoftd/00-21swapper/010:50:160
37419955,0cyclictest0-21swapper/208:34:182
37419954,0cyclictest9533-21grep08:24:112
37379955,0cyclictest0-21swapper/112:27:091
37379955,0cyclictest0-21swapper/112:04:081
37379955,0cyclictest0-21swapper/112:01:461
37379955,0cyclictest0-21swapper/111:38:261
37379955,0cyclictest0-21swapper/108:04:011
37359955,0cyclictest0-21swapper/010:32:550
37359955,0cyclictest0-21swapper/010:08:200
37359955,0cyclictest0-21swapper/008:59:450
37359950,0cyclictest0-21swapper/010:56:340
339950,0migration/39241-21kthreadcore12:19:103
339950,0migration/37032-21kthreadcore09:29:143
339950,0migration/36415-21munin-node07:54:253
339950,0migration/35921-21taskset07:32:093
339950,0migration/35253-21kthreadcore10:09:163
339950,0migration/327401-21latency_hist08:13:483
339950,0migration/321503-21sh09:18:543
339950,0migration/321413-21fschecks_time08:09:123
339950,0migration/319339-21kthreadcore08:29:223
339950,0migration/318112-21kthreadcore12:04:103
339950,0migration/314723-21sh12:00:093
339950,0migration/314103-21ksmtuned10:36:513
339950,0migration/311185-21kthreadcore07:59:213
28415050,0irq/27-eth0-tx-12050-21ssh10:55:452
28415050,0irq/27-eth0-tx-0-21swapper/212:19:432
28415050,0irq/27-eth0-tx-0-21swapper/212:04:022
28415050,0irq/27-eth0-tx-0-21swapper/211:39:012
28415050,0irq/27-eth0-tx-0-21swapper/211:34:112
28415050,0irq/27-eth0-tx-0-21swapper/211:04:142
28415050,0irq/27-eth0-tx-0-21swapper/211:00:442
28415050,0irq/27-eth0-tx-0-21swapper/210:05:092
28415050,0irq/27-eth0-tx-0-21swapper/210:01:422
28415050,0irq/27-eth0-tx-0-21swapper/209:45:082
28415050,0irq/27-eth0-tx-0-21swapper/209:14:192
28415050,0irq/27-eth0-tx-0-21swapper/209:11:062
28415050,0irq/27-eth0-tx-0-21swapper/209:01:452
28415050,0irq/27-eth0-tx-0-21swapper/007:34:140
28415050,0irq/27-eth0-tx-0-21swapper/007:34:140
28415050,0irq/27-eth0-tx-0-21swapper/007:05:520
28405050,0irq/26-eth0-rx-5260-21cut10:29:163
28405050,0irq/26-eth0-rx-4667-21latency_hist09:28:483
28405050,0irq/26-eth0-rx-28867-21if_eth109:44:063
28405050,0irq/26-eth0-rx-25849-21ntp_kernel_pll_09:39:213
28405050,0irq/26-eth0-rx-20691-21ssh09:16:483
28405050,0irq/26-eth0-rx-19691-21smartctl11:24:043
28405050,0irq/26-eth0-rx-10484-21if_eth1.211:39:073
28405050,0irq/26-eth0-rx-0-21swapper/312:09:083
28405050,0irq/26-eth0-rx-0-21swapper/311:50:083
28405050,0irq/26-eth0-rx-0-21swapper/311:15:213
28405050,0irq/26-eth0-rx-0-21swapper/311:08:193
28405050,0irq/26-eth0-rx-0-21swapper/310:56:463
28405050,0irq/26-eth0-rx-0-21swapper/310:24:033
28405050,0irq/26-eth0-rx-0-21swapper/309:59:443
28405050,0irq/26-eth0-rx-0-21swapper/307:19:013
259950,0migration/22339-21kthreadcore12:14:112
249950,0watchdog/27046-21fschecks_time07:34:072
249950,0watchdog/27046-21fschecks_time07:34:072
249950,0watchdog/20-21swapper/212:24:072
249950,0watchdog/20-21swapper/212:12:552
249950,0watchdog/20-21swapper/211:54:192
249950,0watchdog/20-21swapper/211:29:352
249950,0watchdog/20-21swapper/211:19:032
249950,0watchdog/20-21swapper/211:14:432
249950,0watchdog/20-21swapper/210:51:352
249950,0watchdog/20-21swapper/210:39:312
249950,0watchdog/20-21swapper/210:36:392
249950,0watchdog/20-21swapper/210:29:472
249950,0watchdog/20-21swapper/210:11:312
249950,0watchdog/20-21swapper/209:04:472
249950,0watchdog/20-21swapper/207:30:512
249950,0watchdog/20-21swapper/207:13:512
179950,0migration/17847-21kthreadcore11:14:091
179950,0migration/17316-21taskset08:19:231
179950,0migration/15305-21latency_hist10:53:481
179950,0migration/128111-21latency_hist11:08:481
169950,0watchdog/119352-21tr11:44:151
169950,0watchdog/114080-21sed07:39:111
169950,0watchdog/111387-21grep12:19:111
149950,0migration/09935-21sh11:15:230
149950,0migration/05084-21kthreadcore10:09:160
149950,0migration/04010-21kthreadcore12:14:110
149950,0migration/03460-21kthreadcore11:34:160
149950,0migration/0323-21sh11:29:510
149950,0migration/025730-21kthreadcore08:34:110
149950,0migration/023804-21kthreadcore09:59:170
149950,0migration/023105-21fschecks_time10:44:070
149950,0migration/02215-21fschecks_count08:19:070
149950,0migration/021128-21cpuspeed_turbos08:09:090
149950,0migration/020929-21fschecks_time09:39:090
149950,0migration/019196-21kthreadcore12:04:110
149950,0migration/01891-21kthreadcore11:54:100
149950,0migration/017490-21aten2.4-expect11:44:050
149950,0migration/01629-21kthreadcore09:24:150
149950,0migration/015876-21if_eth1.310:39:130
149950,0migration/015526-21kthreadcore08:04:290
149950,0migration/015289-21kthreadcore09:34:140
149950,0migration/013323-21kthreadcore07:09:110
149950,0migration/011013-21kthreadcore08:49:100
149950,0migration/010924-21kthreadcore07:59:210
37419943,0cyclictest3110-21irqbalance08:39:232
37419943,0cyclictest2590-21awk08:19:102
37419943,0cyclictest15090-21awk08:04:272
37419943,0cyclictest10408-21awk08:49:072
37379944,0cyclictest0-21swapper/110:59:391
37379943,0cyclictest8290-21awk10:34:141
37379943,0cyclictest6445-21snmpd11:55:551
37379943,0cyclictest6445-21snmpd10:27:101
37379943,0cyclictest6445-21snmpd08:55:101
37379943,0cyclictest6445-21snmpd07:27:551
37379943,0cyclictest633-21awk10:29:111
37379943,0cyclictest4357-21awk09:49:091
37379943,0cyclictest3425-21lldpd12:15:521
37379943,0cyclictest27869-21kworker/1:211:22:501
37379943,0cyclictest27869-21kworker/1:211:04:321
37379943,0cyclictest27869-21kworker/1:210:00:121
37379943,0cyclictest27209-21awk10:04:121
37379943,0cyclictest25863-21awk11:49:031
37359944,0cyclictest0-21swapper/011:51:170
37359943,0cyclictest6445-21snmpd11:04:400
37359943,0cyclictest6445-21snmpd09:08:100
37359943,0cyclictest4324-21awk09:49:080
37359943,0cyclictest3425-21lldpd10:23:250
37359943,0cyclictest31807-21sadc10:28:480
37359943,0cyclictest3110-21irqbalance11:59:130
37359943,0cyclictest3110-21irqbalance08:30:530
37359943,0cyclictest29139-21awk11:09:060
37359943,0cyclictest16983-21awk08:54:080
37359943,0cyclictest15577-21awk12:24:050
37359943,0cyclictest13891-21awk07:39:100
339940,0migration/332599-21taskset07:51:033
339940,0migration/327166-21missed_timers08:34:133
339940,0migration/320996-21kthreadcore10:19:123
339940,0migration/316275-21kthreadcore07:39:143
339940,0migration/313987-21kthreadcore11:19:123
339940,0migration/311159-21kthreadcore07:34:193
339940,0migration/311159-21kthreadcore07:34:193
28415040,0irq/27-eth0-tx-0-21swapper/210:14:032
28415040,0irq/27-eth0-tx-0-21swapper/008:44:030
28415040,0irq/27-eth0-tx-0-21swapper/008:14:070
28415040,0irq/27-eth0-tx-0-21swapper/007:19:080
28405040,0irq/26-eth0-rx-7505-21sendmail_mailst11:34:273
28405040,0irq/26-eth0-rx-0-21swapper/311:56:183
28405040,0irq/26-eth0-rx-0-21swapper/311:33:103
28405040,0irq/26-eth0-rx-0-21swapper/311:09:093
28405040,0irq/26-eth0-rx-0-21swapper/310:58:513
28405040,0irq/26-eth0-rx-0-21swapper/310:44:563
28405040,0irq/26-eth0-rx-0-21swapper/310:42:083
28405040,0irq/26-eth0-rx-0-21swapper/310:14:153
28405040,0irq/26-eth0-rx-0-21swapper/309:01:123
28405040,0irq/26-eth0-rx-0-21swapper/308:44:033
28405040,0irq/26-eth0-rx-0-21swapper/308:18:503
259940,0migration/227845-21kthreadcore07:49:132
259940,0migration/221374-21kthreadcore07:44:172
259940,0migration/218169-21kthreadcore07:39:142
249940,0watchdog/20-21swapper/209:25:152
249940,0watchdog/20-21swapper/208:54:112
249940,0watchdog/20-21swapper/208:44:392
249940,0watchdog/20-21swapper/208:29:112
249940,0watchdog/20-21swapper/208:09:232
249940,0watchdog/20-21swapper/207:58:552
249940,0watchdog/20-21swapper/207:23:512
249940,0watchdog/20-21swapper/207:19:072
249940,0watchdog/20-21swapper/207:04:232
249940,0watchdog/20-21swapper/207:02:552
179940,0migration/15971-21fschecks_count10:54:081
179940,0migration/132749-21hddtemp_smartct07:29:061
179940,0migration/131559-21kthreadcore09:44:101
179940,0migration/127530-21kthreadcore12:09:101
179940,0migration/122742-21latency_hist08:58:481
179940,0migration/122571-21kthreadcore07:44:171
179940,0migration/122232-21kthreadcore08:09:231
179940,0migration/121427-21taskset08:31:481
179940,0migration/118321-21kthreadcore10:19:111
179940,0migration/117611-21taskset07:12:061
179940,0migration/116817-21kthreadcore10:39:161
179940,0migration/116486-21kthreadcore09:14:131
179940,0migration/11557-21latency_hist10:08:481
179940,0migration/113899-21kthreadcore08:49:121
179940,0migration/111291-21kthreadcore11:39:111
179940,0migration/111179-21kthreadcore08:24:151
179940,0migration/110918-21timerwakeupswit09:29:281
179940,0migration/110527-21kthreadcore09:09:091
169940,0watchdog/17047-21fschecks_time07:34:071
169940,0watchdog/17047-21fschecks_time07:34:071
169940,0watchdog/16547-21runrttasks06:59:191
169940,0watchdog/1626-21ssh11:30:591
169940,0watchdog/129311-21ssh09:23:551
169940,0watchdog/11942-21irqstats07:54:111
169940,0watchdog/115067-21ssh10:14:231
169940,0watchdog/113651-21munin-node09:34:111
169940,0watchdog/10-21swapper/111:24:111
169940,0watchdog/10-21swapper/109:54:031
149940,0migration/0986-21kthreadcore08:39:170
149940,0migration/028395-21fschecks_count09:44:030
149940,0migration/024384-21df_abs12:09:010
149940,0migration/021462-21kthreadcore07:14:150
149940,0migration/019481-21fschecks_time11:24:040
149940,0migration/015092-21kthreadcore10:59:100
149940,0migration/012364-21irqcore09:54:130
149940,0migration/011462-21kthreadcore08:24:150
37419932,0cyclictest12171-21awk07:09:072
37379933,0cyclictest27869-21kworker/1:210:43:541
37379933,0cyclictest18489-21kworker/1:009:39:021
37379933,0cyclictest18489-21kworker/1:009:18:481
37379933,0cyclictest18489-21kworker/1:008:43:481
37379933,0cyclictest18489-21kworker/1:008:38:481
37379933,0cyclictest18489-21kworker/1:008:33:501
37379933,0cyclictest18489-21kworker/1:008:13:561
37379933,0cyclictest18489-21kworker/1:007:59:061
37379933,0cyclictest18489-21kworker/1:007:48:591
37379933,0cyclictest18489-21kworker/1:007:13:511
37379933,0cyclictest18489-21kworker/1:007:03:531
37379932,0cyclictest31396-21awk09:04:091
37359932,0cyclictest6631-21cut09:09:030
37359932,0cyclictest6445-21snmpd09:16:400
37359932,0cyclictest10411-21awk11:39:060
37359930,0cyclictest14439-21cut11:19:130
37359930,0cyclictest14404-21munin-run12:23:480
37359930,0cyclictest0-21swapper/009:29:140
339930,0migration/32436-21runrttasks08:43:203
28415030,0irq/27-eth0-tx-0-21swapper/007:49:020
28415030,0irq/27-eth0-tx-0-21swapper/007:03:190
28405030,0irq/26-eth0-rx-0-21swapper/308:55:353
28405030,0irq/26-eth0-rx-0-21swapper/307:09:013
28405030,0irq/26-eth0-rx-0-21swapper/307:05:303
28405030,0irq/26-eth0-rx-0-21swapper/306:59:223
149930,0migration/031567-21runrttasks07:26:550
28405020,0irq/26-eth0-rx-0-21swapper/308:50:153
28405020,0irq/26-eth0-rx-0-21swapper/308:23:483
28405020,0irq/26-eth0-rx-0-21swapper/308:04:053
28405020,0irq/26-eth0-rx-0-21swapper/307:44:053
28405020,0irq/26-eth0-rx-0-21swapper/307:28:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional