You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 22:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Core i3-2310E @2100 MHz, Linux 5.10.47-rt46 (Profile)

Latency plot of shadow in rack #8, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by a total of 2 SMIs that occured during the measurement.
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot2s.osadl.org (updated Sun Dec 04, 2022 12:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
533499111100,10cyclictest6423-21needreboot07:11:031
4973210041,47sleep00-21swapper/007:09:350
41892970,1sleep04194-21idleruntime-cro09:45:450
99712910,0sleep30-21swapper/311:44:513
62822900,0sleep20-21swapper/209:18:012
475829046,33sleep30-21swapper/307:06:493
489528946,32sleep10-21swapper/107:08:361
479428947,31sleep20-21swapper/207:07:162
246962880,0sleep10-21swapper/110:02:531
203232880,1sleep30-21swapper/312:21:113
53802820,0sleep00-21swapper/010:15:060
5336993630,2cyclictest0-21swapper/307:11:033
5335993530,3cyclictest6422-21uname07:11:032
5333993530,4cyclictest6424-21needreboot07:11:030
239932300,1sleep30-21swapper/310:31:093
533699291,1cyclictest0-21swapper/307:35:473
120672260,0sleep30-21swapper/309:51:173
74842250,1sleep00-21swapper/012:39:230
123462240,1sleep211154-21/usr/sbin/munin08:31:082
533499200,5cyclictest13378-21tr07:25:591
533399205,8cyclictest23035-21munin-run10:30:440
533499190,5cyclictest5502-21seq09:16:591
533499190,4cyclictest13804-21grep10:51:031
533499190,4cyclictest13804-21grep10:51:031
533499190,4cyclictest10892-21tr11:16:151
153232190,0sleep20-21swapper/210:22:582
533499184,12cyclictest0-21swapper/111:07:231
533499176,9cyclictest8260-21sh11:42:371
533499175,4cyclictest2275-21apt10:40:591
533499173,4cyclictest5476-21if_err_enp5s009:46:041
533499170,4cyclictest30374-21/usr/sbin/munin10:36:071
533499170,4cyclictest20313-21ssh11:53:231
533499165,3cyclictest7588-21kworker/u8:011:21:051
533499165,3cyclictest25288-21cstates07:51:021
533499164,5cyclictest0-21swapper/109:32:531
533499164,4cyclictest17727-21latency_hist07:35:441
533499164,4cyclictest17727-21latency_hist07:35:441
533499164,3cyclictest11272-21ssh12:14:081
533499161,6cyclictest4206-21awk09:45:451
533499161,6cyclictest27084-21ls09:05:441
533499161,3cyclictest21131-21cron10:00:441
232102160,0sleep20-21swapper/211:56:092
533499157,6cyclictest23149-21fschecks_count08:56:001
533499156,7cyclictest29398-21diskmemload10:20:511
533499155,3cyclictest8197-21grep10:16:141
533499154,3cyclictest2559-21grep08:11:011
533499154,2cyclictest8970-21cpuspeed_turbos09:20:591
533499153,4cyclictest27634-21cat09:05:571
533499153,3cyclictest29357-21idleruntime-cro08:00:441
533499153,3cyclictest2444-21ssh11:37:251
533499153,3cyclictest12430-21/usr/sbin/munin08:31:091
533499152,4cyclictest0-21swapper/111:05:291
533499152,3cyclictest30393-21grep09:11:001
533499152,3cyclictest0-21swapper/108:26:031
533499151,5cyclictest4545-21gltestperf11:11:021
533499151,5cyclictest27051-21head09:36:081
533499151,5cyclictest26092-21cron11:30:441
533499151,5cyclictest20914-21timerwakeupswit10:56:181
533499151,5cyclictest15029-21grep08:36:101
533499151,4cyclictest31649-21cron08:05:451
533499151,4cyclictest29978-21unixbench_multi12:01:111
533499151,4cyclictest29104-21unixbench_multi10:06:151
533499151,4cyclictest23160-21fschecks_time11:56:091
533499151,4cyclictest19665-21kernelversion12:21:051
533499150,6cyclictest29654-21kworker/1:112:10:121
533499150,5cyclictest3397-21tr12:36:011
533499150,5cyclictest15750-21/usr/sbin/munin09:26:121
533499150,5cyclictest14209-21pmu-power12:16:131
533499150,5cyclictest13405-21ssh11:46:501
533499150,5cyclictest12741-21cron07:25:441
533499150,4cyclictest5871-21idleruntime-cro10:15:441
533499150,3cyclictest232-21systemd-journal11:35:441
533699148,5cyclictest23665-21ssh11:27:293
533499144,3cyclictest16321-21grep08:41:011
533499144,2cyclictest32359-21cut08:06:001
533499144,2cyclictest20989-21gltestperf08:51:011
533499143,2cyclictest7841-21ssh10:45:521
533499142,2cyclictest0-21swapper/107:45:561
533499141,5cyclictest19143-21needreboot08:46:041
533499141,5cyclictest18085-21grep10:26:021
533499141,4cyclictest8871-21fschecks_time07:15:591
533499141,3cyclictest23790-21grep10:31:071
533499140,5cyclictest20630-21grep07:40:581
533499140,5cyclictest10050-21ssh09:50:491
533499140,4cyclictest19082-21cut07:36:031
533499140,2cyclictest0-21swapper/112:34:401
148322140,1sleep00-21swapper/010:22:200
533699139,3cyclictest10079-21ssh12:12:363
533699137,3cyclictest0-21swapper/311:10:053
533699133,5cyclictest0-21swapper/311:01:093
533699132,6cyclictest26003-21kworker/u8:110:11:003
533599131,10cyclictest24595-21grep12:26:002
533499131,4cyclictest7000-21df_inode08:21:031
533499130,4cyclictest4268-21cut08:15:571
533499130,4cyclictest24468-21sh12:26:001
244722130,0sleep00-21swapper/010:02:360
533699128,2cyclictest11341-21cpu08:31:003
533699126,4cyclictest29600-21ls12:30:443
533699126,4cyclictest29535-21sort09:10:443
533699126,4cyclictest28734-21systemctl10:06:123
533699125,5cyclictest0-21swapper/312:18:463
533699124,7cyclictest0-21swapper/309:35:053
533699124,7cyclictest0-21swapper/307:48:103
533699123,8cyclictest0-21swapper/311:17:053
533699121,8cyclictest18479-21grep10:26:053
533699121,6cyclictest32548-21ssh12:32:103
533699121,4cyclictest15879-21cpuspeed_turbos07:31:023
533699121,4cyclictest15879-21cpuspeed_turbos07:31:023
533699120,10cyclictest1891-21gdbus08:10:453
533599125,5cyclictest15640-21awk07:31:012
533599125,5cyclictest15640-21awk07:31:012
533599124,7cyclictest0-21swapper/207:16:022
533599120,10cyclictest355-21wc08:06:042
533599120,10cyclictest3465-21grep12:36:012
533399128,3cyclictest13078-21latency_hist10:50:450
533399128,3cyclictest12-21ksoftirqd/009:00:440
533399127,4cyclictest15967-21awk08:40:580
533399126,4cyclictest30935-21swap08:01:100
533399121,3cyclictest18796-21munin-run10:55:440
533399121,3cyclictest18796-21munin-run10:55:440
533399120,11cyclictest29398-21diskmemload09:19:510
533399120,10cyclictest5353-21apt-get11:40:570
533399120,10cyclictest30845-21if_enp3s012:31:090
533699117,3cyclictest43450irq/33-enp3s0-r09:41:053
533699117,3cyclictest2437-21kernelversion12:06:043
533699117,2cyclictest570-21runrttasks10:23:053
533699117,2cyclictest18097-21apt08:45:563
533699116,4cyclictest9082-21ntpq10:46:063
533699116,3cyclictest31663-21idleruntime-cro08:05:453
533699116,3cyclictest29398-21diskmemload12:03:443
533699116,3cyclictest27193-21grep11:31:043
533699115,5cyclictest0-21swapper/307:44:313
533699115,4cyclictest15915-21grep11:21:053
533699115,3cyclictest21147-21latency_hist10:00:443
533699114,5cyclictest14779-21sed10:51:123
533699114,5cyclictest14779-21sed10:51:123
533699114,5cyclictest11505-21cat07:21:043
533699114,5cyclictest0-21swapper/309:26:343
533699114,5cyclictest0-21swapper/307:51:363
533699112,2cyclictest141rcu_preempt11:45:593
533699111,8cyclictest30317-21ps10:36:073
533699111,8cyclictest28878-21sed07:56:103
533699111,3cyclictest17376-21fschecks_count11:51:073
533699111,1cyclictest0-21swapper/311:39:283
533699110,9cyclictest26035-21cat09:01:073
533699110,9cyclictest24438-21seq08:56:513
533699110,5cyclictest7539-21perf10:45:443
533699110,10cyclictest4562-21apt-get09:45:573
533599118,2cyclictest11983-21unixbench_singl09:51:152
533599116,4cyclictest24950-21wc10:31:212
533599116,4cyclictest232-21systemd-journal12:23:482
533599116,4cyclictest1153-21cut11:36:102
533599116,3cyclictest8554-21mailstats12:11:072
533599115,5cyclictest9493-21which09:21:032
533599115,4cyclictest25448-21latency_hist09:35:452
533599115,4cyclictest20158-21munin-plugin-st08:50:442
533599114,2cyclictest0-21swapper/212:18:372
533599114,1cyclictest0-21swapper/211:44:292
533599114,1cyclictest0-21swapper/210:37:482
533599114,1cyclictest0-21swapper/210:12:482
533599113,3cyclictest30943-21ssh09:40:382
533599113,2cyclictest0-21swapper/211:28:332
533599113,2cyclictest0-21swapper/211:21:312
533599113,2cyclictest0-21swapper/209:28:342
533599111,9cyclictest0-21swapper/210:28:442
533399116,4cyclictest9880-21grep11:16:050
533399115,4cyclictest27345-21cut07:55:590
533399115,4cyclictest24613-21cut12:26:000
533399114,5cyclictest19207-21pmu-power07:36:060
533399111,9cyclictest28400-21runrttasks11:31:160
533399111,9cyclictest15110-21gltestperf09:26:040
533399111,3cyclictest12-21ksoftirqd/010:36:030
60382100,1sleep20-21swapper/209:46:122
533699108,1cyclictest0-21swapper/308:25:473
533699107,2cyclictest5790-21unixbench-2d08:16:083
533699106,3cyclictest0-21swapper/309:13:523
533699105,4cyclictest9258-21entropy09:21:023
533699105,4cyclictest22089-21df10:01:063
533699105,3cyclictest9000-21cut07:16:003
533699105,3cyclictest4670-21sh12:36:123
533699105,3cyclictest12754-21latency_hist07:25:443
533699104,5cyclictest0-21swapper/308:21:073
533699104,4cyclictest3737-21smartctl09:16:003
533699103,5cyclictest575-21snmpd11:56:003
533699103,5cyclictest232-21systemd-journal09:40:453
533699103,4cyclictest0-21swapper/308:51:003
533699101,7cyclictest7107-21head10:16:023
533699100,8cyclictest43450irq/33-enp3s0-r10:56:053
533699100,3cyclictest4554-21hddtemp_smartct11:11:033
533599108,1cyclictest0-21swapper/211:55:352
533599108,1cyclictest0-21swapper/209:00:342
533599107,2cyclictest307232sleep212:02:032
533599106,3cyclictest570-21runrttasks12:32:032
533599106,3cyclictest31656-21latency_hist08:05:452
533599105,4cyclictest527-21dbus-daemon10:06:132
533599105,4cyclictest14297-21cut07:26:072
533599105,3cyclictest6862-21cpuspeed_turbos08:21:022
533599105,2cyclictest20045-21kworker/u8:210:03:392
533599104,5cyclictest3852-21apt-get11:10:582
533599104,4cyclictest31944-21forks09:41:022
533599104,1cyclictest0-21swapper/210:15:462
533599103,1cyclictest0-21swapper/209:59:262
533599102,4cyclictest32049-21mailstats11:06:122
533599102,2cyclictest0-21swapper/210:51:262
533599102,2cyclictest0-21swapper/210:51:262
533599102,2cyclictest0-21swapper/210:41:182
533599101,8cyclictest27564-21tr07:55:592
533599101,4cyclictest4139-21rm12:07:152
533599100,9cyclictest11903-21if_enp5s011:46:032
533599100,4cyclictest29777-21ssh09:10:562
533599100,4cyclictest19246-21/usr/sbin/munin10:56:172
533599100,1cyclictest0-21swapper/211:34:152
533399107,2cyclictest29398-21diskmemload11:52:230
533399107,2cyclictest29398-21diskmemload09:38:180
533399107,2cyclictest1915-21acpi08:10:560
533399107,2cyclictest16570-21grep09:56:070
533399106,3cyclictest4284-21aten2_r8power_e08:15:570
533399106,3cyclictest29398-21diskmemload09:34:580
533399106,3cyclictest13262-21cstates12:16:040
533399106,3cyclictest1-21systemd11:01:160
533399106,2cyclictest570-21runrttasks10:41:080
533399105,4cyclictest16853-21cat11:21:150
533399104,5cyclictest575-21snmpd07:43:160
533399104,5cyclictest11876-21awk08:31:030
533399104,4cyclictest28453-21open_inodes09:06:040
533399103,5cyclictest22843-21cpu07:45:580
533399102,2cyclictest141rcu_preempt12:00:440
533399102,1cyclictest141rcu_preempt09:12:490
533399101,2cyclictest141rcu_preempt09:22:360
533399101,2cyclictest0-21swapper/009:51:300
533399100,9cyclictest0-21swapper/010:31:010
533399100,9cyclictest0-21swapper/010:19:410
533399100,9cyclictest0-21swapper/007:34:210
533399100,9cyclictest0-21swapper/007:34:210
256182100,1sleep00-21swapper/011:30:060
53369996,2cyclictest16914-21open_files08:41:073
53369991,6cyclictest13732-21grep08:35:593
53369990,3cyclictest556-21ntp_states08:06:063
53359995,3cyclictest25014-21tr07:51:002
53359995,2cyclictest26202-21ntp_states11:01:142
53359994,4cyclictest9783-21latency08:26:052
53359994,4cyclictest575-21snmpd09:06:102
53359994,4cyclictest0-21swapper/211:17:352
53359994,3cyclictest22897-21grep07:45:582
53359994,3cyclictest20648-21grep08:50:582
53359991,7cyclictest0-21swapper/208:41:112
53359991,2cyclictest13454-21/usr/sbin/munin08:36:002
53359991,2cyclictest0-21swapper/210:46:072
53359990,8cyclictest0-21swapper/207:24:042
53359990,4cyclictest2504-21gltestperf08:11:012
53359990,3cyclictest4218-21/usr/sbin/munin08:16:022
53359990,3cyclictest20353-21/usr/sbin/munin07:40:562
53359990,1cyclictest0-21swapper/209:04:242
53339996,2cyclictest7610-21df12:10:590
53339996,2cyclictest6012-21sh11:11:370
53339996,2cyclictest5166-21ssh09:46:020
53339996,2cyclictest29398-21diskmemload10:05:570
53339996,2cyclictest20792-21sh12:21:310
53339996,2cyclictest20478-21ntp_states10:56:140
53339996,2cyclictest13729-21ssh11:47:160
53339995,3cyclictest570-21runrttasks08:53:410
53339995,3cyclictest570-21runrttasks08:07:490
53339994,4cyclictest8867-21fschecks_time07:15:590
53339994,4cyclictest575-21snmpd09:01:420
53339994,4cyclictest14577-21sh08:36:070
53339994,3cyclictest25221-21cut07:51:010
53339994,3cyclictest10496-21unixbench_singl08:26:110
53339993,5cyclictest575-21snmpd11:35:470
53339993,5cyclictest575-21snmpd08:45:450
53339993,4cyclictest6852-21grep08:21:020
53339993,4cyclictest11428-21cat07:21:030
53339992,6cyclictest30870-21tr11:06:010
53339992,5cyclictest14510-21switchtime07:26:090
53339991,1cyclictest141rcu_preempt12:04:330
53339990,7cyclictest1348-21ssh12:05:460
53359986,1cyclictest18030-21/usr/sbin/munin07:35:582
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional