You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-08 - 23:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by a total of 2 SMIs that occured during the measurement.
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack8slot2s.osadl.org (updated Fri Dec 08, 2023 12:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2862899141139,1cyclictest0-21swapper/207:12:492
2862699134132,1cyclictest0-21swapper/007:12:490
2862899131128,2cyclictest3748-21apt-get07:27:432
2862699123121,1cyclictest0-21swapper/007:27:430
608521030,1sleep30-21swapper/311:42:473
298892970,1sleep30-21swapper/311:34:143
2809929246,34sleep20-21swapper/207:09:252
2787029046,31sleep00-21swapper/007:08:140
2625629047,32sleep30-21swapper/307:07:533
2802128946,32sleep10-21swapper/107:08:241
201902840,0sleep20-21swapper/210:57:172
28629998280,1cyclictest0-21swapper/307:12:493
12602800,0sleep30-21swapper/309:25:123
228312750,0sleep10-21swapper/109:48:191
28629997068,1cyclictest0-21swapper/307:27:433
180802340,8sleep122-21ksoftirqd/110:18:161
118722290,1sleep00-21swapper/011:18:150
309802250,0sleep00-21swapper/012:03:170
275972250,1sleep30-21swapper/311:03:153
305742240,1sleep20-21swapper/212:33:092
28627992420,2cyclictest23037-21ssh11:28:131
12452231,15sleep10-21swapper/111:08:481
2862699226,9cyclictest1422-21awk07:22:420
2862699196,7cyclictest25590-21munin-run11:02:420
2862699196,7cyclictest25590-21munin-run11:02:420
2862799163,8cyclictest0-21swapper/112:01:111
2862999156,7cyclictest13129-21missed_timers11:48:123
2862999150,1cyclictest0-21swapper/307:18:013
2862699151,7cyclictest7159-21munin-run08:42:420
2862799144,7cyclictest3762-21sed10:38:171
2862799144,5cyclictest16132chrt10:01:211
2862799144,5cyclictest0-21swapper/107:13:081
2862799143,4cyclictest25554-1kworker/u9:209:23:351
2862799143,2cyclictest141rcu_preempt09:28:011
2862799142,8cyclictest21338-21ssh11:55:351
2862699144,2cyclictest316232unixbench_multi10:33:130
2862699140,8cyclictest30842-21apt-get11:07:420
2862999136,5cyclictest514-21dbus-daemon09:42:423
2862999136,5cyclictest240-21systemd-journal12:42:423
2862999136,5cyclictest240-21systemd-journal12:42:423
2862999136,5cyclictest20507-21cron10:22:423
2862999134,5cyclictest0-21swapper/310:58:093
2862999134,5cyclictest0-21swapper/310:58:093
2862999130,8cyclictest28921-21ssh10:31:413
2862799137,5cyclictest0-21swapper/108:04:511
2862799135,3cyclictest0-21swapper/108:47:471
2862799134,4cyclictest13019-21ssh11:19:251
2862799131,2cyclictest0-21swapper/112:29:151
2862799131,2cyclictest0-21swapper/110:29:271
2862799131,2cyclictest0-21swapper/107:39:411
2862999125,6cyclictest0-21swapper/309:49:043
2862999125,5cyclictest0-21swapper/311:15:553
2862999125,5cyclictest0-21swapper/309:07:123
2862999124,7cyclictest0-21swapper/309:02:183
2862999124,3cyclictest0-21swapper/310:39:593
2862999123,6cyclictest15476-21cron11:22:433
2862999123,4cyclictest0-21swapper/308:56:173
2862999122,7cyclictest14443-21sh11:21:193
2862999122,5cyclictest0-21swapper/309:33:073
2862899128,2cyclictest22345-21cut11:28:062
2862899125,5cyclictest13145-21grep10:48:172
2862899120,4cyclictest16536-21latency_hist09:02:422
2862799129,2cyclictest0-21swapper/109:33:151
2862799127,4cyclictest22-21ksoftirqd/109:02:421
2862799127,3cyclictest530-21runrttasks09:14:511
2862799126,4cyclictest547-21snmpd10:34:141
2862799126,4cyclictest31796-21head07:18:141
2862799125,5cyclictest530-21runrttasks10:07:441
2862799125,5cyclictest0-21swapper/110:02:521
2862799124,7cyclictest1804-21kworker/u8:012:38:051
2862799124,7cyclictest1804-21kworker/u8:012:38:051
2862799124,6cyclictest32615-21ssh12:05:211
2862799124,6cyclictest12955-21ssh12:17:151
2862799124,5cyclictest0-21swapper/111:36:571
2862799124,5cyclictest0-21swapper/107:43:191
2862799124,3cyclictest22-21ksoftirqd/110:26:531
2862799124,3cyclictest0-21swapper/110:53:121
2862799123,8cyclictest0-21swapper/108:41:191
2862799123,7cyclictest5438-21if_enp4s011:13:091
2862799123,5cyclictest0-21swapper/112:21:171
2862799123,4cyclictest0-21swapper/109:53:081
2862799122,6cyclictest0-21swapper/112:10:371
2862799122,5cyclictest0-21swapper/111:48:091
2862799122,5cyclictest0-21swapper/107:48:091
2862799121,8cyclictest27515-21needreboot09:18:091
2862799121,8cyclictest27515-21needreboot09:18:091
2862799121,6cyclictest18620-21ssh11:24:171
2862799121,6cyclictest0-21swapper/110:46:191
2862799121,2cyclictest0-21swapper/111:38:051
2862699127,4cyclictest7231-21diskmemload11:14:170
2862699126,4cyclictest547-21snmpd07:57:140
2862999117,2cyclictest7146-21sendmail08:42:423
2862999116,4cyclictest240-21systemd-journal08:32:423
2862999115,5cyclictest0-21swapper/310:54:303
2862999115,5cyclictest0-21swapper/310:01:283
2862999115,4cyclictest8622-21needreboot10:08:113
2862999115,4cyclictest28014-21latency_hist08:17:433
2862999115,4cyclictest24924-21smartctl11:58:153
2862999115,4cyclictest22198-21cstates09:13:073
2862999115,4cyclictest17925-21sort09:43:233
2862999114,6cyclictest4908-21ssh10:03:493
2862999114,5cyclictest845-21tr11:38:013
2862999114,5cyclictest0-21swapper/312:17:363
2862999114,5cyclictest0-21swapper/308:01:403
2862999113,3cyclictest0-21swapper/309:30:093
2862999113,3cyclictest0-21swapper/307:34:123
2862999112,4cyclictest4526-21ssh12:08:323
2862999111,8cyclictest27704-21ssh12:31:333
2862999111,5cyclictest0-21swapper/312:03:093
2862999111,3cyclictest32549-21sh08:27:423
2862999110,2cyclictest0-21swapper/312:17:493
2862899119,1cyclictest21449-21diskmemload10:00:572
2862899118,2cyclictest0-21swapper/207:20:182
2862899117,3cyclictest18053-21cpuspeed_turbos11:53:032
2862899117,2cyclictest7058-21sh10:07:422
2862899117,2cyclictest2697-21tr12:38:042
2862899117,2cyclictest2697-21tr12:38:042
2862899117,2cyclictest13028-21latency08:53:092
2862899116,4cyclictest4251-21grep12:08:152
2862899116,4cyclictest26021-21tr10:28:052
2862899116,3cyclictest2746-21needreboot07:23:102
2862899115,4cyclictest3305-21wc10:38:142
2862899115,4cyclictest22201-21cut08:03:422
2862899115,4cyclictest21643-21grep09:48:092
2862899115,4cyclictest17907-21tr09:03:092
2862899115,4cyclictest12778-21cut07:43:112
2862899114,5cyclictest7745-21grep08:43:052
2862899114,2cyclictest141rcu_preempt11:35:342
2862899111,9cyclictest0-21swapper/212:00:372
2862899111,8cyclictest11319-21apt-config07:42:422
2862899111,3cyclictest15944-21sh09:42:422
2862899111,2cyclictest0-21swapper/210:43:452
2862799116,3cyclictest11359-21sh10:48:001
2862799115,5cyclictest0-21swapper/108:53:501
2862799115,5cyclictest0-21swapper/108:24:291
2862799115,4cyclictest17678-21processes09:43:211
2862799114,6cyclictest0-21swapper/110:58:171
2862799114,6cyclictest0-21swapper/110:58:171
2862799114,5cyclictest23926-21cut08:08:091
2862799114,5cyclictest1195-21ssh12:36:471
2862799114,5cyclictest10619-21ssh11:46:311
2862799114,5cyclictest0-21swapper/108:29:411
2862799114,4cyclictest0-21swapper/111:03:051
2862799113,5cyclictest0-21swapper/109:40:451
2862799113,5cyclictest0-21swapper/109:03:071
2862799113,3cyclictest0-21swapper/112:23:091
2862799113,2cyclictest0-21swapper/107:31:491
2862799112,7cyclictest28764-21gltestperf08:18:061
2862799112,4cyclictest530-21runrttasks07:53:191
2862799112,2cyclictest0-21swapper/107:33:311
2862799111,8cyclictest240-21systemd-journal08:47:141
2862799110,3cyclictest0-21swapper/107:25:351
2862699118,2cyclictest12-21ksoftirqd/007:42:420
2862699117,2cyclictest317502sleep008:23:140
2862699116,4cyclictest5767-21cron11:42:420
2862699116,4cyclictest240-21systemd-journal10:43:140
2862699115,4cyclictest3193-21ntpq10:38:130
2862699115,4cyclictest20532-21basename10:22:420
2862699115,4cyclictest13248-21vmstat10:48:180
2862699115,4cyclictest12782-21hddtemp_smartct08:53:060
2862699114,5cyclictest16893-21cut09:43:140
2862699112,5cyclictest141rcu_preempt11:52:140
2862699111,4cyclictest89682/usr/sbin/munin10:08:140
2862699110,4cyclictest21449-21diskmemload12:11:490
2862999106,3cyclictest34-21ksoftirqd/307:53:033
2862999106,2cyclictest21449-21diskmemload10:48:083
2862999106,2cyclictest0-21swapper/307:38:013
2862999105,3cyclictest28701-21gltestperf08:18:063
2862999105,3cyclictest25499-21sh11:30:563
2862999105,3cyclictest22785-21sh12:25:133
2862999105,3cyclictest20856-21sh10:22:543
2862999104,5cyclictest17582-21ssh11:52:523
2862999104,5cyclictest0-21swapper/310:43:003
2862999104,5cyclictest0-21swapper/307:25:303
2862999104,4cyclictest547-21snmpd08:42:503
2862999104,4cyclictest24372-21latency08:08:133
2862999104,4cyclictest21761-21gltestperf08:03:383
2862999103,5cyclictest547-21snmpd08:49:323
2862999102,3cyclictest0-21swapper/309:53:073
2862999101,7cyclictest27027-21gltestperf09:18:053
2862999101,7cyclictest27027-21gltestperf09:18:053
2862999101,5cyclictest31634-21unixbench_multi10:33:133
2862999101,2cyclictest0-21swapper/307:47:043
2862999100,9cyclictest0-21swapper/310:13:093
2862999100,4cyclictest3256-21tune2fs08:33:093
2862999100,4cyclictest29962-21irqstats12:33:033
2862899106,3cyclictest19606-21sh11:25:372
2862899106,3cyclictest12511-21kworker/u8:012:18:092
2862899105,3cyclictest7085-21sh09:32:482
2862899105,3cyclictest530-21runrttasks11:44:422
2862899105,3cyclictest530-21runrttasks11:18:202
2862899105,3cyclictest530-21runrttasks08:48:182
2862899105,3cyclictest29898-21ssh10:33:002
2862899105,3cyclictest28-21ksoftirqd/208:28:092
2862899105,3cyclictest26394-21grep09:53:062
2862899105,3cyclictest26052-21tr08:13:052
2862899105,3cyclictest21854-21sh09:13:042
2862899105,3cyclictest21449-21diskmemload09:17:542
2862899105,3cyclictest21449-21diskmemload09:17:542
2862899105,3cyclictest21160-21cpuspeed_turbos10:58:052
2862899105,3cyclictest21160-21cpuspeed_turbos10:58:052
2862899105,3cyclictest20079-21sh10:21:592
2862899105,3cyclictest14632-21gltestperf07:48:072
2862899105,3cyclictest11926-21sh11:47:582
2862899104,5cyclictest32277-21kworker/u8:209:31:332
2862899104,4cyclictest8855-21sed12:13:092
2862899104,4cyclictest3840-21tr08:33:142
2862899104,4cyclictest32055-21sort11:08:062
2862899104,4cyclictest31035-21sort08:23:062
2862899104,4cyclictest28245-21cut08:18:042
2862899104,4cyclictest26452-21ssh12:29:512
2862899104,4cyclictest24806-21wc08:08:182
2862899103,5cyclictest8169-21gltestperf10:08:062
2862899103,5cyclictest547-21snmpd11:12:452
2862899103,5cyclictest547-21snmpd11:03:572
2862899103,5cyclictest22711-21grep10:23:182
2862899103,5cyclictest21359-21users12:23:182
2862899103,5cyclictest20429-21vmstat07:58:162
2862899103,5cyclictest19513-21aten2.4-expect09:08:042
2862899103,5cyclictest18140-21awk09:43:242
2862899103,5cyclictest11809-21tr10:13:052
2862899101,7cyclictest0-21swapper/208:38:062
2862899100,9cyclictest0-21swapper/207:53:072
2862899100,8cyclictest5779-21latency_hist11:42:422
2862899100,2cyclictest0-21swapper/209:25:422
2862799105,4cyclictest0-21swapper/110:12:491
2862799104,5cyclictest0-21swapper/109:07:531
2862799104,5cyclictest0-21swapper/108:32:431
2862799104,4cyclictest19365-21gltestperf07:58:071
2862799101,1cyclictest0-21swapper/108:12:471
2862699107,2cyclictest26930-21ntp_kernel_pll_10:28:130
2862699107,2cyclictest21449-21diskmemload12:28:490
2862699107,2cyclictest21449-21diskmemload12:15:000
2862699107,2cyclictest21449-21diskmemload11:46:030
2862699107,2cyclictest21449-21diskmemload10:04:270
2862699107,2cyclictest21449-21diskmemload09:39:480
2862699107,2cyclictest21449-21diskmemload09:13:400
2862699107,2cyclictest20158-21threads07:58:150
2862699106,3cyclictest3860-21if_err_enp5s009:28:070
2862699106,3cyclictest12-21ksoftirqd/010:54:070
2862699106,2cyclictest19009-21sh11:53:150
2862699105,4cyclictest21055-21aten2.4_r8power10:23:050
2862699105,4cyclictest19946-21cat12:23:060
2862699105,4cyclictest15988-21apt-get11:23:020
2862699105,3cyclictest25772-21aten2.4-expect09:53:020
2862699104,4cyclictest31503-21load09:58:110
2862699104,4cyclictest240-21systemd-journal09:02:420
2862699104,4cyclictest14060-21aten2.4-expect07:48:040
2862699103,5cyclictest28098-21aten2.4_r8power08:18:030
2862699103,1cyclictest12-21ksoftirqd/011:08:160
2862699102,2cyclictest141rcu_preempt11:57:430
2862699101,7cyclictest0-21swapper/008:37:420
2862699101,2cyclictest21452-21munin-run09:12:420
2862699100,3cyclictest21449-21diskmemload12:18:570
262682100,1sleep00-21swapper/011:31:580
286299995,2cyclictest31949-21expr11:08:053
286299993,4cyclictest514-21dbus-daemon09:07:433
286299990,3cyclictest14811-21if_enp4s007:48:093
286289994,3cyclictest530-21runrttasks07:35:442
286289993,4cyclictest29654-21ssh12:03:052
286269996,2cyclictest8554-21ntp_states08:43:120
286269996,2cyclictest7183-21acpi09:32:590
286269996,2cyclictest530-21runrttasks08:03:050
286269996,2cyclictest23518-21aten2.4_r8power08:08:060
286269996,2cyclictest2293-21sh12:37:570
286269996,2cyclictest2293-21sh12:37:570
286269996,2cyclictest17751-21latency09:03:070
286269995,3cyclictest27537-21needreboot09:18:090
286269995,3cyclictest27537-21needreboot09:18:090
286269995,3cyclictest27117-21sshd11:32:480
286269995,3cyclictest26880-1kworker/u9:012:32:440
286269995,3cyclictest2067-21grep08:28:140
286269995,2cyclictest12202-21grep07:43:070
286269994,4cyclictest7008-21cpuspeed_turbos07:33:010
286269994,4cyclictest31755-21interrupts09:23:110
286269994,4cyclictest26895-21ntpq08:13:130
286269994,4cyclictest11286-21ls08:48:220
286269993,5cyclictest547-21snmpd09:47:460
286269993,4cyclictest1955-21cpuspeed_turbos07:23:040
286269990,4cyclictest21449-21diskmemload10:12:540
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional