You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-23 - 09:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by a total of 2 SMIs that occured during the measurement.
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack8slot2s.osadl.org (updated Sat Sep 23, 2023 00:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
306599139131,7cyclictest0-21swapper/119:11:411
1836521050,0sleep10-21swapper/122:36:591
306499104102,1cyclictest0-21swapper/019:11:410
325112950,0sleep30-21swapper/322:50:483
93529441,39sleep20-21swapper/219:07:032
270429147,32sleep00-21swapper/019:10:300
273729047,32sleep30-21swapper/319:10:583
253428945,32sleep10-21swapper/119:08:191
149412880,0sleep10-21swapper/123:36:231
145532830,0sleep00-21swapper/000:39:350
3067997270,1cyclictest0-21swapper/319:20:533
3064994926,22cyclictest0-21swapper/023:56:550
3066994644,1cyclictest0-21swapper/219:20:532
302162330,0chrt0-21swapper/221:12:392
106992310,0sleep30-21swapper/321:27:073
193702300,1sleep30-21swapper/319:42:113
271922280,0sleep30-21swapper/321:42:343
271922280,0sleep30-21swapper/321:42:343
306499250,23cyclictest19306-21sh22:06:590
163882250,0sleep30-21swapper/322:02:543
306599241,22cyclictest0-21swapper/119:36:591
3064992321,1cyclictest0-21swapper/020:06:570
41432220,0sleep00-21swapper/023:25:360
306799210,1cyclictest0-21swapper/321:56:583
306699210,1cyclictest0-21swapper/223:56:562
306499204,8cyclictest1199-21awk00:26:390
306499180,10cyclictest25332-21basename21:06:390
306499175,6cyclictest19790-21perf21:36:390
306499174,7cyclictest30871-21perf23:51:390
3067991612,1cyclictest0-21swapper/319:11:413
306699165,6cyclictest0-21swapper/223:31:502
306499164,10cyclictest24683-21awk19:56:390
306499161,8cyclictest658-21munin-run22:51:390
306699158,5cyclictest5858-21cron22:56:392
306599153,11cyclictest0-21swapper/121:44:411
306599153,11cyclictest0-21swapper/121:44:411
306599151,1cyclictest0-21swapper/123:12:061
306499153,3cyclictest27669-21diskmemload22:30:340
200242150,1sleep30-21swapper/322:07:033
306799147,5cyclictest1-21systemd20:21:393
306699147,5cyclictest18846-21cron22:06:392
306699144,8cyclictest0-21swapper/222:16:422
306699144,5cyclictest0-21swapper/220:12:062
306599143,6cyclictest14659-21ls19:32:181
306599141,2cyclictest0-21swapper/121:51:221
306499147,5cyclictest29592-21dump-pmu-power20:06:390
113882140,1sleep20-21swapper/221:58:212
306799138,3cyclictest21477-21grep20:57:013
306799135,3cyclictest23567-21ssh21:39:503
306799134,7cyclictest7580-21ssh23:59:453
306799134,4cyclictest28759-21ssh23:48:303
306699136,5cyclictest23235-21ntpq23:12:072
306699134,4cyclictest0-21swapper/222:36:432
306699134,4cyclictest0-21swapper/220:47:482
306599138,3cyclictest5605-21tr23:27:001
306599137,4cyclictest4479-21cat23:56:581
306599136,5cyclictest7768-21cut00:32:091
306599134,8cyclictest0-21swapper/122:06:361
306599134,8cyclictest0-21swapper/120:41:211
306599134,4cyclictest0-21swapper/120:14:321
306599133,5cyclictest0-21swapper/122:11:331
306599131,5cyclictest0-21swapper/100:38:351
306499136,5cyclictest18672-21grep23:07:110
306499136,5cyclictest14672-21if_err_eno120:42:050
306499135,6cyclictest8680-21ssh21:24:350
306499135,6cyclictest867-21sh21:48:450
306499135,6cyclictest5874-21tune2fs23:27:020
306499135,6cyclictest30717-21ssh21:13:300
306499135,6cyclictest27963-21ssh21:43:440
306499135,6cyclictest27963-21ssh21:43:440
306499135,6cyclictest26316-21grep22:12:140
306499134,7cyclictest8839-21ssh22:58:130
306499132,7cyclictest29409-21ssh23:17:380
306499131,4cyclictest0-21swapper/000:07:020
306499131,11cyclictest0-21swapper/000:31:570
306799127,3cyclictest29815-21unixbench-2d21:12:133
306799126,4cyclictest7918-21systemctl00:32:103
306799126,4cyclictest514-21dbus-daemon00:41:393
306799126,4cyclictest18573-21head22:37:013
306799125,6cyclictest0-21swapper/320:41:513
306799124,6cyclictest13484-21grep19:32:053
306799124,5cyclictest0-21swapper/322:33:433
306799123,4cyclictest0-21swapper/322:22:243
306799121,6cyclictest0-21swapper/320:33:143
306799120,5cyclictest15814-21fschecks_time21:32:013
306699129,2cyclictest0-21swapper/223:27:092
306699127,3cyclictest530-21runrttasks21:30:062
306699127,3cyclictest5184-21cron19:16:382
306699127,3cyclictest13561-21cron20:41:392
306699126,4cyclictest26439-21systemctl19:57:102
306699126,4cyclictest240-21systemd-journal20:11:392
306699126,4cyclictest19079-21sed19:42:092
306699125,6cyclictest0-21swapper/221:51:412
306699125,5cyclictest19756-21cron21:36:392
306699124,3cyclictest0-21swapper/222:24:362
306599126,4cyclictest4850-21idleruntime-cro23:26:391
306599125,6cyclictest0-21swapper/123:06:351
306599125,5cyclictest0-21swapper/120:32:321
306599124,6cyclictest29761-21tail22:16:571
306599123,4cyclictest9365-21fschecks_time21:57:021
306599122,9cyclictest17517-21grep23:07:011
306599120,7cyclictest5394-21grep20:22:081
306599120,6cyclictest0-21swapper/122:47:121
306599120,11cyclictest0-21swapper/123:46:581
306499126,4cyclictest3206-21ssh21:51:540
306499125,5cyclictest4143-21cat22:22:180
306499125,5cyclictest31978-21sed23:52:040
306499125,5cyclictest27669-21sh21:11:390
306499125,5cyclictest24404-21aten2.4_r8power00:16:580
306499125,5cyclictest22992-21sendmail21:01:390
306499125,5cyclictest22507-21ssh22:41:290
306499125,5cyclictest19999-21awk19:46:390
306499125,5cyclictest19378-21diskmemload00:11:520
306499125,5cyclictest18274-21dump-pmu-power20:51:380
306499125,5cyclictest17621-21cron19:41:390
306499125,5cyclictest14110-21/usr/sbin/munin22:32:120
306499125,5cyclictest12306-21which23:02:010
306499125,5cyclictest11075-21grep21:27:110
306499125,5cyclictest10500-21ssh23:32:010
306499125,5cyclictest0-21swapper/020:39:270
306499124,6cyclictest6004-21munin-node19:16:410
306499124,6cyclictest5730-21ssh00:31:060
306499124,6cyclictest3745-21grep20:17:150
306499124,6cyclictest19738-21awk20:52:130
306499124,6cyclictest1629-21seq21:17:060
306499124,6cyclictest14523-21cstates22:02:010
306499124,6cyclictest12149-21ssh21:59:280
306499124,6cyclictest0-21swapper/020:00:520
306499123,7cyclictest1860-21awk20:16:390
306499123,4cyclictest24455-21ssh23:45:000
306799117,2cyclictest16446-21sh23:06:393
306799116,3cyclictest4748-21tr20:22:023
306799116,3cyclictest27233-21sendmail20:01:393
306799116,3cyclictest1444-21tr22:52:023
306799115,4cyclictest433-21wc23:52:143
306799114,5cyclictest607-21gdbus21:52:133
306799114,5cyclictest28408-21cut20:02:043
306799114,5cyclictest240-21systemd-journal00:26:393
306799114,5cyclictest0-21swapper/320:06:583
306799114,4cyclictest17525-21systemctl20:47:113
306799112,7cyclictest25590-21turbostat23:46:393
306799111,8cyclictest25842-21tr21:06:583
306799111,4cyclictest32578-21aten2.4_r8power23:21:593
306799111,3cyclictest5626-21tr23:27:003
306799111,2cyclictest0-21swapper/322:58:163
306799110,7cyclictest19478-21ssh23:40:213
306699116,4cyclictest23508-21cut21:01:582
306699116,4cyclictest12550-21tr00:37:082
306699116,3cyclictest8102-21grep22:57:132
306699116,3cyclictest530-21runrttasks21:23:382
306699116,3cyclictest2815-21idleruntime20:17:072
306699116,3cyclictest27669-21diskmemload00:27:102
306699116,3cyclictest13574-21dump-pmu-power20:46:342
306699115,5cyclictest0-21swapper/220:27:062
306699115,4cyclictest644-21ls22:51:392
306699115,4cyclictest26420-21cut21:42:092
306699115,4cyclictest26420-21cut21:42:092
306699115,4cyclictest10618-21basename19:26:382
306699115,4cyclictest1014-21cat23:22:072
306699114,5cyclictest547-21snmpd20:24:262
306699114,5cyclictest20958-21tail23:42:022
306699114,5cyclictest1892-21wc21:17:102
306699114,5cyclictest15451-21cut19:36:392
306699113,3cyclictest0-21swapper/200:32:402
306699111,8cyclictest24571-21tr00:16:592
306599117,2cyclictest18786-21cron00:11:381
306599116,3cyclictest9779-21sh21:26:271
306599116,3cyclictest30951-21ntp_kernel_pll_20:07:071
306599115,5cyclictest0-21swapper/119:31:361
306599115,4cyclictest29870-21grep21:12:131
306599115,4cyclictest20325-21basename19:46:591
306599114,5cyclictest0-21swapper/122:12:241
306599114,5cyclictest0-21swapper/121:08:131
306599114,5cyclictest0-21swapper/120:58:001
306599114,5cyclictest0-21swapper/119:19:361
306599114,2cyclictest0-21swapper/123:38:521
306599113,7cyclictest0-21swapper/121:36:581
306599113,5cyclictest0-21swapper/121:27:191
306599113,5cyclictest0-21swapper/120:54:061
306599113,3cyclictest24619-1kworker/u9:021:52:121
306599113,2cyclictest0-21swapper/122:52:341
306599112,6cyclictest0-21swapper/100:27:111
306599111,8cyclictest27774-21grep20:01:591
306599111,8cyclictest2426-21grep20:17:031
306499117,2cyclictest15557-21apt23:36:580
306499115,5cyclictest9734-21tr00:02:010
306499114,5cyclictest8363-21/usr/sbin/munin20:27:150
306499114,5cyclictest4542-21aten2.4_r8power20:22:000
306499114,5cyclictest29984-21cut22:16:580
306499114,5cyclictest24758-21ssh23:13:300
306499114,5cyclictest22887-21ssh22:41:420
306499114,5cyclictest21892-21unixbench_multi19:47:130
306499114,5cyclictest20006-21ssh21:36:410
306499114,5cyclictest13215-21/usr/sbin/munin19:32:040
306499114,5cyclictest11888-21cat19:27:090
306499113,6cyclictest10160-21missed_timers20:32:070
306499110,6cyclictest9013-21/usr/sbin/munin19:22:030
306799107,2cyclictest40550irq/33-enp3s0-r22:27:103
306799106,2cyclictest242582sleep323:12:453
306799106,2cyclictest19886-21sh23:08:593
306799106,2cyclictest10604-21sh19:26:383
306799105,4cyclictest23203-21head19:52:023
306799105,3cyclictest3932-21sh21:19:423
306799105,3cyclictest27669-21diskmemload00:07:083
306799105,3cyclictest26647-21sh22:12:423
306799105,3cyclictest20858-21gltestperf19:47:023
306799105,3cyclictest16773-21proc_pri19:37:133
306799105,3cyclictest1504-21sh00:26:503
306799105,3cyclictest0-21swapper/322:19:063
306799104,4cyclictest32268-21aten2.4_r8power20:12:013
306799104,4cyclictest30668-21cut21:46:593
306799104,4cyclictest10964-21aten2.4_r8power19:27:003
306799103,5cyclictest547-21snmpd22:42:073
306799102,6cyclictest8903-21munin-run20:31:393
306799102,4cyclictest20169-21/usr/sbin/munin00:12:013
306799101,4cyclictest27989-21ssh00:20:243
306799100,4cyclictest9912-21/usr/sbin/munin00:02:013
306799100,4cyclictest27344-21cut23:16:573
306799100,4cyclictest18958-21expr20:52:013
306799100,4cyclictest10429-21/usr/sbin/munin23:32:103
306699108,1cyclictest0-21swapper/200:07:092
306699107,2cyclictest14950irq/32-ahci[00023:17:062
306699106,2cyclictest29204-21uptime20:02:112
306699106,2cyclictest27669-21diskmemload23:48:112
306699105,4cyclictest0-21swapper/221:00:152
306699105,3cyclictest23792-21head22:42:042
306699105,3cyclictest19092-21md.daily00:11:482
306699105,3cyclictest0-21swapper/221:46:442
306699104,4cyclictest31277-21acpi23:51:582
306699104,4cyclictest26103-21cut21:07:002
306699104,4cyclictest240-21systemd-journal20:53:092
306699104,4cyclictest20313-21dump-pmu-power23:41:392
306699104,4cyclictest19474-21tail22:07:002
306699104,4cyclictest14038-21ntpq22:32:112
306699104,4cyclictest11982-21ntp_kernel_pll_19:27:112
306699103,5cyclictest547-21snmpd23:07:562
306699103,5cyclictest547-21snmpd21:38:022
306699103,5cyclictest547-21snmpd19:54:522
306699103,5cyclictest547-21snmpd00:24:252
306699103,5cyclictest514-21dbus-daemon00:02:092
306699102,4cyclictest0-21swapper/219:47:042
306699100,4cyclictest17639-21latency_hist19:41:392
306599106,3cyclictest13462-21gltestperf22:32:051
306599105,4cyclictest3418-21if_enp3s022:22:101
306599105,4cyclictest0-21swapper/120:44:361
306599105,3cyclictest21208-21grep00:12:111
306599104,5cyclictest8174-21df22:27:011
306599104,5cyclictest0-21swapper/121:02:061
306599104,5cyclictest0-21swapper/119:22:051
306599104,4cyclictest7116-21cut20:27:001
306599104,4cyclictest16853-21grep20:47:041
306599104,4cyclictest0-21swapper/123:42:031
306599104,4cyclictest0-21swapper/100:04:501
306599103,6cyclictest0-21swapper/122:56:401
306599103,6cyclictest0-21swapper/121:33:101
306599103,5cyclictest0-21swapper/119:58:071
306599103,1cyclictest0-21swapper/121:16:411
306599102,3cyclictest23917-21kworker/u8:123:16:441
306599102,2cyclictest0-21swapper/100:18:581
306599101,7cyclictest31929-21grep23:52:041
306599101,7cyclictest0-21swapper/119:41:591
306599101,6cyclictest25768-21ssh22:43:281
306599100,5cyclictest29400-21grep00:22:011
306499104,4cyclictest3654-21ssh22:53:160
116612100,1sleep20-21swapper/223:01:572
30679994,4cyclictest5498-21basename21:21:563
30679994,4cyclictest11496-21aten2.4_r8power20:37:003
30679994,3cyclictest23303-21aten2.4_r8power21:01:573
30669994,3cyclictest24859-21tr22:12:022
30669993,4cyclictest7729-21aten2.4-expect22:26:592
30669993,4cyclictest10751-21unixbench-2d20:32:132
30659991,7cyclictest0-21swapper/119:52:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional