You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-10-07 - 12:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackaslot8s.osadl.org (updated Fri Oct 07, 2022 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11162178107,27sleep20-21swapper/219:09:272
10072177105,28sleep00-21swapper/019:07:590
10232174102,29sleep10-21swapper/119:08:131
8452162112,27sleep30-21swapper/319:05:583
116327557,11sleep00-21swapper/019:10:000
23042740,1chrt2305-21/usr/sbin/munin23:25:193
36912590,2sleep30-21swapper/300:30:293
130692590,2sleep20-21swapper/219:35:182
120362530,3chrt12-21ksoftirqd/023:35:160
120362530,3chrt12-21ksoftirqd/023:35:150
132922520,1sleep10-21swapper/119:35:201
142872490,4sleep20-21swapper/223:05:282
31142470,1sleep20-21swapper/221:23:132
193402440,1sleep30-21swapper/319:50:153
1437994411,23cyclictest11247-21sh22:01:232
143799438,19cyclictest22675-21sh23:14:362
265962410,1sleep10-21swapper/120:05:271
265962410,1sleep10-21swapper/120:05:261
143899414,24cyclictest55350irq/142-enp3s0-21:35:153
1438994118,7cyclictest11616-21perl00:05:213
143699398,11cyclictest30376-21ssh21:18:251
143799380,7cyclictest29686-21ssh21:17:272
1436993811,12cyclictest20892-21diskmemload21:53:411
143799379,13cyclictest30753-21sh21:49:572
143799371,5cyclictest2110-21sh22:24:402
1437993710,15cyclictest20892-21diskmemload22:46:232
143699379,12cyclictest30100-21ssh00:25:131
143699379,11cyclictest4259-21ssh21:24:501
143699379,10cyclictest11699-21awk21:30:311
1436993711,16cyclictest24167-21sh00:18:471
1436993711,10cyclictest32200-21sh22:21:221
143799361,3cyclictest20892-21diskmemload21:12:072
143799360,6cyclictest21472-21ssh22:11:032
143699368,10cyclictest17660-21ssh00:10:461
1436993611,15cyclictest2239-21sh22:55:131
1437993514,18cyclictest14381-21cat22:05:152
143699359,11cyclictest23881-21ssh22:14:311
143699359,11cyclictest23741-21ssh21:42:261
143699359,11cyclictest20248-21sed19:50:271
143699359,10cyclictest24728-21ssh23:46:211
143699358,11cyclictest23117-21ssh21:10:411
143699358,10cyclictest15641-21sed22:05:291
1436993511,9cyclictest5328-21ssh23:28:371
1436993511,10cyclictest9505-21seq00:03:121
143899342,13cyclictest7544-21ssh23:00:173
143799341,3cyclictest923-21kworker/u8:022:29:402
143799340,6cyclictest19942-21ssh23:41:532
143699349,11cyclictest1701-21cron22:55:001
143699348,9cyclictest19186-21fschecks_time22:40:181
143699348,8cyclictest30812-21ssh23:52:311
143699348,8cyclictest16688-21ssh23:08:321
143699348,8cyclictest11948-21ssh22:02:231
143699347,10cyclictest29358-21ssh21:47:561
1436993411,9cyclictest32696-21ssh23:23:561
1436993411,9cyclictest26619-21sh22:46:491
143699339,8cyclictest9127-21ssh23:31:301
143699339,15cyclictest25106-21seq20:03:491
143699338,8cyclictest4248-21ssh00:31:011
143699338,11cyclictest8844-21sh21:28:501
143699338,10cyclictest23447-21sort23:15:141
143699337,18cyclictest28749-21ssh22:19:011
143699336,16cyclictest0-21swapper/119:10:011
1436993311,8cyclictest6951-21chrt21:57:361
1436993311,8cyclictest17759-21sh21:36:311
1436993310,6cyclictest13344-21ssh22:35:111
143799328,21cyclictest6999-21cat23:30:182
143799321,5cyclictest32759-21cut20:20:222
143799320,5cyclictest12448-21switchtime00:05:312
143799320,3cyclictest32344-21taskset23:23:262
143699327,9cyclictest20892-21diskmemload23:04:421
1436993210,9cyclictest20892-21diskmemload00:20:071
143599320,29cyclictest8053-21ssh21:27:420
143799318,21cyclictest14996-21expr19:40:132
143799318,20cyclictest23039-21cron20:04:552
1437993110,18cyclictest5491-21if_enp3s021:25:212
143799310,9cyclictest8850-21memory20:40:222
143799310,6cyclictest19960-21ls21:05:272
143799310,6cyclictest14378-21taskset20:54:182
143799310,5cyclictest8806-21chrt00:02:022
143699319,9cyclictest498-21dbus-daemon23:35:261
143699319,9cyclictest498-21dbus-daemon23:35:251
143699319,7cyclictest1314-21taskset20:24:171
143699318,9cyclictest12190-21sh22:33:521
143699317,7cyclictest0-21swapper/123:56:211
143699317,14cyclictest0-21swapper/123:41:001
143699316,8cyclictest21331-21ssh23:12:411
1436993110,8cyclictest641-21runrttasks20:56:381
143799302,25cyclictest2261-21sed23:55:292
1437993015,13cyclictest3947-21sh00:30:322
1437993011,16cyclictest0-21swapper/221:35:142
143799300,5cyclictest8846-21rm00:36:072
143699308,8cyclictest4986-21ssh22:26:131
143699308,8cyclictest10928-21ssh00:39:341
143699308,10cyclictest22699-21taskset19:55:371
143899290,26cyclictest8972-21apt-get22:00:153
143799297,19cyclictest25934-21gltestperf20:05:202
143799297,19cyclictest25934-21gltestperf20:05:192
143799297,19cyclictest1822-21fschecks_count23:25:162
143799295,21cyclictest14883-21ls20:55:172
1437992914,12cyclictest27145-21sh22:16:412
1437992914,12cyclictest19766-21open_inodes22:40:252
1437992913,14cyclictest28178-21latency_hist23:20:002
143799291,25cyclictest32148-21rm00:26:152
1437992912,14cyclictest15679-21sh22:36:182
1437992910,15cyclictest4370-21idleruntime20:30:242
143799290,5cyclictest22433-21/usr/sbin/munin23:45:222
143799290,3cyclictest31206-21sh23:53:022
143799290,3cyclictest15683-21chrt23:38:192
143799290,3cyclictest15683-21chrt23:38:182
143699295,9cyclictest0-21swapper/120:25:251
1436992913,13cyclictest12658-21grep00:05:311
143599291,4cyclictest32763-21ssh22:22:100
143599291,26cyclictest0-21swapper/021:20:180
143599291,25cyclictest31110-21ssh21:19:280
143799289,17cyclictest9668-21unixbench-2d22:30:302
143799289,16cyclictest21920-21load19:55:192
143799286,19cyclictest4036-21expr21:55:132
143799281,24cyclictest11827-21ssh23:04:102
143799280,4cyclictest22286-21ssh00:15:392
143799280,26cyclictest24499-21rm21:43:312
143799280,25cyclictest6083-21rm22:58:352
143699289,5cyclictest5432-21taskset19:19:071
143699287,8cyclictest16355-21seq19:41:381
1435992822,4cyclictest0-21swapper/021:33:200
143599281,26cyclictest0-21swapper/022:06:020
143599281,25cyclictest29199-21ssh22:19:370
143599281,25cyclictest0-21swapper/022:04:210
1438992723,2cyclictest0-21swapper/321:21:263
1438992722,3cyclictest0-21swapper/321:27:373
143899271,24cyclictest0-21swapper/323:22:123
143899270,24cyclictest20892-21diskmemload21:30:593
143799279,15cyclictest0-21swapper/200:20:032
143799277,17cyclictest6048-21expr19:20:162
143799275,18cyclictest17506-21if_err_enp3s019:45:182
143799270,4cyclictest481-21unixbench-2d21:50:472
143799270,4cyclictest4278-21irqstats19:15:232
143799270,3cyclictest32165-21ssh22:52:142
143799270,3cyclictest28167-21hddtemp_smartct20:10:182
1435992723,2cyclictest23336-21ssh22:13:440
1435992721,4cyclictest20892-21diskmemload00:16:570
1435992720,4cyclictest15690-21diskstats21:35:170
143599271,25cyclictest0-21swapper/023:24:140
143599271,24cyclictest0-21swapper/023:06:160
143899261,2cyclictest0-21swapper/321:16:483
143899261,24cyclictest0-21swapper/323:50:483
143899261,23cyclictest0-21swapper/322:06:173
143899261,23cyclictest0-21swapper/300:16:573
143799269,14cyclictest10905-21fw_packets19:30:202
1437992623,2cyclictest0-21swapper/221:30:182
143799260,3cyclictest30305-21hddtemp_smartct20:15:192
143799260,3cyclictest17875-21ssh00:11:062
143799260,3cyclictest10570-21df_inode20:45:182
143699267,7cyclictest10006-21apt-get19:30:021
143699265,13cyclictest0-21swapper/120:15:031
143699263,14cyclictest0-21swapper/120:40:221
1436992613,10cyclictest17748-21/usr/sbin/munin21:00:291
1436992613,10cyclictest17748-21/usr/sbin/munin21:00:281
1435992624,1cyclictest0-21swapper/023:03:560
1435992617,6cyclictest14002-21hddtemp_smartct22:35:180
143599261,24cyclictest0-21swapper/023:19:410
143599261,24cyclictest0-21swapper/023:14:520
1438992522,2cyclictest0-21swapper/323:36:303
1438992522,2cyclictest0-21swapper/323:36:303
143899251,23cyclictest0-21swapper/322:13:543
143899250,24cyclictest0-21swapper/323:17:333
143799253,16cyclictest0-21swapper/219:50:142
143799252,6cyclictest16847-21find21:00:192
143799252,6cyclictest16847-21find21:00:182
143799251,3cyclictest3353-21cron19:15:002
143799250,6cyclictest6782-21uname20:35:232
143699254,8cyclictest0-21swapper/119:20:221
1436992519,4cyclictest7811-21turbostat.cron20:40:001
1436992515,7cyclictest12686-21cut20:50:171
1435992523,1cyclictest0-21swapper/021:49:060
143599251,1cyclictest0-21swapper/022:34:170
143599250,24cyclictest0-21swapper/021:58:210
226382240,11sleep012-21ksoftirqd/022:43:390
1438992422,1cyclictest0-21swapper/322:45:553
1438992422,1cyclictest0-21swapper/322:23:173
143899241,22cyclictest0-21swapper/323:05:593
143899241,1cyclictest0-21swapper/323:13:263
143899240,22cyclictest0-21swapper/322:15:043
143799240,5cyclictest9833-21runrttasks19:27:592
1436992414,7cyclictest8283-21/usr/sbin/munin19:25:141
1436992413,8cyclictest4318-21sed20:30:241
143899230,1cyclictest0-21swapper/322:42:123
143899230,1cyclictest0-21swapper/321:58:233
143899230,1cyclictest0-21swapper/321:49:173
143799231,17cyclictest1646-21apt-get20:25:142
1436992315,6cyclictest27990-21fschecks_count20:10:181
1436992314,7cyclictest10391-21tr20:45:171
1436992310,10cyclictest18988-21taskset21:05:151
143599237,4cyclictest790250lldpd21:13:580
1435992313,7cyclictest1967-21df20:25:160
1438992213,6cyclictest4201-21grep20:30:223
1435992220,1cyclictest0-21swapper/023:50:480
1435992220,1cyclictest0-21swapper/023:49:080
143899211,14cyclictest30113-21sh00:25:143
1436992111,8cyclictest17133-21expr19:45:151
143599219,10cyclictest1699-21idleruntime-cro22:55:010
143599212,13cyclictest1443-21ssh23:25:120
1435992113,6cyclictest7238-21ls20:35:280
1435992112,6cyclictest11123-21seq00:05:170
45112200,2sleep30-21swapper/323:58:223
143899207,11cyclictest14925-21ssh22:35:273
143899203,7cyclictest16597-21turbostat21:00:013
143899202,12cyclictest15867-21apt-get00:10:153
1438992014,3cyclictest18762-21sh19:50:003
1438992012,6cyclictest7663-21cron19:25:013
1438992011,7cyclictest23190-21cut23:45:173
143599207,10cyclictest24018-21ssh22:45:160
1435992013,5cyclictest12638-21ls20:50:160
1435992012,6cyclictest19961-21awk19:50:230
1435992012,6cyclictest15056-21forks20:55:200
177722190,3chrt17734-21ssh23:40:180
143899193,10cyclictest6199-21ssh00:00:203
143899191,8cyclictest11643-21sort23:35:003
1438991912,4cyclictest31206-21/usr/sbin/munin21:50:423
1438991911,5cyclictest28136-21sed20:10:193
1438991911,5cyclictest2638-21sh22:55:163
143599199,7cyclictest1929-21cat23:55:260
143599192,12cyclictest10440-21apt-get19:30:170
1435991913,4cyclictest2168-21sh00:30:020
1435991912,5cyclictest9424-21ls19:25:260
1435991912,5cyclictest25863-21cut20:05:180
1435991912,5cyclictest25863-21cut20:05:180
1435991911,6cyclictest6945-21cat23:30:180
1435991911,6cyclictest5807-21cat20:35:000
1435991911,6cyclictest2781-21fw_packets22:55:170
143599191,15cyclictest201762dump-pmu-power00:15:000
1435991911,5cyclictest18124-21timerandwakeup21:00:340
1435991911,5cyclictest18124-21timerandwakeup21:00:340
1435991910,6cyclictest932-21grep20:20:280
1435991910,6cyclictest7079-21cat19:20:270
143899188,7cyclictest11140-21latency_hist00:40:003
143899187,3cyclictest141rcu_preempt22:29:593
143899182,11cyclictest18579-21ssh23:40:263
1438991813,3cyclictest16844-21/usr/sbin/munin21:00:353
1438991813,3cyclictest16844-21/usr/sbin/munin21:00:343
1438991813,3cyclictest11409-21swap20:45:283
1438991811,5cyclictest26553-21awk00:20:263
1438991811,5cyclictest11643-21wc19:30:293
1438991811,4cyclictest4910-21sort19:15:293
1438991811,4cyclictest20892-21diskmemload22:32:063
1438991811,4cyclictest20892-21diskmemload21:41:163
1438991810,6cyclictest7245-21wc20:35:273
1438991810,6cyclictest1973-21expr19:10:173
143599182,11cyclictest2741-21apt-get22:25:120
1435991813,3cyclictest6830-21idleruntime00:00:250
1435991813,3cyclictest5549-21idleruntime-cro19:20:000
1435991813,3cyclictest32072-21kernelversion21:50:360
1435991813,3cyclictest29935-21sh00:25:000
1435991813,3cyclictest13250-21kernelversion19:35:210
1435991811,5cyclictest9522-21grep20:40:290
1435991811,5cyclictest7298-21expr00:35:170
1435991811,5cyclictest20888-21ls21:40:160
1435991811,5cyclictest17962-21gzip19:45:250
143599181,12cyclictest641-21runrttasks00:20:040
1435991810,6cyclictest22651-21awk19:55:280
1435991810,6cyclictest11536-21cat20:45:290
143899179,6cyclictest15862-21mailstats19:40:223
1438991712,3cyclictest8670-21hddtemp_smartct19:25:183
1438991712,3cyclictest641-21runrttasks21:11:233
1438991710,4cyclictest30616-21timerandwakeup22:50:273
143599172,11cyclictest19005-21apt-get21:05:160
143599172,10cyclictest29748-21apt-get20:15:140
1435991710,5cyclictest28388-21sed20:10:210
1435991710,5cyclictest16227-21awk19:40:250
143899169,5cyclictest13840-21expr19:35:273
143899168,5cyclictest9351-21systemd-journal21:10:003
1438991612,2cyclictest21145-21/usr/sbin/munin19:55:173
143899161,10cyclictest12129-21apt-get20:50:033
1438991610,4cyclictest8999-21ntp_offset20:40:243
1438991610,4cyclictest1621-21/usr/sbin/munin20:25:233
143599168,6cyclictest23561-21expr20:00:180
143899158,5cyclictest26909-21awk20:05:303
143899158,5cyclictest26909-21awk20:05:293
143899158,4cyclictest30291-21tail20:15:183
1438991510,3cyclictest31886-21/usr/sbin/munin20:20:253
1438991510,3cyclictest23994-21kernelversion20:00:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional