You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-19 - 18:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Atom(TM) Processor E3940 @ 1.60GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack5slot3s.osadl.org (updated Sun Sep 19, 2021 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
266562236161,23sleep00-21swapper/007:07:060
266092233165,56sleep20-21swapper/207:06:312
266002204169,23sleep30-21swapper/307:06:243
265732202165,24sleep10-21swapper/107:06:031
2714099105102,2cyclictest0-21swapper/212:22:042
2714099105102,2cyclictest0-21swapper/212:22:042
27140997471,2cyclictest0-21swapper/211:48:572
285502550,2sleep00-21swapper/007:10:270
2714599422,11cyclictest31083-21/usr/sbin/munin08:30:253
2712999417,31cyclictest27153-21munin-run12:00:000
27145994014,6cyclictest28653-21sendmail_mailtr12:00:333
27140993936,2cyclictest0-21swapper/211:08:042
2712999395,31cyclictest3938-21perf09:55:010
2712999391,10cyclictest6144-21/usr/sbin/munin12:25:250
2712999381,19cyclictest29357-21kworker/0:112:05:200
2714599372,11cyclictest17495-21/usr/sbin/munin10:25:243
2712999373,12cyclictest14028-21kworker/0:311:35:310
2712999371,10cyclictest26623-21/usr/sbin/munin08:20:190
2712999369,4cyclictest11540-21/usr/sbin/munin12:35:220
2712999368,13cyclictest5167-21awk08:45:010
2712999361,5cyclictest11008-21wc07:45:010
27129993611,3cyclictest5376-21/usr/sbin/munin08:45:210
2712999351,6cyclictest26960-21tail09:30:440
2712999348,2cyclictest86550irq/126-enp1s0-11:45:290
2712999341,20cyclictest6259-21kworker/0:107:45:280
27129993411,3cyclictest18727-21/usr/sbin/munin09:15:270
2712999336,3cyclictest14041-21/usr/sbin/munin11:30:180
2712999332,20cyclictest26991-21kworker/0:012:12:510
2712999331,7cyclictest15904-21tr10:20:230
27129993312,3cyclictest19155-21/usr/sbin/munin11:40:180
27129993312,2cyclictest836-21runrttasks08:55:010
27129993312,2cyclictest836-21runrttasks08:55:010
27129993311,2cyclictest15035-21crond10:20:010
27129993310,3cyclictest26231-21cat10:45:020
2714599328,7cyclictest0-21swapper/309:08:173
27145993220,3cyclictest0-21swapper/308:53:543
27145993220,3cyclictest0-21swapper/308:53:533
27145993219,3cyclictest0-21swapper/307:29:543
2712999329,3cyclictest906-21/usr/sbin/munin08:35:190
2712999327,2cyclictest0-21swapper/012:31:410
2712999321,2cyclictest0-21swapper/007:25:370
27129993211,3cyclictest23804-21gltestperf09:25:160
27129993211,3cyclictest17540-21cat08:00:010
2712999321,12cyclictest12148-21taskset11:25:140
2714599319,4cyclictest2209-21sensors_temp08:35:293
2712999319,3cyclictest25151-21if_enp2s008:15:210
2712999316,5cyclictest0-21swapper/007:15:300
2712999312,3cyclictest86750irq/128-enp1s0-10:05:270
2712999311,5cyclictest0-21swapper/009:58:410
27129993111,3cyclictest25010-21date11:55:010
27129993110,3cyclictest4400-21/usr/sbin/munin07:30:320
27129993110,3cyclictest18053-21cat08:00:260
2712999310,10cyclictest6259-21kworker/0:107:50:330
27145993017,3cyclictest0-21swapper/307:17:393
2714599301,18cyclictest0-21swapper/311:55:203
2712999309,4cyclictest6607-21/usr/sbin/munin11:10:290
2712999309,4cyclictest6607-21/usr/sbin/munin11:10:290
2712999309,3cyclictest8482-21sort10:05:020
2712999309,3cyclictest8482-21sort10:05:010
2712999309,3cyclictest29438-21gltestperf08:25:190
2712999308,4cyclictest23453-21sendmail_mailtr08:10:390
2712999301,6cyclictest22024-21cut08:10:010
2712999301,5cyclictest30369-21taskset09:40:160
13112300,3sleep20-21swapper/207:20:332
2714599299,7cyclictest16850-21fschecks_time11:35:243
2712999299,3cyclictest27803-21cstates09:35:170
2712999297,1cyclictest0-21swapper/010:55:170
2712999295,6cyclictest0-21swapper/012:20:180
2712999295,6cyclictest0-21swapper/012:20:180
2712999294,9cyclictest6620-21/usr/sbin/munin07:35:360
2712999294,2cyclictest0-21swapper/010:30:390
2712999293,6cyclictest1973-21smart_sda11:00:280
2712999292,2cyclictest836-21runrttasks11:07:360
2712999292,11cyclictest15859-21unixbench_singl09:05:320
2712999291,6cyclictest0-21swapper/009:10:170
2712999291,3cyclictest0-21swapper/011:17:050
2712999290,9cyclictest0-21swapper/012:15:200
27145992824,2cyclictest0-21swapper/308:05:023
27145992816,7cyclictest0-21swapper/308:13:453
27145992815,3cyclictest0-21swapper/311:47:313
2714599280,12cyclictest0-21swapper/308:55:113
2712999289,3cyclictest11684-21cut11:25:010
2712999286,19cyclictest31741-21gltestperf08:30:270
2712999281,9cyclictest0-21swapper/010:10:350
2712999281,8cyclictest0-21swapper/012:00:190
2712999281,3cyclictest0-21swapper/010:35:290
2712999281,3cyclictest0-21swapper/009:20:510
27129992810,3cyclictest27903-21sed10:45:290
27129992810,3cyclictest12244-21cstates09:00:300
27129992810,3cyclictest11544-21sshd08:56:430
2714599279,4cyclictest5495-21cat09:55:273
27145992715,3cyclictest0-21swapper/309:03:033
27145992714,5cyclictest0-21swapper/308:40:253
2714599270,13cyclictest0-21swapper/312:17:273
2713699271,8cyclictest0-21swapper/112:32:281
27136992716,3cyclictest18655-21date08:00:401
2713699271,20cyclictest9587-21kworker/u8:211:35:261
2714599267,3cyclictest0-21swapper/310:46:263
2714599264,3cyclictest0-21swapper/311:33:083
2713699266,11cyclictest31588-21unixbench-2d07:15:351
2712999261,8cyclictest0-21swapper/010:50:170
2712999261,6cyclictest0-21swapper/009:46:340
2712999261,4cyclictest18211-21basename10:25:200
2714599258,3cyclictest24010-21ntp_states11:50:253
2714599254,2cyclictest0-21swapper/309:10:243
2714599252,11cyclictest32315-21/usr/sbin/munin07:20:183
2714599251,6cyclictest0-21swapper/310:20:363
27145992514,5cyclictest0-21swapper/312:26:013
2714599251,12cyclictest3576-21sshd09:50:433
2714599251,11cyclictest0-21swapper/307:45:493
27140992512,6cyclictest28583-21cut12:00:332
2714099251,10cyclictest0-21swapper/208:20:402
27136992512,11cyclictest6074-21kworker/u8:010:15:021
2713699251,17cyclictest21939-21chrt08:10:001
27129992511,3cyclictest0-21swapper/007:20:390
2714599249,3cyclictest29843-21/usr/sbin/munin09:40:243
2714599247,14cyclictest0-21swapper/312:30:233
2714599245,3cyclictest0-21swapper/309:19:093
2714599241,5cyclictest848-21sssd_nss09:20:263
27145992415,4cyclictest0-21swapper/308:07:293
27145992412,3cyclictest346-21chrt10:59:573
2714599241,14cyclictest0-21swapper/308:18:093
2714599241,11cyclictest0-21swapper/311:14:543
2714599241,11cyclictest0-21swapper/311:14:533
27140992417,4cyclictest26775-21timerandwakeup09:30:442
2713699243,9cyclictest0-21swapper/112:15:351
2713699241,17cyclictest0-21swapper/110:37:491
15502240,1sleep30-21swapper/311:00:223
2714599239,3cyclictest0-21swapper/311:24:053
2714599236,4cyclictest7293-21memory10:00:213
2714599236,4cyclictest7293-21memory10:00:213
2714599236,11cyclictest0-21swapper/308:48:213
2714599232,10cyclictest29287-21hddtemp_smartct10:50:173
2714599231,3cyclictest30407-21if_err_enp2s012:05:193
2714599231,19cyclictest0-21swapper/311:17:073
2714599231,11cyclictest0-21swapper/310:41:333
2714599231,10cyclictest4397-21runrttasks07:30:173
27145992310,9cyclictest12725-21ls11:25:193
27145992310,7cyclictest0-21swapper/308:22:133
27136992312,3cyclictest1748-21irqstats08:35:221
2713699231,19cyclictest23047-21cpuspeed_turbos11:50:141
2713699231,19cyclictest119132sleep110:10:351
2713699231,17cyclictest0-21swapper/107:55:181
2713699230,6cyclictest0-21swapper/110:03:561
2713699230,6cyclictest0-21swapper/110:03:551
2714599226,4cyclictest0-21swapper/307:40:233
2714599222,2cyclictest86750irq/128-enp1s0-10:31:433
2714599221,7cyclictest581-21systemd-journal12:25:023
2714599221,7cyclictest581-21systemd-journal12:25:013
27145992211,3cyclictest0-21swapper/308:28:283
2714599221,11cyclictest23943-21if_err_enp1s009:25:173
2714599221,10cyclictest0-21swapper/312:13:493
2714599221,10cyclictest0-21swapper/310:07:093
2714599220,16cyclictest0-21swapper/310:18:333
2714599220,12cyclictest0-21swapper/312:36:533
2714599220,12cyclictest0-21swapper/309:33:573
2714599220,10cyclictest0-21swapper/307:57:253
2714099222,11cyclictest0-21swapper/209:45:132
27136992213,3cyclictest0-21swapper/108:11:591
2713699221,18cyclictest5029-21open_files12:20:261
2713699221,18cyclictest5029-21open_files12:20:261
2713699221,18cyclictest23977-21irqstats09:25:191
2713699221,18cyclictest20291-21chrt11:42:451
2713699221,16cyclictest0-21swapper/108:49:441
2713699221,15cyclictest0-21swapper/108:15:291
2713699221,10cyclictest30855-21perf08:30:001
2713699220,20cyclictest0-21swapper/108:52:081
2713699220,20cyclictest0-21swapper/108:52:081
2714599211,5cyclictest0-21swapper/309:38:183
2714599211,10cyclictest0-21swapper/307:35:013
2714599210,10cyclictest0-21swapper/311:05:173
27140992115,3cyclictest20466-21cat10:30:282
2713699219,3cyclictest581-21systemd-journal10:30:021
2713699218,12cyclictest18225-21kworker/u8:208:41:151
2713699212,11cyclictest28897-21smart_sda09:35:291
27136992115,3cyclictest3074-21grep11:05:151
2713699211,18cyclictest6877-21sshd11:11:421
2713699211,18cyclictest6877-21sshd11:11:421
2713699211,18cyclictest0-21swapper/111:56:191
2713699211,18cyclictest0-21swapper/111:47:191
2713699211,15cyclictest0-21swapper/112:39:161
2713699211,15cyclictest0-21swapper/110:56:561
2713699211,15cyclictest0-21swapper/110:45:291
2713699211,11cyclictest30186-21df_inode09:40:161
2713699210,6cyclictest20416-21kworker/u8:112:08:321
61272201,2sleep10-21swapper/109:59:561
2714599208,3cyclictest0-21swapper/310:10:463
2714599207,2cyclictest19485-21strings11:40:213
2714599201,16cyclictest32062-21/usr/sbin/munin09:45:183
2714599201,10cyclictest0-21swapper/310:37:023
2714099203,14cyclictest11251-21tail12:35:192
2714099201,5cyclictest19167-21taskset10:26:012
27140992013,4cyclictest22592-21tail10:35:202
2713699209,3cyclictest13252-21/usr/sbin/munin09:00:421
2713699208,3cyclictest4170-21munin-run07:30:001
2713699208,3cyclictest0-21swapper/110:23:461
2713699202,3cyclictest86750irq/128-enp1s0-09:11:471
27136992017,2cyclictest0-21swapper/107:37:001
2713699201,18cyclictest0-21swapper/109:20:251
2713699201,18cyclictest0-21swapper/107:31:291
2713699201,17cyclictest0-21swapper/112:27:131
2713699201,17cyclictest0-21swapper/107:42:371
2713699201,15cyclictest0-21swapper/110:44:371
2713699201,11cyclictest28610-21cat08:24:561
2713699201,10cyclictest1900-21cat09:50:011
2713699200,5cyclictest0-21swapper/110:34:011
2713699200,15cyclictest0-21swapper/110:54:021
9652190,4sleep123-21ksoftirqd/111:00:161
2714599193,3cyclictest0-21swapper/307:52:063
2714599190,15cyclictest0-21swapper/307:10:113
2714099196,2cyclictest0-21swapper/208:31:052
2714099192,15cyclictest0-21swapper/210:46:322
2714099192,15cyclictest0-21swapper/207:45:222
2714099191,7cyclictest0-21swapper/212:30:302
27136991915,3cyclictest0-21swapper/111:22:141
2713699191,2cyclictest0-21swapper/109:33:211
2713699191,2cyclictest0-21swapper/109:19:511
2713699191,17cyclictest0-21swapper/111:31:561
2713699191,17cyclictest0-21swapper/107:10:281
2713699191,16cyclictest0-21swapper/109:50:171
2713699191,16cyclictest0-21swapper/108:55:101
2713699191,10cyclictest910-21missed_timers07:20:251
2713699191,10cyclictest0-21swapper/111:15:171
2713699190,6cyclictest0-21swapper/111:29:001
2713699190,2cyclictest0-21swapper/107:51:091
105512191,2sleep10-21swapper/110:09:321
2714099189,7cyclictest14514-21fschecks_time11:30:172
2714099186,3cyclictest8159-21tail07:35:392
2714099185,12cyclictest0-21swapper/208:37:452
2714099182,15cyclictest0-21swapper/212:08:372
2714099182,15cyclictest0-21swapper/210:11:202
2714099182,14cyclictest0-21swapper/208:01:472
2714099182,10cyclictest0-21swapper/212:11:242
2714099181,4cyclictest0-21swapper/210:53:132
27140991813,3cyclictest20558-21grep08:05:202
27140991812,3cyclictest23103-21sed09:25:022
2714099181,11cyclictest0-21swapper/209:43:352
2713699187,3cyclictest14411-21cut09:05:151
2713699181,9cyclictest32169-21ntp_states08:30:371
2713699181,2cyclictest0-21swapper/112:00:151
2713699181,10cyclictest11145-21seq07:45:181
2713699180,17cyclictest0-21swapper/112:11:301
2714099172,5cyclictest0-21swapper/207:55:252
2714099172,14cyclictest0-21swapper/210:22:262
2714099171,8cyclictest0-21swapper/211:35:322
2714099171,7cyclictest0-21swapper/209:57:322
2714099171,6cyclictest9948-21/usr/sbin/munin11:20:162
2714099171,6cyclictest0-21swapper/211:58:332
2714099171,6cyclictest0-21swapper/211:43:152
2714099171,6cyclictest0-21swapper/210:40:262
2714099171,6cyclictest0-21swapper/209:27:202
2714099171,5cyclictest0-21swapper/210:19:402
2714099171,4cyclictest5797-21sed11:10:172
2714099171,4cyclictest5797-21sed11:10:172
2714099171,4cyclictest0-21swapper/209:35:272
27140991713,3cyclictest0-21swapper/207:50:502
27140991712,3cyclictest5643-21cat08:45:162
27140991712,3cyclictest1767-21ntpq11:00:242
2714099171,15cyclictest0-21swapper/209:13:222
27140991711,4cyclictest5942-21grep07:30:412
2714099171,14cyclictest0-21swapper/208:10:052
27140991711,3cyclictest3174-21ls07:25:342
27140991711,3cyclictest25270-21cat08:15:232
102222170,3sleep20-21swapper/210:05:322
2714099164,5cyclictest2553-21gltestperf09:50:192
2714099161,9cyclictest0-21swapper/212:15:112
2714099161,9cyclictest0-21swapper/207:40:222
2714099161,6cyclictest3157-21/usr/sbin/munin08:40:232
2714099161,6cyclictest0-21swapper/211:50:202
2714099161,6cyclictest0-21swapper/211:15:202
2714099161,4cyclictest3264-21kworker/u8:111:25:152
2714099161,2cyclictest0-21swapper/209:15:352
27140991611,3cyclictest6843-21grep10:00:172
27140991611,3cyclictest6843-21grep10:00:172
27140991611,3cyclictest6672-21fschecks_time12:25:182
27140991611,3cyclictest31485-21gltestperf10:55:172
27140991611,3cyclictest30619-21fschecks_time07:15:182
27140991611,3cyclictest10046-21cat08:55:122
27140991610,4cyclictest0-21swapper/208:50:262
27140991610,4cyclictest0-21swapper/208:50:262
2714099159,3cyclictest836-21runrttasks07:10:162
2714099151,6cyclictest28853-21/usr/sbin/munin08:25:182
27140991512,2cyclictest0-21swapper/209:03:182
27140991510,3cyclictest14276-21/usr/sbin/munin09:05:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional