You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-27 - 15:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Mar 27, 2023 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
87802242170,23sleep20-21swapper/207:05:202
97812238161,24sleep10-21swapper/107:08:131
97582236162,23sleep30-21swapper/307:07:563
98422229152,24sleep00-21swapper/007:09:010
3159221580,3sleep01001499cyclictest07:55:200
10024998379,3cyclictest0-21swapper/212:22:382
10024995956,2cyclictest0-21swapper/207:43:472
98542530,4sleep339-21rcuc/308:20:173
75372480,2sleep00-21swapper/010:40:270
10024994845,2cyclictest0-21swapper/210:27:242
26182460,1sleep30-21swapper/310:30:233
1001499379,13cyclictest21672-21awk07:35:000
1001499357,13cyclictest3124-21perf09:20:020
1001999341,23cyclictest0-21swapper/107:36:321
1001499345,13cyclictest26684-21perf09:00:000
1001999320,26cyclictest0-21swapper/112:10:141
1001999320,26cyclictest0-21swapper/112:10:141
1001499328,11cyclictest2156-21awk11:45:020
1001499324,13cyclictest17554-21awk12:20:010
1001499314,12cyclictest20091-21idleruntime-cro08:45:000
1001999308,13cyclictest32286-21kworker/u8:111:40:191
279612290,2sleep21002499cyclictest09:00:222
1001999281,23cyclictest24235-21tr07:40:181
1001999281,21cyclictest0-21swapper/110:50:221
10014992814,7cyclictest836-21runrttasks09:54:330
1001499281,12cyclictest25948-21kworker/0:208:00:250
1001999271,21cyclictest0-21swapper/108:22:261
1001999271,18cyclictest16385-21irqstats12:15:181
1001999271,17cyclictest32566-21smart_sda09:10:251
190872261,21sleep319088-21sshd11:09:463
190872261,21sleep319088-21sshd11:09:463
1001999262,12cyclictest8082-21diskstats09:30:161
10019992615,3cyclictest29448-21df_inode09:05:151
10019992612,8cyclictest0-21swapper/111:12:431
1001999261,20cyclictest0-21swapper/112:06:131
1001999261,11cyclictest6543-21munin-run11:55:001
10019992610,9cyclictest0-21swapper/108:30:171
1001499269,2cyclictest29420-21kworker/0:010:27:360
1001499267,16cyclictest10333-21awk10:50:010
1001499261,9cyclictest0-21swapper/011:25:290
10031992521,2cyclictest32754-21kernelversion10:25:263
1002499253,4cyclictest0-21swapper/211:22:392
1001999253,15cyclictest0-21swapper/111:59:181
10019992516,6cyclictest22187-21grep12:30:151
1001999251,20cyclictest0-21swapper/107:14:591
10019992510,3cyclictest32303-21latency_hist11:40:011
1001999250,20cyclictest0-21swapper/109:36:161
1001999250,16cyclictest0-21swapper/107:50:171
1001499252,4cyclictest10617-21cpuspeed_turbos10:50:130
10031992422,1cyclictest0-21swapper/308:15:253
10031992420,2cyclictest13763-21fschecks_count12:10:163
10031992420,2cyclictest13763-21fschecks_count12:10:163
10031992418,4cyclictest0-21swapper/312:00:153
10031992410,6cyclictest0-21swapper/311:25:093
10031992410,5cyclictest0-21swapper/309:20:243
10024992419,3cyclictest0-21swapper/208:05:132
1002499241,6cyclictest26406-21kworker/2:110:04:122
1002499241,12cyclictest780-21rngd12:06:472
1001999245,3cyclictest0-21swapper/110:18:561
1001999245,10cyclictest0-21swapper/108:49:491
1001999244,8cyclictest0-21swapper/109:25:121
1001999242,4cyclictest17272-21munin-run07:25:021
1001999241,9cyclictest0-21swapper/111:30:271
1001999241,6cyclictest0-21swapper/110:46:081
1001999241,4cyclictest24775-21kworker/u8:210:41:071
1001999241,3cyclictest14635-21expr09:45:131
10019992411,8cyclictest0-21swapper/112:03:591
1001999241,17cyclictest0-21swapper/111:22:021
1001999241,17cyclictest0-21swapper/110:23:581
1001999241,17cyclictest0-21swapper/110:05:141
1001999241,17cyclictest0-21swapper/109:17:381
1001999241,17cyclictest0-21swapper/108:57:101
10019992411,3cyclictest0-21swapper/112:35:201
1001999241,11cyclictest3025-21chrt10:30:281
1001999241,11cyclictest27830-21missed_timers09:00:191
1001999241,11cyclictest26293-21diskstats11:25:151
1001999241,11cyclictest18842-21ntp_states12:20:221
1001999241,11cyclictest0-21swapper/110:35:251
1001999241,10cyclictest23490-21kworker/u8:208:42:211
1001499242,10cyclictest30099-21kworker/0:307:51:310
1001499240,11cyclictest28265-21perf07:50:000
10031992321,1cyclictest0-21swapper/309:25:223
1003199231,10cyclictest22149-21gunzip10:00:253
1002499231,7cyclictest0-21swapper/212:33:182
1002499231,4cyclictest0-21swapper/211:29:572
10024992313,7cyclictest6536-40dnf11:55:012
1002499231,15cyclictest0-21swapper/210:53:432
1001999232,9cyclictest32286-21kworker/u8:111:45:021
1001999232,4cyclictest0-21swapper/109:52:141
1001999231,6cyclictest0-21swapper/109:40:581
1001999231,6cyclictest0-21swapper/109:23:261
10019992316,4cyclictest14170-21ntp_states07:15:231
10019992312,7cyclictest0-21swapper/111:15:411
10019992312,3cyclictest19541-21irqstats09:55:191
10019992311,7cyclictest0-21swapper/108:37:341
10019992311,7cyclictest0-21swapper/107:25:571
10019992311,3cyclictest0-21swapper/108:27:171
1001999231,10cyclictest0-21swapper/108:05:461
1001499237,3cyclictest13376-21/usr/sbin/munin12:10:160
1001499237,3cyclictest13376-21/usr/sbin/munin12:10:160
1001499234,8cyclictest23128-21/usr/sbin/munin10:05:130
1001499234,7cyclictest15831-21unixbench_multi09:45:270
1001499231,8cyclictest0-21swapper/011:24:030
1001499230,2cyclictest12050irq/123-ahci[0010:30:290
1001499230,12cyclictest0-21swapper/011:08:400
1001499230,12cyclictest0-21swapper/011:08:400
260742221,17sleep326144-21irqstats10:10:213
10031992219,2cyclictest0-21swapper/310:40:273
10031992219,1cyclictest0-21swapper/309:45:233
10031992218,2cyclictest21563-21smart_sda08:45:263
10031992217,3cyclictest0-21swapper/308:30:253
10031992214,5cyclictest27960-21ntp_states09:00:223
1003199221,15cyclictest0-21swapper/312:20:443
1003199221,10cyclictest4339-21fschecks_count10:35:163
10024992217,3cyclictest0-21swapper/209:07:272
1002499221,6cyclictest0-21swapper/211:15:232
1002499221,10cyclictest0-21swapper/210:16:242
1001999221,6cyclictest0-21swapper/111:05:581
1001999221,6cyclictest0-21swapper/111:05:581
1001999221,6cyclictest0-21swapper/110:00:031
10019992215,2cyclictest0-21swapper/111:00:401
10019992211,7cyclictest0-21swapper/110:55:491
10019992211,3cyclictest0-21swapper/108:19:011
1001999221,11cyclictest24775-21kworker/u8:210:10:031
1001999221,10cyclictest31925-21/usr/sbin/munin10:25:261
1001999221,10cyclictest23429-21sed08:50:191
1001999221,10cyclictest23429-21sed08:50:191
1001999220,17cyclictest0-21swapper/107:57:321
1001999220,11cyclictest0-21swapper/108:00:221
1001499227,8cyclictest86750irq/128-enp1s0-12:30:240
1001499224,8cyclictest6232-21grep09:25:190
1001499223,2cyclictest23562-21latency_hist11:20:020
1001499223,11cyclictest2696-21expr11:45:140
1001499223,10cyclictest29103-21/usr/sbin/munin09:05:270
1001499222,4cyclictest9088-21munin-run08:20:010
1001499222,2cyclictest628750kworker/u8:111:58:020
1001499222,13cyclictest19858-21fschecks_time11:10:160
1001499222,10cyclictest2258-21kworker/0:108:05:270
1001499221,6cyclictest0-21swapper/012:05:040
10014992211,3cyclictest823-21snmpd11:53:460
1001499221,12cyclictest0-21swapper/009:13:250
1001499221,11cyclictest0-21swapper/009:23:150
1001499220,9cyclictest18816-21sed07:25:250
1001499220,5cyclictest0-21swapper/010:37:310
1001499220,10cyclictest10406-21timerwakeupswit12:00:270
1003199218,2cyclictest6849-21sshd08:14:413
1003199212,2cyclictest86850irq/129-enp1s0-11:44:503
10031992119,1cyclictest0-21swapper/307:35:273
1003199211,8cyclictest16209-21smart_sda11:00:263
10031992118,2cyclictest0-21swapper/312:30:283
1003199211,7cyclictest0-21swapper/311:48:183
1003199211,6cyclictest0-21swapper/309:35:363
10031992116,3cyclictest9264-21grep09:30:303
1003199211,5cyclictest0-21swapper/311:15:413
1003199211,17cyclictest0-21swapper/312:25:263
1003199211,11cyclictest0-21swapper/310:16:243
1003199211,10cyclictest413-21/usr/sbin/munin08:00:263
1003199211,10cyclictest0-21swapper/312:09:173
1002499215,8cyclictest10303-21cat10:49:562
1002499211,7cyclictest31305-21/usr/sbin/munin09:10:132
1002499211,6cyclictest0-21swapper/210:37:352
1002499211,6cyclictest0-21swapper/210:14:372
1002499211,5cyclictest0-21swapper/211:13:252
1002499211,4cyclictest0-21swapper/209:30:212
1001999211,17cyclictest31073-21diskmemload12:25:111
10019992111,2cyclictest0-21swapper/108:11:241
1001999210,9cyclictest0-21swapper/107:35:011
1001499218,3cyclictest11060-21kworker/0:108:27:130
1001499217,9cyclictest0-21swapper/009:31:020
1001499215,4cyclictest14948-21/usr/sbin/munin11:00:150
1001499213,2cyclictest86850irq/129-enp1s0-10:58:410
1001499213,10cyclictest24365-21/usr/sbin/munin12:35:240
1001499212,9cyclictest12231-21kworker/0:107:21:370
1001499212,6cyclictest823-21snmpd10:13:250
1001499212,4cyclictest18727-21/usr/sbin/munin09:55:220
10014992116,3cyclictest10683-21grep07:10:170
1001499211,13cyclictest0-21swapper/010:18:060
1003199208,3cyclictest86850irq/129-enp1s0-08:38:373
1003199208,3cyclictest0-21swapper/310:20:253
1003199206,7cyclictest0-21swapper/311:50:143
1003199201,7cyclictest20268-21memory11:10:203
1003199201,7cyclictest0-21swapper/311:36:363
1003199201,6cyclictest18648-21latency_hist09:55:023
1003199201,6cyclictest0-21swapper/312:16:593
1003199201,6cyclictest0-21swapper/311:20:113
10031992016,2cyclictest25356-21smart_sda07:40:323
1003199201,4cyclictest0-21swapper/310:45:053
1002499201,9cyclictest0-21swapper/209:20:252
1002499201,7cyclictest3959-21users11:45:272
1002499201,7cyclictest0-21swapper/212:36:472
1002499201,7cyclictest0-21swapper/209:43:052
10024992017,2cyclictest0-21swapper/210:08:122
1002499201,6cyclictest0-21swapper/211:30:172
1002499201,6cyclictest0-21swapper/211:08:352
1002499201,6cyclictest0-21swapper/211:08:352
1002499201,6cyclictest0-21swapper/211:00:442
1002499201,6cyclictest0-21swapper/209:56:232
1002499201,5cyclictest0-21swapper/212:19:422
1002499201,5cyclictest0-21swapper/212:04:192
1002499201,4cyclictest14510-21cstates09:45:142
1002499201,4cyclictest0-21swapper/210:59:152
10024992013,4cyclictest848-21sssd_nss09:00:012
10024992013,4cyclictest7584-21tune2fs08:15:172
1002499201,16cyclictest17781-21mailstats09:50:252
1001999200,14cyclictest0-21swapper/107:47:231
1001499207,10cyclictest31316-21users10:20:290
1001499204,5cyclictest31073-21diskmemload11:35:330
1001499202,4cyclictest25590-21kworker/0:211:34:260
1001499201,5cyclictest0-21swapper/009:35:380
1001499201,11cyclictest15910-21/usr/sbin/munin08:35:180
1001499201,11cyclictest13101-21/usr/sbin/munin07:15:260
1001499201,11cyclictest12137-21/usr/sbin/munin09:40:230
1001499201,11cyclictest0-21swapper/012:25:050
184702191,2sleep20-21swapper/207:25:212
1003199199,3cyclictest31284-21grep07:55:183
1003199198,9cyclictest13499-21awk08:30:023
1003199194,3cyclictest86850irq/129-enp1s0-09:40:283
1003199191,7cyclictest13474-21idleruntime10:55:213
1003199191,7cyclictest0-21swapper/312:38:383
10031991917,1cyclictest0-21swapper/310:05:253
1003199191,6cyclictest0-21swapper/309:57:073
1003199191,6cyclictest0-21swapper/309:13:043
1003199191,5cyclictest0-21swapper/310:52:023
10031991912,4cyclictest19843-21grep07:30:133
1003199191,17cyclictest0-21swapper/309:18:153
1003199191,16cyclictest0-21swapper/307:25:183
1003199191,15cyclictest15745-21grep07:20:203
1002499195,4cyclictest23490-21kworker/u8:208:03:502
1002499191,6cyclictest31503-21pmu-power11:35:332
1002499191,6cyclictest0-21swapper/211:55:522
1002499191,6cyclictest0-21swapper/210:45:002
1002499191,6cyclictest0-21swapper/209:15:182
1002499191,6cyclictest0-21swapper/207:50:452
1002499191,5cyclictest0-21swapper/212:11:132
1002499191,5cyclictest0-21swapper/212:11:132
1002499191,5cyclictest0-21swapper/209:36:552
1002499191,5cyclictest0-21swapper/207:46:502
1002499191,4cyclictest1096-21missed_timers11:40:212
1002499191,4cyclictest0-21swapper/210:20:352
1002499191,16cyclictest32053-21cat07:55:302
10024991911,5cyclictest12462-21missed_timers08:25:212
1002499191,10cyclictest0-21swapper/208:42:222
1002499191,10cyclictest0-21swapper/207:20:202
1001499195,2cyclictest8835-21kworker/0:208:31:130
1001499193,9cyclictest20298-21kworker/0:109:04:120
1001499191,10cyclictest17772-21/usr/sbin/munin12:20:220
1001499191,10cyclictest0-21swapper/010:02:310
297762181,2sleep30-21swapper/311:30:533
1003199182,10cyclictest41-21ksoftirqd/308:09:313
1003199181,7cyclictest0-21swapper/308:42:223
1003199181,6cyclictest24395-21NetworkManager08:54:163
1003199181,6cyclictest24395-21NetworkManager08:54:163
1003199181,6cyclictest0-21swapper/311:55:373
1002499182,4cyclictest0-21swapper/208:35:152
1002499181,6cyclictest0-21swapper/207:32:512
1002499181,5cyclictest0-21swapper/210:31:032
1002499181,5cyclictest0-21swapper/207:17:492
1002499181,4cyclictest0-21swapper/209:25:282
10024991813,3cyclictest22498-21/usr/sbin/munin08:50:132
10024991813,3cyclictest22498-21/usr/sbin/munin08:50:132
10024991811,4cyclictest6174-21/usr/sbin/munin08:10:262
1002499180,6cyclictest0-21swapper/212:25:172
1001499186,2cyclictest6605-21kworker/0:008:14:310
1001499184,6cyclictest20304-21/usr/sbin/munin08:45:260
1001499182,4cyclictest9-21rcu_preempt07:37:080
1001499181,9cyclictest22498-21/usr/sbin/munin08:50:140
1001499181,9cyclictest22498-21/usr/sbin/munin08:50:140
1001499181,5cyclictest0-21swapper/008:23:300
1001499181,5cyclictest0-21swapper/007:42:510
1003199171,15cyclictest0-21swapper/309:07:113
1003199171,15cyclictest0-21swapper/308:58:113
10031991711,3cyclictest13879-21if_err_enp1s007:15:183
1002499171,6cyclictest0-21swapper/208:22:102
10024991711,3cyclictest10220-21basename07:10:122
1003199161,5cyclictest0-21swapper/307:50:453
1003199161,5cyclictest0-21swapper/307:45:143
1003199161,4cyclictest0-21swapper/307:10:203
10024991612,2cyclictest20486-21cstates08:45:142
10024991611,3cyclictest22068-21cstates07:35:142
10024991611,3cyclictest14779-21ntp_states08:30:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional