You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 23:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 04, 2022 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
82452237161,24sleep00-21swapper/007:08:580
81362236163,25sleep20-21swapper/207:07:362
81072236168,21sleep10-21swapper/107:07:141
80522235164,58sleep30-21swapper/307:06:323
2049321510,4sleep2842399cyclictest11:15:222
1005321440,4sleep1841999cyclictest10:50:291
1005321440,4sleep1841999cyclictest10:50:291
930921080,3sleep0841199cyclictest10:50:190
930921080,3sleep0841199cyclictest10:50:190
8423991071,105cyclictest0-21swapper/207:30:022
307052650,3sleep030706-21turbostat.cron11:40:010
86492590,4sleep08651-21missed_timers08:20:220
8423995339,3cyclictest8395-21kworker/2:007:24:402
204602530,3sleep320464-21pmu-power10:00:323
8423994631,7cyclictest22271-21kworker/2:112:13:042
8423994529,2cyclictest22271-21kworker/2:111:48:482
8423994432,3cyclictest22271-21kworker/2:112:26:202
306582440,2sleep20-21swapper/207:55:282
169072430,3sleep1841999cyclictest12:20:201
8423994229,2cyclictest22271-21kworker/2:110:15:222
842399401,29cyclictest8395-21kworker/2:007:17:032
8428993821,10cyclictest0-21swapper/308:10:433
8428993817,7cyclictest0-21swapper/311:48:473
8423993828,3cyclictest22271-21kworker/2:108:33:522
8423993828,2cyclictest22271-21kworker/2:111:06:262
8423993826,7cyclictest22271-21kworker/2:109:36:062
842399382,27cyclictest22271-21kworker/2:110:47:322
842399382,27cyclictest22271-21kworker/2:110:11:562
842399381,28cyclictest22271-21kworker/2:110:50:162
842399381,28cyclictest22271-21kworker/2:110:50:162
8423993728,2cyclictest22271-21kworker/2:109:01:062
842399372,33cyclictest22271-21kworker/2:111:52:262
8428993621,3cyclictest0-21swapper/312:26:493
8423993626,3cyclictest22271-21kworker/2:109:18:042
8423993626,2cyclictest22271-21kworker/2:110:35:002
8423993625,3cyclictest22271-21kworker/2:112:24:282
842399362,11cyclictest22271-21kworker/2:110:05:482
842399361,22cyclictest22271-21kworker/2:109:31:402
842399361,14cyclictest22271-21kworker/2:110:45:002
8428993526,3cyclictest0-21swapper/307:24:393
8423993524,3cyclictest22271-21kworker/2:108:24:522
8423993523,2cyclictest22271-21kworker/2:108:51:382
8423993510,3cyclictest22271-21kworker/2:112:00:342
8428993419,10cyclictest0-21swapper/309:41:033
8423993424,2cyclictest22271-21kworker/2:108:46:202
8423993423,3cyclictest22271-21kworker/2:111:01:002
8423993423,3cyclictest22271-21kworker/2:110:39:242
8423993422,3cyclictest22271-21kworker/2:111:12:362
8423993422,2cyclictest22271-21kworker/2:110:00:462
8423993416,3cyclictest22271-21kworker/2:107:41:542
842399341,10cyclictest22271-21kworker/2:111:24:502
8428993320,4cyclictest0-21swapper/312:13:033
842399332,29cyclictest22271-21kworker/2:110:28:202
8423993322,3cyclictest22271-21kworker/2:112:33:222
842399331,9cyclictest22271-21kworker/2:109:42:082
8423993319,7cyclictest22271-21kworker/2:112:35:482
841199336,12cyclictest27280-21munin-run09:05:010
8428993217,9cyclictest0-21swapper/311:53:213
8428993215,8cyclictest0-21swapper/312:00:513
8423993223,2cyclictest22271-21kworker/2:107:47:472
8423993222,3cyclictest8395-21kworker/2:007:35:482
8423993222,3cyclictest8395-21kworker/2:007:35:472
8423993222,3cyclictest22271-21kworker/2:108:36:402
8423993222,2cyclictest22271-21kworker/2:111:29:182
842399322,20cyclictest22271-21kworker/2:110:22:182
842399322,14cyclictest22271-21kworker/2:109:45:042
842399322,14cyclictest22271-21kworker/2:109:45:042
8423993220,3cyclictest8395-21kworker/2:007:27:482
8423993216,2cyclictest22271-21kworker/2:109:28:502
8428993119,3cyclictest0-21swapper/309:31:573
8428993113,7cyclictest0-21swapper/312:24:273
8423993121,3cyclictest22271-21kworker/2:109:57:062
842399311,28cyclictest22271-21kworker/2:111:33:042
8423993112,7cyclictest22271-21kworker/2:108:19:362
8419993112,9cyclictest14435-21kworker/u8:209:59:401
8428993018,3cyclictest0-21swapper/311:58:113
8428993018,3cyclictest0-21swapper/311:58:113
8428993018,3cyclictest0-21swapper/308:07:453
842899301,19cyclictest0-21swapper/310:58:553
842399302,13cyclictest22271-21kworker/2:112:06:542
8423993019,3cyclictest8395-21kworker/2:007:14:282
8423993019,3cyclictest22271-21kworker/2:108:11:362
8423993014,3cyclictest22271-21kworker/2:112:17:522
842399301,20cyclictest22271-21kworker/2:109:52:242
841199304,13cyclictest15675-21munin-run07:25:010
8411993011,16cyclictest22872-21munin-run08:55:010
8428992915,3cyclictest0-21swapper/310:12:233
8428992911,7cyclictest0-21swapper/312:33:213
8423992918,2cyclictest22271-21kworker/2:109:10:442
842399290,20cyclictest22271-21kworker/2:111:38:122
841199292,14cyclictest8730-21munin-run10:50:000
841199290,7cyclictest11214-21kworker/0:012:13:430
8428992818,3cyclictest0-21swapper/310:47:513
8423992818,2cyclictest22271-21kworker/2:108:00:042
8423992814,3cyclictest22271-21kworker/2:108:27:082
8423992813,3cyclictest22271-21kworker/2:108:59:322
842399281,13cyclictest22271-21kworker/2:111:55:202
842399281,13cyclictest22271-21kworker/2:111:55:202
8423992811,3cyclictest22271-21kworker/2:108:43:002
8423992811,3cyclictest22271-21kworker/2:108:43:002
841999281,12cyclictest859-21perf08:05:021
841999281,11cyclictest780-21rngd08:38:101
8428992718,2cyclictest0-21swapper/310:15:213
8428992717,3cyclictest0-21swapper/310:36:093
8423992716,2cyclictest22271-21kworker/2:109:09:102
8423992715,2cyclictest22271-21kworker/2:111:40:022
8423992715,2cyclictest22271-21kworker/2:109:23:002
8423992714,3cyclictest22271-21kworker/2:107:54:362
841999277,13cyclictest0-21swapper/109:29:441
841999273,2cyclictest12888-21grep12:10:241
841999272,13cyclictest15335-21awk11:05:011
841999271,17cyclictest3347-21grep11:50:221
841999270,10cyclictest0-21swapper/108:27:521
8428992612,3cyclictest0-21swapper/308:35:473
8423992611,3cyclictest22271-21kworker/2:110:56:522
8423992611,2cyclictest22271-21kworker/2:108:06:582
841999262,21cyclictest20674-21munin-run08:50:011
841999261,17cyclictest18447-21diskstats07:30:191
841999261,17cyclictest12746-21gltestperf08:30:191
841999260,9cyclictest0-21swapper/111:45:151
841199261,12cyclictest7494-21munin-run08:20:000
8428992515,3cyclictest0-21swapper/309:50:573
8428992514,3cyclictest0-21swapper/311:06:253
8428992514,3cyclictest0-21swapper/308:16:333
8428992514,3cyclictest0-21swapper/307:50:553
8428992512,6cyclictest0-21swapper/310:42:513
841999259,4cyclictest28971-21latency_hist07:55:021
841999255,9cyclictest780-21rngd07:59:361
841999252,16cyclictest823-21snmpd10:05:241
8419992519,3cyclictest25702-21munin-run10:15:021
841999250,17cyclictest21811-21load08:50:201
842899246,6cyclictest0-21swapper/311:04:213
8428992416,4cyclictest0-21swapper/307:10:123
8428992412,3cyclictest0-21swapper/311:29:173
842899241,16cyclictest0-21swapper/309:16:593
842899241,14cyclictest0-21swapper/309:14:113
841999242,17cyclictest0-21swapper/112:02:031
841999241,8cyclictest0-21swapper/111:18:321
841999241,4cyclictest0-21swapper/110:45:491
841999241,17cyclictest0-21swapper/110:55:281
841999241,17cyclictest0-21swapper/110:33:081
8419992410,5cyclictest21531-21cat12:30:281
841199244,6cyclictest19049-40dnf09:59:390
841199240,6cyclictest0-21swapper/011:26:250
299032241,3sleep10-21swapper/109:10:141
842899236,4cyclictest0-21swapper/308:50:113
842899233,2cyclictest86650irq/127-enp1s0-09:08:473
8428992312,3cyclictest0-21swapper/307:38:553
8428992312,3cyclictest0-21swapper/307:38:553
8428992311,3cyclictest0-21swapper/309:37:473
842899230,3cyclictest0-21swapper/308:49:313
841999235,7cyclictest2697-21df_inode10:35:151
841999233,14cyclictest86850irq/129-enp1s0-10:16:071
841999232,18cyclictest24458-21munin-run07:45:011
841999231,9cyclictest0-21swapper/109:40:431
841999231,7cyclictest0-21swapper/111:56:031
841999231,7cyclictest0-21swapper/111:56:031
8419992314,3cyclictest0-21swapper/110:44:181
841999231,3cyclictest14240-21grep07:20:181
841999231,19cyclictest1677-21perl08:05:171
841999231,19cyclictest10020-21grep07:10:241
841999231,17cyclictest0-21swapper/111:44:081
841999231,11cyclictest0-21swapper/108:20:401
841199231,5cyclictest0-21swapper/010:16:430
842899229,2cyclictest0-21swapper/311:42:413
842899228,9cyclictest0-21swapper/310:50:333
842899228,9cyclictest0-21swapper/310:50:333
842899228,3cyclictest0-21swapper/311:16:353
842899224,2cyclictest86850irq/129-enp1s0-07:34:033
842899221,6cyclictest0-21swapper/307:57:053
8428992211,5cyclictest0-21swapper/310:08:273
8428992211,3cyclictest0-21swapper/312:37:353
8428992211,3cyclictest0-21swapper/309:24:273
842899220,13cyclictest0-21swapper/310:25:093
841999223,16cyclictest221ktimersoftd/110:05:021
841999222,15cyclictest0-21swapper/112:29:291
841999222,15cyclictest0-21swapper/111:28:171
841999222,15cyclictest0-21swapper/111:12:501
841999222,15cyclictest0-21swapper/109:16:061
8419992213,3cyclictest0-21swapper/111:33:311
841999221,19cyclictest0-21swapper/111:35:191
841999221,19cyclictest0-21swapper/109:05:131
841999221,18cyclictest25573-21cstates09:00:151
841999221,18cyclictest23030-21cstates12:35:161
841999221,15cyclictest0-21swapper/109:30:391
841999221,10cyclictest17465-21grep07:25:281
841199226,8cyclictest14700-21awk09:50:000
841199226,8cyclictest14700-21awk09:50:000
841199225,5cyclictest0-21swapper/009:32:450
841199225,15cyclictest0-21swapper/011:53:430
841199221,15cyclictest0-21swapper/010:35:270
841199220,13cyclictest0-21swapper/009:37:460
198332220,4sleep011-21rcuc/012:25:280
842899218,4cyclictest0-21swapper/309:57:053
842899217,7cyclictest0-21swapper/309:29:513
842899211,6cyclictest13009-21cstates09:45:143
842899211,6cyclictest13009-21cstates09:45:143
8428992115,4cyclictest29387-21tail11:35:183
842899211,13cyclictest0-21swapper/312:19:353
842899210,3cyclictest0-21swapper/307:47:013
842899210,3cyclictest0-21swapper/307:28:293
841999215,8cyclictest0-21swapper/109:36:501
841999212,6cyclictest29474-21diskmemload11:06:041
841999212,18cyclictest0-21swapper/107:17:111
841999211,5cyclictest0-21swapper/109:51:581
841999211,4cyclictest11475-21kworker/u8:012:15:531
841999211,18cyclictest0-21swapper/108:10:211
841999211,15cyclictest0-21swapper/112:06:371
841999211,15cyclictest0-21swapper/111:20:541
841999211,15cyclictest0-21swapper/107:49:081
841999211,11cyclictest581-21systemd-journal09:50:011
841999211,11cyclictest581-21systemd-journal09:50:011
841999211,11cyclictest31396-21ntp_states10:25:271
841999211,11cyclictest20867-21perl07:35:151
841999211,11cyclictest20867-21perl07:35:151
841199215,6cyclictest0-21swapper/009:29:080
841199212,7cyclictest0-21swapper/010:30:220
841199211,9cyclictest30025-21kworker/0:010:40:030
841199211,11cyclictest31935-21sshd10:25:380
842899206,8cyclictest0-21swapper/310:21:313
842899206,3cyclictest0-21swapper/311:31:453
842899203,5cyclictest10114-21fschecks_time12:05:163
842899202,3cyclictest86850irq/129-enp1s0-08:40:573
842899202,3cyclictest86850irq/129-enp1s0-08:40:573
842899200,3cyclictest0-21swapper/309:04:293
842899200,3cyclictest0-21swapper/308:34:593
8419992016,3cyclictest0-21swapper/108:59:131
8419992013,3cyclictest0-21swapper/108:18:441
841999201,14cyclictest0-21swapper/108:43:381
841999201,14cyclictest0-21swapper/108:43:381
8419992010,3cyclictest29391-21smart_sda10:20:301
841199209,3cyclictest10534-21/usr/sbin/munin09:40:190
841199206,5cyclictest0-21swapper/010:58:110
841199202,5cyclictest11567-1kworker/u9:010:07:590
841199201,8cyclictest0-21swapper/011:00:120
841199201,4cyclictest0-21swapper/011:47:580
8411992014,3cyclictest28236-21cpuspeed_turbos10:20:130
841199201,3cyclictest0-21swapper/012:06:480
841199201,11cyclictest18706-21/usr/sbin/munin08:45:160
841199201,11cyclictest0-21swapper/012:19:390
841199201,11cyclictest0-21swapper/011:43:360
841199201,11cyclictest0-21swapper/010:11:120
841199201,11cyclictest0-21swapper/009:16:420
841199201,10cyclictest0-21swapper/011:07:000
841199200,10cyclictest4896-21kworker/u8:111:55:030
841199200,10cyclictest4896-21kworker/u8:111:55:030
842899192,2cyclictest86650irq/127-enp1s0-08:25:593
842899191,11cyclictest0-21swapper/311:20:093
842899191,10cyclictest17756-21/usr/sbin/munin11:10:143
8428991910,7cyclictest0-21swapper/310:31:053
842899190,3cyclictest0-21swapper/308:56:233
842899190,3cyclictest0-21swapper/308:20:273
842899190,3cyclictest0-21swapper/307:16:033
842899190,12cyclictest0-21swapper/308:04:373
841999190,18cyclictest0-21swapper/109:20:481
841199199,6cyclictest0-21swapper/009:09:560
841199198,6cyclictest0-21swapper/008:39:550
841199198,3cyclictest24680-21/usr/sbin/munin07:45:180
841199196,5cyclictest0-21swapper/010:04:570
841199193,15cyclictest11-21rcuc/011:20:010
841199192,4cyclictest15277-21diskstats09:50:160
841199191,9cyclictest0-21swapper/008:07:260
841199191,4cyclictest0-21swapper/012:34:140
841199191,10cyclictest823-21snmpd07:42:360
841199191,10cyclictest29474-21diskmemload11:30:220
841199191,10cyclictest0-21swapper/012:37:440
841199191,10cyclictest0-21swapper/011:24:030
841199190,7cyclictest0-21swapper/012:00:400
176582191,2sleep00-21swapper/007:27:320
8428991812,3cyclictest23268-21cat07:40:183
841199181,9cyclictest0-21swapper/011:11:080
841199181,5cyclictest0-21swapper/012:23:290
8411991812,3cyclictest29074-21/usr/sbin/munin07:55:200
841199181,14cyclictest11886-21dump-pmu-power08:30:000
841199181,12cyclictest0-21swapper/009:14:510
34982181,2sleep00-21swapper/009:22:290
198802180,2sleep00-21swapper/007:32:500
841199177,7cyclictest522-21sshd08:01:370
841199177,2cyclictest87950irq/132-enp2s0-08:32:590
841199172,13cyclictest13012-21grep07:15:270
841199171,13cyclictest16511-21/usr/sbin/munin08:40:180
841199171,13cyclictest16511-21/usr/sbin/munin08:40:180
234412170,13sleep00-21swapper/008:55:150
219162171,2sleep00-21swapper/007:35:480
219162171,2sleep00-21swapper/007:35:480
841199161,6cyclictest8616-21/usr/sbin/munin07:10:110
841199160,10cyclictest0-21swapper/008:10:140
272272160,2sleep00-21swapper/007:50:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional