You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 08:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 27, 2024 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
242022235163,24sleep30-21swapper/319:09:203
241492232161,57sleep20-21swapper/219:08:392
239072225193,20sleep10-21swapper/119:05:311
241272203166,24sleep00-21swapper/019:08:200
1802521850,4sleep02434599cyclictest21:20:140
640121660,4sleep22435599cyclictest23:20:202
24355991080,106cyclictest0-21swapper/222:32:532
24355998884,3cyclictest0-21swapper/221:50:532
19092710,1sleep30-21swapper/323:10:163
204102640,2sleep20-21swapper/220:10:242
277792590,2sleep00-21swapper/000:10:100
265992560,3sleep341-21ksoftirqd/320:25:173
129792470,2sleep230-21rcuc/219:55:152
2435099381,11cyclictest0-21swapper/123:34:201
2434599389,14cyclictest20871-21idleruntime-cro23:55:000
2434599389,13cyclictest5847-21awk19:40:000
2435099361,6cyclictest0-21swapper/120:52:521
2434599359,13cyclictest14291-21munin-run23:40:010
2434599357,13cyclictest4896-21awk22:05:020
2435099343,10cyclictest0-21swapper/123:56:281
2435099341,7cyclictest0-21swapper/121:28:181
2434599346,13cyclictest5502-21munin-run23:20:010
2435599332,20cyclictest10753-21kworker/2:122:21:112
2435099331,14cyclictest15181-21switchtime22:25:261
2435099331,14cyclictest0-21swapper/123:29:241
24355993222,2cyclictest10753-21kworker/2:120:52:522
2435599321,19cyclictest10753-21kworker/2:119:27:122
2435099321,13cyclictest0-21swapper/123:46:021
246282310,5sleep00-21swapper/021:35:150
2435599312,19cyclictest10753-21kworker/2:122:55:402
2435599312,19cyclictest10753-21kworker/2:122:55:402
2435099311,12cyclictest0-21swapper/100:12:161
2434599313,13cyclictest8047-21awk19:45:000
2435599304,7cyclictest10753-21kworker/2:123:49:442
2435599301,19cyclictest10753-21kworker/2:100:18:322
2435099305,3cyclictest0-21swapper/119:48:101
2435099305,3cyclictest0-21swapper/119:48:101
2435099301,9cyclictest0-21swapper/122:32:101
2435099301,5cyclictest0-21swapper/123:00:121
2434599307,11cyclictest4267-21perf20:50:000
2434599307,11cyclictest4267-21perf20:50:000
2435599292,10cyclictest10753-21kworker/2:123:10:202
24355992918,3cyclictest10753-21kworker/2:100:06:162
24355992918,3cyclictest10753-21kworker/2:100:06:162
24355992912,3cyclictest10753-21kworker/2:123:20:022
2435099291,15cyclictest0-21swapper/122:07:541
2435099291,10cyclictest0-21swapper/119:42:481
2435099290,14cyclictest0-21swapper/121:30:041
2435099290,11cyclictest0-21swapper/119:55:381
2434599291,15cyclictest29416-21munin-run19:20:020
2435099288,3cyclictest0-21swapper/120:43:441
2435099288,2cyclictest0-21swapper/120:29:461
2435099285,3cyclictest0-21swapper/120:48:441
2435099285,3cyclictest0-21swapper/120:48:441
2435099281,9cyclictest0-21swapper/120:05:301
2435099281,14cyclictest0-21swapper/121:41:281
2435099281,12cyclictest0-21swapper/119:19:421
2435099281,12cyclictest0-21swapper/100:02:401
2435099281,11cyclictest0-21swapper/122:21:241
2435099281,10cyclictest0-21swapper/121:37:301
2435099281,10cyclictest0-21swapper/120:19:341
2435099281,10cyclictest0-21swapper/100:36:261
2434599281,12cyclictest30039-21awk20:35:000
2435599277,4cyclictest10753-21kworker/2:120:02:062
2435599272,18cyclictest10753-21kworker/2:121:16:502
2435599272,12cyclictest10753-21kworker/2:122:10:232
24355992717,2cyclictest10753-21kworker/2:119:18:252
24355992714,3cyclictest10753-21kworker/2:121:11:122
2435099274,3cyclictest0-21swapper/100:16:001
2435099271,9cyclictest0-21swapper/122:35:121
2435099271,9cyclictest0-21swapper/100:09:061
2435099271,9cyclictest0-21swapper/100:09:061
2435099271,10cyclictest0-21swapper/121:02:221
2435099271,10cyclictest0-21swapper/119:25:101
2435099270,4cyclictest0-21swapper/119:32:421
2435099270,10cyclictest0-21swapper/122:48:441
2435099270,10cyclictest0-21swapper/119:12:071
2434599279,15cyclictest9905-21perf23:30:010
2434599274,11cyclictest16499-21perf23:45:000
2436299264,15cyclictest0-21swapper/300:05:193
2436299264,15cyclictest0-21swapper/300:05:193
24355992613,3cyclictest10753-21kworker/2:119:32:422
24355992612,3cyclictest10753-21kworker/2:121:30:182
24355992610,3cyclictest10753-21kworker/2:120:05:262
2435099268,3cyclictest14350-21ntp_states21:10:261
2435099264,3cyclictest0-21swapper/119:39:581
2435099261,13cyclictest0-21swapper/123:18:121
2435099261,13cyclictest0-21swapper/122:41:081
2435099261,13cyclictest0-21swapper/121:23:541
2435099261,13cyclictest0-21swapper/121:18:121
2435099261,11cyclictest0-21swapper/123:06:181
2435099261,10cyclictest0-21swapper/121:08:121
24350992610,3cyclictest6801-21ntp_states23:20:281
2434599267,16cyclictest30643-21munin-run21:50:010
2434599261,6cyclictest0-21swapper/000:28:010
2435599258,2cyclictest10753-21kworker/2:123:00:122
2435599255,9cyclictest30-21rcuc/219:11:552
2435599255,11cyclictest32-21ksoftirqd/219:24:442
2435599254,7cyclictest30-21rcuc/221:00:442
2435599253,8cyclictest10753-21kworker/2:122:05:282
2435599252,4cyclictest10753-21kworker/2:120:17:002
24355992513,3cyclictest10753-21kworker/2:123:51:302
24355992510,3cyclictest10753-21kworker/2:122:50:002
2435599250,15cyclictest10753-21kworker/2:121:41:192
2435099257,2cyclictest0-21swapper/100:35:021
2435099256,4cyclictest0-21swapper/119:50:141
2435099255,3cyclictest0-21swapper/120:34:081
2435099252,4cyclictest0-21swapper/120:12:141
24350992512,3cyclictest0-21swapper/122:04:481
2435099251,15cyclictest0-21swapper/122:53:041
2435099251,10cyclictest0-21swapper/121:59:001
2435099250,10cyclictest0-21swapper/100:25:101
2434599257,15cyclictest8344-21idleruntime-cro00:40:010
151032250,3sleep115107-21gltestperf23:40:181
2436299241,10cyclictest0-21swapper/322:35:273
2435599244,2cyclictest86750irq/128-enp1s0-23:38:102
2435599243,8cyclictest10753-21kworker/2:120:34:162
2435599243,4cyclictest10753-21kworker/2:120:43:082
2435599242,3cyclictest86550irq/126-enp1s0-21:06:242
24355992411,2cyclictest10753-21kworker/2:123:32:162
2435599241,10cyclictest10753-21kworker/2:122:37:282
2435099247,3cyclictest20661-21taskset23:52:241
2435099247,3cyclictest0-21swapper/123:11:401
2435099245,3cyclictest0-21swapper/120:00:041
2435099244,3cyclictest0-21swapper/120:36:181
2435099242,12cyclictest31608-21munin-run19:25:011
2435099241,11cyclictest0-21swapper/123:36:181
2434599241,11cyclictest20275-21perf22:40:000
2436299234,10cyclictest0-21swapper/323:01:473
2436299231,13cyclictest0-21swapper/321:10:203
2436299231,11cyclictest0-21swapper/321:45:193
2435599239,3cyclictest86650irq/127-enp1s0-20:29:462
2435599236,3cyclictest10753-21kworker/2:121:47:512
2435599235,5cyclictest10753-21kworker/2:100:10:162
2435599234,8cyclictest10753-21kworker/2:122:00:552
2435099238,3cyclictest28355-21sensors22:55:281
2435099238,3cyclictest28355-21sensors22:55:281
2435099237,3cyclictest0-21swapper/100:22:541
2435099236,4cyclictest10415-21memory22:15:181
2435099231,15cyclictest0-21swapper/122:10:081
2435099231,11cyclictest0-21swapper/120:58:441
2436299226,3cyclictest0-21swapper/320:50:113
2436299226,2cyclictest529-21/usr/sbin/munin00:20:233
2435599228,3cyclictest10753-21kworker/2:120:21:382
2435599226,8cyclictest30-21rcuc/223:25:242
2435599226,8cyclictest30-21rcuc/219:50:142
2435599226,2cyclictest10753-21kworker/2:100:22:162
2435599225,7cyclictest10753-21kworker/2:122:18:592
2435599223,9cyclictest32-21ksoftirqd/220:56:142
2435599222,8cyclictest32-21ksoftirqd/220:36:162
2435599222,3cyclictest10753-21kworker/2:121:55:112
2435599222,10cyclictest10753-21kworker/2:100:02:102
2435599221,17cyclictest10753-21kworker/2:121:36:152
2435599221,11cyclictest9321-21ntp_states19:45:212
2435599221,11cyclictest9321-21ntp_states19:45:212
2435099224,3cyclictest0-21swapper/121:52:501
2434599222,17cyclictest15257-21munin-run21:15:010
2434599221,5cyclictest0-21swapper/021:50:280
2436299217,3cyclictest0-21swapper/319:34:393
2436299215,9cyclictest0-21swapper/321:19:593
2436299215,3cyclictest0-21swapper/320:10:193
2436299211,9cyclictest0-21swapper/320:59:203
2436299210,12cyclictest0-21swapper/322:15:133
2435599216,3cyclictest32-21ksoftirqd/220:45:322
2435599216,3cyclictest32-21ksoftirqd/220:45:322
2435599216,2cyclictest32-21ksoftirqd/219:44:362
2435599214,8cyclictest32-21ksoftirqd/223:56:522
2435599214,5cyclictest10753-21kworker/2:123:40:122
2435599213,3cyclictest10753-21kworker/2:100:33:342
2435099214,3cyclictest0-21swapper/121:45:141
2434599216,3cyclictest30757-21kernelversion00:15:170
2434599215,14cyclictest3317-21awk23:15:010
2434599213,10cyclictest21102-21/usr/sbin/munin23:55:270
2434599211,5cyclictest0-21swapper/022:10:130
24345992111,3cyclictest86750irq/128-enp1s0-22:21:520
2436299209,3cyclictest41-21ksoftirqd/321:54:313
2436299209,3cyclictest14634-21snmpd22:10:083
2436299204,3cyclictest0-21swapper/300:02:313
2436299201,9cyclictest0-21swapper/320:00:243
2436299201,6cyclictest0-21swapper/323:28:503
2436299201,5cyclictest0-21swapper/322:06:133
2436299201,12cyclictest0-21swapper/321:23:213
2436299201,11cyclictest16708-21/usr/sbin/munin23:45:193
2436299201,11cyclictest0-21swapper/300:15:123
2436299200,8cyclictest14287-21sadc23:40:013
2436299200,7cyclictest0-21swapper/300:34:333
2435599209,2cyclictest10753-21kworker/2:122:29:512
2435599206,3cyclictest10753-21kworker/2:119:35:222
2435599205,8cyclictest32-21ksoftirqd/222:42:042
2435599203,2cyclictest10753-21kworker/2:121:27:362
2435599201,2cyclictest86750irq/128-enp1s0-21:20:422
24355992010,2cyclictest86850irq/129-enp1s0-23:06:282
2435099203,3cyclictest0-21swapper/120:24:121
2434599209,3cyclictest14634-21snmpd21:40:020
2434599201,5cyclictest0-21swapper/021:30:180
2434599201,4cyclictest0-21swapper/023:45:470
2434599201,4cyclictest0-21swapper/000:23:260
2434599201,12cyclictest0-21swapper/022:33:300
2434599201,11cyclictest25858-21/usr/sbin/munin20:25:280
2434599201,10cyclictest0-21swapper/020:57:180
73442191,3sleep00-21swapper/023:20:420
2436299199,2cyclictest39-21rcuc/323:05:233
2436299198,3cyclictest14512-21/usr/sbin/munin23:40:273
2436299194,4cyclictest0-21swapper/322:00:123
2436299192,7cyclictest41-21ksoftirqd/319:10:173
2436299191,6cyclictest0-21swapper/321:28:233
2436299191,15cyclictest12156-21ntp_states21:05:273
2436299191,12cyclictest0-21swapper/323:15:203
2436299191,11cyclictest0-21swapper/321:37:303
2436299191,10cyclictest29635-21/usr/sbin/munin19:20:153
2436299191,10cyclictest20836-21diskstats22:40:133
2436299191,10cyclictest0-21swapper/322:56:593
2436299191,10cyclictest0-21swapper/322:56:593
2436299191,10cyclictest0-21swapper/322:32:193
2436299190,5cyclictest0-21swapper/300:12:393
2436299190,11cyclictest0-21swapper/322:21:573
2436299190,11cyclictest0-21swapper/320:15:193
2435599195,7cyclictest32-21ksoftirqd/200:38:562
2435599193,8cyclictest10753-21kworker/2:100:29:002
2435599191,3cyclictest86750irq/128-enp1s0-22:50:102
2434599198,10cyclictest0-21swapper/021:20:010
2434599191,7cyclictest0-21swapper/023:03:570
2434599191,6cyclictest0-21swapper/000:30:400
2434599191,5cyclictest0-21swapper/023:33:150
2434599191,10cyclictest23650-21/usr/sbin/munin20:20:280
57182181,2sleep30-21swapper/323:20:093
2436299189,2cyclictest3871-21seq00:29:213
2436299188,2cyclictest3862-21/usr/sbin/munin19:35:133
2436299185,4cyclictest6412-21diskstats19:40:143
2436299184,2cyclictest87950irq/132-enp2s0-21:30:113
2436299183,8cyclictest12050irq/123-ahci[0021:45:013
2436299181,9cyclictest0-21swapper/323:57:353
2436299181,6cyclictest0-21swapper/323:33:183
24362991812,3cyclictest7769-21grep00:35:293
24362991811,2cyclictest86550irq/126-enp1s0-21:58:003
2436299181,12cyclictest0-21swapper/323:54:043
2436299181,11cyclictest0-21swapper/322:45:243
2434599187,3cyclictest14634-21snmpd00:01:200
2434599182,6cyclictest20174-21kworker/u8:122:50:240
24345991813,3cyclictest14264-21grep22:25:130
24345991812,3cyclictest9943-21sh21:00:220
2434599181,10cyclictest0-21swapper/019:30:130
2434599180,10cyclictest17055-21/usr/sbin/munin20:05:280
2436299174,3cyclictest0-21swapper/322:25:213
2436299171,6cyclictest0-21swapper/322:50:083
24362991713,3cyclictest0-21swapper/320:41:383
24362991712,3cyclictest395-21nfs4_client19:25:223
24362991712,3cyclictest28742-21hwlatdetect20:30:153
24362991712,3cyclictest2284-21/usr/sbin/munin20:45:173
24362991712,3cyclictest2284-21/usr/sbin/munin20:45:173
24362991712,3cyclictest18301-21sendmail_mailst20:05:253
2436299171,14cyclictest0-21swapper/321:00:153
2436299171,13cyclictest30253-21/usr/sbin/munin20:35:263
2434599177,2cyclictest6817-21sshd22:06:510
2434599176,8cyclictest11185-21grep21:05:140
2434599175,3cyclictest0-21swapper/022:55:140
2434599175,3cyclictest0-21swapper/022:55:140
2434599171,6cyclictest6039-21grep20:50:270
2434599171,6cyclictest0-21swapper/021:56:260
2434599171,5cyclictest20434-21grep21:25:140
24345991713,2cyclictest0-21swapper/022:40:140
24345991712,3cyclictest14439-21seq19:57:490
2434599171,15cyclictest0-21swapper/020:02:170
24345991711,3cyclictest29977-21seq19:20:120
2434599171,13cyclictest24549-21/usr/sbin/munin19:10:140
24345991711,3cyclictest22801-21grep20:15:250
24345991710,4cyclictest23307-21grep22:45:140
123522170,2sleep00-21swapper/019:54:120
324452160,2sleep00-21swapper/020:40:090
2436299165,3cyclictest10450-21/usr/sbin/munin19:50:163
24362991612,3cyclictest0-21swapper/319:56:583
24362991612,3cyclictest0-21swapper/319:47:043
24362991612,3cyclictest0-21swapper/319:47:043
2436299161,10cyclictest0-21swapper/319:17:043
24362991610,3cyclictest23799-21cpuspeed_turbos20:20:113
2434599161,6cyclictest25528-21/usr/sbin/munin00:05:090
2434599161,6cyclictest25528-21/usr/sbin/munin00:05:090
2434599161,5cyclictest0-21swapper/022:15:130
24345991612,3cyclictest0-21swapper/023:05:050
24345991612,3cyclictest0-21swapper/019:48:230
24345991612,3cyclictest0-21swapper/019:48:230
2434599161,12cyclictest30845-21head20:35:160
24345991610,3cyclictest19786-21fschecks_time20:10:130
2434599151,5cyclictest0-21swapper/019:25:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional