You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-23 - 12:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Apr 23, 2024 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
68192240165,24sleep10-21swapper/119:09:371
65432238164,24sleep30-21swapper/319:06:053
65032210177,22sleep00-21swapper/019:05:340
66682201165,24sleep20-21swapper/219:07:412
6951997369,3cyclictest0-21swapper/222:27:102
6951997369,3cyclictest0-21swapper/200:13:542
186442590,4sleep1694399cyclictest20:45:301
49172570,4sleep1694399cyclictest23:55:261
194212530,1sleep30-21swapper/319:35:163
35932460,1sleep03595-21pmu-power21:25:280
147112460,2sleep10-21swapper/121:50:251
6951994340,2cyclictest0-21swapper/223:31:072
62192430,5sleep232-21ksoftirqd/200:00:122
695599419,6cyclictest14936-21expr00:20:143
695599419,6cyclictest14936-21expr00:20:143
695199419,2cyclictest8965-21kworker/2:219:32:392
6955994013,3cyclictest12063-21irqstats21:45:193
695199391,25cyclictest0-21swapper/222:15:182
695199374,7cyclictest20400-21chrt23:19:432
695199374,7cyclictest20400-21chrt23:19:432
6951993711,4cyclictest14205-21basename19:25:012
6955993615,3cyclictest5037-21turbostat.cron22:45:013
6955993613,20cyclictest86650irq/127-enp1s0-19:40:173
695199362,14cyclictest5334-21sort20:15:302
695199355,3cyclictest0-21swapper/221:45:192
695199351,6cyclictest3583-21cat23:55:032
6951993511,4cyclictest5183-21grep22:45:032
695599345,12cyclictest780-21rngd20:46:253
6955993412,3cyclictest366-21processes23:45:243
6955993410,5cyclictest5765-21grep22:45:173
695199341,6cyclictest3690-21/usr/sbin/munin23:55:242
695599335,19cyclictest780-21rngd22:54:593
695599334,10cyclictest780-21rngd20:03:513
695599331,7cyclictest0-21swapper/323:54:153
695599331,4cyclictest780-21rngd20:13:293
695199333,14cyclictest0-21swapper/219:56:482
695199331,6cyclictest0-21swapper/222:50:242
695199331,5cyclictest0-21swapper/221:16:272
6951993312,4cyclictest10021-21unixbench_singl20:25:302
695199331,21cyclictest0-21swapper/219:50:222
693899339,11cyclictest6641-21awk21:35:010
695599324,11cyclictest780-21rngd20:56:233
695599322,3cyclictest19113-21/usr/sbin/munin00:30:213
6955993216,3cyclictest31242-21sshd23:42:473
6955993214,4cyclictest12211-21grep20:30:283
6955993212,3cyclictest15952-21grep19:25:273
695599321,10cyclictest14074-21/usr/sbin/munin23:05:243
695199325,11cyclictest780-21rngd20:57:122
695199322,3cyclictest0-21swapper/223:13:462
695199321,7cyclictest0-21swapper/222:35:302
695199321,6cyclictest0-21swapper/200:31:342
695199321,6cyclictest0-21swapper/200:19:292
695199321,5cyclictest0-21swapper/222:00:482
695599315,13cyclictest780-21rngd21:30:553
695599313,14cyclictest0-21swapper/321:24:513
695599311,6cyclictest0-21swapper/321:55:433
6955993112,5cyclictest26472-21ntp_states19:50:213
695599311,15cyclictest0-21swapper/323:12:473
695599311,12cyclictest0-21swapper/323:59:433
695199314,3cyclictest86850irq/129-enp1s0-21:50:162
6951993121,2cyclictest0-21swapper/221:35:172
695199311,6cyclictest0-21swapper/222:06:352
695199311,6cyclictest0-21swapper/200:21:312
695199311,6cyclictest0-21swapper/200:21:312
695199311,5cyclictest27365-21idleruntime22:20:192
695199311,5cyclictest0-21swapper/200:39:282
695199311,13cyclictest0-21swapper/222:14:062
695199311,13cyclictest0-21swapper/222:14:062
695199311,10cyclictest0-21swapper/223:49:422
201862310,3sleep00-21swapper/000:30:250
695599306,10cyclictest780-21rngd19:23:313
695599301,2cyclictest0-21swapper/320:41:483
6955993011,3cyclictest18101-21ls22:00:123
695599301,13cyclictest0-21swapper/320:25:333
695599301,12cyclictest5333-21ls20:15:293
695199301,4cyclictest0-21swapper/223:05:422
695199301,20cyclictest0-21swapper/219:10:292
695199301,12cyclictest0-21swapper/200:07:112
693899302,13cyclictest23816-21/usr/sbin/munin21:00:120
235112300,4sleep3401ktimersoftd/323:25:173
695599298,3cyclictest30105-21smart_sda22:25:253
695599291,6cyclictest0-21swapper/322:18:073
695599290,5cyclictest0-21swapper/322:35:273
695199299,3cyclictest0-21swapper/220:11:422
695199298,7cyclictest24883-21ntp_states21:00:212
695199291,19cyclictest0-21swapper/219:39:492
695199291,13cyclictest0-21swapper/222:30:052
695199291,11cyclictest0-21swapper/220:45:492
694399291,16cyclictest0-21swapper/100:22:111
694399291,16cyclictest0-21swapper/100:22:111
6955992810,3cyclictest25978-21memory23:30:213
695199285,10cyclictest780-21rngd23:25:072
695199282,10cyclictest0-21swapper/220:00:182
695199281,5cyclictest0-21swapper/221:57:062
695199281,3cyclictest0-21swapper/223:02:422
695199281,3cyclictest0-21swapper/222:56:382
695199281,3cyclictest0-21swapper/221:31:412
695199281,12cyclictest0-21swapper/220:40:542
695199281,11cyclictest0-21swapper/220:30:462
694399283,14cyclictest23811-40dnf21:00:121
181752280,3sleep121-21rcuc/100:25:281
695599279,3cyclictest12600-21cat00:15:093
695599276,3cyclictest0-21swapper/319:32:393
6955992711,6cyclictest86650irq/127-enp1s0-22:05:153
695599271,13cyclictest0-21swapper/321:51:153
695599271,11cyclictest780-21rngd21:00:013
695599271,11cyclictest0-21swapper/300:14:113
695599270,11cyclictest0-21swapper/320:20:373
695199274,3cyclictest0-21swapper/220:22:462
695199273,4cyclictest20465-21kworker/u8:221:22:532
695199271,12cyclictest0-21swapper/222:49:022
695199271,10cyclictest0-21swapper/223:20:382
695199271,10cyclictest0-21swapper/200:25:182
694399271,6cyclictest29540-21sshd19:59:481
693899279,15cyclictest23282-21munin-run00:40:010
218902270,1sleep00-21swapper/019:40:220
695599268,3cyclictest7487-21diskstats19:10:133
695599264,13cyclictest780-21rngd19:56:513
695599264,10cyclictest0-21swapper/300:37:233
695599263,12cyclictest0-21swapper/321:43:273
695599261,3cyclictest0-21swapper/300:04:323
6955992612,3cyclictest22831-21cat23:25:013
695599261,10cyclictest0-21swapper/300:25:173
695599260,14cyclictest0-21swapper/320:09:413
695599260,14cyclictest0-21swapper/320:09:413
695599260,14cyclictest0-21swapper/319:48:373
695199267,3cyclictest0-21swapper/223:38:142
695199263,3cyclictest0-21swapper/221:26:132
695199261,9cyclictest0-21swapper/219:18:052
695199261,17cyclictest0-21swapper/219:25:502
695199261,12cyclictest0-21swapper/221:44:452
695199261,10cyclictest0-21swapper/220:54:212
695199261,10cyclictest0-21swapper/219:41:172
694399261,10cyclictest28625-21perf22:25:011
695599252,3cyclictest0-21swapper/321:07:493
695599252,3cyclictest0-21swapper/321:07:493
695199252,3cyclictest0-21swapper/220:08:062
695199252,3cyclictest0-21swapper/220:08:062
695199251,13cyclictest0-21swapper/220:38:142
694399251,17cyclictest24187-21sshd22:14:481
694399251,17cyclictest24187-21sshd22:14:481
693899252,7cyclictest7-21ksoftirqd/000:16:180
6938992511,3cyclictest20399-21kworker/0:019:14:370
693899250,4cyclictest0-21swapper/021:14:160
695599246,3cyclictest0-21swapper/321:19:093
695599241,13cyclictest0-21swapper/321:37:073
695599241,12cyclictest780-21rngd21:10:553
695599241,12cyclictest0-21swapper/322:57:073
695599241,11cyclictest0-21swapper/322:22:333
695599240,16cyclictest0-21swapper/322:30:373
695199248,3cyclictest0-21swapper/223:41:012
695199246,2cyclictest0-21swapper/221:10:112
694399249,6cyclictest0-21swapper/100:15:211
694399242,16cyclictest86850irq/129-enp1s0-22:58:001
694399241,6cyclictest0-21swapper/121:21:401
694399241,16cyclictest987-21kworker/u8:123:47:431
693899245,10cyclictest32618-21/usr/sbin/munin21:20:230
693899244,13cyclictest25317-21sed22:15:210
693899241,12cyclictest3480-21perf23:55:010
6938992410,8cyclictest88150irq/134-enp2s0-22:30:310
107192240,3sleep121-21rcuc/100:10:151
695599235,3cyclictest0-21swapper/300:05:573
695599232,13cyclictest780-21rngd19:16:093
695599231,14cyclictest0-21swapper/320:35:033
695599231,11cyclictest0-21swapper/323:19:233
695599231,11cyclictest0-21swapper/323:19:233
695199231,9cyclictest0-21swapper/221:09:502
695199231,9cyclictest0-21swapper/221:09:492
695199231,11cyclictest0-21swapper/219:47:342
694399236,2cyclictest25592-21pmu-power22:15:251
694399233,6cyclictest0-21swapper/122:06:131
694399231,3cyclictest21620-21/usr/sbin/munin20:55:251
694399231,3cyclictest19447-40dnf22:00:311
694399231,16cyclictest0-21swapper/121:38:091
694399230,17cyclictest0-21swapper/120:04:201
693899237,7cyclictest28787-21cat22:25:010
693899233,2cyclictest11356-21kworker/0:021:45:210
6938992316,4cyclictest22248-21munin-node22:10:100
6938992316,4cyclictest22248-21munin-node22:10:100
693899231,11cyclictest0-21swapper/000:25:240
695599229,2cyclictest86650irq/127-enp1s0-21:25:233
695599224,3cyclictest0-21swapper/323:04:213
695599223,11cyclictest0-21swapper/323:35:073
694399223,7cyclictest7550-40dnf00:01:041
694399221,7cyclictest0-21swapper/122:28:581
6943992216,3cyclictest7154-21cpu19:10:101
694399221,4cyclictest10884-21cat20:30:131
694399221,4cyclictest0-21swapper/123:54:001
694399221,4cyclictest0-21swapper/123:11:161
694399221,4cyclictest0-21swapper/122:49:571
694399221,3cyclictest26922-21unixbench_multi19:50:271
694399221,2cyclictest16910-21smartctl21:55:291
694399221,2cyclictest0-21swapper/122:31:491
694399221,15cyclictest0-21swapper/121:33:051
694399221,15cyclictest0-21swapper/119:35:451
694399221,10cyclictest2983-21grep20:10:251
6943992210,3cyclictest13844-21sh23:05:011
694399220,17cyclictest0-21swapper/122:51:351
694399220,15cyclictest9266-21kworker/u8:223:25:161
693899224,6cyclictest86550irq/126-enp1s0-23:15:140
693899224,6cyclictest86550irq/126-enp1s0-23:15:140
693899223,10cyclictest14634-21snmpd19:35:030
693899221,14cyclictest0-21swapper/023:28:290
693899221,13cyclictest0-21swapper/022:52:180
693899220,18cyclictest14634-21snmpd20:55:180
695599217,12cyclictest86650irq/127-enp1s0-22:10:153
695599217,12cyclictest86650irq/127-enp1s0-22:10:153
695599215,3cyclictest0-21swapper/320:51:413
694399214,7cyclictest21-21rcuc/123:30:221
694399211,4cyclictest20834-21kworker/u8:200:37:281
6943992114,2cyclictest0-21swapper/121:27:421
6943992113,2cyclictest0-21swapper/119:43:501
694399211,2cyclictest0-21swapper/120:35:551
694399211,15cyclictest0-21swapper/123:43:581
694399211,15cyclictest0-21swapper/123:23:021
694399211,15cyclictest0-21swapper/123:17:181
694399211,15cyclictest0-21swapper/123:17:181
694399211,15cyclictest0-21swapper/121:43:401
694399211,15cyclictest0-21swapper/120:54:551
694399211,15cyclictest0-21swapper/119:47:181
694399211,15cyclictest0-21swapper/119:27:461
694399211,15cyclictest0-21swapper/100:34:011
694399211,15cyclictest0-21swapper/100:06:221
694399211,11cyclictest15653-21grep23:05:291
694399211,11cyclictest11099-21ntp_states19:15:231
6943992110,3cyclictest30771-21df_inode21:15:151
694399210,6cyclictest0-21swapper/122:35:161
694399210,16cyclictest0-21swapper/120:08:161
694399210,16cyclictest0-21swapper/120:08:161
694399210,15cyclictest0-21swapper/121:08:331
694399210,15cyclictest0-21swapper/121:08:331
693899219,4cyclictest81ktimersoftd/020:35:280
693899219,3cyclictest9057-21/usr/sbin/munin21:40:310
693899217,6cyclictest19271-21timerwakeupswit22:00:280
693899216,2cyclictest86550irq/126-enp1s0-23:50:020
693899215,7cyclictest5894-21/usr/sbin/munin00:00:180
693899215,7cyclictest28112-21latency23:35:190
693899213,10cyclictest27609-21acpi19:55:140
693899212,9cyclictest29163-21kworker/0:223:40:250
693899211,9cyclictest0-21swapper/023:09:180
693899211,6cyclictest0-21swapper/022:40:000
693899211,5cyclictest14341-21kworker/u8:200:24:330
693899211,5cyclictest14341-21kworker/u8:200:24:330
693899211,15cyclictest0-21swapper/022:45:450
6938992111,2cyclictest10625-21missed_timers22:55:220
693899210,5cyclictest7-21ksoftirqd/021:15:490
694399209,3cyclictest7317-21ntp_states20:20:231
6943992017,2cyclictest0-21swapper/119:31:491
694399201,15cyclictest0-21swapper/121:11:371
694399201,14cyclictest0-21swapper/121:46:561
694399201,14cyclictest0-21swapper/120:18:131
694399200,15cyclictest0-21swapper/122:41:201
693899209,3cyclictest14634-21snmpd19:50:060
693899204,3cyclictest20399-21kworker/0:019:25:570
693899203,2cyclictest14634-21snmpd20:25:120
693899203,11cyclictest86550irq/126-enp1s0-00:10:170
693899202,5cyclictest8364-21cstates00:05:130
693899201,5cyclictest3690-21/usr/sbin/munin23:55:200
693899201,4cyclictest1624-21munin-run20:10:010
693899201,4cyclictest1624-21munin-run20:10:010
693899201,4cyclictest0-21swapper/023:21:480
693899201,14cyclictest0-21swapper/021:54:330
693899201,11cyclictest6853-21/usr/sbin/munin21:35:220
693899201,10cyclictest0-21swapper/021:59:490
693899200,6cyclictest14634-21snmpd22:25:350
6943991914,3cyclictest13705-21grep19:20:281
6943991912,2cyclictest0-21swapper/120:43:561
694399191,12cyclictest0-21swapper/123:36:111
694399190,18cyclictest0-21swapper/120:25:141
693899198,3cyclictest25059-21/usr/sbin/munin23:30:160
693899191,9cyclictest10638-21/usr/sbin/munin20:30:240
693899191,10cyclictest4043-21/usr/sbin/munin20:15:290
693899191,10cyclictest14634-21snmpd19:25:010
693899191,10cyclictest0-21swapper/023:14:090
693899191,10cyclictest0-21swapper/022:05:570
693899190,8cyclictest23390-21kworker/0:021:08:070
693899190,8cyclictest23390-21kworker/0:021:08:070
693899190,10cyclictest0-21swapper/023:00:230
693899183,2cyclictest31540-21kworker/0:220:10:270
693899181,6cyclictest17227-21/usr/sbin/munin20:45:170
6938991812,3cyclictest6578-21sh20:20:120
6938991812,3cyclictest15020-21seq20:40:040
693899181,11cyclictest0-21swapper/022:40:190
693899178,2cyclictest0-21swapper/020:00:250
693899172,7cyclictest5750irq/9-acpi19:30:180
693899171,8cyclictest86750irq/128-enp1s0-19:15:110
693899171,8cyclictest23207-21/usr/sbin/munin19:45:190
693899171,6cyclictest19855-21cut20:50:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional