You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-03-03 - 21:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun Mar 03, 2024 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
254942239164,24sleep30-21swapper/307:08:473
255562234162,58sleep10-21swapper/107:09:331
254562205166,26sleep00-21swapper/007:08:160
254512204166,26sleep20-21swapper/207:08:132
3000421520,4sleep22569799cyclictest07:15:322
3074621440,4sleep32570299cyclictest12:11:383
68612610,1sleep10-21swapper/108:50:281
4022590,5sleep339-21rcuc/308:35:333
323012490,1sleep20-21swapper/208:35:242
2570299431,33cyclictest0-21swapper/312:38:203
25687994216,14cyclictest31762-21/usr/sbin/munin11:00:310
25702993825,8cyclictest0-21swapper/311:26:143
2568799389,13cyclictest27594-21basename09:40:010
261742360,4sleep12569299cyclictest09:35:241
25687993612,15cyclictest22570-21awk10:40:260
25687993612,13cyclictest8726-21ntpq12:35:320
2570299351,15cyclictest0-21swapper/310:12:483
2570299351,15cyclictest0-21swapper/310:12:483
2570299351,14cyclictest0-21swapper/311:40:303
2568799358,18cyclictest0-21swapper/009:15:210
2568799358,17cyclictest13058-21basename10:20:190
25687993511,14cyclictest7165-21cpuspeed_turbos11:20:210
2570299341,30cyclictest0-21swapper/311:59:363
2570299341,22cyclictest0-21swapper/308:25:363
2570299341,14cyclictest3597-21memory11:10:303
2569299340,17cyclictest0-21swapper/109:01:381
2568799349,12cyclictest16780-21memory11:40:290
2568799349,11cyclictest10844-21/usr/sbin/munin10:15:250
25687993423,2cyclictest4950-21kworker/0:207:39:150
2570299331,19cyclictest0-21swapper/310:09:083
2570299331,13cyclictest0-21swapper/311:47:163
2570299330,22cyclictest0-21swapper/311:50:203
2569299331,18cyclictest0-21swapper/108:30:521
2568799339,13cyclictest14390-21diskmemload09:40:180
2568799339,12cyclictest3325-21gltestperf11:10:230
2568799338,14cyclictest836-21runrttasks09:52:260
2568799332,22cyclictest3105-21kworker/0:108:47:470
25687993313,7cyclictest11285-21latency_hist11:30:020
25687993313,12cyclictest1165-21cpuspeed_turbos12:20:210
25687993312,8cyclictest22076-21latency09:25:300
2568799331,21cyclictest26935-21kworker/0:008:33:230
25687993311,7cyclictest27723-21grep08:25:240
25687993311,13cyclictest30131-21ntp_states12:10:320
2570299321,16cyclictest0-21swapper/309:42:043
2570299321,13cyclictest0-21swapper/312:32:003
2570299321,11cyclictest0-21swapper/311:08:423
2569299321,22cyclictest0-21swapper/110:35:371
2568799329,12cyclictest8327-21memory07:40:270
25687993212,17cyclictest2657-21tail09:55:240
25687993211,10cyclictest22429-21seq11:55:050
25687993211,10cyclictest10389-21ls09:00:200
25687993210,9cyclictest18402-21/usr/sbin/munin08:05:340
147012320,4sleep22569799cyclictest10:21:142
2570299311,19cyclictest0-21swapper/308:04:203
2570299311,19cyclictest0-21swapper/308:04:203
2570299311,15cyclictest0-21swapper/309:24:123
2570299311,14cyclictest0-21swapper/309:33:083
2570299311,13cyclictest0-21swapper/310:35:443
2570299311,13cyclictest0-21swapper/310:15:383
2570299311,13cyclictest0-21swapper/308:53:373
2570299311,12cyclictest0-21swapper/312:23:003
2569799310,13cyclictest0-21swapper/211:35:182
2569299311,11cyclictest0-21swapper/109:56:201
2568799312,8cyclictest23650-21kworker/0:210:46:010
25687993115,8cyclictest17047-21sshd10:28:130
2568799311,22cyclictest3105-21kworker/0:108:44:590
25687993111,7cyclictest21742-21sed11:50:340
2570299301,19cyclictest0-21swapper/309:09:223
2570299301,19cyclictest0-21swapper/309:09:223
2570299301,18cyclictest0-21swapper/312:06:283
2570299301,15cyclictest0-21swapper/310:04:583
2570299301,14cyclictest0-21swapper/311:17:303
2570299301,14cyclictest0-21swapper/311:17:303
2570299301,13cyclictest0-21swapper/310:57:003
2570299301,13cyclictest0-21swapper/308:30:163
2570299301,11cyclictest0-21swapper/310:20:043
2570299301,11cyclictest0-21swapper/308:20:503
2570299301,11cyclictest0-21swapper/307:59:003
2569299308,12cyclictest86650irq/127-enp1s0-11:16:231
2569299308,12cyclictest86650irq/127-enp1s0-11:16:231
2568799307,10cyclictest5510-21smart_sda10:00:360
2568799306,10cyclictest12050irq/123-ahci[0010:15:000
2568799306,10cyclictest12050irq/123-ahci[0010:15:000
25687993013,5cyclictest4897-21taskset07:34:120
2570299291,16cyclictest0-21swapper/310:54:323
2570299291,14cyclictest0-21swapper/311:01:203
2570299291,14cyclictest0-21swapper/307:10:233
2570299291,12cyclictest0-21swapper/309:50:363
2570299291,12cyclictest0-21swapper/307:17:383
2570299291,11cyclictest0-21swapper/311:35:423
2570299291,11cyclictest0-21swapper/307:27:323
2570299291,10cyclictest0-21swapper/312:00:203
2570299291,10cyclictest0-21swapper/311:31:303
2569299291,10cyclictest0-21swapper/111:08:311
2569299291,10cyclictest0-21swapper/108:26:281
2569299290,9cyclictest0-21swapper/112:14:271
2568799297,9cyclictest32058-21cat12:15:260
2568799292,22cyclictest351-21kworker/0:107:25:150
2568799292,15cyclictest0-21swapper/012:30:210
25687992920,5cyclictest26071-21ntp_states08:20:290
25687992913,13cyclictest7262-21timerwakeupswit08:50:330
25687992912,14cyclictest823-21snmpd12:09:140
25687992911,9cyclictest86550irq/126-enp1s0-12:00:250
2568799291,19cyclictest25677-21cyclictest09:31:320
25687992910,7cyclictest20602-21/usr/sbin/munin08:10:280
10962290,4sleep12569299cyclictest09:50:381
2570299289,8cyclictest0-21swapper/309:25:103
2570299281,19cyclictest0-21swapper/309:11:203
2570299281,17cyclictest0-21swapper/310:30:203
2570299281,13cyclictest0-21swapper/312:17:523
2570299281,12cyclictest0-21swapper/309:56:263
2570299281,12cyclictest0-21swapper/309:35:063
2570299281,11cyclictest0-21swapper/312:27:043
2570299281,11cyclictest0-21swapper/307:31:243
2570299281,10cyclictest0-21swapper/311:24:443
2570299280,19cyclictest0-21swapper/307:54:143
2570299280,10cyclictest0-21swapper/308:18:183
25692992817,2cyclictest88150irq/134-enp2s0-10:13:591
25692992817,2cyclictest88150irq/134-enp2s0-10:13:591
25692992813,7cyclictest20940-21systemd-journal09:24:431
2569299281,12cyclictest0-21swapper/107:13:541
2568799287,19cyclictest88250irq/135-enp2s0-11:05:270
25687992819,6cyclictest15665-21/usr/sbin/munin09:10:270
25687992814,11cyclictest14802-21/usr/sbin/munin07:55:270
25687992811,14cyclictest4865-21sshd11:15:130
25687992811,14cyclictest4865-21sshd11:15:130
25687992810,6cyclictest12365-21cut07:50:230
298812270,4sleep00-21swapper/010:55:350
2570299277,9cyclictest0-21swapper/308:57:423
2570299271,13cyclictest0-21swapper/309:48:203
2570299271,13cyclictest0-21swapper/307:23:283
2570299271,11cyclictest0-21swapper/310:48:003
2569799273,11cyclictest25677-21cyclictest11:06:042
2569299271,8cyclictest0-21swapper/107:22:361
2569299271,11cyclictest0-21swapper/110:59:291
2568799278,9cyclictest86750irq/128-enp1s0-08:55:330
2568799278,17cyclictest0-21swapper/009:05:240
2568799278,17cyclictest0-21swapper/009:05:240
2568799278,13cyclictest16979-21kworker/u8:010:34:330
2568799275,13cyclictest0-21swapper/007:20:260
25687992719,5cyclictest30821-21/usr/sbin/munin09:45:230
25687992719,5cyclictest28605-21irqstats07:15:050
25687992719,5cyclictest20287-21smart_sda09:20:370
25687992718,6cyclictest7368-21memory10:05:290
25687992718,6cyclictest3296-21/usr/sbin/munin12:25:250
25687992718,6cyclictest31591-21/usr/sbin/munin08:35:180
25687992718,6cyclictest25892-21/usr/sbin/munin07:10:340
25687992718,6cyclictest16869-21hddtemp_smartct08:00:220
25687992718,6cyclictest16869-21hddtemp_smartct08:00:220
25687992718,6cyclictest14013-21diskstats11:35:230
25687992717,6cyclictest19986-21diskstats10:35:220
2570299261,12cyclictest0-21swapper/307:47:183
2570299261,12cyclictest0-21swapper/307:45:003
2570299261,11cyclictest0-21swapper/310:40:443
2570299260,13cyclictest0-21swapper/307:36:343
2570299260,10cyclictest0-21swapper/310:29:343
2569299261,9cyclictest0-21swapper/108:59:141
2569299261,11cyclictest0-21swapper/108:40:361
2569299260,10cyclictest20049-21latency_hist11:50:011
25687992619,4cyclictest26590-21diskstats10:50:210
25687992619,4cyclictest12685-21/usr/sbin/munin11:30:310
25687992617,6cyclictest9665-21cpu07:45:200
25687992617,6cyclictest23163-21/usr/sbin/munin08:15:230
25687992617,6cyclictest18122-21/usr/sbin/munin11:45:180
2570299251,15cyclictest0-21swapper/308:40:263
2570299251,12cyclictest0-21swapper/308:08:463
2569299251,9cyclictest0-21swapper/112:37:051
2569299251,9cyclictest0-21swapper/110:50:151
2569299251,9cyclictest0-21swapper/108:12:551
25692992512,5cyclictest28536-21kworker/u8:012:09:431
2569299251,22cyclictest0-21swapper/112:28:031
2569299251,10cyclictest0-21swapper/112:24:511
2569299251,10cyclictest0-21swapper/111:53:031
2569299251,10cyclictest0-21swapper/109:47:221
25692992510,2cyclictest21-21rcuc/111:05:001
2569299250,18cyclictest0-21swapper/109:11:481
2570299249,3cyclictest0-21swapper/309:01:183
2570299241,12cyclictest0-21swapper/308:45:363
2570299241,10cyclictest0-21swapper/308:15:003
2569299243,11cyclictest21424-21kworker/1:207:46:571
2569299241,9cyclictest0-21swapper/108:01:311
2569299241,9cyclictest0-21swapper/108:01:311
2569299241,8cyclictest0-21swapper/112:16:251
2569299241,7cyclictest0-21swapper/107:54:271
2569299241,17cyclictest0-21swapper/111:30:261
2569299241,12cyclictest0-21swapper/110:23:391
2569299241,11cyclictest0-21swapper/111:26:111
2569299241,10cyclictest0-21swapper/111:12:471
2570299237,3cyclictest17435-21tail09:15:253
2569799230,10cyclictest0-21swapper/210:30:282
2569299239,3cyclictest0-21swapper/108:48:441
2569299238,4cyclictest0-21swapper/108:19:121
2569299237,3cyclictest15597-21diskstats10:25:251
2569299231,9cyclictest0-21swapper/112:01:551
2569299231,8cyclictest0-21swapper/107:55:151
2569299231,6cyclictest0-21swapper/108:37:001
2569299231,13cyclictest0-21swapper/109:07:221
2569299231,13cyclictest0-21swapper/109:07:221
2569299231,10cyclictest0-21swapper/107:26:071
25692992310,3cyclictest29627-21grep07:15:271
2569299230,15cyclictest23-21ksoftirqd/111:20:201
2569299230,10cyclictest0-21swapper/111:41:131
2569799228,6cyclictest20166-21pmu-power09:20:362
2569299228,4cyclictest0-21swapper/112:30:431
2569299227,3cyclictest0-21swapper/107:39:591
2569299225,4cyclictest0-21swapper/109:25:061
2569299225,3cyclictest0-21swapper/111:59:551
2569299225,3cyclictest0-21swapper/109:44:521
2569299223,8cyclictest0-21swapper/110:31:591
2569299223,5cyclictest86750irq/128-enp1s0-10:00:381
2569299221,8cyclictest0-21swapper/110:09:041
2569299221,6cyclictest86750irq/128-enp1s0-11:38:481
2569299221,12cyclictest0-21swapper/110:40:231
2569299220,8cyclictest0-21swapper/108:07:231
270302210,4sleep230-21rcuc/212:05:192
2569299211,9cyclictest0-21swapper/108:22:081
2569299211,12cyclictest24109-21sed09:30:291
2569299211,11cyclictest0-21swapper/109:16:181
2569799204,7cyclictest26241-21/usr/sbin/munin10:50:372
2569799201,11cyclictest0-21swapper/207:38:042
2569799201,10cyclictest0-21swapper/212:01:522
2569299205,3cyclictest0-21swapper/107:42:471
2569299200,9cyclictest0-21swapper/110:47:371
158782201,2sleep20-21swapper/207:59:122
89922190,2sleep20-21swapper/211:24:452
2569799195,5cyclictest3772-21if_enp2s007:30:232
2569799194,5cyclictest18428-21uniq11:45:212
2569799194,3cyclictest0-21swapper/210:12:242
2569799194,3cyclictest0-21swapper/210:12:242
2569799191,6cyclictest0-21swapper/211:51:182
2569799191,5cyclictest0-21swapper/211:25:282
25697991913,3cyclictest27621-21grep08:25:242
2569799190,8cyclictest0-21swapper/209:36:562
2569299191,16cyclictest0-21swapper/107:30:411
61072181,2sleep20-21swapper/210:03:522
295322181,2sleep20-21swapper/209:42:052
2569799189,2cyclictest7452-21awk12:35:002
2569799183,9cyclictest0-21swapper/212:25:332
2569799183,7cyclictest0-21swapper/212:38:262
2569799182,9cyclictest0-21swapper/209:16:182
2569799182,6cyclictest6737-21memory08:50:272
2569799181,8cyclictest7412-21/usr/sbin/munin07:40:352
2569799181,7cyclictest31284-21gltestperf11:00:242
2569799181,7cyclictest1408-21/usr/sbin/munin08:40:322
2569799181,7cyclictest0-21swapper/210:05:372
2569799181,7cyclictest0-21swapper/209:26:182
2569799181,6cyclictest29416-21needreboot10:55:292
2569799181,6cyclictest0-21swapper/209:30:362
2569799181,5cyclictest0-21swapper/210:40:032
2569799181,5cyclictest0-21swapper/209:45:302
25697991813,3cyclictest11912-21cpuspeed_turbos07:50:212
25697991812,3cyclictest2679-21/usr/sbin/munin11:10:212
2569799181,15cyclictest12596-21sed11:30:292
2569799181,12cyclictest0-21swapper/211:41:122
2569799181,10cyclictest0-21swapper/212:11:182
2569799181,10cyclictest0-21swapper/209:53:302
2569299180,2cyclictest0-21swapper/110:15:031
231132180,5sleep230-21rcuc/211:55:232
65862170,2sleep20-21swapper/211:17:022
65862170,2sleep20-21swapper/211:17:022
2569799171,9cyclictest0-21swapper/209:05:232
2569799171,9cyclictest0-21swapper/209:05:232
2569799171,6cyclictest0-21swapper/212:20:172
2569799171,4cyclictest0-21swapper/208:08:192
2569799171,2cyclictest86550irq/126-enp1s0-10:15:202
25697991712,3cyclictest25226-21expr08:20:212
25697991712,3cyclictest16214-21uptime09:10:352
25697991711,3cyclictest23296-21fschecks_time08:15:242
25697991710,5cyclictest848-21sssd_nss07:45:242
2569799170,5cyclictest6962-1kworker/u9:110:46:282
223502171,2sleep20-21swapper/208:12:292
312762160,2sleep20-21swapper/208:34:132
2569799169,4cyclictest26543-21gltestperf07:10:232
2569799164,4cyclictest780-21rngd09:00:522
2569799163,7cyclictest25677-21cyclictest10:25:212
2569799161,7cyclictest0-21swapper/209:56:182
2569799161,7cyclictest0-21swapper/208:45:272
25697991612,3cyclictest0-21swapper/207:22:272
2569799161,14cyclictest0-21swapper/210:35:182
25697991611,3cyclictest8023-21/usr/sbin/munin08:55:302
25697991611,3cyclictest16379-21cstates08:00:202
25697991611,3cyclictest16379-21cstates08:00:202
15892160,2sleep20-21swapper/207:25:282
2569799151,6cyclictest31260-21/usr/sbin/munin12:15:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional