You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-26 - 01:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Thu Apr 25, 2024 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
288991940,84ptp4l0-21swapper/107:05:451
3882760,1sleep100-21swapper/1007:05:312
94480750,70rtkit-daemon0-21swapper/607:06:148
63426553,10sleep70-21swapper/707:08:539
2987326251,9sleep00-21swapper/007:05:100
68826050,8sleep50-21swapper/507:09:397
49626050,8sleep110-21swapper/1107:06:573
3040825745,9sleep40-21swapper/407:05:136
37912530,0sleep80-21swapper/812:05:1410
37912530,0sleep80-21swapper/812:05:1410
111799222,7cyclictest0-21swapper/410:04:596
111799222,7cyclictest0-21swapper/408:15:016
111799222,15cyclictest0-21swapper/412:10:016
111799222,15cyclictest0-21swapper/412:10:006
109899211,14cyclictest0-21swapper/111:55:011
109899211,14cyclictest0-21swapper/111:55:001
112699203,10cyclictest0-21swapper/608:15:288
288991180,1ptp4l21-21ksoftirqd/110:36:461
113099183,10cyclictest0-21swapper/708:35:009
113099181,3cyclictest0-21swapper/709:20:179
110499184,9cyclictest0-21swapper/212:05:014
110499184,9cyclictest0-21swapper/212:05:014
109899187,7cyclictest0-21swapper/112:00:011
109899187,7cyclictest0-21swapper/112:00:011
288991170,0ptp4l0-21swapper/109:45:141
288991170,0ptp4l0-21swapper/108:50:161
288991170,0ptp4l0-21swapper/107:45:221
113099173,8cyclictest0-21swapper/710:10:009
113099173,8cyclictest0-21swapper/709:30:009
113099172,9cyclictest0-21swapper/710:35:009
111199173,9cyclictest0-21swapper/309:00:525
111199173,8cyclictest0-21swapper/310:25:015
111199173,8cyclictest0-21swapper/308:20:015
111199172,9cyclictest0-21swapper/310:35:015
110499173,9cyclictest0-21swapper/211:05:004
110499172,9cyclictest0-21swapper/210:10:004
110499172,8cyclictest0-21swapper/208:50:004
109499170,15cyclictest0-21swapper/012:35:180
109499170,10cyclictest0-21swapper/008:30:010
4032167,7sleep80-21swapper/807:05:4210
288991160,1ptp4l21-21ksoftirqd/112:35:121
288991160,1ptp4l21-21ksoftirqd/112:15:141
288991160,1ptp4l21-21ksoftirqd/111:45:481
288991160,1ptp4l21-21ksoftirqd/111:45:481
288991160,1ptp4l21-21ksoftirqd/109:50:161
288991160,1ptp4l21-21ksoftirqd/109:30:251
288991160,1ptp4l21-21ksoftirqd/109:25:181
288991160,1ptp4l21-21ksoftirqd/109:00:381
288991160,1ptp4l21-21ksoftirqd/108:35:291
288991160,1ptp4l21-21ksoftirqd/108:25:011
288991160,0ptp4l0-21swapper/109:10:271
288991160,0ptp4l0-21swapper/108:15:111
288991160,0ptp4l0-21swapper/107:20:181
114299161,8cyclictest0-21swapper/907:45:0111
112699163,9cyclictest0-21swapper/611:05:008
112699160,11cyclictest2992-21switchtime11:20:248
111799163,8cyclictest0-21swapper/412:20:376
111199163,8cyclictest0-21swapper/307:40:015
111199162,8cyclictest0-21swapper/308:50:005
110499162,8cyclictest0-21swapper/212:25:014
109499163,8cyclictest0-21swapper/007:10:010
288991150,1ptp4l21-21ksoftirqd/110:53:231
288991150,1ptp4l21-21ksoftirqd/110:25:261
288991150,1ptp4l21-21ksoftirqd/109:40:291
288991150,1ptp4l21-21ksoftirqd/109:05:311
288991150,0ptp4l0-21swapper/112:30:171
288991150,0ptp4l0-21swapper/111:40:161
288991150,0ptp4l0-21swapper/111:25:171
1146991514,1cyclictest0-21swapper/1010:35:262
113099153,7cyclictest0-21swapper/709:34:599
113099153,7cyclictest0-21swapper/709:00:009
113099151,8cyclictest0-21swapper/707:55:019
112699150,12cyclictest0-21swapper/609:38:278
1121991513,1cyclictest0-21swapper/510:25:187
1117991512,2cyclictest0-21swapper/407:45:186
111199153,6cyclictest0-21swapper/311:15:015
111199152,8cyclictest0-21swapper/309:10:285
111199152,8cyclictest0-21swapper/308:20:265
111199150,3cyclictest0-21swapper/310:10:195
110499153,8cyclictest0-21swapper/208:45:004
110499153,7cyclictest0-21swapper/212:30:294
110499151,9cyclictest0-21swapper/208:50:314
109899153,7cyclictest0-21swapper/107:10:261
109499153,7cyclictest0-21swapper/012:05:010
109499153,7cyclictest0-21swapper/012:05:000
109499152,8cyclictest0-21swapper/007:45:000
102742150,0sleep00-21swapper/010:05:290
288991140,1ptp4l21-21ksoftirqd/112:25:301
288991140,1ptp4l21-21ksoftirqd/111:00:261
288991140,1ptp4l21-21ksoftirqd/110:55:231
288991140,1ptp4l21-21ksoftirqd/110:20:211
288991140,1ptp4l21-21ksoftirqd/110:01:281
288991140,1ptp4l21-21ksoftirqd/109:55:201
288991140,1ptp4l21-21ksoftirqd/109:35:211
288991140,1ptp4l21-21ksoftirqd/108:40:111
288991140,0ptp4l0-21swapper/107:35:171
288991140,0ptp4l0-21swapper/107:15:021
277812140,0sleep100-21swapper/1009:05:202
114299143,7cyclictest0-21swapper/911:10:0111
114299142,8cyclictest0-21swapper/909:50:3011
113599143,7cyclictest0-21swapper/811:50:0010
113599143,7cyclictest0-21swapper/811:50:0010
113599143,7cyclictest0-21swapper/810:50:0110
113599142,8cyclictest0-21swapper/811:40:0010
113099142,7cyclictest0-21swapper/709:55:009
113099142,7cyclictest0-21swapper/709:45:299
111799142,8cyclictest0-21swapper/410:25:266
111199142,8cyclictest0-21swapper/307:30:005
111199142,7cyclictest0-21swapper/309:15:005
111199140,8cyclictest0-21swapper/307:15:015
110499143,8cyclictest0-21swapper/209:30:004
109899142,7cyclictest0-21swapper/111:15:001
109499142,7cyclictest0-21swapper/007:35:010
1094991411,2cyclictest26899-21turbostat11:55:000
1094991411,2cyclictest26899-21turbostat11:54:590
6932136,5sleep90-21swapper/907:09:4311
288991130,6ptp4l21-21ksoftirqd/112:00:261
288991130,6ptp4l21-21ksoftirqd/112:00:251
288991130,3ptp4l9499-21fw_conntrack11:30:181
288991130,2ptp4l1295-21df_inode11:20:121
288991130,1ptp4l21-21ksoftirqd/110:05:161
288991130,1ptp4l21-21ksoftirqd/108:08:421
288991130,0ptp4l0-21swapper/110:30:191
1146991312,1cyclictest0-21swapper/1009:40:172
114299133,6cyclictest0-21swapper/909:10:0011
114299132,5cyclictest0-21swapper/911:35:0011
113599132,7cyclictest0-21swapper/808:05:2610
113599130,12cyclictest0-21swapper/812:15:3410
113099133,6cyclictest0-21swapper/711:55:269
113099133,6cyclictest0-21swapper/711:55:269
113099130,13cyclictest0-21swapper/712:39:119
112699133,4cyclictest0-21swapper/609:34:598
112699132,8cyclictest0-21swapper/607:15:288
112699132,7cyclictest0-21swapper/609:59:598
111799130,5cyclictest0-21swapper/412:10:186
111799130,5cyclictest0-21swapper/412:10:186
111199133,6cyclictest0-21swapper/311:55:015
111199133,6cyclictest0-21swapper/311:55:015
110499130,13cyclictest0-21swapper/207:44:584
109499133,7cyclictest0-21swapper/009:15:000
109499132,7cyclictest0-21swapper/011:30:300
109499132,7cyclictest0-21swapper/010:55:250
109499132,6cyclictest0-21swapper/008:00:000
288991120,1ptp4l21-21ksoftirqd/108:55:511
288991120,1ptp4l21-21ksoftirqd/108:30:261
288991120,1ptp4l21-21ksoftirqd/107:50:221
288991120,0ptp4l0-21swapper/112:05:181
288991120,0ptp4l0-21swapper/112:05:181
288991120,0ptp4l0-21swapper/111:38:061
288991120,0ptp4l0-21swapper/107:29:051
229682120,0sleep50-21swapper/507:35:177
114299122,6cyclictest0-21swapper/909:35:0111
113599129,2cyclictest28774-21missed_timers10:30:2110
113599122,6cyclictest0-21swapper/808:40:0010
113599120,7cyclictest0-21swapper/812:30:0010
113599120,12cyclictest0-21swapper/807:50:1310
113599120,11cyclictest11917-21sed07:20:2010
113099122,8cyclictest0-21swapper/708:10:019
113099122,6cyclictest0-21swapper/708:00:019
1130991210,1cyclictest0-21swapper/711:19:599
112699125,1cyclictest0-21swapper/611:41:388
112699122,5cyclictest0-21swapper/610:15:008
112699122,5cyclictest0-21swapper/607:25:318
1126991210,1cyclictest29556-21basename09:10:008
1121991210,1cyclictest28969-21turbostat07:45:017
111799121,7cyclictest0-21swapper/410:20:006
111199122,7cyclictest0-21swapper/311:50:015
111199122,7cyclictest0-21swapper/311:50:005
111199122,6cyclictest0-21swapper/311:10:325
111199121,3cyclictest0-21swapper/310:56:355
110499123,8cyclictest0-21swapper/211:50:004
110499123,8cyclictest0-21swapper/211:50:004
110499123,8cyclictest0-21swapper/208:30:254
109899129,2cyclictest29580-21cstates11:15:131
109899122,6cyclictest0-21swapper/108:15:011
109499121,7cyclictest0-21swapper/007:50:260
109499120,6cyclictest0-21swapper/012:25:010
288991110,2ptp4l17065-21cut10:15:191
288991110,1ptp4l8655-21memory12:10:241
288991110,1ptp4l8655-21memory12:10:231
288991110,1ptp4l23137-21irqrtprio11:05:211
288991110,1ptp4l21-21ksoftirqd/110:10:391
288991110,0ptp4l0-21swapper/112:20:311
288991110,0ptp4l0-21swapper/110:46:491
288991110,0ptp4l0-21swapper/108:25:141
288991110,0ptp4l0-21swapper/108:00:161
272291110,0getstats0-21swapper/211:15:224
272291110,0getstats0-21swapper/209:55:214
114699118,2cyclictest29065-21irqrtprio11:55:182
114699118,2cyclictest29065-21irqrtprio11:55:172
114699112,6cyclictest0-21swapper/1009:54:592
114699112,5cyclictest0-21swapper/1012:25:012
114699110,1cyclictest0-21swapper/1007:55:112
114699110,10cyclictest0-21swapper/1008:55:142
113599118,2cyclictest32665-21cat11:20:0010
113599112,6cyclictest0-21swapper/808:20:0010
113599110,1cyclictest0-21swapper/812:30:3010
113599110,1cyclictest0-21swapper/808:30:1010
113099118,2cyclictest9902-21turbostat07:20:019
113099110,1cyclictest0-21swapper/709:41:529
112699118,2cyclictest28682-21latency10:30:208
112699112,8cyclictest31609-21sed09:10:188
112699112,6cyclictest0-21swapper/610:55:018
112699111,6cyclictest0-21swapper/608:00:318
112699110,10cyclictest0-21swapper/611:50:148
112699110,10cyclictest0-21swapper/611:50:138
111799118,2cyclictest7886-21awk11:29:596
111799110,3cyclictest6992-21cat08:40:006
111199116,4cyclictest28514-21aten2_r3power_p09:50:125
111199112,7cyclictest0-21swapper/309:35:325
111199110,8cyclictest94550avahi-daemon11:03:085
109899112,6cyclictest0-21swapper/110:45:001
109899112,5cyclictest0-21swapper/107:35:011
109899110,11cyclictest0-21swapper/109:15:261
109499117,3cyclictest5077-21pool09:05:470
109499110,10cyclictest19424-21/usr/sbin/munin08:55:130
291091100,1phc2sys0-21swapper/211:55:184
291091100,1phc2sys0-21swapper/211:55:184
288991100,2ptp4l21-21ksoftirqd/107:55:221
288991100,1ptp4l21-21ksoftirqd/109:20:101
288991100,0ptp4l0-21swapper/108:45:131
288991100,0ptp4l0-21swapper/107:40:171
115199109,1cyclictest0-21swapper/1111:48:303
115199109,1cyclictest0-21swapper/1111:48:293
115199109,1cyclictest0-21swapper/1107:33:523
115199100,9cyclictest32494-21switchtime09:10:273
115199100,3cyclictest2964-21ls12:05:113
115199100,3cyclictest2964-21ls12:05:103
115199100,10cyclictest0-21swapper/1109:57:103
114699109,1cyclictest0-21swapper/1011:45:142
114699109,1cyclictest0-21swapper/1011:45:132
114699106,3cyclictest6899-21cpuspeed_turbos07:15:122
114699102,6cyclictest0-21swapper/1011:00:002
114699100,4cyclictest0-21swapper/1008:53:242
114699100,10cyclictest0-21swapper/1011:43:092
114299109,1cyclictest0-21swapper/910:43:1611
114299109,1cyclictest0-21swapper/908:24:3111
114299109,1cyclictest0-21swapper/908:00:3911
114299106,3cyclictest22237-21if_err_p34p109:40:1811
114299103,7cyclictest0-21swapper/912:04:5511
114299103,7cyclictest0-21swapper/912:04:5411
114299102,7cyclictest0-21swapper/907:25:0111
114299101,8cyclictest22189-21grep07:35:1211
114299100,4cyclictest32423-21swap09:10:2611
114299100,10cyclictest0-21swapper/912:31:4011
113599109,1cyclictest0-21swapper/811:31:3910
113599109,1cyclictest0-21swapper/810:35:0110
113599107,2cyclictest21946-21cut08:15:2010
113599106,4cyclictest0-21swapper/812:00:1710
113599106,4cyclictest0-21swapper/812:00:1610
113599106,3cyclictest31061-21cut07:45:2110
113599104,1cyclictest0-21swapper/809:27:4910
113599101,6cyclictest0-21swapper/812:15:0110
113599101,6cyclictest0-21swapper/812:15:0010
113599101,6cyclictest0-21swapper/809:00:0010
113599100,9cyclictest8435-21grep09:20:2710
113599100,9cyclictest457-21unixbench-2d09:10:2810
113599100,9cyclictest29403-21sort11:55:1910
113599100,9cyclictest29403-21sort11:55:1910
113599100,9cyclictest29240-21df09:50:1210
113599100,9cyclictest2054-21phc2sys09:55:2410
113599100,10cyclictest0-21swapper/810:06:5010
113099108,1cyclictest0-21swapper/711:20:119
113099107,2cyclictest0-21swapper/712:20:369
113099106,3cyclictest5197-21munin-node10:45:119
113099106,3cyclictest30897-21awk10:35:109
113099106,3cyclictest26011-21sed11:50:299
113099106,3cyclictest26011-21sed11:50:299
113099106,3cyclictest15393-21/usr/sbin/munin10:15:109
113099106,3cyclictest1033-21runrttasks08:40:329
113099105,4cyclictest10551-21/usr/sbin/munin12:15:319
113099100,9cyclictest17949-21cpuspeed_turbos11:00:139
113099100,3cyclictest17449-21fschecks_time08:10:209
113099100,10cyclictest0-21swapper/707:34:359
112699106,3cyclictest16108-21smartctl10:55:238
112699102,6cyclictest0-21swapper/612:15:018
112699102,6cyclictest0-21swapper/612:15:008
112699100,9cyclictest7450-21cat09:20:208
112699100,9cyclictest27724-21/usr/sbin/munin12:35:208
112699100,9cyclictest24454-21/usr/sbin/munin10:25:198
112699100,9cyclictest1033-21runrttasks10:09:468
112699100,2cyclictest5607-21perl11:25:178
112699100,10cyclictest0-21swapper/612:04:018
112699100,10cyclictest0-21swapper/612:04:008
112699100,10cyclictest0-21swapper/611:35:438
112199109,1cyclictest0-21swapper/511:26:597
112199109,1cyclictest0-21swapper/507:57:447
112199108,1cyclictest0-21swapper/511:35:177
112199106,4cyclictest0-21swapper/511:09:547
112199100,9cyclictest1033-21runrttasks11:40:517
112199100,1cyclictest0-21swapper/509:55:187
111799109,1cyclictest0-21swapper/409:25:496
111799109,1cyclictest0-21swapper/409:15:486
111799108,1cyclictest0-21swapper/409:00:526
111799100,9cyclictest8071-21grep10:05:136
111799100,10cyclictest0-21swapper/412:38:456
111799100,10cyclictest0-21swapper/410:49:486
111199109,1cyclictest0-21swapper/310:01:145
111199107,2cyclictest20967-21timerwakeupswit12:25:305
111199107,2cyclictest0-21swapper/311:29:595
111199100,9cyclictest28957-21fschecks_time08:25:155
111199100,9cyclictest28641-21switchtime09:05:295
111199100,2cyclictest30371-21fschecks_count07:45:155
111199100,1cyclictest0-21swapper/312:09:575
111199100,1cyclictest0-21swapper/312:09:565
111199100,1cyclictest0-21swapper/310:45:155
110499109,1cyclictest0-21swapper/209:45:124
110499107,2cyclictest0-21swapper/211:25:104
110499107,2cyclictest0-21swapper/211:15:014
110499107,2cyclictest0-21swapper/207:45:144
110499106,3cyclictest855-21expr10:35:274
110499106,3cyclictest31192-21cut09:10:154
110499101,8cyclictest0-21swapper/211:38:434
110499101,4cyclictest0-21swapper/209:22:204
110499100,9cyclictest29099-21turbostat12:40:004
110499100,9cyclictest0-21swapper/208:00:224
110499100,3cyclictest21864-21/usr/sbin/munin07:35:284
110499100,2cyclictest0-21swapper/210:10:454
109499106,3cyclictest21353-21cstates09:40:130
109499105,4cyclictest12633-21needreboot12:15:240
109499105,4cyclictest10996-21expr07:20:130
109499102,7cyclictest0-21swapper/008:25:010
109499100,9cyclictest9245-21grep08:40:180
109499100,9cyclictest22488-21cpuspeed_turbos12:30:120
109499100,9cyclictest15852-21irqrtprio07:25:230
109499100,9cyclictest11098-21gunzip08:00:290
109499100,4cyclictest12813-21cpuspeed08:05:100
109499100,3cyclictest32266-21/usr/sbin/munin09:55:210
109499100,3cyclictest19790-21/usr/sbin/munin11:45:150
109499100,3cyclictest19790-21/usr/sbin/munin11:45:150
109499100,3cyclictest1642-21perl11:20:130
109499100,3cyclictest14441-21grep11:35:250
109499100,1cyclictest0-21swapper/012:25:060
109499100,1cyclictest0-21swapper/011:56:010
109499100,1cyclictest0-21swapper/011:56:010
11519998,1cyclictest0-21swapper/1111:00:403
11519992,7cyclictest0-21swapper/1108:35:163
11519991,5cyclictest0-21swapper/1107:50:013
11519990,8cyclictest4877-21grep11:25:133
11519990,8cyclictest26442-21ls11:50:313
11519990,8cyclictest26442-21ls11:50:313
11519990,8cyclictest26422-21gltestperf11:10:163
11519990,8cyclictest19355-21grep08:10:313
11519990,8cyclictest17073-21ptp4l08:50:273
11519990,8cyclictest1585-21latency_hist10:40:003
11519990,8cyclictest13408-21df08:05:113
11519990,3cyclictest18794-21latency_hist10:20:003
11519990,1cyclictest0-21swapper/1108:00:243
11469998,1cyclictest0-21swapper/1009:15:122
11469998,1cyclictest0-21swapper/1007:32:302
11469995,3cyclictest1431-21cat12:00:262
11469995,3cyclictest1431-21cat12:00:262
11469995,3cyclictest1051-21cut09:15:002
11469993,5cyclictest2649-21cstates10:40:122
11469992,4cyclictest0-21swapper/1009:25:002
11469990,9cyclictest0-21swapper/1008:20:262
11469990,8cyclictest27309-21cpuspeed_turbos10:30:112
11469990,8cyclictest20734-21cpuspeed_turbos08:15:112
11469990,1cyclictest0-21swapper/1007:40:172
11429996,2cyclictest10181-21diskstats09:25:1311
11429995,3cyclictest554-21cpu11:20:1011
11429994,4cyclictest12538-21fschecks_count10:10:1411
11429993,2cyclictest23948-21fschecks_count10:25:1511
11429991,7cyclictest0-21swapper/911:58:1411
11429991,7cyclictest0-21swapper/911:58:1411
11429991,7cyclictest0-21swapper/909:20:2311
11429990,8cyclictest6926-21cpuspeed_turbos07:15:1211
11429990,8cyclictest25094-21cut11:50:2211
11429990,8cyclictest25094-21cut11:50:2111
11429990,8cyclictest15144-21expr12:20:1911
11429990,4cyclictest26963-21irqstats11:10:2111
11429990,2cyclictest30692-21irqrtprio11:15:2111
11429990,2cyclictest0-21swapper/912:15:3311
11429990,1cyclictest0-21swapper/908:10:2211
11359996,2cyclictest25595-21chrt09:03:0610
11359995,3cyclictest18160-21proc_pri11:40:2610
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional