You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 13:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Tue Nov 18, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911000,90ptp4l0-21swapper/107:08:331
94980770,72rtkit-daemon0-21swapper/407:07:476
465127158,11sleep60-21swapper/607:08:078
448126755,9sleep110-21swapper/1107:05:363
161492670,0sleep70-21swapper/710:50:209
469326654,10sleep70-21swapper/707:08:459
455726654,10sleep50-21swapper/507:06:427
470926554,9sleep100-21swapper/1007:09:002
473326452,10sleep00-21swapper/007:09:140
70532620,2sleep333-21ksoftirqd/312:05:005
70532620,2sleep333-21ksoftirqd/312:05:005
326042580,0sleep50-21swapper/509:50:117
72122560,0sleep50-21swapper/509:15:207
242052530,0sleep50-21swapper/510:20:157
275991260,3ptp4l21-21ksoftirqd/108:00:151
20092240,0chrt2008-21perl11:15:153
518299224,11cyclictest0-21swapper/308:45:255
148012210,0sleep20-21swapper/212:15:014
148012210,0sleep20-21swapper/212:15:014
521099186,9cyclictest25147-21kworker/u24:110:25:0011
520099184,8cyclictest0-21swapper/710:50:009
518299183,9cyclictest0-21swapper/309:45:005
517199174,8cyclictest0-21swapper/107:15:011
5171991713,3cyclictest131rcu_preempt09:40:161
521999164,7cyclictest0-21swapper/1111:35:003
521999162,8cyclictest0-21swapper/1111:50:003
518299163,8cyclictest0-21swapper/311:30:015
518299163,8cyclictest0-21swapper/309:39:595
517199160,10cyclictest0-21swapper/109:20:001
47872167,7sleep80-21swapper/807:09:5810
275991160,1ptp4l27521-21latency_hist11:50:001
275991160,1ptp4l21-21ksoftirqd/112:05:231
275991160,1ptp4l21-21ksoftirqd/112:05:231
275991160,0ptp4l0-21swapper/112:25:251
275991160,0ptp4l0-21swapper/109:35:261
275991160,0ptp4l0-21swapper/109:25:171
521999153,7cyclictest0-21swapper/1107:45:253
521999151,2cyclictest0-21swapper/1110:35:193
5219991511,2cyclictest0-21swapper/1111:10:173
520099153,7cyclictest0-21swapper/707:15:019
518299153,7cyclictest0-21swapper/310:05:265
518299153,7cyclictest0-21swapper/307:50:015
518299153,6cyclictest0-21swapper/310:34:285
275991150,1ptp4l21-21ksoftirqd/111:00:141
275991150,1ptp4l21-21ksoftirqd/109:00:181
275991150,1ptp4l21-21ksoftirqd/108:45:161
275991150,1ptp4l21-21ksoftirqd/107:40:201
275991150,0ptp4l0-21swapper/111:10:201
275991150,0ptp4l0-21swapper/109:20:171
521999143,6cyclictest0-21swapper/1107:45:013
521999142,8cyclictest0-21swapper/1111:25:003
521999142,7cyclictest0-21swapper/1112:40:003
521999142,7cyclictest0-21swapper/1108:45:003
521099149,3cyclictest0-21swapper/909:35:2211
521099144,6cyclictest0-21swapper/907:40:0111
520099143,6cyclictest0-21swapper/711:30:009
519199142,8cyclictest0-21swapper/509:20:007
519199142,8cyclictest0-21swapper/507:35:017
518299143,7cyclictest0-21swapper/311:35:015
518299143,6cyclictest0-21swapper/309:35:005
518299142,7cyclictest0-21swapper/311:10:005
518299141,8cyclictest0-21swapper/310:00:005
275991140,2ptp4l19931-21date09:30:311
275991140,0ptp4l0-21swapper/112:30:261
275991140,0ptp4l0-21swapper/112:00:171
275991140,0ptp4l0-21swapper/112:00:171
275991140,0ptp4l0-21swapper/107:35:191
520099132,6cyclictest0-21swapper/708:40:009
519199132,7cyclictest0-21swapper/509:00:007
519199131,8cyclictest0-21swapper/511:55:007
519199131,8cyclictest0-21swapper/511:55:007
518299133,7cyclictest0-21swapper/311:30:005
518299132,7cyclictest0-21swapper/310:00:005
518299132,6cyclictest0-21swapper/309:45:275
517199132,7cyclictest0-21swapper/109:05:001
517199132,6cyclictest0-21swapper/108:20:001
517199132,6cyclictest0-21swapper/107:35:011
275991130,1ptp4l21-21ksoftirqd/110:33:371
275991130,0ptp4l0-21swapper/111:30:071
275991130,0ptp4l0-21swapper/111:10:001
275991130,0ptp4l0-21swapper/110:30:001
275991130,0ptp4l0-21swapper/108:05:201
275991130,0ptp4l0-21swapper/107:25:251
264662130,0sleep20-21swapper/211:05:174
244982130,1sleep7561rcuc/708:15:159
521999122,7cyclictest0-21swapper/1111:05:003
521099122,6cyclictest0-21swapper/909:15:0011
520099123,7cyclictest0-21swapper/708:25:009
520099123,6cyclictest0-21swapper/711:40:269
519199121,7cyclictest0-21swapper/508:15:267
517199122,6cyclictest0-21swapper/108:30:001
5166991210,1cyclictest0-21swapper/009:20:000
46412126,5sleep90-21swapper/907:07:5811
275991120,3ptp4l6639-21irqrtprio07:50:171
275991120,1ptp4l21-21ksoftirqd/111:35:151
275991120,1ptp4l21-21ksoftirqd/110:40:291
275991120,0ptp4l0-21swapper/110:20:201
275991120,0ptp4l0-21swapper/109:49:321
275991120,0ptp4l0-21swapper/108:15:191
275991120,0ptp4l0-21swapper/107:11:031
521999111,6cyclictest0-21swapper/1111:50:303
521999111,6cyclictest0-21swapper/1111:50:303
521599110,10cyclictest13471-21irqrtprio10:05:202
520099119,1cyclictest0-21swapper/709:55:009
520099112,6cyclictest0-21swapper/710:25:009
519199111,6cyclictest0-21swapper/509:05:007
519199110,10cyclictest7644-21tr12:05:187
519199110,10cyclictest7644-21tr12:05:187
518799110,10cyclictest10648-21cpuspeed_turbos10:45:146
517799117,3cyclictest25691-21cut11:45:194
516699110,2cyclictest0-21swapper/009:55:150
516699110,11cyclictest0-21swapper/010:50:000
275991110,2ptp4l32550-21expr11:55:141
275991110,2ptp4l32550-21expr11:55:141
275991110,2ptp4l23060-21sed10:20:001
275991110,2ptp4l21292-21grep11:40:171
275991110,2ptp4l13450-21rt-features08:40:321
275991110,1ptp4l21-21ksoftirqd/111:15:131
275991110,1ptp4l21-21ksoftirqd/107:45:251
275991110,1ptp4l0-21swapper/111:20:151
275991110,0ptp4l0-21swapper/109:10:201
275991110,0ptp4l0-21swapper/108:55:151
150512110,0sleep70-21swapper/708:00:259
521999103,6cyclictest0-21swapper/1112:20:003
521999103,6cyclictest0-21swapper/1112:20:003
521599100,2cyclictest5787-21/usr/sbin/munin09:15:152
521599100,1cyclictest0-21swapper/1009:10:192
521099101,6cyclictest25147-21kworker/u24:111:05:0011
521099101,6cyclictest0-21swapper/908:45:0011
521099101,5cyclictest0-21swapper/908:45:0011
521099100,6cyclictest25147-21kworker/u24:112:10:0111
521099100,6cyclictest25147-21kworker/u24:112:10:0111
521099100,2cyclictest10037-21tr09:20:1411
519199101,5cyclictest0-21swapper/511:40:017
519199100,5cyclictest0-21swapper/510:00:007
517799100,9cyclictest7884-21switchtime07:10:314
516699100,1cyclictest0-21swapper/008:45:000
275991100,2ptp4l4856-21/usr/sbin/munin08:30:191
275991100,2ptp4l20048-21/usr/sbin/munin08:50:271
275991100,1ptp4l21-21ksoftirqd/112:14:451
275991100,1ptp4l21-21ksoftirqd/112:14:451
275991100,1ptp4l20748-21sed10:55:241
275991100,1ptp4l17683-21sed10:55:001
275991100,1ptp4l1206-21cat12:40:001
275991100,1ptp4l11641-21latency_hist10:05:001
275991100,1ptp4l10159-21awk11:25:191
275991100,0ptp4l0-21swapper/112:20:201
275991100,0ptp4l0-21swapper/110:10:141
275991100,0ptp4l0-21swapper/109:56:231
275991100,0ptp4l0-21swapper/109:52:011
275991100,0ptp4l0-21swapper/108:14:261
275991100,0ptp4l0-21swapper/107:55:191
9498090,2rtkit-daemon0-21swapper/708:07:219
52199996,2cyclictest0-21swapper/1110:00:173
52199992,6cyclictest0-21swapper/1110:55:003
52199992,6cyclictest0-21swapper/1108:50:343
52159998,1cyclictest0-21swapper/1012:10:172
52159998,1cyclictest0-21swapper/1012:10:172
52159998,1cyclictest0-21swapper/1008:55:152
52159998,1cyclictest0-21swapper/1008:45:142
52109991,5cyclictest0-21swapper/908:10:0011
52109990,9cyclictest0-21swapper/909:25:1711
52109990,1cyclictest0-21swapper/910:10:1411
52069997,1cyclictest0-21swapper/809:30:0010
52009990,8cyclictest0-21swapper/711:05:009
52009990,8cyclictest0-21swapper/710:14:539
51969990,8cyclictest28567-21open_inodes09:00:218
51919997,1cyclictest0-21swapper/511:20:187
51919996,2cyclictest1203-21diskmemload09:25:167
51879997,1cyclictest30545-21tune2fs07:40:136
51879990,1cyclictest0-21swapper/411:05:006
51879990,1cyclictest0-21swapper/409:50:326
51879990,1cyclictest0-21swapper/409:05:266
51829997,1cyclictest0-21swapper/307:25:155
51829995,3cyclictest7867-21needreboot09:15:225
51829993,4cyclictest0-21swapper/311:10:145
51779998,1cyclictest0-21swapper/207:35:154
51779990,9cyclictest0-21swapper/212:30:174
51779990,9cyclictest0-21swapper/212:05:304
51779990,9cyclictest0-21swapper/212:05:304
51779990,9cyclictest0-21swapper/210:40:264
51779990,8cyclictest20492-21cut08:50:294
51779990,8cyclictest11518-21/usr/sbin/munin10:45:174
51779990,1cyclictest0-21swapper/207:50:124
51669998,1cyclictest0-21swapper/012:35:170
51669990,9cyclictest0-21swapper/010:10:240
51669990,9cyclictest0-21swapper/008:10:000
51669990,1cyclictest0-21swapper/009:20:170
51669990,1cyclictest0-21swapper/008:55:270
27599190,1ptp4l28247-21cpuspeed_turbos11:50:191
27599190,1ptp4l28247-21cpuspeed_turbos11:50:191
27599190,1ptp4l21-21ksoftirqd/110:35:001
27599190,0ptp4l0-21swapper/112:15:161
27599190,0ptp4l0-21swapper/112:15:161
27599190,0ptp4l0-21swapper/110:45:141
27599190,0ptp4l0-21swapper/110:05:141
27599190,0ptp4l0-21swapper/108:35:181
27599190,0ptp4l0-21swapper/107:20:121
9498080,0rtkit-daemon0-21swapper/509:39:357
52199987,1cyclictest0-21swapper/1110:10:003
52199986,1cyclictest0-21swapper/1108:50:003
52199980,8cyclictest0-21swapper/1108:35:213
52199980,7cyclictest0-21swapper/1108:04:113
52199980,1cyclictest0-21swapper/1111:05:233
52199980,1cyclictest0-21swapper/1109:20:253
52199980,1cyclictest0-21swapper/1107:35:133
52199980,1cyclictest0-21swapper/1107:10:163
52159987,1cyclictest0-21swapper/1011:10:242
52159987,1cyclictest0-21swapper/1010:19:582
52159987,1cyclictest0-21swapper/1007:35:012
52159986,1cyclictest0-21swapper/1007:40:122
52159980,8cyclictest0-21swapper/1011:55:222
52159980,8cyclictest0-21swapper/1011:55:222
52159980,8cyclictest0-21swapper/1008:00:222
52159980,8cyclictest0-21swapper/1007:25:202
52159980,7cyclictest0-21swapper/1010:56:402
52159980,7cyclictest0-21swapper/1010:39:592
52109981,5cyclictest0-21swapper/910:00:2611
52109981,5cyclictest0-21swapper/909:55:2411
52109981,4cyclictest0-21swapper/911:50:0011
52109980,8cyclictest0-21swapper/912:30:2611
52109980,1cyclictest0-21swapper/910:15:0211
52109980,1cyclictest0-21swapper/908:30:1111
52069987,1cyclictest0-21swapper/812:00:2310
52069987,1cyclictest0-21swapper/812:00:2310
52069987,1cyclictest0-21swapper/811:15:1510
52069987,1cyclictest0-21swapper/810:30:2510
52069987,1cyclictest0-21swapper/810:05:1410
52069987,1cyclictest0-21swapper/809:40:1910
52069987,1cyclictest0-21swapper/809:30:2510
52069987,1cyclictest0-21swapper/808:35:2310
52069987,1cyclictest0-21swapper/808:25:2710
52069987,1cyclictest0-21swapper/807:45:2510
52069980,8cyclictest0-21swapper/810:40:1710
52069980,8cyclictest0-21swapper/810:00:1810
52069980,8cyclictest0-21swapper/808:40:3210
52069980,8cyclictest0-21swapper/808:30:1410
52069980,8cyclictest0-21swapper/807:10:2710
52069980,7cyclictest0-21swapper/812:36:0510
52069980,7cyclictest0-21swapper/807:44:2010
52069980,7cyclictest0-21swapper/807:27:4110
52069980,1cyclictest0-21swapper/812:30:1810
52069980,1cyclictest0-21swapper/811:45:2310
52069980,1cyclictest0-21swapper/809:55:1310
52069980,1cyclictest0-21swapper/809:51:3010
52069980,1cyclictest0-21swapper/808:55:0010
52069980,1cyclictest0-21swapper/807:20:1110
52009980,8cyclictest0-21swapper/711:35:169
52009980,7cyclictest1203-21diskmemload09:41:279
52009980,1cyclictest0-21swapper/712:25:149
52009980,1cyclictest0-21swapper/712:05:309
52009980,1cyclictest0-21swapper/712:05:309
52009980,1cyclictest0-21swapper/711:15:229
52009980,1cyclictest0-21swapper/707:34:089
51969987,1cyclictest0-21swapper/611:00:268
51969980,8cyclictest0-21swapper/611:15:188
51969980,1cyclictest0-21swapper/611:05:218
51969980,1cyclictest0-21swapper/609:50:368
51969980,1cyclictest0-21swapper/608:45:148
51969980,1cyclictest0-21swapper/607:40:228
51969980,1cyclictest0-21swapper/607:10:318
51919980,8cyclictest0-21swapper/507:25:187
51919980,1cyclictest0-21swapper/510:40:167
51919980,1cyclictest0-21swapper/510:35:217
51919980,1cyclictest0-21swapper/508:30:267
51919980,1cyclictest0-21swapper/508:00:177
51879987,1cyclictest0-21swapper/412:00:236
51879987,1cyclictest0-21swapper/412:00:236
51879987,1cyclictest0-21swapper/411:40:006
51879987,1cyclictest0-21swapper/410:35:176
51879987,1cyclictest0-21swapper/409:20:166
51879986,1cyclictest15347-21fw_forwarded_lo10:50:166
51879980,8cyclictest0-21swapper/410:25:196
51879980,8cyclictest0-21swapper/408:50:346
51879980,8cyclictest0-21swapper/408:10:196
51879980,7cyclictest21106-21aten2_r3power_c09:35:166
51879980,2cyclictest12839-21cat07:15:286
51879980,1cyclictest0-21swapper/409:10:146
51829987,1cyclictest0-21swapper/312:30:185
51829985,2cyclictest19035-21turbostat10:19:555
51829985,2cyclictest0-21swapper/308:20:125
51829981,6cyclictest0-21swapper/310:25:005
51829980,8cyclictest0-21swapper/312:35:205
51829980,1cyclictest0-21swapper/312:10:255
51829980,1cyclictest0-21swapper/312:10:255
51829980,1cyclictest0-21swapper/309:25:005
51829980,1cyclictest0-21swapper/308:40:355
51779987,1cyclictest0-21swapper/211:30:114
51779987,1cyclictest0-21swapper/211:25:164
51779987,1cyclictest0-21swapper/211:20:214
51779987,1cyclictest0-21swapper/211:02:364
51779987,1cyclictest0-21swapper/209:05:144
51779987,1cyclictest0-21swapper/208:40:354
51779980,8cyclictest0-21swapper/212:35:224
51779980,8cyclictest0-21swapper/211:40:184
51779980,8cyclictest0-21swapper/211:35:144
51779980,8cyclictest0-21swapper/210:50:264
51779980,8cyclictest0-21swapper/210:30:194
51779980,8cyclictest0-21swapper/209:25:264
51779980,8cyclictest0-21swapper/207:45:154
51779980,7cyclictest4191-21/usr/sbin/munin09:10:224
51779980,2cyclictest19675-21grep12:20:164
51779980,1cyclictest29051-21perl11:50:214
51779980,1cyclictest29051-21perl11:50:214
51779980,1cyclictest0-21swapper/210:04:364
51779980,1cyclictest0-21swapper/207:55:014
51779980,1cyclictest0-21swapper/207:44:274
51669987,1cyclictest0-21swapper/012:20:210
51669987,1cyclictest0-21swapper/011:25:210
51669987,1cyclictest0-21swapper/011:04:590
51669987,1cyclictest0-21swapper/010:00:260
51669987,1cyclictest0-21swapper/009:30:310
51669987,1cyclictest0-21swapper/009:25:110
51669987,1cyclictest0-21swapper/007:55:270
51669987,1cyclictest0-21swapper/007:25:160
51669986,1cyclictest0-21swapper/012:10:190
51669986,1cyclictest0-21swapper/012:10:190
51669985,2cyclictest0-21swapper/012:00:180
51669985,2cyclictest0-21swapper/012:00:180
51669980,8cyclictest0-21swapper/010:05:210
51669980,8cyclictest0-21swapper/007:20:200
51669980,1cyclictest0-21swapper/008:30:230
2462281,5sleep30-21swapper/307:05:165
52199976,1cyclictest0-21swapper/1112:30:293
52199976,1cyclictest0-21swapper/1112:25:133
52199976,1cyclictest0-21swapper/1111:40:493
52199976,1cyclictest0-21swapper/1110:10:373
52199976,1cyclictest0-21swapper/1109:30:313
52199976,1cyclictest0-21swapper/1109:15:223
52199976,1cyclictest0-21swapper/1109:05:153
52199976,1cyclictest0-21swapper/1109:00:183
52199976,1cyclictest0-21swapper/1108:25:193
52199976,1cyclictest0-21swapper/1108:07:033
52199976,1cyclictest0-21swapper/1107:50:023
52199976,1cyclictest0-21swapper/1107:23:073
52199976,1cyclictest0-21swapper/1107:15:103
52199974,2cyclictest31182-21cpuspeed10:30:133
52199974,2cyclictest29098-21load10:25:193
52199974,2cyclictest24158-21/usr/sbin/munin08:15:163
52199974,2cyclictest19572-21aten2_r3power_c10:15:133
52199974,2cyclictest1020-21runrttasks12:20:443
52199973,3cyclictest22063-21cut08:10:203
52199973,3cyclictest21141-21wget09:35:163
52199970,7cyclictest0-21swapper/1109:50:543
52199970,7cyclictest0-21swapper/1109:45:273
52199970,7cyclictest0-21swapper/1109:40:303
52199970,7cyclictest0-21swapper/1109:25:133
52199970,7cyclictest0-21swapper/1109:10:263
52199970,7cyclictest0-21swapper/1108:59:413
52199970,7cyclictest0-21swapper/1107:31:383
52199970,1cyclictest0-21swapper/1112:10:033
52199970,1cyclictest0-21swapper/1112:10:033
52199970,1cyclictest0-21swapper/1112:05:543
52199970,1cyclictest0-21swapper/1112:05:543
52199970,1cyclictest0-21swapper/1112:02:503
52199970,1cyclictest0-21swapper/1112:02:503
52199970,1cyclictest0-21swapper/1111:59:253
52199970,1cyclictest0-21swapper/1111:59:253
52199970,1cyclictest0-21swapper/1111:35:253
52199970,1cyclictest0-21swapper/1111:25:073
52199970,1cyclictest0-21swapper/1110:52:343
52199970,1cyclictest0-21swapper/1110:45:243
52199970,1cyclictest0-21swapper/1110:42:503
52199970,1cyclictest0-21swapper/1110:20:353
52199970,1cyclictest0-21swapper/1109:56:583
52199970,1cyclictest0-21swapper/1108:30:113
52199970,1cyclictest0-21swapper/1108:20:033
52199970,1cyclictest0-21swapper/1107:55:113
52199970,1cyclictest0-21swapper/1107:29:283
52159976,1cyclictest0-21swapper/1012:25:182
52159976,1cyclictest0-21swapper/1012:20:102
52159976,1cyclictest0-21swapper/1012:05:092
52159976,1cyclictest0-21swapper/1012:05:092
52159976,1cyclictest0-21swapper/1012:00:522
52159976,1cyclictest0-21swapper/1012:00:522
52159976,1cyclictest0-21swapper/1011:45:042
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional