You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 17:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Mon Nov 24, 2025 12:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911060,96ptp4l0-21swapper/107:08:031
862526755,10sleep50-21swapper/507:07:587
874326554,9sleep00-21swapper/007:09:410
866826553,10sleep70-21swapper/707:08:369
859226352,9sleep110-21swapper/1107:07:283
724826352,8sleep40-21swapper/407:05:206
860026250,9sleep60-21swapper/607:07:358
852426251,9sleep100-21swapper/1007:06:272
283652570,0sleep30-21swapper/311:45:145
913999242,17cyclictest0-21swapper/409:45:016
228642200,1sleep9681rcuc/909:30:1711
913999191,17cyclictest0-21swapper/410:30:186
912099191,16cyclictest0-21swapper/011:35:160
315902190,0sleep110-21swapper/1110:25:133
913599183,9cyclictest0-21swapper/307:25:015
913099183,9cyclictest0-21swapper/209:30:014
913099183,10cyclictest0-21swapper/210:25:014
913099183,10cyclictest0-21swapper/209:10:244
913099182,10cyclictest0-21swapper/207:30:004
914499170,16cyclictest0-21swapper/511:45:187
913599172,9cyclictest0-21swapper/312:20:005
913599171,10cyclictest0-21swapper/309:45:005
913099173,9cyclictest0-21swapper/212:30:004
913099173,10cyclictest0-21swapper/212:15:004
913099172,9cyclictest0-21swapper/212:10:004
912099172,9cyclictest0-21swapper/009:15:010
917399163,8cyclictest0-21swapper/1111:50:003
915799163,9cyclictest0-21swapper/809:40:0210
914899162,8cyclictest0-21swapper/610:55:018
914899162,8cyclictest0-21swapper/609:30:008
914499164,8cyclictest0-21swapper/509:15:017
913999160,2cyclictest0-21swapper/411:50:186
913599164,7cyclictest0-21swapper/307:40:005
913599161,9cyclictest0-21swapper/312:25:005
913099163,8cyclictest0-21swapper/207:50:004
913099163,8cyclictest0-21swapper/207:45:004
913099162,9cyclictest0-21swapper/207:25:004
913099162,2cyclictest0-21swapper/207:10:174
912699163,8cyclictest0-21swapper/112:05:011
912099163,8cyclictest0-21swapper/010:15:010
912099163,8cyclictest0-21swapper/010:05:010
912099162,9cyclictest0-21swapper/009:15:020
912099162,8cyclictest0-21swapper/009:05:010
912099162,10cyclictest0-21swapper/010:05:250
85222167,7sleep80-21swapper/807:06:2510
275991160,1ptp4l21-21ksoftirqd/109:15:131
275991160,1ptp4l21-21ksoftirqd/107:15:171
275991160,0ptp4l0-21swapper/111:30:131
275991160,0ptp4l0-21swapper/108:20:141
275991160,0ptp4l0-21swapper/107:55:131
275991160,0ptp4l0-21swapper/107:55:131
915799152,8cyclictest0-21swapper/811:25:2510
914899152,9cyclictest0-21swapper/609:55:018
914899152,9cyclictest0-21swapper/608:35:008
914899151,10cyclictest13120-21munin-node07:15:018
914499153,7cyclictest0-21swapper/510:25:017
913599153,7cyclictest0-21swapper/309:35:015
912699153,7cyclictest0-21swapper/107:20:011
912699150,3cyclictest0-21swapper/109:00:231
912099153,7cyclictest0-21swapper/012:20:000
912099153,7cyclictest0-21swapper/011:04:210
912099152,8cyclictest0-21swapper/012:15:000
912099152,8cyclictest0-21swapper/011:35:000
912099152,8cyclictest0-21swapper/007:45:000
912099151,9cyclictest0-21swapper/011:45:000
275991150,1ptp4l21-21ksoftirqd/111:10:151
275991150,1ptp4l21-21ksoftirqd/110:50:151
275991150,1ptp4l21-21ksoftirqd/108:05:131
275991150,1ptp4l21-21ksoftirqd/108:05:131
275991150,0ptp4l0-21swapper/110:00:141
275991150,0ptp4l0-21swapper/109:55:141
275991150,0ptp4l0-21swapper/109:40:131
917399143,7cyclictest0-21swapper/1107:30:003
917399142,7cyclictest0-21swapper/1108:55:003
916399142,7cyclictest0-21swapper/911:30:0011
914499142,8cyclictest0-21swapper/507:55:247
914499142,8cyclictest0-21swapper/507:55:247
914499141,10cyclictest30237-21kworker/u24:212:10:007
913999142,7cyclictest0-21swapper/410:00:016
913599143,7cyclictest0-21swapper/309:05:265
913099142,7cyclictest0-21swapper/208:40:014
912699143,7cyclictest0-21swapper/110:15:011
912699143,6cyclictest0-21swapper/108:15:011
912699142,8cyclictest0-21swapper/112:25:241
912699142,7cyclictest0-21swapper/111:00:251
912099143,7cyclictest0-21swapper/012:05:000
305772140,0sleep80-21swapper/812:30:1210
304972140,0sleep100-21swapper/1009:40:192
275991140,6ptp4l21-21ksoftirqd/112:05:161
275991140,0ptp4l0-21swapper/108:55:191
251142140,0sleep80-21swapper/812:20:2310
917399134,6cyclictest30237-21kworker/u24:211:05:003
917399132,7cyclictest0-21swapper/1108:15:013
917399132,7cyclictest0-21swapper/1108:15:003
917399132,7cyclictest0-21swapper/1107:35:243
917399130,7cyclictest0-21swapper/1112:04:553
914899133,7cyclictest0-21swapper/610:35:018
913999130,8cyclictest0-21swapper/410:55:016
913599132,7cyclictest0-21swapper/312:30:235
913599131,8cyclictest0-21swapper/310:30:015
913599130,12cyclictest0-21swapper/308:25:005
913099130,2cyclictest0-21swapper/207:30:184
912699133,6cyclictest0-21swapper/108:03:171
912699133,6cyclictest0-21swapper/108:03:171
912699132,7cyclictest0-21swapper/111:30:001
912099139,3cyclictest31264-21irqrtprio11:05:190
912099133,6cyclictest0-21swapper/009:50:010
78882131,8sleep90-21swapper/907:05:2611
275991130,1ptp4l21-21ksoftirqd/112:18:411
275991130,1ptp4l21-21ksoftirqd/112:13:161
275991130,1ptp4l21-21ksoftirqd/111:46:371
275991130,1ptp4l21-21ksoftirqd/110:43:061
275991130,1ptp4l21-21ksoftirqd/110:31:051
275991130,1ptp4l21-21ksoftirqd/108:35:181
275991130,0ptp4l0-21swapper/109:20:221
275991130,0ptp4l0-21swapper/109:10:131
205682130,0sleep30-21swapper/308:05:145
205682130,0sleep30-21swapper/308:05:135
917399122,6cyclictest0-21swapper/1112:40:003
917399122,6cyclictest0-21swapper/1107:35:003
916999123,5cyclictest0-21swapper/1011:00:022
915799121,6cyclictest0-21swapper/809:15:0110
914899120,2cyclictest774-21missed_timers10:25:198
914499123,6cyclictest0-21swapper/511:25:007
914499122,6cyclictest0-21swapper/510:10:017
913999121,6cyclictest0-21swapper/411:45:256
913999121,6cyclictest0-21swapper/408:35:016
913999121,6cyclictest0-21swapper/407:15:246
913999120,7cyclictest0-21swapper/411:20:006
913599120,11cyclictest10772-21latency_hist08:35:015
912699122,7cyclictest0-21swapper/110:55:011
912099120,3cyclictest15676-21cstates08:40:160
275991120,1ptp4l21-21ksoftirqd/111:40:161
275991120,1ptp4l21-21ksoftirqd/109:45:261
275991120,1ptp4l21-21ksoftirqd/109:25:141
275991120,1ptp4l21-21ksoftirqd/108:32:261
275991120,0ptp4l0-21swapper/111:56:261
275991120,0ptp4l0-21swapper/110:45:131
275991120,0ptp4l0-21swapper/108:11:261
275991120,0ptp4l0-21swapper/108:11:251
917399112,5cyclictest0-21swapper/1110:05:013
916399112,6cyclictest0-21swapper/909:15:2411
916399111,6cyclictest30237-21kworker/u24:212:10:0011
916399110,10cyclictest1702-21date09:05:0111
915399110,2cyclictest8663-21tr09:55:139
914899112,6cyclictest0-21swapper/607:45:018
914899110,2cyclictest18705-21tr07:20:118
914899110,10cyclictest15283-21cut09:20:198
914899110,10cyclictest14755-21memory07:55:198
914899110,10cyclictest14755-21memory07:55:198
914499111,6cyclictest0-21swapper/507:55:017
914499111,6cyclictest0-21swapper/507:55:017
913099119,1cyclictest5413-21diskmemload09:41:534
912699117,2cyclictest0-21swapper/112:34:201
912099112,8cyclictest9255-21latency08:30:250
912099112,8cyclictest19683-21sed07:20:160
912099110,9cyclictest10974-21missed_timers11:20:200
912099110,1cyclictest0-21swapper/012:22:510
912099110,1cyclictest0-21swapper/008:15:150
912099110,1cyclictest0-21swapper/007:55:120
912099110,1cyclictest0-21swapper/007:55:120
912099110,1cyclictest0-21swapper/007:50:250
912099110,1cyclictest0-21swapper/007:50:250
912099110,10cyclictest30333-21chrt10:20:250
912099110,10cyclictest0-21swapper/010:25:220
275991110,3ptp4l21-21ksoftirqd/107:40:211
275991110,2ptp4l4641-21awk09:50:131
275991110,2ptp4l31406-21irqstats07:35:181
275991110,1ptp4l21-21ksoftirqd/111:05:261
275991110,1ptp4l21-21ksoftirqd/110:36:221
275991110,1ptp4l21-21ksoftirqd/110:15:201
275991110,1ptp4l21-21ksoftirqd/107:12:551
275991110,0ptp4l0-21swapper/111:35:241
275991110,0ptp4l0-21swapper/108:45:131
275991110,0ptp4l0-21swapper/107:34:331
917399107,2cyclictest18440-21turbostat12:15:003
916999109,1cyclictest741rcuc/1008:35:162
916999108,1cyclictest0-21swapper/1008:10:182
916999108,1cyclictest0-21swapper/1008:10:182
916999101,6cyclictest0-21swapper/1008:45:002
916999100,9cyclictest0-21swapper/1011:50:232
916399102,5cyclictest0-21swapper/911:10:0011
915799109,1cyclictest0-21swapper/809:50:0010
915799108,1cyclictest0-21swapper/811:45:1610
915799102,5cyclictest0-21swapper/811:35:0010
915399107,2cyclictest0-21swapper/711:50:179
914499102,5cyclictest0-21swapper/510:05:017
914499100,9cyclictest19330-21fw_conntrack10:50:147
914499100,7cyclictest0-21swapper/509:50:017
913999100,3cyclictest8166-21irqrtprio10:35:186
913599108,1cyclictest0-21swapper/311:00:135
913599107,2cyclictest0-21swapper/311:35:195
913099102,5cyclictest0-21swapper/210:35:164
913099100,2cyclictest0-21swapper/207:15:014
912699102,5cyclictest0-21swapper/109:40:021
912099104,5cyclictest25019-21irqrtprio10:15:180
912099101,8cyclictest6814-21cut07:45:210
912099101,8cyclictest5646-21cat08:25:200
912099101,8cyclictest26436-21expr07:30:150
912099100,9cyclictest9594-21if_enp4s009:55:170
912099100,9cyclictest8149-21interrupts10:35:180
912099100,9cyclictest7903-21cpuspeed_turbos12:00:110
912099100,9cyclictest717-21gltestperf11:50:170
912099100,9cyclictest6614-21chrt11:55:360
912099100,9cyclictest5160-21taskset11:15:070
912099100,9cyclictest5010-21expr09:50:130
912099100,9cyclictest32420-21tr08:20:140
912099100,9cyclictest32374-21latency_hist09:45:010
912099100,9cyclictest31532-21grep09:00:210
912099100,9cyclictest31095-21expr12:30:120
912099100,9cyclictest30404-21sed11:45:220
912099100,9cyclictest29113-21timerandwakeup08:55:240
912099100,9cyclictest2851-21cpuspeed10:30:140
912099100,9cyclictest2630-21irqrtprio11:10:200
912099100,9cyclictest24362-21sed08:50:150
912099100,9cyclictest23861-21latency07:25:190
912099100,9cyclictest2352-21cpuspeed_turbos12:35:140
912099100,9cyclictest21684-21expr09:30:120
912099100,9cyclictest20899-21cpuspeed_turbos08:05:140
912099100,9cyclictest20899-21cpuspeed_turbos08:05:140
912099100,9cyclictest19269-21cpuspeed_turbos08:45:140
912099100,9cyclictest19156-21expr08:00:240
912099100,9cyclictest19156-21expr08:00:230
912099100,9cyclictest17564-21cpu09:25:140
912099100,9cyclictest15840-21iwlist07:15:180
912099100,9cyclictest15215-21tune2fs10:45:150
912099100,9cyclictest13150-21expr11:25:130
912099100,9cyclictest12504-21gltestperf08:35:160
912099100,9cyclictest10962-21cpuspeed_turbos10:40:140
912099100,7cyclictest0-21swapper/010:56:320
912099100,5cyclictest27053-21cpuspeed_turbos12:25:130
912099100,4cyclictest26463-21if_err_enp1s0.109:35:190
912099100,4cyclictest25480-21grep08:10:150
912099100,4cyclictest25480-21grep08:10:150
912099100,2cyclictest9432-21sshd07:10:120
912099100,1cyclictest0-21swapper/010:50:060
912099100,1cyclictest0-21swapper/009:20:120
912099100,1cyclictest0-21swapper/007:35:450
275991100,5ptp4l21-21ksoftirqd/108:27:181
275991100,4ptp4l21-21ksoftirqd/108:51:211
275991100,3ptp4l4253-21phc2sys09:05:211
275991100,2ptp4l32753-21diskstats11:50:151
275991100,2ptp4l24018-21gltestperf12:20:151
275991100,1ptp4l31841-21cstates10:25:131
275991100,1ptp4l0-21swapper/111:15:161
275991100,1ptp4l0-21swapper/107:45:251
91739996,2cyclictest20740-21timerandwakeup10:50:233
91739990,8cyclictest0-21swapper/1109:50:153
91699998,1cyclictest741rcuc/1010:00:192
91699997,1cyclictest0-21swapper/1009:37:122
91699992,6cyclictest8550-21tr09:55:132
91699991,8cyclictest741rcuc/1010:40:182
91699990,1cyclictest0-21swapper/1011:20:232
91699990,1cyclictest0-21swapper/1008:05:202
91699990,1cyclictest0-21swapper/1008:05:202
91639998,1cyclictest0-21swapper/908:05:0011
91639998,1cyclictest0-21swapper/908:05:0011
91639996,2cyclictest13222-21fw_forwarded_lo10:00:1711
91639996,2cyclictest10257-21sshd07:50:1611
91639996,2cyclictest10257-21sshd07:50:1611
91639992,5cyclictest0-21swapper/907:30:0011
91639991,6cyclictest0-21swapper/910:20:0111
91639991,5cyclictest0-21swapper/907:20:0011
91639990,9cyclictest0-21swapper/910:25:0111
91639990,8cyclictest5413-21diskmemload10:55:2411
91579991,5cyclictest0-21swapper/807:25:0010
91579990,1cyclictest0-21swapper/811:30:1810
91539990,9cyclictest0-21swapper/708:45:189
91539990,1cyclictest0-21swapper/707:35:199
91539990,1cyclictest0-21swapper/707:20:179
91489992,5cyclictest0-21swapper/608:05:018
91489992,5cyclictest0-21swapper/608:05:018
91489992,4cyclictest30237-21kworker/u24:208:20:018
91489990,9cyclictest0-21swapper/612:35:168
91489990,9cyclictest0-21swapper/609:30:188
91449990,8cyclictest4173-21sshd12:35:207
91449990,2cyclictest20533-21diskstats10:10:167
91399990,8cyclictest19834-21sed09:25:256
91359990,1cyclictest0-21swapper/308:50:125
91309996,2cyclictest0-21swapper/208:15:224
91309990,8cyclictest0-21swapper/211:10:004
91309990,8cyclictest0-21swapper/209:45:204
27599190,1ptp4l22137-21cpu07:25:131
27599190,1ptp4l21-21ksoftirqd/111:20:131
27599190,1ptp4l21-21ksoftirqd/110:05:151
27599190,1ptp4l21-21ksoftirqd/109:30:131
27599190,1ptp4l21-21ksoftirqd/108:40:011
27599190,1ptp4l21-21ksoftirqd/107:50:141
27599190,1ptp4l21-21ksoftirqd/107:50:131
27599190,0ptp4l0-21swapper/112:35:141
27599190,0ptp4l0-21swapper/110:20:131
91739980,8cyclictest0-21swapper/1110:55:123
91739980,8cyclictest0-21swapper/1107:55:183
91739980,8cyclictest0-21swapper/1107:55:183
91699987,1cyclictest75-21ksoftirqd/1007:24:382
91699987,1cyclictest741rcuc/1011:30:232
91699987,1cyclictest741rcuc/1008:46:092
91699987,1cyclictest0-21swapper/1011:16:052
91699987,1cyclictest0-21swapper/1010:30:202
91699987,1cyclictest0-21swapper/1009:50:142
91699987,1cyclictest0-21swapper/1007:35:232
91699986,2cyclictest741rcuc/1009:23:062
91699986,2cyclictest0-21swapper/1011:39:282
91699985,2cyclictest8395-21tune2fs08:30:222
91699985,2cyclictest3314-21awk09:45:272
91699985,2cyclictest25333-21fschecks_time11:40:152
91699985,2cyclictest22478-21cut09:30:152
91699984,2cyclictest30302-21date09:00:012
91699981,1cyclictest741rcuc/1008:15:442
91699980,8cyclictest0-21swapper/1007:40:442
91699980,1cyclictest0-21swapper/1007:50:172
91699980,1cyclictest0-21swapper/1007:50:172
91699980,1cyclictest0-21swapper/1007:25:222
91639982,4cyclictest0-21swapper/908:50:0111
91639981,5cyclictest0-21swapper/908:40:0111
91639980,1cyclictest29765-21sshd08:58:2011
91639980,1cyclictest0-21swapper/912:20:1511
91639980,1cyclictest0-21swapper/912:10:0211
91639980,1cyclictest0-21swapper/911:55:2311
91639980,1cyclictest0-21swapper/910:30:2311
91639980,1cyclictest0-21swapper/910:10:1811
91639980,1cyclictest0-21swapper/908:30:3211
91639980,1cyclictest0-21swapper/907:10:1311
91579987,1cyclictest0-21swapper/811:40:1310
91579987,1cyclictest0-21swapper/807:10:2210
91579985,2cyclictest1009-21snmpd09:31:5010
91579980,1cyclictest0-21swapper/811:05:2110
91539987,1cyclictest0-21swapper/712:05:139
91539987,1cyclictest0-21swapper/711:45:209
91539987,1cyclictest0-21swapper/711:00:189
91539987,1cyclictest0-21swapper/708:05:219
91539987,1cyclictest0-21swapper/708:05:219
91539985,2cyclictest1020-21runrttasks11:55:439
91539980,8cyclictest0-21swapper/710:00:499
91539980,8cyclictest0-21swapper/708:35:009
91539980,8cyclictest0-21swapper/708:00:159
91539980,8cyclictest0-21swapper/708:00:159
91539980,1cyclictest0-21swapper/709:40:419
91539980,1cyclictest0-21swapper/707:59:119
91539980,1cyclictest0-21swapper/707:59:119
91539980,1cyclictest0-21swapper/707:40:179
91539980,0cyclictest0-21swapper/707:30:139
91489987,1cyclictest0-21swapper/612:05:168
91489987,1cyclictest0-21swapper/611:10:238
91489987,1cyclictest0-21swapper/610:15:258
91489987,1cyclictest0-21swapper/607:10:218
91489986,1cyclictest0-21swapper/610:05:168
91489980,1cyclictest0-21swapper/611:05:188
91489980,1cyclictest0-21swapper/608:26:208
91449986,1cyclictest0-21swapper/511:35:137
91449985,2cyclictest11855-21expr08:35:137
91449981,5cyclictest0-21swapper/511:55:007
91449980,1cyclictest0-21swapper/512:14:257
91449980,1cyclictest0-21swapper/508:25:187
91399987,1cyclictest0-21swapper/411:05:266
91399987,1cyclictest0-21swapper/410:20:216
91399980,8cyclictest0-21swapper/407:55:156
91399980,8cyclictest0-21swapper/407:55:156
91399980,3cyclictest22248-21latency_hist08:50:016
91399980,2cyclictest1242-40dnf09:03:206
91399980,1cyclictest0-21swapper/410:05:146
91399980,1cyclictest0-21swapper/408:35:146
91399980,1cyclictest0-21swapper/408:00:146
91399980,1cyclictest0-21swapper/408:00:146
91399980,1cyclictest0-21swapper/407:20:166
91359987,1cyclictest0-21swapper/310:10:185
91359987,1cyclictest0-21swapper/310:00:175
91359986,1cyclictest0-21swapper/310:31:155
91359980,8cyclictest0-21swapper/310:40:145
91359980,8cyclictest0-21swapper/310:20:185
91359980,8cyclictest0-21swapper/309:55:205
91359980,7cyclictest0-21swapper/309:46:325
91359980,1cyclictest0-21swapper/311:15:195
91359980,1cyclictest0-21swapper/309:16:205
91359980,1cyclictest0-21swapper/308:15:195
91309986,1cyclictest5413-21diskmemload10:44:134
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional