You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-01 - 14:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Mon Jun 01, 2026 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2834911000,89ptp4l0-21swapper/107:05:201
2300027965,12sleep70-21swapper/707:09:339
2288927052,10sleep40-21swapper/407:07:536
2275026958,9sleep00-21swapper/007:05:490
2285526856,10sleep110-21swapper/1107:07:243
2272726852,8sleep50-21swapper/507:05:307
1975426655,9sleep100-21swapper/1007:05:112
2274126553,10sleep60-21swapper/607:05:438
91992500,0sleep10-21swapper/107:30:161
52982500,0sleep50-21swapper/509:30:207
191962490,0sleep81009-21runrttasks07:45:0010
2339399233,16cyclictest0-21swapper/411:35:016
2339399232,8cyclictest0-21swapper/408:55:006
2339399212,14cyclictest0-21swapper/407:20:006
2341699193,9cyclictest0-21swapper/912:05:0011
2342199180,1cyclictest0-21swapper/1009:12:192
283491170,0ptp4l0-21swapper/111:05:161
2342199170,3cyclictest0-21swapper/1007:55:152
2338799174,8cyclictest0-21swapper/312:20:015
2337899170,3cyclictest246311switchtime09:55:261
2337299172,8cyclictest0-21swapper/012:30:020
283491160,1ptp4l21-21ksoftirqd/108:25:131
283491160,1ptp4l21-21ksoftirqd/107:50:241
283491160,0ptp4l0-21swapper/110:25:191
283491160,0ptp4l0-21swapper/110:25:191
283491160,0ptp4l0-21swapper/107:35:181
269662160,0sleep00-21swapper/012:05:190
2342199160,1cyclictest0-21swapper/1009:15:152
2341699163,7cyclictest0-21swapper/911:05:0011
2340699160,15cyclictest0-21swapper/709:15:009
2339399162,13cyclictest0-21swapper/411:55:016
2338799163,8cyclictest0-21swapper/310:00:005
2338799163,8cyclictest0-21swapper/309:45:005
2338799163,7cyclictest0-21swapper/309:20:005
2337899160,2cyclictest3572-21sort08:05:201
2337299163,7cyclictest0-21swapper/010:45:010
2337299160,2cyclictest0-21swapper/011:40:150
229192167,7sleep80-21swapper/807:08:2110
283491150,1ptp4l21-21ksoftirqd/112:15:201
283491150,0ptp4l0-21swapper/110:45:201
283491150,0ptp4l0-21swapper/110:15:251
283491150,0ptp4l0-21swapper/110:15:241
2342199150,2cyclictest0-21swapper/1008:40:162
2341699153,7cyclictest0-21swapper/908:25:0011
2341699152,7cyclictest0-21swapper/908:55:2411
2341699150,2cyclictest0-21swapper/911:10:1611
2338799153,8cyclictest0-21swapper/308:30:005
2338799153,7cyclictest0-21swapper/308:45:005
2338799152,8cyclictest0-21swapper/311:00:245
2338799152,8cyclictest0-21swapper/309:00:005
2337899150,15cyclictest0-21swapper/111:50:111
2337299153,7cyclictest0-21swapper/007:20:000
56432140,0sleep20-21swapper/208:50:134
283491140,1ptp4l21-21ksoftirqd/110:30:211
283491140,1ptp4l21-21ksoftirqd/110:30:211
283491140,0ptp4l0-21swapper/110:40:001
2341699143,6cyclictest0-21swapper/911:25:2411
23406991412,1cyclictest0-21swapper/710:30:159
23406991412,1cyclictest0-21swapper/710:30:159
2339399142,6cyclictest0-21swapper/409:15:256
2337299142,8cyclictest0-21swapper/008:20:000
2337299142,7cyclictest0-21swapper/011:10:000
2337299142,7cyclictest0-21swapper/008:25:000
283491130,1ptp4l21-21ksoftirqd/109:45:111
283491130,0ptp4l0-21swapper/111:19:311
23416991311,1cyclictest0-21swapper/908:45:0011
2341699130,12cyclictest0-21swapper/908:00:0011
2340699130,3cyclictest18993-21latency_hist12:40:019
2340699130,13cyclictest0-21swapper/712:15:009
23397991311,1cyclictest0-21swapper/511:40:007
2339399133,6cyclictest0-21swapper/408:05:276
2338799132,7cyclictest0-21swapper/310:40:015
2338799132,6cyclictest0-21swapper/308:20:005
2338799132,6cyclictest0-21swapper/307:10:005
23378991310,3cyclictest0-21swapper/109:15:001
2337899130,9cyclictest16266-21expr12:35:121
2337899130,1cyclictest0-21swapper/110:55:561
2337899130,1cyclictest0-21swapper/107:20:211
2337299131,7cyclictest0-21swapper/012:25:000
283491120,3ptp4l2404-21expr11:35:101
283491120,2ptp4l27604-21perl11:25:151
283491120,0ptp4l0-21swapper/112:05:271
283491120,0ptp4l0-21swapper/111:55:081
283491120,0ptp4l0-21swapper/111:23:411
283491120,0ptp4l0-21swapper/108:21:581
283491120,0ptp4l0-21swapper/107:45:541
2342199120,11cyclictest0-21swapper/1012:04:552
2341699122,6cyclictest0-21swapper/910:15:0011
2341699122,6cyclictest0-21swapper/910:15:0011
2341699121,6cyclictest0-21swapper/908:30:0011
2340699120,1cyclictest0-21swapper/711:38:489
2340699120,1cyclictest0-21swapper/711:10:259
2340699120,12cyclictest0-21swapper/711:16:309
2340699120,12cyclictest0-21swapper/710:26:149
2340699120,12cyclictest0-21swapper/710:26:149
2340699120,11cyclictest19789-21kworker/u24:011:20:319
2340399122,6cyclictest0-21swapper/610:30:008
2340399122,6cyclictest0-21swapper/610:30:008
2339399122,6cyclictest0-21swapper/412:00:236
2338799123,6cyclictest0-21swapper/307:45:005
2337899120,1cyclictest0-21swapper/108:53:181
2337899120,12cyclictest0-21swapper/111:43:091
2337899120,12cyclictest0-21swapper/109:51:331
2337899120,11cyclictest0-21swapper/108:10:331
2337299122,6cyclictest0-21swapper/012:35:230
2337299122,6cyclictest0-21swapper/011:25:010
47792110,0sleep20-21swapper/209:30:174
283491110,3ptp4l2819-21irqrtprio10:10:141
283491110,3ptp4l2819-21irqrtprio10:10:141
283491110,2ptp4l31868-21fschecks_time08:00:131
283491110,2ptp4l30910-21irqrtprio08:40:171
283491110,2ptp4l2171-21irqrtprio08:45:191
283491110,2ptp4l13286-21ls11:50:011
283491110,1ptp4l0-21swapper/108:35:181
283491110,0ptp4l0-21swapper/112:30:121
283491110,0ptp4l0-21swapper/112:29:561
283491110,0ptp4l0-21swapper/110:43:491
283491110,0ptp4l0-21swapper/108:55:201
2342599110,6cyclictest0-21swapper/1110:29:553
2342599110,6cyclictest0-21swapper/1110:29:553
2342199110,1cyclictest0-21swapper/1010:11:252
2342199110,1cyclictest0-21swapper/1010:11:252
2341699113,4cyclictest0-21swapper/910:05:0011
2341699112,6cyclictest0-21swapper/912:10:2211
2341699110,7cyclictest0-21swapper/911:00:0011
2341699110,5cyclictest0-21swapper/908:55:0011
2340699119,2cyclictest0-21swapper/709:37:179
2340699119,2cyclictest0-21swapper/709:29:359
2340699118,3cyclictest0-21swapper/710:00:249
2340699113,2cyclictest3902-21cpuspeed_turbos10:55:139
2340699113,2cyclictest2546-21cpuspeed_turbos08:05:159
2340699110,9cyclictest0-21swapper/709:47:429
2340699110,4cyclictest6982-21fschecks_time11:40:139
2340699110,4cyclictest12845-21chrt11:48:249
2340699110,3cyclictest7760-21cpuspeed_turbos11:00:139
2340699110,3cyclictest26668-21chrt07:10:569
2340699110,3cyclictest24824-21irqrtprio07:50:189
2340699110,3cyclictest16998-21latency_hist11:55:009
2340699110,3cyclictest1544-21fschecks_count08:45:159
2340699110,3cyclictest13411-21latency_hist08:20:009
2340699110,1cyclictest0-21swapper/710:20:189
2340699110,1cyclictest0-21swapper/710:20:189
2340699110,1cyclictest0-21swapper/710:10:189
2340699110,1cyclictest0-21swapper/710:10:189
2340699110,11cyclictest0-21swapper/711:55:309
2340699110,10cyclictest0-21swapper/711:32:419
2340699110,10cyclictest0-21swapper/709:30:199
2340399112,6cyclictest0-21swapper/609:25:258
2340399110,3cyclictest32545-21cpu07:20:118
2339399118,2cyclictest0-21swapper/411:15:146
2339399110,10cyclictest0-21swapper/411:05:006
2339399110,10cyclictest0-21swapper/410:10:116
2339399110,10cyclictest0-21swapper/410:10:116
2338799112,6cyclictest0-21swapper/311:15:005
2337899119,2cyclictest0-21swapper/107:58:161
2337899113,7cyclictest7581-21latency_hist12:25:001
2337899112,8cyclictest31277-21timerandwakeup12:10:221
2337899110,4cyclictest15345-21sort11:10:121
2337899110,11cyclictest0-21swapper/109:40:101
2337899110,11cyclictest0-21swapper/109:20:541
2337299112,8cyclictest0-21swapper/012:30:220
283491100,2ptp4l25925-21ptp4l07:10:201
283491100,1ptp4l7709-21awk11:00:131
283491100,1ptp4l30648-21grep11:30:141
283491100,1ptp4l17279-21irqstats07:40:171
283491100,1ptp4l13085-21cstates09:00:141
283491100,1ptp4l0-21swapper/109:35:181
283491100,0ptp4l0-21swapper/110:50:131
283491100,0ptp4l0-21swapper/110:00:131
283491100,0ptp4l0-21swapper/108:30:131
283491100,0ptp4l0-21swapper/107:15:021
2342599100,9cyclictest6625-21taskset10:55:263
2342599100,4cyclictest1770-21awk08:05:003
2342199100,1cyclictest0-21swapper/1011:03:242
2341699102,5cyclictest0-21swapper/911:55:0111
2341699102,5cyclictest0-21swapper/911:25:0011
2341699102,5cyclictest0-21swapper/908:10:0111
2341699100,9cyclictest0-21swapper/909:35:1811
2341699100,1cyclictest24226-21strings09:55:2111
2340699109,1cyclictest0-21swapper/711:05:169
2340699109,1cyclictest0-21swapper/709:59:379
2340699109,1cyclictest0-21swapper/709:06:409
2340699109,1cyclictest0-21swapper/708:21:339
2340699108,2cyclictest0-21swapper/710:44:329
2340699101,2cyclictest12221-21cpuspeed_turbos07:35:129
2340699100,9cyclictest0-21swapper/708:57:409
2340699100,4cyclictest685-21systemd-journal09:25:009
2340699100,4cyclictest400-21sed08:00:189
2340699100,4cyclictest27845-21cat11:25:169
2340699100,4cyclictest22581-21cut12:00:179
2340699100,4cyclictest21971-21grep08:30:129
2340699100,4cyclictest19319-21irqrtprio08:25:189
2340699100,4cyclictest1543-21irqrtprio07:20:169
2340699100,3cyclictest9639-21sed12:25:189
2340699100,3cyclictest7379-21fschecks_time08:10:159
2340699100,3cyclictest6238-21missed_timers12:20:199
2340699100,3cyclictest29791-21/usr/sbin/munin10:05:199
2340699100,3cyclictest29481-21grep08:40:139
2340699100,3cyclictest18751-21chrt07:43:479
2340699100,3cyclictest13359-21irqrtprio12:30:169
2340699100,3cyclictest1007-21snmpd09:15:189
2340699100,2cyclictest1009-21runrttasks08:50:169
2340699100,2cyclictest1007-21snmpd12:08:249
2340699100,1cyclictest0-21swapper/710:48:339
2340699100,1cyclictest0-21swapper/710:19:219
2340699100,1cyclictest0-21swapper/710:19:209
2340699100,1cyclictest0-21swapper/709:00:449
2340699100,1cyclictest0-21swapper/707:59:129
2340699100,10cyclictest0-21swapper/710:51:469
2340399102,5cyclictest0-21swapper/610:20:238
2340399102,5cyclictest0-21swapper/610:20:238
2340399102,5cyclictest0-21swapper/608:45:258
2340399101,6cyclictest24141-21kworker/u24:008:35:008
2340399101,6cyclictest0-21swapper/609:50:008
2340399100,9cyclictest18195-21fschecks_count11:55:158
2340399100,9cyclictest0-21swapper/612:10:138
2339799100,9cyclictest0-21swapper/510:35:007
2339799100,9cyclictest0-21swapper/510:35:007
2339399108,1cyclictest0-21swapper/408:35:136
2338799107,2cyclictest0-21swapper/310:50:125
2337899102,7cyclictest31639-21cpuspeed09:25:131
2337899102,7cyclictest23767-21date09:15:001
2337899102,7cyclictest17613-21irqrtprio09:05:171
2337899100,4cyclictest9771-21cat10:20:131
2337899100,4cyclictest9771-21cat10:20:131
2337899100,4cyclictest7434-21sort07:30:001
2337899100,4cyclictest32068-21grep10:05:201
2337899100,4cyclictest3107-21cpuspeed09:30:111
2337899100,4cyclictest24870-21cut12:05:011
2337899100,2cyclictest10517-21taskset08:15:131
227982105,4sleep90-21swapper/907:06:3411
28359190,3phc2sys7735-21cat10:15:234
28359190,3phc2sys7735-21cat10:15:234
234219995,3cyclictest23983-21sed08:30:242
234219990,8cyclictest6049-21sshd09:30:292
234169992,4cyclictest0-21swapper/909:20:0011
234169990,9cyclictest0-21swapper/908:35:2511
234169990,8cyclictest1731-21sshd08:04:3911
234169990,1cyclictest0-21swapper/910:15:1711
234169990,1cyclictest0-21swapper/910:15:1611
234169990,1cyclictest0-21swapper/909:15:2011
234169990,1cyclictest0-21swapper/907:20:0111
234129998,1cyclictest0-21swapper/812:15:1910
234129998,1cyclictest0-21swapper/810:10:0910
234129998,1cyclictest0-21swapper/810:10:0910
234129997,1cyclictest0-21swapper/809:40:1710
234129990,8cyclictest0-21swapper/807:22:0210
234129990,1cyclictest0-21swapper/809:05:1210
234129990,1cyclictest0-21swapper/807:15:2110
234069990,3cyclictest8775-21fschecks_count07:30:149
234069990,3cyclictest529-21cpuspeed_turbos12:15:119
234069990,3cyclictest5099-21taskset07:25:149
234069990,3cyclictest27029-21irqrtprio08:35:189
234069990,3cyclictest21715-21switchtime07:45:239
234069990,3cyclictest20642-21wget10:35:139
234069990,3cyclictest19805-21ps09:50:189
234069990,2cyclictest28559-21perl07:15:029
234069990,2cyclictest12712-21cat09:40:189
234039992,6cyclictest115650irq/18-p34p110:56:178
234039990,9cyclictest0-21swapper/610:50:188
234039990,9cyclictest0-21swapper/608:40:178
234039990,8cyclictest23973-21cpuspeed09:15:128
234039990,8cyclictest1336-21grep12:15:138
234039990,8cyclictest12028-21cstates11:05:128
234039990,8cyclictest0-21swapper/611:25:008
234039990,8cyclictest0-21swapper/611:10:148
234039990,8cyclictest0-21swapper/609:05:148
233979997,1cyclictest0-21swapper/510:55:157
233979990,1cyclictest0-21swapper/511:00:197
233939990,9cyclictest0-21swapper/412:30:206
233829997,1cyclictest30114-21cpuspeed_turbos10:05:114
233829990,9cyclictest0-21swapper/207:45:004
233829990,8cyclictest9345-21idleruntime12:25:164
233829990,1cyclictest15201-21aten2_r3power_v11:10:114
233729993,5cyclictest0-21swapper/010:23:500
233729993,5cyclictest0-21swapper/010:23:500
233729991,5cyclictest0-21swapper/008:15:010
233729990,8cyclictest0-21swapper/010:35:130
233729990,1cyclictest0-21swapper/010:00:230
234259987,1cyclictest0-21swapper/1109:50:203
234259987,1cyclictest0-21swapper/1109:25:253
234259986,1cyclictest1449-21df_inode08:45:153
234259980,8cyclictest0-21swapper/1109:40:123
234259980,1cyclictest0-21swapper/1109:35:243
234219987,1cyclictest0-21swapper/1008:02:462
234219981,4cyclictest0-21swapper/1007:15:252
234219980,1cyclictest0-21swapper/1011:10:252
234169987,1cyclictest0-21swapper/907:35:1611
234169980,8cyclictest0-21swapper/910:05:1911
234169980,8cyclictest0-21swapper/909:30:1511
234169980,8cyclictest0-21swapper/908:30:1211
234169980,1cyclictest0-21swapper/912:30:1211
234129987,1cyclictest0-21swapper/811:30:1310
234129986,1cyclictest0-21swapper/807:35:2010
234129980,8cyclictest0-21swapper/812:25:1410
234129980,7cyclictest0-21swapper/812:21:0310
234129980,7cyclictest0-21swapper/811:39:2610
234129980,7cyclictest0-21swapper/811:11:3710
234129980,7cyclictest0-21swapper/810:56:4010
234129980,7cyclictest0-21swapper/810:35:1010
234129980,3cyclictest0-21swapper/811:59:5510
234129980,1cyclictest0-21swapper/808:50:2310
234129980,1cyclictest0-21swapper/808:05:2210
234039987,1cyclictest0-21swapper/609:35:138
234039987,1cyclictest0-21swapper/607:40:168
234039982,5cyclictest115650irq/18-p34p111:46:128
234039982,5cyclictest115650irq/18-p34p111:35:538
234039982,5cyclictest115650irq/18-p34p111:15:298
234039982,5cyclictest115650irq/18-p34p110:15:568
234039982,5cyclictest115650irq/18-p34p110:15:568
234039982,5cyclictest115650irq/18-p34p110:05:588
234039982,5cyclictest115650irq/18-p34p109:56:048
234039981,6cyclictest115650irq/18-p34p109:55:008
234039981,6cyclictest115650irq/18-p34p109:30:568
234039980,8cyclictest0-21swapper/612:00:128
234039980,8cyclictest0-21swapper/611:25:018
234039980,8cyclictest0-21swapper/610:40:148
234039980,8cyclictest0-21swapper/610:30:208
234039980,8cyclictest0-21swapper/610:30:198
234039980,8cyclictest0-21swapper/608:50:188
234039980,8cyclictest0-21swapper/607:15:018
234039980,7cyclictest8018-21/usr/sbin/munin11:40:198
234039980,7cyclictest24908-21hddtemp_smartct07:10:158
234039980,7cyclictest23365-21/usr/sbin/munin10:35:258
234039980,7cyclictest17757-21/usr/sbin/munin12:35:188
234039980,7cyclictest0-21swapper/610:00:158
234039980,3cyclictest28392-21/usr/sbin/munin09:20:128
234039980,2cyclictest14284-21df_inode11:50:128
234039980,1cyclictest0-21swapper/612:25:208
234039980,1cyclictest0-21swapper/611:30:208
234039980,1cyclictest0-21swapper/608:55:158
233979987,1cyclictest0-21swapper/511:10:167
233979987,1cyclictest0-21swapper/510:20:127
233979987,1cyclictest0-21swapper/510:20:117
233979987,1cyclictest0-21swapper/507:45:197
233979987,1cyclictest0-21swapper/507:30:237
233979980,8cyclictest0-21swapper/510:50:207
233979980,8cyclictest0-21swapper/509:05:007
233979980,7cyclictest19432-21diskmemload09:54:457
233939987,1cyclictest0-21swapper/409:50:176
233939987,1cyclictest0-21swapper/408:30:156
233939985,2cyclictest3558-21turbostat12:20:006
233939982,3cyclictest0-21swapper/411:14:246
233939980,8cyclictest0-21swapper/408:25:216
233939980,1cyclictest0-21swapper/408:15:166
233939980,1cyclictest0-21swapper/407:50:176
233879987,1cyclictest0-21swapper/308:35:195
233879980,8cyclictest0-21swapper/312:25:005
233879980,8cyclictest0-21swapper/307:25:135
233879980,8cyclictest0-21swapper/307:15:195
233829987,1cyclictest0-21swapper/211:50:204
233829986,1cyclictest0-21swapper/208:45:144
233829985,2cyclictest22350-21cstates08:30:134
233829980,8cyclictest0-21swapper/209:50:184
233829980,8cyclictest0-21swapper/208:25:234
233829980,1cyclictest0-21swapper/212:15:164
233829980,1cyclictest0-21swapper/209:11:204
233729987,1cyclictest0-21swapper/009:10:150
233729987,1cyclictest0-21swapper/007:30:180
233729986,1cyclictest0-21swapper/007:45:000
233729985,2cyclictest10178-21turbostat09:44:550
233729982,3cyclictest0-21swapper/010:13:320
233729982,3cyclictest0-21swapper/010:13:320
233729981,4cyclictest0-21swapper/011:00:350
233729980,8cyclictest0-21swapper/010:45:140
233729980,8cyclictest0-21swapper/010:15:130
233729980,8cyclictest0-21swapper/010:15:130
233729980,8cyclictest0-21swapper/008:40:190
233729980,8cyclictest0-21swapper/007:45:250
233729980,1cyclictest0-21swapper/009:45:130
233729980,1cyclictest0-21swapper/009:25:130
22955281,6sleep30-21swapper/307:08:525
234259976,1cyclictest0-21swapper/1112:35:263
234259976,1cyclictest0-21swapper/1112:32:143
234259976,1cyclictest0-21swapper/1112:20:473
234259976,1cyclictest0-21swapper/1112:11:183
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional