You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 12:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Fri Mar 27, 2026 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2834911250,58ptp4l0-21swapper/119:06:141
940801010,96rtkit-daemon0-21swapper/1119:08:543
2748127767,8sleep60-21swapper/619:05:248
2824127058,10sleep70-21swapper/719:07:139
2837126856,9sleep50-21swapper/519:09:127
2837026755,10sleep40-21swapper/419:09:126
2819826756,9sleep100-21swapper/1019:06:402
2818826756,9sleep00-21swapper/019:06:300
2882899250,16cyclictest0-21swapper/1022:25:242
2878699212,12cyclictest510-21ps19:15:011
2878699191,17cyclictest0-21swapper/119:25:151
2878699183,9cyclictest0-21swapper/100:10:011
2879199170,1cyclictest0-21swapper/222:25:234
2880499163,8cyclictest0-21swapper/522:50:007
2878699163,8cyclictest0-21swapper/123:50:011
2878699163,8cyclictest0-21swapper/123:05:001
283491160,1ptp4l21-21ksoftirqd/122:30:171
283491160,1ptp4l21-21ksoftirqd/119:20:011
283491160,1ptp4l0-21swapper/120:55:131
283491160,1ptp4l0-21swapper/120:45:211
283491160,0ptp4l0-21swapper/123:15:141
267442160,0sleep00-21swapper/022:10:010
267442160,0sleep00-21swapper/022:10:000
28832991514,1cyclictest0-21swapper/1123:20:013
28832991513,1cyclictest0-21swapper/1120:35:013
2881399153,6cyclictest0-21swapper/722:05:009
2881399153,6cyclictest0-21swapper/722:05:009
2881399152,8cyclictest0-21swapper/719:45:009
2879199150,14cyclictest0-21swapper/219:25:154
2878699153,8cyclictest0-21swapper/123:40:001
2878699152,7cyclictest0-21swapper/122:27:261
283491150,5ptp4l0-21swapper/120:05:161
283491150,0ptp4l0-21swapper/119:50:251
283222157,6sleep80-21swapper/819:08:2610
259462151,10sleep90-21swapper/919:05:1511
2883299140,13cyclictest7675-21awk22:35:003
2883299140,13cyclictest0-21swapper/1100:15:143
2880499142,7cyclictest0-21swapper/522:45:017
2880499142,7cyclictest0-21swapper/521:05:017
2878699142,7cyclictest0-21swapper/120:30:011
2878699141,8cyclictest0-21swapper/123:45:011
283491140,1ptp4l21-21ksoftirqd/121:41:481
283491140,1ptp4l21-21ksoftirqd/121:41:481
283491140,1ptp4l0-21swapper/121:35:211
2883299132,6cyclictest0-21swapper/1121:30:003
2883299130,2cyclictest4387-21latency_hist23:55:003
2880499133,6cyclictest0-21swapper/523:55:017
2880499132,7cyclictest31112-21kworker/u24:100:40:007
2880499132,7cyclictest0-21swapper/519:50:257
2880499130,8cyclictest0-21swapper/520:55:007
2880499130,12cyclictest0-21swapper/522:35:007
2879199130,12cyclictest0-21swapper/200:15:174
2878699132,8cyclictest0-21swapper/123:10:001
2878699132,7cyclictest0-21swapper/122:15:001
283491130,5ptp4l21-21ksoftirqd/100:00:211
283491130,4ptp4l21-21ksoftirqd/119:35:241
283491130,1ptp4l21-21ksoftirqd/123:10:171
283491130,1ptp4l21-21ksoftirqd/119:13:441
283491130,0ptp4l0-21swapper/100:35:171
283491130,0ptp4l0-21swapper/100:25:181
251132130,1sleep61009-21runrttasks20:25:198
2883299123,8cyclictest0-21swapper/1100:05:153
2883299122,6cyclictest0-21swapper/1120:20:003
2883299122,6cyclictest0-21swapper/1100:00:003
2881899122,7cyclictest0-21swapper/821:50:0110
2881899122,7cyclictest0-21swapper/821:50:0010
2881899122,6cyclictest0-21swapper/821:35:0010
2881399124,5cyclictest0-21swapper/700:40:019
2880999129,2cyclictest0-21swapper/620:25:008
2880499123,5cyclictest0-21swapper/521:35:017
2880499122,6cyclictest0-21swapper/521:05:237
2880499122,6cyclictest0-21swapper/520:20:267
2880499121,7cyclictest0-21swapper/523:00:247
283491120,3ptp4l534-21/usr/sbin/munin21:20:191
283491120,2ptp4l30288-21perl21:15:131
283491120,1ptp4l21-21ksoftirqd/123:55:181
283491120,1ptp4l21-21ksoftirqd/120:40:221
283491120,1ptp4l21-21ksoftirqd/100:16:381
283491120,1ptp4l0-21swapper/121:25:121
283491120,1ptp4l0-21swapper/120:00:121
283491120,0ptp4l0-21swapper/123:50:191
283491120,0ptp4l0-21swapper/123:20:121
283491120,0ptp4l0-21swapper/119:45:141
283491120,0ptp4l0-21swapper/119:30:171
283491120,0ptp4l0-21swapper/100:20:221
94080110,1rtkit-daemon32747-21munin-node21:12:457
2883299112,6cyclictest0-21swapper/1123:00:003
2883299112,4cyclictest23100-21phc2sys-jitter19:40:203
2883299112,3cyclictest11891-21irqrtprio23:20:163
2883299112,2cyclictest19855-21cstates20:20:133
2883299111,9cyclictest0-21swapper/1121:15:163
2883299111,2cyclictest0-21swapper/1100:35:133
2883299110,4cyclictest27321-21missed_timers21:10:183
2883299110,2cyclictest9913-21fw_conntrack00:00:173
2883299110,2cyclictest3008-21sed00:35:013
2883299110,2cyclictest0-21swapper/1123:40:163
2883299110,2cyclictest0-21swapper/1120:50:243
2883299110,2cyclictest0-21swapper/1119:20:113
2883299110,1cyclictest0-21swapper/1120:08:063
2883299110,1cyclictest0-21swapper/1120:00:183
2883299110,1cyclictest0-21swapper/1100:10:163
28828991110,1cyclictest0-21swapper/1022:45:172
2881899113,5cyclictest0-21swapper/823:20:0010
2881899113,5cyclictest0-21swapper/820:50:0010
2881899113,5cyclictest0-21swapper/820:40:0110
2881899112,6cyclictest0-21swapper/800:25:0010
2881899112,5cyclictest0-21swapper/819:45:0110
2881399112,6cyclictest0-21swapper/721:35:009
2881399112,6cyclictest0-21swapper/721:00:009
2881399112,5cyclictest0-21swapper/700:20:019
2881399110,10cyclictest19083-21sshd23:30:149
2881399110,10cyclictest17284-21irqrtprio22:45:179
2880499113,4cyclictest0-21swapper/520:40:237
2880499111,6cyclictest0-21swapper/522:20:017
283491110,4ptp4l21-21ksoftirqd/121:30:151
283491110,3ptp4l21-21ksoftirqd/123:25:111
283491110,2ptp4l8845-21expr22:35:131
283491110,2ptp4l21-21ksoftirqd/123:30:241
283491110,2ptp4l0-21swapper/120:20:161
283491110,1ptp4l21-21ksoftirqd/120:35:131
283491110,1ptp4l0-21swapper/122:55:201
283491110,1ptp4l0-21swapper/121:05:171
283491110,1ptp4l0-21swapper/120:50:121
283491110,1ptp4l0-21swapper/120:25:151
283491110,1ptp4l0-21swapper/119:40:141
283491110,1ptp4l0-21swapper/100:10:141
283491110,0ptp4l0-21swapper/122:50:161
283491110,0ptp4l0-21swapper/120:15:061
2883299103,6cyclictest17988-21fschecks_count19:35:163
2883299102,2cyclictest24752-21/usr/sbin/munin19:45:103
2883299102,2cyclictest1717-21aten2_r3power_e19:15:123
2883299101,3cyclictest32712-21grep19:55:113
2883299101,3cyclictest30012-21cut19:10:153
2883299100,4cyclictest9517-21aten2_r3power_c19:25:103
2883299100,4cyclictest14160-21cut19:30:133
2883299100,4cyclictest1177-21expr21:20:123
2883299100,3cyclictest7134-21fschecks_count20:45:143
2883299100,3cyclictest32282-21head23:05:153
2883299100,3cyclictest29395-21cut22:20:163
2883299100,3cyclictest28783-21fschecks_count00:25:163
2883299100,3cyclictest23440-21grep21:05:173
2883299100,3cyclictest23298-21cat20:25:133
2883299100,3cyclictest22557-21cut22:00:113
2883299100,3cyclictest22557-21cut22:00:103
2883299100,3cyclictest20056-21switchtime21:00:233
2883299100,3cyclictest1550-21tail22:25:173
2883299100,3cyclictest15101-21grep20:55:163
2883299100,3cyclictest13236-21gltestperf22:40:153
2883299100,3cyclictest12235-21cstates20:10:113
2883299100,3cyclictest10131-21irqrtprio21:30:173
2883299100,2cyclictest30664-21gltestperf23:45:143
2883299100,2cyclictest29842-21idleruntime19:50:163
2883299100,2cyclictest201412sleep1121:48:453
2883299100,2cyclictest201412sleep1121:48:453
2883299100,2cyclictest16855-21grep22:45:153
2883299100,2cyclictest0-21swapper/1123:35:133
2883299100,2cyclictest0-21swapper/1123:30:143
2883299100,2cyclictest0-21swapper/1123:25:123
2883299100,2cyclictest0-21swapper/1123:10:153
2883299100,2cyclictest0-21swapper/1123:00:143
2883299100,2cyclictest0-21swapper/1122:50:173
2883299100,2cyclictest0-21swapper/1122:35:133
2883299100,2cyclictest0-21swapper/1121:40:143
2883299100,2cyclictest0-21swapper/1121:40:133
2883299100,2cyclictest0-21swapper/1121:35:153
2883299100,2cyclictest0-21swapper/1120:40:143
2883299100,2cyclictest0-21swapper/1120:35:123
2883299100,2cyclictest0-21swapper/1100:20:123
2883299100,1cyclictest0-21swapper/1122:20:003
2882899100,9cyclictest25865-21switchtime22:55:232
2882899100,9cyclictest25183-21fw_conntrack00:20:142
2882899100,1cyclictest0-21swapper/1020:40:152
2881899102,5cyclictest0-21swapper/822:05:0010
2881899102,5cyclictest0-21swapper/822:05:0010
2881899101,6cyclictest28872-21kworker/u24:123:55:0110
2881899101,5cyclictest0-21swapper/820:55:2510
2881399100,3cyclictest17565-21/usr/sbin/munin00:10:159
2880499102,5cyclictest0-21swapper/522:55:007
2880499102,5cyclictest0-21swapper/520:55:007
2880499101,7cyclictest28872-21kworker/u24:123:55:017
2880499101,5cyclictest27260-21kworker/u24:200:30:007
2879199103,6cyclictest0-21swapper/200:10:234
2879199100,1cyclictest0-21swapper/220:40:154
2878699100,3cyclictest12003-21/usr/sbin/munin22:40:151
2878299100,9cyclictest3959-21basename20:00:100
283491100,4ptp4l21-21ksoftirqd/100:33:281
283491100,1ptp4l21-21ksoftirqd/122:00:101
283491100,1ptp4l21-21ksoftirqd/122:00:101
283491100,0ptp4l0-21swapper/122:20:161
283491100,0ptp4l0-21swapper/121:10:251
283491100,0ptp4l0-21swapper/120:10:141
278212100,3sleep757-21ksoftirqd/723:00:139
288329990,3cyclictest27492-21sh22:15:003
288329990,3cyclictest21819-21taskset21:58:113
288329990,3cyclictest21819-21taskset21:58:113
288329990,2cyclictest27116-21date22:10:003
288329990,1cyclictest0-21swapper/1121:51:593
288329990,1cyclictest0-21swapper/1121:51:583
288289998,1cyclictest0-21swapper/1021:40:222
288289998,1cyclictest0-21swapper/1021:40:222
288289990,8cyclictest0-21swapper/1021:15:002
288189998,1cyclictest0-21swapper/822:50:1110
288189998,1cyclictest0-21swapper/800:35:2510
288189998,1cyclictest0-21swapper/800:30:1410
288189998,1cyclictest0-21swapper/800:25:1510
288189998,1cyclictest0-21swapper/800:15:1310
288189998,1cyclictest0-21swapper/800:10:1310
288189992,5cyclictest0-21swapper/819:55:2210
288189991,5cyclictest0-21swapper/820:55:0010
288189990,9cyclictest0-21swapper/823:25:2210
288189990,5cyclictest0-21swapper/823:15:0010
288189990,4cyclictest15895-21munin-run00:10:0110
288189990,1cyclictest0-21swapper/821:00:1810
288189990,1cyclictest0-21swapper/819:35:2110
288139995,3cyclictest706-21cstates19:55:119
288139992,5cyclictest0-21swapper/722:25:249
288139992,5cyclictest0-21swapper/720:45:249
288139992,4cyclictest0-21swapper/721:45:019
288139992,4cyclictest0-21swapper/721:45:009
288139990,1cyclictest0-21swapper/723:27:309
288099998,1cyclictest0-21swapper/619:15:008
288099990,9cyclictest0-21swapper/620:45:248
288099990,8cyclictest0-21swapper/619:55:178
288099990,1cyclictest0-21swapper/621:35:178
288049990,7cyclictest27260-21kworker/u24:200:30:007
288009990,3cyclictest18749-21date21:40:236
288009990,3cyclictest18749-21date21:40:236
287959996,2cyclictest2631-21turbostat00:39:555
287959990,8cyclictest0-21swapper/300:10:125
287919998,1cyclictest0-21swapper/221:20:124
287919993,5cyclictest0-21swapper/222:20:004
287919993,5cyclictest0-21swapper/200:35:004
287919992,6cyclictest0-21swapper/221:40:244
287919992,6cyclictest0-21swapper/221:40:234
287919990,9cyclictest0-21swapper/222:20:184
287919990,8cyclictest0-21swapper/221:05:224
287919990,1cyclictest0-21swapper/223:25:104
287919990,1cyclictest0-21swapper/220:05:214
287829990,8cyclictest0-21swapper/023:35:000
28349190,1ptp4l24810-21diskmemload22:17:001
28349190,1ptp4l21-21ksoftirqd/122:45:111
28349190,1ptp4l21-21ksoftirqd/122:05:011
28349190,1ptp4l21-21ksoftirqd/122:05:011
28349190,1ptp4l21-21ksoftirqd/121:55:011
28349190,1ptp4l21-21ksoftirqd/121:55:011
28349190,1ptp4l21-21ksoftirqd/121:50:001
28349190,1ptp4l21-21ksoftirqd/121:50:001
28349190,1ptp4l21-21ksoftirqd/121:48:161
28349190,1ptp4l21-21ksoftirqd/121:48:161
28349190,1ptp4l21-21ksoftirqd/121:00:011
28349190,1ptp4l21-21ksoftirqd/119:55:001
28204293,5sleep30-21swapper/319:06:465
288329980,1cyclictest0-21swapper/1122:05:053
288289987,1cyclictest0-21swapper/1023:40:162
288289987,1cyclictest0-21swapper/1020:55:192
288289987,1cyclictest0-21swapper/1020:00:242
288289980,8cyclictest0-21swapper/1022:11:572
288289980,8cyclictest0-21swapper/1021:00:152
288289980,8cyclictest0-21swapper/1000:05:202
288289980,1cyclictest0-21swapper/1022:40:122
288239987,1cyclictest0-21swapper/921:43:4511
288239987,1cyclictest0-21swapper/921:43:4511
288239987,1cyclictest0-21swapper/920:05:1911
288239987,1cyclictest0-21swapper/919:20:0111
288239987,1cyclictest0-21swapper/900:20:2311
288239980,8cyclictest0-21swapper/922:35:1511
288239980,8cyclictest0-21swapper/919:10:1411
288239980,7cyclictest0-21swapper/919:33:5511
288239980,1cyclictest0-21swapper/923:29:5911
288239980,1cyclictest0-21swapper/919:55:2311
288239980,1cyclictest0-21swapper/919:43:2311
288189987,1cyclictest0-21swapper/819:35:0010
288189982,5cyclictest0-21swapper/820:45:0010
288189982,4cyclictest0-21swapper/823:10:0010
288189982,4cyclictest0-21swapper/821:40:0010
288189982,4cyclictest0-21swapper/800:05:0010
288189981,4cyclictest0-21swapper/823:20:2310
288189980,8cyclictest0-21swapper/823:45:2110
288189980,8cyclictest0-21swapper/823:40:1910
288189980,8cyclictest0-21swapper/822:45:2110
288189980,8cyclictest0-21swapper/820:25:2110
288139985,2cyclictest32187-21munin-run22:25:009
288139980,8cyclictest0-21swapper/723:50:139
288139980,8cyclictest0-21swapper/722:35:159
288139980,8cyclictest0-21swapper/721:40:129
288139980,8cyclictest0-21swapper/721:40:119
288139980,8cyclictest0-21swapper/719:15:019
288139980,7cyclictest8628-21/usr/sbin/munin23:15:199
288139980,7cyclictest32653-21/usr/sbin/munin00:30:159
288139980,7cyclictest27333-21tr20:30:129
288139980,7cyclictest22370-21/usr/sbin/munin22:50:239
288139980,7cyclictest0-21swapper/700:25:189
288139980,3cyclictest30051-21cstates23:45:129
288139980,1cyclictest5730-21aten2_r3power_e19:20:109
288139980,1cyclictest23070-21smartctl23:35:149
288139980,1cyclictest0-21swapper/723:57:159
288139980,1cyclictest0-21swapper/722:30:209
288139980,1cyclictest0-21swapper/719:45:159
288139980,1cyclictest0-21swapper/700:00:119
288099987,1cyclictest0-21swapper/623:15:188
288099986,2cyclictest10575-21open_files21:30:208
288099986,1cyclictest0-21swapper/621:07:238
288099980,8cyclictest0-21swapper/622:20:168
288099980,7cyclictest24810-21diskmemload22:14:038
288099980,1cyclictest0-21swapper/623:00:238
288099980,1cyclictest0-21swapper/622:45:208
288049986,1cyclictest0-21swapper/520:35:177
288049980,3cyclictest0-21swapper/521:45:007
288049980,3cyclictest0-21swapper/521:45:007
288049980,1cyclictest0-21swapper/523:05:127
288049980,1cyclictest0-21swapper/519:40:117
288009987,1cyclictest0-21swapper/400:25:166
288009986,1cyclictest14750irq/20-ata_piix21:20:006
288009985,2cyclictest9552-21turbostat20:54:556
288009980,7cyclictest0-21swapper/422:42:566
288009980,7cyclictest0-21swapper/419:50:156
288009980,7cyclictest0-21swapper/419:44:466
288009980,7cyclictest0-21swapper/400:23:286
287959987,1cyclictest0-21swapper/323:00:115
287959987,1cyclictest0-21swapper/322:45:245
287959987,1cyclictest0-21swapper/300:30:225
287959980,8cyclictest0-21swapper/322:40:255
287959980,7cyclictest0-21swapper/319:55:185
287959980,1cyclictest0-21swapper/323:40:225
287959980,1cyclictest0-21swapper/322:20:285
287959980,1cyclictest0-21swapper/322:04:235
287959980,1cyclictest0-21swapper/322:04:235
287959980,1cyclictest0-21swapper/321:35:125
287959980,1cyclictest0-21swapper/321:30:145
287959980,1cyclictest0-21swapper/300:05:155
287919985,2cyclictest0-21swapper/223:50:144
287919984,3cyclictest0-21swapper/200:29:554
287919980,8cyclictest0-21swapper/220:30:254
287919980,8cyclictest0-21swapper/220:00:144
287919980,1cyclictest0-21swapper/221:25:164
287829987,1cyclictest0-21swapper/022:30:150
287829987,1cyclictest0-21swapper/020:55:130
287829980,8cyclictest0-21swapper/020:00:000
287829980,7cyclictest0-21swapper/023:06:240
287829980,1cyclictest0-21swapper/023:45:240
287829980,1cyclictest0-21swapper/022:55:210
287829980,1cyclictest0-21swapper/000:35:190
26920280,1sleep2261rcuc/219:05:204
3557270,0sleep90-21swapper/900:35:1311
288289976,1cyclictest0-21swapper/1023:50:452
288289976,1cyclictest0-21swapper/1023:15:032
288289976,1cyclictest0-21swapper/1023:10:292
288289976,1cyclictest0-21swapper/1023:00:222
288289976,1cyclictest0-21swapper/1022:35:032
288289976,1cyclictest0-21swapper/1022:16:062
288289976,1cyclictest0-21swapper/1021:35:242
288289976,1cyclictest0-21swapper/1021:30:242
288289976,1cyclictest0-21swapper/1021:06:472
288289976,1cyclictest0-21swapper/1020:46:342
288289976,1cyclictest0-21swapper/1020:31:012
288289976,1cyclictest0-21swapper/1020:22:532
288289976,1cyclictest0-21swapper/1020:10:052
288289976,1cyclictest0-21swapper/1020:05:292
288289976,1cyclictest0-21swapper/1019:50:152
288289976,1cyclictest0-21swapper/1019:46:572
288289976,1cyclictest0-21swapper/1019:42:432
288289976,1cyclictest0-21swapper/1019:20:572
288289976,1cyclictest0-21swapper/1019:15:122
288289976,1cyclictest0-21swapper/1019:10:182
288289976,1cyclictest0-21swapper/1000:00:102
288289974,2cyclictest979-21NetworkManager21:16:512
288289973,3cyclictest6968-21sendmail_mailtr21:25:202
288289970,7cyclictest0-21swapper/1023:45:122
288289970,7cyclictest0-21swapper/1023:35:132
288289970,7cyclictest0-21swapper/1023:30:192
288289970,7cyclictest0-21swapper/1023:05:202
288289970,7cyclictest0-21swapper/1022:30:152
288289970,7cyclictest0-21swapper/1021:45:052
288289970,7cyclictest0-21swapper/1021:45:052
288289970,7cyclictest0-21swapper/1021:20:232
288289970,7cyclictest0-21swapper/1019:35:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional