You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 09:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Thu Dec 04, 2025 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275991890,62ptp4l0-21swapper/119:07:511
928126654,10sleep50-21swapper/519:08:097
926026554,9sleep100-21swapper/1019:07:502
938226453,9sleep110-21swapper/1119:09:403
939126352,9sleep70-21swapper/719:09:489
610026251,9sleep00-21swapper/019:05:040
922626049,9sleep40-21swapper/419:07:196
35592600,0sleep90-21swapper/921:05:1411
925625951,6sleep60-21swapper/619:07:468
9790993224,8cyclictest0-21swapper/1023:25:222
979099240,1cyclictest0-21swapper/1021:20:182
976399242,9cyclictest0-21swapper/400:35:006
976399242,9cyclictest0-21swapper/400:35:006
976399212,15cyclictest0-21swapper/420:20:006
976399212,14cyclictest0-21swapper/422:35:006
152002210,0sleep8621rcuc/823:25:1610
317102200,0chrt31709-21ls21:00:1311
20402200,0sleep70-21swapper/723:50:189
979699191,17cyclictest0-21swapper/1123:05:173
978699183,8cyclictest0-21swapper/919:30:0111
978699182,8cyclictest0-21swapper/923:30:0011
75012180,0sleep110-21swapper/1121:50:213
143482180,0sleep9681rcuc/900:05:2211
979099179,8cyclictest0-21swapper/1021:45:242
978699173,8cyclictest0-21swapper/922:05:0011
978199174,8cyclictest0-21swapper/821:55:2310
977799174,8cyclictest0-21swapper/700:30:249
977799174,8cyclictest0-21swapper/700:30:249
977799173,9cyclictest0-21swapper/722:40:009
977799173,8cyclictest0-21swapper/720:50:019
977799173,8cyclictest0-21swapper/720:30:019
977299172,9cyclictest0-21swapper/622:50:248
975999174,7cyclictest0-21swapper/320:30:015
975499173,9cyclictest0-21swapper/223:40:004
975099174,8cyclictest0-21swapper/123:15:001
275991170,0ptp4l0-21swapper/119:45:241
110992170,0sleep90-21swapper/922:40:0011
979699161,3cyclictest0-21swapper/1119:35:183
979699160,16cyclictest0-21swapper/1122:10:203
978699163,8cyclictest0-21swapper/920:25:0111
978199162,8cyclictest0-21swapper/800:30:0010
978199162,8cyclictest0-21swapper/800:30:0010
977799163,8cyclictest0-21swapper/719:35:019
977799162,9cyclictest0-21swapper/720:25:019
977799162,9cyclictest0-21swapper/719:40:019
977299162,7cyclictest0-21swapper/622:20:008
976899163,7cyclictest0-21swapper/523:25:007
975999163,7cyclictest0-21swapper/323:51:545
975999161,9cyclictest0-21swapper/321:55:005
975499163,7cyclictest0-21swapper/200:00:264
975099163,8cyclictest0-21swapper/123:19:591
975099162,8cyclictest0-21swapper/123:35:261
975099161,2cyclictest0-21swapper/119:15:181
61082167,7sleep80-21swapper/819:05:1210
275991160,0ptp4l0-21swapper/122:35:121
275991160,0ptp4l0-21swapper/122:05:251
181632160,0sleep80-21swapper/822:45:2410
978699153,7cyclictest0-21swapper/922:45:0011
978699152,7cyclictest0-21swapper/922:10:0011
977799153,7cyclictest0-21swapper/719:55:019
977299153,8cyclictest0-21swapper/622:10:008
977299153,7cyclictest0-21swapper/620:55:008
977299152,7cyclictest0-21swapper/619:35:018
976899153,8cyclictest0-21swapper/523:00:007
975999153,7cyclictest0-21swapper/320:20:015
975999152,8cyclictest0-21swapper/320:05:265
975499153,8cyclictest0-21swapper/222:54:594
975499152,8cyclictest0-21swapper/223:05:004
975499152,8cyclictest0-21swapper/222:15:004
975099153,6cyclictest0-21swapper/100:30:011
975099153,6cyclictest0-21swapper/100:30:011
277972150,0sleep90-21swapper/919:30:1511
275991150,2ptp4l31467-21entropy23:05:141
275991150,2ptp4l28308-21latency_hist22:20:001
275991150,1ptp4l2821-21proc_pri20:20:231
275991150,0ptp4l0-21swapper/121:50:181
275991150,0ptp4l0-21swapper/120:40:151
210872150,0sleep30-21swapper/323:35:005
979099142,7cyclictest0-21swapper/1023:30:232
978199142,8cyclictest0-21swapper/823:55:0110
978199142,7cyclictest0-21swapper/822:00:2310
9781991410,3cyclictest0-21swapper/820:30:1710
977799142,7cyclictest0-21swapper/721:45:009
977799142,7cyclictest0-21swapper/721:35:249
977799142,7cyclictest0-21swapper/700:05:019
977299144,6cyclictest0-21swapper/621:30:008
977299142,8cyclictest0-21swapper/622:50:008
976399142,8cyclictest0-21swapper/421:15:006
975999143,7cyclictest0-21swapper/320:45:015
975999143,7cyclictest0-21swapper/319:45:015
975999142,7cyclictest0-21swapper/323:15:245
975999142,7cyclictest0-21swapper/321:45:005
975999142,7cyclictest0-21swapper/321:30:255
975999142,7cyclictest0-21swapper/320:50:015
975499142,8cyclictest0-21swapper/222:00:004
975499142,8cyclictest0-21swapper/200:15:004
975099143,7cyclictest0-21swapper/121:30:001
975099143,6cyclictest0-21swapper/100:40:011
975099143,6cyclictest0-21swapper/100:40:011
975099142,6cyclictest0-21swapper/123:55:241
974599143,8cyclictest26576-21kworker/u24:220:35:010
974599142,7cyclictest0-21swapper/021:45:250
974599142,7cyclictest0-21swapper/020:20:010
311052140,0sleep80-21swapper/821:00:0010
275991140,0ptp4l0-21swapper/123:40:161
275991140,0ptp4l0-21swapper/122:30:141
275991140,0ptp4l0-21swapper/120:30:221
979699132,6cyclictest0-21swapper/1100:15:003
979699130,8cyclictest0-21swapper/1122:30:003
978199133,6cyclictest0-21swapper/819:10:2310
978199131,7cyclictest0-21swapper/800:00:0010
977799132,6cyclictest0-21swapper/719:45:019
977299132,7cyclictest0-21swapper/623:05:248
977299132,6cyclictest0-21swapper/623:55:008
976899133,6cyclictest0-21swapper/520:55:247
976899132,7cyclictest0-21swapper/522:04:597
976899132,7cyclictest0-21swapper/521:50:007
975099132,7cyclictest0-21swapper/123:45:231
975099132,6cyclictest0-21swapper/121:20:001
975099132,6cyclictest0-21swapper/120:15:011
974599132,7cyclictest0-21swapper/021:55:000
974599130,1cyclictest0-21swapper/000:05:190
327032130,0sleep10-21swapper/100:30:161
327032130,0sleep10-21swapper/100:30:151
275991130,0ptp4l0-21swapper/123:05:011
275991130,0ptp4l0-21swapper/120:27:371
275991130,0ptp4l0-21swapper/120:15:181
275991130,0ptp4l0-21swapper/119:41:351
275991130,0ptp4l0-21swapper/119:10:111
978699122,6cyclictest0-21swapper/920:51:1711
978699122,6cyclictest0-21swapper/920:50:0011
977799122,6cyclictest0-21swapper/720:40:019
977799120,8cyclictest0-21swapper/723:50:009
977799120,5cyclictest0-21swapper/722:50:009
977299122,6cyclictest0-21swapper/621:40:008
976899122,6cyclictest0-21swapper/500:15:017
976899122,6cyclictest0-21swapper/500:05:007
976899121,7cyclictest23688-21kworker/u24:122:55:007
976399122,6cyclictest0-21swapper/420:50:256
976399120,4cyclictest0-21swapper/400:00:186
975999122,6cyclictest0-21swapper/320:00:255
975499122,7cyclictest0-21swapper/220:15:014
975499122,7cyclictest0-21swapper/200:40:014
975499122,7cyclictest0-21swapper/200:40:014
975499122,6cyclictest0-21swapper/221:00:004
975099122,6cyclictest0-21swapper/119:25:221
975099122,6cyclictest0-21swapper/100:24:591
974599121,6cyclictest0-21swapper/023:00:000
275991120,6ptp4l0-21swapper/122:28:221
275991120,3ptp4l0-21swapper/119:20:231
275991120,2ptp4l16144-21/usr/sbin/munin00:10:231
275991120,0ptp4l0-21swapper/123:50:131
275991120,0ptp4l0-21swapper/123:22:461
275991120,0ptp4l0-21swapper/121:56:001
275991120,0ptp4l0-21swapper/120:47:401
275991120,0ptp4l0-21swapper/100:02:491
979699118,2cyclictest19657-21hddtemp_smartct21:25:153
979699112,6cyclictest0-21swapper/1121:00:243
979699112,6cyclictest0-21swapper/1119:50:013
979699110,5cyclictest5633-21munin-run23:14:593
979699110,10cyclictest0-21swapper/1121:09:593
979099112,6cyclictest0-21swapper/1020:10:012
979099110,10cyclictest30906-21awk00:30:002
979099110,10cyclictest30906-21awk00:30:002
977799119,1cyclictest0-21swapper/700:09:549
975999111,6cyclictest0-21swapper/323:40:015
975999110,10cyclictest5523-21irqrtprio22:30:185
975499112,6cyclictest0-21swapper/221:24:594
975499112,6cyclictest0-21swapper/219:45:234
975499110,10cyclictest15788-21grep21:20:174
275991110,2ptp4l31599-21fschecks_count19:35:151
275991110,2ptp4l3107-21irqstats21:45:191
275991110,1ptp4l0-21swapper/122:00:141
275991110,1ptp4l0-21swapper/121:30:181
275991110,0ptp4l0-21swapper/122:55:111
275991110,0ptp4l0-21swapper/122:45:241
275991110,0ptp4l0-21swapper/121:38:461
199212110,0chrt0-21swapper/1019:20:152
979699102,4cyclictest0-21swapper/1120:35:013
979099102,5cyclictest0-21swapper/1000:25:012
978699102,5cyclictest0-21swapper/919:35:2511
978199102,6cyclictest0-21swapper/820:30:0010
977799103,4cyclictest0-21swapper/719:25:019
977299100,9cyclictest28973-21cat20:15:008
977299100,3cyclictest9944-21runrttasks21:10:568
977299100,2cyclictest17293-21fw_forwarded_lo00:10:178
976899102,5cyclictest0-21swapper/519:20:017
976399108,1cyclictest0-21swapper/419:40:246
976399100,9cyclictest0-21swapper/420:20:186
976399100,9cyclictest0-21swapper/420:10:156
975499108,1cyclictest30115-21perl23:45:144
975499102,5cyclictest0-21swapper/220:45:014
975499102,4cyclictest0-21swapper/222:40:014
975499100,10cyclictest0-21swapper/219:10:014
974599100,9cyclictest0-21swapper/022:35:000
974599100,1cyclictest0-21swapper/023:55:010
275991100,1ptp4l31299-21if_err_p34p121:40:171
275991100,1ptp4l29708-21sed19:30:251
275991100,1ptp4l24556-21df_inode20:50:141
275991100,1ptp4l21133-21gltestperf00:15:181
275991100,1ptp4l19326-21cut20:00:181
275991100,1ptp4l15275-21cat19:55:181
275991100,1ptp4l12180-21ps19:50:201
275991100,0ptp4l0-21swapper/123:30:171
275991100,0ptp4l0-21swapper/122:10:231
97969998,1cyclictest0-21swapper/1122:20:003
97909991,5cyclictest0-21swapper/1020:40:012
97909991,5cyclictest0-21swapper/1019:30:012
97909991,5cyclictest0-21swapper/1019:15:012
97869996,2cyclictest29842-21cut00:25:2211
97869996,2cyclictest29842-21cut00:25:2211
97869993,3cyclictest691ksoftirqd/922:20:0011
97869991,5cyclictest0-21swapper/920:10:0111
97869990,8cyclictest16632-21irqrtprio19:15:1911
97819990,9cyclictest0-21swapper/820:40:0110
97819990,9cyclictest0-21swapper/819:15:0210
97779997,1cyclictest0-21swapper/720:54:599
97779992,5cyclictest0-21swapper/723:05:009
97779990,9cyclictest0-21swapper/719:15:019
97779990,1cyclictest0-21swapper/722:10:239
97729996,2cyclictest4435-21/usr/sbin/munin23:10:208
97729996,2cyclictest2362-21sed20:20:218
97689992,5cyclictest0-21swapper/521:35:007
97639990,1cyclictest0-21swapper/422:35:146
97639990,1cyclictest0-21swapper/421:05:216
97639990,1cyclictest0-21swapper/420:00:226
97639990,1cyclictest0-21swapper/400:20:166
97599996,2cyclictest30903-21turbostat00:30:005
97599996,2cyclictest30903-21turbostat00:29:595
97599992,6cyclictest0-21swapper/320:35:255
97549998,1cyclictest0-21swapper/223:00:004
97549996,2cyclictest28605-21turbostat20:19:564
97549992,4cyclictest0-21swapper/219:40:244
97549991,6cyclictest0-21swapper/223:30:004
97549990,8cyclictest4042-21cpuspeed22:30:134
97549990,8cyclictest23128-21expr21:30:154
97549990,8cyclictest0-21swapper/222:25:244
97549990,8cyclictest0-21swapper/222:20:144
97549990,8cyclictest0-21swapper/221:35:164
97549990,8cyclictest0-21swapper/221:00:154
97549990,8cyclictest0-21swapper/219:50:234
97549990,8cyclictest0-21swapper/219:20:154
97549990,8cyclictest0-21swapper/200:05:174
97549990,1cyclictest8578-21fschecks_count22:35:154
97549990,1cyclictest26062-21cut22:15:214
97549990,1cyclictest0-21swapper/223:10:244
97549990,1cyclictest0-21swapper/200:20:254
97459997,1cyclictest27615-21wget20:55:130
97459996,2cyclictest5631-21turbostat19:45:010
97459996,2cyclictest20912-21sshd23:34:410
97459996,2cyclictest0-21swapper/000:25:000
97459992,5cyclictest0-21swapper/020:45:010
97459992,4cyclictest0-21swapper/021:20:000
97459990,1cyclictest0-21swapper/022:45:000
97459990,1cyclictest0-21swapper/000:15:260
27599190,1ptp4l12878-21cstates00:05:161
27599190,0ptp4l0-21swapper/123:25:121
27599190,0ptp4l0-21swapper/122:50:131
27599190,0ptp4l0-21swapper/122:40:121
27599190,0ptp4l0-21swapper/122:20:121
27599190,0ptp4l0-21swapper/121:20:131
27599190,0ptp4l0-21swapper/121:09:591
27599190,0ptp4l0-21swapper/121:05:141
27599190,0ptp4l0-21swapper/121:00:131
27599190,0ptp4l0-21swapper/120:55:141
27599190,0ptp4l0-21swapper/120:35:131
27599190,0ptp4l0-21swapper/120:05:011
97969987,1cyclictest0-21swapper/1121:55:243
97969987,1cyclictest0-21swapper/1121:35:173
97969987,1cyclictest0-21swapper/1119:40:193
97969987,1cyclictest0-21swapper/1119:25:113
97969987,1cyclictest0-21swapper/1100:15:143
97969987,1cyclictest0-21swapper/1100:05:263
97969985,2cyclictest9794-21netstat22:35:203
97969980,8cyclictest0-21swapper/1123:35:213
97969980,8cyclictest0-21swapper/1123:00:013
97969980,8cyclictest0-21swapper/1120:15:173
97969980,8cyclictest0-21swapper/1119:50:163
97969980,1cyclictest0-21swapper/1123:45:123
97969980,1cyclictest0-21swapper/1122:40:243
97909987,1cyclictest0-21swapper/1023:40:282
97909987,1cyclictest0-21swapper/1023:05:142
97909987,1cyclictest0-21swapper/1022:35:142
97909987,1cyclictest0-21swapper/1021:54:592
97909987,1cyclictest0-21swapper/1021:10:252
97909981,5cyclictest0-21swapper/1023:10:012
97909980,8cyclictest0-21swapper/1022:00:132
97909980,8cyclictest0-21swapper/1019:35:192
97909980,1cyclictest0-21swapper/1023:20:252
97909980,1cyclictest0-21swapper/1022:30:392
97909980,1cyclictest0-21swapper/1019:50:202
97909980,1cyclictest0-21swapper/1019:40:142
97909980,1cyclictest0-21swapper/1000:15:192
97869985,2cyclictest2846-21ps21:45:1911
97869981,4cyclictest0-21swapper/921:50:2511
97869980,1cyclictest0-21swapper/920:10:2111
97869980,1cyclictest0-21swapper/919:40:1511
97819987,1cyclictest0-21swapper/822:40:1510
97819987,1cyclictest0-21swapper/800:30:2410
97819987,1cyclictest0-21swapper/800:30:2310
97819980,1cyclictest0-21swapper/822:30:1910
97819980,1cyclictest0-21swapper/821:16:2810
97819980,1cyclictest0-21swapper/800:05:2810
97779987,1cyclictest0-21swapper/722:55:149
97779987,1cyclictest0-21swapper/722:05:189
97779987,1cyclictest0-21swapper/721:00:179
97779987,1cyclictest0-21swapper/720:15:189
97779985,2cyclictest19439-21munin-run00:15:009
97779984,3cyclictest12247-21irqrtprio21:15:199
97779980,8cyclictest0-21swapper/721:10:249
97779980,1cyclictest0-21swapper/723:05:159
97779980,1cyclictest0-21swapper/720:40:189
97779980,1cyclictest0-21swapper/719:55:269
97779980,1cyclictest0-21swapper/700:15:249
97729980,7cyclictest26308-21awk23:00:008
97729980,1cyclictest0-21swapper/622:25:168
97729980,1cyclictest0-21swapper/620:55:188
97689987,1cyclictest0-21swapper/523:35:187
97689987,1cyclictest0-21swapper/523:10:207
97689980,8cyclictest0-21swapper/500:30:207
97689980,8cyclictest0-21swapper/500:30:207
97689980,1cyclictest0-21swapper/520:31:207
97689980,1cyclictest0-21swapper/519:45:197
97639987,1cyclictest0-21swapper/423:05:126
97639985,2cyclictest20959-21turbostat20:09:566
97639980,8cyclictest0-21swapper/421:20:256
97639980,7cyclictest21232-21cat23:35:006
97639980,7cyclictest0-21swapper/422:10:256
97639980,3cyclictest9455-21latency_hist23:20:006
97639980,3cyclictest17127-21latency_hist20:00:016
97639980,3cyclictest13492-21latency_hist19:15:006
97639980,1cyclictest0-21swapper/422:25:206
97639980,1cyclictest0-21swapper/420:25:156
97639980,1cyclictest0-21swapper/400:10:196
97599987,1cyclictest0-21swapper/323:55:135
97599987,1cyclictest0-21swapper/321:35:175
97599980,8cyclictest0-21swapper/322:00:185
97599980,7cyclictest0-21swapper/321:00:155
97599980,1cyclictest0-21swapper/319:20:235
97549987,1cyclictest0-21swapper/223:20:184
97549987,1cyclictest0-21swapper/221:15:144
97549987,1cyclictest0-21swapper/220:30:204
97549980,8cyclictest0-21swapper/223:55:244
97549980,8cyclictest0-21swapper/223:50:194
97549980,8cyclictest0-21swapper/223:40:174
97549980,8cyclictest0-21swapper/223:05:124
97549980,8cyclictest0-21swapper/200:15:214
97549980,7cyclictest18787-21latency_hist22:50:004
97549980,3cyclictest6526-21/usr/sbin/munin21:50:164
97549980,2cyclictest15168-21/usr/sbin/munin19:15:184
97549980,2cyclictest13440-21fw_conntrack20:35:164
97549980,2cyclictest0-21swapper/220:50:194
97549980,1cyclictest0-21swapper/221:10:134
97549980,1cyclictest0-21swapper/220:25:224
97549980,1cyclictest0-21swapper/220:20:164
97549980,1cyclictest0-21swapper/219:35:144
97549980,1cyclictest0-21swapper/200:25:254
97549980,1cyclictest0-21swapper/200:25:254
97459987,1cyclictest0-21swapper/023:20:180
97459987,1cyclictest0-21swapper/022:35:140
97459987,1cyclictest0-21swapper/021:40:160
97459984,3cyclictest16650-21cut00:10:150
97459980,8cyclictest0-21swapper/023:25:130
97459980,8cyclictest0-21swapper/020:35:150
97459980,8cyclictest0-21swapper/020:20:260
97459980,8cyclictest0-21swapper/019:50:130
97459980,8cyclictest0-21swapper/019:15:150
97459980,1cyclictest0-21swapper/019:25:510
9186281,5sleep30-21swapper/319:06:435
8444281,5sleep90-21swapper/919:05:2511
97969976,1cyclictest0-21swapper/1123:55:263
97969976,1cyclictest0-21swapper/1123:30:493
97969976,1cyclictest0-21swapper/1123:20:523
97969976,1cyclictest0-21swapper/1122:00:063
97969976,1cyclictest0-21swapper/1121:45:433
97969976,1cyclictest0-21swapper/1121:40:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional