You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Fri May 09, 2025 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2759911080,58ptp4l0-21swapper/119:09:311
94980790,74rtkit-daemon0-21swapper/1019:06:512
674027058,9sleep00-21swapper/019:05:070
909726755,9sleep50-21swapper/519:05:237
235302660,0sleep80-21swapper/821:30:1710
981326453,9sleep110-21swapper/1119:06:543
980926452,10sleep70-21swapper/719:06:499
980626453,9sleep40-21swapper/419:06:466
984926251,9sleep60-21swapper/619:07:248
294952550,0sleep110-21swapper/1120:15:133
111772540,0sleep10-21swapper/123:20:241
20222530,0sleep30-21swapper/319:40:015
224882230,0sleep50-21swapper/522:10:207
10438992120,1cyclictest0-21swapper/1000:10:182
1041599182,10cyclictest0-21swapper/523:00:247
1044299173,7cyclictest0-21swapper/1123:00:013
1044299170,17cyclictest0-21swapper/1100:05:163
1042499172,8cyclictest0-21swapper/722:10:009
1042499172,10cyclictest0-21swapper/700:05:009
1041599173,8cyclictest0-21swapper/519:10:237
1041599173,8cyclictest0-21swapper/519:10:227
1039999173,9cyclictest0-21swapper/223:10:004
1039999173,10cyclictest0-21swapper/219:30:004
1039999172,9cyclictest0-21swapper/222:55:014
1039599173,9cyclictest0-21swapper/123:55:001
1039599173,8cyclictest0-21swapper/123:50:011
1039599172,9cyclictest0-21swapper/121:55:001
98102167,7sleep80-21swapper/819:06:5110
275991160,1ptp4l21-21ksoftirqd/119:15:211
275991160,1ptp4l21-21ksoftirqd/119:15:201
275991160,0ptp4l0-21swapper/122:40:241
1043399161,9cyclictest0-21swapper/919:25:0111
1043399160,3cyclictest0-21swapper/921:00:1911
1043399160,10cyclictest0-21swapper/919:40:0111
1042999164,7cyclictest0-21swapper/821:30:0110
1042499163,8cyclictest0-21swapper/722:45:259
1042499162,9cyclictest5984-21kworker/u24:019:15:019
1042499162,9cyclictest5984-21kworker/u24:019:15:019
1040499162,8cyclictest0-21swapper/323:50:005
1039999163,8cyclictest0-21swapper/223:45:004
1039999163,8cyclictest0-21swapper/221:15:004
1039999163,8cyclictest0-21swapper/200:15:004
1039999162,9cyclictest0-21swapper/219:10:004
275991150,1ptp4l21-21ksoftirqd/122:55:171
275991150,0ptp4l0-21swapper/122:50:181
275991150,0ptp4l0-21swapper/119:30:001
1043399153,8cyclictest0-21swapper/923:05:0011
1043399153,7cyclictest0-21swapper/922:25:0011
1043399153,7cyclictest0-21swapper/920:00:0011
1042999153,8cyclictest0-21swapper/820:25:0110
1042999153,7cyclictest0-21swapper/823:55:0010
1042999152,8cyclictest0-21swapper/822:20:0010
1042999152,8cyclictest0-21swapper/820:50:0110
1042999150,9cyclictest0-21swapper/820:30:0110
1042999150,14cyclictest0-21swapper/823:45:1810
1042499151,9cyclictest0-21swapper/722:45:009
10420991512,2cyclictest0-21swapper/622:35:008
1040499153,7cyclictest0-21swapper/323:30:005
1040499152,8cyclictest0-21swapper/321:20:015
1039999153,8cyclictest0-21swapper/222:10:004
1039999153,8cyclictest0-21swapper/219:45:004
1039999153,8cyclictest0-21swapper/200:25:014
1039999152,8cyclictest0-21swapper/222:00:274
1039599153,8cyclictest0-21swapper/121:40:001
1039599153,8cyclictest0-21swapper/121:15:001
275991140,1ptp4l21-21ksoftirqd/120:05:241
275991140,0ptp4l0-21swapper/120:40:231
275991140,0ptp4l0-21swapper/120:00:181
1044299143,8cyclictest0-21swapper/1119:40:013
1044299143,6cyclictest0-21swapper/1100:24:213
1044299142,7cyclictest0-21swapper/1123:45:013
1043399142,7cyclictest0-21swapper/922:10:2411
1043399142,7cyclictest0-21swapper/919:10:0111
1043399142,7cyclictest0-21swapper/919:10:0011
1042999142,7cyclictest0-21swapper/819:40:0010
1041599143,8cyclictest0-21swapper/521:15:027
1041599143,6cyclictest0-21swapper/520:55:017
1041599142,8cyclictest0-21swapper/522:35:017
1041599142,7cyclictest0-21swapper/519:20:017
1041599142,7cyclictest0-21swapper/519:20:007
1040499143,6cyclictest0-21swapper/322:34:595
1040499142,7cyclictest0-21swapper/322:50:005
1040499141,2cyclictest0-21swapper/300:00:205
1039999142,8cyclictest0-21swapper/222:05:014
1039599143,7cyclictest0-21swapper/123:55:001
275991130,1ptp4l21-21ksoftirqd/122:25:221
275991130,0ptp4l0-21swapper/123:00:231
275991130,0ptp4l0-21swapper/121:25:251
275991130,0ptp4l0-21swapper/119:50:221
1044299132,7cyclictest0-21swapper/1120:30:003
1044299132,6cyclictest0-21swapper/1121:50:003
1044299131,7cyclictest0-21swapper/1120:25:013
1043399132,6cyclictest0-21swapper/923:30:0111
1042999132,7cyclictest0-21swapper/823:25:0010
1042499133,6cyclictest0-21swapper/721:10:019
1041599130,3cyclictest0-21swapper/523:35:187
10399991311,1cyclictest581-21turbostat21:45:004
1039599132,7cyclictest0-21swapper/121:15:241
275991120,1ptp4l21-21ksoftirqd/123:43:151
275991120,1ptp4l21-21ksoftirqd/122:17:411
275991120,1ptp4l21-21ksoftirqd/120:35:001
275991120,1ptp4l21-21ksoftirqd/120:14:271
275991120,0ptp4l0-21swapper/123:38:141
275991120,0ptp4l0-21swapper/121:55:181
275991120,0ptp4l0-21swapper/121:24:111
275991120,0ptp4l0-21swapper/120:55:191
1043399122,7cyclictest0-21swapper/900:25:2511
1043399122,6cyclictest0-21swapper/923:40:0011
1042999121,6cyclictest0-21swapper/823:05:0110
1042499122,7cyclictest0-21swapper/721:25:019
1041599122,6cyclictest0-21swapper/523:45:017
1041599122,6cyclictest0-21swapper/521:24:097
1041599122,6cyclictest0-21swapper/520:15:237
1041599122,6cyclictest0-21swapper/520:00:007
1041599121,6cyclictest0-21swapper/500:35:017
1040999120,11cyclictest0-21swapper/423:00:136
1040499122,6cyclictest0-21swapper/321:30:005
1040499122,6cyclictest0-21swapper/321:15:015
278952110,0sleep90-21swapper/920:55:1411
275991110,4ptp4l21-21ksoftirqd/119:30:121
275991110,2ptp4l30812-21/usr/sbin/munin00:30:181
275991110,1ptp4l21-21ksoftirqd/122:05:201
275991110,1ptp4l21-21ksoftirqd/120:25:161
275991110,1ptp4l21-21ksoftirqd/120:20:021
275991110,0ptp4l0-21swapper/121:48:261
275991110,0ptp4l0-21swapper/120:45:161
275991110,0ptp4l0-21swapper/100:15:181
275991110,0ptp4l0-21swapper/100:00:221
1044299112,5cyclictest0-21swapper/1123:55:003
1044299110,5cyclictest0-21swapper/1122:09:553
10438991110,1cyclictest0-21swapper/1023:45:162
10438991110,1cyclictest0-21swapper/1023:05:172
10438991110,1cyclictest0-21swapper/1022:40:192
10438991110,1cyclictest0-21swapper/1022:30:152
10438991110,1cyclictest0-21swapper/1021:59:242
1043899110,1cyclictest18138-21irqrtprio22:05:192
1043899110,1cyclictest0-21swapper/1023:50:132
1043899110,1cyclictest0-21swapper/1022:04:052
1043899110,1cyclictest0-21swapper/1021:25:152
1043899110,11cyclictest0-21swapper/1022:15:162
1043399110,10cyclictest8732-21sed21:10:1911
1042499111,6cyclictest0-21swapper/723:35:009
1042499110,10cyclictest27579-21munin-run22:19:599
10409991110,1cyclictest0-21swapper/422:06:046
1040999110,1cyclictest0-21swapper/423:13:226
1040999110,1cyclictest0-21swapper/422:28:446
1040999110,1cyclictest0-21swapper/421:34:166
1040999110,10cyclictest0-21swapper/423:52:076
1040999110,10cyclictest0-21swapper/423:20:376
1040999110,10cyclictest0-21swapper/422:50:436
1040999110,10cyclictest0-21swapper/421:56:256
1040999110,10cyclictest0-21swapper/421:22:126
1040499117,3cyclictest31915-21expr21:00:145
1040499112,5cyclictest0-21swapper/319:20:205
1040499110,1cyclictest25914-21cpuspeed_turbos20:10:155
1039999110,10cyclictest0-21swapper/223:24:194
1039599110,10cyclictest6546-21df_inode23:15:121
78912101,6sleep90-21swapper/919:05:1611
275991100,7ptp4l0-21swapper/123:25:161
275991100,4ptp4l21-21ksoftirqd/120:17:181
275991100,1ptp4l31343-21perl23:05:161
275991100,1ptp4l2548-21cpuspeed_turbos00:35:141
275991100,1ptp4l0-21swapper/100:25:001
275991100,0ptp4l0-21swapper/123:30:161
275991100,0ptp4l0-21swapper/122:10:161
275991100,0ptp4l0-21swapper/121:05:171
275991100,0ptp4l0-21swapper/119:35:161
275991100,0ptp4l0-21swapper/119:20:211
275991100,0ptp4l0-21swapper/119:10:131
275991100,0ptp4l0-21swapper/119:10:121
1044299100,3cyclictest18042-21runrttasks22:46:093
1043899109,1cyclictest0-21swapper/1023:55:032
1043899109,1cyclictest0-21swapper/1023:42:412
1043899109,1cyclictest0-21swapper/1023:25:282
1043899109,1cyclictest0-21swapper/1023:10:292
1043899109,1cyclictest0-21swapper/1023:00:212
1043899109,1cyclictest0-21swapper/1022:55:122
1043899109,1cyclictest0-21swapper/1022:21:492
1043899109,1cyclictest0-21swapper/1022:11:442
1043899109,1cyclictest0-21swapper/1021:45:242
1043899109,1cyclictest0-21swapper/1021:22:512
1043899109,1cyclictest0-21swapper/1000:35:052
1043899109,1cyclictest0-21swapper/1000:25:282
1043899109,1cyclictest0-21swapper/1000:21:002
1043899109,1cyclictest0-21swapper/1000:15:172
1043899100,1cyclictest0-21swapper/1023:35:292
1043899100,1cyclictest0-21swapper/1022:25:582
1043899100,1cyclictest0-21swapper/1021:30:112
1043899100,10cyclictest0-21swapper/1023:30:392
1043899100,10cyclictest0-21swapper/1023:20:142
1043899100,10cyclictest0-21swapper/1023:15:292
1043899100,10cyclictest0-21swapper/1022:50:352
1043899100,10cyclictest0-21swapper/1022:45:272
1043899100,10cyclictest0-21swapper/1022:35:012
1043899100,10cyclictest0-21swapper/1021:52:212
1043899100,10cyclictest0-21swapper/1021:40:442
1043899100,10cyclictest0-21swapper/1021:35:482
1043899100,10cyclictest0-21swapper/1000:31:002
1043899100,10cyclictest0-21swapper/1000:05:522
1043899100,10cyclictest0-21swapper/1000:00:542
1043399102,6cyclictest0-21swapper/923:30:2611
1043399100,9cyclictest30340-21awk19:35:0011
1042999100,9cyclictest0-21swapper/820:00:0010
1042999100,5cyclictest0-21swapper/820:45:0110
1042999100,5cyclictest0-21swapper/819:30:0010
1042499100,9cyclictest2936-21grep23:10:169
1042499100,9cyclictest0-21swapper/720:45:009
1040999109,1cyclictest0-21swapper/423:55:586
1040999109,1cyclictest0-21swapper/422:55:376
1040999109,1cyclictest0-21swapper/422:40:486
1040999109,1cyclictest0-21swapper/422:00:476
1040999109,1cyclictest0-21swapper/421:50:136
1040999109,1cyclictest0-21swapper/421:25:306
1040999109,1cyclictest0-21swapper/400:32:236
1040999109,1cyclictest0-21swapper/400:05:146
1040999100,9cyclictest0-21swapper/419:35:016
1040999100,1cyclictest0-21swapper/423:45:406
1040999100,1cyclictest0-21swapper/422:10:066
1040999100,1cyclictest0-21swapper/421:45:316
1040999100,1cyclictest0-21swapper/421:42:226
1040999100,1cyclictest0-21swapper/400:35:366
1040999100,1cyclictest0-21swapper/400:15:106
1040999100,10cyclictest0-21swapper/423:40:116
1040999100,10cyclictest0-21swapper/423:35:166
1040999100,10cyclictest0-21swapper/423:30:036
1040999100,10cyclictest0-21swapper/423:26:356
1040999100,10cyclictest0-21swapper/423:17:456
1040999100,10cyclictest0-21swapper/423:05:216
1040999100,10cyclictest0-21swapper/422:45:346
1040999100,10cyclictest0-21swapper/422:35:096
1040999100,10cyclictest0-21swapper/422:30:546
1040999100,10cyclictest0-21swapper/422:20:576
1040999100,10cyclictest0-21swapper/422:16:026
1040999100,10cyclictest0-21swapper/421:36:336
1040999100,10cyclictest0-21swapper/400:26:566
1040999100,10cyclictest0-21swapper/400:21:136
1040999100,10cyclictest0-21swapper/400:11:206
1040999100,10cyclictest0-21swapper/400:00:476
1040499100,1cyclictest0-21swapper/322:11:135
27599190,1ptp4l21-21ksoftirqd/123:10:131
27599190,1ptp4l21-21ksoftirqd/122:35:131
27599190,1ptp4l21-21ksoftirqd/122:20:111
27599190,1ptp4l21-21ksoftirqd/122:00:151
27599190,1ptp4l21-21ksoftirqd/121:40:151
27599190,1ptp4l21-21ksoftirqd/121:30:151
27599190,1ptp4l21-21ksoftirqd/120:50:121
27599190,1ptp4l21-21ksoftirqd/120:35:101
27599190,1ptp4l21-21ksoftirqd/100:10:181
27599190,1ptp4l21-21ksoftirqd/100:05:181
27599190,1ptp4l15748-21expr22:45:141
27599190,1ptp4l0-21swapper/100:25:131
27599190,0ptp4l0-21swapper/122:30:001
27599190,0ptp4l0-21swapper/121:00:151
27599190,0ptp4l0-21swapper/119:55:141
27599190,0ptp4l0-21swapper/119:45:141
18194290,0sleep00-21swapper/020:00:120
104429991,6cyclictest5984-21kworker/u24:019:15:003
104429991,6cyclictest5984-21kworker/u24:019:15:003
104429991,5cyclictest0-21swapper/1123:25:013
104429991,5cyclictest0-21swapper/1123:20:013
104339997,1cyclictest0-21swapper/919:45:2111
104339990,8cyclictest9679-21gunzip22:35:2211
104339990,8cyclictest0-21swapper/919:55:0011
104299996,2cyclictest32497-21idleruntime-cro20:20:0010
104299990,8cyclictest0-21swapper/822:10:0010
104299990,8cyclictest0-21swapper/821:45:2310
104299990,8cyclictest0-21swapper/800:00:2210
104249991,5cyclictest0-21swapper/720:20:229
104249991,5cyclictest0-21swapper/719:45:019
104249990,9cyclictest0-21swapper/719:45:019
104249990,8cyclictest22667-21awk00:19:599
104249990,8cyclictest0-21swapper/722:14:559
104209998,1cyclictest51-21ksoftirqd/619:40:228
104159997,1cyclictest0-21swapper/522:24:597
104159990,8cyclictest30991-21smartctl21:40:167
104099990,8cyclictest0-21swapper/421:05:166
104049998,1cyclictest0-21swapper/323:55:165
104049990,9cyclictest0-21swapper/322:40:195
104049990,9cyclictest0-21swapper/320:45:135
104049990,2cyclictest5845-21fw_forwarded_lo20:25:165
104049990,1cyclictest0-21swapper/322:05:185
104049990,1cyclictest0-21swapper/300:10:185
103999998,1cyclictest0-21swapper/223:34:594
103999990,9cyclictest0-21swapper/221:25:194
103999990,9cyclictest0-21swapper/220:20:214
103999990,8cyclictest24854-21perl20:50:124
103999990,8cyclictest11301-21/usr/sbin/munin19:50:134
103999990,8cyclictest0-21swapper/221:05:144
103999990,3cyclictest32523-21/usr/sbin/munin21:00:164
103999990,3cyclictest14579-21/usr/sbin/munin20:35:194
103959990,1cyclictest0-21swapper/119:40:121
103909998,1cyclictest0-21swapper/021:55:270
103909996,2cyclictest16380-21awk20:40:120
103909990,9cyclictest0-21swapper/023:40:170
103909990,9cyclictest0-21swapper/022:45:160
103909990,9cyclictest0-21swapper/021:30:130
103909990,1cyclictest0-21swapper/023:10:130
9725281,5sleep30-21swapper/319:05:335
104429982,4cyclictest0-21swapper/1123:35:003
104429980,1cyclictest0-21swapper/1121:15:153
104389985,2cyclictest30336-21turbostat19:35:002
104389980,8cyclictest0-21swapper/1019:55:162
104389980,8cyclictest0-21swapper/1019:45:182
104339982,5cyclictest0-21swapper/921:05:2511
104339981,7cyclictest69-21ksoftirqd/922:04:5711
104339980,8cyclictest0-21swapper/922:34:5911
104339980,1cyclictest0-21swapper/923:45:1811
104339980,1cyclictest0-21swapper/923:05:2611
104339980,1cyclictest0-21swapper/921:40:1511
104339980,1cyclictest0-21swapper/920:40:1711
104299982,5cyclictest0-21swapper/800:40:0010
104299980,7cyclictest0-21swapper/800:15:4610
104299980,1cyclictest0-21swapper/823:40:0010
104299980,1cyclictest0-21swapper/822:25:0010
104299980,1cyclictest0-21swapper/821:35:2110
104249987,1cyclictest0-21swapper/723:25:199
104249982,3cyclictest0-21swapper/720:10:019
104249981,5cyclictest0-21swapper/722:30:009
104249980,8cyclictest0-21swapper/723:15:179
104249980,8cyclictest0-21swapper/721:00:139
104249980,8cyclictest0-21swapper/719:35:179
104249980,8cyclictest0-21swapper/700:05:229
104209987,1cyclictest0-21swapper/622:25:148
104209987,1cyclictest0-21swapper/622:15:158
104209987,1cyclictest0-21swapper/619:15:188
104209987,1cyclictest0-21swapper/619:15:188
104209986,1cyclictest6341-21diskmemload22:11:528
104209986,1cyclictest0-21swapper/619:10:218
104209986,1cyclictest0-21swapper/619:10:218
104209985,2cyclictest9716-21cpuspeed23:20:178
104209980,1cyclictest0-21swapper/620:55:168
104209980,1cyclictest0-21swapper/620:13:298
104159987,1cyclictest0-21swapper/521:30:227
104159987,1cyclictest0-21swapper/500:05:237
104159986,1cyclictest0-21swapper/521:00:217
104159980,1cyclictest0-21swapper/523:50:147
104159980,1cyclictest0-21swapper/523:25:167
104159980,1cyclictest0-21swapper/519:35:237
104099980,3cyclictest26836-21cat19:30:016
104099980,3cyclictest22962-21cat19:25:006
104049987,1cyclictest0-21swapper/323:05:215
104049987,1cyclictest0-21swapper/321:50:235
104049987,1cyclictest0-21swapper/319:55:145
104049981,4cyclictest0-21swapper/320:10:005
104049980,8cyclictest0-21swapper/322:20:135
104049980,1cyclictest0-21swapper/323:35:235
104049980,1cyclictest0-21swapper/300:27:205
103999987,1cyclictest0-21swapper/219:20:194
103999987,1cyclictest0-21swapper/200:30:234
103999986,1cyclictest4171-21turbostat20:29:554
103999980,8cyclictest0-21swapper/223:10:184
103999980,7cyclictest0-21swapper/223:25:184
103999980,7cyclictest0-21swapper/223:00:134
103999980,2cyclictest10870-21/usr/sbin/munin21:55:234
103999980,1cyclictest0-21swapper/222:50:134
103999980,1cyclictest0-21swapper/222:20:164
103999980,1cyclictest0-21swapper/221:50:204
103999980,1cyclictest0-21swapper/220:45:264
103999980,1cyclictest0-21swapper/219:30:174
103909987,1cyclictest0-21swapper/022:50:170
103909987,1cyclictest0-21swapper/022:35:190
103909987,1cyclictest0-21swapper/021:10:150
103909987,1cyclictest0-21swapper/020:10:180
103909985,2cyclictest1007-21timerandwakeup19:35:240
103909984,3cyclictest17390-21ls20:00:000
103909980,8cyclictest0-21swapper/023:45:220
103909980,8cyclictest0-21swapper/022:00:230
103909980,8cyclictest0-21swapper/020:50:090
103909980,7cyclictest0-21swapper/021:37:430
104429976,1cyclictest0-21swapper/1122:50:153
104429976,1cyclictest0-21swapper/1122:13:353
104429976,1cyclictest0-21swapper/1122:01:123
104429976,1cyclictest0-21swapper/1121:42:303
104429976,1cyclictest0-21swapper/1121:22:243
104429976,1cyclictest0-21swapper/1121:10:083
104429976,1cyclictest0-21swapper/1120:01:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional