You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 12:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Sat Jul 12, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
949801050,90rtkit-daemon0-21swapper/1019:08:242
275991900,80ptp4l0-21swapper/119:09:581
1885827058,9sleep110-21swapper/1119:05:233
1957026654,10sleep50-21swapper/519:06:057
1955226454,8sleep00-21swapper/019:05:480
1955626353,8sleep40-21swapper/419:05:526
1967526153,6sleep70-21swapper/719:07:319
1961226052,6sleep60-21swapper/619:06:428
10682590,0sleep30-21swapper/320:05:205
10682590,0sleep30-21swapper/320:05:205
302442570,0sleep80-21swapper/823:30:2410
19012540,0sleep50-21swapper/522:15:017
2025899360,1cyclictest0-21swapper/1020:00:242
2023199211,7cyclictest0-21swapper/420:40:016
2022699204,9cyclictest0-21swapper/321:00:015
2023599181,3cyclictest0-21swapper/519:40:217
275991170,1ptp4l21-21ksoftirqd/120:10:011
275991170,1ptp4l21-21ksoftirqd/120:10:011
2025399173,8cyclictest0-21swapper/921:20:0111
2022699175,8cyclictest17241-21kworker/u24:100:10:005
2022699173,8cyclictest0-21swapper/300:20:005
275991160,1ptp4l21-21ksoftirqd/120:15:181
275991160,1ptp4l21-21ksoftirqd/120:15:181
275991160,1ptp4l21-21ksoftirqd/119:15:231
275991160,0ptp4l0-21swapper/120:25:181
275991160,0ptp4l0-21swapper/120:25:181
2026299162,8cyclictest0-21swapper/1121:30:013
2025399163,8cyclictest0-21swapper/923:45:0011
2025399162,9cyclictest0-21swapper/921:25:0111
2022699164,7cyclictest0-21swapper/319:30:005
20226991612,3cyclictest0-21swapper/322:00:205
20214991614,1cyclictest17032-21turbostat22:39:551
196482167,7sleep80-21swapper/819:07:0810
275991150,0ptp4l0-21swapper/123:40:131
275991150,0ptp4l0-21swapper/122:20:201
275991150,0ptp4l0-21swapper/121:50:161
275991150,0ptp4l0-21swapper/120:50:151
275991150,0ptp4l0-21swapper/119:25:231
2025399153,7cyclictest0-21swapper/923:25:2311
2025399153,7cyclictest0-21swapper/922:50:0111
2023199153,7cyclictest0-21swapper/423:40:236
2022699153,7cyclictest0-21swapper/320:35:015
2022699152,7cyclictest0-21swapper/322:15:015
154782150,0sleep90-21swapper/922:30:1811
89092140,0sleep110-21swapper/1123:05:123
275991140,1ptp4l21-21ksoftirqd/119:10:181
275991140,0ptp4l0-21swapper/120:00:251
2026299143,7cyclictest0-21swapper/1121:45:243
2026299142,7cyclictest0-21swapper/1123:40:003
2025399143,7cyclictest0-21swapper/920:10:0011
2025399143,7cyclictest0-21swapper/920:10:0011
2025399142,8cyclictest0-21swapper/900:15:0011
2025399142,7cyclictest0-21swapper/919:40:0111
2024099142,8cyclictest0-21swapper/622:45:018
2024099140,8cyclictest0-21swapper/621:30:018
2023599143,7cyclictest0-21swapper/521:40:017
2023599142,7cyclictest0-21swapper/519:20:007
2023599142,7cyclictest0-21swapper/500:30:007
2022699143,6cyclictest0-21swapper/322:10:015
2022699143,6cyclictest0-21swapper/319:50:015
2022699142,7cyclictest0-21swapper/300:05:005
2020999142,8cyclictest0-21swapper/022:40:270
2020999142,8cyclictest0-21swapper/020:00:250
275991130,1ptp4l21-21ksoftirqd/122:57:391
275991130,1ptp4l21-21ksoftirqd/122:02:331
275991130,1ptp4l21-21ksoftirqd/121:08:511
275991130,1ptp4l21-21ksoftirqd/120:46:011
275991130,1ptp4l21-21ksoftirqd/119:36:171
275991130,0ptp4l0-21swapper/100:35:221
2026299133,6cyclictest0-21swapper/1119:40:013
2025899132,6cyclictest0-21swapper/1020:05:252
2025899132,6cyclictest0-21swapper/1020:05:252
2025899130,6cyclictest11639-21munin-run23:10:012
2025899130,12cyclictest0-21swapper/1022:20:002
2025399132,7cyclictest0-21swapper/923:00:0111
2025399132,6cyclictest0-21swapper/923:05:2311
2025399131,8cyclictest0-21swapper/923:10:2711
2023599132,7cyclictest0-21swapper/521:32:177
2023599132,7cyclictest0-21swapper/521:20:017
2023599132,7cyclictest0-21swapper/520:10:017
2023599132,7cyclictest0-21swapper/520:10:017
2023199132,7cyclictest0-21swapper/423:00:246
2023199130,9cyclictest0-21swapper/422:35:256
2022699132,7cyclictest0-21swapper/322:55:015
2020999132,8cyclictest0-21swapper/021:55:010
2020999132,8cyclictest0-21swapper/021:50:010
2020999132,8cyclictest0-21swapper/021:10:010
2020999132,7cyclictest0-21swapper/022:15:010
2020999130,9cyclictest0-21swapper/022:55:250
275991120,2ptp4l1974-21df00:20:131
275991120,1ptp4l21-21ksoftirqd/120:12:071
275991120,1ptp4l21-21ksoftirqd/120:12:071
275991120,0ptp4l0-21swapper/123:23:061
275991120,0ptp4l0-21swapper/121:20:041
2026299120,7cyclictest0-21swapper/1122:20:003
2025399129,2cyclictest20182-21turbostat21:15:0011
2025399121,7cyclictest0-21swapper/922:20:0111
2025399120,11cyclictest0-21swapper/922:44:5611
2024099122,6cyclictest0-21swapper/600:10:258
2023599122,6cyclictest0-21swapper/522:45:017
2023199121,7cyclictest0-21swapper/423:45:236
2022699123,6cyclictest0-21swapper/322:20:015
2021499123,3cyclictest12198-21if_err_enp1s020:20:191
2021499123,3cyclictest12198-21if_err_enp1s020:20:191
2020999122,8cyclictest0-21swapper/021:30:010
2020999121,7cyclictest0-21swapper/023:55:240
275991110,2ptp4l8811-21fw_conntrack21:40:161
275991110,2ptp4l8474-21cpu23:05:121
275991110,2ptp4l7594-21unixbench-2d19:30:241
275991110,2ptp4l30250-21tune2fs19:20:131
275991110,2ptp4l25246-21users20:35:261
275991110,2ptp4l12967-21diskstats23:10:171
275991110,2ptp4l12228-21cstates21:45:131
275991110,2ptp4l11376-21diskstats23:50:161
275991110,1ptp4l21-21ksoftirqd/100:06:111
275991110,0ptp4l0-21swapper/122:10:161
275991110,0ptp4l0-21swapper/121:31:541
2026299110,10cyclictest0-21swapper/1123:59:553
2026299110,10cyclictest0-21swapper/1100:15:153
2025899110,6cyclictest0-21swapper/1020:25:242
2025899110,6cyclictest0-21swapper/1020:25:242
2025399110,10cyclictest4325-21fw_conntrack20:10:1511
2025399110,10cyclictest4325-21fw_conntrack20:10:1511
2024999119,1cyclictest0-21swapper/800:23:5610
2024499119,1cyclictest0-21swapper/721:50:009
2024099111,6cyclictest0-21swapper/622:15:018
2024099110,7cyclictest0-21swapper/620:09:558
2024099110,7cyclictest0-21swapper/620:09:558
2023599112,5cyclictest0-21swapper/523:35:007
2023599110,7cyclictest0-21swapper/523:15:017
2022199110,2cyclictest0-21swapper/222:15:014
2021499114,6cyclictest12259-21switchtime22:25:231
2021499110,3cyclictest19371-21cut23:20:001
2021499110,10cyclictest16868-21expr19:45:161
2020999113,7cyclictest0-21swapper/019:20:000
275991100,3ptp4l18310-21if_p34p121:10:181
275991100,2ptp4l10984-21kernelversion21:00:201
275991100,1ptp4l26243-21perl22:45:151
275991100,1ptp4l26198-21cpuspeed_turbos00:10:141
275991100,1ptp4l21629-21sed19:50:191
275991100,1ptp4l19919-21df21:55:181
275991100,1ptp4l16356-21cat23:55:201
2026299108,1cyclictest0-21swapper/1100:00:193
2026299102,6cyclictest0-21swapper/1121:45:013
2026299100,9cyclictest24221-21missed_timers20:35:203
2025899100,9cyclictest23669-21users21:15:262
2025899100,1cyclictest0-21swapper/1023:35:192
2025399100,9cyclictest0-21swapper/919:10:1311
2025399100,2cyclictest7381-21proc_pri00:25:2111
2025399100,1cyclictest0-21swapper/923:00:2411
2025399100,10cyclictest0-21swapper/919:40:2011
2024999106,3cyclictest13758-21sed23:10:2110
2024499107,2cyclictest0-21swapper/722:55:139
2024499107,2cyclictest0-21swapper/719:31:539
2024499107,2cyclictest0-21swapper/719:20:009
2024099102,5cyclictest0-21swapper/600:05:008
2024099101,6cyclictest0-21swapper/621:25:018
2024099100,1cyclictest7467-21timerwakeupswit19:30:248
2023599108,1cyclictest0-21swapper/523:20:157
2023599107,2cyclictest15274-21latency_hist21:50:017
2023599103,5cyclictest0-21swapper/500:30:267
2022699100,9cyclictest25915-21needreboot19:55:225
2022199107,2cyclictest0-21swapper/219:55:174
2022199100,9cyclictest0-21swapper/220:35:154
2021499104,1cyclictest5456-21cpuspeed_turbos00:25:141
2021499102,3cyclictest5162-21cat23:00:131
2021499102,2cyclictest31844-21cat21:30:011
2021499102,2cyclictest24268-21expr19:55:161
2021499101,3cyclictest27478-21cat20:40:191
2021499100,3cyclictest9205-21date21:00:011
2021499100,3cyclictest8206-21cat23:45:161
2021499100,3cyclictest4644-21df21:35:161
2021499100,3cyclictest3243-21fw_forwarded_lo22:15:181
2021499100,3cyclictest31829-21nfs4_client00:15:221
2021499100,3cyclictest31415-21cpu23:35:171
2021499100,3cyclictest29824-21dnf22:50:171
2021499100,3cyclictest28566-21cat22:05:191
2021499100,3cyclictest27561-21grep23:30:121
2021499100,3cyclictest21373-21cstates21:15:141
2021499100,3cyclictest18720-21cat20:30:121
2021499100,3cyclictest18720-21cat20:30:121
2021499100,3cyclictest15587-21runrttasks19:43:391
2021499100,3cyclictest15061-21if_enp1s0.122:30:171
2021499100,3cyclictest11311-21sendmail_mailqu00:30:231
2020999102,7cyclictest0-21swapper/020:40:260
2020999100,2cyclictest0-21swapper/021:25:160
202589998,1cyclictest0-21swapper/1019:20:282
202539998,1cyclictest0-21swapper/921:25:1711
202539991,5cyclictest0-21swapper/922:15:0111
202539990,9cyclictest0-21swapper/921:00:1611
202539990,9cyclictest0-21swapper/920:30:0011
202539990,9cyclictest0-21swapper/920:30:0011
202539990,8cyclictest0-21swapper/922:54:5611
202539990,1cyclictest0-21swapper/919:30:1311
202539990,1cyclictest0-21swapper/900:15:1511
202499990,9cyclictest0-21swapper/820:55:2310
202499990,8cyclictest0-21swapper/822:45:2510
202499990,8cyclictest0-21swapper/819:30:2410
202449998,1cyclictest0-21swapper/719:55:029
202449990,8cyclictest0-21swapper/722:25:179
202449990,8cyclictest0-21swapper/719:45:189
202449990,3cyclictest0-21swapper/722:24:569
202409998,1cyclictest0-21swapper/622:45:158
202409998,1cyclictest0-21swapper/600:15:268
202409992,5cyclictest0-21swapper/620:10:248
202409992,5cyclictest0-21swapper/620:10:248
202409991,5cyclictest0-21swapper/619:25:008
202409990,1cyclictest0-21swapper/621:15:198
202359998,1cyclictest0-21swapper/521:55:227
202269996,2cyclictest5896-21users23:40:245
202269992,5cyclictest0-21swapper/300:40:005
202269990,8cyclictest0-21swapper/322:46:085
202219992,4cyclictest0-21swapper/223:43:194
202219990,8cyclictest0-21swapper/223:10:204
202219990,1cyclictest0-21swapper/223:25:144
202149990,3cyclictest23738-21cpu23:25:111
202149990,3cyclictest21622-21ls22:40:151
202149990,3cyclictest18486-21grep00:00:131
202099992,4cyclictest0-21swapper/021:00:160
202099990,1cyclictest0-21swapper/019:10:140
202629987,1cyclictest0-21swapper/1123:50:163
202629987,1cyclictest0-21swapper/1123:45:143
202629987,1cyclictest0-21swapper/1121:10:163
202629987,1cyclictest0-21swapper/1120:30:173
202629987,1cyclictest0-21swapper/1120:30:173
202629987,1cyclictest0-21swapper/1119:10:133
202629980,8cyclictest0-21swapper/1121:20:223
202629980,1cyclictest0-21swapper/1123:25:163
202589987,1cyclictest0-21swapper/1023:40:202
202589987,1cyclictest0-21swapper/1023:30:192
202589987,1cyclictest0-21swapper/1020:35:152
202589987,1cyclictest0-21swapper/1019:50:132
202589987,1cyclictest0-21swapper/1019:10:232
202589980,8cyclictest0-21swapper/1023:15:192
202589980,8cyclictest0-21swapper/1022:10:162
202589980,8cyclictest0-21swapper/1021:10:232
202589980,8cyclictest0-21swapper/1000:10:192
202589980,1cyclictest0-21swapper/1021:20:162
202539987,1cyclictest0-21swapper/921:30:1511
202539987,1cyclictest0-21swapper/919:50:2211
202539987,1cyclictest0-21swapper/919:45:1511
202539987,1cyclictest0-21swapper/900:20:1411
202539980,8cyclictest0-21swapper/920:30:2411
202539980,8cyclictest0-21swapper/920:30:2411
202539980,8cyclictest0-21swapper/900:35:1311
202539980,1cyclictest0-21swapper/923:35:2911
202539980,1cyclictest0-21swapper/922:20:2811
202539980,1cyclictest0-21swapper/922:00:2111
202539980,1cyclictest0-21swapper/921:45:2011
202539980,1cyclictest0-21swapper/900:05:1611
202499987,1cyclictest0-21swapper/823:55:1910
202499987,1cyclictest0-21swapper/820:25:1310
202499987,1cyclictest0-21swapper/820:25:1310
202499980,8cyclictest0-21swapper/823:45:2210
202499980,1cyclictest0-21swapper/823:30:0010
202499980,1cyclictest0-21swapper/820:45:2610
202499980,1cyclictest0-21swapper/820:15:1510
202499980,1cyclictest0-21swapper/820:15:1510
202499980,1cyclictest0-21swapper/800:05:1310
202449987,1cyclictest0-21swapper/723:55:139
202449987,1cyclictest0-21swapper/723:50:279
202449987,1cyclictest0-21swapper/722:50:169
202449987,1cyclictest0-21swapper/720:55:219
202449987,1cyclictest0-21swapper/719:55:169
202449987,1cyclictest0-21swapper/700:00:239
202449986,1cyclictest16357-21diskmemload21:20:469
202449980,8cyclictest0-21swapper/723:20:229
202449980,8cyclictest0-21swapper/723:15:259
202409987,1cyclictest0-21swapper/621:45:138
202409987,1cyclictest0-21swapper/620:50:148
202409987,1cyclictest0-21swapper/620:45:198
202409986,1cyclictest91172sleep622:21:148
202409985,2cyclictest10290-21turbostat20:20:008
202409985,2cyclictest10290-21turbostat20:20:008
202409981,5cyclictest0-21swapper/621:15:018
202409981,4cyclictest0-21swapper/623:25:008
202409980,8cyclictest0-21swapper/622:55:268
202409980,8cyclictest0-21swapper/619:55:168
202409980,8cyclictest0-21swapper/619:10:188
202409980,5cyclictest0-21swapper/600:32:328
202409980,1cyclictest0-21swapper/622:05:138
202409980,1cyclictest0-21swapper/620:42:248
202409980,1cyclictest0-21swapper/619:25:208
202359987,1cyclictest441rcuc/522:05:147
202359987,1cyclictest441rcuc/519:20:197
202359985,2cyclictest2845-21netstat19:25:207
202359980,4cyclictest45-21ksoftirqd/500:00:007
202359980,1cyclictest0-21swapper/523:55:007
202359980,1cyclictest0-21swapper/522:55:017
202359980,1cyclictest0-21swapper/521:50:247
202359980,1cyclictest0-21swapper/521:25:237
202359980,1cyclictest0-21swapper/520:00:207
202319980,7cyclictest0-21swapper/420:45:136
202319980,1cyclictest0-21swapper/422:25:196
202269987,1cyclictest0-21swapper/323:55:155
202269987,1cyclictest0-21swapper/300:15:005
202269981,6cyclictest0-21swapper/323:35:005
202269981,6cyclictest0-21swapper/319:30:015
202269980,8cyclictest0-21swapper/323:15:145
202219985,2cyclictest0-21swapper/219:25:204
202219982,1cyclictest27-21ksoftirqd/221:22:044
202219980,8cyclictest0-21swapper/221:55:194
202219980,8cyclictest0-21swapper/221:35:174
202219980,8cyclictest0-21swapper/219:35:144
202219980,8cyclictest0-21swapper/200:05:234
202219980,7cyclictest0-21swapper/222:55:184
202219980,7cyclictest0-21swapper/222:00:324
202219980,7cyclictest0-21swapper/221:45:154
202219980,7cyclictest0-21swapper/219:45:184
202219980,7cyclictest0-21swapper/200:35:194
202219980,2cyclictest16689-21/usr/sbin/munin22:30:264
202219980,2cyclictest0-21swapper/223:45:234
202219980,2cyclictest0-21swapper/223:00:114
202219980,2cyclictest0-21swapper/221:16:334
202219980,2cyclictest0-21swapper/221:10:254
202219980,2cyclictest0-21swapper/220:31:294
202219980,2cyclictest0-21swapper/220:31:294
202219980,2cyclictest0-21swapper/219:15:164
202219980,1cyclictest0-21swapper/223:20:134
202219980,1cyclictest0-21swapper/222:40:154
202219980,1cyclictest0-21swapper/220:25:174
202219980,1cyclictest0-21swapper/220:25:174
202219980,1cyclictest0-21swapper/219:40:194
202219980,1cyclictest0-21swapper/200:10:184
202099986,1cyclictest16357-21diskmemload23:02:240
202099986,1cyclictest0-21swapper/022:20:150
202099980,1cyclictest8146-21sed19:35:000
202099980,1cyclictest0-21swapper/022:35:130
19239280,0sleep90-21swapper/922:35:2011
17351282,4sleep30-21swapper/319:05:155
16547281,5sleep90-21swapper/919:05:0811
26223270,0sleep545-21ksoftirqd/500:10:157
202629976,1cyclictest0-21swapper/1123:16:153
202629976,1cyclictest0-21swapper/1122:30:393
202629976,1cyclictest0-21swapper/1122:05:393
202629976,1cyclictest0-21swapper/1122:01:263
202629976,1cyclictest0-21swapper/1121:55:173
202629976,1cyclictest0-21swapper/1121:51:343
202629976,1cyclictest0-21swapper/1121:35:493
202629976,1cyclictest0-21swapper/1121:30:353
202629976,1cyclictest0-21swapper/1120:45:323
202629976,1cyclictest0-21swapper/1120:25:463
202629976,1cyclictest0-21swapper/1120:25:463
202629976,1cyclictest0-21swapper/1120:16:583
202629976,1cyclictest0-21swapper/1120:16:583
202629976,1cyclictest0-21swapper/1120:05:203
202629976,1cyclictest0-21swapper/1120:05:203
202629976,1cyclictest0-21swapper/1120:01:593
202629976,1cyclictest0-21swapper/1119:56:243
202629976,1cyclictest0-21swapper/1119:31:183
202629976,1cyclictest0-21swapper/1100:30:113
202629976,1cyclictest0-21swapper/1100:20:063
202629974,2cyclictest24490-21switchtime00:05:233
202629973,3cyclictest32320-21cut22:10:193
202629970,7cyclictest0-21swapper/1123:40:583
202629970,7cyclictest0-21swapper/1123:12:123
202629970,7cyclictest0-21swapper/1123:01:243
202629970,7cyclictest0-21swapper/1122:56:423
202629970,7cyclictest0-21swapper/1122:51:383
202629970,7cyclictest0-21swapper/1122:45:483
202629970,7cyclictest0-21swapper/1122:37:243
202629970,7cyclictest0-21swapper/1121:05:163
202629970,7cyclictest0-21swapper/1120:55:583
202629970,7cyclictest0-21swapper/1120:52:123
202629970,7cyclictest0-21swapper/1120:20:063
202629970,7cyclictest0-21swapper/1120:20:063
202629970,7cyclictest0-21swapper/1120:10:183
202629970,7cyclictest0-21swapper/1120:10:183
202629970,7cyclictest0-21swapper/1119:52:183
202629970,7cyclictest0-21swapper/1119:15:413
202629970,7cyclictest0-21swapper/1100:10:163
202629970,1cyclictest0-21swapper/1123:31:023
202629970,1cyclictest0-21swapper/1123:20:273
202629970,1cyclictest0-21swapper/1122:40:573
202629970,1cyclictest0-21swapper/1122:26:173
202629970,1cyclictest0-21swapper/1122:20:383
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional