Is there an example of this anywhere? This seems to be the only detail that might drastically improve our production performance.
Not quite clear, I think, because of the way that AbpStringLocalizerFactory itself gets "registered".
Yes, that's where I'm trying to get to - the problem is that I don't know what these implementations are actually doing, so I can't just go replacing them. I understand they are pro-related, which is why I can't find non-forked examples, but something seems to be unnecessarily delayed in production by that code.
Moving to 9.2.2 is a little better, but still confusing why there are positives on the list: https://www.virustotal.com/gui/file/6fd3da75a2e2ff69257f96ab3b02a9432697cff78caa319290b28accedffbf9a?nocache=1
I have done that, and it still happens. Since the prevalence of supply chain/dependency hijacking, I'm not sure that answer is sufficient as is especially with the result from VirusTotal: https://www.virustotal.com/gui/file/d190aea0d64b850d5dfe19b2786283ab7f7dc4b72e4f6c5e0d3ae58b51c5ac48/detection
It appears to be an issue with Aspire and ABP. Now that I'm pretty sure this is the issue, I'll be able to fix up my repro and send that to you.
Alternatively, your sample should have the same exact issue. The easiest way to see this is to have something in web call an app service where you either intentionally or unintentionally have code that takes long, like adding await Task.Delay(TimeSpan.FromSeconds(30)); - then you should see the exact same issue.
Once I understood where the Polly retry was being caused, I was able to modify the AddServiceDefaults() extension method like so:
public static TBuilder AddServiceDefaults<TBuilder>(this TBuilder builder) where TBuilder : IHostApplicationBuilder
{
builder.ConfigureOpenTelemetry();
builder.AddDefaultHealthChecks();
builder.Services.AddServiceDiscovery();
builder.Services.ConfigureHttpClientDefaults(http =>
{
// Turn on resilience by default
http.AddStandardResilienceHandler(resilience =>
{
// Set timeouts
resilience.AttemptTimeout.Timeout = TimeSpan.FromMinutes(2);
resilience.TotalRequestTimeout.Timeout = TimeSpan.FromMinutes(5); // Must be > attempt timeout
// Configure retry (either set to 1+ or disable the strategy)
resilience.Retry.MaxRetryAttempts = 1; // Minimum allowed value
// Configure circuit breaker
resilience.CircuitBreaker.SamplingDuration = TimeSpan.FromMinutes(5); // Must be ≥ 2× attempt timeout
});
// Turn on service discovery by default
http.AddServiceDiscovery();
});
// Uncomment the following to restrict the allowed schemes for service discovery.
// builder.Services.Configure<ServiceDiscoveryOptions>(options =>
// {
// options.AllowedSchemes = ["https"];
// });
return builder;
}
Specifically, the section AddStandardResilienceHandler is what needed to be modified.
Web Log: [2025-05-27 16:36:40.353] [Warning] () <CabMD.Web.Pages.Files.EdtIndexModel> === PAGE HANDLER START === [2025-05-27 16:36:40.354] [Warning] () <CabMD.Web.Pages.Files.EdtIndexModel> CorrelationId: "66a388fe", Time: 16:36:40.354 [2025-05-27 16:36:40.354] [Warning] () <CabMD.Web.Pages.Files.EdtIndexModel> Request: Page=1, PageSize=10, Skip=0 [2025-05-27 16:37:10.917] [Warning] () <CabMD.Web.Pages.Files.EdtIndexModel> === PAGE HANDLER END === [2025-05-27 16:37:10.917] [Warning] () <CabMD.Web.Pages.Files.EdtIndexModel> CorrelationId: "66a388fe", ResultCount: 10 [2025-05-27 16:37:10.935] [Warning] () <CabMD.PerformanceDiagnosticMiddleware> Slow request: "POST" "/Files/Edt" ("files/edt") took 30798ms (exceeds tolerable threshold of 500ms) [Status: 200]
API Log:
[2025-05-27 16:36:40.733] [Warning] () <CabMD.Controllers.Edt.EdtFileController> === API CONTROLLER START ===
[2025-05-27 16:36:40.733] [Warning] () <CabMD.Controllers.Edt.EdtFileController> RequestId: "6eca2ca5", Time: 16:36:40.733
[2025-05-27 16:36:40.734] [Warning] () <CabMD.Controllers.Edt.EdtFileController> HttpContext.TraceIdentifier: "0HNCTBOB4A9NI:00000026"
[2025-05-27 16:36:40.734] [Warning] () <CabMD.Controllers.Edt.EdtFileController> Input: Skip=0, Max=10, Sorting=null
[2025-05-27 16:36:40.736] [Warning] () <CabMD.Controllers.Edt.EdtFileController> Headers: "X-Requested-With=XMLHttpRequest"
[2025-05-27 16:36:40.745] [Warning] () <CabMD.Edt.EdtFileAppService> === APP SERVICE START ===
[2025-05-27 16:36:40.745] [Warning] () <CabMD.Edt.EdtFileAppService> RequestId: "dfc9379f", CorrelationId: "no-correlation", Time: 16:36:40.745
[2025-05-27 16:36:40.745] [Warning] () <CabMD.Edt.EdtFileAppService> Method: "CabMD.Edt.EdtFileAppService".GetListAsync, Hash: 42651975
[2025-05-27 16:36:40.745] [Warning] () <CabMD.Edt.EdtFileAppService> Is Proxy: False
[2025-05-27 16:36:40.745] [Warning] () <CabMD.Edt.EdtFileAppService> UOW ID: d451c0ca-00d6-4aeb-91d5-e175cf90ddf5, UOW IsActive: True
[2025-05-27 16:36:40.745] [Warning] () <CabMD.Edt.EdtFileAppService> Input: Skip=0, Max=10
[2025-05-27 16:36:40.749] [Warning] () <CabMD.Edt.EdtFileAppService> Call Stack: "<GetListAsync>d__13.MoveNext -> AsyncMethodBuilderCore.Start -> EdtFileAppService.GetListAsync -> IReadOnlyAppService4_GetListAsync_7.InvokeMethodOnTarget -> AbstractInvocation.Proceed -> ProceedInfo.Invoke -> <ProceedAsynchronous>d__141.MoveNext -> AsyncMethodBuilderCore.Start -> AsyncInterceptorBase.ProceedAsynchronous -> <ProceedAsync>d__7.MoveNext"
[2025-05-27 16:37:10.831] [Warning] () <CabMD.Controllers.Edt.EdtFileController> === API CONTROLLER END ===
[2025-05-27 16:37:10.832] [Warning] () <CabMD.Controllers.Edt.EdtFileController> RequestId: "6eca2ca5", ResultCount: 10
Generally, in the host apps - there is something similar to the following:
Log.Information("Starting web host.");
var builder = WebApplication.CreateBuilder(args);
builder.AddRedisClient("redis");
builder.AddRedisDistributedCache("redis");
builder.AddServiceDefaults(); // Bam, issue - if this is the default AddServiceDefaults and nothing modifies the resiliancy
I'm 99% certain it has nothing to do with the actual client control (kendo). It's making a single call to the API, then the app service portion is being repeatedly called.
I only see a single rest call to the page handler, which is pending the entire time that I get stopped multiple times at the app service layer.
Yep, I see it's up - thank you for the quick response.
Please don't close this ticket. It is unresolved. Any news?
I'll note, though, while I'm responding, that I see very similar caching issues on a completely separate project. It feels like a cache stampede issue causing a race condition on the lock internal to the implementation, which then causes the cache to need to be loaded again, etc.
I've been side tracked on the AbpCachingPlayground while we separated out the aspire k6 components (https://github.com/kfrancis/k6-aspire-hosting), but I'll be getting back to it shortly to test my hypothesis.
Just a heads up while you look into this, we are working on a bit of an "abp caching playground" to assist from our side: https://github.com/Clinical-Support-Systems/abp-caching-playground
It's meant to help us determine how changes in the caching implementation change the overall caching health/throughput, but also (hopefully) expose possible issues with the caching implementation.
Cool things:
