Hello everyone,
I'd like to ask for help debugging ASP.NET Core Web API.
That's the situation:
We've got an authorization service that exposes multiple RESTful APIs to check if authenticated user can access certain business objects. In my case the business is object is a concept of market and they are authorized by project. So API call would look as follows:
GET https://authorization.service.com/markets?project=123456
Successful response will return array of markets, unauthorized response will return a 403 status code.
It all works great when I call it through Swagger, call it from Postman - responses are quick and come back in tens of milliseconds.
Now my other service is calling this service in authorization filter and based on response either continues or responds with Forbidden.
I've implemented a reusable client for HTTP calls:
public class ApiClient : IApiClient
{
private readonly HttpClient _client;
private readonly JsonSerializer _jsonSerializer;
public ApiClient(HttpClient httpClient, JsonSerializer jsonSerializer)
{
_client = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
_jsonSerializer = jsonSerializer ?? throw new ArgumentNullException(nameof(jsonSerializer));
}
public async Task<T> GetDataAsync<T>(string uri, string accessToken)
where T : class
{
var request = CreateRequest(uri, accessToken);
var response = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
if (!response.IsSuccessStatusCode)
{
return default(T);
}
using (var responseStream = await response.Content.ReadAsStreamAsync())
using (var streamReader = new StreamReader(responseStream))
using (var jsonTextReader = new JsonTextReader(streamReader))
{
return _jsonSerializer.Deserialize<T>(jsonTextReader);
}
}
private HttpRequestMessage CreateRequest(string uri, string accessToken)
{
var locationUri = new Uri(uri);
var message = new HttpRequestMessage(HttpMethod.Get, locationUri);
message.Headers.Add("Authorization", $"Bearer {accessToken}");
return message;
}
}
My authorization filter looks as follows:
public class AuthorizeProjectFilter : IAsyncActionFilter
{
private readonly IApiClient _apiClient;
private readonly IMarketsContext _marketsContext;
public AuthorizeProjectFilter(IApiClient apiClient, IMarketsContext marketsContext)
{
_apiClient = apiClient ?? throw new ArgumentNullException(nameof(apiClient));
_marketsContext = marketsContext ?? throw new ArgumentNullException(nameof(marketsContext));
}
public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
{
if (
context.HttpContext.Request.Query.TryGetValue(nameof(RequestBase.Project), out var value) &&
context.HttpContext.Request.Headers.TryGetValue("Authorization", out var authorizationHeader)
)
{
var project = value.ToString();
var accessToken = authorizationHeader.ToString().Split(" ")[1];
var markets = await _apiClient.GetDataAsync<IEnumerable<Market>>($"{_authorizationServiceOptions.Authorization}/markets?project={project}", accessToken);
var authorized = markets.Any();
if (!authorized)
{
context.Result = new ForbidResult();
}
else
{
_marketsContext.Markets = markets;
await next();
}
}
}
}
And this works great locally, requests are quick when I call this service, however when I deploy this to Azure App Service - responses take 20+ seconds, sometimes more than a minute. App plan isn't also the cheapest one and it should be fast.
That doesn't seem like a cold start issue because I can keep on calling the service - it still remains slow.
I've run a trace on Azure diagnostics and got the following:
Seems like most of the time is spent on BLOCKED_TIME within system.net.http and I'm literally lost here.
Tracing has also caught some exceptions with the following message:
An attempt was made to access a socket in a way forbidden by its access permissions
Also my HttpClient is added using IHttpClientFactory:
services
.AddHttpClient<IApiClient, ApiClient>(client =>
{
client.DefaultRequestHeaders.Add("Accept", "application/json");
})
.ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
{
UseProxy = false
});
I'm literally lost now, I couldn't find anything on Google that would've helped me.
Please help :)
Top comments (8)
Took ages, but I finally managed to find that the issue is corporate VNet that application is connected to. 🙂
Any possibility you could provide the details as to what was wrong with the VNet? We are currently experiencing this exact issue.
Hey, yes.
So the actual issue was VNet and that it used our own DNS.
Any outgoing external request would be routed through a service that would check if address is not blacklisted by our IT department. Can't remember how it was called, but it was this exact service that caused it.
Hope that helps. If you need more details, let me know.
I tried removing the route table from the Subnet the App Service was running on, but still encountered the same issue. Was there another change needing to be made, perhaps on the VNET/subnet itself, or the way the company created it internally?
To be honest we went with a totally different approach.
The only solution I was told was not to check these outgoing addresses at the company level, which we obviously didn't do.
The service we called was internal and we implemented event driven architecture to not rely on it and to be more fault tolerant.
I think the "socket access" error is the big clue. It's probably not your code (although full disclosure, it's late here, I can't sleep... so your code hit my TL;DR filter). :-)
But it definitely feels like an OS level issue and not an issue with your code. Assuming you're running some form of *nix (including MacOS), most users and OS processes are not allowed to run on any port lower than 1024, without human intervention on the config files.
Other alternative ideas:
Since the error specifically mentions ports, I'd start there. You're using an already occupied port, or you have a firewall setting blocking the port... etc.
And, of course, I missed the fact you're running in Azure. In that case, definitely check the firewall. Azure VMs are pretty strict about the ports that they can use. (I'm assuming you're in a VM of some sort and not an Azure Serverless Function type thing).
You can open ports from the Azure portal.... I don't remember the exact nav-path in the Azure portal, but Googling "open port on azure firewall" or some such should get you there.
It's an azure app service, not a virtual machine. It really is IIS as service for a single application to run. I'll see what possible network restrictions are there.