Unable to deserialize content from batch request, Graph API appears to be returning HTML which is not handled by SDK

[FionaUoN]

Describe the bug

We are making batched requests into the Graph API using the SDK and after a certain number of requests, the Graph API seems to be returning an HTML response, causing the SDK to throw an exception.

Stack trace is as follows: :

Error occurred while getting batch response: Unable to deserialize content.. Exception: Microsoft.Graph.ClientException: Unable to deserialize content.---> System.Text.Json.JsonReaderException: '<' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 137.at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan1 bytes)at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)at System.Text.Json.Utf8JsonReader.ReadSingleSegment()at System.Text.Json.Utf8JsonReader.Read()at System.Text.Json.JsonDocument.Parse(ReadOnlySpan1 utf8JsonSpan, JsonReaderOptions readerOptions, MetadataDb& database, StackRowStack& stack)at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedArrayPoolBytes, PooledByteBufferWriter extraPooledByteBufferWriter)at System.Text.Json.JsonDocument.ParseAsyncCore(Stream utf8Json, JsonDocumentOptions options, CancellationToken cancellationToken)at Microsoft.Graph.BatchResponseContent.GetBatchResponseContentAsync()

Expected behavior

When the Graph API returns HTML in this way, the SDK should handle the exception and return the HTML to the caller in a structured way, to enable us to triage the issue.
We are currently working with Microsoft support teams, but without the ability to obtain what the actual HTML response from the underlying Graph API was, we cannot pinpoint the issue.

How to reproduce

We are making hundreds of batched requests, 20 at a time, from an Azure Function in C#.

The request being made was [https://graph.microsoft.com/v1.0/users/{useremailaddress}/events/?$filter=singleValueExtendedProperties/Any(ep](https://graph.microsoft.com/v1.0/users/%7buseremailaddress%7d/events/?$filter=singleValueExtendedProperties/Any(ep): ep/id eq 'String {redacted} Name {redacted}'and ep/value ne null)&$expand=singleValueExtendedProperties($filter = id eq 'String {redacted} Name {redacted}')&top=1000

SDK Version

5.56.0

Latest version known to work for scenario above?

No response

Known Workarounds

No response

Debug output

Click to expand log

```

</details>


### Configuration

_No response_

### Other information

_No response_

[shemogumbe]

Hello @FionaUoN and thanks for raising this,

To help reproduce this, do you mind sharing the setup of your batchrequestSteps, content and collection.

Do some requests in the batch get executed and only this one fails?

[FionaUoN]

This is the SDK usage where we're calling into the Graph API and it works fine for a few hundred batches and then throws the exception mentioned above:

 GraphServiceClient graphClient = await GetGraphApiClient();
                    var retryPolicy = Policy
                      .Handle<ServiceException>(ex => ex.ResponseStatusCode == Convert.ToInt32(HttpStatusCode.TooManyRequests) ||
                       ex.ResponseStatusCode == Convert.ToInt32(HttpStatusCode.RequestTimeout) || ex.ResponseStatusCode == Convert.ToInt32(HttpStatusCode.Conflict) ||
                       ex.ResponseStatusCode == Convert.ToInt32(HttpStatusCode.InternalServerError) ||
                       ex.ResponseStatusCode == Convert.ToInt32(HttpStatusCode.BadGateway) || ex.ResponseStatusCode == Convert.ToInt32(HttpStatusCode.ServiceUnavailable) ||
                       ex.ResponseStatusCode == Convert.ToInt32(HttpStatusCode.GatewayTimeout))
                      .Or<SocketException>()
                      .Or<HttpRequestException>()
                      .OrInner<TaskCanceledException>()
                      .WaitAndRetryAsync(_config.NumberOfRetries, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt) * retryTime.Value),
                      (exception, timeSpan, retryCount, _) =>
                      {
                          if (exception is not null)
                          {
                              _logger.LogFunctionRetryAttempt(timeSpan, retryAttempt, _config.NumberOfRetries);
                              _logger.LogError($"Exception: {exception}," +
                                  $" RetryCount: {retryCount}, Timespan: {timeSpan}");
                          }
                      });

var batchRequestContentCollection = new BatchRequestContentCollection(graphClient);
batchRequestSteps.ForEach(x => batchRequestContentCollection.AddBatchRequestStep(x));

await retryPolicy.ExecuteAsync(async () =>
{
    batchResponse = await graphClient.Batch.PostAsync(batchRequestContentCollection);
});

[andrueastman]

@shemogumbe I think this can be resolved together with #2660

Essentially, at this point here, we need to validate that
https://github.com/microsoftgraph/msgraph-sdk-dotnet-core/blob/6eb564892b0b417cd6b49341af77448aae65cf0e/src/Microsoft.Graph.Core/Requests/BatchRequestBuilder.cs#L61

• The status code is a success code otherwise throw and error with the response information if an error mapping is passed.
• Validate the response content is json as well. This is the error observed from https://github.com/microsoftgraph/msgraph-sdk-dotnet-core/blob/6eb564892b0b417cd6b49341af77448aae65cf0e/src/Microsoft.Graph.Core/Requests/Content/BatchResponseContent.cs#L230

Thoughts? Would you be willing to resolve this?

[FionaUoN]

@andrueastman @shemogumbe I think crucially for us, we really need the content of the response from the Graph API even if it is HTML. So even if there is some network / firewall in between causing the HTML response, we really need to see that response as the caller, if possible please.

[andrueastman]

@FionaUoN Is this a scenario you can replicate consistently?
Would using the guidance here on logging request be helpful in your scenario? https://github.com/microsoftgraph/msgraph-sdk-dotnet-core/blob/main/docs/logging-requests.md

Validate the response content is json as well.

To be clear, what I mean here is the change would validate the response header and have the resulting response as a property in the error thrown.

[FionaUoN]

Consistently in that it happens for data of this volume but for different emails / batch number each time.
We weren't aware of the additional logging no, we will look at implementing that, thank you!