ASP File Streams

Intro

As software developers, we frequently encounter scenarios where reading from or writing to streams is an essential part of our daily tasks. Whether it's processing data from files, network communication, or manipulating memory buffers, working with streams lies at the heart of many programming tasks. Mastering the art of handling file streams efficiently is not just a skill but a necessity.

This article is not about covering all stream-related scenarios — the point of focus will be how to use file streams in the following scenario:

• We have our own blob service
• We have multiple layers above this blob service that will have to forward the stream

Hypothetical Setup

Services — imagine the following scenario:

• We have a Gateway
• The Gateway forwards the request to Service A
• Service A forwards the request to ServiceBlobs

ServiceBlobs can leverage any storage medium — in this example MongoGridFS is used to store files.

Requirements:

• The user can send large files
• The memory footprint must be small
• Files should be verified

About Streams

.NET streams are a fundamental abstraction for handling input and output operations, providing a uniform way to read from and write to various data sources and destinations. They represent a sequence of bytes and are commonly used for tasks such as reading from files, network communication, memory operations, and more.

One specific stream is HttpRequestStream, used in the context of HTTP requests. When you make an HTTP request with the HttpClient class, the request body can be accessed through this stream, allowing you to write data you want to send in the body of the HTTP request.

Api Design

When transmitting files over the web, developers face a choice between sending the file as binary content in the request body or using multipart form data. Each has its own trade-offs.

Sending file as binary content — the file is sent directly as binary data in the HTTP request body. Straightforward and efficient, with no additional encoding overhead.

Using multipart form data — the file is packaged alongside form fields into a multipart message. Commonly used for HTML form submissions with file uploads.

Why multipart is problematic for large files:

1. Memory consumption — the entire file must be loaded into memory or temporary storage before transmission
2. Network overhead — additional headers and multipart formatting increase request sizes and network traffic
3. Server processing — parsing multipart messages on the server is resource-intensive for large files

For large files, sending as binary content in the request body is the more efficient alternative — minimizing memory consumption, network overhead, and server processing requirements.

Analysis

[Gateway] API Endpoint

The choice of sending the file as binary content in the request body was clear: avoid large memory consumption and enable simple stream buffering.HttpRequestBody is un-seekable by default for efficiency reasons — this behavior was leveraged by simply passing the stream forward, keeping the endpoint lightweight with no data piling in memory.

// HttpRequestBody (HttpRequestStream) — non-seekable by default
httpContext.Request.Body.CanRead   // true
httpContext.Request.Body.CanSeek   // false
httpContext.Request.Body.CanWrite  // false

// Accessing Length or Position throws NotSupportedException
httpContext.Request.Body.Length;   // NotSupportedException
httpContext.Request.Body.Position; // NotSupportedException

[Gateway → Service A] Service Client

The client had to be equally lightweight — no reading from the stream, no extracting information. Simply pass the input stream directly in the HTTP request using StreamContent:

HttpRequestMessage request = new (...);

request.Content = new StreamContent(httpContext.Request.Body);

[Blobs Service]

The blob service uploads content to MongoDB using GridFS. To perform content validation without reading the entire stream into memory, the stream first needs to be made seekable:

HttpContext.Request.EnableBuffering();

With buffering enabled, the stream can be partially read for validation and then seeked back to the beginning before being written to storage:

using MemoryStream memoryStream = new MemoryStream();
await data.Stream.CopyToAsync(memoryStream, 1024);
data.Stream.Seek(0, SeekOrigin.Begin);

await _blobContentStorage.WriteAsync(storeId, new BlobWriteStream
{
    FileName = fileName,
    Content = data.Stream,
});

Reading the first few bytes allows checking the file's magic byte signature — verifying that the content matches the file extension provided. Any form of content validation can be performed at this stage, just bear in mind the performance cost.

Example JPEG magic byte signatures:

new byte[] { 0xFF, 0xD8, 0xFF, 0xE0 },
new byte[] { 0xFF, 0xD8, 0xFF, 0xE1 },
new byte[] { 0xFF, 0xD8, 0xFF, 0xE8 },
new byte[] { 0xFF, 0xD8, 0xFF, 0xE2 },
new byte[] { 0xFF, 0xD8, 0xFF, 0xE3 }