Intro

In the realm of extensive datasets — with MongoDB as the focus here — it's imperative to understand the database's response patterns to the queries our applications launch against it. A task as "innocent" as "deleting all records associated with user x" turns out to be far from trivial at scale.

Research and community consensus (see the MongoDB forums) consistently highlights four problems with large single-call deletes:

Performance impact — increased resource usage can slow other operations or make the server unresponsive
Locking — write locks during a mass delete block other operations for its duration
Network traffic — large batches generate significant traffic between app and database
Timeouts — very large operations can exceed connection or execution timeouts, leaving deletions incomplete

A methodical batching approach addresses all of these:

Identify and batch entities — use MongoDB's FindAsync batch options to group results into manageable chunks
Sequential deletion — process one batch at a time for a controlled, system-friendly pace
Pause between batches — brief intervals ease load and maintain operational continuity
Background processing — delegate to a background worker (Hangfire) to avoid taxing the primary app flow

Batching The Mongo Cursor

MongoDB's FindAsync yields an IAsyncCursor — a low-level cursor pointing to a Current batch with a mechanism to advance to the next. Three goals drove a custom enumerator on top of it:

Item-centric iteration — yield single items, not entire batches
Automated batch loading — fetch the next batch automatically when the current one is exhausted
Decouple from IAsyncCursor — expose a universal IAsyncEnumerable so callers aren't aware of MongoDB internals

The BatchAsyncEnumerable wrapper accepts the IAsyncCursor and creates the enumerator:

csharp

public sealed class BatchAsyncEnumerable<TDocument> : IAsyncEnumerable<TDocument>
    where TDocument : BaseDocument
{
    private readonly IAsyncCursor<TDocument> _mongoAsyncCursor;

    public BatchAsyncEnumerable(IAsyncCursor<TDocument> mongoAsyncCursor)
    {
        _mongoAsyncCursor = mongoAsyncCursor;
    }

    public IAsyncEnumerator<TDocument> GetAsyncEnumerator(CancellationToken cancellationToken = default)
    {
        return new BatchAsyncEnumerator<TDocument>(_mongoAsyncCursor);
    }
}

The enumerator itself maintains a batch list and an index into it, fetching the next batch from MongoDB only when the current one is fully consumed:

csharp

public sealed class BatchAsyncEnumerator<TDocument> : IAsyncEnumerator<TDocument>
    where TDocument : BaseDocument
{
    private readonly IAsyncCursor<TDocument> _asyncCursor;
    private List<TDocument> currentBatch;
    private int currentBatchIndex;

    public BatchAsyncEnumerator(IAsyncCursor<TDocument> asyncCursor)
    {
        _asyncCursor = asyncCursor;
        currentBatchIndex = -1;
    }

    public TDocument Current
    {
        get
        {
            if (currentBatch == null || currentBatch.Count == 0)
                throw new Exception("Current item is null");

            if (currentBatchIndex == -1)
                throw new ArgumentOutOfRangeException("Current item cannot be retrieved. Try using MoveNextAsync");

            return currentBatch[currentBatchIndex];
        }
    }

    public ValueTask DisposeAsync()
    {
        _asyncCursor.Dispose();
        return new ValueTask();
    }

    public async ValueTask<bool> MoveNextAsync()
    {
        if (currentBatch == null)
        {
            if (await _asyncCursor.MoveNextAsync().ConfigureAwait(false))
                return GetCurrentBatch();

            currentBatch = null;
            currentBatchIndex = -1;
            return false;
        }
        else
        {
            if (currentBatchIndex < currentBatch.Count - 1)
            {
                currentBatchIndex++;
                return true;
            }
            else
            {
                if (await _asyncCursor.MoveNextAsync().ConfigureAwait(false))
                    return GetCurrentBatch();

                currentBatch = null;
                currentBatchIndex = -1;
                return false;
            }
        }
    }

    private bool GetCurrentBatch()
    {
        if (_asyncCursor.Current == null)
        {
            currentBatch = null;
            currentBatchIndex = -1;
            return false;
        }

        currentBatch = _asyncCursor.Current.ToList();

        if (currentBatch.Count == 0)
        {
            currentBatch = null;
            currentBatchIndex = -1;
            return false;
        }

        currentBatchIndex = 0;
        return true;
    }
}

Repository usage — note that passing an expression rather than a FilterDefinition carries a minor performance overhead from expression compilation:

csharp

public async Task<IAsyncEnumerable<TDocument>> FindManyAsync(Expression<Func<TDocument, bool>> filter)
{
    var asyncCursor = await _mongoCollection.FindAsync(filter, new FindOptions<TDocument>
    {
        BatchSize = 1000
    });
    return new BatchAsyncEnumerable<TDocument>(asyncCursor);
}

Partitioning

The batch enumerator lives at the repository layer, yielding individual items. To process items in bulk at the service layer, a second enumerator is layered on top: it accepts an IAsyncEnumerable<T> and re-groups items into partitions of a given size, returning IAsyncEnumerable<IEnumerable<T>>. It's advisable (though not required) to align the MongoDB BatchSize with the partitionSize here to avoid unnecessary item re-shuffling.

csharp

public class PartitionAsyncEnumerable<T> : IAsyncEnumerable<IEnumerable<T>>
{
    private readonly IAsyncEnumerable<T> _enumerable;
    private readonly int _partitionSize;

    public PartitionAsyncEnumerable(IAsyncEnumerable<T> enumerable, int partitionSize)
    {
        _enumerable = enumerable;
        _partitionSize = partitionSize;
    }

    public IAsyncEnumerator<IEnumerable<T>> GetAsyncEnumerator(CancellationToken cancellationToken = default)
    {
        return new PartitionAsyncEnumerator<T>(_enumerable, _partitionSize);
    }
}

public class PartitionAsyncEnumerator<T> : IAsyncEnumerator<IEnumerable<T>>
{
    private readonly IAsyncEnumerable<T> _enumerable;
    private readonly IAsyncEnumerator<T> _enumerator;
    private readonly int _partitionSize;
    private List<T> _current = new();

    public IEnumerable<T> Current => _current;

    public PartitionAsyncEnumerator(IAsyncEnumerable<T> enumerable, int partitionSize)
    {
        _enumerable = enumerable;
        _partitionSize = partitionSize;
        _enumerator = _enumerable.GetAsyncEnumerator();
    }

    public ValueTask DisposeAsync() => _enumerator.DisposeAsync();

    public async ValueTask<bool> MoveNextAsync()
    {
        _current.Clear();
        while (await _enumerator.MoveNextAsync())
        {
            _current.Add(_enumerator.Current);
            if (_current.Count == _partitionSize)
                break;
        }

        return _current.Count > 0;
    }
}

Batch Processing

A convenience extension method wires the partitioner directly onto any IAsyncEnumerable<T>:

csharp

public static class ExtensionsEnumerable
{
    public static IAsyncEnumerator<IEnumerable<T>> Partition<T>(
        this IAsyncEnumerable<T> enumerable, int partSize)
    {
        return new PartitionAsyncEnumerator<T>(enumerable, partSize);
    }
}

The BatchProcessAsync helper composes the enumerable, partitions it, and introduces a pause after each partition — the pause gives MongoDB breathing room between bursts:

csharp

public static class Batching
{
    public static async Task BatchProcessAsync<T>(
        Func<ushort, Task<IAsyncEnumerable<T>>> enumerableTask,
        Func<List<T>, Task> execution,
        ushort batchSize = 1000)
    {
        var enumerable = await enumerableTask(batchSize);
        IAsyncEnumerator<IEnumerable<T>> enumerator = enumerable.Partition<T>(batchSize);

        while (await enumerator.MoveNextAsync())
        {
            await execution(enumerator.Current.ToList());
            await Pauser.PauseAsync();
        }

        await Pauser.PauseAsync();
    }
}

The Hangfire background job ties everything together — one call handles enumeration, partitioning, deletion, and pacing:

csharp

public async Task ProcessAsync(JobRemove message)
{
    await Batching.BatchProcessAsync(
        async batchSize => await _repository.FindAsync(message.SomeId, batchSize),
        async entities =>
        {
            await DeleteAsync(entities);
            await Pauser.PauseAsync(); // brief Thread.Sleep to destress the database
        },
        batchSize: 1000);
}

Mongo Batch Processes

Intro

Batching The Mongo Cursor

Partitioning

Batch Processing