Efficient Scalability and Concurrency implementing the Lease Management as a Locking Pattern

Introduction

The challenge lies in effectively managing a large volume of events while ensuring parallel processing without duplicates. In event-driven architecture, it often becomes necessary to handle high-throughput or high-concurrency data processing.

We encountered a problem when waves of entities with the same identifier arrived together. This issue prevented us from processing creations or updates against the database due to conflicts with unique fields. To address this, I implemented a Lease pattern that enables sequential processing of events related to the same entity, without sacrificing the ability to process and scale out in parallel for events involving other entities.

The lease pattern, a technique that involves atomic operations and entity locking, ensures that while multiple events can be processed in parallel, the same entity is never processed concurrently. Through the integration of Azure Service Bus and Azure Storage, this pattern allows for improved performance, scalability, and effective handling of high-concurrency data processing tasks.

Background

In our high-throughput data processing environment, we faced a significant challenge when handling simultaneous events for the same entity. The system was designed to create a new entity in the database if it didn’t already exist or update it if exist. However, when multiple events for the same entity were processed concurrently, the first event would create the entity successfully, while the subsequent ones would fail due to an existing entry. This issue extended to related tables with unique relational indexes, necessitating manual intervention and increased workload.

Azure Service Bus offered a solid foundation with its robust capabilities for handling asynchronous data and state transfers. Still, we needed an additional layer of control to manage concurrent processing for the same entity effectively. The solution? The lease pattern, in conjunction with Azure Storage. This approach has transformed our system, creating a more efficient and scalable environment for high-throughput data processing. The ensuing sections will explore this implementation and its benefits in more detail.

Understanding the Lease Pattern

The lease pattern is based on the concept of “leasing” or “locking” a resource for a specific period, ensuring exclusive access to the lease holder during this time. In the context of our problem, the “resource” is the entity that is being processed.

When an event arrives, the system first tries to acquire a lease for the related entity. If the lease is acquired successfully, the system proceeds to process the event, safe in the knowledge that no other event will interfere with the same entity during the lease period. If the entity already has a lease (i.e., it’s being processed by another event), the system will either queue the incoming event for later processing or discard it, depending on the specific requirements.

This strategy effectively resolves the issue we were facing with Azure Service Bus. By implementing the lease pattern, we can process multiple events in parallel without worrying about duplications or conflicts for the same entity. The use of atomic operations ensures that the leasing process itself is free from concurrency issues.

Using Azure Storage for Lease Implementation

For the purposes of our problem, Azure Blob Storage and its leasing capabilities are particularly relevant. Azure Blob Storage allows us to create and manage blobs (Binary Large Objects) that can hold large amounts of unstructured data. Blob Storage provides a feature to create leases on blobs that act as locks, preventing others from modifying the blob for the duration of the lease. This feature can be effectively utilized to implement the lease pattern.

In our case, each entity corresponds to a blob in Azure Storage. When an event arrives for processing, our system attempts to acquire a lease on the corresponding blob. If successful, the system processes the event, secure in the knowledge that no other process will modify this entity during the lease period. If the blob is already leased, the incoming event is either queued for later processing or discarded.

By using Azure Storage in this manner, we create a distributed, scalable leasing mechanism for our entities, perfectly complementing the capabilities of Azure Service Bus and facilitating high-throughput data processing. In the next section, we will delve into the details of how this solution is implemented.

Implementation in .Net

Step 1: Setting Up Azure Blob Storage To begin with, we set up Azure Blob Storage, creating a container where each blob corresponds to an entity in our system. These blobs will act as “locks” in our lease pattern implementation.

Certainly. In our implementation, we use Azure Blob Storage configured with a ‘hot’ access tier. This configuration is crucial to our solution’s efficiency and cost-effectiveness.

Azure Blob Storage offers different access tiers — hot, cool, and archive — each designed for different use cases. The ‘hot’ access tier is optimized for storing data that is accessed frequently. In our case, since we’re continuously interacting with the blobs (acquiring and releasing leases), this access tier is the most appropriate.

Notably, while the ‘hot’ tier has slightly higher storage costs compared to ‘cool’ and ‘archive’ tiers, it has the lowest transaction costs. Given that we’re not storing data in the blobs but rather performing numerous transactions (lease acquisitions and releases), the ‘hot’ tier is more cost-effective for our use case. We efficiently manage high volumes of incoming events, ensuring optimal performance and minimizing costs by using Azure Blob Storage in the ‘hot’ access tier.
1- Create a storage account — Azure Storage | Microsoft Learn
2- Manage blob containers using the Azure portal — Azure Storage | Microsoft Learn

Step 2: We must install the NuGet Gallery | Azure.Storage.Blobs 12.16.0 to use it in our solution.
I am going to resume it in the essential parts.

The code below add the BlobLeaseService (Custom service in our solution) with the specific configuration for our BlobServiceClient having it set through dependency injection.

var blobServiceClient = new BlobServiceClient(configuration.GetSection("BlobServiceClient:ConnectionString").Value);
    services.AddSingleton(blobServiceClient);
    services.AddSingleton<IBlobLeaseService>(sp => new BlobLeaseService(sp.GetRequiredService<BlobServiceClient>(), "entity-leasing"));

And here how our service looks.

public class BlobLeaseService : IBlobLeaseService
{
    private readonly BlobServiceClient _blobServiceClient;
    private readonly string _containerName;

    public BlobLeaseService(BlobServiceClient blobServiceClient, string containerName)
    {
        _blobServiceClient = blobServiceClient; 
        _containerName = containerName;
    }   

Step 3: Event Arrival and Lease Acquisition When an event arrives for processing, the system first tries to acquire a lease on the corresponding blob in Azure Storage. This lease acts as a lock, ensuring exclusive access to the lease holder during the lease period.

public async Task<string?> TryAcquireLeaseAsync(string leaseBlobName, CancellationToken cancellationToken)
{
    var containerClient = _blobServiceClient.GetBlobContainerClient(_containerName);

    if (!await containerClient.ExistsAsync(cancellationToken))
    {
        await containerClient.CreateAsync(cancellationToken: cancellationToken);
    }

    var blobClient = containerClient.GetBlobClient(leaseBlobName);

    if (!await blobClient.ExistsAsync(cancellationToken))
    {   
        using var emptyStream = new MemoryStream();
        await blobClient.UploadAsync(emptyStream, cancellationToken);
    }

    var leaseClient = blobClient.GetBlobLeaseClient();
    
    try
    {
        var thirtySeconds = TimeSpan.FromSeconds(30);
        var lease = await leaseClient.AcquireAsync(thirtySeconds, cancellationToken: cancellationToken);
        return lease.Value.LeaseId;
    }
    catch
    {
        return null;
    }
}

In essence, we’re dealing with an atomic operation, meaning the first event to query a given entity will acquire the lease lock immediately. Consequently, any subsequent events targeting the same entity will encounter the lease as already locked.

Step 4: Lease Verification If the lease acquisition is successful (i.e., no other process currently holds a lease on the blob), the system proceeds to process the event. It does this secure in the knowledge that no other process will modify the corresponding entity during the lease period.

Step 5: Handling Locked Entities In cases where the blob is already under lease (indicating another process is handling an event related to this entity), the system follows a predefined protocol. Depending on the specific requirements, it either queues the incoming event for later processing or discards it.

var leaseBlobName = $"{entityNumber}-lease";
var leaseId = await _blobLeaseService.TryAcquireLeaseAsync(leaseBlobName, cancellationToken);

if (leaseId == null)
{
    await messageActions.AbandonMessageAsync(request.EventMessage, cancellationToken: cancellationToken);
    return;
}

In my specific scenario, I choose to abandon the message if the lease is already locked, returning it to the queue for future processing. Given our infrastructure’s efficiency, events are typically processed in under 0.3 seconds. Therefore, in most cases, an initially abandoned event is processed successfully on the second attempt without significant delay.

Step 6: Event Processing and Lease Release Once the event is processed, the system releases the lease, allowing other processes to acquire it for future events related to the same entity. This step is crucial to ensure smooth operation and prevent indefinite blocking of entities.

var leaseBlobName = $"{entityNumber}-lease";
var leaseId = await _blobLeaseService.TryAcquireLeaseAsync(leaseBlobName, cancellationToken);

if (leaseId == null)
{
    //Handling AcquiredLease
    return;
}  

try
{
   //Event processing
}
finally
{
    await _blobLeaseService.ReleaseLease(leaseBlobName, leaseId, cancellationToken);
}

There are essentially two methods for managing leases in the processing flow. The first option utilizes a try-finally block, ensuring that the lease is released after event processing, regardless of the outcome.

The second option is to manually release the lease in all sections of your code that could potentially break, return, or throw an exception. This method, however, can be inefficient and less reliable due to the need to manage lease releases in multiple places.

I configure each lease with a hard-coded duration of 30 seconds. According to the official documentation BlobLeaseClient.AcquireAsync Method (Azure.Storage.Blobs.Specialized) — Azure for .NET Developers | Microsoft Learn, the lease duration can range between 15 and 60 seconds, or it can be set to ‘infinite’. However, an ‘infinite’ lease duration wasn’t an acceptable option for us due to the risk of retaining leases indefinitely.

Upon completion of event processing, it’s crucial to release the lease. This action makes the lease available for the next event related to the same entity, thereby maintaining a smooth flow of event processing without unnecessary delays.

Performance and Scalability

Lease pattern, in tandem with Azure Service Bus and Azure Storage, has a profound effect on both the performance and scalability of our high-throughput data processing system.

Performance: We’ve eliminated the issue of concurrent processing of the same entity, thus reducing the occurrence of failed operations and unnecessary database retries. This significantly enhances the overall efficiency of the system.

With Azure Storage’s ‘hot’ access tier, we optimize for transaction costs, further improving the performance. Given the high frequency of lease operations (acquiring and releasing), the lower transaction costs in the ‘hot’ tier make this option not only feasible but also economically efficient.

Scalability: The lease pattern provides a scalable solution to manage a growing volume of events. Azure Storage offers virtually limitless storage, meaning we can comfortably accommodate an increasing number of entities and their corresponding blobs.

Conclusion and Future Work

The lease pattern, facilitated through Azure Service Bus and Azure Storage, has proven to be a powerful solution for our high-throughput data processing needs. By ensuring that only one event is processed at a time for a given entity, we’ve effectively eliminated conflicts and reduced unnecessary database retries. The result is a significantly improved performance and scalability in our system.

However, while the current implementation meets our needs, there is always room for further optimization and enhancements.

Additionally, more sophisticated lease management strategies could be explored. Currently, we employ a simple lease management strategy where incoming events for an entity with a locked lease are immediately abandoned and retried later.

Maintaining system resiliency is also a vital consideration. Striking a balanced approach to retries and reprocessing across all layers within the solution is crucial to ensure seamless operation. For a more in-depth discussion on maintaining downstream resiliency, effective retry strategies, and the use of exponential backoff in Azure Functions with Azure Database for PostgreSQL and HttpClients, I invite you to read my previous post:

Downstream Resiliency, retries and exponential backoff for Azure Functions with Azure Database-PostgreSQL and HttpClients.

I hope you’ve found this post insightful and practical. If you’ve enjoyed this post or have any ideas for improving the strategies I’ve discussed, I’d love to hear from you. Your feedback and ideas not only help me refine my current practices but also inspire new topics to dive into. Please feel free to leave a comment or write to me directly.

Thanks for reading, and until next time! Adrià.