Our .NET App Crashed on 11.11. Here's the Queue Pattern That Fixed It

1. The Monday brief

I joined Lion City Logistics, a delivery startup in Singapore. Three weeks earlier, their tracking app (ParcelTrack) went down for hours during the 11.11 sale.

On my first Monday, the CTO sent me this message:

"Move ParcelTrack to AWS Singapore. It must handle traffic spikes. Keep secrets out of the code. Tell us when something breaks, before customers notice. Deploy automatically when we merge to main on Bitbucket. We need a UAT environment and a production environment."

First, I wanted to know what actually broke. If I move a broken design to the cloud, I just get a broken design that costs more money.

2. What went wrong

I read the old code and the incident notes. I found five problems.

Problems 1 and 2 caused the crash. The other three made it worse and made recovery hard.

3. The one rule I decided to follow

I wrote one sentence on a sticky note and put it on my monitor:

The API only does fast things. Anything slow, heavy, or likely to fail goes on a queue. A separate worker does it.

4. Why I ran the app on my laptop first

I could have started clicking in AWS right away. But I have learned this the hard way: if you debug code problems and cloud problems at the same time, you cannot tell which one is failing.

So first I made the new design work on my laptop. Then I moved it to AWS, knowing the code was fine.

To do this, I run six containers. Some were new to me, so here is what each one is.

The trick that makes this work

I want the same code on my laptop and in AWS. So I use one setting to choose where my code talks to. I show it in Stage 3.

5. Get the code

The folder looks like this:

parceltrack-aws-tutorial/
├── src/
│   ├── ParcelTrack.Core/      shared model, database, email, AWS setup
│   ├── ParcelTrack.Api/       the web API (fast things only)
│   ├── ParcelTrack.Worker/    background jobs (slow things)
│   └── ParcelTrack.Web/       the web page (Blazor)
├── scripts/                   LocalStack setup + a quick test
├── docker-compose.yml         starts all six containers
└── README.md

To run everything:

docker compose up --build

The first build takes a few minutes. Then open:

At the end of each stage, I can run something and see it work. If a stage breaks, I know which new piece caused it.

6. Building it in five stages

Stage 1: The database and the API

Goal: create a parcel and list parcels. Nothing else yet.

Files: Core/Parcel.cs, Core/ParcelDb.cs, Api/Program.cs, Api/Dockerfile

I put the data model in a separate Core project. Both the API and the worker need to read the same tables. If I define a parcel in one place, the two cannot disagree.

This C# class becomes a database table:

public class Parcel
{
    public Guid Id { get; set; }
    public string TrackingNumber { get; set; } = "";
    public string Recipient { get; set; } = "";
    public string CustomerEmail { get; set; } = "";
    public string Status { get; set; } = "Created";   // Created -> InTransit -> Delivered
    public DateTime CreatedAt { get; set; }
    public DateTime UpdatedAt { get; set; }
}

This is how Entity Framework talks to the database. DbSet<Parcel> means "a table of parcels":

public class ParcelDb(DbContextOptions<ParcelDb> options) : DbContext(options)
{
    public DbSet<Parcel> Parcels => Set<Parcel>();
    public DbSet<Invoice> Invoices => Set<Invoice>();
}

In the API's Program.cs, the first lines connect to the database:

builder.Services.AddDbContext<ParcelDb>(o =>
    o.UseNpgsql(builder.Configuration.GetConnectionString("Default")));

Look at where the connection string comes from. It is not in the code. It comes from configuration. This is my first step to fix problem 4 (the password in the repo).

Then there are the endpoints. The important ones:

app.MapHealthChecks("/health");      // "Am I alive?" The load balancer will ask this later.

app.MapGet("/api/parcels", ...);     // list parcels
app.MapPost("/api/parcels", ...);    // create a parcel

The Dockerfile uses two stages. The first stage has the full .NET SDK and builds the app. The second stage copies only the result onto a small image. The final image is much smaller and has no compiler in it.

I copy the .csproj files and run restore before I copy the source code. Docker saves that step in its cache, so later builds take seconds.

ASP.NET Core and environment variables: in docker-compose.yml you see ConnectionStrings__Default. The double underscore __ means a nested setting. It becomes ConnectionStrings:Default. ECS and EKS will use the same method later.

Check it works:

curl -X POST localhost:8080/api/parcels \
  -H "Content-Type: application/json" \
  -d '{"recipient":"Tan Wei Ming","email":"tan@example.com"}'

curl localhost:8080/api/parcels

You should see your parcel with a tracking number like LCL482910.

Stage 2: The web page (what the user sees)

Goal: a page that talks to the API.

Files: Web/Program.cs, Web/Pages/Home.razor, Web/wwwroot/appsettings.json

I used Blazor WebAssembly. It is C#, which my team already knows. It also builds into plain static files. Static files need no server. This will be useful when I use S3 and CloudFront later.

Three things matter on this page.

1. The API address is a setting, not hard-coded. It lives in wwwroot/appsettings.json:

{ "ApiUrl": "http://localhost:8080" }

Later, I can point the same build at UAT or production by changing this one file.

2. The page refreshes by itself every 3 seconds. So you can watch the status change:

timer = new Timer(async _ =>
{
    await Load();
    await InvokeAsync(StateHasChanged);
}, null, 3000, 3000);

3. The button that creates a parcel:

var res = await Http.PostAsJsonAsync("api/parcels", new { Recipient = recipient, Email = email });

My first problem here: CORS. My browser refused to let the page on localhost:8081 call the API on localhost:8080. This is a browser safety rule. The API must specify which websites are allowed. I fixed it in the API:

builder.Services.AddCors(o => o.AddDefaultPolicy(p =>
    p.WithOrigins(origins).AllowAnyHeader().AllowAnyMethod()));
// ...
app.UseCors();

origins comes from a setting (AllowedOrigins). UAT and production will have different website addresses, so I did not want to hard-code it.

Check it works: open http://localhost:8081, type a name and email, and click Create parcel. A new row appears.

Stage 3: The queue

Goal: when someone asks for an invoice, the API puts a ticket on a queue and replies at once.

Files: scripts/localstack-init.sh, Core/AwsExtensions.cs, the /invoice endpoint in Api/Program.cs

In docker-compose.yml, LocalStack runs a script when it is ready. The script creates my queues and buckets:

awslocal s3 mb s3://parceltrack-invoices-local
awslocal sqs create-queue --queue-name parceltrack-invoices-dlq
awslocal sqs create-queue --queue-name parceltrack-invoices \
  --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:parceltrack-invoices-dlq\",\"maxReceiveCount\":\"3\"}","VisibilityTimeout":"60"}'

To write this script, I had to learn three SQS ideas:

Now the endpoint. This is the heart of the fix:

app.MapPost("/api/parcels/{id:guid}/invoice", async (
    Guid id, CreateInvoice req, ParcelDb db, IAmazonSQS sqs, IConfiguration cfg) =>
{
    // One invoice per parcel. A customer who clicks twice gets the SAME invoice back.
    var invoice = await db.Invoices.FirstOrDefaultAsync(i => i.ParcelId == id);

    if (invoice is null)
    {
        invoice = new Invoice { /* ... Status = "Pending" ... */ };
        db.Invoices.Add(invoice);
        try { await db.SaveChangesAsync(); }
        catch (DbUpdateException)
        {
            // Two requests at the same moment: the unique index lets only one in
            db.ChangeTracker.Clear();
            invoice = await db.Invoices.FirstAsync(i => i.ParcelId == id);
        }
    }

    // Put a ticket on the queue. Only the ID, not all the data.
    if (invoice.Status == "Pending")
        await sqs.SendMessageAsync(cfg["InvoiceQueueUrl"],
            JsonSerializer.Serialize(new InvoiceRequested(invoice.Id)));

    // Reply right away. 202 Accepted means "I got it, but it is not finished."
    return Results.Accepted($"/api/invoices/{invoice.Id}", new { invoice.Id, invoice.Status });
});

Notice the first lines. On 11.11, customers pressed retry when the site was slow. Without these lines, each retry would make a new invoice and a new email. So the API itself must be safe against duplicates. The rule is enforced by the database, not by my code alone:

b.Entity<Invoice>().HasIndex(i => i.ParcelId).IsUnique();   // one invoice per parcel

(I added this after a reader asked about duplicates. See the section "Duplicate emails, properly" in Stage 4.)

I only put the ID in the message. The database stays the one true source of data. Messages stay small.

Now the one setting I promised. In Core/AwsExtensions.cs, all AWS clients are created in one place:

var url = cfg["Aws:ServiceUrl"];     // set on my laptop, empty in AWS

s.AddSingleton<IAmazonSQS>(_ => !string.IsNullOrEmpty(url)
    ? new AmazonSQSClient(new AmazonSQSConfig { ServiceURL = url })   // talk to LocalStack
    : new AmazonSQSClient());                                          // talk to real AWS

• On my laptop, Aws:ServiceUrl points to LocalStack.
• In AWS, I leave it empty. The SDK then uses the real service and the IAM role of the running container. The S3 client does the same, with one extra setting: ForcePathStyle = true for LocalStack. Real S3 uses addresses like bucket.s3.amazonaws.com. LocalStack uses localhost:4566/bucket.

Check it works:

./scripts/smoke-test.sh

The invoice request should return HTTP 202 in a few milliseconds.

Stage 4: The worker

Goal: read the tickets, make the PDF, store it in S3, and email the link.

Files: Worker/InvoiceWorker.cs, Worker/InvoicePdf.cs, Core/Email.cs, Worker/ParcelProgressWorker.cs

A .NET Worker Service is a console app that runs background tasks. The template gives you a Worker.cs that only prints the time. I replaced it with InvoiceWorker. It has two parts.

Part 1: the loop that takes tickets

var res = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
    QueueUrl = queueUrl,
    MaxNumberOfMessages = 5,
    WaitTimeSeconds = 20        // long polling
}, ct);

foreach (var msg in res.Messages)
{
    try
    {
        var job = JsonSerializer.Deserialize<InvoiceRequested>(msg.Body)!;
        var finished = await Process(job.InvoiceId, ct);
        if (finished)
            await sqs.DeleteMessageAsync(queueUrl, msg.ReceiptHandle, ct);   // done: remove the ticket
    }
    catch (Exception ex)
    {
        log.LogError(ex, "Invoice job failed");   // do NOT delete: it will come back and retry
    }
}

Two things to notice:

• Long polling (WaitTimeSeconds = 20): SQS waits up to 20 seconds for a message before it answers "nothing". This means fewer empty calls and a smaller bill.
• I delete the message only when the job is really finished. This one rule is the whole retry system. No delete means the message comes back. (Process returns false when another worker is busy with the same invoice. Then I leave the message alone.)

Part 2: the real work

async Task Process(Guid invoiceId, CancellationToken ct)
{
    var inv = await db.Invoices.FindAsync([invoiceId], ct);

    if (inv.Status == "Sent") return;          // see "idempotency" below

    // 1. Make the PDF and store it in S3
    var pdf = InvoicePdf.Render(inv);
    // ... upload to S3, set Status = "Generated"

    // 2. Make a link that works for 24 hours, and email it
    var link = s3.GetPreSignedURL(/* bucket, key, expires in 24h */);
    await email.SendAsync(inv.CustomerEmail, "Your invoice", $"Download: {link}", ct);

    // 3. Mark as Sent
}

• Pre-signed URL. A temporary link that lets someone download one private file for a short time. The bucket stays private. The customer still gets a working link.

Duplicate emails, properly (a reader's question)

A reader pointed out a hole in my first version, and they were right. SQS standard queues deliver messages at least once. That means the same message can arrive twice. My first version only checked Status == "Sent" before sending, and saved Sent after. Look at the gap:

send email ──► (crash here) ──► save "Sent"

If the worker dies in that gap, the email is out but not recorded. The message comes back and the customer gets a second email. Two workers can also both pass the check at the same moment. And on the API side, a customer retry made a second invoice. So there are three ways to get duplicates:

Why "claim before send" and not just "write a key first"? If I record "sending" and then crash before the email goes out, the retry would see the record and skip it. The email would never be sent. That trades a duplicate for a lost email, which is worse. So the claim is a short lease (45 seconds), not a permanent mark:

• A worker claims the email and sends it.
• If it dies, the claim expires after 45 seconds. The message returns after 60 seconds (the queue's visibility timeout), and another worker can claim it.
• The lease must be shorter than the visibility timeout. If not, the returning message would find the claim still active and give up. The honest limit. Email is an outside system. I cannot make "send the email" and "save to my database" happen as one all-or-nothing step. So if a worker dies in that tiny gap, a duplicate is still possible. I cannot remove it. I can only make it rare, and make it harmless: the second email contains a link to the same invoice PDF, so the customer sees the same file twice and nothing is charged or created twice.

What about a FIFO queue? SQS FIFO queues can drop duplicate sends using a MessageDeduplicationId (for example the parcel ID). That is useful, but it only remembers for 5 minutes, and it does not help when a message is delivered twice after a worker crash. A database rule that never expires (the unique index) is stronger, so I use the database. FIFO is a good extra layer if you need strict ordering too.

Swappable email. My worker calls email.SendAsync(...). It never says which email system. There are two classes behind one interface, and a setting picks one:

s.AddSingleton<IEmailSender>(sp => cfg["Email:Provider"] == "ses"
    ? ActivatorUtilities.CreateInstance<SesEmailSender>(sp)      // AWS (Tutorial 3)
    : ActivatorUtilities.CreateInstance<SmtpEmailSender>(sp));   // Mailpit (my laptop)

This is the same idea as LocalStack: the worker code does not change between my laptop and AWS.

The simulator. There are no real drivers in my demo. So I added a second small worker, ParcelProgressWorker. It moves parcels from Created to InTransit to Delivered every few seconds. This makes the page look alive.

It is not part of the real design, so I put it behind a switch:

if (builder.Configuration.GetValue<bool>("Simulator:Enabled"))
    builder.Services.AddHostedService<ParcelProgressWorker>();

I turn it on locally and in UAT. I leave it off in production, where real driver updates will come in.

One more fix I made. At first, the worker crashed if LocalStack was not ready yet. Now the worker catches the error, waits 5 seconds, and tries again.

Stage 5: Put it all together in Docker Compose

Six services in one file is a lot to read. So I wrote the settings that the API and worker share just once:

x-app-env: &app-env                         # a reusable block
  ConnectionStrings__Default: "Host=db;Database=parceltrack;Username=postgres;Password=localdev"
  Aws__ServiceUrl: http://localstack:4566   # "talk to the fake AWS"
  AWS_ACCESS_KEY_ID: test                   # LocalStack accepts any key
  AWS_SECRET_ACCESS_KEY: test
  AWS_REGION: us-east-1
  InvoiceQueueUrl: http://localstack:4566/000000000000/parceltrack-invoices
  InvoiceBucket: parceltrack-invoices-local
  Email__Provider: smtp                     # "use Mailpit, not SES"
  Smtp__Host: mailpit
  Smtp__Port: "1025"

Each service then reuses the block:

  api:
    environment: *app-env
  worker:
    environment:
      <<: *app-env                          # everything above...
      Simulator__Enabled: "true"            # ...plus one more setting

One thing that confused me: inside Docker Compose, containers find each other by service name. So the API connects to Host=db and http://localstack:4566. It never uses localhost. Inside a container, localhost means the container itself.

7. The 11.11 test

Now I tested the exact thing that broke before.

First the normal flow:

1. Create a parcel on the page.
2. Click Email invoice.
3. Watch the badge go Pending then Generated then Sent.
4. Open http://localhost:8025. The email is there. Now the disaster:

docker compose stop mailpit        # the email server "dies"

I created and invoiced 20 parcels. The page stayed fast, because the API only writes tickets. The worker printed errors. The tickets waited in the queue.

Then:

docker compose start mailpit       # the email server comes back

In about a minute, all 20 emails arrived.

In the old system, this exact situation took the whole site down. This is when I felt ready to show my boss.

8. From the brief to an AWS shopping list

Now that the code works, I can choose AWS services. For each need, I asked: what do I need, and which service gives it to me?

This is the target picture:

Why two ways to run containers? The CTO asked for EKS. But ECS Fargate is simpler and cheaper. It has no cluster to manage. EKS is more portable, and many employers ask for it. I will build both and compare them in a later post.

9. UAT and production

My rule: same Terraform code, different settings. If UAT is built differently from production, a passing UAT test means little.

In a real company, I would put UAT and prod in two separate AWS accounts. This is safer. To keep this series cheap, I will use one account with two clearly separate environments. They will have separate networks, separate state files, and uat- and prod- name prefixes. I will point out where you would split accounts.

The Terraform folder will look like this (I build it in the next post):

infra/
├── bootstrap/          the bucket that stores Terraform state
├── modules/            network, ecr, rds, sqs, ecs, eks ... written once
└── envs/
    ├── uat/            small and cheap
    └── prod/           safe and strong

Next: Tutorial 2

I will set up my AWS account safely (SSO and a budget alarm). I will learn Terraform properly. Then I will build the shared base for UAT and production: remote state, network (VPC), ECR, and S3.