kunalships.dev

$ cat ARCHITECTURE.md

How Filedrop works

Filedrop is a serverless file-delivery pipeline on AWS. You POST a filename and email to /uploads, PUT the bytes to a presigned S3 URL, and a signed download link is emailed back. Under the hood, one request touches seven AWS services: API Gateway v2, four Lambdas, S3, EventBridge, DynamoDB, SNS, SQS, and SES. This page walks through the whole flow — what happens where, why the pieces are wired the way they are, and where the failure modes live.

The full flow, at a glance

Browser
   │  POST /uploads (email, filename, size)
   ▼
API Gateway (HTTP API)
   │
   ▼
request_upload Lambda
   │  1. validate email/filename/size
   │  2. reserve email (24h) — conditional PutItem
   │  3. write AWAITING_UPLOAD row (48h TTL)
   │  4. sign PUT URL (15 min)
   │
   ▼
DynamoDB (uploads + email-index)
   │
Browser ──── PUT ────► S3 (uploads/{id}/{filename})
                          │
                          │  Object Created event
                          ▼
                     EventBridge
                          │
                          ▼
                  process Lambda
                     │  1. HEAD object, re-check size/ext
                     │  2. status = UPLOADED  IF  status = AWAITING_UPLOAD  ← idempotency
                     │  3. SNS publish + attributes
                     ▼
                  SNS filedrop-events ───────┬───────────────┐
                                             │               │
                             filter:         │               │  no filter
                             file_uploaded   ▼               ▼
                                      SQS notify-queue   SQS audit-queue
                                             │               │
                                             ▼               ▼
                                      notify Lambda   audit Lambda
                                       │                     │
                                       ▼                     ▼
                              SES SendEmail         DynamoDB filedrop-audit
                                       │
                                       ▼
                              status = NOTIFIED  IF  status = UPLOADED

Browser polls GET /uploads/{id}/status → returns presigned GET URL once UPLOADED
ASCII rendered so it stays readable without a mermaid runtime.

Why this shape

There are simpler designs — you could point S3 straight at a single Lambda and skip the message brokers. This one exists in that shape because Filedrop is a portfolio piece meant to show three things: an event-driven fan-out pattern, at-least-once delivery discipline, and the failure-handling that keeps the whole thing quiet in production. Each hop in the flow earns its keep against a specific failure mode.

Step 1 — request_upload (POST /uploads)

The browser sends a JSON body with the email, filename, content type, and declared size. The Lambda does four things:

  1. Validation. Email regex, filename length + path-traversal guard, extension allowlist (pdf, txt, md, png, jpg, zip, csv), max size 25 MB.
  2. Anti-abuse gate. A conditional PutItem on the filedrop-email-index table with attribute_not_exists(email). DynamoDB TTL clears the row after 24 h, so genuine users can retry tomorrow. Duplicate within the window → HTTP 429.
  3. Persist slot. Write an AWAITING_UPLOAD row to the uploads table with the same 48h TTL — abandoned slots clean themselves up.
  4. Sign PUT URL. Presign a 15-minute PUT URL for uploads/{upload_id}/{filename} with the client's Content-Type baked into the signature.

The response returns the presigned URL. The Lambda never sees the file bytes. That's the point of presigned URLs: S3 becomes the upload endpoint, and Lambda is just a signing service.

The presigned-URL gotcha we hit

SigV4 presigning against S3 needs three things set on the boto3 client: signature_version="s3v4", region_name (Lambda's AWS_REGION env var), and Config(s3={"addressing_style": "virtual"}). Without the third, boto3 emits the legacy bucket.s3.amazonaws.com global host even in regional accounts. S3 returns a 307 redirect to the regional host, the browser follows it, but host is in SignedHeaders — so the resigned request fails and you get an opaque 403.

Step 2 — Direct PUT to S3

The browser PUTs the file bytes straight at the presigned URL. The bucket is block-all-public-access, encrypted at rest (SSE-S3), and has a 7-day lifecycle rule on the uploads/ prefix so orphaned files evict themselves.

Step 3 — EventBridge → process Lambda

The bucket has EventBridge notifications enabled (off by default on new buckets). An EventBridge rule matches source=aws.s3, detail-type=Object Created, and detail.object.key prefix uploads/. The rule targets the process Lambda.

Why EventBridge instead of S3 → Lambda direct? Three reasons:

Step 4 — Idempotency via conditional DynamoDB writes

S3 and EventBridge both deliver at-least-once. If process fires twice on the same object, the second run must be a no-op — not a second SNS publish, not a second email.

The idempotency gate is one line of DynamoDB:

ddb.update_item(
    Key={"upload_id": upload_id},
    UpdateExpression="SET #s = :up, actual_size = :sz, ...",
    ConditionExpression="#s = :aw",   # only fires if status = AWAITING_UPLOAD
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={
        ":up": UploadStatus.UPLOADED.value,
        ":aw": UploadStatus.AWAITING_UPLOAD.value,
        ...
    },
)

The first delivery flips the row. The second delivery hits ConditionalCheckFailedException, we catch it, log duplicate_event_suppressed, and return success. No dedup table needed — the existing state transitions (AWAITING_UPLOAD → UPLOADED → NOTIFIED) are the dedup keys.

Same pattern in the notify Lambda: SET status = NOTIFIED IF status = UPLOADED. Same catch-and-log behaviour.

Step 5 — SNS fan-out

process publishes to SNS filedrop-events with two message attributes: event_type (either file_uploaded or file_rejected) and content_type. Two SQS queues subscribe:

Each queue has its own DLQ with maxReceiveCount=3. After three failed deliveries, the message lands in the DLQ and a CloudWatch alarm on queue depth publishes to the filedrop-alarms SNS topic.

Step 6 — notify Lambda + SES email

notify signs a fresh 24-hour presigned GET URL, renders a text + HTML email (filename, size, content type, expiry timestamp), and calls SES SendEmail.

Two SES gotchas Filedrop actually hit in production:

Step 7 — audit Lambda

Append-only writes to a separate filedrop-audit DynamoDB table keyed by upload_id + emitted_at. Since it subscribes without a filter, both file_uploaded and file_rejected events accrue rows — handy for post-hoc review without touching the main uploads table.

Step 8 — The status poll (GET /uploads/{id}/status)

A client-side fallback for surfacing the download link without depending on email (SES sandbox blocks arbitrary recipients). The get_upload_status Lambda reads the uploads row, and only when status is UPLOADED or NOTIFIED does it sign a fresh GET URL. A caller can poll every couple of seconds until it gets a URL, then hand the user the download button.

IAM per Lambda

Least-privilege is the default here. Each Lambda role only carries the actions it actually uses, scoped to the specific resource:

Lambda Grants
request_upload dynamodb:PutItem on uploads + email-index; s3:PutObject on bucket/uploads/* (needed to presign PUT URLs).
get_upload_status dynamodb:GetItem on uploads; s3:GetObject on bucket/uploads/*.
process s3:GetObject + PutObjectTagging on uploads/*; dynamodb:UpdateItem on uploads; sns:Publish on the events topic.
notify s3:GetObject on uploads/*; dynamodb:GetItem/UpdateItem on uploads; ses:SendEmail/SendRawEmail scoped to the sender identity + config-set.
audit dynamodb:PutItem on the audit table.

Failure handling

Observability

What this is not

This is a portfolio piece, not a production file-delivery service. A few things I'd add before pointing real users at it:

Read next

Deploy your own copy — step-by-step guide →

Source code on GitHub →