Architecting Systems #5: System Design Overview

Pre-signed uploads, ephemeral GPU workers, leases, and HLS over CloudFront — and the cases that shape the design.

This blog is part of the Architecting Systems series, where we are breaking down the different ideas and decisions that go behind architecting a system. In the previous parts, we covered requirements and system design guidelines, communication and flow design, and the role of buffers, events, streaming systems, blob stores, and CDNs.

In this part we bring those ideas together and design a video-streaming platform. A creator uploads a raw .mp4, the system processes it, and the processed output is served to millions of users. The system has to accept the upload, transcode it before it becomes streamable, and deliver it reliably across devices and network conditions.

We will build the system in order — upload, worker, transcode, delivery — and handle each failure where it can occur.

Control Plane and Data Plane Split

The design separates lightweight coordination from heavy media processing.

Control plane coordinates; the data plane does the expensive work and only while there is expensive work to do.

The control plane stays serverless. API Gateway and Lambda handle upload coordination, S3 events and EventBridge carry state changes, and DynamoDB is the source of truth for job metadata, processing state, ownership, leases, and retries. None of it holds a machine open waiting for work.

The data plane is where heavy work takes place. Transcoding and HLS generation run on ephemeral, GPU-backed EC2 instances running ECS worker containers. Each worker is launched for one specific job, reads that job directly from DynamoDB, and terminates when the job reaches a terminal state.

Which means,

• Coordination runs on services that cost nothing when idle, and expensive machines exist only while a video is being processed. Super good for infrequent requests, bad for high throughput.
• Instance type can be chosen per job, based on file size or expected processing complexity.
• DynamoDB gives one consistent view of job state and worker ownership, with conditional writes as the concurrency primitive.
• S3 and CloudFront handle durability and global reach without the application participating in the byte path.
• Uploading, processing, and streaming stay separable, so each can be scaled, failed, and reasoned about on its own.

Why There Is No Queue

In Part 3 of this series we spent a long time on queues, pub-sub, logs, visibility timeouts, and dead-letter queues. So the missing box in the diagram above is the obvious question: where is the SQS queue between upload and worker?

There isn’t one, in this case.

A queue solves two problems. It decouples producer from consumer, and it hands out work to a pool of consumers that do not know in advance what they will get. This system has neither problem. There is no consumer pool, because a worker is created for exactly one job and dies with it. And DynamoDB already holds the complete job state, so a queue message would carry nothing the worker cannot read for itself from the row it was named after. The message would be a pointer to the truth sitting next to the truth.

Removing the queue means that the worker does a strongly consistent read on its own key instead of polling a shared queue and filtering messages it does not care about. Visibility timeouts are replaced by a lease you own and can inspect. The dead-letter queue is replaced by a terminal failed status on the row itself, which is strictly more useful, because the failure sits with the metadata, the progress, and the attempt history rather than in a separate queue you have to correlate back.

Like with all trade-offs, there are downsides.

• Latency per job. Every upload requires a cold EC2 launch plus ECS cluster registration and image pull. A warm pool draining a queue starts work in milliseconds. If your product promises fast turnaround on short clips, consider an auto scaling worker pool with few always on workers.
• No concurrency ceiling. A queue with a fixed consumer pool is a natural throttle. One instance per video is not. A hundred creators uploading at once launches a hundred GPU instances and bills you for all of them.
• No backpressure. There is nowhere for work to wait. A queue absorbs a spike; this design converts it directly into spend.
• API limits become correctness problems. RunInstances throttling and per-account instance and vCPU limits now surface as jobs that never get a worker, which we discuss in Section D. With a queue, capacity exhaustion just means the queue gets deeper.

Reach for SQS plus a warm pool when jobs are small and frequent, when launch latency dominates processing time, or when you need a hard ceiling on concurrent spend. Keep the queue-less design when jobs are large and infrequent, when per-job machine sizing matters, and when processing time makes a two-minute launch irrelevant.

Job States

Before the flow, the state machine. Every transition in this design is a conditional DynamoDB write, and the condition is almost always “the current status is the one I expect.”

Note that file-processing-ready is reachable twice: once on the happy path from file-uploaded, and once from job-received when a lease expires and the job goes back into the pool.

A. Input Processing

The upload path: pre-signed URL, direct PUT to S3, event-driven verification.

1. Upload Request and Pre-Signed URL Creation

The flow begins when the client wants to upload. Rather than streaming the video through the backend, it asks for permission to upload directly to S3.

The client asks for an upload URL; Lambda writes the job row before returning it.

The client computes an actionKey and calls api.system.com/media/upload, backed by API Gateway and Lambda. The Lambda takes the actionKey and the caller's identity, generates a pre-signed S3 URL, and writes a DynamoDB item holding the actionKey, the userId, the expected object key, and status url-created. The key is strict:

uploads/{userId}/{actionKey}/source.mp4

Writing the row before returning the URL matters. The pre-signed URL is a capability that exists in the world the moment it leaves the Lambda, and if the row is not there when the object lands, the upload event has nothing to verify against.

A pre-signed PUT constrains the key and nothing else. The client can upload forty gigabytes, or a ZIP file renamed to .mp4, and you will find out by paying a GPU instance to discover it. If you need limits at the edge, use a POST policy with content-length-range, or sign Content-Type and Content-Length as required condition headers so a mismatched request is rejected by S3 rather than by your worker. Either way the worker should ffprobe the source before transcoding and fail the job cheaply if it is not a video it can handle.

2. Direct Video Upload to S3

The client PUTs the file straight to the pre-signed URL. This keeps the backend out of the heavy file-transfer path.

The client uploads directly to S3. The backend never sees the bytes.

A single PUT handles small videos. Anything larger uses multipart upload, with the backend signing a URL per part. As we covered in Part 4, a multipart bucket needs a lifecycle rule that aborts incomplete uploads, because abandoned parts are invisible in the console and billed indefinitely.

3. Detecting and Admitting the Upload

S3 emits an object-created event through EventBridge into a validation Lambda.

The bucket is configured to emit events through EventBridge. Note that this is a bucket-level setting, and you cannot enable notifications for a prefix. Prefix filtering belongs in the EventBridge rule’s event pattern, which is where you keep worker traffic from firing on the processed/ path.

The event invokes a Lambda that parses the actionKey out of the object key and checks DynamoDB: the job exists, the object key matches the one that was issued, and the upload belongs to the user it was issued to. If all three hold, the job moves to file-uploaded and then, conditionally, to file-processing-ready, initialising an empty processByResolution map alongside attemptCount, leaseExpiryTime, and beingProcessedBy. Lambda then launches an EC2 worker for that actionKey.

Once the upload is verified, the job is admitted and a worker is launched for its actionKey.

The condition on the current status is what makes this safe. S3 event delivery is at-least-once, and duplicate events are ordinary. A Lambda that only admits a job when the job is not already admitted will do nothing on the second delivery, which is exactly what you want.

No queue message is written. DynamoDB already holds the complete job state, and the worker is launched knowing the one key it is responsible for.

B. Picking Up the Job

Worker lifecycle: launch, claim, lease, terminate.

1. Launching the Worker Instance

One EC2 instance joins the ECS cluster and runs exactly one worker container.

Lambda launches a GPU-backed EC2 instance for the specific actionKey, sizing the instance type from the source file size or expected complexity. The instance's user-data script joins it to an ECS cluster, where a daemon service runs exactly one worker container per instance, pulled from ECR. The ECR repository keeps a retention policy of the latest five images.

One uploaded video creates one EC2 instance, one ECS worker container, and one processing job.

You can pass the actionKey into the container through user data, where the same script that joins the cluster writes the key to a file on disk, or through an instance tag exposed in instance metadata.

If the worker is supposed to read the key from metadata, configure these two launch parameters:

• MetadataOptions.InstanceMetadataTags=enabled, with the tag passed in TagSpecifications at launch. It cannot be turned on after the instance exists.
• HttpPutResponseHopLimit=2. The default is 1, and with bridge networking the container's request to IMDS crosses the docker bridge, which is one hop, so the IMDSv2 token PUT is silently dropped and the container simply never gets a token. This is the single most common way this pattern fails on first deploy. Remember too that IMDSv2 requires the token round-trip, so a bare curl of the metadata path will not work.

2. Claiming the Job

The worker reads its own key and claims the job with a conditional write.

Because the worker was launched for a specific actionKey, it does not discover work. It performs a strongly consistent read on that exact key, and it never polls a shared queue or filters messages meant for someone else.

Having read the job, the worker claims it with a conditional update, moving the status to job-received, recording its worker ID in beingProcessedBy, setting leaseExpiryTime, incrementing attemptCount, and stamping the start time. attemptCount has exactly one writer, and this is it. Counting attempts at claim time means the number always reflects how many workers have actually held the job, and never double-counts a failure that also gets recorded on the way out.

Get Susmit’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The update is conditional on the job being unowned, and the next section covers exactly what “unowned” means. If the claim fails, the worker reads the state and gets out of the way:

• Another worker holds a live lease, so terminate.
• The job is already processing-completed, so terminate.
• The job is failed, so terminate.
• The job does not exist, so record the provisioning error and terminate.

3. Leases, Ownership, and Worker Safety

The lease: temporary ownership, renewed by heartbeat, expiring on death.

A worker does not own a job. It holds a lease on one, and the lease has to be renewed for as long as processing continues.

DynamoDB carries the ownership fields alongside the job:

• beingProcessedBy, the current owner's worker ID
• leaseExpiryTime, when that ownership lapses
• attemptCount, how many workers have claimed this job
• lastHeartbeatAt, when the owner last checked in
• processingStartedAt, when the current attempt began

A claim succeeds only when the status is file-processing-ready and no live lease exists, meaning either beingProcessedBy is unset or leaseExpiryTime is in the past. There is no now() in a condition expression, so the caller passes its own timestamp in as an expression value and the condition compares against that. Clock skew between callers becomes a real, if small, part of your correctness argument.

While processing is healthy the worker extends its lease, conditional on beingProcessedBy still being its own ID, so a worker that has been superseded discovers it on the next heartbeat rather than at the end of a long transcode. The heartbeat must run on its own thread. A blocking ffmpeg invocation will happily starve a heartbeat loop for twenty minutes and cause a worker to lose a lease on a job it is actively, successfully processing.

When a worker dies, freezes, or loses the network, the lease simply stops being renewed and the job becomes recoverable once it expires. This is stronger than reacting to EC2 termination events, because it also covers the cases EC2 has no opinion about: a hung process, a deadlocked application, a partial network partition where the instance is healthy and the worker is not.

4. Terminating the Worker

The worker terminates its own instance with TerminateInstances once the job reaches a terminal state, and it does so on every exit path, not only on success. Put the call in a finally. A daemon service restarts any container that exits, so a worker that dies without terminating its instance comes back as a fresh container holding the same actionKey, and an uncaught exception, an OOM kill, or a non-zero ffmpeg exit will loop.

That loop is worse than it looks if worker identity is the instance ID, because the restarted container passes the ownership check against itself and re-claims a job an orphaned process may still be working on, with attemptCount climbing on an instance that never dies. So worker identity should be a per-container UUID generated at boot rather than the EC2 instance ID. A restarted container is then a distinct owner and has to wait out the previous lease exactly like a replacement worker on a new instance.

5. Telling the Creator

DynamoDB tracks the job through six states, and none of that is worth anything if the creator cannot see it. Expose a GET /media/{actionKey} endpoint through the same API Gateway and Lambda, authorised against userId, returning the status, the completed entries in processByResolution, and the playback URL once the job is processing-completed. Clients poll it while the upload page is open.

For anything longer-running than a page visit, the terminal transitions should also emit an EventBridge event. The same mechanism that alerts your team on failed can drive an email, a push notification, or a webhook to the creator on processing-completed.

C. Processing the Job: The Simple Path

Transcode, segment, package, upload.

1. Transcoding and HLS Generation

One source file becomes a resolution ladder of HLS segments and playlists.

With the job claimed, the worker downloads the source, probes it, and transcodes it into a resolution ladder, typically 360p, 720p, and 1080p, using GPU acceleration. Each rendition is divided into HLS segments, and the packager emits a media playlist per resolution plus one master playlist listing them all.

The client loads only the master playlist. The player picks a rendition from it based on measured bandwidth and device capability, and switches between them mid-stream as conditions change. This is how you are able to continue watching when passing through bad network conditions at the expense of viewing at a lower resolution.

2. Storing the Processed Output

Processed output is written to a separate prefix and tracked per resolution in DynamoDB.

Processed files go to a separate prefix, keeping raw uploads and streamable output cleanly apart. Each rendition gets its own subdirectory, with the master playlist at the top:

processed/{userId}/{actionKey}/master.m3u8
processed/{userId}/{actionKey}/360p/playlist.m3u8
processed/{userId}/{actionKey}/360p/segment_00001.ts
processed/{userId}/{actionKey}/720p/...
processed/{userId}/{actionKey}/1080p/...

As each rendition finishes, the worker conditionally sets that rendition’s entry in the processByResolution map — 360p, 720p, 1080p — to the playlist path it just wrote. Tracking progress per rendition means a retried job can skip work that is already done rather than re-encoding the whole ladder.

“Already done” needs a definition, though, because trusting a half-written rendition is how you ship a video that stalls at the four-minute mark. A rendition counts as complete when its media playlist parses, the segment count matches what the source duration and target segment length imply, and every segment referenced by the playlist exists in S3 at non-zero size. Anything short of that, re-encode the rendition. If you would rather not write the validator, the safe default is to treat partial output as absent.

Once every required rendition is complete, the worker writes the master playlist path and moves the job to processing-completed.

D. Failure, Retry, and Recovery

The paths a job takes when something goes wrong.

1. Recording Failure and Retrying

Worker-reported failure, and EventBridge-driven replacement when the worker cannot report at all.

When processing fails, the worker writes what it knows: the failure reason and timestamp, the stage it reached, the resolutions it completed, and its worker ID. The worker writes its terminal state before it dies. The lease covers the case where it can’t.

Infrastructure failures are that case, whether a spot reclaim, a hardware fault, or a kernel panic. EC2 state changes flow through EventBridge into a Lambda, which maps the instance ID back to the job, confirms the job has not reached a terminal state, and confirms the lease has expired. If the attempt limit has not been reached, it conditionally moves the job back to file-processing-ready and launches a replacement worker with the same actionKey. The replacement reads the job directly from DynamoDB and claims it in the ordinary way, incrementing attemptCount as it does.

The lease is still the primary recovery mechanism. EC2 and EventBridge events only let the system react faster than lease expiry would allow, so they are an optimisation on top of it and never a substitute. A retry must not begin while the previous worker still holds a valid lease, or an infrastructure event becomes the thing that puts two workers on one video.

2. Permanently Failed Jobs

The terminal failure state and its operational surface.

Without a queue there is no dead-letter queue to move failures into, and none is needed. When the attempt limit is reached, the job is conditionally moved to failed, carrying the final failure reason, the last worker ID, and the attempt count. It is no longer eligible for automatic provisioning, and an EventBridge event or SNS notification puts it in front of a human, whether that is Slack, email, or an incident-management system.

A dedicated failure state provides the same operational visibility as a dead-letter queue while keeping job state, metadata, progress, and failure information in one place.

Finding those jobs needs an index. A global secondary index on status and updatedAt lets operators and scheduled recovery processes query for failed jobs, expired leases, jobs awaiting processing, and jobs stuck beyond their expected duration.

One caveat: a status partition key has a handful of values, so it concentrates writes on a handful of partitions, and file-processing-ready and job-received churn on every job in the system. Shard the key, using something like file-processing-ready#3 with a small random suffix and fanning the recovery query across shards, or make the index sparse by writing a needsRecovery attribute only on the states you actually scan for, so completed jobs fall out of the index entirely.

3. Recovering Jobs That Were Never Started

A scheduled sweep for jobs whose worker never launched.

A job can be marked file-processing-ready and never get a worker at all, because RunInstances was throttled, there was no GPU capacity in the AZ, or the instance failed during startup before the container ever ran. Nothing has failed in a way that emits an event, so nothing wakes up.

A scheduled EventBridge rule invokes a recovery Lambda that queries the index for jobs sitting in file-processing-ready longer than they should, checks that no live worker is associated with each, and launches one. A conditional update or a provisioning token keeps two overlapping recovery executions from launching duplicate instances for the same key.

This sweep is not how workers find work. It exists solely to repair jobs whose original provisioning failed, and if it is doing anything on a normal day, something upstream is broken.

E. Delivery

CloudFront in front of a private S3 origin.

1. Serving Through CloudFront

Origin Access Control keeps the bucket private while CloudFront serves the stream.

Processed output is served through CloudFront, not from S3 directly. The bucket stays private and CloudFront reads from it through Origin Access Control, so the only way to reach a segment is a CloudFront URL.

1.5. Content Types and CORS

Segments and playlists need correct MIME types, application/vnd.apple.mpegurl for .m3u8 and video/MP2T for .ts. Set them on upload from the worker, or force them with a CloudFront response headers policy. And because the player fetches segments with XHR or fetch from your application's origin, the distribution needs CORS headers. Without them the master playlist loads and every segment request fails, which looks like a broken encode and is not.

Worth knowing that HLS alone is not universally native. Safari plays it directly, while other browsers need hls.js or an equivalent, which is also the library that will surface the CORS failure above as a console error rather than a silent stall.

2. Private Video Access

Signed URLs and signed cookies for authorised playback.

For private, paid, or member-only content, CloudFront access itself has to be gated. Signed CloudFront URLs give short-lived access to a single object, while signed cookies are the right tool when the client needs a whole prefix, which is exactly what a stream of hundreds of segments is. The backend issues either only after confirming the viewer is authorised to watch that video.

This matters most for private uploads, paid content, course platforms, and creator-only access controls, which is to say anywhere the CloudFront URL leaking is the same thing as the content leaking.

Conclusion

The control plane coordinates, holds state, and provisions workers. The data plane does expensive work, and only while there is expensive work to do. DynamoDB is the source of truth, S3 holds raw and processed bytes, and CloudFront delivers the result at scale.

But almost none of the difficulty in this system is in the boxes. It is in what happens between them: a duplicate S3 event, a lease that expires while an orphaned process keeps writing, a container that restarts and re-claims its own job, a partial ladder that looks complete, an instance that never launched and never failed. The simple path is a diagram anyone can draw. The robust path, with its idempotency, conditional writes, ownership, leases, attempt limits, partial progress, terminal failure, and orphan recovery, is the system.

Next up, we will be touching upon databases.