Distributed Job Processing System
The point of this project is not to collect infrastructure logos. It is to expose the operational behavior behind a distributed job from submission through execution, retry, and diagnosis.
The challenge
Complexity lives in the failure paths
Asynchronous work introduces partial failures, duplicated delivery, invisible queues, and deployment concerns that a synchronous demo can avoid. The system needed to make those concerns observable without pretending a portfolio workload was production traffic.
System design
Boundaries before components
Key decisions
Tradeoffs made explicit
Separate dispatch from durable state
PostgreSQL owns the job record while RabbitMQ and Celery handle task delivery. The UI reads authoritative status through the API.
Make failure a first-class state
Retry budgets, failed jobs, dead-letter behavior, correlation identifiers, and health endpoints are part of the product surface rather than hidden implementation details.
Keep local and cloud paths coherent
Docker Compose supports the smallest useful stack, while Kubernetes overlays, Helm-managed monitoring, Terraform, and GitHub Actions extend the same system into deployment practice.
Evidence & validation
What can be inspected
- A React dashboard submits work, filters recent jobs, and follows selected jobs until completion or failure.
- The repository includes deterministic data seeding and Playwright end-to-end coverage for the local stack.
- Local Kubernetes uses k3d, ingress-nginx, and the Prometheus Operator stack; cloud infrastructure is represented with EKS-focused Terraform.
Outcomes
- Creates one coherent environment for learning queueing, state, retries, observability, Kubernetes, and infrastructure automation.
- Keeps source, deployment instructions, and operating assumptions publicly inspectable.
Honest limits
- This is a portfolio and learning system, not evidence of real production traffic or availability.
- The next planned distributed-systems step is a transactional outbox and Kafka lifecycle event stream while retaining Celery and RabbitMQ for dispatch.