Polishing diamonds: velocity is an architecture problem (Part 1)
Published On
Back in 2022, in the good old days, we wrote a blog post that got quite a bit of traction: “A magical AWS serverless developer experience”. **If you haven’t read it, have a look first; it explains how we got here.
At the time, our stance was very deliberate:
Local emulation for serverless is an anti-pattern. Avoid it.
Every engineer gets their own personal AWS account.
Live Lambda development (via SST) proxies code to real AWS infrastructure.
Invert the testing pyramid into a testing diamond (10% e2e tests, 70% integration tests, 20% unit tests), all running against live cloud resources.
With 7 engineers and around 600 integration tests running in under two minutes, it was genuinely brilliant. Four years later, that same setup had become the thing slowing us down: by this summer, every push waited almost half an hour for integration test results. This post is about what broke, and what we replaced it with.
Serverless wasn’t the problem
We’re still believers in serverless. Its elasticity is unmatched. Our customers vary in size, and support traffic is spiky by nature. With that traffic profile, having nearly immediate access to extra compute and database capacity is a game changer.
Yes, our AWS bill grows as we scale the business, but the unit economics are reasonable and, today, the trade-offs make sense.
What broke was our testing and deployment model at scale.
What broke
As Plain grew, testing against real AWS infrastructure started creating serious drag:
Deploying a complete stage to AWS via SST, CDK, and CloudFormation meant waiting longer and longer just to start running tests.
CI was constantly exposed to failure modes that had nothing to do with application logic: eventual consistency delays, AWS API rate limits, AWS resource quotas and network flakiness, among others.
Testing event-driven systems against real AWS EventBridge buses and SQS queues meant relying on messy polling loops and negative assertions. It’s “easy” to know something has happened, but not that it hasn’t.
Then there’s the elephant in the room, because nowadays you can’t write anything without mentioning it: AI (LLMs). Building our own internal software factory and leaning hard into agent-centric development put massive pressure on these foundational ideas.
When LLMs speed up coding and review cycles, your CI pipeline quickly becomes your primary bottleneck. Waiting 15, 20, or 30 minutes on cloud infrastructure runs brings the whole factory to a halt, and kills the fast feedback loop that agents depend on.
This isn’t a problem you can brute-force by buying larger, faster runners (thanks Depot!) or adding more concurrency. Our bottleneck was never compute. It was waiting for CloudFormation state transitions, or sleeping through arbitrary retry intervals in polling loops. Throwing compute at that doesn’t move the needle.
Project Loopback
So, how do we fix it?
The naive answer is: just mock every AWS call. But then who maintains those mocks? How do you ensure they expose the right failure modes? You just end up trading one nightmare for another.
Instead, we decided to tackle this with Project Loopback. The core idea: stop asking a single integration test to prove application logic AND infrastructure wiring at the same time.
So we split them.
Application logic is now covered by what we coined as “local integration tests”. We added local to tell them apart from our standard integration tests, because the two have co-existed for a while.
Plain services now follow a hexagonal architecture. Production and test run the exact same router, handlers, and aggregates. The test adapter binds persistence to a few locally-running containers (postgres, DynamoDB, and S3) and uses in-process adapters for events and queues. Published events are recorded synchronously, so there’s no more polling and no more eventual consistency races.
Infrastructure wiring is now covered by **OpenTofu and selective verification.** Instead of relying on integration tests to prove our cloud plumbing is sound, our confidence in the infrastructure now comes from:
Service blueprints, defined with codegen and static validation of each service's capabilities and bindings (more on this in a future post).
Targeted AWS contract tests for subtle cloud behavior our services rely on but that we can't emulate locally.
A lean set of end-to-end tests reserved for complete user flows and features that cross multiple system boundaries, such as inbound and outbound messaging.
Goodbye SST, hello OpenTofu
On the infrastructure tooling side, this meant saying goodbye to SST and going all-in on OpenTofu.
We really loved SST. Their team built pioneering developer abstractions that served us incredibly well for years.
But as Plain matures, we want fewer abstraction layers in our supply chain. We were also stuck on an older version of SST that required a non-trivial migration to upgrade. If we were signing up for a significant migration anyway, it made sense to take a step back and look at the bigger picture. OpenTofu gives us a thin layer on top of AWS APIs (moving us away from CDK/CloudFormation) and access to a big global talent pool.
The shift from TypeScript to HCL isn't a problem either. AI helps scaffold our service blueprints (YAML files), and an in-house compiler we built deterministically generates the underlying HCL for us.
The result
Unit tests and local integration tests now carry 90%+ of the confidence for a PR, in seconds, while a thin, selective suite keeps our AWS infrastructure wiring in check.
Most importantly, this has unlocked a massive step-change in development speed. Shipping changes with high confidence is fast and fun again.
It goes without saying that this has been a huge effort. It would easily have taken multiple quarters if we hadn't had agents doing the heavy lifting.
We didn't abandon our original testing diamond. It had just gathered too much grit to shine in this new, agent-driven world. Project Loopback has been about cutting away the dead weight and polishing the diamond until it gave us the instant feedback we needed.
By the numbers
Some stats we gathered after we finished the project:
28 min → 2.5 min: median time from push to integration test results
79% → 0.5% retried builds: most green builds needed a retry to get there. Now almost none do
50 → 6 CI runners: about 200 runner-minutes per push on 8vCPU machines, down to about 11 on 4vCPU machines
197 services, one blueprint each: blueprints are the source for our codegen, which writes 77% of our OpenTofu templates (about 65k lines of them)
What’s next
This is part one of a series on what we learned rebuilding our testing foundations and, more broadly, the way our team ships features to Plain.
In the next post, I'll share how we landed on our service architecture, what it gives us and why it's simpler than it sounds, with a bunch of concrete examples. I'll also cover the tooling that got us there in a fraction of the time we expected.
Stay tuned!