Most of my career has been spent on both sides of a line that many teams treat as a wall: writing software, and getting it to run in production. The second half is where projects quietly succeed or fail. Code that works on a laptop is a draft. Code that deploys safely, recovers on its own and can be changed on a Friday without fear is a product.
So when AWS ran Accelerate Platform Modernization with AI on Amazon EKS, an online lab on Tuesday 13 October (12:30 in the afternoon here in Nairobi), I signed up for one reason: I wanted to see how AWS thinks applications should be deployed and managed in production at scale, and what a modern platform team actually hands its developers.
This is not a walkthrough. AWS publishes the workshop itself, and I've linked it at the end. These are my notes on what we did, what I took from it, and where I think it applies.
What we built in two hours#
Each participant got a temporary AWS account with an entire platform already running: three Amazon EKS clusters in Auto Mode (a hub that runs the platform tools, and DEV and PROD clusters for workloads), with more than fifty add-ons kept in sync by Argo CD. On top of it sat a developer portal (Backstage), Git (GitLab), single sign-on (Keycloak federated into IAM Identity Center), and the controllers that turn Kubernetes objects into real AWS resources (kro and ACK).
Then we used it the way a developer would, in three modules:
- Platform foundations. We requested an S3 bucket through a form, deployed an application from a manifest of about a dozen lines, and saw how a whole new EKS cluster can be ordered through a merge request.
- Application delivery. The backend of a fictional unicorn e-commerce shop, a Rust service backed by DynamoDB, taken from source code to running in DEV and PROD: a CI/CD pipeline ordered from Backstage, an image built by Argo Workflows and pushed to Amazon ECR, and one deployment manifest that brought its database, IAM role and Pod Identity with it.
- Progressive delivery. A Java service and the Rust service released as canaries with Argo Rollouts, behind functional, performance and metrics gates that roll back a bad version on their own, then promoted from DEV to PROD with Kargo.
Throughout, an AI agent in the terminal that knew the platform was a command away.
Seven ideas stayed with me.
1. The portal should never touch the cloud#
When I clicked Create on the S3 template, Backstage didn't call AWS. It wrote a change to Git. Argo CD noticed the commit and synced it into the cluster, kro expanded it, and ACK created the bucket.
It sounds like an indirection, and it is the most important design decision on the platform. Because Git is the only door, every request is reviewable, every change has an author and a timestamp, and every mistake has the same fix: revert the commit. The larger example made this obvious. Ordering a new EKS cluster didn't create one. It opened a merge request touching four files, which a platform admin reviewed and merged before anything was built.
The CI pipeline followed the same rule, which I liked even more. When Argo Workflows finished building the Rust image and pushed it to Amazon ECR, it didn't deploy anything. Its last step was to update the image tag in the deployment manifest in Git. Argo CD saw that commit and rolled the new version out. Even the robot goes through the front door.
As a DevOps engineer, this is the rule I'd carry into any system, platform or not: automation should change the source of truth, never the world directly. Let something else make the world match.
2. A cluster is an API, not a place where tools run#
Argo CD, kro and ACK ran as EKS managed capabilities. One small command made the point better than any slide:
kubectl get pods -n ack-system
# No resources found
The ACK controllers weren't in the cluster at all. AWS runs them. What lives in your cluster are only their APIs, the custom resource definitions such as an ACK Bucket or a kro ResourceGraphDefinition.
Anyone who has upgraded and patched in-cluster tooling will appreciate what that removes. But the bigger lesson is conceptual. Kubernetes is, at heart, an API for declaring what you want plus controllers that make it so. Where the controllers physically run is an implementation detail. Once you see a cluster that way, a platform stops being "a pile of tools" and becomes a set of APIs you design.
3. Reconciliation changes what "deployed" means#
The S3 exercise ended by asking us to notice what we hadn't done. No state file. No plan and apply. No pipeline to detect drift. The bucket's desired state lives in the cluster, and a controller compares it with reality continuously. Delete the bucket by hand and it comes back.
That landed because of my own recent work. Alex, the multi-agent financial planner I built during the Andela AI Engineering programme, was built on AWS with its infrastructure split across separate Terraform stacks (SageMaker, ingestion, the researcher, the database, the agents and the frontend), each planned and applied in order. That model works, and for large, deliberate changes I still like it. But it treats deployment as an event: you apply, and the state is correct until someone changes something.
Reconciliation treats deployment as a condition that is enforced all the time. For the stream of small resources developers request every day, that is the better contract. The workshop was fair about the boundary, and I agree with it: account baselines, organisation-wide networking and the bootstrap that creates the clusters still belong in Terraform. Bootstrap with Terraform; hand day-two provisioning to controllers.
4. Abstraction is the product#
This is the whole manifest a developer writes to get an application:
apiVersion: kro.run/v1alpha1
kind: AppmodService
metadata:
name: hello-world
namespace: default
spec:
image: public.ecr.aws/nginx/nginx:alpine
image_name: hello-world
replicas: 2
port: 80
targetPort: 80
ingress:
enabled: true
path: /hello-world
From that, kro produced an Argo Rollout with a canary strategy, the networking, an ingress on an internal load balancer and the secrets for metrics. The platform team had written the expansion once, in a ResourceGraphDefinition. The platform shipped six of them, from S3Bucket to EksCluster to RayService for machine learning workloads.
What impressed me wasn't the tool. It was the division of labour the lab kept spelling out under the heading who owns what. The platform team owns the template, the defaults, the security settings and the wiring. The developer owns a name, an image and a few choices. That ratio is the whole point of platform engineering, and it is a design problem more than a technical one. The workshop was honest that the hard parts are adoption, choosing the right level of abstraction, and helping a developer debug a layer they can't see.
The same thinking showed up in security. Each cluster's ACK controller had no broad AWS permissions of its own. To create a bucket it assumed a narrow, shared role that could only manage S3, and new clusters were trusted automatically by naming convention.
Permissions defined per AWS service, not per cluster, is the kind of decision that decides whether a platform still makes sense at forty clusters.
5. Order outcomes, not parts#
The second module opened with a sentence every DevOps engineer will recognise: deploying "Hello World" is easy, and real deployments are not. A real service needs a build pipeline, an image registry, a database, IAM permissions, secrets and routing, and each of those is a place where a developer usually stops and waits for someone else.
The lab's answer was to let the developer order outcomes. In Backstage I filled in one form, Deploy CI/CD Pipeline (GitOps), with the application name rust. That committed a single CICDPipeline object, and kro expanded it into everything a pipeline actually is: Argo Workflow templates, an Argo Events source and sensor that trigger a build on every git push, the Amazon ECR repository, and the IAM role, Pod Identity and RBAC the pipeline needs to run. I asked for a pipeline, not for its parts.
Deployment worked the same way. The lab let each of us choose KubeVela or kro, which produce the same running application from different manifest formats, and either way one manifest declared what the service needed: its image, port, ingress and a DynamoDB table. From that, the platform created the table through ACK, a service account with exactly the IAM permissions for that table, EKS Pod Identity so the pods get short-lived AWS credentials without a stored key anywhere, and an Argo Rollout to run the service.
The lab summed up the deal as a contract, and it is the cleanest statement of platform engineering I've seen:
- The developer provides the code, a Dockerfile at the root of the repository, and one manifest that says what the application needs.
- The platform decides how to build it, provision it and deploy it.
The Dockerfile is a smart choice of boundary. The container is the one standard every language shares, so the same golden path works for Rust, Next.js or Java without the platform knowing anything about the language. (The Rust image itself was a nice example of doing it well: a multi-stage build that compiles a static binary, then copies it into an empty scratch image, so production ships nothing but the program.)
It wasn't seamless, and I count that as part of the lesson. The GitOps application didn't create the namespace on the workload clusters, so we created it by hand. The load balancer's address was cached in a shell variable when the terminal started, so a terminal opened too early saw an empty URL. The Argo CD token expired and needed refreshing. None of these is serious, but each is the kind of small friction that sends a developer to ask for help. For a platform team, every one of them is a ticket worth closing, because the path is only golden if nobody has to step off it.
6. Releases should earn their way to production#
At the end of the second module, we promoted the Rust service to production by copying its manifest into the prod folder, changing a few values and committing. It worked, and it is honest GitOps. But the decision to promote was simply a person deciding the version looked fine.
The third module is the one I came for, because it replaces that judgement with evidence. It separates three jobs that are easy to blur together:
- Argo CD keeps each environment matching what Git says.
- Argo Rollouts shifts live traffic to a new version gradually, instead of all at once.
- Kargo moves a verified version from one environment to the next.
Of the usual strategies (A/B testing, blue/green and canary), the platform uses canary, and the platform team hides all of Argo Rollouts inside the same AppmodService abstraction. By default a new version gets 20% of traffic, then 40%, 60%, 80% and finally 100%, with an optional functional gate early and a performance gate late. A small demo made it visible: a grid of blue squares slowly turning yellow as traffic moved to the new version.
Then we broke things on purpose, which is where the real teaching was.
A gate that says no. The Java service's functional gate checked that the page was red. We changed the page to orange and pushed. The pipeline built the image, the canary started, the gate failed, and the rollout marked itself Degraded and went back to the last stable version. The page stayed red. Nothing was broken: the gate had done exactly its job, because the code had changed and the definition of "correct" hadn't. Once we updated the gate to expect orange, the same image sailed through. The lesson is that tests and code have to change together, and the gate belongs in the same manifest as the service it protects.
A gate that reads metrics. For the Rust service we went further. Its manifest declared five rules over the service's own Prometheus metrics, such as "fail if the average request duration rises above half a second", and the platform turned them into analysis that runs during the rollout. Then we uncommented a three-second sleep in one of the request handlers. A simple pass/fail test would still have passed: the page loads, just slowly. The metrics didn't. The duration rule tripped, and the rollout rolled back before the slow version took real traffic. That is the difference between does it work and does it work well enough, and only measurement can tell them apart.
Promotion with a paper trail. Finally, Kargo. A Warehouse watches Amazon ECR for new images. The DEV stage promotes automatically. The PROD stage waits for a person to click Promote, and what gets promoted is the same image that already passed in DEV, not a rebuild. Kargo writes the new tag into the production manifest in Git and asks Argo CD to sync. So the decision to go to production is still human, but it is a decision about a verified artefact, and Git records who made it.
The lesson is that deploying and releasing are different things. Deploying puts new code on servers. Releasing exposes it to users, and that should be decided by evidence, not by whoever pressed the button. Once gates and metrics sit in the path, a Friday deploy stops being an act of courage.
Two practical notes for anyone building this. The lab kept application code and deployment manifests in one repository for simplicity, and admitted the cost: every code change triggers two syncs, one for the commit and one for the new image tag. In production, keep code and configuration in separate repositories. And the AWS material made a point I'll repeat to anyone funding a platform: each of these capabilities produces a signal. Canary rollouts shrink the blast radius, gates protect quality without slowing delivery, metrics lower the change failure rate, and auditable promotion shortens lead time. Measure them, because, as the lab quoted from the State of Platform Engineering report, 45% of platform teams don't measure success at all.
7. A platform makes AI safer#
The lab came with Kiro CLI and an agent pre-configured for this platform, and I used it as I worked through the exercises. AWS framed AI in platform engineering as two capabilities: a chatbot over playbooks, onboarding and troubleshooting docs, and an assistant that generates platform artefacts such as manifests, ResourceGraphDefinitions and Backstage templates.
The most instructive exercise was small. We asked the agent to write the Rust service's deployment manifest, and the prompt was carefully built. Before anything else, it told the agent to read the platform's own component and trait schemas so it would use the right field names, and only then listed the requirements: image, ports, replicas, environment variables, readiness probe, resource limits, ingress path, the service account and the DynamoDB table. Then came the step I liked best: we compared the generated file with the platform's reference manifest using yq and diff.
It was close, and it was not identical. A few properties needed adjusting. The lab's own conclusion was the right one: the agent is excellent at the skeleton and at discovering how a platform's APIs fit together, and a human still has to validate the result. For the real deployment we used the platform's tested template, not the generated one.
Two habits from that exercise are worth stealing for any AI-assisted infrastructure work. Ground the agent in your real schemas before asking it to write anything, and diff its output against a known-good reference instead of trusting it by eye.
The Rust case study made a broader point too. Modernising a system often means adopting a language your team doesn't know yet, and AI has made that far cheaper than it used to be. But it moves the risk rather than removing it: a team can now produce code in an unfamiliar language faster than it can judge it.
The connection I keep coming back to is the reverse of the usual worry. Ask an AI agent to write raw Kubernetes manifests for a production service and you get two hundred lines that someone has to review line by line. Ask it for an AppmodService and you get a dozen, because the rollout strategy, networking and secrets are already encoded by the platform. The change still goes through Git, still gets reviewed, and is still reconciled by controllers that only allow what the platform allows.
A golden path does for an AI agent exactly what it does for a new hire: it makes the right thing the easy thing. As more code is written by agents, I suspect that will become one of the strongest arguments for building a platform at all.
Where I'd use this, and where I wouldn't#
Not every team needs Backstage, Argo CD, kro and three clusters. A small product run by one team can carry this machinery more easily than it can justify it. The full platform pays off when there are many teams, many services and the same requests arriving every week.
But several of the ideas scale down well, and those are what I'm taking into my own work:
- Git as the only door for infrastructure and deployment changes, even on a two-person project.
- Reconciliation over one-off scripts wherever the tooling allows it.
- Templates for the requests that repeat, including the pipeline itself, so that the second service is a copy of a good first one rather than a fresh guess.
- A clear developer contract: code, a Dockerfile and one manifest in; a running, observable service out.
- Narrow, assumed roles instead of one powerful credential, and short-lived pod identities instead of stored keys.
- Deploy and release as separate steps, with gates and metrics in the path, and rollback that doesn't need a human.
- Application code and deployment configuration in separate repositories.
- Abstractions an AI agent can safely target, because agents are already writing a growing share of the code.
What's next#
This was the two-hour edition. The full workshop has two more modules, and both are on my list. AI/ML delivery applies the same platform ideas to model serving and MLOps, including agentic workloads, which is where my own work increasingly lives. Measuring platform success covers DORA metrics (deployment frequency, lead time for changes, change failure rate and time to recover), which is how a platform team proves it is worth having.
If you want to try it yourself, AWS publishes a self-paced version you can run in your own account: Platform Engineering on Amazon EKS.