Deploying k3s on AWS: what actually broke
Most k3s-on-AWS tutorials show you the happy path: provision, install, join, done. Here's what actually happened building Capstone Phoenix — a real production-style deployment, not a demo.
1. The account tier ceiling
Partway through provisioning, Terraform runs started failing against AWS account-level resource limits I hadn't accounted for. The fix wasn't more resources — it was resizing the cluster topology to fit what the account tier actually permitted, and being deliberate about which nodes justified their cost.
2. Private vs. public IP for agent nodes
k3s agent nodes join the cluster using an address you specify at join time. I had this pointed at the wrong IP class, which meant node-to-node traffic was trying to route somewhere it couldn't reach. The symptom was vague — nodes just wouldn't stay Ready — until I traced it back to the join configuration and switched to the private IP for internal cluster traffic.
3. Environment variable naming drift
This one looked like an application bug. The Flask API was failing to start, and the first instinct is to blame the code. It wasn't the code — it was a mismatch between the environment variable names Terraform and Ansible were setting on the host and the names the Flask app actually expected. Infrastructure and application config had drifted apart silently.
4. A moving allowlist
Working from a home connection without a static IP meant the AWS security group controlling SSH and admin access needed updating every time my IP changed. Not a deep technical problem, but a recurring one — the kind of thing that's invisible in a tutorial and constant in practice.
Takeaway
None of these were exotic failures. They were the ordinary friction of running infrastructure as code against a real cloud account with real limits — which is exactly why they're worth writing down. Full build details are in the Capstone Phoenix case study.