Automating My Tekton on EKS Lab with Terraform, ChatGPT, and Codex

In Part 1 of this series, I built a working Tekton CI/CD pipeline on Amazon ECR and EKS and then spent time understanding how all of the pieces fit together.

The pipeline worked. The application deployed. The public endpoint worked. The OpenRouter API call worked.

But there was one fairly obvious problem:

I did not want to pay to keep the entire AWS environment running when I was not using it.

An EKS cluster, EC2 worker nodes, a NAT Gateway, load balancing, storage, and the rest of the supporting infrastructure can add up quickly. For this lab, leaving everything running continuously could cost somewhere in the neighborhood of $250–$300 per month.

That makes very little sense for something I may use heavily for a few days, then ignore for a week.

My goal became simple:

I wanted to be able to shut the entire lab down when I was finished and bring it back later with as little effort as possible.

The end result was exactly that.

I now have two PowerShell scripts:

stop-lab.ps1
start-lab.ps1

One command shuts the lab down safely.

One command brings it back.

That should reduce my ongoing cost to something more like $10–$20 per month, depending on how often I actually run it.

But getting there was more interesting than simply writing two scripts.

First: Put the Infrastructure Under Terraform

The original lab had been built incrementally, using tools such as eksctl, kubectl, AWS CLI commands, and Tekton manifests.

That was fine for learning, but it meant the infrastructure lifecycle was not fully reproducible.

So the first step was moving the AWS foundation into Terraform.

Terraform now owns the major infrastructure components:

• VPC

• public and private subnets

• Internet Gateway

• NAT Gateway

• route tables

• EKS cluster

• managed EC2 worker node group

• EKS add-ons

• cluster IAM roles

• EKS access configuration

The final Terraform plan contained 38 managed resources.

Once the environment was rebuilt using Terraform, the important test was not just:

terraform apply

It was also:

terraform destroy

A lab is not truly disposable unless you know that both directions work.

We eventually proved the full cycle:

terraform destroy
terraform apply

and rebuilt the complete environment successfully.

Terraform Was Only Part of the Problem

One of the more useful lessons from this project was that Terraform did not own everything.

Some resources were created by Kubernetes after the cluster came up.

For example, this Kubernetes Service:

type: LoadBalancer

caused AWS to create a real Elastic Load Balancer.

Likewise, the Tekton workspace PersistentVolumeClaim caused Kubernetes and the EBS CSI driver to provision a real EBS volume.

Those resources existed in AWS, but Terraform had not created them.

That distinction became very important during shutdown.

If I simply ran:

terraform destroy

without first cleaning up the Kubernetes-owned resources, I could leave resources behind or create dependencies that interfered with destruction of the VPC.

So the shutdown order mattered. A lot.

The Safe Shutdown Sequence

Before Terraform destroys the AWS foundation, the script performs cleanup at the Kubernetes layer.

At a high level:

Delete Tekton PipelineRuns
        ↓
Delete public LoadBalancer Service
        ↓
Verify AWS ELB is gone
        ↓
Delete Tekton workspace PVC
        ↓
Verify EBS volume is gone
        ↓
terraform destroy
        ↓
Verify EKS cluster is gone

One detail turned out to be especially important.

The PipelineRuns must be deleted before the PVC.

During manual testing, I tried deleting the Tekton workspace PVC and it sat in:

Terminating

for more than ten minutes.

The reason became obvious when I described the PVC:

Finalizers: [kubernetes.io/pvc-protection]

Several old Tekton Pods still referenced the workspace.

As soon as I deleted the old PipelineRuns, those Pods disappeared and Kubernetes immediately allowed the PVC deletion to finish.

That is the kind of dependency I wanted to understand manually before automating anything.

Then Came Codex

Once the lifecycle was proven manually, I opened a Codex task and asked it to automate what we had just learned.

My instruction was essentially:

Create understandable, incremental PowerShell automation. Do not turn this into an elaborate framework.

Codex created:

stop-lab.ps1
start-lab.ps1

and updated the repository README.

What surprised me was how quickly it produced a substantial first version.

Roughly twelve minutes.

Not twelve minutes to generate a trivial shell script.

It had to understand the repository, the Terraform layout, the Kubernetes manifests, Tekton resources, AWS CLI usage, local secret handling, Pod Identity, and the order of operations.

This was my first experience watching an agent perform sustained repository work of this kind, and I found it genuinely remarkable.

A Dry Run Is Not the Same as Reality

We tested both scripts incrementally.

First:

-DryRun

Then parser checks.

Then controlled live testing.

Everything looked good.

Until the first real shutdown.

The script successfully:

• found the PipelineRuns

• deleted them

• deleted the public Kubernetes Service

• 
  

Then it stopped.

Importantly, it stopped before deleting the PVC, the EBS volume, or the Terraform stack.

The failure was subtle.

The script was checking whether the AWS Elastic Load Balancer had disappeared.

AWS CLI correctly returned:

LoadBalancerNotFound

That should have meant:

Good. The load balancer is gone.

But Windows PowerShell treated the AWS CLI stderr output as a terminating:

NativeCommandError

So the script never got a chance to interpret the expected "not found" result as success.

This is exactly the kind of thing that syntax checking and dry-run mode may never expose.

The logic was correct.

The real runtime behavior was different.

The Script Failed Safely

This is where the guardrails mattered.

Instead of continuing blindly after an unexpected error, the script stopped.

Nothing expensive or destructive happened after the unexpected condition.

We manually verified that the old ELB really was gone.

Codex then repaired the expected-failure handling for:

• ELB deletion checks

• EBS volume deletion checks

• final EKS deletion checks

• 
  

The repaired script passed additional tests.

Then we reran it.

This time it recognized the partial state correctly:

• there were no PipelineRuns left

• the public Service was already gone

• the PVC still existed

• 
  

It continued from there, removed the workspace storage, verified the EBS volume had disappeared, and destroyed all 38 Terraform-managed resources.

The final output was:

Lab stopped successfully.
The EKS cluster, public ELB, and workspace EBS volume are gone.

That was the shutdown half proven.

Starting the Lab Again

The startup script performs the reverse operation, but startup is more involved than simply running Terraform.

Terraform rebuilds the AWS foundation.

After that, Kubernetes and Tekton still need to be rebuilt.

The script performs roughly this sequence:

terraform apply
        ↓
Update kubeconfig
        ↓
Wait for EKS nodes
        ↓
Install Tekton Pipelines v1.15.0
        ↓
Restore StorageClass and PVC
        ↓
Restore Secrets
        ↓
Restore ServiceAccounts and RBAC
        ↓
Recreate EKS Pod Identity association
        ↓
Restore Tekton Tasks and Pipeline
        ↓
Create Deployment
        ↓
Create public Service
        ↓
Discover new ELB hostname
        ↓
Set APP_URL
        ↓
Run real PipelineRun
        ↓
Verify Deployment
        ↓
Verify HTTP response

There are several interesting details buried in that sequence.

Secrets Without Printing Secrets

The application needs two Kubernetes Secrets.

One contains the SSH key used by Tekton to clone the GitHub repository.

The other contains the OpenRouter API key.

The OpenRouter key is stored locally using Windows DPAPI protection.

The startup script restores the Kubernetes Secret without printing the plaintext API key to the screen.

That may seem like a small detail, but it is exactly the kind of behavior I want from automation that handles credentials.

Convenience should not mean dumping secrets into terminal history.

Recreating Pod Identity

Another subtle dependency is the EKS Pod Identity association.

The IAM role:

TektonBuildEcrPushRole

survives the destruction of the cluster.

But the association between that IAM role and the Kubernetes ServiceAccount:

default / tekton-build

does not.

It is cluster-specific.

So every time the cluster is recreated, the startup script has to recreate that Pod Identity association.

Without it, the Tekton build task would not be able to authenticate to ECR and push the new container image.

Again, that is something we discovered by manually rebuilding the environment before automation.

The Full Automated Start Worked

We finally ran the complete startup script.

It rebuilt the infrastructure.

The EKS nodes became Ready.

Tekton installed successfully.

Storage, Secrets, RBAC, and Pod Identity were restored.

The script launched a real Tekton PipelineRun:

tekton-eks-lab-pipeline-run-2fdwk

The pipeline successfully:

• cloned the application from GitHub

• built the container image

• authenticated to ECR

• pushed the image

• updated the Kubernetes Deployment

• 
  

The Deployment completed its rollout.

The application Pod became healthy.

The script received:

HTTP 200

from the new AWS load balancer.

The final message was:

Lab started successfully.

I then opened the application myself and sent a real request through OpenRouter.

It worked.

That was the proof I wanted.

Why I Still Tested Everything Manually

It would be easy to look at this and conclude:

AI wrote the automation. Done.

That would be the wrong lesson.

The more important sequence was:

Understand manually
        ↓
Automate
        ↓
Dry run
        ↓
Test incrementally
        ↓
Find real-world failure
        ↓
Repair
        ↓
Retest
        ↓
Trust

Codex dramatically reduced the amount of time required to create the scripts.

But it did not eliminate the need for engineering judgment.

In fact, the better I understood the system, the more effectively I could evaluate what Codex produced.

I did not need to understand every PowerShell detail.

But I did understand enough of the architecture to know that:

• PipelineRuns should disappear before the PVC

• Kubernetes-created AWS resources should disappear before Terraform destroys the VPC

• ECR must survive

• Pod Identity must be recreated

• Secrets should never be printed

• destructive automation should verify its target

• expected "not found" responses can mean success

• the final proof must include a real working application

• 
  

That level of understanding changes the relationship with AI considerably.

The goal is not necessarily to write every line yourself.

The goal is to understand what must be true.

Guardrails Over Cleverness

One thing I deliberately asked for was straightforward automation.

I did not want a generalized lifecycle framework.

I wanted scripts that were easy to read, easy to troubleshoot, and difficult to misuse.

The shutdown script includes preflight checks and verifies the AWS identity and Kubernetes target.

Before destructive work, it requires typing the exact cluster name unless automatic approval is explicitly requested.

It also checks that Terraform state does not include the ECR repository.

Then, after each important destructive action, it verifies that the expected resource is really gone.

That is not sophisticated abstraction.

It is just disciplined automation.

For this kind of lab, I think that is far more valuable.

The Cost Difference Changes the Lab

This may be the most practical result of the entire exercise.

Previously, leaving the lab running continuously could cost a few hundred dollars per month.

Now I can shut it down when I am finished and recreate it when I want to continue learning.

That changes the economic model from:

"Do I really want to keep this environment around?"

to:

"I can keep the code indefinitely and only pay when I am actively using it."

Infrastructure as Code makes the environment disposable without making the work disposable.

That is a powerful distinction.

Preserving the Work

The completed automation was committed to GitHub as:

439f497
Automate Tekton EKS lab lifecycle

The commit includes:

start-lab.ps1
stop-lab.ps1
README updates

The repository is here:

https://github.com/kmunson007/tekton-eks-lab

At this point the lifecycle portion of the lab is complete.

I can build it.

I can destroy it.

I can rebuild it.

And I can do either with one command!!

Next: Security

Now that the environment is reproducible, I can move on to the part I am most interested in.

Security.

The third post in this series will focus on hardening the software supply chain and the Kubernetes/Tekton environment.

Some of the areas I plan to explore include:

• SAST

• software composition analysis

• container image scanning

• SBOM generation

• image signing

• provenance

• secrets scanning

• IAM least privilege

• Tekton ServiceAccount security

• Kubernetes RBAC

• admission controls

  
  

The basic pipeline works.

The infrastructure lifecycle works.

Now the question becomes:

How much security can I realistically add to this pipeline to turn this learning lab into a reasonable facsimile of an enterprise platform?

I can't wait, post coming soon!