From CI/CD to DevSecOps: Securing My Tekton-on-EKS Pipeline

Over the past several weeks, I have been building a hands-on CI/CD lab using Tekton Pipelines running on Amazon EKS.

The original goal was fairly straightforward: take an application stored in GitHub, automatically build a container image, push that image to Amazon ECR, and deploy it to Kubernetes.

Eventually the basic pipeline looked something like this:

GitHub
   ↓
Tekton
   ↓
BuildKit
   ↓
Amazon ECR
   ↓
Amazon EKS / Kubernetes
   ↓
Application Load Balancer
   ↓
HTTPS Application

Getting that entire chain working was an accomplishment by itself.

But once the pipeline worked, I wanted to answer a different question:

What would it take to make this pipeline meaningfully more secure?

That became Phase 3 of the project.

And it also changed how I thought about CI/CD security.

Security Scanning Is Not the Same as Security Enforcement

One of the biggest lessons from this phase was surprisingly simple:

A security scanner finding something does not necessarily mean anything has been secured.

A scanner can produce a beautiful report containing vulnerabilities, exposed credentials, or insecure code and then allow the software to continue directly into production.

For a CI/CD pipeline to actually enforce security policy, there has to be a decision point.

In my Tekton pipeline, that decision is represented by the success or failure of a Tekton Task.

Conceptually:

Security scanner
      ↓
Finding detected
      ↓
Does it violate policy?
      ↓
     YES
      ↓
Return non-zero exit code
      ↓
Tekton Task fails
      ↓
Downstream Tasks do not execute

That distinction between detection and enforcement became the central theme of the security phase.

Gate 1: Gitleaks — Stop Secrets Early

The first security control I added was Gitleaks, an open-source secret-scanning tool.

Its job is to examine the source repository for things that look like credentials: API keys, tokens, passwords, private keys, and other secrets that should never have been committed to source control.

I deliberately placed Gitleaks immediately after source checkout:

GitHub
   ↓
Clone source
   ↓
Gitleaks
   ↓
Continue only if clean

There is a reason for putting it so early.

If a developer accidentally commits a credential, I want the pipeline to stop before that source code progresses any farther through the software supply chain.

I also tested the control instead of simply assuming that it worked.

I temporarily introduced a fake credential that Gitleaks would detect. The scanner returned a failure, Tekton marked the Task unsuccessful, and the remaining build and deployment Tasks did not run.

That gave me confidence that Gitleaks was not merely producing information—it was acting as a real pipeline gate.

Gitleaks runs immediately after source checkout, before the image is built. If committed source contains a detected secret, the pipeline terminates before that code can move farther down the supply chain.

Gate 2: Semgrep — Static Application Security Testing

The next control was Semgrep, which I used for Static Application Security Testing, or SAST.

Where Gitleaks primarily asks:

“Did someone accidentally commit a secret?”

Semgrep asks a different type of question:

“Does the source code contain a potentially dangerous programming pattern?”

I created a small Semgrep policy for the Python application and configured the pipeline to run both Semgrep's Python rules and my local security policy.

The resulting flow became:

Clone
   ↓
Gitleaks
   ↓
Semgrep
   ↓
Build

Again, the important part was not simply running Semgrep.

The important part was configuring the scanner so that a policy violation would return a non-zero exit code.

Tekton sees that exit status as a failed Task and stops the pipeline.

To validate the control, I created a temporary test branch containing intentionally unsafe Python code using:

subprocess.run(..., shell=True)

My Semgrep rule detected it.

The Semgrep Task failed.

And Tekton skipped both the image build and the deployment.

Semgrep isn't useful merely because it identifies an insecure construct. I configured it so a blocking finding returns a nonzero exit code. Tekton sees that Task failure and does not execute downstream build or deployment Tasks.

That negative test was particularly valuable because it demonstrated that the security architecture behaved the way I intended it to behave.

Gate 3: Aqua Security Trivy — Software Composition Analysis

The third major security control was Trivy, the open-source vulnerability scanner maintained by Aqua Security.

This one was especially interesting to me because Aqua Security is a technology I encounter professionally, so I wanted hands-on experience with one of the open-source components in that ecosystem.

For this phase I used Trivy primarily for Software Composition Analysis, or SCA.

SCA looks at the third-party software dependencies used by an application and asks:

“Do any of these components contain known vulnerabilities?”

That immediately exposed another interesting issue in my application.

My original Python dependency file looked roughly like this:

streamlit
openai
python-dotenv

That tells Python which packages I want, but it does not define the exact dependency versions that will ultimately be installed.

That is not ideal for reproducible builds—and it also gives a vulnerability scanner less precise information to work with.

So I introduced a common Python dependency-management pattern:

requirements.in
        ↓
   pip-compile
        ↓
requirements.txt

requirements.in describes my direct application dependencies.

requirements.txt contains the fully resolved and pinned dependency tree.

For example:

urllib3==2.7.0

instead of simply allowing the package manager to select some compatible version later.

I converted the application from loosely specified Python dependencies to a fully pinned dependency set using pip-compile. That improved build reproducibility and gave Trivy the exact software inventory it needed for meaningful SCA (Software Composition Analysis).

This turned out to be both a security improvement and a software-engineering improvement.

Visibility Versus Enforcement

For Trivy, I deliberately separated visibility from enforcement.

The first Trivy scan reports vulnerabilities at all severity levels:

UNKNOWN
LOW
MEDIUM
HIGH
CRITICAL

But I did not want every finding to automatically stop the pipeline.

Instead, I created a second enforcement pass.

The policy for Version 1 is:

LOW / MEDIUM
     ↓
Report

HIGH / CRITICAL
     ↓
Fail pipeline

In simplified form, the Tekton Task performs:

trivy fs --scanners vuln .

followed by:

trivy fs \
  --scanners vuln \
  --severity HIGH,CRITICAL \
  --exit-code 1 .

The first scan provides broad visibility.

The second scan enforces policy.

I configured Aqua Trivy to separate vulnerability visibility from enforcement. The pipeline reports dependency vulnerabilities at all severities, but only HIGH and CRITICAL findings fail the Tekton Task and prevent the image from being built or deployed.

That model feels much closer to how a real enterprise security program [should] operate.

Security policy is rarely:

“No findings of any kind are permitted.”

Instead, organizations typically have thresholds, risk tolerances, exceptions, remediation timelines, and escalation criteria.

Then I Deliberately Broke It

One of my favorite parts of the entire security phase was the negative testing.

A security control that has never been tested is difficult to trust.

So I created a temporary Git branch and deliberately changed one dependency:

urllib3==2.7.0

to:

urllib3==2.6.3

At the time of the test, Trivy identified two known HIGH-severity vulnerabilities in that version of urllib3.

The visibility scan reported both vulnerabilities.

Then the enforcement scan ran.

It found the same two HIGH findings and returned exit code 1.

Tekton responded exactly as intended:

Clone        SUCCESS
Gitleaks     SUCCESS
Semgrep      SUCCESS
Trivy SCA    FAILED

Build        SKIPPED
Deploy       SKIPPED

The Tekton PipelineRun summary showed:

Tasks Completed: 4
Failed: 1
Skipped: 2

And explicitly identified the skipped Tasks:

build-push
deploy

That was the real proof of the control.

I validated the Trivy SCA gate by deliberately introducing two HIGH-severity dependency vulnerabilities on a disposable branch. Trivy failed the Tekton Task, and Tekton automatically skipped both the container build and deployment stages.

After the test, I deleted the temporary branch and returned the application repository to the known-good dependency version.

The Secure Pipeline

At the end of this phase, the CI/CD flow looks approximately like this:

                    GitHub
                       │
                       ▼
                 Clone Source
                       │
                       ▼
               ┌─────────────┐
               │  Gitleaks   │
               │Secret Scan  │
               └──────┬──────┘
                      │
                      ▼
               ┌─────────────┐
               │   Semgrep   │
               │    SAST     │
               └──────┬──────┘
                      │
                      ▼
               ┌─────────────┐
               │    Trivy    │
               │     SCA     │
               └──────┬──────┘
                      │
             Security gates pass
                      │
                      ▼
                   BuildKit
                      │
                      ▼
                  Amazon ECR
                      │
                      ▼
                  Kubernetes
                      │
                      ▼
                    EKS
                      │
                      ▼
                     ALB
                      │
                      ▼
                  HTTPS App

Each security Task runs before the container is built.

If any blocking security condition is detected, Tekton's dependency graph prevents downstream Tasks from running.

That gives the pipeline an important property:

Known-bad software should fail before it becomes a deployable artifact.

Making the Security Controls Reproducible

Another lesson from this lab was that manually applying a security control once is not enough.

This environment was deliberately designed so that I could destroy the AWS infrastructure when I was not using it and recreate it later using Terraform and startup automation.

That meant the security controls had to survive the same lifecycle.

Gitleaks, Semgrep, and Trivy therefore became part of the lab's startup process.

When the environment is recreated, the Tekton Tasks and Pipeline definitions are restored automatically.

This may sound like a relatively small detail, but I think it represents an important infrastructure-as-code principle:

Security configuration should be reproducible along with the infrastructure it protects.

Otherwise, rebuilding an environment can quietly rebuild it without the controls that existed before.

What I Learned

I started this project primarily because I wanted more hands-on experience with Tekton, Kubernetes, EKS, and CI/CD.

I ended up learning considerably more.

The Version 1 lab now touches:

Terraform
AWS networking
Amazon EKS
Kubernetes
IAM
Pod Identity
Tekton
GitHub
BuildKit
Amazon ECR
Application Load Balancers
ACM / TLS
DNS
Gitleaks
Semgrep
Aqua Sec Trivy

But the most useful lesson from the security phase was more conceptual.

DevSecOps is not simply:

CI/CD + scanners

A more useful model is:

CI/CD
   +
Security analysis
   +
Security policy
   +
Automated enforcement

The scanner provides information.

The policy decides what matters.

The pipeline enforces the decision.

That distinction is something I understood intellectually before doing this lab, but implementing and deliberately breaking the controls made it much more concrete.

Version 1

There are many things I could still add.

Container-image scanning, SBOM generation, image signing, provenance, admission policies, deeper IAM hardening, and additional Kubernetes security controls would all be reasonable next steps.

But there is also value in declaring something finished.

So I am calling this Version 1 of the Tekton-on-EKS lab.

It began as an experiment to understand how Tekton could build and deploy an application on AWS.

It ended as a small but functioning DevSecOps environment where:

secrets, insecure source-code patterns, and vulnerable application dependencies can stop software from progressing through the delivery pipeline.

For a lab environment, that feels like a very good place to stop.

And after a couple of very long weeks, I am perfectly happy to call that Version 1.

Postmortem Summary:

Time spent on all three phases of this project, approximately 70 hours.

Tokens spent on all three phases (5.6 Sol set to high), approximately 40 million. Could be more. The largest 1 day for token usage during this project was 14.5 million.

Used this model in:

1. OpenAI Codex inside of VSCode (Codex IDE Extension)

2. The ChatGPT Desktop User Interface with ChatGPT selected (Prompt and response)

3. The ChatGPT Desktop User Interface with Codex selected - Codex built most of the Start and Stop scrips to setup and teardown the lab for cost management