DevSecOps Pipeline Documentation

⎈

Pipeline Flowchart

Visual representation of the full DevSecOps flow — from feature branch to production.

DEV

feature branch — code

Merge Request feature → dev · Code Review + Approve

Merged to dev branch

DEV Pipeline — dev branch

secret_scan

GitLeaks · TruffleHog

blocks

sast

Semgrep · Bandit · njsscan · gosec

sca

pip-audit · npm audit · govulncheck

test

Unit Tests + Coverage

blocks

build

Docker Build + Push to Harbor

blocks

container_scan

Trivy · Grype

sbom

Syft — CycloneDX + SPDX

blocks

sign

Cosign — Image Signing

blocks

promote

Auto-push to staging + save image SHA

blocks

FAIL

Fix code
re-push to dev

DEV Pipeline
result?

PASS

Auto-promote
push to staging

Auto-promoted to staging branch

STG Pipeline — staging branch (same image from DEV)

deploy_stg

Deploy SAME image → ArgoCD → K3s

blocks

smoke_stg

/health + /api checks

blocks

dast_stg

OWASP ZAP — Baseline Scan

quality_gate_stg

SonarQube — Coverage >= 70%

blocks

defectdojo_stg

Download dev artifacts via API + Upload all reports

promote_to_production

Manual — Lead/SRE approves → auto-merge staging → main

manual

FAIL

Fix → push to dev
re-flow to staging

STG Pipeline
result?

PASS

promote_to_production
Manual — Lead clicks Play
Auto-merges staging → main

Auto-merged to main branch

PROD Pipeline — main branch (same image from DEV)

deploy_prod

Deploy SAME image → ArgoCD → Production

manual

smoke_prod

/health + /api — triggers rollback if fails

blocks

dast_prod

ZAP DAST — Baseline only

defectdojo_prod

Upload PROD reports

rollback

Revert to previous image tag via ArgoCD

emergency

LIVE

https://{service}.apps.yhakkache.tech

◫

Service Overview

user-service

Language	Python 3.11
Framework	Flask
Port	:5000
Unit Tests	26 tests (99% coverage)
SAST Tools	Semgrep, Bandit
SCA Tools	pip-audit
DefectDojo	Engagement #1
Image Size	~150 MB

wallet-service

Language	Go 1.24
Framework	net/http (stdlib)
Port	:8081
Unit Tests	17 tests (80% coverage)
SAST Tools	Semgrep, gosec, go vet
SCA Tools	govulncheck
DefectDojo	Engagement #3
Image Size	~15 MB (multi-stage)

transaction-service

Language	Node.js 18
Framework	Express.js
Port	:3000
Unit Tests	22 tests (100% coverage)
SAST Tools	Semgrep, njsscan, ESLint
SCA Tools	npm audit
DefectDojo	Engagement #2
Image Size	~120 MB

frontend

Type	Static Dashboard
Server	Nginx 1.25 (Alpine)
Port	:80
Pipeline	3 stages (build → deploy → verify)
Deploy	GitOps + ArgoCD (STG + PROD)
Health	/health endpoint
Image Size	~25 MB

⟿

Pipeline Architecture

Each microservice follows the Build Once, Deploy Everywhere model: DEV → STG → PROD. The DEV pipeline builds the image once and runs all security scans. Staging and Production deploy the same image — no rebuild. This ensures reproducibility and supply chain integrity.

DEV — Scan & Build

1. Secret Scan 2. SAST 3. SCA 4. Unit Tests 5. Docker Build 6. Container Scan 7. SBOM 8. Image Signing 9. Auto-Promote

→

Staging — Same Image

1. Deploy STG 2. Smoke Test 3. DAST (ZAP) 4. Quality Gate 5. DefectDojo 6. Promote Prod 🔒

→

Production — Same Image

1. Deploy PROD 🔒 2. Smoke Test 3. DAST Baseline 4. DefectDojo 5. Rollback 🔒

Blocks on failure Allowed to fail Manual trigger

◈

Stage Details

Secret Scanning Detect hardcoded credentials before code is built ▼

First line of defense. If secrets are found, the pipeline stops immediately — no image is ever built from compromised code.

Job 1: GitLeaks Shared

Scans the current source tree with 1000+ regex patterns for API keys, tokens, and passwords.

Image: zricethezav/gitleaks:v8.18.0

Output: gitleaks-report.json

Failure: Blocks pipeline

Job 2: TruffleHog Shared

Scans git history and uses entropy analysis to detect random strings that are likely tokens — only flags verified secrets (confirmed still active). Uses --only-verified and --exclude-paths=.git to reduce false positives.

Image: trufflesecurity/trufflehog:3.63.0

Output: trufflehog-report.json

Failure: Allowed (false positives)

SAST — Static Application Security Testing Analyze source code without executing it ▼

Examines code patterns to find vulnerabilities. Each language gets its own specialized scanner in addition to shared Semgrep analysis.

Job 3: Semgrep Shared

Multi-language SAST engine with --config=auto that applies the right ruleset (p/python, p/golang, p/javascript, p/owasp-top-10).

Image: semgrep/semgrep:latest

Output: semgrep-report.json

Failure: Allowed (false positives)

Job 4: Language-Specific SAST Per-Language

Detail

Python — Bandit

Go — gosec

Node.js — njsscan

Image

python:3.11-slim

golang:1.24-alpine

python:3.11-slim

Detects

eval(), subprocess, pickle, weak crypto

Hardcoded creds, SQL concat, weak crypto

eval(), child_process, prototype pollution

Output

bandit-report.json

gosec-report.json

njsscan-report.json

Job 5: Linter / Extra SAST Go & Node.js only

Go: go vet — printf format mismatches, unreachable code, mutex issues.
Node.js: eslint + eslint-plugin-security — object injection, non-literal regexp (ReDoS), eval patterns.

SCA — Software Composition Analysis Scan third-party dependencies for known vulnerabilities ▼

~80% of modern applications consist of third-party code. SCA checks those dependencies against vulnerability databases.

Detail

Python

Node.js

Primary Tool

pip-audit (PyPI/OSV DB)

govulncheck (vuln.go.dev)

npm audit (npm Advisory)

Secondary Tool

—

Why This Tool

PyPI + OSV DB coverage

Official Go tool; only flags called functions

Built into npm; covers deep dep trees

Unit Tests + Coverage Verify application logic before building ▼

Detail

Python

Node.js

Framework

pytest + pytest-cov

go test (built-in)

Jest + Supertest

Tests

26 tests, 7 groups

17 tests, 8 groups

22 tests, 7 groups

Coverage

99%

80%

100%

Coverage Format

Cobertura XML

coverage.out → Cobertura

LCOV

Report

junit-report.xml

junit.xml

Race Detection

—

✓ -race flag

—

Failure

Blocks pipeline

Docker Build + Push to Harbor Build container image and push to private registry ▼

Job: docker_build Shared

Image: docker:27-cli

Registry: devsecops-demo-harbor.yhakkache.tech

Tags: ${IMAGE_NAME}:${CI_COMMIT_SHORT_SHA} + :latest

Method: Host Docker socket mount (/var/run/docker.sock) — no DinD service

Dockerfile

Python

Node.js

Base Image

python:3.11-slim

Multi-stage: golang:1.24 → alpine:3.18

node:18-alpine

Result Size

~150 MB

~15 MB

~120 MB

Install

pip install

go build → binary only

npm ci --production

Container Scanning Scan the built image for OS-level and library vulnerabilities ▼

Job 9: Trivy (Aqua Security) Shared

Image: aquasec/trivy:latest

Severity: HIGH, CRITICAL

Output: trivy-report.json

Job 10: Grype (Anchore) Shared

Image: alpine:3.18 + curl install grype (anchore image is distroless — no shell)

Output: grype-report.json

Failure: Allowed

SBOM — Software Bill of Materials Full inventory of every component in the image ▼

Produces a complete list of all packages, libraries, and their versions — essential for supply chain security and compliance. CycloneDX (OWASP) + SPDX (ISO). Retained 30 days.

Job 11: Syft (Anchore) Shared

Image: alpine:3.18 + curl install syft (anchore image is distroless — no shell)

Outputs: sbom-cyclonedx.json, sbom-spdx.json

Retention: 30 days

Image Signing — Cosign Cryptographically sign the image to prevent tampering ▼

Signs the Docker image with a private key via Sigstore Cosign, then verifies the signature. Kubernetes can enforce that only signed images are deployed.

Job 12: Cosign (Sigstore) Shared

Image: bitnami/cosign:latest

Sign: cosign sign --key env://COSIGN_KEY

Verify: cosign verify --key env://COSIGN_PUB

Promote — Auto-push to Staging Push code to staging branch + save image tag for reuse ▼

Final DEV stage. Uses git push to auto-promote code from dev to staging branch, which triggers the STG pipeline. Also saves $IMAGE_TAG to .promote-image-tag artifact so staging/prod deploy the exact same image — no rebuild.

Job 13: promote_to_staging

Mechanism: git push origin HEAD:staging

Artifact: .promote-image-tag (contains $CI_COMMIT_SHORT_SHA)

Failure: Blocks pipeline

Deploy to Staging (GitOps) Update image tag in GitOps repo → ArgoCD auto-syncs ▼

Uses the GitOps pattern: the pipeline clones the infra-gitops repo using oauth2:${GITLAB_PAT} (cross-project access), reads the image tag from .promote-image-tag, updates the newTag in the Kustomize overlay, commits, and pushes. ArgoCD watches the repo and automatically deploys to K3s.

Job 14: deploy_staging

Image: alpine:3.18

Mechanism: sed → update kustomization.yaml → git push

Wait: 30s for ArgoCD sync

Failure: Blocks pipeline

Smoke Test (Staging) Verify the deployed application is alive and responding ▼

Uses wget --no-check-certificate on alpine:3.18 instead of curl (Alpine musl libc has TLS read issues with HAProxy). Runner uses network_mode: host for direct access to the cluster network.

Job 15: smoke_test_stg Shared

Image: alpine:3.18

Tool: wget --spider --no-check-certificate

Target: https://stg-${APP_NAME}.apps.yhakkache.tech

Failure: Blocks pipeline

Endpoint

Python

Node.js

Health

/health

API

/api/users

/api/wallets

/api/logs

DAST — Dynamic Application Security Testing Attack the running application to find runtime vulnerabilities ▼

OWASP ZAP performs a baseline scan against the live staging application, testing for SQL injection, XSS, CSRF, missing security headers, and information disclosure.

Job 16: OWASP ZAP (Staging) Shared

ZAP Docker image requires /zap/wrk to exist for file-based output. The job creates this directory with mkdir -p /zap/wrk, then copies reports to the build directory for artifact collection.

Image: ghcr.io/zaproxy/zaproxy:stable

Mode: Baseline scan (5–10 min)

Workdir: mkdir -p /zap/wrk → cp /zap/wrk/zap-*.* .

Outputs: zap-stg-report.json, zap-stg-report.html

SonarQube Quality Gate Code quality decision point — pass or no production ▼

SonarQube v10.7.0 (self-hosted) analyzes code quality, coverage, bugs, and security hotspots. Configuration is read from sonar-project.properties at the repo root. With sonar.qualitygate.wait=true, the pipeline blocks until the result is returned.

Job 17: sonarqube_check Shared

Image: sonarsource/sonar-scanner-cli:latest

Instance: https://devsecops-demo-sonar.yhakkache.tech

Config: sonar-project.properties (projectKey, sources, exclusions, test paths, JUnit report)

Quality Gate: Sonar way (built-in, compliant)

Failure: Blocks pipeline — no production without passing

Metric	Threshold
Coverage	≥ 70%
Critical Bugs	0
Security Hotspots	All reviewed
Code Smells	Acceptable level
Duplicated Code	< 3%

DefectDojo — Vulnerability Management Centralize all scan reports for tracking and triage ▼

Runs on the staging branch. Since scan artifacts are produced on the dev branch (different pipeline), the job downloads them via the GitLab API: curl --header "PRIVATE-TOKEN: ${GITLAB_PAT}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/artifacts/dev/download?job={JOB_NAME}" → unzip. The ZAP STG report is available from the current pipeline's dast_stg stage. Each service maps to a dedicated DefectDojo engagement.

Cross-Pipeline Artifact Download Architecture Note

GitLab limits needs: to 5 entries, and needs:project cannot use $CI_PROJECT_PATH variables. The solution: download artifacts from the latest successful dev pipeline via the GitLab REST API using GITLAB_PAT, then unzip them into the build directory before uploading to DefectDojo.

Report	Scan Type	Source Stage
`gitleaks-report.json`	Gitleaks Scan	Secret Scan
`trufflehog-report.json`	Trufflehog Scan	Secret Scan
`semgrep-report.json`	Semgrep JSON Report	SAST
`bandit / gosec / njsscan`	Per-language scan type	SAST
`pip-audit / govulncheck / npm-audit`	Per-language scan type	SCA
`trivy-report.json`	Trivy Scan	Container Scan
`grype-report.json`	Anchore Grype	Container Scan
`sbom-cyclonedx.json`	CycloneDX Scan	SBOM
`zap-stg-report.json`	ZAP Scan	DAST

Promote to Production 🔒 Manual lead approval — merge staging → main ▼

The gateway to production. Runs on the staging branch with when: manual. A Tech Lead or SRE must click "Play" in the GitLab UI. The job then auto-merges staging into main via git push origin staging:main, triggering the PROD pipeline.

Job: promote_to_production Shared

Branch: staging

Trigger: Manual — Lead/SRE approval

Mechanism: git push origin staging:main

Auth: oauth2:${GITLAB_PAT}

Result: Triggers PROD pipeline on main branch

Deploy to Production 🔒 Manual approval required — human-in-the-loop ▼

Runs on main branch. Same GitOps mechanism as staging — clones infra-gitops via oauth2:${GITLAB_PAT}, reads the staging image tag from the staging kustomization overlay, and updates the production overlay. Requires when: manual — a team member must explicitly click "Deploy". ArgoCD auto-syncs the production namespace.

Smoke Test (Production) Verify production is alive — failure triggers rollback consideration ▼

Same health + API checks as staging, using wget on alpine:3.18. Targets https://${APP_NAME}.apps.yhakkache.tech (no prod- prefix — matches SSL certificate SANs). If this fails, the rollback job becomes immediately relevant.

DAST (Production Baseline) Lightweight security scan on production ▼

Baseline-only ZAP scan on production using mkdir -p /zap/wrk. A full scan would generate too much traffic and risk performance impact on live users.

DefectDojo (Production Reports) Upload production ZAP scan to DefectDojo ▼

Uploads the production ZAP report to DefectDojo for tracking alongside staging findings. Uses python:3.11-slim with apt-get install curl (Alpine curl has musl TLS issues with HAProxy).

Rollback 🔒 Emergency — revert to previous version ▼

Manual trigger, emergency-only. Clones infra-gitops via oauth2:${GITLAB_PAT} and uses CI_COMMIT_BEFORE_SHA to roll back to the previous image tag. ArgoCD detects the change and redeploys the old version.

≡

Tool Comparison Across Pipelines

#	Stage	Python	Go	Node.js	Shared?
1-2	Secret Scan	GitLeaks + TruffleHog	GitLeaks + TruffleHog	GitLeaks + TruffleHog	✓
3	SAST (generic)	Semgrep	Semgrep	Semgrep	✓
4	SAST (lang)	Bandit	gosec	njsscan	✗
5	Linter	—	go vet	ESLint + security	✗
6	SCA (primary)	pip-audit	govulncheck	npm audit	✗
7	SCA (secondary)	—	—	—	—
8	Tests	pytest (26)	go test (17)	Jest (22)	✗
9	Build	Docker CLI (socket)	Docker CLI (socket)	Docker CLI (socket)	✓
10-11	Container Scan	Trivy + Grype	Trivy + Grype	Trivy + Grype	✓
12	SBOM	Syft	Syft	Syft	✓
13	Sign	Cosign	Cosign	Cosign	✓
14	Deploy	GitOps + ArgoCD	GitOps + ArgoCD	GitOps + ArgoCD	✓
15	Promote Prod	git push (manual)	git push (manual)	git push (manual)	✓
16	Smoke	wget (Alpine)	wget (Alpine)	wget (Alpine)	✓
17	DAST	OWASP ZAP	OWASP ZAP	OWASP ZAP	✓
18	Quality Gate	SonarQube	SonarQube	SonarQube	✓
19	Vuln Mgmt	DefectDojo	DefectDojo	DefectDojo	✓

⬡

Pipeline Artifacts

Every pipeline run produces the following artifacts, stored for 7 days (SBOM for 30 days).

gitleaks-report.json

Secret Scan · GitLeaks · 7d

trufflehog-report.json

Secret Scan · TruffleHog · 7d

semgrep-report.json

SAST · Semgrep · 7d

bandit / gosec / njsscan

SAST · Per-language · 7d

pip-audit / govulncheck / npm-audit

SCA · Per-language · 7d

coverage.xml / .out / lcov

Tests · Coverage · 7d

junit-report.xml

Tests · JUnit · 7d

trivy-report.json

Container Scan · Trivy · 7d

grype-report.json

Container Scan · Grype · 7d

sbom-cyclonedx.json

SBOM · CycloneDX (OWASP) · 30d

sbom-spdx.json

SBOM · SPDX (ISO) · 30d

zap-*-report.json/html

DAST · OWASP ZAP · 7d

⚿

CI/CD Variables

Configured in GitLab → Settings → CI/CD → Variables.

Variable	Description	Masked	Used By
`HARBOR_USER`	Harbor registry username	✓	Build, Container Scan, SBOM, Sign
`HARBOR_PASSWORD`	Harbor registry password	✓	Build, Container Scan, SBOM, Sign
`SONAR_HOST_URL`	SonarQube instance URL	✗	Quality Gate
`SONAR_TOKEN`	SonarQube project analysis token	✓	Quality Gate
`DEFECTDOJO_TOKEN`	DefectDojo API token	✓	DefectDojo uploads
`COSIGN_KEY`	Cosign private key (signing)	✓	Image signing
`COSIGN_PASSWORD`	Cosign key passphrase	✓	Image signing
`COSIGN_PUB`	Cosign public key (verification)	✗	Image verification
`GITLAB_PAT`	Personal Access Token for cross-project access	✓	Promote, Deploy STG, Deploy PROD, Rollback, DefectDojo artifact download

Infrastructure

Self-hosted on DigitalOcean + GCP, interconnected via Tailscale mesh VPN.

GitLab.com

Source control & CI/CD engine (SaaS)

SaaS + Self-Hosted Runner

Harbor

Private container registry

Self-Hosted

SonarQube

Code quality & coverage

Self-Hosted

DefectDojo

Vulnerability management

Self-Hosted

K3s Cluster

Kubernetes (master + worker)

Self-Hosted

ArgoCD

GitOps CD — auto-syncs from infra-gitops repo

Self-Hosted

⊘

Pipeline Stop Points

These jobs block the pipeline on failure — the security and quality gates.

#	Stage	Job	Rationale
1	Secret Scan	GitLeaks	Secrets in code must never be built into an image
2	Secret Scan	TruffleHog	allow_failure — only verified secrets; false positives common
3	Test	Unit Tests	Broken logic must not be deployed
4	Build	Docker Build	No image = nothing to deploy
5	SBOM	Syft	Blocks pipeline — supply chain transparency is mandatory
6	Sign	Cosign	Blocks pipeline — unsigned images must not be deployed
7	Deploy STG	GitOps Deploy	Must deploy to staging before validation
8	Smoke STG	Health Check	Application must be alive before further testing
9	Quality Gate	SonarQube	Code quality threshold — no production without passing
10	Promote PROD	promote_to_production	Manual — Lead/SRE approves merge to main
11	Deploy PROD	GitOps Deploy	Manual — requires human approval
12	Smoke PROD	Health Check	Production failure = immediate rollback consideration
13	Rollback	Rollback	Manual — emergency revert