Objective: Master the long-term management of your infrastructure. We cover advanced log analysis using Gemini 1.5 Pro, cost tracking, emergency “Break Glass” procedures, and the maintenance rhythm.

1. Log Analysis with AI

When Vercel throws a 500, you get a graph. When Docker throws a 500, you get a 100MB log file. We use Gemini to read it.

The Workflow

1. Dump: docker logs --since 1h <container> > crash.log.

2. Analyze: Pipe to Gemini.

“Analyze this log. Find the stack trace. Correlate with Postgres logs. Explain the root cause.”

2. Cost Analysis (The Victory Lap)

Scenario: 500k Users/mo.

• Vercel: ~$550/mo.

• Self-Hosted: ~$17.25/mo.

• Savings: ~$6,393/yr.

Use this to justify your next hardware upgrade.

3. Emergency Procedures (”Break Glass”)

A. Disk Full

Symptom: DB stops writing. Fix:

1. ncdu /.

2. Truncate Docker logs: truncate -s 0 ....

3. Prune images: docker system prune.

B. 502 Bad Gateway

Symptom: Site down. Fix:

1. Check Traefik logs.

2. Restart App container.

3. Check OOM: dmesg | grep killed.

4. TLS Expiry Monitoring

Cloudflare handles the Edge certificate, but in Full (Strict) mode, it validates your Origin Certificate (managed by Traefik on apps-vps). If Traefik fails to renew, your site goes down (Error 526).

We monitor the Origin Cert directly.

Agent: Claude Code Prompt:

“Create a script /opt/monitoring/scripts/check_coolify_origin_cert.sh.

1. It should connect to localhost:443 (Traefik).

2. Extract the expiration date of the certificate.

3. If expiry < 7 days, send an alert to Alertmanager/SES. Add this to the daily cron.”

5. The Maintenance Rhythm

• Daily: Check Backup emails.

• Monthly: Restore Drill (Automated).

• Quarterly: OS Updates (apt upgrade). Reboot.

6. Conclusion

You are no longer a tenant. You are the landlord. You own the metal. You own the data. You are the Architect.

Objective: Execute the Go-Live sequence. We will perform a zero-downtime DNS switch using Cloudflare, ensure email deliverability with DKIM/SPF/DMARC, and verify SSL propagation.

1. The Pre-Flight Checklist (T-Minus 24 Hours)

1. TTL: Lower DNS TTL to 60s.

2. Freeze: Code freeze on Vercel.

3. Dry Run: Run the import script on db-vps (delete test data first).

2. The Cutover Sequence (T-Minus 0)

Step 1: Maintenance Mode

We stop writes to the old DB. Agent: Claude Code Prompt:

“Create a maintenance page. Configure Vercel to redirect all traffic to it (or break the DB connection).”

Step 2: The Migration

1. Export: Run export.sh.

2. Transfer: scp to db-vps.

3. Import: Run import.sh.

4. Verify: Check row counts.

Step 3: The Switch

Agent: Claude Code (Local) Prompt:

“Create a script switch_dns.sh using curl against Cloudflare API. Update the A record for @ to the Apps VPS IP (144.126.x.x). Ensure proxied: true.”

Step 4: Validation

Check https://app.yourdomain.com. Test Login. Test DB write.

3. Email Deliverability (DKIM/SPF)

Your app sends emails. If you don’t fix DNS, they go to Spam.

DKIM

AWS gives you 3 CNAMEs. Crucial: These must be DNS Only (Grey Cloud) in Cloudflare. If proxied, validation fails.

SPF & DMARC

Prompt:

“Create a script setup_email_dns.sh using curl to add:

1. SPF TXT: v=spf1 include:amazonses.com ~all.

2. DMARC TXT: v=DMARC1; p=none; rua=mailto:admin@.... Ensure proxied: false.”

4. Context Update

Agent: Codex Prompt:

“System is LIVE. DNS points to VPS. Email authenticated. Update INFRASTRUCTURE_CONTEXT.md.”

Next Step: You are live. Now you live with it. Go to 14. Day 2 Operations.

Objective: Configure the application environment. We will securely map secrets in Coolify, update OAuth providers to trust our new domain, and deploy persistent background workers to handle long-running tasks.

1. Secrets Management with Bitwarden (BWS)

Stop pasting .env files into web forms. We use Bitwarden Secrets Manager (bws) to maintain a single source of truth for secrets.

The Workflow

1. Local: You update secrets in Bitwarden.

2. Inject: You use bws to generate the env block for Coolify.

Agent: Claude Code (Local) Prompt:

“I need to generate the environment variables for Coolify using bws.

1. List secrets in project ‘My App’: bws secret list --project-id <ID>.

2. Format them as KEY=VALUE.

3. Replace DATABASE_URL with the production internal URL (PgBouncer).

4. Output the block so I can paste it into Coolify (or script the API call if Coolify API is available).”

Note: We also maintain a custom tool bws-init (see your 20 - Development/21 - Projects/Bws Init.md) to initialize this structure.

The Map

• DATABASE_URL: postgres://user:pass@100.x.y.z:6432/db (Use Tailscale IP + PgBouncer Port).

• NEXTAUTH_URL: https://app.yourdomain.com (Must match exactly).

• NEXTAUTH_SECRET: Keep the old one to preserve user sessions.

2. Fixing OAuth (The “Login Failed” Error)

Google and GitHub only trust the domains you whitelist. Action:

1. Go to Google Cloud Console -> Credentials.

2. Add https://app.yourdomain.com/api/auth/callback/google.

3. Repeat for GitHub.

3. Background Workers (The Superpower)

Vercel limits functions to 60s. We can now run tasks for hours.

Redis

Deploy Redis on apps-vps via Coolify (Internal access only).

The Worker Service

1. Clone your Web App in Coolify.

2. Change Start Command: npm run worker.

3. This runs the same code but starts the queue processor instead of the web server.

Dashboard

Mount Bull Board at /admin/queues. Security: Protect this route with Middleware!

4. Context Update

Agent: Codex Prompt:

“Production Config Complete. Secrets managed via bws. OAuth updated. Workers active. Update INFRASTRUCTURE_CONTEXT.md.”

Next Step: The system is built. The data is migrated. It is time to flip the switch. Go to 13. The Cutover Event.

Objective: Extract your entire database from Supabase (including Auth users), sanitize it, and hydrate your new Postgres instance. This is the most dangerous phase; we will script it meticulously.

1. The Export Strategy

Supabase locks down superuser access. pg_dumpall fails. We need a surgical extraction.

The Export Script (export.sh)

Agent: Claude Code Prompt:

“Write export.sh accepting SUPABASE_URL.

1. Dump auth schema (Schema + Data).

2. Dump public schema (Schema + Data).

3. Exclude: storage, realtime, graphql (Supabase internals).

4. Use --clean --if-exists flags.”

2. The Import Logic

A. Roles

Supabase uses anon, authenticated, service_role. We must create these in our new DB, or the import will fail.

Prompt:

“Create setup_roles.sql. Create roles if not exist. Grant usage on public schema.”

B. Extensions

Supabase has 50 extensions. We only need uuid-ossp and pgcrypto. We install these manually in the import script.

3. The Sequence Trap

Critical: When you insert data with explicit IDs, Postgres sequences do not update. Next insert -> Duplicate Key Error.

Prompt:

“Write a SQL snippet to reset all sequences in public schema to MAX(id).”

4. The Master Import Script (import.sh)

Prompt:

“Combine everything into import.sh on db-vps.

1. Create Roles.

2. Create Extensions.

3. Import Auth.

4. Import Public.

5. Reset Sequences.

6. Run ANALYZE.”

5. Verification

Prompt:

“Run import. Compare SELECT count(*) FROM auth.users between Supabase and Local.”

Next Step: Data is live. App is ready. Let’s wire them together. Go to 12. Production Configuration.

Objective: Transform your Vercel-optimized Next.js application into a lean, mean Docker container. We will use Multi-Stage builds and output: standalone to shrink the image size from 1GB+ to <100MB, ensuring fast deployments and minimal attack surface.

1. The “Standalone” Magic

Vercel’s platform automatically traces your code dependencies. When self-hosting, we must replicate this. Next.js has a feature called Output Standalone. It creates a .next/standalone folder containing a minimal server.js and only the necessary node_modules.

Configuration

Agent: Claude Code Prompt:

“Edit next.config.mjs. Add output: ‘standalone’. Explain why this reduces image size.”

2. The Multi-Stage Dockerfile

We don’t want our production image to contain TypeScript compilers, Webpack, or dev dependencies. We use a 3-Stage Build:

1. Deps: Install dependencies (cached layer).

2. Builder: Copy source, run npm run build.

3. Runner: Copy only .next/standalone, public, and .next/static.

Agent: Claude Code Prompt:

“Create a production Dockerfile. Base: node:20-alpine. User: nextjs (Non-root security). Env: NEXT_TELEMETRY_DISABLED=1. Expose: 3000. Command: node server.js.”

3. The Build-Time vs. Run-Time Trap

NEXT_PUBLIC_ Variables: These are inlined into the JavaScript bundle at Build Time. If you build your image on GitHub Actions, NEXT_PUBLIC_API_URL is baked in. Solution:

• Either build the image on the server (Coolify does this).

• Or use “Runtime Configuration” patterns (advanced). For this course, we rely on Coolify building the image, so secrets are injected during the build.

4. Local Verification

Prompt:

“Build locally: docker build -t my-app . Run: docker run -p 3000:3000 my-app Verify size: docker images | grep my-app.”

Next Step: Code is ready. Data is trapped in Supabase. Go to 11. The Great Data Migration.

Objective: Implement a “Paranoid” backup strategy. We will script automated nightly dumps to Cloudflare R2 (S3 Compatible) and, crucially, create an automated “Restore Drill” that verifies the backup integrity every month.

1. The Hybrid Backup Strategy

We do not rely on a single backup method. We use a Hybrid Strategy:

1. WAL-G (Primary): Continuously streams Write Ahead Logs (WAL) to Cloudflare R2.

   • Benefit: Point-in-Time Recovery (PITR). Can restore to any second.

   • Speed: Extremely fast binary restore.

2. Logical Dumps (Secondary): Nightly pg_dumpall to local disk (retained 7 days).

   • Benefit: Portable. Human-readable SQL. Can restore to different Postgres versions.

   • Safety: If R2 credentials fail, we have local copies.

2. R2 Integration

Cloudflare R2 has zero egress fees. It is perfect for both WAL archives and dump storage.

Agent: Claude Code Prompt:

“Create R2 bucket postgres-backups via Cloudflare Dashboard or API (curl). Install aws-cli and wal-g binary on db-vps. Configure wal-g environment variables (WALG_S3_PREFIX, AWS_ACCESS_KEY_ID, etc.) pointing to the R2 bucket.”

3. The Backup Script (backup.sh) - Logical

We place this in /opt/postgres/scripts/backup.sh.

Features:

1. Dump: docker exec ... pg_dumpall | gzip.

2. Upload: (Optional) Upload to R2 (separate prefix) or keep local.

3. Retention: Delete local > 7 days.

4. Logging: Write to /var/log/postgres_backup.log.

4. The Restore Drill (restore_drill.sh)

This is the most important script you will ever write. It runs on the 1st of the month via cron (0 4 1 * *).

Workflow (WAL-G Validation):

1. Use wal-g backup-fetch to download the latest base backup to a scratch directory (/tmp/walg-restore-test).

2. Run pg_controldata on the fetched data directory to validate the backup integrity.

3. Check that Database cluster state shows a valid state.

4. Log results to /var/log/postgres-restore-drill.log.

5. Clean up scratch directory (use --keep flag to preserve for deeper inspection).

Agent: Claude Code Prompt:

“Create /opt/postgres/scripts/restore_drill.sh:

#!/bin/bash

set -euo pipefail

LOG=/var/log/postgres-restore-drill.log

SCRATCH=/tmp/walg-restore-test

KEEP=false

[[ “${1:-}” == “--keep” ]] && KEEP=true

echo “$(date): Starting restore drill” >> $LOG

rm -rf $SCRATCH && mkdir -p $SCRATCH

# Fetch latest backup

docker exec postgres_db wal-g backup-fetch $SCRATCH LATEST 2>>$LOG

# Validate with pg_controldata

RESULT=$(docker exec postgres_db pg_controldata $SCRATCH 2>>$LOG | grep ‘Database cluster state’ || echo ‘FAILED’)

echo “$(date): $RESULT” >> $LOG

if [[ “$KEEP” == “false” ]]; then

rm -rf $SCRATCH

echo “$(date): Cleanup complete” >> $LOG

fi

1. Make executable: chmod +x /opt/postgres/scripts/restore_drill.sh

2. Add monthly cron: 0 4 1 * * root /opt/postgres/scripts/restore_drill.sh >> /var/log/postgres-restore-drill-cron.log 2>&1

3. Test manually: /opt/postgres/scripts/restore_drill.sh --keep to verify.”

For deeper validation, use --keep and spin up a temporary Postgres container on a different port to run actual queries against the restored data.

5. Restore Drill Monitoring

A backup that isn’t tested is worthless. We need alerts if the drill stops running.

Agent: Claude Code Prompt:

“Create /opt/postgres/scripts/restore_drill_monitor.sh:

#!/bin/bash

LOG=/var/log/postgres-restore-drill.log

WARN_DAYS=35

if [[ ! -f $LOG ]]; then

echo “No restore drill log found!”

exit 1

fi

LAST_RUN=$(stat -c %Y $LOG)

NOW=$(date +%s)

DAYS_AGO=$(( ($NOW - $LAST_RUN) / 86400 ))

if [[ $DAYS_AGO -gt $WARN_DAYS ]]; then

# Post alert to Alertmanager

curl -s -X POST http://localhost:9093/api/v2/alerts \

-H ‘Content-Type: application/json’ \

-d ‘[{”labels”:{”alertname”:”RestoreDrillStale”,”severity”:”warning”},”annotations”:{”summary”:”Restore drill not run in ‘$DAYS_AGO’ days”}}]’

fi

1. Add daily cron: /etc/cron.d/restore_drill_monitor

2. This fires a RestoreDrillStale alert if no drill has run in 35+ days.”

6. Context Update

Agent: Codex Prompt:

“Backups & Drills Configured. Target: Cloudflare R2. Schedule: Daily logical backup (0 2 * * *), Monthly restore drill (0 4 1 * *). Monitoring: RestoreDrillStale alert fires if drill is >35 days old. Update INFRASTRUCTURE_CONTEXT.md.”

Next Step: The infrastructure is complete. Time to move the Application. Go to 10. Containerizing Next.js.

Objective: Architect the dedicated db-vps. We will deploy PostgreSQL 17 via Docker Compose, tune it for our specific hardware using AI calculations, and implement PgBouncer to solve the “Thundering Herd” problem of serverless connections.

Note: PostgreSQL 17 (released September 2024) brings improvements including incremental backup support, enhanced COPY performance, and improved JSON support.

1. The “Pet” Philosophy

We treat db-vps differently.

• No Auto-Updates: We pin Docker image versions (postgres:17-alpine).

• No Abstractions: We use raw docker-compose.yml.

• Manual Tuning: We edit postgresql.conf.

2. The Docker Stack

Directory Structure

/opt/postgres

├── data/ (Volume)

├── config/ (postgresql.conf, pg_hba.conf, pgbouncer.ini)

└── docker-compose.yml

Agent: Claude Code Prompt:

“Create this structure on db-vps. Write docker-compose.yml: Services: postgres (17-alpine), pgbouncer (edoburu/pgbouncer). Critical: Set shm_size: 1gb for Postgres (default 64m crashes). Network: Bind to 127.0.0.1 or specific Tailscale IP.”

3. AI-Driven Configuration Tuning

Default Postgres is tuned for 512MB RAM. We have 8GB.

Agent: Gemini CLI Prompt:

“I have 8GB RAM, 6 vCPU. Workload: Web App (Read Heavy). Generate postgresql.conf. Calculate:

• shared_buffers (2GB)

• effective_cache_size (6GB)

• maintenance_work_mem (512MB)

• work_mem (16MB)

• max_connections (100 - Let PgBouncer handle the rest). Provide the file content.”

4. PgBouncer: The Connection Multiplexer

Next.js Serverless functions open a new TCP connection for every request. 1000 requests = 1000 Connections = Crash.

PgBouncer maintains a pool of connections to Postgres. It shares them among clients.

Configuration (pgbouncer.ini)

• Mode: transaction. (Client keeps connection only for query duration).

• Max Client Conn: 400. (Allows high concurrency from Next.js).

• Default Pool Size: 40. (Enough real connections for 6 vCPU).

Agent: Claude Code Prompt:

“Generate pgbouncer.ini and userlist.txt. Use pool_mode = transaction. Set max_client_conn = 400. Set default_pool_size = 40. Set listen_port = 6543. Verify connection from apps-vps using psql -h db-vps -p 6543.”

5. Context Update

Agent: Codex Prompt:

“Database Active. PostgreSQL 17 Tuned (direct connections on port 5432). PgBouncer listening on Port 6543 (Tailscale) for pooled connections. Update INFRASTRUCTURE_CONTEXT.md.”

Note: Port 6543 is used to avoid conflicts with any local PostgreSQL installations that might use the default 6432.

Next Step: The DB is fast. Is it safe? Go to 09. Data Durability Strategy.

Objective: Configure Cloudflare as our global shield using curl and the Cloudflare API. We will implement “Full (Strict)” SSL, configure HSTS, and verify that Coolify’s internal Traefik proxy correctly handles certificate generation behind Cloudflare’s proxy.

1. The Edge Strategy

We are not exposing our server directly. We are putting Cloudflare in front.

• DDoS Protection: Cloudflare absorbs attacks. Our IP is hidden.

• Performance: CDN caches static assets at the edge.

• SSL Offloading: Cloudflare handles the heavy crypto.

The Tool: Cloudflare API (via curl). We don’t click buttons in the UI. We write Infrastructure as Code using shell scripts.

2. DNS Configuration via API

We will manage our DNS records programmatically using the Cloudflare API. This allows us to version control our infrastructure.

Step 1: The Setup Script

Create a script setup_dns.sh locally. You will need your Zone ID and API Token.

Agent: Claude Code (Local) Prompt:

“Create a bash script setup_dns.sh to add Cloudflare DNS records using curl. Variables: ZONE_ID, TOKEN, IP=144.126.x.x, TAILSCALE_IP=100.x.y.z. Records to Create:

1. Root (@): A record to $IP, proxied (true).

2. Wildcard (*): CNAME to @, proxied (true).

3. Coolify (coolify): A record to $TAILSCALE_IP, proxied (false) - DNS Only.

The script should use: curl -X POST “https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records” ... “

Step 2: Execution

Run the script locally: export TOKEN=... ZONE_ID=...; bash setup_dns.sh

3. SSL Architecture: “Full (Strict)”

This is the most misunderstood setting.

• Flexible: Encrypted to Cloudflare, Unencrypted to Server. BAD.

• Full: Encrypted to Server, but accepts Self-Signed certs. OKAY.

• Full (Strict): Encrypted to Server, requires Valid Trusted Cert. BEST.

We choose Full (Strict). This means our apps-vps MUST have a valid Let’s Encrypt certificate.

Traefik & Let’s Encrypt

Coolify uses Traefik. Traefik requests certs. The Magic: Cloudflare is smart enough to pass /.well-known/acme-challenge requests through to the origin, even when proxied.

4. Deployment Verification

Let’s prove it works.

Agent: Claude Code Prompt:

“Deploy a ‘Hello World’ Next.js app in Coolify. Domain: test.app.com. Verify:

1. curl -I https://test.app.com. Check for cf-cache-status.

2. Check SSL issuer: openssl s_client .... Should be Let’s Encrypt (on the backend) or Cloudflare (on the frontend).

3. Verify HTTP->HTTPS redirect.”

5. HSTS & Security Headers

HSTS (HTTP Strict Transport Security) tells browsers: “Never talk to me via HTTP again.” Enable this in Cloudflare Dashboard -> SSL/TLS -> Edge Certificates. Warning: Once enabled, you cannot downgrade to HTTP for months.

6. TLS Certificate Monitoring

You now have two layers of certificates to monitor:

Agent: Claude Code Prompt:

“We need to monitor both certificate layers.

1. Edge (Cloudflare): Use the TLS expiry script from Module 06 pointing at your public domain.

2. Origin (Traefik): Create check_coolify_origin_cert.sh to inspect Traefik’s cert files or connect directly to http://localhost:8000 (bypassing Cloudflare).

3. Add to cron alongside the edge check. Why both? Cloudflare can mask origin cert failures until you disable proxy mode.”

Reference: See 06. Observability Stack for the full monitoring script.

7. Context Update

Agent: Codex Prompt:

“Edge Layer Active. DNS: Managed via Cloudflare API scripts. SSL: Full (Strict). Security: Break-glass account created, 2FA active. Update INFRASTRUCTURE_CONTEXT.md.”

Next Step: The App Server is ready. Now for the heavy lifting. Go to 08. High-Performance Database.

Objective: Deploy a “Survival” monitoring stack. If Coolify crashes, we need to know why. Therefore, we run monitoring outside of Coolify using standard Docker Compose. We will implement Prometheus, Node Exporter, Uptime Kuma, and Alertmanager.

1. The “Separate Stack” Philosophy

Why not run Monitoring inside Coolify?

• Circular Dependency: If Coolify’s database fails, Coolify goes down. If Coolify manages monitoring, monitoring goes down. You are flying blind.

• Resources: Monitoring needs guaranteed CPU. We don’t want a Next.js memory leak to kill Prometheus.

2. The Stack Architecture

We deploy to /opt/monitoring on apps-vps. Note: While we run this on the same Host (Apps VPS) for cost efficiency, it runs as a completely separate Docker Compose stack from Coolify. If Coolify’s Docker network implodes, this stack (on the host network or its own bridge) survives.

• Prometheus: The brain. Scrapes metrics every 15s.

• Node Exporter: The sensor. Reports CPU, RAM, Disk, Network.

• Uptime Kuma: The dashboard. Pretty graphs for “Is Google Up?”

• Alertmanager: The dispatcher. Sends emails via SES.

• Custom Scripts: We add bash scripts to monitor TLS expiry and Backup freshness.

• UptimeRobot (External): Third-party HTTP(S) checks every 60s with email alerts (contact: chris@heavisidegroup.com). Managed via API, not stored in Coolify.

3. Deployment via AI

Step 1: Directory Setup

Agent: Claude Code Prompt:

“SSH into apps-vps. Create /opt/monitoring. Create subdirs prometheus, alertmanager, uptime-kuma-data, scripts. Chown to deploy user.”

Step 2: Docker Compose

Prompt:

“Create /opt/monitoring/docker-compose.yml. Services: prometheus (9090), node-exporter (9100), alertmanager (9093), uptime-kuma (3001). Security: Bind ports to the Tailscale IP (100.x.x.x), not 127.0.0.1. This allows direct access from your Tailnet while keeping them off the public internet. Example: - ‘100.77.226.26:9090:9090’ Volumes: Map config files and persistent data.”

Alternative: Bind to 127.0.0.1 and use SSH tunnels. Tailscale binding is simpler and equally secure within your mesh.

Step 3: Config Files

Prompt:

“Create prometheus.yml scraping localhost:9100. Create alertmanager.yml configured for AWS SES (SMTP). Secrets: Use environment variables or placeholders for SMTP creds.”

4. Alerting Pipelines

Graphs are for post-mortems. Alerts are for now. For critical infrastructure, you need an automated pager.

A. AWS SES Setup - Your Email Gateway

To send reliable emails from your server, you need an SMTP relay. AWS SES is highly recommended for its reliability, low cost, and deliverability. We will manage this through the AWS CLI.

Step 1: Configure AWS CLI Credentials (Local)

You’ll need an AWS IAM user with permissions to manage SES identities and send emails. For simplicity in this guide, we’ll assume your aws configure is set up with an Administrator Access Key. In production, use a more granular IAM policy.

Agent: Claude Code (Local, on your Admin Laptop) Prompt:

“I need to configure my local aws-cli with credentials for an IAM user that has permissions to verify SES identities and send emails.

1. Instruct me on how to obtain an AWS Access Key ID and Secret Access Key for an IAM user (e.g., via AWS Console -> IAM -> Users -> Security credentials).

2. Guide me through running aws configure to set up these credentials locally.

3. Remind me to set a default region (e.g., us-east-1) and output format (e.g., json).”

Step 2: Verify Email Identity via AWS CLI

Before Alertmanager can send emails, your sending email address must be verified with SES.

Agent: Claude Code (Local, using aws-cli via deploy user with AWS credentials configured) Prompt:

“I need to verify an email identity for AWS SES for sending alerts. Use aws-cli to verify alerts@yourdomain.com. Command: aws ses verify-email-identity --email-address alerts@yourdomain.com Explain that I will receive a verification email and need to click a link. Then, explain how to check the verification status: aws ses list-identities --query ‘Identities[*]’.”

B. The Rules (rules.yml)

Agent: Claude Code Prompt:

“Create prometheus/rules.yml. Rules:

1. InstanceDown: up == 0 for 1m.

2. HighCPU: Usage > 90% for 5m.

3. DiskSpace: Free < 10%. Configure Alertmanager to route these to my email.”

Step 1: Verify Email Identity via AWS CLI

Before Alertmanager can send emails, your sending email address must be verified with SES.

Agent: Claude Code (Local, using aws-cli via deploy user with AWS credentials configured) Prompt:

“I need to verify an email identity for AWS SES for sending alerts. Use aws-cli to verify alerts@yourdomain.com. Command: aws ses verify-email-identity --email-address alerts@yourdomain.com Explain that I will receive a verification email and need to click a link.”

5. TLS Certificate Monitoring

Cloudflare handles edge certificates, but you still need to monitor:

1. Edge certificates (Cloudflare-issued): Rarely expire but can have issues

2. Origin certificates (Traefik/Let’s Encrypt): Expire every 90 days

Agent: Claude Code Prompt:

“Create /opt/monitoring/scripts/check_tls_expiry.sh:

#!/bin/bash

DOMAIN=$1

WARN_DAYS=${2:-21}

EXPIRY=$(echo | openssl s_client -servername $DOMAIN -connect $DOMAIN:443 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)

EXPIRY_EPOCH=$(date -d “$EXPIRY” +%s)

NOW_EPOCH=$(date +%s)

DAYS_LEFT=$(( ($EXPIRY_EPOCH - $NOW_EPOCH) / 86400 ))

if [ $DAYS_LEFT -lt $WARN_DAYS ]; then

echo “WARNING: $DOMAIN expires in $DAYS_LEFT days”

exit 1

fi

echo “OK: $DOMAIN expires in $DAYS_LEFT days”

1. Make executable: chmod +x /opt/monitoring/scripts/check_tls_expiry.sh

2. Add cron: /etc/cron.d/tls_expiry_check

0 6 * * * root /opt/monitoring/scripts/check_tls_expiry.sh yourdomain.com 21 >> /var/log/tls-expiry-check.log 2>&1

3. For Traefik origin certs, create a similar script checking the cert files directly.”

6. External Monitoring (UptimeRobot)

Internal monitoring can’t catch issues affecting external access (DNS, Cloudflare outages, etc.). Use an external service.

Setup:

1. Create free account at UptimeRobot

2. Add HTTP(S) monitors for each public domain

3. Configure email alerts (same address as Alertmanager)

4. Optional: Use UptimeRobot API to manage monitors as code:

curl -X POST “https://api.uptimerobot.com/v2/newMonitor” \

-d “api_key=YOUR_KEY” \

-d “friendly_name=My App” \

-d “url=https://myapp.com” \

-d “type=1” # HTTP(S)

Alternative services: Pingdom, Better Uptime, or self-hosted with a separate VPS in a different region.

7. Alertmanager Email Formatting

Customize Alertmanager to send clear, actionable alerts.

Agent: Claude Code Prompt:

“Update /opt/monitoring/alertmanager.yml:

global:

smtp_smarthost: ‘email-smtp.us-east-1.amazonaws.com:587’

smtp_from: ‘alerts@yourdomain.com’

smtp_auth_username: ‘{{ env “ALERT_SMTP_USER” }}’

smtp_auth_password: ‘{{ env “ALERT_SMTP_PASS” }}’

route:

receiver: ‘email-alerts’

group_by: [’alertname’]

group_wait: 30s

group_interval: 5m

repeat_interval: 4h

receivers:

- name: ‘email-alerts’

email_configs:

- to: ‘ops@yourdomain.com’

send_resolved: true

headers:

Subject: ‘[infra] {{ .GroupLabels.alertname }}’

Store SMTP credentials in /opt/monitoring/.env (chmod 600). Reload: curl -s -X POST http://localhost:9093/-/reload”

8. Verification

Agent: Claude Code Prompt:

“Deploy: docker compose up -d. Test: Stop node-exporter. Wait 2 mins. Did I get an email with [infra] subject? Restart node-exporter.”

9. Context Update

Agent: Codex Prompt:

“Monitoring Active. Access: Tailscale IPs - Prometheus (9090), Uptime Kuma (3001), Alertmanager (9093). Alerts: SES Email with [infra] prefix. External: UptimeRobot for public domain checks. TLS: Daily cron checks for certificate expiry. Update INFRASTRUCTURE_CONTEXT.md.”

Next Step: We have eyes. Now let’s secure the front door. Go to 07. The Edge Layer.

Objective: Transform the raw apps-vps into a robust PaaS (Platform as a Service) using Docker Engine and Coolify. We will replace the functionality of Vercel with open-source software you control.

1. The Architecture: “PaaS in a Box”

We are building a stack that mimics Vercel’s core features:

1. Git Push -> Deploy: Handled by Coolify + GitHub App.

2. SSL Management: Handled by Traefik (part of Coolify) + Let’s Encrypt.

3. Environment Management: Handled by Coolify UI.

4. Containerization: Handled by Docker.

Why Docker Engine?

We use the Official Docker Engine, not the Ubuntu “Snap” package.

• Snap: Auto-updates can break your db. Permission issues with volumes.

• Official: Stable, predictable, standard paths (/var/lib/docker).

2. Preparing the Host

Agent: Claude Code Prompt:

“SSH into apps-vps.

1. Remove Snaps: sudo apt remove docker.io docker-doc ....

2. Install Official: Add Docker GPG key and repo. Install docker-ce docker-ce-cli containerd.io.

3. User Group: sudo usermod -aG docker deploy. (Requires re-login).

4. Log Rotation: Create /etc/docker/daemon.json.

{ “log-driver”: “json-file”, “log-opts”: { “max-size”: “10m”, “max-file”: “3” } }

Why?: Without this, a looping error fills your disk in 24 hours.

5. Restart Docker.”

3. Installing Coolify

Coolify is our orchestrator.

Agent: Claude Code Prompt:

“SSH into apps-vps. Run the install script: curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash. Wait for completion. Verify containers: docker ps | grep coolify.”

4. Initial Configuration

1. Access: Two paths, one public, one private:

   • Public (Proxied): https://coolify.yourdomain.com (via Cloudflare proxy on 443, fronted by Traefik). Use this for normal admin access; enforce 2FA.

   • Private (Direct): http://100.x.y.z:8000 (Tailscale IP only).

   • Security Rule: UFW must deny 8000 on eth0 and allow 8000 on tailscale0. Keep 80/443 open. This keeps the raw admin port off the public internet while preserving a VPN backdoor.

2. Register: Create Admin account.

3. Security Hardening (Critical):

   • 2FA: Enable Two-Factor Authentication immediately (Profile -> Security).

   • Break-Glass Account: Create a second admin user (e.g., breakglass@yourdomain.com) with a generated 32-char password. Store this offline (e.g., on paper or in a separate vault). If 2FA fails or SSO breaks, this is your only way back in.

4. Select Server: Choose “Local Docker”.

5. GitHub Integration:

   • Go to Sources -> Add GitHub.

   • Create GitHub App (Follow Coolify prompts).

   • Crucial: Grant access ONLY to specific repos (not your whole org).

5. Coolify Internal Tuning (Horizon Queues)

Coolify uses Laravel Horizon to manage background jobs (deployments, server checks). By default, a heavy deployment can clog the queue, preventing other tasks (like backups) from running. We will tune the worker pools.

Agent: Claude Code Prompt:

“SSH into apps-vps. I need to check the Horizon supervisor configuration inside the Coolify container.

1. Run docker exec coolify php artisan horizon:supervisors.

2. We want to ensure there are dedicated queues.

   • deployments: For long-running builds.

   • priority: For server health checks.

   • general: For everything else.

3. If they are not split, we might need to edit the docker-compose.yml or environment variables for Coolify to set HORIZON_BALANCE=simple or custom supervisor configurations. (Note: In recent Coolify versions, this is auto-managed, but verifying 3 separate pools is best practice.)

4. Also, check if balanceMaxShift is set to 2 (allows workers to shift pools quickly).”

6. Docker Firewall Workaround (Critical)

Warning: Docker bypasses UFW by default! A simple ufw deny 8000 will NOT block Docker-exposed ports.

Docker manipulates iptables directly via the DOCKER chain, which is processed before UFW rules. To properly restrict Docker ports, we must use the DOCKER-USER chain.

Agent: Claude Code Prompt:

“SSH into apps-vps. We need to block port 8000 on the public interface while allowing it on Tailscale.

1. Create iptables rules:

# Drop 8000 from public interface

sudo iptables -I DOCKER-USER -i eth0 -p tcp --dport 8000 -j DROP

# Allow 8000 from Tailscale

sudo iptables -I DOCKER-USER -i tailscale0 -p tcp --dport 8000 -j ACCEPT

2. Persist rules across reboot:

sudo apt install iptables-persistent netfilter-persistent

sudo netfilter-persistent save

3. Verify: Run sudo iptables -L DOCKER-USER -n -v to see the rules.”

Note: The interface name may vary (eth0, ens3, etc.). Check with ip addr.

7. Security Verification

Ensure the Coolify Dashboard is NOT exposed to the wild.

Agent: Claude Code Prompt:

“Check public access to port 8000. nc -zv <PUBLIC_IP> 8000. It MUST fail (Timeout/Refused). If it succeeds, check DOCKER-USER iptables rules immediately.”

8. Redis Queue (Optional)

For applications using background jobs (BullMQ, Sidekiq, etc.), deploy Redis on the Apps VPS.

Agent: Claude Code Prompt:

“SSH into apps-vps. Create /opt/redis/docker-compose.yml:

services:

redis:

container_name: redis-broker

image: redis:7.4-alpine

restart: unless-stopped

command: redis-server --appendonly yes --requirepass <STRONG_PASSWORD>

volumes:

- ./data:/data

ports:

- ‘100.x.x.x:6379:6379’ # Tailscale IP only

1. Generate password: openssl rand -base64 32

2. Create directory: mkdir -p /opt/redis/data

3. Deploy: cd /opt/redis && docker compose up -d

4. Apply system tuning:

   • Add vm.overcommit_memory = 1 to /etc/sysctl.d/99-redis.conf

   • Disable Transparent Huge Pages (THP) via systemd service

5. Verify: redis-cli -h 100.x.x.x -a <password> ping”

For external access (e.g., Vercel during migration), consider adding mTLS on a separate port (6380).

9. Context Update

Agent: Codex Prompt:

“Coolify Installed & Hardened. Dashboard: http://apps-vps:8000 (Tailscale-only via DOCKER-USER rules). Security: 2FA Enabled, Break-glass account created. Tuning: Horizon queues verified. Redis: Available at 100.x.x.x:6379 (optional). Update INFRASTRUCTURE_CONTEXT.md.”

Next Step: We have a platform. Now we need eyes on it. Go to 06. Observability Stack.

Objective: Connect the isolated apps-vps and db-vps using Tailscale, creating a private, encrypted mesh network. We will implement “Zero Trust” principles by configuring Firewalls and Tailscale ACLs to ensure the Database is only accessible by the App Server, and never the public internet.

1. The “Public IP” Problem

In a traditional setup, to let your App Server talk to your Database:

1. You open Port 5432 on the DB Server’s Public IP.

2. You rely on IP Whitelisting (pg_hba.conf or UFW).

3. The Risk: If you mess up the whitelist, or if IPs change, your Database is exposed to the entire internet. IP spoofing is hard, but misconfiguration is easy.

The Solution: Overlay Networks. We will create a virtual network (100.x.y.z) that exists on top of the public internet. Traffic is encrypted (WireGuard). The DB Server listens only on this virtual interface.

2. Installing Tailscale via AI

We need to install the agent on 3 nodes: Admin Laptop, Apps VPS, DB VPS.

Step 1: Server Installation

Agent: Claude Code Prompt:

“SSH into apps-vps. Install Tailscale using the official script: curl -fsSL https://tailscale.com/install.sh | sh Run sudo tailscale up. Important: Capture the Auth URL and show it to me.”

Click the URL to authorize. Repeat for db-vps.

Step 2: MagicDNS & Naming

1. Go to Tailscale Admin Console.

2. Rename the machines to apps-vps and db-vps (removing the random suffixes).

3. Enable MagicDNS. This allows us to ping db-vps instead of memorizing IPs.

3. The Firewall Trust (The “Air Gap”)

Currently, UFW on db-vps blocks everything (except SSH on Public IP). It also blocks the Tailscale interface by default. We need to tell UFW: “Trust the Mesh.”

Agent: Claude Code Prompt:

“SSH into db-vps. We need to allow traffic from the Tailscale interface (tailscale0).

1. Check interface name: ip addr show tailscale0.

2. Add rule: sudo ufw allow in on tailscale0.

   • Why: This allows port 5432 (and others) ONLY if the packet comes decrypted from the VPN.

3. Reload UFW.

4. Verify with sudo ufw status verbose.”

Repeat for apps-vps (allows SSH from laptop via VPN).

4. Tailscale ACLs (Advanced Security)

For true “Zero Trust,” we don’t just trust everyone on the VPN. We restrict access.

Agent: Gemini CLI Prompt:

“I have 3 nodes: admin, apps, db. Generate a Tailscale ACL JSON policy. Rules:

1. Admin: Access to *:* (Everything).

2. Apps: Access to tag:db:5432 (Postgres) and tag:db:6543 (PgBouncer).

3. Isolation: db cannot initiate connections to apps (only reply).

4. Default: Deny all.”

Paste this JSON into Tailscale Admin Console -> Access Controls.

5. Connectivity Verification

Let’s prove it works.

Agent: Claude Code Prompt:

“SSH into apps-vps.

1. Ping the DB: ping -c 4 db-vps.

2. Check SSH port via Mesh: nc -zv db-vps 22.

3. Check Public IP for 5432 (Should fail/timeout). Report results.”

6. SSH Lockdown (Critical Security Step)

Now that the mesh is operational, we block public SSH entirely. All SSH access will be Tailscale-only.

Agent: Claude Code Prompt (Apps VPS):

“SSH into apps-vps via Tailscale (ssh deploy@100.x.x.x).

1. Remove public SSH rule: sudo ufw delete allow 22/tcp.

2. Verify: sudo ufw status verbose - Port 22 should only appear for tailscale0.

3. Test from outside Tailscale (e.g., your phone on mobile data): ssh deploy@<PUBLIC_IP> should timeout.

4. Test from inside Tailscale: ssh deploy@apps-vps should work. Report results.”

Repeat for db-vps. The DB VPS already had no public ports - this confirms the lockdown.

Warning: Before running this, ensure you have VNC access via Contabo control panel as a backup!

7. Context Update

Agent: Codex Prompt:

“Mesh Network Active & SSH Locked Down. Apps VPS: 100.x.x.x — Public SSH blocked, Tailscale-only DB VPS: 100.y.y.y — No public ports whatsoever Firewall: in on tailscale0 ALLOWED. Public port 22 DENIED. Update INFRASTRUCTURE_CONTEXT.md.”

Next Step: The network is ready. Time to build the Platform. Go to 05. The App Engine.

Objective: Go from “Purchase Confirmation Email” to a fully hardened, secure server environment. We will use AI agents to generate safe provisioning scripts and configure “Default Deny” firewalls.

1. Buying the Metal: Provider Selection

AI cannot buy servers for you (yet). You must do this manually.

The Contabo Configuration (Recommended)

Why Contabo? Best Price/Performance ratio for raw compute.

1. Apps VPS: Cloud VPS 20 SSD (6 vCPU, 16GB RAM, 200GB NVMe).

2. DB VPS: Cloud VPS 20 SSD (6 vCPU, 16GB RAM, 200GB NVMe) - dedicated resources for Postgres.

3. Region: Choose the one closest to your users. Crucial: Both VPSs MUST be in the same region to minimize latency.

4. Storage: NVMe. Never choose SSD if NVMe is an option. Database performance depends on IOPS.

5. OS: Ubuntu 24.04 LTS. Use the latest Long Term Support.

Note: Product names vary by promotion. The key specs are 6+ vCPU, 16GB+ RAM, and NVMe storage.

After purchase, wait 15-30 mins for the email with IPs and Root Passwords.

2. The “First Boot” Ritual

Warning: The moment you log in as root, you are vulnerable. If you edit SSH configs incorrectly, you lock yourself out. Strategy: We use Claude Code to generate a “One-Shot” setup script that creates a sudo user and installs keys before we touch any configs.

Step 1: Generate Infrastructure Keys

Don’t use your personal GitHub key. Create a dedicated key for this cluster.

Agent: Claude Code Prompt:

“Check ~/.ssh/. Do I have a key named id_ed25519_vps? If not, run ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_vps -C ‘admin@infra’. Display the public key content so I can copy it.”

Step 2: The Provisioning Script

Agent: Claude Code Prompt:

“I have a fresh Ubuntu 24.04 VPS. Write a bash script to initialize a user. Variables: USER=deploy, PUBKEY=<PASTE_KEY_HERE>. Actions:

1. Create user $USER with bash shell.

2. Add to sudo group.

3. Create .ssh dir with 700 permissions.

4. Create authorized_keys with 600 permissions and append the key.

5. Configure sudoers to allow $USER sudo without password (optional but useful for agents). Show me the script.”

Action: SSH into root@<IP>, paste the script, run it. Repeat for both servers.

Step 3: Verify Access

Agent: Claude Code Prompt:

“Verify access to 144.126.x.x using the new user. ssh -i ~/.ssh/id_ed25519_vps deploy@144.126.x.x ‘whoami’ It should return ‘deploy’.”

3. Agent-Driven OS Hardening

Now that we have secure access, we lock the doors.

A. SSH Hardening (sshd_config)

Default Ubuntu allows Root Login and Password Auth. We want neither.

Agent: Claude Code Prompt:

“SSH into apps-vps. We are hardening SSH.

1. Backup config: sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.

2. Edit /etc/ssh/sshd_config:

   • PermitRootLogin no

   • PasswordAuthentication no

   • ChallengeResponseAuthentication no

   • UsePAM yes

   • X11Forwarding no

3. Critical: Run sudo sshd -t to verify syntax. If error, abort.

4. Restart SSH.

5. Suicide Check: Immediately try to open a second SSH connection in the background. If it fails, revert the config.”

Repeat for db-vps.

B. Firewall Strategy (UFW)

We adopt a “Default Deny” posture.

Apps VPS Rules: Agent: Claude Code Prompt:

“Configure UFW on apps-vps.

1. sudo ufw default deny incoming.

2. sudo ufw default allow outgoing.

3. sudo ufw allow 22/tcp (SSH).

4. sudo ufw allow 80/tcp (HTTP).

5. sudo ufw allow 443/tcp (HTTPS).

6. Enable and verify.”

DB VPS Rules: Prompt:

“Configure UFW on db-vps.

1. sudo ufw default deny incoming.

2. sudo ufw default allow outgoing.

3. sudo ufw allow 22/tcp (SSH).

4. Do NOT open port 5432 yet.

5. Enable and verify.”

4. System Updates & Hygiene

Prompt:

“SSH into both servers.

1. Run sudo apt update && sudo apt upgrade -y.

2. Install tools: curl wget git vim htop net-tools ncdu.

3. Set Hostnames: sudo hostnamectl set-hostname apps-vps (and db-vps).

4. Set Timezone: sudo timedatectl set-timezone UTC.

5. Setup Swap: If no swap, create a 4GB swapfile (essential for stability).”

5. Context Update

Agent: Codex Prompt:

“Provisioning & Hardening Complete. Apps VPS: 144.126.x.x (User: deploy). Ports 22/80/443 Open. DB VPS: 144.126.y.y (User: deploy). Port 22 Open. SSH Hardening: Keys only. Root disabled. Update INFRASTRUCTURE_CONTEXT.md.”

Important: After completing the Tailscale mesh in the next module, we will block public SSH entirely (port 22 on the public interface). All SSH access will then be Tailscale-only, adding another layer of security. For now, port 22 remains open on the public IP for initial setup only.

Next Step: The servers are secure islands. We need to build a bridge. Go to 04. The Private Mesh.

Objective: Perform a rigorous financial and operational analysis of the “Exit.” We will define the “Safe Harbor” 2-node architecture, calculate the precise ROI of leaving managed cloud services, and understand why “owning the metal” is the ultimate competitive advantage for a scaling startup.

1. The Economics of Exit: Escaping the Success Tax

Vercel and Supabase are fantastic “Day 0” tools. They offer zero-config deployment. But their business model depends on the “Success Tax.” As you scale, their pricing curves become punitive.

The “Success Tax” Breakdown (Scenario: Moderate B2B SaaS)

Traffic: 1TB Bandwidth, 500k Users, 50GB DB.

The Self-Hosted Bill (Monthly)

We replace “Variable Usage Pricing” with “Fixed Capacity Pricing.”

The Savings: ~90% to 97%. The ROI: If migration takes 10 hours, and you save $530/mo ($6360/yr), your hourly rate for this work is $636/hr.

2. The Operational Case: Visibility & Control

Saving money is great. But Control is better.

The “Black Box” Problem

On Vercel, if your API is slow, you see a chart saying “High Duration.”

• Is it the CPU?

• Is it a “Noisy Neighbor” on the AWS Lambda host?

• Is it a network timeout? You cannot know. You don’t have htop. You don’t have root.

The “Glass Box” Solution

On your VPS:

1. Unified Logs: You run docker logs -f and see Nginx, Next.js, and Redis logs interleaved. You see the causal chain of errors immediately.

2. Deep Metrics: You install Node Exporter. You see “CPU Core 2 is pinned by ffmpeg.” You fix the code. You don’t just pay for a bigger tier.

3. No Vendor Lock-in: You use standard Docker images. If Contabo doubles prices, you rsync your volumes to Hetzner in 1 hour. You are sovereign.

3. The “Safe Harbor” Architecture

We do not dump everything on one server. That is a Single Point of Failure (SPOF). We build a 2-Node Cluster connected by a private mesh.

Why Two Nodes?

1. The “OOM Killer” Protection: Next.js is memory hungry. If it eats all RAM, Linux kills the biggest process. On a single server, that’s often Postgres. By separating them, your App can crash without taking down your Data.

2. The Security Air Gap: The App Server must be exposed to the internet (Port 443). The DB Server never needs to be. By isolating the DB on a separate node with NO public ports, we create a massive security barrier.

The Diagram

graph TD

User((User)) -->|HTTPS:443| CF[Cloudflare Proxy]

CF -->|HTTPS:443| AppsVPS[Apps VPS (Public)]

subgraph “Public Internet”

AppsVPS

end

subgraph “Private Mesh (Tailscale 100.x.x.x)”

AppsVPS -- “Postgres (Port 6432)” --> DBVPS[DB VPS (Private)]

Laptop[Admin] -- “SSH (22)” --> AppsVPS

Laptop -- “SSH (22)” --> DBVPS

end

subgraph “Apps VPS (The Cattle)”

Coolify[Coolify PaaS]

Traefik[Traefik Proxy]

NextJS[Next.js App]

Redis[Redis Cache]

end

subgraph “DB VPS (The Pet)”

PgBouncer[Connection Pooler]

Postgres[Postgres 16]

WALG[Backup Agent]

end

WALG -.->|Encrypted Backup| R2[Cloudflare R2]

Node Specs

• Apps VPS: High CPU (for SSR rendering). Run Coolify.

• DB VPS: High RAM (for caching). Fast NVMe. Run Docker Compose.

4. The Edge Layer: Cloudflare API

We are not just buying servers. We are putting Cloudflare in front of them.

• DDoS Protection: Hides our real IP.

• SSL: “Strict” mode handles crypto at the edge.

• Management: We use API scripts (curl) to manage DNS as code.

5. The Network: Tailscale Mesh

How do servers talk if the DB has no public ports? Tailscale.

• It creates a virtual network (100.x.y.z).

• Traffic is encrypted (WireGuard).

• We configure the DB Firewall to Drop All from Public IPs and Accept All from Tailscale IPs.

Next Step: We have the blueprint. Let’s buy the metal. Go to 03. The Metal Foundation.

Objective: Establish the tooling, environment, and mental model required to build enterprise-grade infrastructure using AI agents. We will configure the “Triad” of tools (Codex, Claude, Gemini) and integrate them with an Obsidian-based “Context System.”

1. The Paradigm Shift: Architect vs. Operator

In the traditional DevOps world, you are the Operator.

• You memorize flags for tar, rsync, systemctl, and iptables.

• You manually edit config files in nano, hoping you didn’t miss a semicolon.

• Your “state” is stored in your brain or a stale Wiki.

• Bottleneck: Your typing speed and your memory.

In the AI-Native Ops world, you are the Architect.

• You define the Goal: “I want a secure, high-availability Postgres cluster.”

• You define the Constraints: “It must use Docker, allow access only via Tailscale, and back up to R2.”

• The AI Agents handle the Implementation.

• Bottleneck: Your ability to clearly articulate your intent.

This shift requires a new toolchain. We are replacing your memory with Codex, your hands with Claude, and your research skills with Gemini.

2. The Toolchain: The Triad & The Utility Knives

We will use three distinct CLI agents. Using the wrong one for a task leads to frustration. These are supplemented by powerful utility CLIs for specific cloud providers.

A. Codex CLI (The Autonomous Agent)

OpenAI’s official CLI tool, powered by GPT-5.1-Codex-Max (as of November 2025).

• Role: The Autonomous Executor & Context Manager.

• Superpower: Can work autonomously for extended periods (24+ hours), managing long-term memory, project context, and executing complex multi-step tasks. Supports multi-context window operation.

• Weakness: Expensive for massive log dumps; requires careful supervision for critical operations.

• When to use:

  • “What was the IP address of the database server?”

  • “Remind me of the architecture decision we made about Redis.”

  • “Update the documentation to reflect the new firewall rules.”

  • Extended autonomous coding sessions.

Installation:

# Via npm (recommended)

npm i -g @openai/codex

# Or via Homebrew on macOS

brew install --cask codex

# Authenticate

codex auth login

# Alias for speed

alias cx=”codex”

Reference: OpenAI Codex

B. Claude Code (The Engineer)

This is the star of the show. claude-code is an official tool from Anthropic designed specifically for execution. Current version: 2.0.54, powered by Claude Sonnet 4.5 and Claude Opus 4.5.

• Role: The Execution Engine.

• Superpower: It can SSH into servers, run commands, read files, edit files, and fix its own errors. It acts as a Senior Engineer paired with you. Features include checkpoints, /rewind command for session recovery, and VS Code extension integration.

• Weakness: Can get stuck in loops if a command fails repeatedly. Needs supervision.

• When to use:

  • “SSH into apps-vps and install Docker.”

  • “Write a Python script to migrate this data and run it.”

  • “Edit nginx.conf to enable gzip compression.”

Installation:

# Requires Node.js 18+

npm install -g @anthropic-ai/claude-code

claude auth login

# Alias for speed

alias c=”claude”

VS Code Extension (Beta): Search “Claude Code” in the VS Code marketplace for IDE integration.

Reference: Claude Code Documentation

C. Gemini CLI (The Strategist)

Gemini 3 Pro (as of November 2025) features thinking models with adaptive reasoning and massive context windows.

• Role: The Big Brain / Debugger.

• Superpower: Absorbing massive amounts of data (Logs, Documentation) and finding the needle in the haystack. Features Deep Think mode for complex reasoning tasks (available to AI Ultra subscribers).

• Weakness: Tool use (running commands) is less mature than Claude Code.

• When to use:

  • “Here is a 50MB log file. Find the error.”

  • “Read this 100-page PDF on PgBouncer and tell me how to configure TLS.”

  • “Design a high-availability architecture for this app.”

Installation: Use the Google Gen AI SDK or a wrapper.

pip install google-generativeai

# You will need a simple python wrapper script

# export GOOGLE_API_KEY=”...”

# alias g=”python3 ~/scripts/gemini.py”

Reference: Gemini API Documentation

D. Cloudflare API (The Edge Manager)

We manage our DNS and Edge infrastructure as code using the official API.

• Role: DNS & Storage Manager.

• Superpower: Managing Cloudflare resources programmatically.

• When to use:

  • “Update the DNS record for app.example.com to point to the new VPS.”

  • “Create a new R2 bucket for backups.”

Installation: No installation required. We use curl (pre-installed on most systems).

E. AWS CLI (The Cloud Swiss Army Knife)

This CLI allows command-line interaction with a vast array of AWS services, including SES (Simple Email Service).

• Role: AWS Resource Manager.

• Superpower: Automating the configuration and verification of AWS services directly from the terminal.

• When to use:

  • “Verify an email identity in SES.”

  • “List domains managed by Route53.”

  • “Configure an S3 bucket policy.”

Installation:

sudo apt update

sudo apt install awscli -y

aws configure # Follow prompts for Access Key ID, Secret Access Key, Region

F. Bitwarden Secrets Manager (BWS)

Your secrets should not live in text files or git. We use Bitwarden Secrets Manager (bws) to inject secrets directly into our environment.

• Role: Secrets Vault.

• Superpower: Centralized, encrypted management of API keys and database credentials.

• When to use:

  • “Inject the production database URL into the deployment script.”

  • “Rotate the Stripe API key across all services.”

Installation:

# Download from https://bitwarden.com/help/secrets-manager-cli/

# Authenticate

bws login

3. What is Obsidian?

Obsidian is a local-first, Markdown-based knowledge management application that serves as the foundation for our Context System. Unlike cloud-based tools, your notes are stored as plain text files on your device, ensuring privacy, offline access, and no vendor lock-in.

Key Features:

• Local Storage: All notes are Markdown files on your filesystem

• Bidirectional Links: [[Wiki-style links]] connect related notes

• Graph View: Visualizes connections between your notes

• Plugin Ecosystem: Thousands of community plugins extend functionality

• Cross-Platform: Available on macOS, Windows, Linux, iOS, and Android

Why Obsidian for Infrastructure?

1. AI-Friendly: Plain text files can be read by any AI agent

2. Version Control: Markdown files work perfectly with Git

3. No Lock-in: Your data is yours forever

4. Offline-First: Works without internet connection

Pricing (as of February 2025):

• Personal use: Free (including commercial use)

• Sync service: Optional paid add-on

• Publish service: Optional paid add-on

Installation: Download from obsidian.md

4. The “Context-First” Workflow with Obsidian

The biggest mistake engineers make with AI is Ambiguity. If you type “Install Postgres,” the AI guesses. You want “Postgres 17 on Ubuntu 24.04 via Docker with specific tuning.”

To solve this, we use a Context System anchored in your Obsidian Vault.

Step 1: Create Your Context Root

Your Obsidian vault is the “Source of Truth.” Create a folder: 00 - Inbox/AI Context (or where you store project specs).

Step 2: The Infrastructure Context File

Create 00 - Inbox/AI Context/INFRASTRUCTURE_CONTEXT.md. This file is the “Bible.”

Template:

# Project Infrastructure Context

## Status

- **Phase**: 1 (Foundation)

- **Last Updated**: {{date:YYYY-MM-DD}}

## Architecture Overview

We are building a “Safe Harbor” 2-node cluster.

- **Node 1 (Apps VPS)**: Public Gateway. Runs Coolify, Traefik, Next.js.

- **Node 2 (DB VPS)**: Private Vault. Runs Postgres, Redis, PgBouncer.

- **Network**: Tailscale Mesh (100.x range) for internal communication.

- **Edge**: Cloudflare (DNS, Proxy, R2).

- **Email**: AWS SES.

## Server Inventory

| Role | Hostname | Public IP | Tailscale IP | User | OS |

| :--- | :--- | :--- | :--- | :--- | :--- |

| App Gateway | `apps-vps` | TBD | TBD | `deploy` | Ubuntu 24.04 |

| Database | `db-vps` | TBD | TBD | `deploy` | Ubuntu 24.04 |

## Credentials & Access

- **Contabo**: (Link to [[Contabo Credentials]])

- **Tailscale**: (Link to [[Tailscale Admin]])

- **Cloudflare API Token**: (Link to [[Cloudflare Credentials]])

- **AWS Credentials**: (Link to [[AWS CLI Credentials]])

- **SSH Keys**: `~/.ssh/id_ed25519_vps`

## Constraints

1. **Security**:

- `db-vps` must NEVER have public ports open (except SSH via VPN).

- Root login disabled.

- SSH Key Auth only.

2. **Tooling**:

- **Orchestration**: Coolify (Apps), Docker Compose (DB).

- **Scripting**: Bash for simple tasks, Python for complex logic.

- **DNS**: Managed via Cloudflare API (curl).

- **AWS Services**: Managed via `awscli`.

- **Secrets**: Managed via `bws` CLI.

3. **Paths**:

- Persistent Data: `/opt/<service>/data`

- Configs: `/opt/<service>/config`

- Scripts: `/opt/<service>/scripts`

## Secrets Policy

- **NEVER store actual secrets in this file.**

- Use placeholders: `{{BITWARDEN_LOOKUP}}`.

- Store secrets in Bitwarden Secrets Manager. Use `bws` to retrieve them.

Step 3: The “Prime” Command

Before any session, you force the AI to read this file.

With Claude Code:

claude --context “00 - Inbox/AI Context/INFRASTRUCTURE_CONTEXT.md” “SSH into apps-vps and...”

5. Setting Up Local Control Center

A. SSH Config (~/.ssh/config)

Your agents need this to connect without typing IPs.

# Defaults

Host *

User deploy

IdentityFile ~/.ssh/id_ed25519

ServerAliveInterval 60

# The Apps Server

Host apps-vps

HostName 144.126.x.x # We will fill this later

User deploy

IdentityFile ~/.ssh/id_ed25519_vps

# The Database Server (Via Tailscale)

Host db-vps

HostName 100.x.y.z

User deploy

IdentityFile ~/.ssh/id_ed25519_vps

B. Local Secrets

Set API keys as environment variables, not text.

export CLOUDFLARE_API_TOKEN=”...”

export AWS_ACCESS_KEY_ID=”...”

export AWS_SECRET_ACCESS_KEY=”...” # For aws-cli

export AWS_DEFAULT_REGION=”us-east-1” # For aws-cli

6. The Execution Loop

For every module, follow this loop:

1. Intent (Gemini): “I need to set up backups. What is the best practice for Postgres on Docker?”

2. Plan (Claude): “Create a plan to implement WAL-G. List scripts and cron jobs.”

3. Review (Human): Check against constraints. “Wait, use R2, not S3.”

4. Execute (Claude): “SSH into db-vps and run the plan.”

5. Verify (Codex/Obsidian): “Update INFRASTRUCTURE_CONTEXT.md with the backup schedule.”

Next Step: We have the tools. Now let’s understand the mission. Go to 02. The Case for Self-Hosting.

If you are building software today, you probably started on Vercel or Supabase. And why wouldn’t you? They are miraculous. You push code, and it’s live. You click a button, and you have a database. It feels like magic.

Until the bill arrives.

The pricing models of managed cloud services are designed to capture your upside. They are a “Success Tax.” A viral post, a DDOS attack, or simply a slightly inefficient database query can turn your $20/month hobby bill into a $500/month crisis.

But the cost isn’t the only problem. It’s the control. When your API is slow on Vercel, you are debugging a black box. You cannot see the CPU steal time. You cannot see the neighbor process on the Lambda host. You are a tenant, not an owner.

The “Hard” Way is Now the Easy Way

Historically, the alternative—managing your own Linux servers (VPS)—was reserved for bearded sysadmins who memorized iptables syntax and enjoyed suffering.

Artificial Intelligence has changed this equation.

Tools like Claude Code, Gemini, and Codex have turned the Command Line Interface (CLI) into a natural language conversation. You no longer need to memorize how to write a systemd unit file. You simply need to intent it.

This series is a Masterclass. It is a step-by-step blueprint to take you from a “Vercel Consumer” to an “AI-Native Systems Architect.”

We will build a production-grade infrastructure on commodity hardware (Contabo/Hetzner) that rivals the reliability of AWS, for 1/10th the cost.

The Curriculum

This series is broken into 14 in-depth guides. Each one includes the theory, the execution steps, and the exact AI Prompts you need to copy-paste to get the job done.

Phase 1: The Foundation

01. The AI-Native Ops Workflow We stop typing commands manually. We set up the “Triad” of AI agents (Codex, Claude, Gemini) and build an Obsidian-based “Context System” that serves as the brain of our infrastructure.

02. The Case for Self-Hosting A rigorous financial and operational analysis. We calculate the ROI of leaving the cloud ($6,000/year savings for a moderate app) and define our “Safe Harbor” 2-node architecture.

03. The Metal Foundation We go shopping for bare metal. I guide you through purchasing high-performance NVMe VPSs and performing the “First Boot” ritual using AI to secure SSH and harden the OS.

04. The Private Mesh Connecting our isolated servers using Tailscale. We implement a Zero-Trust mesh network that allows our servers to talk securely without ever opening a database port to the public internet.

Phase 2: The Platform

05. The App Engine Turning a raw Linux server into a PaaS. We install Docker and Coolify, effectively giving you your own private Vercel dashboard for a fraction of the price.

06. Observability Stack Flying blind is dangerous. We deploy a “Survival” monitoring stack (Prometheus, Grafana, Uptime Kuma) that lives outside our platform, ensuring we get alerted via AWS SES even if everything else fails.

07. The Edge Layer Configuring Cloudflare as our global shield. We use the wrangler CLI to manage DNS as code, implement “Full (Strict)” SSL, and set up HSTS for banking-grade security.

Phase 3: The Database

08. High-Performance Database The crown jewels. We deploy a dedicated Postgres 16 node, tune it for our specific hardware using AI calculations, and implement PgBouncer to handle thousands of concurrent connections.

09. Data Durability Strategy Backups are nothing without restoration. We script automated nightly dumps to Cloudflare R2 and create an automated “Restore Drill” that verifies our backups actually work every single month.

Phase 4: The Application

10. Containerizing Next.js A deep dive into Docker optimization. We use Multi-Stage builds and output: standalone to shrink our Next.js application image from 1GB to <100MB.

11. The Great Data Migration The scary part. We script the surgical extraction of data from Supabase (including the Auth schema), sanitize it, and hydrate our new Postgres instance without losing a single user.

12. Production Configuration Re-wiring the application for its new home. We manage secrets securely, fix OAuth callbacks for Google/GitHub, and deploy persistent Background Workers (Redis/BullMQ) to handle heavy tasks.

Phase 5: Operations

13. The Cutover Event The Go-Live runbook. We script a zero-downtime DNS switchover using Cloudflare, ensure email deliverability (DKIM/SPF), and flip the switch.

14. Day 2 Operations Living with the machine. We cover log analysis using Gemini’s massive context window, cost tracking, and the “Break Glass” emergency procedures you need when the pager goes off.

Are you ready to own your infrastructure?

Start with Part 1: The AI-Native Ops Workflow