Security Hardening

Overview

This playbook covers how to secure a SaaS application in production on a dedicated server/VPS behind Cloudflare. It is based on an architecture audited and tested under real-world conditions (L7 DDoS attacks, automated scans, SSH brute force).

Reference stack: Next.js / Node.js, nginx, PostgreSQL, Redis, Cloudflare (Free/Pro). Adaptable to any modern web stack.

The core principle is defense in depth — every layer assumes the previous one can be bypassed. No single measure is sufficient on its own.

1. Firewall — Default-Deny Policy

The firewall must block everything by default and only allow the strict minimum:

Inbound policy:  DENY (default)
Outbound policy: ALLOW (default)

Allowed ports:
  22/tcp    → from admin IPs only (or Anywhere if fail2ban)
  80/tcp    → from Cloudflare IPs only
  443/tcp   → from Cloudflare IPs only

Cloudflare IPs to allow

IPv4:

173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22,
141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20,
197.234.240.0/22, 198.41.128.0/17, 162.158.0.0/15, 104.16.0.0/13,
104.24.0.0/14, 172.64.0.0/13, 131.0.72.0/22

IPv6:

2400:cb00::/32, 2606:4700::/32, 2803:f800::/32,
2405:b500::/32, 2405:8100::/32, 2a06:98c0::/29, 2c0f:f248::/32

Internal services — Localhost only

No backend service should listen on 0.0.0.0:

Validation: run sudo ss -tlnp — no line should show 0.0.0.0:3000, 0.0.0.0:5432, or 0.0.0.0:6379.

Inter-server communication

If a service runs on a separate server (Discord bot, background worker, etc.):

• Never communicate over plain HTTP on the public network
• Use WireGuard/VPN between servers
• Connection URLs use internal VPN IPs (10.0.0.x)

2. Cloudflare — CDN and Edge Protection

DNS

• All A/AAAA/CNAME records must be proxied through Cloudflare (orange cloud icon)
• Use Cloudflare Email Routing for MX records (no mail server exposing your IP)
• SPF: v=spf1 include:_spf.mx.cloudflare.net ~all
• Never create subdomains like "direct", "origin", or "ssh" pointing to the real IP

Authenticated Origin Pulls (AOP)

Cryptographically verifies that every HTTPS request to the origin comes from Cloudflare. Even if someone discovers the IP, they cannot bypass the CDN.

Cloudflare side:

• Dashboard → SSL/TLS → Origin Server → Authenticated Origin Pulls → ON

nginx side:

ssl_client_certificate /etc/ssl/cloudflare/authenticated_origin_pull_ca.pem;
ssl_verify_client on;
 
# Reject without revealing info (silently close connection)
error_page 495 496 =444 /;

Important: The default AOP certificate is shared across all Cloudflare tenants. An attacker can create their own Cloudflare account, point a domain at your IP, enable AOP, and route traffic through the Cloudflare network — your nginx accepts it because the certificate is the same.

Both AOP and a UFW firewall are required:

• AOP alone → bypassable via another Cloudflare account
• Firewall alone → bypassable if Cloudflare IP ranges change
• Both together → solid defense-in-depth

For maximum protection, use custom AOP certificates (per-zone or per-hostname), unique to your account.

Recommended settings

3. nginx — Reverse Proxy Hardening

Global config

# /etc/nginx/nginx.conf
worker_processes auto;
events {
    worker_connections 4096;
    multi_accept on;
    use epoll;
}
http {
    server_tokens off;   # Hide nginx version
}

Anti-Slowloris timeouts

# /etc/nginx/conf.d/security.conf
client_body_timeout 10s;
client_header_timeout 10s;
keepalive_timeout 65s;   # 30s is too low with HTTP/2 + Cloudflare (nginx default is 75s)
send_timeout 10s;
 
client_header_buffer_size 1k;
large_client_header_buffers 4 8k;
client_max_body_size 10m;
 
proxy_hide_header X-Powered-By;
proxy_hide_header Server;

Rate limiting

# /etc/nginx/conf.d/rate-limiting.conf
limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=auth:10m rate=5r/s;
limit_req_zone $binary_remote_addr zone=api:10m rate=20r/s;
limit_req_zone $binary_remote_addr zone=strict:10m rate=2r/s;
limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;
 
limit_req_log_level warn;
limit_conn_log_level warn;

Server blocks

# Block direct IP access (no domain name)
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    return 444;
}
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    ssl_reject_handshake on;  # Reject TLS without revealing a certificate
}
 
# HTTP → HTTPS redirect
server {
    listen 80;
    server_name example.com *.example.com;
    return 301 https://$host$request_uri;
}
 
# Main server
server {
    listen 443 ssl http2;
    server_name example.com *.example.com;
 
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
 
    # AOP
    ssl_client_certificate /etc/ssl/cloudflare/authenticated_origin_pull_ca.pem;
    ssl_verify_client on;
    error_page 495 496 =444 /;
 
    # Real IP from Cloudflare
    set_real_ip_from 173.245.48.0/20;
    # ... all CF ranges
    real_ip_header CF-Connecting-IP;
 
    include snippets/security-headers.conf;
}

Proxy cache — DDoS absorber

A short-lived cache (5s) absorbs floods during the auto-detection window:

proxy_cache_path /var/cache/nginx/app
    levels=1:2
    keys_zone=app_cache:10m
    max_size=100m
    inactive=10m
    use_temp_path=off;

# Inside the server block location /
proxy_cache app_cache;
proxy_cache_key "$host$request_uri";
proxy_cache_valid 200 5s;
proxy_cache_valid 301 302 30s;
proxy_cache_methods GET HEAD;
 
# Serve stale content when backend is overwhelmed
proxy_cache_use_stale error timeout updating http_502 http_503 http_504;
proxy_cache_background_update on;
 
# Ignore Cache-Control only (allows caching SSR pages marked no-cache)
proxy_ignore_headers Cache-Control;
 
# Do NOT cache authenticated requests (personalized content)
proxy_cache_bypass $cookie_session_token $http_authorization;
proxy_no_cache $cookie_session_token $http_authorization;

How it works:

• Normal traffic: re-generates every 5s (transparent)
• Under flood: proxy_cache_use_stale updating serves the last valid content during refresh
• Authenticated users: always bypass (fresh content)

Do not use proxy_cache_lock on — the polling interval is a fixed 500ms in nginx and not configurable, adding 500ms latency to every waiting request. proxy_cache_use_stale updating is the better approach: one request refreshes the cache, the rest receive stale content instantly.

Never proxy_ignore_headers Set-Cookie — nginx uses Set-Cookie presence to decide not to cache a response. Ignoring it causes responses with session cookies to be cached and served to other users, leaking sessions.

4. Automated DDoS Protection

The problem

Distributed L7 attacks (residential botnets, 4000+ IPs, ~400 req/s) pass through Cloudflare because each individual IP stays below per-IP thresholds. Per-IP rate limiting is useless against this pattern.

The solution

A bash script monitors aggregate throughput (not per-IP) and toggles Cloudflare Under Attack Mode via the API.

Detection parameters

Attack flow

T=0s    Flood starts (400 req/s)
        → nginx cache absorbs (serves stale, app server protected)

T=15s   Monitor detects → enables UAM via Cloudflare API
        → JS challenge blocks HTTP bots
        → Push alert sent (ntfy/Slack)

T=135s+ Normal traffic + minimum duration reached
        → 5 min cooldown starts

T=435s  60 consecutive checks < 80 req/s → UAM disabled
        → Resolution alert sent

Result: zero downtime. The cache absorbs during the 15s detection window, then UAM blocks everything.

Cloudflare prerequisites

• API Token with Zone Settings: Edit permission (scoped to specific zone)
• Zone ID (visible in dashboard Overview)

systemd service

[Unit]
Description=DDoS Auto-Detection Monitor
After=nginx.service network-online.target
 
[Service]
Type=simple
ExecStart=/bin/bash /path/to/ddos-monitor.sh
Restart=always
RestartSec=10
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/tmp
ReadOnlyPaths=/var/log/nginx /path/to/.env
MemoryMax=64M
CPUQuota=5%
StandardOutput=journal
SyslogIdentifier=ddos-monitor
 
[Install]
WantedBy=multi-user.target

5. Application Rate Limiting (Redis)

nginx rate limiting is a first pass (per-IP, stateless). Application-level rate limiting adds an intelligent layer: sliding window, per-endpoint limits, and ban escalation.

Sliding Window algorithm (Redis Sorted Sets)

Atomic pipeline:
1. ZREMRANGEBYSCORE key 0 (now - window)    // Remove expired entries
2. ZCARD key                                 // Count active entries
3. ZADD key now "now:random"                 // Add current request
4. EXPIRE key windowSeconds                  // Auto-cleanup

Advantages over fixed windows:

• No "burst at boundary" (2x allowed traffic at window edge)
• More accurate counting
• Smoother user experience

Limits by endpoint type

Auth endpoint limits

Auth endpoints need their own stricter limits:

In-memory fallback

If Redis is unavailable:

• Fall back to an in-memory Map<string, { count, resetTime }>
• Lazy cleanup every 60s
• Graceful degradation — rate limiting works but resets on restart

IP ban escalation

When the same IP triggers too many rate limits:

Implementation:

• Violations tracked in a Redis Sorted Set ip-ban:{ip} (sliding window)
• Bans stored in a TTL key ip-ban:banned:{ip}
• Check at the start of every request → 403 if banned
• Tracking is fire-and-forget (no added latency)
• Fail open: if Redis is down, nobody gets banned

6. Database Security

PostgreSQL

• listen_addresses = 'localhost' in postgresql.conf
• No trust connections — strong passwords only
• statement_timeout = '30s' to prevent infinite queries
• ORM with parameterized queries (Drizzle, Prisma) — never concatenate SQL
• Multi-tenant isolation: every query filters by tenant_id
• FOR UPDATE locks in critical transactions (stock, balance)
• INSERT-before-UPDATE pattern for financial operations (prevents double-crediting)

Redis

• bind 127.0.0.1 ::1 (localhost only)
• requirepass with a strong password
• Connection URL: redis://:password@localhost:6379/0
• Timeouts: connectTimeout: 10s, commandTimeout: 5s
• Retry: exponential backoff (50ms → 100ms → 200ms... max 2s)
• Key prefix: app: to avoid collisions
• maxRetriesPerRequest: 3
• Auto-reconnect on READONLY error (failover)
• Separate client for BLPOP workers (don't block the main client)

7. SSRF Protection

When the application makes HTTP requests to user-provided URLs (webhooks, callbacks), validate the URL to block internal targets.

Ranges to block

IPv4:

0.0.0.0/8        This network
10.0.0.0/8       RFC 1918
100.64.0.0/10    CGNAT
127.0.0.0/8      Loopback
169.254.0.0/16   Link-local
172.16.0.0/12    RFC 1918
192.168.0.0/16   RFC 1918
224.0.0.0/4      Multicast + Reserved (224+)

IPv6:

::1, ::          Loopback
fe80::/10        Link-local
fc00::/7         ULA (fd prefix included)
::ffff:x.x.x.x  IPv4-mapped (check the v4 part)

Hostnames:

localhost, *.localhost, *.local, *.internal

Cloud metadata:

169.254.169.254
metadata.google.internal
metadata.internal

Schemes: allow only http: and https:. Block file:, ftp:, gopher:, data:, etc.

Fail-safe: invalid URL = blocked.

8. Authentication and Sessions

Session configuration

Timing-safe secret comparisons

Always use timing-safe comparison for secrets to prevent timing attacks:

import { timingSafeEqual } from 'crypto';
 
function safeCompare(a: string, b: string): boolean {
  const bufA = Buffer.from(a);
  const bufB = Buffer.from(b);
  if (bufA.length !== bufB.length) {
    timingSafeEqual(bufA, bufA); // dummy comparison to keep constant time
    return false;
  }
  return timingSafeEqual(bufA, bufB);
}

Used for: CRON_SECRET, BOT_API_KEY, webhook signatures.

Endpoint protection matrix

Health endpoint

Without auth  → { "status": "ok" }
With auth     → { "status": "healthy", "checks": { "database": "ok", "redis": "ok" }, "latencyMs": 3 }

Never publicly expose: DB type, Redis status, latency, or versions.

Startup secret validation

if (process.env.NODE_ENV === 'production') {
  const required = ['BETTER_AUTH_SECRET', 'ENCRYPTION_KEY', 'CRON_SECRET', 'REDIS_URL'];
  for (const key of required) {
    if (!process.env[key]) {
      console.error(`Missing required secret: ${key}`);
      process.exit(1);
    }
  }
}

9. Webhook Security

Inbound webhooks (receiving)

Each provider has its own signing method:

Common rules:

• Dedicated rate limiting (WEBHOOK: 20 req/min)
• Idempotency by event_id (no double-processing)
• Atomic transactions (INSERT-before-UPDATE)
• Security flag logging on signature failure

Outbound webhooks (sending)

• SSRF validation on URL (section 7)
• Strict timeout: AbortController with 5-30s
• Bot detection: return 204 for link preview bots (Discord, Slack, Twitter) to avoid triggering actions
• If URL is user-controlled → your server IP will be visible (see section 13)
• Restrict accepted domains when possible (e.g. only discord.com/api/webhooks/)

10. Monitoring

Key principle: monitoring must NEVER depend on Cloudflare

Monitoring runs on the same server → it must check via localhost, not via the public domain. Otherwise:

• Under Attack Mode blocks checks → false "major outage"
• Cloudflare down = monitoring down
• Public status page shows outage when the server is actually up

Architecture

Monitoring Worker (same server)
├── HTTP checks via http://127.0.0.1:3000 (+ Host header)
├── Redis check via localhost:6379
├── PostgreSQL check via localhost:5432
├── Disk space check
└── SSL certificate check

Results → DB (status_checks)
Transitions → Alerts (ntfy / Slack / email)

Localhost check implementation

Node.js fetch (undici) does not allow overriding the Host header. Use http.request:

import http from 'http';
 
function checkLocalService(path: string, hostHeader: string): Promise<number> {
  return new Promise((resolve) => {
    const req = http.request({
      hostname: '127.0.0.1',
      port: 3000,
      path,
      method: 'GET',
      timeout: 10000,
      headers: {
        'Host': hostHeader,
        'Authorization': `Bearer ${CRON_SECRET}`,
      },
    }, (res) => {
      res.resume();
      resolve(res.statusCode ?? 0);
    });
    req.on('error', () => resolve(0));
    req.on('timeout', () => { req.destroy(); resolve(0); });
    req.end();
  });
}

Recommended settings

11. Fail2ban

Jails

[DEFAULT]
banaction = nftables
backend = systemd
ignoreip = 127.0.0.1/8 ::1
bantime = 3600
findtime = 600
maxretry = 5
 
[sshd]
enabled = true
maxretry = 4
bantime = 7200           # 2h
 
[nginx-limit-req]
enabled = true
logpath = /var/log/nginx/error.log
maxretry = 10
findtime = 60
bantime = 600            # 10 min
 
[nginx-bad-request]
enabled = true
logpath = /var/log/nginx/access.log
maxretry = 10
findtime = 60
bantime = 3600           # 1h
 
[nginx-botsearch]
enabled = true
logpath = /var/log/nginx/access.log
maxretry = 5
findtime = 300
bantime = 86400          # 24h
 
[recidive]
enabled = true
logpath = /var/log/fail2ban.log
banaction = nftables[type=allports]
maxretry = 3
findtime = 86400         # 24h
bantime = 604800         # 1 week — ban all ports

The recidive jail is powerful: an attacker banned 3+ times in 24h by any jail gets banned on ALL ports for 1 week.

Fail2ban limitations: fail2ban parses logs in Python and runs iptables/nftables commands for each ban. At high volume (100k+ req/min), it falls behind and accumulates fatal lag. Fail2ban is a brute-force tool, NOT a DDoS tool. For DDoS, use automated detection + Cloudflare UAM (section 4) and/or kernel-level solutions (nftables sets, CrowdSec).

12. Security Headers

# /etc/nginx/snippets/security-headers.conf
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "geolocation=(), microphone=(), camera=(), payment=(), interest-cohort=()" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

Content-Security-Policy

• default-src 'self'
• Only allow third-party domains you actually use (Stripe, analytics, etc.)
• No * in any directive
• wss://*.yourdomain.com instead of wss: (restrict WebSockets)
• No sandbox/dev URLs in production
• frame-ancestors 'none'
• upgrade-insecure-requests

13. IP Leak Vectors

Defense in depth — 5 layers even if IP is known

Layer 1: UFW                          → Only Cloudflare IPs allowed
Layer 2: AOP (ssl_verify_client)      → nginx rejects without CF certificate
Layer 3: ssl_reject_handshake         → No cert revealed to scanners
Layer 4: error_page 495/496 =444      → Connection closed without response
Layer 5: fail2ban                     → Bans direct connection attempts

14. Encryption

Symmetric encryption (sensitive data in DB)

• Algorithm: AES-256-GCM
• Format: iv:authTag:ciphertext
• Random IV for every encryption
• Auth tag validates ciphertext integrity

Key and token generation

Critical environment variables

BETTER_AUTH_SECRET        Session signing (required)
ENCRYPTION_KEY            AES-256-GCM (exactly 64 hex chars)
CRON_SECRET               Bearer token for cron/health
PAYMENT_GATEWAY_KEY       Payment config encryption
REDIS_URL                 Redis connection string
BOT_API_KEY               Discord bot authentication

15. Queues and Workers

• Separate Redis client for BLPOP workers (don't block the main client)
• Retry with exponential backoff: 1s → 2s → 4s → 8s → 16s → max 60s
• Dead-letter queue for permanently failed notifications
• Auto-disable: a Discord webhook that fails 3+ times gets disabled in DB
• Graceful shutdown: SIGTERM → flush queue → close connections
• Fire-and-forget for notifications (don't block the critical path)

16. Validation Checklist

Network and Firewall

• UFW enabled, deny incoming default policy
• Ports 80/443 restricted to Cloudflare IPs only
• SSH restricted or protected by fail2ban
• PostgreSQL listening on localhost only
• Redis listening on localhost only
• App server listening on localhost only
• No other ports exposed (ss -tlnp)

Cloudflare

• All DNS records behind proxy (orange cloud)
• SSL/TLS mode Full (Strict)
• Authenticated Origin Pulls enabled (dashboard + nginx)
• Bot Fight Mode enabled
• API Token created for DDoS monitor

nginx

• server_tokens off
• Default server: 444 (HTTP) + ssl_reject_handshake on (HTTPS)
• real_ip_header CF-Connecting-IP with all CF ranges
• Rate limiting per endpoint type
• limit_conn per IP
• Proxy cache with proxy_cache_use_stale
• Security headers (HSTS, CSP, X-Frame-Options, etc.)
• proxy_hide_header X-Powered-By and Server

Application

• ORM with parameterized queries
• Redis sliding window rate limiting on all endpoints
• IP ban escalation
• SSRF protection on all user-controlled URLs
• Webhook signatures verified (HMAC, timing-safe)
• Cron endpoints protected by secret
• /api/health reveals nothing without auth
• No stack traces in error responses
• Sessions: HttpOnly, Secure, SameSite
• Secrets validated at startup
• Timing-safe comparisons for all secrets

Monitoring

• Health checks via localhost (not public domain)
• Worker uses http.request (not fetch) for Host header
• Alerts on transitions only (not every check)
• Status page independent of Cloudflare