How to Set Up DDoS Protection for Your VPS

A DDoS attack floods your VPS with junk traffic until the network card can’t keep up or the services crash. Unlike a targeted hack, there’s no exploit to patch — you’re just getting buried under volume. The fix isn’t a single tool. It’s a combination of network-level filtering, server hardening, and knowing when to hand off to your hosting provider.

Know What You’re Dealing With

Before throwing mitigation tools at the problem, figure out the attack type. Run this on your VPS:

tcpdump -nn -c 1000 -i eth0 | awk '{print $3}' | sort | uniq -c | sort -rn | head -20

This shows the top source IPs hitting your box. If you see thousands of unique IPs each sending a few packets, that’s a distributed attack — you can’t IP-ban your way out. If it’s a smaller set of IPs sending SYN floods or UDP amplification, local filtering can help.

Check your connection states:

ss -s
netstat -an | awk '/tcp/ {print $6}' | sort | uniq -c | sort -rn

A normal server has mostly ESTABLISHED connections. If you see tens of thousands of SYN_RECV, someone is SYN flooding you.

Step 1: Kernel-Level SYN Flood Mitigation

SYN floods are the most common VPS-level DDoS. The Linux kernel has built-in SYN cookies that deflect them. Check if they’re on:

sysctl net.ipv4.tcp_syncookies
# If 0, enable it:
sysctl -w net.ipv4.tcp_syncookies=1
# Also raise the SYN backlog:
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
sysctl -w net.ipv4.tcp_synack_retries=2

Make these permanent by adding them to /etc/sysctl.d/99-ddos.conf and running sysctl -p.

Drop some aggressive defaults too:

net.ipv4.conf.all.rp_filter=1
net.ipv4.tcp_syn_retries=3
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_tw_reuse=1

Step 2: Rate Limiting with iptables or nftables

For connection floods (not SYN floods), rate limiting per IP works well. With iptables:

# Limit new connections per source IP to 20/minute
iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -m recent --set
iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 20 -j DROP

# Limit SSH
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 5 -j DROP

If you’re on nftables, the logic is the same, just different syntax. Save rules with iptables-save > /etc/iptables/rules.v4 (Debian/Ubuntu) or service iptables save (CentOS).

Step 3: Use Fail2ban for Application-Layer Floods

Fail2ban watches log files and bans IPs that hit too fast. It’s not a DDoS silver bullet, but it handles HTTP floods where bots hammer your web server:

# /etc/fail2ban/jail.local
[nginx-limit-req]
enabled = true
maxretry = 30
findtime = 60
bantime = 3600

Make sure your nginx has limit_req_zone configured, otherwise fail2ban has nothing to parse.

Step 4: Front It with Cloudflare or a CDN

This is the part that actually handles large volumetric attacks. A VPS with 1 Gbps uplink will go down at 2-3 Gbps of junk traffic. Cloudflare’s free plan absorbs that before it reaches your server.

Switch your DNS to Cloudflare, enable the “Under Attack” mode during an active attack, and configure rate limiting rules. The free tier handles most Layer 3/4 floods. For Layer 7 (HTTP) attacks, you may need the Pro plan’s WAF rules.

Other options: your hosting provider’s DDoS protection (Vultr, Hetzner, OVH all have something), AWS Shield if you’re on AWS, or a dedicated scrubbing service like Path.net.

Step 5: Nginx Hardening for Layer 7 Attacks

If bots are hitting specific URLs (like /xmlrpc.php or /wp-login.php), block them at the web server:

# nginx.conf http block
limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=login:10m rate=1r/s;

# server block
location / {
    limit_req zone=general burst=20 nodelay;
}
location /wp-login.php {
    limit_req zone=login burst=5 nodelay;
}
location = /xmlrpc.php {
    deny all;
}

When Local Mitigation Isn’t Enough

If your upstream bandwidth is saturated, no amount of server-side config will help. The packets are already arriving. At that point you need upstream filtering — either your provider’s DDoS protection or a CDN in front. Contact your provider’s support; most will null-route the attacked IP temporarily, which takes your service down but protects their network.

For more server hardening guides, visit fenij.com.

FAQ

Q: Will SYN cookies slow down my server?
A: Negligibly. SYN cookies only activate under SYN flood conditions. Normal traffic goes through the standard handshake. The CPU cost of generating cookies is trivial on modern hardware.

Q: Cloudflare is in front, do I still need iptables rate limiting?
A: Yes. Cloudflare handles volumetric attacks, but if an attacker finds your origin IP (through DNS history, email headers, or a subdomain not proxied), they bypass Cloudflare entirely. Rate limiting is your backstop.

Q: How do I hide my origin IP behind Cloudflare?
A: Don’t put the origin IP in any DNS record that isn’t proxied (orange cloud). Check crt.sh for certificate transparency leaks. Don’t send email directly from the web server — use a separate mail service. Scan your DNS records for anything pointing to the origin.