yourgeek.it_

~/ real-client-ip-behind-proxy

Logs Full of Cloudflare IPs: Finding the Real Client

#networking#security#infrastructure#troubleshooting

Logs full of Cloudflare IPs

At a company I worked for, the web servers sat behind Cloudflare. One day I opened the Apache access logs to find my own requests, from my own PC, and couldn’t. The same handful of addresses was on every line, and none of them was mine, or any visitor’s: they all belonged to Cloudflare. It had been like that for a few weeks.

On another setup the logs were even less helpful. Apache sat behind an nginx reverse proxy on the same host, and every request came from a private 172.x address: the proxy itself, talking to Apache over an internal network.

The cause is the same in both cases, and it isn’t a bug. When a proxy sits in front of your application, the TCP connection your application accepts is opened by the proxy, not by the client. The address on the socket is correct: it’s the last machine that talked to you. It just isn’t the one you care about.

What breaks when every client is the proxy

The logs are the first thing to go, and the first thing you notice. An access log where every request comes from a handful of proxy addresses can’t answer the question you opened it for: who did this? You can’t follow one visitor through a session, you can’t tell a crawler from a user, and during an incident you can’t pull the attacker’s requests out of the noise.

Anything that makes a decision based on the client address breaks too, and it breaks quietly:

  • Rate limits count every visitor as the same client. Set the limit low and the whole site is throttled as soon as traffic picks up. Set it high enough that this never happens and it never stops anyone.
  • Bans have the same problem. fail2ban, or any tool that reads the logs and blocks an address after too many failed logins, will eventually block the proxy, and with it every user. If the proxy’s address is excluded so this can’t happen, nobody is ever blocked.
  • Allowlists for an admin area either let nobody in or let everyone in, depending on whether the proxy’s address is on the list.
  • Geolocation reports where the proxy is, not where the user is.

None of these produces an error. The application keeps running, the requests keep being served, and the numbers look plausible. That is why the logs are often the only place you see it.

Read the header your proxy writes

The proxy knows who the client is, because it accepted the client’s connection. Since it can’t pass that on through the socket, it passes it on as a request header. There are a few of them, and which one you get depends on the proxy:

  • X-Forwarded-For is the most common. It’s a comma-separated list: each proxy appends the address it received the request from, so the first entry is the original client and the following ones are the proxies along the way.
  • X-Real-IP holds a single address. It’s a convention, not a standard: nginx sets it if you configure it to, and Traefik sets it by default.
  • CF-Connecting-IP is Cloudflare’s own: a single address, the client as Cloudflare saw it.
  • Forwarded is the standardized version (RFC 7239). It carries the same information in a structured format. I rarely find it in the wild.

The rule that follows: when a request comes from a proxy you trust, the client address is in one of these headers, not on the socket. When it doesn’t, the socket address is still the only fact you have, and you keep it. Which proxies to trust is what the rest of this post is about.

The best place to apply it is usually the web server, not the application code. Apache, nginx and Traefik can all replace the socket address with the one from the header before anything else sees the request. The access log, the rate limiter and every application behind it then get the right address, without each one parsing headers on its own.

In Apache it’s mod_remoteip. Behind Cloudflare:

RemoteIPHeader CF-Connecting-IP
# One line per range Cloudflare publishes
RemoteIPTrustedProxy 173.245.48.0/20

Use %a in the LogFormat: it reports the address mod_remoteip resolved.

In nginx it’s the realip module:

# One line per range Cloudflare publishes
set_real_ip_from 173.245.48.0/20;
real_ip_header CF-Connecting-IP;

In Traefik it’s an option on the entry point:

entryPoints:
  websecure:
    address: ":443"
    forwardedHeaders:
      trustedIPs:
        - "173.245.48.0/20"

All three configurations have two parts: which header to read, and a list of addresses. The second part is the one that matters.

Anyone can write that header

A request header is text the sender chooses. Nothing stops a client from sending one of these headers itself:

curl -H "X-Forwarded-For: 203.0.113.7" https://example.com/login

An application that reads X-Forwarded-For blindly now believes the request came from 203.0.113.7. The attacker picks the address, and the decisions from the previous section turn against you:

  • a new fake address on every request, and the rate limit never triggers
  • the address of the office, and the allowlist on the admin area lets them in
  • someone else’s address, and that’s who ends up in your logs and your ban list

Proxies don’t clean this up for you. When a request already has an X-Forwarded-For, Cloudflare appends the address that connected to it instead of replacing the header. For a client talking to Cloudflare directly, that’s the real address, and the application receives 203.0.113.7, <real client>. Take the first entry, as most examples online do, and you take the one the attacker wrote.

CF-Connecting-IP is safer, because Cloudflare overwrites it. But only if the request really went through Cloudflare. A server behind Cloudflare still has its own public address, and that address can be found: in old DNS records, in certificates, or by scanning, which finds an exposed server in minutes. Anyone who reaches the origin directly can send CF-Connecting-IP with any value they like, and the origin has no way to tell it from the real one.

So a header is only as trustworthy as the machine that sent it. The list of addresses in the configurations above is exactly that check: read the header only when the connection comes from one of your proxies, otherwise keep the socket address. It’s also why those lists need the proxy’s actual ranges, kept current, and not a catch-all like 0.0.0.0/0.

The other half is closing the door around the proxy. If the origin accepts web traffic only from the proxy’s addresses, enforced outside the machine at the cloud firewall, nobody can talk to it directly, and the header can’t arrive from anyone else. I’ve seen the same log symptom from the start of this post on servers that accepted web traffic from anyone, directly. On those, making the logs right by trusting the header, without closing the port first, would only have made them believable.

Counting hops

With one proxy, the choice is easy. With two or more, X-Forwarded-For turns into a list, and which entry you take decides whether you get the client, a proxy, or the attacker’s fake.

Take a common chain: Cloudflare, then Traefik on the server, then the application. All the addresses in this example are from the ranges reserved for documentation, including the one standing in for Cloudflare. A client at 198.51.100.20 sends a request with a forged header, and Traefik is configured to trust Cloudflare’s ranges. Each hop adds what it saw:

HopConnection comes fromX-Forwarded-For after this hop
Client—203.0.113.7 (forged)
Cloudflare198.51.100.20203.0.113.7, 198.51.100.20
Traefika Cloudflare node (192.0.2.10)203.0.113.7, 198.51.100.20, 192.0.2.10

The application receives all three. The only entries it can rely on are the ones written by machines it trusts, and those are always at the end of the list, because each hop appends. So the list is read from the right:

  1. The last entry is the Cloudflare node Traefik received the request from. It’s in a trusted range: skip it.
  2. The next one, 198.51.100.20, is what Cloudflare saw. It isn’t a proxy of yours: that’s the client.
  3. Everything to its left was written by the client and is ignored.

That is the rule: from the right, skip the addresses of your own proxies, and the first one left is the client. nginx does exactly this with real_ip_recursive on, and mod_remoteip walks the list the same way through its trusted proxies.

The tempting shortcut is to count instead: “we have two proxies, take the second entry from the right”. It works until someone adds a load balancer, or a request takes a different path, and then the count is silently off by one. A list of trusted ranges adapts to the topology; a fixed count assumes it never changes.

The opposite mistake is also easy. Without trustedIPs, Traefik doesn’t trust the incoming X-Forwarded-For at all: it discards it and writes its own, containing only the Cloudflare node. Nothing is forgeable any more, but the client is gone from the header too, and X-Real-IP holds the same Cloudflare address. In that setup the only header still carrying the client is CF-Connecting-IP, because Traefik leaves headers it doesn’t manage alone.

The 172.x case from the start is the same pattern with a private address: nginx is the trusted proxy, on an internal network, and Apache reads the client from the header only when the connection comes from nginx.

A small tool behind two proxies

myconf.it is a set of free tools for sysadmins that I run. One of them, IP geolocation, tells you where an address is. The page runs on Cloudflare, and the lookup itself is done by an API on my server, behind Cloudflare and then Traefik: the chain from the previous section.

On the Cloudflare side, the client address comes from CF-Connecting-IP. There it is set by Cloudflare’s edge itself, and there is no origin behind it to reach directly. The page passes the address to the API explicitly, as a parameter.

The API also had a fallback for calls without that parameter: work out the caller’s address from the headers. This is how it looked:

fn client_ip(headers: &HeaderMap) -> Option<String> {
    for key in ["cf-connecting-ip", "x-real-ip", "x-forwarded-for"] {
        if let Some(v) = headers.get(key).and_then(|v| v.to_str().ok()) {
            let first = v.split(',').next().unwrap_or("").trim();
            if let Ok(ip) = first.parse::<IpAddr>() {
                return Some(ip.to_string());
            }
        }
    }
    None
}

CF-Connecting-IP first is right. The two fallbacks are the problems of this post in a few lines. Behind a proxy that rewrites the header, X-Real-IP and X-Forwarded-For contain a Cloudflare address, so a fallback would have geolocated Cloudflare. And split(',').next() takes the first entry of X-Forwarded-For: in any setup that keeps the incoming header, that’s the one the client wrote.

Requests that come through Cloudflare always carry CF-Connecting-IP, so in practice the fallbacks were never reached. They were dead code, but wrong dead code, and code like that gets copied. While writing this post I removed them:

fn client_ip(headers: &HeaderMap) -> Option<String> {
    headers
        .get("cf-connecting-ip")
        .and_then(|v| v.to_str().ok())
        .and_then(|v| v.trim().parse::<IpAddr>().ok())
        .map(|ip| ip.to_string())
}

One header, the one this chain actually fills in. If it’s missing, the API doesn’t guess: it answers with an error and asks for the address as a parameter.

The address doesn’t protect anything here, either. The rate limit on the lookup is tied to the account making the call, not to its address. A forged header would only change which address gets looked up, and the parameter already lets anyone look up any address. That’s why getting it wrong cost nothing on this tool. On a login form, it would.

What to take from it

  • Behind a proxy, the socket address is the proxy. On connections from it, read the client from the header it writes.
  • Fix it once in the web server or the ingress (mod_remoteip, realip, trustedIPs), not in every application.
  • Trust the header only on connections from your own proxies, listed by their real ranges.
  • Close the origin to everyone but the proxy, at the cloud firewall, or the header is just a suggestion.
  • Read X-Forwarded-For from the right, skipping trusted proxies. Never take the first entry.
  • Don’t count hops: the topology will change and the count won’t.
  • Ask what each IP-based decision protects. If the answer is “a login”, the client address must be right.

← all posts