Traefik + Cloudflare: Keeping the origin server a secret

I recently started using Cloudflare proxying for all of the services I host under *.benscobie.com to reduce the attack surface on the origin server. In order for this to work effectively we need to ensure we aren’t leaking the origin server’s IP address in any way. This post details the steps I took to mitigate this when using traefik as a reverse proxy.

UPDATE 18/02/2024: Since writing this article I have switched to using Cloudflare Tunnels which removes some complexity mentioned in this article.

A note about my chosen Cloudflare setup

First off I think it is important to note that I am using the Full (strict) encryption mode in Cloudflare. I already had traefik setup to automatically generate SSL certificates via LetsEncrypt so it made the most sense to me. I am now also using a modern SSL setup as my only clients in the Cloudflare proxy scenario are the proxy servers themselves which support TLS 1.3 just fine.

Personally I think you should be using either Full or Full (strict) encryption modes so that your client’s connections are secure all of the way to your origin server. Though if you are determined to use the less secure modes, you can utilise the IPWhiteList middleware in traefik and whitelist Cloudflares ranges to only allow Cloudflare proxy traffic.

Now onto the fun stuff…

Securing HTTPS traffic

You might think that simply switching your DNS to Cloudflare, enabling proxying and leaving it at that is good enough. However, if you’re using Cloudflare to hide your origin server you’ll want to go a step further as a determined attacker could still potentially discover your origin server.

To demonstrate this we can run the following curl command:

curl --resolve www.benscobie.com:443:originserverip https://www.benscobie.com/

What we’re doing here is sending an SNI-compliant request directly to the origin web server (traefik). And surprise surprise we get a response:

What does this mean for us? Simply put, an attacker could iterate over the entire worlds IP space (really not that hard to do these days if using IPv4!) and send the same exact request. Eventually they’ll hit your origin server and/or Cloudflare, and as their IP ranges are public it won’t be hard to figure out which of those is the origin server.

To fix this we’re going to enable Cloudflare’s authenticated origin pulls. With this we can validate that requests come from the Cloudflare network.

Navigate to the setup page, under the section Zone-Level Authenticated Origin Pull using Cloudflare certificate, download the .PEM file and put this somewhere where traefik can access it.

Next, configure traefik client authentication (mTLS). I’m using yml dynamic configuration which looks like the following:

tls:
  options:
    default:
      sniStrict: true
      clientAuth:
        caFiles:
          - /opt/traefik/certs/authenticated_origin_pull_ca.pem
        clientAuthType: RequireAndVerifyClientCert

I’m also enabling strict SNI checking here to stop traefik responding with a default certificate when non-SNI requests are received.

With all of that in place and running the same command again, we get the following generic error that doesn’t leak any information about the site:

curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_CERT_UNKNOWN (0x80090327) - An unknown error occurred while processing the certificate.

So that’s HTTPS traffic sorted out, but what about HTTP traffic?

Securing HTTP traffic

Let’s run the following command to send a HTTP request directly to our origin server:

curl -v -H "Host: benscobie.com" http://originserverip

The response looks like this:

> GET / HTTP/1.1
> Host: www.benscobie.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Location: https://www.benscobie.com/
< Date: Sat, 27 Aug 2022 16:46:53 GMT
< Content-Length: 17
< Content-Type: text/plain; charset=utf-8
<
Moved Permanently* Connection #0 to host x.x.x.x left intact

What’s happening here is that the browser is being told to redirect to our site using the HTTPs protocol, is a pretty standard thing to do these days. This falls foul to the same attack I mentioned previously whereby an attacker can iterate over IPs until they hit a response like this. They’d need only to validate that the Location contains the target domain.

Cloudflare can handle the redirection of HTTP to HTTPS for us so enable that. Once you’ve done this there shouldn’t be any reason to respond to HTTP traffic from the origin server, you can simply stop traefik listening on port 80 or disable that entrypoint in the traefik router.

Closing words

That’s it for the traefik and Cloudflare setup. There are more ways for an attacker to discover your origin server but resolving these is not in scope, but here are a few to keep in mind:

• Sending emails directly from your origin server will leak its IP address. Use an email sending service like SendGrid or Mailgun instead.
• Making server-side network requests based on user input could leak your origin server’s IP address. E.g. you have a utility to set a profile image based on a user provided URL.

I hope that this helped you in some way. Let me know if there’s something I could do to further improve my security posture here as I’m always looking to learn.