How to Read a Remote IP Address Behind Ingress and Load Balancer in OKE

Introduction

Oracle Container Engine for Kubernetes, also known as OKE, is a powerful Kubernetes managed service, and the tip of the spear of the cloud-native stack in Oracle Cloud Infrastructure. It’s tightly integrated with core cloud components, such as load balancers and networking. Every modern cloud-native deployment would imply your containerized application executes in Kubernetes, exposed to the world through an ingress controller and load balancer on top of it. Even more, ingress would take care of the certificates for each domain and path with the cert-manager. OKE is integrated with OCI Load Balancer, and you can easily install multiple ingress controllers, such as Ingress Nginx Controller.

In advanced scenarios, you might want to know a remote client IP address from the container or ingress controller. It might be necessary for blue/green deployments, where you would allow specific source IP addresses to preview some canary features. Typical Ingress Nginx Controller installation is cloud-agnostic and spins default OCI Load Balancer settings. OCI Load Balancer is a flexible proxy-based load balancer operating on layer 4 and layer 7 of the OSI model. Default settings imply that the load balancer listener doesn’t propagate remote IP addresses through x-real-ip or x-forwarded-for headers, since the listener initially uses TCP. No headers on a TCP level, right? You might then use the OCI console (or add custom annotations to ingress controller manifests) and change the load balancer listener to use HTTP or HTTPS. That would fix the problem and expose remote IP addresses in headers but potentially lead you to another problem — forcing SSL termination on the load balancer level. If you would like to use cert-manager, it’s not the happiest option.

A solution is to implement OCI Network Load Balancer in front of the ingress controller. OCI Network Load Balancer is an alternative solution, a non-proxy load-balancer operating on layer 4 of the OSI model. It implies it wouldn’t do SSL termination, but it’s capable of forwarding client source addresses if configured to do so.

Install Ingress Nginx Controller with OCI Network Load Balancer in Front

1

Follow Oracle’s official guide for setting up Ingress Nginx Controller: https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengsettingupingresscontroller.htm

2

When you come to the step of deploying ingress-nginx-controller, just before applying kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.3/deploy/static/provider/cloud/deploy.yaml make a stop and download the deployment manifest instead. Modify it by adding annotations to provision OCI Network Load Balancer instead of the default one (marked bold in the snippet below).

apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.23.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-nlb
  namespace: ingress-nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
    oci-network-load-balancer.oraclecloud.com/is-preserve-source: "true"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

Custom annotation oci.oraclecloud.com/load-balancer-type: “nlb” and specification property externalTrafficPolicy: Local will enable remote IP address passthrough inside the Nginx Controller, all the way to the container.

3

Execute the downloaded and updated manifest with kubectl apply -f downloaded_manifest.yaml, and complete guide from step one.

4

Since specification property externalTrafficPolicy: Local brings remote IP address as the source address for your worker nodes, make sure you update the worker node subnet security list accordingly. You will most probably need to add a security list ingress rule allowing source 0.0.0.0/0 (remote client IP) to reach worker node ports (30000–32767). You can find more info in the official docs.

5

Invoke your application and check if you can read x-real-ip or x-forwarded-for headers from the container. In my case, it works like a charm, writing remote client IP addresses in headers.

{"host":"remoteip-nlb.micro.ivandelic.com","x-real-ip":"74.34.156.180","x-forwarded-for":"74.34.156.180","x-forwarded-host":"remoteip-nlb.micro.ivandelic.com","x-forwarded-port":"443","x-forwarded-proto":"https"...}

What If You Are Behind Cloudflare Proxy?

Since we used OCI Network Load Balancer operating on layer 4, all headers set up by Cloudflare are forwarded to Ingress Nginx Controller. This implies the existence of CF-Connecting-IP header in each HTTP request. By default, Nginx uses the content of the header X-Forwarded-For to get information about the client's IP address. It’s inconvenient for our scenario since X-Forwarded-For holds the IP addresses of Cloudflare proxies. The reason for this behavior relies on the fact that OCI Network Load Balancer doesn’t recognize the existence of the Cloudflare proxy, and sets up the proxy IP address as the remote one. The obvious workaround is to configure Nginx to use CF-Connecting-IP the header to get the real remote IP address, instead of X-Forwarded-For.

To make it work, modify the downloaded Ingress Nginx manifest by adding a custom configuration in ConfigMap, as in the example below.

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.23.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  proxy-real-ip-cidr: "173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32"
  use-forwarded-headers: "true"
  forwarded-for-header: "CF-Connecting-IP"

Note: If you haven’t already done so, you can sign up for an Oracle Cloud Free Tier account today. It’s not necessary to follow along to with this post, if you’re curious about how to get started with OCI, signing up is the first step!

If you’re curious about the goings-on of Oracle Developers in their natural habitat, come join us on our public Slack channel!