nginx ingress with istio
Using nginx ingress with istio service mesh
So you started using Kubernetes (which is great). To get traffic into your cluster you need to choose an ingress implementation. Luckily several are available and there are a lot of good reasons to choose nginx as your ingress. After all nginx is a great web server and makes a great ingress for your kubernetes installation.
But now you read about service meshes and want to have a service mesh. We decided to go with istio. There are probably other great service meshes out there (have an eye on conduit ), but for us istio was a great match. If you follow the tutorials they want you to install an istio ingress. But you already have a nice nginx based ingress. And in our case we were using some features available only on nginx and not on istio. So how are we gonna use nginx with istio and vice versa?
First attempt, simply use the sidecar.
Since it is our goal to receive traffic on our nginx ingress and immediately push this traffic into out service mesh
I thought it would be a great idea to simply inject the istio sidecar into our ingress controller deployment.
Thought and done thanks to the great
istio inject command. Everything deployed as expected, but the test site
wouldn’t load. A closer look at the connection via curl showed an interesting error.
* TLSv1.2 (OUT), TLS handshake, Client hello (1): * LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to test.example.com:443 * stopped the pause stream! * Closing connection 0 curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to test.example.com:443
We aren’t even getting the TLS handshake let alone can send our HTTP request. So simply just injecting the istio sidecar wasn’t gonna cut it.
What does the istio sidecar do?
The istio sidecar is simply an envoy proxy with some added logic to generate an envoy
configuration of the state of your service mesh. But all of this is not the part breaking our TLS connection. Istio is
also injecting an init container (actually two of them). The
proxy_init container calls a script which configures the
iptables for this pod. This is done so that istio can act as a transparent proxy. Our application doesn’t have to care
about correctly setting proxy variables, configuring your http client etc. But these iptables rules also make sure that
all incoming traffic to this pod is routed through envoy. In the default use case for the istio sidecar this is totally
fine and what you actually want.
So in essence all of our incoming as well as our outgoing traffic is redirected with the help of iptables to go through our envoy sidecar proxy. But all we want is that our outgoing traffic is redirected.
In case you are interested, the resulting nat table looks like this:
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 129K 7719K ISTIO_REDIRECT all -- any any anywhere anywhere /* istio/install-istio-prerouting */ Chain INPUT (policy ACCEPT 129K packets, 7719K bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 542K packets, 50M bytes) pkts bytes target prot opt in out source destination 444K 27M ISTIO_OUTPUT tcp -- any any anywhere anywhere /* istio/install-istio-output */ Chain POSTROUTING (policy ACCEPT 842K packets, 68M bytes) pkts bytes target prot opt in out source destination Chain ISTIO_OUTPUT (1 references) pkts bytes target prot opt in out source destination 299K 18M ISTIO_REDIRECT all -- any lo anywhere !localhost /* istio/redirect-implicit-loopback */ 143K 8564K RETURN all -- any any anywhere anywhere owner UID match 1337 /* istio/bypass-envoy */ 1282 76920 RETURN all -- any any anywhere localhost /* istio/bypass-explicit-loopback */ 903 54180 ISTIO_REDIRECT all -- any any anywhere 10.0.0.0/8 /* istio/redirect-ip-range-10.0.0.0/8 */ 5 300 RETURN all -- any any anywhere anywhere /* istio/bypass-default-outbound */ Chain ISTIO_REDIRECT (3 references) pkts bytes target prot opt in out source destination 429K 26M REDIRECT tcp -- any any anywhere anywhere /* istio/redirect-to-envoy-port */ redir ports 15001
If we modify the PREROUTING chain we can exclude traffic from the redirect to istio.
Fixing the iptables for usage in an ingress
So now we identified our main problem. Traffic is not hitting nginx directly, but our envoy sidecar proxy. Since we now know that all this magic is realised via iptables, we can start modifying the iptables rules.
A first look at the init script of the proxy_init init container suggests that we could simply set the
ISTIO_LOCAL_EXCLUDE_PORTS environment variable to achieve what we want. At least if we also set
* it should work, but it actually doesn’t. I don’t really know why this doesn’t work, so if you have an idea I would be glad
to hear from you.
But you are not forced to rely on the init script of the proxy_init container. My current, very simplistic solution is to create another init container with the image of the proxy_init container (because it has already all the necessary tools to manipulate iptables) and overwrite the command in the container. This allows us to simply spawn a bash process which we can use to execute an iptables command. The whole construction looks like this:
- name: ingress-iptables args: - -c - iptables -t nat -I PREROUTING -p tcp -m multiport --dports 80,443 -j RETURN command: - /bin/bash image: docker.io/istio/proxy_init:0.4.0 imagePullPolicy: IfNotPresent securityContext: privileged: true
As you can see all this does is to create a rule in the PREROUTING chain of the nat table to exclude traffic received via TCP
on ports 80 and 443. Since we are manipulating iptables the privileged security context is required, but you can probably also
only give the container the
The best of both worlds
After adding this init container you can now use nginx as your ingress. nginx will be configured via its ingress controller and create its requests to upstream services as usual. The only difference now is that all outgoing requests from nginx are routed through envoy into our service mesh.