One way to think about a service mesh is as a domain of control. Within a Kubernetes namespace where Istio sidecar injection is enabled, you can monitor all traffic between Pods, and enforce security policies.
But what about upstream services that live outside the mesh? How do you determine at runtime which services call external APIs? How do you know which database instance your service is writing to? Or how do you ensure that a service inside the mesh is only sending traffic within its own geographic region? Istio egress monitoring can help with this.
Egress means exit. In this case, egress traffic means requests that must exit the Istio mesh.
There was a time when Istio blocked all egress traffic by default. You had to manually create a
ServiceEntry to whitelist every external host your services needed to access. A ServiceEntry adds an external host to Istio's service registry. This changed in Istio 1.3, when the
REGISTRY_ONLY egress policy became
ALLOW_ANY by default. Now, in-mesh services can access external services freely, without the need for a
No matter which Istio egress option you choose for your mesh, Istio can monitor all egress traffic. And you can monitor this egress traffic through your workloads’ sidecar proxies, without needing a dedicated egress gateway proxy. Let's see how it works.
In this example, we've built a website that lets users share recipes. To optimize costs, the web frontend runs outside of Kubernetes as a serverless function. When a user adds a recipe, the frontend creates an ID for that recipe by calling the ID Generator service (
idgen) inside a Kubernetes cluster.
idgen is exposed through the default Istio IngressGateway, and gets random IDs from an external API called
Option 1 - Passthrough
To start, let's use an Istio installation with the default
ALLOW_ANY option for egress. This means that
idgen's requests to
httpbin are allowed with no additional configuration. When
ALLOW_ANY is enabled, Istio uses an Envoy cluster called
PassthroughCluster, enforced by
idgen's sidecar proxy, to monitor the egress traffic.
An Envoy cluster is a backend (or “upstream”) set of endpoints, representing an external service. The Istio sidecar Envoy proxy applies filters to intercepted requests from an application container. Based on these filters, Envoy sends traffic to a specific route. And a route specifies a cluster to send traffic to.
Passthrough cluster is set up so that the backend is the original request destination. So when
ALLOW_ANY is enabled for egress traffic, Envoy will simply “pass through”
idgen's request to
With this configuration, if we send recipe ID requests through the IngressGateway,
idgen can successfully call
httpbin. This traffic appears as
PassthroughCluster traffic in the Kiali service graph - we'll need to add a
ServiceEntry in order for httpbin to get its own service-level telemetry. (We'll do this in a moment.)
But if we drill down in Prometheus, and find the
istio_total_requests metric, we can see that
PassthroughCluster traffic is going to a
Option 2 -
REGISTRY_ONLY, no ServiceEntry
Now let's say that before we add a
ServiceEntry for httpbin, we want to lock down all egress traffic. We can do this by updating the global installation option for outbound traffic to be
REGISTRY_ONLY, and re-applying the Istio install manifests.
Now, a new cluster called
BlackHole comes into play. The Black Hole cluster is a backend without any IP endpoints. Requests routed to the
BlackHoleCluster are dropped by Envoy, and return a
502: Bad Gateway error. In action, the collection of sidecar proxies dropping egress requests is how the
REGISTRY_ONLY policy is enforced.
Once we re-install Istio with the
REGISTRY_ONLY option enabled, and redeploy our
idgen pod, we can see that the
BlackHoleCluster intercepts the requests. A red graph edge means that HTTP requests do not complete - traffic can't get to the desired
In Prometheus, we can see that the
istio_total_requests metric is accounting for
BlackHoleCluster traffic. In practice, you might set an alert on this metric to detect attempted data exfiltration by a service in your cluster. In this mode, Prometheus can tell you both the source and (attempted) destination workload for the blocked request.
Option 3 -
REGISTRY_ONLY with ServiceEntry
Now let's say that we've gotten approval for
idgen to call an external API. We've authorized the creation of a
ServiceEntry to add
httpbin to the Istio Registry:
apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: httpbin-ext spec: hosts: - httpbin.org ports: - number: 80 name: http protocol: HTTP resolution: DNS location: MESH_EXTERNAL
Now, we can see that requests successfully exit the mesh, and are not dropped by the
And note that with a ServiceEntry, Istio treats httpbin as another distinct mesh service, even though it's external to the Kubernetes cluster and not part of the control domain. Now, we can get telemetry specifically for
httpbin, and if we add another external service, it would appear as its own distinct node in the service graph.
To learn more about monitoring egress traffic: