Bring Your Own Prometheus

Prometheus is an open-source monitoring tool. By default, Prometheus is installed alongside Istio, allowing you to use Grafana and Kiali to view metrics for both the Istio control plane and your Envoy-injected workloads.

But what if you’re already running Prometheus on your cluster, or you want to add extra customization to Istio’s Prometheus installation (for instance, add Slack notifications for Istio )?

Not to worry. You can bring your own Prometheus to Istio, with three quick steps.

First, update your Prometheus configuration. Prometheus relies on a scrape config model, where targets represent /metrics endpoints, ingested by the Prometheus server.

We’ll add targets for each of the Istio components, which are scraped through the Kubernetes API server. For instance, here is the configuration for Istio’s Pilot component:

    - job_name: 'pilot'
      - role: endpoints
          - istio-system

      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: istio-pilot;http-monitoring

See configmap.yaml for a full example.

Second, update your Prometheus deployment to mount Istio’s certificates into Prometheus. This allows Prometheus to scrape Istio workloads when mutual TLS is enabled. To do this, mount in the istio.default secret into your Prometheus deployment YAML:

    - name: config-volume
        name: prometheus
    - name: istio-certs
            defaultMode: 420
            optional: true
            secretName: istio.default

See deployment.yaml for the full example.

Once we deploy Prometheus with this new configuration, we have a Deployment and a Service running in a separate monitoring namespace:

$ kubectl get service -n monitoring

NAME         TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)          AGE
prometheus   LoadBalancer   <IP>             9090:32352/TCP   21m

Lastly, update Istio’s configuration to use a custom Prometheus address. Here’s a helm template example using the Istio installation options:

helm template install/kubernetes/helm/istio --name istio --namespace istio-system \
--set prometheus.enabled=false \
--set kiali.enabled=true --set kiali.createDemoSecret=true \
--set kiali.prometheusAddr="http://prometheus.monitoring.svc.cluster.local:9090" \
--set "kiali.dashboard.jaegerURL=http://jaeger-query:16686" \
--set "kiali.dashboard.grafanaURL=http://grafana:3000" \
--set grafana.enabled=true \
--set grafana.datasources.datasources.datasources.url="http://prometheus.monitoring.svc.cluster.local:9090"  > istio.yaml

Once Istio and Prometheus are both installed, and we deploy some Istio-injected workloads to our cluster, we can see that Prometheus is successfully scraping our Istio targets:

Grafana can fetch service-level metrics:

And Kiali can display the service graph: