Istio Service Mesh in Production

Running a service mesh in production is a significant undertaking. After 18 months operating Istio across three Kubernetes clusters, here are the lessons that matter most.

Why Service Mesh?

We adopted Istio to solve three specific problems:

• mTLS everywhere without application changes
• Traffic management for canary deployments
• Observability with distributed tracing

Installation Strategy

We use the Istio operator for reproducible installations:

istioctl install --set profile=production \ -y \ --set meshConfig.accessLogFile=/dev/stdout \ --set values.pilot.resources.requests.cpu=500m

Enable sidecar injection per namespace:

kubectl label namespace production istio-injection=enabled

mTLS Configuration

Start with permissive mode, then migrate to strict:

apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: istio-system spec: mtls: mode: STRICT

Traffic Management

Canary deployments with VirtualService and DestinationRule:

apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: my-app spec: http: - route: - destination: host: my-app subset: stable weight: 90 - destination: host: my-app subset: canary weight: 10

Performance Impact

In our benchmarks, Istio sidecars add approximately:

• 2-3ms latency per hop
• 50-100MB memory per sidecar
• 0.1 CPU cores per sidecar at moderate traffic

Conclusion

Istio delivers real value for security and observability at scale, but requires dedicated operational expertise. Invest in training your team before rolling out to production.