Leveraging Red Hat OpenShift Service Mesh for Secure, Observable, and Resilient Microservices

Problem Statement

In a modern OpenShift 4.x environment, teams want to expose microservices with mutual TLS (mTLS), fine‑grained traffic control, and rich telemetry without rewriting application code or managing complex networking rules. The question is: how can we add a service‑mesh layer that is fully supported by Red Hat, keeps the deployment simple, and still provides robust observability and resilience?

Architecture Note Overview

The smallest viable design is a sidecar‑injected deployment where each application pod runs an Envoy proxy (the istio-proxy container) that handles routing, mTLS, and telemetry. This approach requires no changes to the application containers, keeps trust boundaries clear, and allows OpenShift’s built‑in monitoring stack to surface metrics.

1. Requirements

• OpenShift 4.x with the openshift-service-mesh Operator installed (verified via oc get catalogsource -n openshift-marketplace).
• Cluster‑wide ServiceMeshControlPlane CRD applied to enable the control plane (control‑plane pods, Pilot, Galley, Citadel, and the ingress gateway).
• Namespace‑level annotation sidecar.istio.io/inject: "true" to activate automatic sidecar injection.
• Certificates issued by an internal CA managed by Istio’s Citadel (or an external CA integrated via cert-manager).
• Prometheus and Grafana operators running to collect and display metrics.

2. Smallest Suitable Design

Deploy each microservice as a standard Deployment or StatefulSet. Add the annotation to the namespace (or to individual pods) and let the istio-proxy container be injected automatically. The resulting pod layout looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: web-app
        image: registry.example.com/web-app:latest
        ports:
        - containerPort: 8080

After rollout, oc get pod -n demo -o jsonpath='{.spec.containers[*].name}' should list both web-app and istio-proxy.

3. Trust and Data Boundaries

• Control Plane Boundary: All traffic entering the mesh must pass through the Istio ingress gateway (a Service named istio-ingressgateway in the istio-system namespace). The gateway terminates TLS using a client certificate that the gateway presents to the service.
• Service‑to‑Service Boundary: Within the mesh, mTLS is enforced by default. The Envoy sidecar negotiates a TLS session with the peer sidecar, using certificates signed by Citadel’s internal CA.
• External services not in the mesh can be exposed via Gateway and VirtualService objects that keep them isolated from the internal mTLS policy.

4. Operational Checks

1. Verify CRDs

   oc get crds | grep istio.io
   # Expected output includes istiooperators.istio.io, serviceentries.istio.io, etc.
   

2. Check Sidecar Injection

   oc get pod -n demo -o jsonpath='{.items[*].spec.containers[*].name}'
   # Should contain 'istio-proxy' for each pod.
   

3. Validate Ingress Gateway

   oc get svc -n istio-system istio-ingressgateway -o yaml
   # Confirm that ports 80 and 443 are exposed and TLS is enabled.
   

4. Prometheus Metrics

   oc get --raw /apis/metrics.k8s.io/v1beta1/nodes
   # Look for a counter named istio_requests_total.
   

5. Health Probe

   oc get pod -n istio-system -l app=istiod -o jsonpath='{.items[*].status.containerStatuses[*].ready}'
   # All should return true.
   

5. Failure Modes & Mitigations

• Control Plane Crash: If the istiod pod dies, sidecars lose routing rules. Envoy falls back to direct IP routing, breaking mTLS. Mitigation: Attach a Probe to istiod and configure a retry policy in VirtualService to retry traffic until the control plane recovers.
• Sidecar Resource Limits: The istio-proxy container adds memory and CPU overhead. Over‑constrained pods may be OOM‑killed. Mitigation: Set istio-proxy resource requests/limits in the values.yaml of the Operator or via a Deployment patch.
• Certificate Rotation: Citadel rotates certificates every 90 days by default. If an application fails to refresh its TLS context, connections may fail. Mitigation: Enable sidecar.istio.io/reloadOnUpdate: "true" and monitor the istio-proxy logs for "certificate update" events.
• High Latency from Telemetry: Sampling telemetry at 100% can add latency to high‑throughput services. Mitigation: Reduce sampling rate via the istio-proxy configuration or use sidecar.istio.io/sampling: "0.1" for a 10% sample.

6. Conditions That Would Change the Design

• Legacy Service Not Supporting Sidecar: If a service cannot be redeployed with sidecar injection, expose it via a dedicated Gateway and VirtualService that routes traffic to the legacy pod without mTLS, effectively isolating it from the internal mesh.
• Need for External Traffic Policy: When external clients must bypass the ingress gateway (e.g., for legacy certificates), configure a ServiceEntry that declares the external service and a DestinationRule that disables mTLS for that subset.
• Performance‑Critical Path: If a service path requires zero‑latency, consider using ProxyConfig to disable Envoy’s HTTP/2 layer or route traffic via a direct ServiceEntry that skips the sidecar.

Concrete Example: Deploying the Mesh and a Sample Service

1. Install the Service Mesh Operator from the OperatorHub.

   oc apply -f https://github.com/openshift/openshift-service-mesh-operator/raw/v1.16/manifests/service-mesh-operator.yaml
   

2. Create a ServiceMeshControlPlane resource.

   oc apply -f - <

3. Annotate the target namespace.

   oc annotate namespace demo sidecar.istio.io/inject: "true"
   

4. Deploy the sample application (see Deployment YAML above).
5. Verify sidecar injection and metrics as described in the operational checks.

Practical Verification Checklist

Check	Command	Expected Result
CRDs installed	oc get crds | grep istio.io	List of Istio CRDs present
Sidecar present	oc get pod -n demo -o jsonpath='{.items[*].spec.containers[*].name}'	Contains istio-proxy
Ingress gateway reachable	oc get svc -n istio-system istio-ingressgateway -o yaml	Ports 80/443 exposed, TLS enabled
Metrics scraped	oc get --raw /apis/metrics.k8s.io/v1beta1/nodes	Presence of istio_requests_total
Control plane healthy	oc get pod -n istio-system -l app=istiod -o jsonpath='{.items[*].status.containerStatuses[*].ready}'	All true

Conclusion

By deploying Red Hat OpenShift Service Mesh with sidecar injection, teams can achieve secure, observable, and resilient microservices with minimal application changes. The architecture note outlines the essential requirements, the smallest viable design, trust boundaries, operational checks, failure modes, and when to revisit the design. Following the verification checklist ensures that the mesh is functioning as intended and that the system remains robust against common failure scenarios.