kubernetes_sd_config scrape targets ignoring NetworkPolicy isolation

Short answer

No — kubernetes_sd_config has no supported way to read NetworkPolicy objects, and there is no relabel action that consults them. Service discovery only watches the API server for pods, services, endpoints, and nodes. Filtering targets to match your network isolation is intentionally your job, via relabel_configs keyed on labels or annotations. This answer assumes a recent Prometheus 2.x release; verify against your version's configuration reference.

Why the targets still appear

Discovery answers "what exists in the API?", not "what can I reach?". NetworkPolicy is enforced by your CNI plugin at the network layer, and the API server happily returns endpoints for pods Prometheus cannot connect to. So isolated pods stay in the target list and show as down on connection timeout — expected behavior, not a bug.

A maintainable relabel convention

Since there is no automatic mapping from NetworkPolicy intent to labels, pick one explicit, team-owned convention and enforce it in CI or admission control:

1. Namespace label as the contract. Require every namespace to carry a label such as monitoring.example.com/scrape=allowed|denied. Prometheus joins pod metadata to namespace metadata automatically (__meta_kubernetes_namespace_label_*), so pods inherit the decision.
2. Filter in relabel_configs. Drop anything not explicitly allowed:

relabel_configs:
  - source_labels: [__meta_kubernetes_namespace_label_monitoring_example_com_scrape]
    regex: allowed
    action: keep

1. Keep the label and the policy in sync. The real risk is drift: a namespace labeled allowed whose NetworkPolicy actually blocks scraping. Mitigate by generating both the NetworkPolicy and the namespace label from the same source (Helm values, a GitOps template, or an admission webhook that rejects namespaces missing the label). Treat the label as documentation of intent, not proof of reachability — the up metric remains the ground truth.

Discovery refresh after a policy change

Yes: a target that becomes unreachable stays in the list (failing) until the next discovery refresh, because the pod itself hasn't changed — only the network path has. The watch-based SD usually picks up metadata changes quickly, but nothing in a NetworkPolicy change triggers a target update, because SD never saw the policy in the first place. If you filter by labels, the target only disappears when the label changes.

You can tune how often Prometheus re-lists resources, but be conservative: aggressive refresh intervals increase load on the Kubernetes API server and on Prometheus itself. Only lower it (for example toward 30s) after measuring API server latency and Prometheus CPU, and prefer fixing the label convention over speeding up polling.

Verification

1. Check the Prometheus UI's Service Discovery page: confirm isolated pods appear under discovered targets but are dropped (or kept) by your relabel rule as intended.
2. After applying a blocking NetworkPolicy, run up{job="your-job"} and confirm the target flips to 0 — that confirms the failure is reachability, not discovery.
3. Change the namespace label and confirm the target disappears from Targets within one discovery cycle.