Files
kubernetes/pkg/kubelet
Kubernetes Submit Queue feb4d2004a Merge pull request #31064 from soundcloud/grobie/filter-internal-labels
Automatic merge from submit-queue

Filter internal Kubernetes labels from Prometheus metrics

**What this PR does / why we need it**:

Kubernetes uses Docker labels as storage for some internal labels. The
majority of these labels are not meaningful metric labels and a few of
them are even harmful as they're not static and cause wrong aggregation
results.

This change provides a custom labels func to only attach meaningful
labels to cAdvisor exported metrics.

**Which issue this PR fixes**

google/cadvisor#1312

**Special notes for your reviewer**:

Depends on google/cadvisor#1429. Once that is merged, I'll update the vendor update commit.

**Release note**:

```release-note
Remove environment variables and internal Kubernetes Docker labels from cAdvisor Prometheus metric labels.

Old behavior:

- environment variables explicitly whitelisted via --docker-env-metadata-whitelist were exported as `container_env_*=*`. Default is zero so by default non were exported
- all docker labels were exported as `container_label_*=*`

New behavior:

- Only `container_name`, `pod_name`, `namespace`, `id`, `image`, and `name` labels are exposed
- no environment variables will be exposed ever via /metrics, even if whitelisted
```

---

Given that we have full control over the exported label set, I shortened the pod_name, pod_namespace and container_name label names. Below an example of the change (reformatted for readability).

```
# BEFORE
container_cpu_cfs_periods_total{
  container_label_io_kubernetes_container_hash="5af8c3b4",
  container_label_io_kubernetes_container_name="sync",
  container_label_io_kubernetes_container_restartCount="1",
  container_label_io_kubernetes_container_terminationMessagePath="/dev/termination-log",
  container_label_io_kubernetes_pod_name="popularsearches-web-3165456836-2bfey",
  container_label_io_kubernetes_pod_namespace="popularsearches",
  container_label_io_kubernetes_pod_terminationGracePeriod="30",
  container_label_io_kubernetes_pod_uid="6a291e48-47c4-11e6-84a4-c81f66bdf8bd",
  id="/docker/68e1f15353921f4d6d4d998fa7293306c4ac828d04d1284e410ddaa75cf8cf25",
  image="redacted.com/popularsearches:42-16-ba6bd88",
  name="k8s_sync.5af8c3b4_popularsearches-web-3165456836-2bfey_popularsearches_6a291e48-47c4-11e6-84a4-c81f66bdf8bd_c02d3775"
} 72819

# AFTER
container_cpu_cfs_periods_total{
  container_name="sync",
  pod_name="popularsearches-web-3165456836-2bfey",
  namespace="popularsearches",
  id="/docker/68e1f15353921f4d6d4d998fa7293306c4ac828d04d1284e410ddaa75cf8cf25",
  image="redacted.com/popularsearches:42-16-ba6bd88",
  name="k8s_sync.5af8c3b4_popularsearches-web-3165456836-2bfey_popularsearches_6a291e48-47c4-11e6-84a4-c81f66bdf8bd_c02d3775"
} 72819
```

Feedback requested on:
* Label names. Other suggestions? Should we keep these very long ones?
* Do we need to export io.kubernetes.pod.uid? It makes working with the metrics a bit more complicated and the pod name is already unique at any time (but not over time). The UID is aslo part of `name`.

As discussed with @timstclair, this should be added to v1.4 as the current labels are harmful.

PTAL @jimmidyson @fabxc @vishh
2016-08-25 21:11:10 -07:00
..
2016-07-16 13:48:21 -04:00
2016-08-03 01:14:44 +03:00
2016-08-03 01:14:44 +03:00
2016-08-17 16:21:19 -04:00
2016-07-16 13:48:21 -04:00
2016-08-25 13:22:01 +02:00
2016-07-16 13:48:21 -04:00
2016-08-22 23:35:11 -04:00
2016-07-16 13:48:21 -04:00
2016-08-25 10:57:31 -07:00
2016-03-02 20:46:32 -05:00
2016-08-03 01:14:44 +03:00
2016-08-03 13:19:22 +08:00
2016-08-22 23:35:11 -04:00