<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->

<!-- BEGIN STRIP_FOR_RELEASE -->

<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
     width="25" height="25">

<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>

If you are using a released version of Kubernetes, you should
refer to the docs that go with that version.

<!-- TAG RELEASE_LINK, added by the munger automatically -->
<strong>
The latest release of this document can be found
[here](http://releases.k8s.io/release-1.1/docs/design/secrets.md).

Documentation for other releases can be found at
[releases.k8s.io](http://releases.k8s.io).
</strong>
--

<!-- END STRIP_FOR_RELEASE -->

<!-- END MUNGE: UNVERSIONED_WARNING -->

## Abstract

A proposal for the distribution of [secrets](../user-guide/secrets.md) (passwords, keys, etc) to the Kubelet and to
containers inside Kubernetes using a custom [volume](../user-guide/volumes.md#secrets) type. See the [secrets example](../user-guide/secrets/) for more information.

## Motivation

Secrets are needed in containers to access internal resources like the Kubernetes master or
external resources such as git repositories, databases, etc.  Users may also want behaviors in the
kubelet that depend on secret data (credentials for image pull from a docker registry) associated
with pods.

Goals of this design:

1.  Describe a secret resource
2.  Define the various challenges attendant to managing secrets on the node
3.  Define a mechanism for consuming secrets in containers without modification

## Constraints and Assumptions

*  This design does not prescribe a method for storing secrets; storage of secrets should be
   pluggable to accommodate different use-cases
*  Encryption of secret data and node security are orthogonal concerns
*  It is assumed that node and master are secure and that compromising their security could also
   compromise secrets:
   *  If a node is compromised, the only secrets that could potentially be exposed should be the
      secrets belonging to containers scheduled onto it
   *  If the master is compromised, all secrets in the cluster may be exposed
*  Secret rotation is an orthogonal concern, but it should be facilitated by this proposal
*  A user who can consume a secret in a container can know the value of the secret; secrets must
   be provisioned judiciously

## Use Cases

1.  As a user, I want to store secret artifacts for my applications and consume them securely in
    containers, so that I can keep the configuration for my applications separate from the images
    that use them:
    1.  As a cluster operator, I want to allow a pod to access the Kubernetes master using a custom
        `.kubeconfig` file, so that I can securely reach the master
    2.  As a cluster operator, I want to allow a pod to access a Docker registry using credentials
        from a `.dockercfg` file, so that containers can push images
    3.  As a cluster operator, I want to allow a pod to access a git repository using SSH keys,
        so that I can push to and fetch from the repository
2.  As a user, I want to allow containers to consume supplemental information about services such
    as username and password which should be kept secret, so that I can share secrets about a
    service amongst the containers in my application securely
3.  As a user, I want to associate a pod with a `ServiceAccount` that consumes a secret and have
    the kubelet implement some reserved behaviors based on the types of secrets the service account
    consumes:
    1.  Use credentials for a docker registry to pull the pod's docker image
    2.  Present Kubernetes auth token to the pod or transparently decorate traffic between the pod
        and master service
4.  As a user, I want to be able to indicate that a secret expires and for that secret's value to
    be rotated once it expires, so that the system can help me follow good practices

### Use-Case: Configuration artifacts

Many configuration files contain secrets intermixed with other configuration information.  For
example, a user's application may contain a properties file than contains database credentials,
SaaS API tokens, etc.  Users should be able to consume configuration artifacts in their containers
and be able to control the path on the container's filesystems where the artifact will be
presented.

### Use-Case: Metadata about services

Most pieces of information about how to use a service are secrets.  For example, a service that
provides a MySQL database needs to provide the username, password, and database name to consumers
so that they can authenticate and use the correct database. Containers in pods consuming the MySQL
service would also consume the secrets associated with the MySQL service.

### Use-Case: Secrets associated with service accounts

[Service Accounts](service_accounts.md) are proposed as a
mechanism to decouple capabilities and security contexts from individual human users.  A
`ServiceAccount` contains references to some number of secrets.  A `Pod` can specify that it is
associated with a `ServiceAccount`.  Secrets should have a `Type` field to allow the Kubelet and
other system components to take action based on the secret's type.

#### Example: service account consumes auth token secret

As an example, the service account proposal discusses service accounts consuming secrets which
contain Kubernetes auth tokens.  When a Kubelet starts a pod associated with a service account
which consumes this type of secret, the Kubelet may take a number of actions:

1.  Expose the secret in a `.kubernetes_auth` file in a well-known location in the container's
    file system
2.  Configure that node's `kube-proxy` to decorate HTTP requests from that pod to the
    `kubernetes-master` service with the auth token, e. g. by adding a header to the request
    (see the [LOAS Daemon](http://issue.k8s.io/2209) proposal)

#### Example: service account consumes docker registry credentials

Another example use case is where a pod is associated with a secret containing docker registry
credentials.  The Kubelet could use these credentials for the docker pull to retrieve the image.

### Use-Case: Secret expiry and rotation

Rotation is considered a good practice for many types of secret data.  It should be possible to
express that a secret has an expiry date; this would make it possible to implement a system
component that could regenerate expired secrets.  As an example, consider a component that rotates
expired secrets.  The rotator could periodically regenerate the values for expired secrets of
common types and update their expiry dates.

## Deferral: Consuming secrets as environment variables

Some images will expect to receive configuration items as environment variables instead of files.
We should consider what the best way to allow this is; there are a few different options:

1.  Force the user to adapt files into environment variables.  Users can store secrets that need to
    be presented as environment variables in a format that is easy to consume from a shell:

        $ cat /etc/secrets/my-secret.txt
        export MY_SECRET_ENV=MY_SECRET_VALUE

    The user could `source` the file at `/etc/secrets/my-secret` prior to executing the command for
    the image either inline in the command or in an init script,

2.  Give secrets an attribute that allows users to express the intent that the platform should
    generate the above syntax in the file used to present a secret.  The user could consume these
    files in the same manner as the above option.

3.  Give secrets attributes that allow the user to express that the secret should be presented to
    the container as an environment variable.  The container's environment would contain the
    desired values and the software in the container could use them without accommodation the
    command or setup script.

For our initial work, we will treat all secrets as files to narrow the problem space.  There will
be a future proposal that handles exposing secrets as environment variables.

## Flow analysis of secret data with respect to the API server

There are two fundamentally different use-cases for access to secrets:

1.  CRUD operations on secrets by their owners
2.  Read-only access to the secrets needed for a particular node by the kubelet

### Use-Case: CRUD operations by owners

In use cases for CRUD operations, the user experience for secrets should be no different than for
other API resources.

#### Data store backing the REST API

The data store backing the REST API should be pluggable because different cluster operators will
have different preferences for the central store of secret data.  Some possibilities for storage:

1.  An etcd collection alongside the storage for other API resources
2.  A collocated [HSM](http://en.wikipedia.org/wiki/Hardware_security_module)
3.  A secrets server like [Vault](https://www.vaultproject.io/) or [Keywhiz](https://square.github.io/keywhiz/)
4.  An external datastore such as an external etcd, RDBMS, etc.

#### Size limit for secrets

There should be a size limit for secrets in order to:

1.  Prevent DOS attacks against the API server
2.  Allow kubelet implementations that prevent secret data from touching the node's filesystem

The size limit should satisfy the following conditions:

1.  Large enough to store common artifact types (encryption keypairs, certificates, small
    configuration files)
2.  Small enough to avoid large impact on node resource consumption (storage, RAM for tmpfs, etc)

To begin discussion, we propose an initial value for this size limit of **1MB**.

#### Other limitations on secrets

Defining a policy for limitations on how a secret may be referenced by another API resource and how
constraints should be applied throughout the cluster is tricky due to the number of variables
involved:

1.  Should there be a maximum number of secrets a pod can reference via a volume?
2.  Should there be a maximum number of secrets a service account can reference?
3.  Should there be a total maximum number of secrets a pod can reference via its own spec and its
    associated service account?
4.  Should there be a total size limit on the amount of secret data consumed by a pod?
5.  How will cluster operators want to be able to configure these limits?
6.  How will these limits impact API server validations?
7.  How will these limits affect scheduling?

For now, we will not implement validations around these limits.  Cluster operators will decide how
much node storage is allocated to secrets. It will be the operator's responsibility to ensure that
the allocated storage is sufficient for the workload scheduled onto a node.

For now, kubelets will only attach secrets to api-sourced pods, and not file- or http-sourced
ones.  Doing so would:
  - confuse the secrets admission controller in the case of mirror pods.
  - create an apiserver-liveness dependency -- avoiding this dependency is a main reason to use non-api-source pods.

### Use-Case: Kubelet read of secrets for node

The use-case where the kubelet reads secrets has several additional requirements:

1.  Kubelets should only be able to receive secret data which is required by pods scheduled onto
    the kubelet's node
2.  Kubelets should have read-only access to secret data
3.  Secret data should not be transmitted over the wire insecurely
4.  Kubelets must ensure pods do not have access to each other's secrets

#### Read of secret data by the Kubelet

The Kubelet should only be allowed to read secrets which are consumed by pods scheduled onto that
Kubelet's node and their associated service accounts.  Authorization of the Kubelet to read this
data would be delegated to an authorization plugin and associated policy rule.

#### Secret data on the node: data at rest

Consideration must be given to whether secret data should be allowed to be at rest on the node:

1.  If secret data is not allowed to be at rest, the size of secret data becomes another draw on
    the node's RAM - should it affect scheduling?
2.  If secret data is allowed to be at rest, should it be encrypted?
    1.  If so, how should be this be done?
    2.  If not, what threats exist?  What types of secret are appropriate to store this way?

For the sake of limiting complexity, we propose that initially secret data should not be allowed
to be at rest on a node; secret data should be stored on a node-level tmpfs filesystem.  This
filesystem can be subdivided into directories for use by the kubelet and by the volume plugin.

#### Secret data on the node: resource consumption

The Kubelet will be responsible for creating the per-node tmpfs file system for secret storage.
It is hard to make a prescriptive declaration about how much storage is appropriate to reserve for
secrets because different installations will vary widely in available resources, desired pod to
node density, overcommit policy, and other operation dimensions.  That being the case, we propose
for simplicity that the amount of secret storage be controlled by a new parameter to the kubelet
with a default value of **64MB**.  It is the cluster operator's responsibility to handle choosing
the right storage size for their installation and configuring their Kubelets correctly.

Configuring each Kubelet is not the ideal story for operator experience; it is more intuitive that
the cluster-wide storage size be readable from a central configuration store like the one proposed
in [#1553](http://issue.k8s.io/1553).  When such a store
exists, the Kubelet could be modified to read this configuration item from the store.

When the Kubelet is modified to advertise node resources (as proposed in
[#4441](http://issue.k8s.io/4441)), the capacity calculation
for available memory should factor in the potential size of the node-level tmpfs in order to avoid
memory overcommit on the node.

#### Secret data on the node: isolation

Every pod will have a [security context](security_context.md).
Secret data on the node should be isolated according to the security context of the container.  The
Kubelet volume plugin API will be changed so that a volume plugin receives the security context of
a volume along with the volume spec.  This will allow volume plugins to implement setting the
security context of volumes they manage.

## Community work

Several proposals / upstream patches are notable as background for this proposal:

1.  [Docker vault proposal](https://github.com/docker/docker/issues/10310)
2.  [Specification for image/container standardization based on volumes](https://github.com/docker/docker/issues/9277)
3.  [Kubernetes service account proposal](service_accounts.md)
4.  [Secrets proposal for docker (1)](https://github.com/docker/docker/pull/6075)
5.  [Secrets proposal for docker (2)](https://github.com/docker/docker/pull/6697)

## Proposed Design

We propose a new `Secret` resource which is mounted into containers with a new volume type. Secret
volumes will be handled by a volume plugin that does the actual work of fetching the secret and
storing it. Secrets contain multiple pieces of data that are presented as different files within
the secret volume (example: SSH key pair).

In order to remove the burden from the end user in specifying every file that a secret consists of,
it should be possible to mount all files provided by a secret with a single `VolumeMount` entry
in the container specification.

### Secret API Resource

A new resource for secrets will be added to the API:

```go
type Secret struct {
    TypeMeta
    ObjectMeta

    // Data contains the secret data.  Each key must be a valid DNS_SUBDOMAIN.
    // The serialized form of the secret data is a base64 encoded string,
    // representing the arbitrary (possibly non-string) data value here.
    Data map[string][]byte `json:"data,omitempty"`

    // Used to facilitate programmatic handling of secret data.
    Type SecretType `json:"type,omitempty"`
}

type SecretType string

const (
    SecretTypeOpaque              SecretType = "Opaque"                                 // Opaque (arbitrary data; default)
    SecretTypeServiceAccountToken SecretType = "kubernetes.io/service-account-token"    // Kubernetes auth token
    SecretTypeDockercfg           SecretType = "kubernetes.io/dockercfg"                // Docker registry auth
    // FUTURE: other type values
)

const MaxSecretSize = 1 * 1024 * 1024
```

A Secret can declare a type in order to provide type information to system components that work
with secrets.  The default type is `opaque`, which represents arbitrary user-owned data.

Secrets are validated against `MaxSecretSize`.  The keys in the `Data` field must be valid DNS
subdomains.

A new REST API and registry interface will be added to accompany the `Secret` resource.  The
default implementation of the registry will store `Secret` information in etcd.  Future registry
implementations could store the `TypeMeta` and `ObjectMeta` fields in etcd and store the secret
data in another data store entirely, or store the whole object in another data store.

#### Other validations related to secrets

Initially there will be no validations for the number of secrets a pod references, or the number of
secrets that can be associated with a service account.  These may be added in the future as the
finer points of secrets and resource allocation are fleshed out.

### Secret Volume Source

A new `SecretSource` type of volume source will be added to the `VolumeSource` struct in the
API:

```go
type VolumeSource struct {
    // Other fields omitted

    // SecretSource represents a secret that should be presented in a volume
    SecretSource *SecretSource `json:"secret"`
}

type SecretSource struct {
    Target ObjectReference
}
```

Secret volume sources are validated to ensure that the specified object reference actually points
to an object of type `Secret`.

In the future, the `SecretSource` will be extended to allow:

1.  Fine-grained control over which pieces of secret data are exposed in the volume
2.  The paths and filenames for how secret data are exposed

### Secret Volume Plugin

A new Kubelet volume plugin will be added to handle volumes with a secret source.  This plugin will
require access to the API server to retrieve secret data and therefore the volume `Host` interface
will have to change to expose a client interface:

```go
type Host interface {
    // Other methods omitted

    // GetKubeClient returns a client interface
    GetKubeClient() client.Interface
}
```

The secret volume plugin will be responsible for:

1.  Returning a `volume.Builder` implementation from `NewBuilder` that:
    1.  Retrieves the secret data for the volume from the API server
    2.  Places the secret data onto the container's filesystem
    3.  Sets the correct security attributes for the volume based on the pod's `SecurityContext`
2.  Returning a `volume.Cleaner` implementation from `NewClear` that cleans the volume from the
    container's filesystem

### Kubelet: Node-level secret storage

The Kubelet must be modified to accept a new parameter for the secret storage size and to create
a tmpfs file system of that size to store secret data.  Rough accounting of specific changes:

1.  The Kubelet should have a new field added called `secretStorageSize`; units are megabytes
2.  `NewMainKubelet` should accept a value for secret storage size
3.  The Kubelet server should have a new flag added for secret storage size
4.  The Kubelet's `setupDataDirs` method should be changed to create the secret storage

### Kubelet: New behaviors for secrets associated with service accounts

For use-cases where the Kubelet's behavior is affected by the secrets associated with a pod's
`ServiceAccount`, the Kubelet will need to be changed.  For example, if secrets of type
`docker-reg-auth` affect how the pod's images are pulled, the Kubelet will need to be changed
to accommodate this.  Subsequent proposals can address this on a type-by-type basis.

## Examples

For clarity, let's examine some detailed examples of some common use-cases in terms of the
suggested changes.  All of these examples are assumed to be created in a namespace called
`example`.

### Use-Case: Pod with ssh keys

To create a pod that uses an ssh key stored as a secret, we first need to create a secret:

```json
{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "ssh-key-secret"
  },
  "data": {
    "id-rsa": "dmFsdWUtMg0KDQo=",
    "id-rsa.pub": "dmFsdWUtMQ0K"
  }
}
```

**Note:** The serialized JSON and YAML values of secret data are encoded as
base64 strings.  Newlines are not valid within these strings and must be
omitted.

Now we can create a pod which references the secret with the ssh key and consumes it in a volume:

```json
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "secret-test-pod",
    "labels": {
      "name": "secret-test"
    }
  },
  "spec": {
    "volumes": [
      {
        "name": "secret-volume",
        "secret": {
          "secretName": "ssh-key-secret"
        }
      }
    ],
    "containers": [
      {
        "name": "ssh-test-container",
        "image": "mySshImage",
        "volumeMounts": [
          {
            "name": "secret-volume",
            "readOnly": true,
            "mountPath": "/etc/secret-volume"
          }
        ]
      }
    ]
  }
}
```

When the container's command runs, the pieces of the key will be available in:

    /etc/secret-volume/id-rsa.pub
    /etc/secret-volume/id-rsa

The container is then free to use the secret data to establish an ssh connection.

### Use-Case: Pods with pod / test credentials

This example illustrates a pod which consumes a secret containing prod
credentials and another pod which consumes a secret with test environment
credentials.

The secrets:

```json
{
  "apiVersion": "v1",
  "kind": "List",
  "items":
  [{
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
      "name": "prod-db-secret"
    },
    "data": {
      "password": "dmFsdWUtMg0KDQo=",
      "username": "dmFsdWUtMQ0K"
    }
  },
  {
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
      "name": "test-db-secret"
    },
    "data": {
      "password": "dmFsdWUtMg0KDQo=",
      "username": "dmFsdWUtMQ0K"
    }
  }]
}
```

The pods:

```json
{
  "apiVersion": "v1",
  "kind": "List",
  "items":
  [{
    "kind": "Pod",
    "apiVersion": "v1",
    "metadata": {
      "name": "prod-db-client-pod",
      "labels": {
        "name": "prod-db-client"
      }
    },
    "spec": {
      "volumes": [
        {
          "name": "secret-volume",
          "secret": {
            "secretName": "prod-db-secret"
          }
        }
      ],
      "containers": [
        {
          "name": "db-client-container",
          "image": "myClientImage",
          "volumeMounts": [
            {
              "name": "secret-volume",
              "readOnly": true,
              "mountPath": "/etc/secret-volume"
            }
          ]
        }
      ]
    }
  },
  {
    "kind": "Pod",
    "apiVersion": "v1",
    "metadata": {
      "name": "test-db-client-pod",
      "labels": {
        "name": "test-db-client"
      }
    },
    "spec": {
      "volumes": [
        {
          "name": "secret-volume",
          "secret": {
            "secretName": "test-db-secret"
          }
        }
      ],
      "containers": [
        {
          "name": "db-client-container",
          "image": "myClientImage",
          "volumeMounts": [
            {
              "name": "secret-volume",
              "readOnly": true,
              "mountPath": "/etc/secret-volume"
            }
          ]
        }
      ]
    }
  }]
}
```

The specs for the two pods differ only in the value of the object referred to by the secret volume
source.  Both containers will have the following files present on their filesystems:

    /etc/secret-volume/username
    /etc/secret-volume/password


<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/secrets.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
