Certificates and Trust Management
The components of Charmed Distribution of Kubernetes® (CDK) need to be able to communicate securely over the network. This is accomplished using TLS and public-key encryption with a chain of trust up to a shared root Certificate Authority (CA). However, when the cluster is being brought up or a new unit is being added, the chain of trust and certificates required must be bootstrapped into the machines somehow.
Juju relations
All communication between Juju units and the Juju controller happens over TLS-encrypted websockets. The certificate for the TLS connection to the controller is added as explicitly trusted to each machine as part of the bootstrap process using a combination of cloud-init and SSH.
With this secure channel, Juju charms can communicate with each other using relation data. The data published by one unit is sent to the controller, which then makes it available for all other units on the same relation. The data for each relation is scoped by ID and is only visible to units participating in the specific relation on which it is set. However, it is worth noting that relation data is stored on the controller machine in MongoDB,so for truly sensitive information, proper secret storage engines, such as Vault, and encryption-at-rest should be used.
Managing certificates
Unfortunately, the Juju controller does not provide any mechanisms for generating or distributing additional certificates to be used by the applications on the machines, so they must be managed by the charms via the secure relation data channel. This is done using the tls-certificates interface protocol and a relation to an application providing a Certificate Authority. (This CA could be either a root CA, or an intermediary CA authorised by some other root CA to issue certificates.)
When the relation is established, the root CA’s certificate is sent via the relation and installed as trusted. Then, certificate requests can be issued over the relation and new certificates created by the CA and returned over the relation.
Because all units with a relation to the CA have a chain of trust up to it (or its root CA), they will automatically trust a certificate generated by the CA without requiring anything to be communicated from the unit which holds the certificate. On the other hand, for the certificates to be trusted externally (such as by clients) or by applications without a relation to the CA, the CA will have to be configured as an intermediary CA with a chain of trust up to a globally trusted CA (such as Comodo or DigiCert).
The certificate information will also be included in the Kubernetes config
file that CDK generates to be used by kubelet, so that communications
between the local machine and Kubernetes will be secured.
Server certificates
Each service that will be connected to will need a server certificate identifying and validating it to any clients that wish to connect.
The primary address at which the service can be reached will be its common name; ideally, this will be a fully-qualified domain name (FQDN) which will not change, but for internal service communication, it is often just the ingress address for the unit. Any additional names or addresses by which the service can be reached will be its subject alternative names (SANs).
CDK will manage the server certificates automatically, including
generating the certificate with the proper CN and SANs. However, the
kubernetes-master charm also supports an extra_sans option which
can be used to provide additional names to be added to the SANs list.
Client certificates
In order to provide for two-way security, some services require that clients identify themselves via a client certificate. These are more or less the same as server certificates, but are presented by a client to the service they are connecting to so that the service can validate that client’s identity. Client certificates can only be used to identify a client and will be rejected by clients if presented by a service they are connecting to.
Certificate Authorities for CDK
CDK can use a CA provided by any charm which provides a tls-certificates endpoint. The two current recommendations are EasyRSA and Vault.
EasyRSA
By default, the CDK bundle includes the EasyRSA charm. This is a relatively simple charm which uses OpenVPN’s easy-rsa to provide a CA and sign certificates. This is lightweight and works out of the box without any additional configuration, but it cannot act as an intermediary CA and does not support HA.
Vault
For production systems, it is recommended to replace EasyRSA with the Vault charm. This uses HashiCorp’s Vault to provide either a root or intermediate CA. It can also be deployed HA, as well as provide a secure secrets store which can be used to enable encryption-at-rest for CDK. However, it requires a database to store its data, and (depending on configuration) some manual steps will be required after deployment as well as after any reboot to unseal Vault so that the secrets, such as certificates and signing keys, can be accessed.
See the operations documentation for details on how to deploy Vault as a CA.