How to Renew SSL Certificate of Your OpenShift Cluster – 3.x

By | May 1, 2020

In this Perticular Article, we will see how we can renew the SSL Certificate in OpenShift Cluster v3.6. This implementation steps will also work in v3.10 and above except major version change like v4.0

Note: In v3.10 and above, there is a slight change. Worker nodes triggers a CSR Which you need to approved manually or automatically in order to prevent your cluster health. Please follow the Steps here in this post.

Openshift Enforces us to use the Encrypted data transmission within the cluster as well, so it uses SSL certificate for all Its Components like etcd, router, Web console, Kubeconfig etc. Below are the Steps to Renew the Cluster certificate

prerequisite

  • You need to have Up and Running Openshift Cluster
  • A Valid SSL certificate. Take a Look here for Self Signed Certificate
  • Make sure to generate 2 Certificate, one for OpenShift WebConsole and Other one is for Running Applications.
  • Copy the certificate to a local directory
  • Ensure chattr on /etc/resolv.conf to be removed before starting the playbook
  • Make sure the user you will be using for running the playbook have sudo rights.

Make Sure You Copy the required certificate to the Server and update the OpenShift Configuration (OSEv3) with proper path of the Certificate before running the redeploy_certificates playbook.

##### Router/ Applications ###
 openshift_master_default_subdomain: <your_app_domain>
 openshift_hosted_router_certificate: >
   {
     'certfile': '/home/openshift/Deployment/ssl cert/wildcard/<app_domain>.crt',
     'keyfile': '/home/openshift/Deployment/ssl-cert/wildcard/<app_domain>.key',
     'cafile': '/home/openshift/Deployment/ssl-cert/<RootCA>.crt'
   }
## master WebConsole
 openshift_master_cluster_public_hostname: <your_domain>
 openshift_master_console_port: 443
 openshift_master_named_certificates: >
   [
     {
       'certfile': '/home/openshift/Deployment/ssl-cert/<your_domain>.crt',
       'keyfile': '/home/openshift/Deployment/ssl-cert/<your_domain>.key',
       'cafile':  '/home/openshift/Deployment/ssl-cert/<RootCA>.crt',
       'names': ['your_domain.com']
     }
   ]

Now Verify the Expiry of your SSL Certificate by using below command.

$ for cert in $(for config in $(find /etc/origin/ -name "*yaml"); do file=$(basename $config); awk '/.crt/ { print FILENAME $2 }' $config | sed "s/$file//"; done); do echo $cert; openssl x509 -in $cert -text -noout | grep Validity -A2; done

$ for config in $(find /etc/origin/ -name "*kubeconfig"); do echo "Config: $config";  file=$(basename $config); echo "  File: $file"; awk '/cert/ {print $2}' $config | sed "s/$file//" | base64 -d | openssl x509 -text -noout | grep Validity -A2 ; done 

Now if you wanted to check the expiry of the SSL Certificates Individually then you can use the below command.

$ openssl x509 -text -noout -in devopsage.crt

Now, for being in the safer side it’s better to take the backup of the origin and etcd folder. which you can take using the ansible adhoc command.

$ ansible all -m archive -a 'path=/etc/origin dest=/tmp/origin.certbkp.tgz'
$ ansible masters -m archive -a 'path=/etc/etcd dest=/tmp/etcd.tgz'

It’s time to run the redeploy_certificate playbook to redeploy the new certificate on the entire cluster components.
Note

  • If playbook fails then use verbose mode to capture the detailed output to identify the error and fix it. To use verbose mode, use -vvvv with command.
  • It’s recommended to use tmux or screen so Incase if anything happens with laptop/Pc and need to reboot, the playbook will keep on running in the background.
  •  Make sure to capture the output of playbook for further troubleshooting.
$ ansible-playbook -i inventory_file /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-certificates.yml -vvvv | tee date +%Y%m%d-%H%M%S-redeploy-cert.log

Redeploy Registry Certificate

Check ca.crt and ca.key before registry redeploy being done –  taken backup under /root directory. Verify  before and after expiry date got changed.

Note: Run oc adm commands only from the first master listed in the Ansible host inventory file

$ oc project default
$ export REGISTRY_IP=oc get service docker-registry -o jsonpath='{.spec.clusterIP}'
$ export REGISTRY_HOSTNAME=oc get route/docker-registry -o jsonpath='{.spec.host}'

After exporting the registry IP and Registry hostname, Run the below command to create new registry certificates

Note: Run oc adm commands only from the first master listed in the Ansible host inventory file

$ oc adm ca create-server-cert \
      --signer-cert=/etc/origin/master/ca.crt \
      --signer-key=/etc/origin/master/ca.key \
      --hostnames=$REGISTRY_IP,docker-registry.default.svc,docker-registry.default.svc.cluster.local,$REGISTRY_HOSTNAME \
      --cert=/etc/origin/master/registry.crt \
      --key=/etc/origin/master/registry.key \
      --signer-serial=/etc/origin/master/ca.serial.txt

Now, Replace the new registry certificate and key.

$ oc secret new registry-certificates \
      /etc/origin/master/registry.crt \
      /etc/origin/master/registry.key \
      -o json | oc replace -f -

$ oc secret new registry-certificates-exposed \
      /etc/origin/master/registry.crt \
      /etc/origin/master/registry.key \
      -o json | oc replace -f –

Redeploy registry

$ oc deploy dc/docker-registry --latest

Create a new file that concatenates the generated certificates

Note: Before you generate a new secret, back up the current one

$ oc export secret router-certs > ~/old-router-certs-secret.yaml
$ cat /home/openshift/Deployment/ssl-cert/wildcard/app_domain.crt /etc/origin/master/ca.crt /home/openshift/Deployment/ssl-cert/wildcard/app_domain.key > router_cert.pem

Create a new secret to hold the new certificate and key, and replace the contents of the existing secret

$ oc create secret tls router-certs --cert=router.pem --key=/home/rhelocp/ssl-cert/wildcard/apps_domain.key -o json --dry-run | oc replace -f -

Finally rollout router pods

$ oc rollout latest dc/router

Now to verify whether the certificate got redeployed to the entire cluster components successfully or not use one of the playbook called easy-mode.yaml. This playbook will give you a html file using which you can visualize the certificate update on entire cluster using UI.

$ ansible-playbook -i inventory_file /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/easy-mode.yaml -v

Note

If any of the Component is not updated then redhat provides playbook for deploying certificate to Openshift Components Individually. Just for an example,

To redeploy etcd certificates, run below playbook, specifying your inventory file

$ ansible-playbook -i inventory_file /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-etcd-certificates.yml -vvvv | tee date +%Y%m%d-%H%M%S-redeploy_etcd.log

For more details you can see the redhat documentations. 

References:

https://docs.openshift.com/container-platform/3.6/install_config/redeploying_certificates.html

https://docs.openshift.com/container-platform/3.11/install_config/redeploying_certificates.html


You May also need to look at,

https://www.devopsage.com/how-to-enable-auto-approval-of-csr-in-openshift-v3-10/(opens in a new tab)

Leave a Reply

Your email address will not be published. Required fields are marked *