WORKSHOP.md

Agenda

• Overview
• What is GitOps?
• Why GitOps?
• Sample Application Walkthrough
• Sample App CI Walkthrough
• Sample App GitOps in Action
• Prepare Your Environment
• Initial Deployment
• Making Changes
• What Just Happened?
• Running the Tekton Pipeline
• Sample Infrastructure Demo
• Considerations for production implementations
• Further reading

Overview

In this workshop we will cover GitOps concepts in the context of a sample application with a CI/CD pipeline. We will use Tekton for our CI (Continuous Integration) solution and ArgoCD for CD (Continuous Delivery) solution. There are other tools we could use for CI/CD under a GitOps model on Kubernetes/OpenShift, however Tekton was purpose-built to be a cloud and Kubernetes native CI solution and ArgoCD provides a class 5 ("Autopilot") operator for Kubernetes. The important parts are the functionality, level of automation, and familiarity of Kubernetes objects so as not to distract participants from the purpose of the workshop: Understanding GitOps and its value as a methodology.

This workshop makes use of Ansible to install components and set up the workshop environment. Participants are not required to use the provided partcipant-*.yaml playbooks, they are merely provided to save time. If you find that following along is difficult or if you are pressed for time, run the associated playbook (will be noted in each relevant section) to catch up.

You can see a visual overview of the workshop contents below:

What is GitOps?

GitOps is a continuous delivery methodology for applying configuration based on assets stored in a git repository. Typical implementations of GitOps include manually applying configuration assets or making use of tools which can automatically apply those assets by responding to events.

Why GitOps?

GitOps provides a way for Developers and Operations Engineers to declaratively describe application and infrastructure components and their desired state, provide traceability of changes, and perform operations via pull request, among other things. Developers are already used to managing their application source code via git, as are Operations Engineers who make use of Infrastructure as Code practices. GitOps makes use of all of that and fills the continuous delivery gap.

Sample Application Walkthrough

Our sample-app for today is a simple Go program which periodically prints a greeting to STDOUT. It will continue to run until stopped and can be executed via a container image defined in its Dockerfile. It includes a Makefile to simplify building the binary and the container image.

The desired deployment target of this application is a Kubernetes/OpenShift cluster. The desired configuration for our application is stored in sample-app-config. Within this directory we have defined the objects necessary for the app to run on a cluster: a Deployment in sample-app-deployment.yaml, a Namespace in sample-app-namespace.yaml, and a Networkpolicy in sample-app-networkpolicy.yaml. This sample-app-config would generally be deployed to a separate Git repository, but for our purposes we are maintaining it in a separate directory within this repository.

Sample Application CI Walkthrough

The sample-app-ci directory contains the object manifests which define the CI implementation for our sample application. Because the CI solution uses Tekton, we already have a delcarative definition of the pipeline and its required objects, so there's no need to separate the pipeline pre-requisite infrastructure and components from the pipeline: they are one and the same.

In this case, we have the sub.yaml which defines OpenShift Pipelines Operator subscription which allows us to install the OpenShift Pipelines Operator and have it managed via the OpenShift cluster's Operator Lifecycle Manager. We also have a Persistant Volume Claim as defined in sample-app-pvc.yaml, a custom Task in make-sample-app-task.yaml, the actual Pipeline in pipeline.yaml, and a PipelineRun in run.yaml.

In Tekton, a Pipeline is a pipeline definition componsed of Task objects. Tasks can be executed individually via a TaskRun object, but here we want to use them in a pipeline. In order to execute a Pipeline, Tekton provides a PipelineRun object. We use this to provide values to parameters defined in the Pipeline and its Tasks.

Our Pipeline contains tasks for performing a git-clone, building the application and its container image, pushing the image to an image registry, and committing a change to the sample-app-config/sample-app-deployment.yaml so that it is updated with the image it just built.

Sample App GitOps in Action

DEMO SAMPLE APP RECONCILIATION

1) Prepare Your Environment

For ease of using the commands below, set an environment variable that contains YOUR_USERNAME (replacing "yourusername" with a username of your choice, in lowercase):

NOTE: Your username MUST be entered as all lowercase characters to create a valid namespace. You can choose any username, so long as it will be unique to you.

$ export YOUR_USERNAME=yourusername

NOTE: If you do not have a kubeconfig file in your workspace, be sure to perform an oc login to your destination cluster from your workspace.

$ cd ansible
$ ansible-playbook -i inventory participants-setup.yaml \
  -e kubeconfig=/path/to/kubeconfig \
  -e participant="${YOUR_USERNAME}"

INSTRUCTION: Execute the playbook referenced above during the workshop in the interest of time. The steps executed by the playbook are described below.

Make sure you are logged into the cluster with a user that has self-provisioner set, then execute the following to create the directories to work from and copy over the files we'll be editing:

$ cd /path/to/gitops-workshop
$ mkdir "${YOUR_USERNAME}"-sample-app-config
$ cp -r sample-app-config "${YOUR_USERNAME}"-sample-app-config
$ mkdir "${YOUR_USERNAME}"-customresources
$ cp -r ansible/files/workshop-sample-app-cr.yaml "${YOUR_USERNAME}"-customresources

Next we need to edit the files so that they are personalized and don't cause conflicts with any other workshop participants:

$ cd "${YOUR_USERNAME}"-sample-app-config
$ sed -i 's/sample-app/"${YOUR_USERNAME}"-sample-app/g' sample-app-deployment.yaml sample-app-namespace.yaml sample-app-networkpolicy.yaml
$ cd ../"${YOUR_USERNAME}"-customresources
$ sed -i 's/sample-app/"${YOUR_USERNAME}"-sample-app/g' workshop-sample-app-cr.yaml

Finally, we need to copy and modify the file that will allow us to set up our github and pull secrets:

$ cd /path/to/gitops-workshop/ansible
$ cp secrets.yaml "${YOUR_USERNAME}"-secrets.yaml
$ sed -i 's/sample-app-ci/"${YOUR_USERNAME}"-sample-app-ci/g' "${YOUR_USERNAME}"-secrets.yaml

NOTE: The following is NOT completed by the playbook referenced at the top of this section and must be executed manually.

Create a branch that is named YOUR_USERNAME and push your first commit:

$ cd /path/to/gitops-workshop
$ git checkout -b "${YOUR_USERNAME}"
$ git add .
$ git commit -m "environment set up"
$ git push -u origin "${YOUR_USERNAME}"

That's it, your environment is now prepared for the rest of the workshop content. To recap, we've branched the workshop content, copied and personalized the files with which we'll be working, and established this as our "feature" branch.

2) Initial Deployment

In this section we will tell ArgoCD to deploy our application. We will be using CustomResources, which are instances of a CustomResourceDefinition, to represent our deployments. In this case we will be using the Application "type" object to tell ArgoCD about our sample-app deployment.

NOTE: Execute the following playbook if you have difficulty following along or just wish to save time:

$ cd /path/to/gitops-workshop/ansible
$ ansible-playbook -i inventory participants-deploy.yaml -e kubeconfig=/path/to/kubeconfig -e participant="${YOUR_USERNAME}"

First, let's deploy the application and ci pipeline. You will need to change to the YOUR_USERNAME-customresources directory and use either oc or kubectl to apply the CustomResources depending on the cluster type you're using ("Vanilla" Kubernetes or OpenShift):

$ cd "${YOUR_USERNAME}"-customresources
$ oc apply -f workshop-sample-app-cr.yaml

Inside your cluster console, from the navigation pane you can select Networking -> Routes to view the available URLs for connection to your deployed ArgoCD instance.

NOTE: You will need to have the argocd project selected to view the relevant routes.

Click on the URL in the Location column for the route named workshop-argocd-server. This will open the ArgoCD login page:

In order to login to the ArgoCD server, you will need to retrieve the admin password from the Argo CD deployed secret. Execute the following commands to retrieve the password:

$ oc project argocd
$ export ARGOCD_CLUSTER_NAME=workshop
$ oc get secret $ARGOCD_CLUSTER_NAME-argocd-cluster -o jsonpath='{.data.admin\.password}' | base64 -d

The returned string will be the password for the admin user login to Argo CD.

You will see the Argo CD Applications dashboard, and you should see the pipelines we created for our workshop project:

3) Making Changes

Now comes the fun part. We are going to make some changes to our files in git, commit and push them, and watch how ArgoCD automatically syncs those changes so that we don't have to manually log into the cluster and perform the deployment operation ourselves.

NOTE: Execute the following playbook to save time or if you encounter an error attempting to follow the steps in this section:

$ cd /path/to/gitops-workshop/ansible
$ ansible-playbook -i inventory participants-make-changes.yaml -e kubeconfig=/path/to/kubeconfig -e participant="${YOUR_USERNAME}"

First let's make a small change, adding a more personalized greeting to the sample-app-deployment.yaml manifest. The provided sample-app-deployment.yaml defaults to a very generic greeting, 'Hello participant\!' as described by the file content below:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: sample-app
          image: "quay.io/sscaling/gitops-workshop:v0.0.1"
          command:
            - '/usr/bin/greet'
          args:
            - 'Hello participant\!'
...

To change this, you can use your text editor or execute the following commands:

$ cd /path/to/gitops-workshop/"${YOUR_USERNAME}"-sample-app-config
$ sed -i 's/participant/"${YOUR_USERNAME}"/g' sample-app-deployment.yaml

The result is that the arg on line 25 no longer says 'Hello participant\!', but instead reflects whatever YOUR_USERNAME is. For example: 'Hello btomlins\!'. If YOUR_USERNAME is btomlins, then you should see the following completed file:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: sample-app
          image: "quay.io/sscaling/gitops-workshop:v0.0.1"
          command:
            - '/usr/bin/greet'
          args:
            - 'Hello btomlins\!'
...

In order for ArgoCD to act upon this change, we will need to commit our changes and push them to our remote repo. You can do so using the following commands:

$ cd /path/to/gitops-workshop/"${YOUR_USERNAME}"-sample-app-config
$ git add sample-app-deployment.yaml
$ git commit -m "Add ${YOUR_USERNAME} to deployment"
$ git push -u origin "${YOUR_USERNAME}"

If you observe the ArgoCD Dashboard, you will see the YOUR_USERNAME-sample-app Application recognize the change to the repository, sync the updated file, and apply the change to the deployment:

What Just Happened?

Several things just happened "auto-magically" based on our ArgoCD configuration.

1. ArgoCD detected a new commit to your git repository
   • git fetch
2. ArgoCD pulled the change into our configured Application from the latest revision
   • git merge
3. ArgoCD pushed the updated file to our OCP deployment
   • oc apply -f sample-app-deployment.yaml
4. OCP spun up a new pod containing the updated Application file, and spun down the outdated deployed pod

You can see the updated deployment in your OCP console by navigating to Workloads -> Deployments for your YOUR_USERNAME-sample-app project:

4) Running the Tekton Pipeline

Now we will demonstrate how a Tekton pipeline can be incorporated in the GitOps process to automate the creation of a new image version in our desired image registry.

First, we will deploy our Tekton pipeline. In order to save time, you should run the following playbook:

$ cd /path/to/gitops-workshop/ansible
$ ansible-playbook -i inventory participants-pipeline-deploy.yaml \
  -e kubeconfig=/path/to/kubeconfig \
  -e participant="${YOUR_USERNAME}" \
  -e state=present \ # Use 'absent' to undeploy
  -e internal_registry=$REGISTRY_NAME # Omit this unless using a registry such as Artifactory or Nexus to limit image access

NOTE - This playbook performs several tasks for the participant. The equivalent commands can been seen below:

Copy the necessary CRD files for the Tekton pipeline and its component tasks:

$ cp -r sample-app-ci "${YOUR_USERNAME}"-sample-app-ci

Replace the namespace, git branch, and config path with the participant's respective values:

$ cd ../"${YOUR_USERNAME}"-sample-app-ci
$ sed -i 's/sample-app-ci/"${YOUR_USERNAME}"-sample-app-ci/g' 01-sample-app-ci-namespace.yaml
$ sed -i 's/master/"${YOUR_USERNAME}"/g' 30-pipeline-run.yaml
$ sed -i 's/sample-app-config/"${YOUR_USERNAME}"-sample-app-config/g' 30-pipeline-run.yaml

Deploy the pipeline and its component tasks from the CRDs:

NOTE - If you specified an internal_registry during deployment, the appropriate images used in the pipeline will be updated to the correct paths.

$ oc create -f 01-sample-app-ci-namespace.yaml
$ oc create -f 02-sample-app-pvc.yaml
$ oc create -f 10-buildah-task.yaml
$ oc create -f 10-git-cli-task.yaml
$ oc create -f 10-git-clone-task.yaml
$ oc create -f 10-make-sample-app-task.yaml
$ oc create -f 10-version-increment-task.yaml
$ oc create -f 20-pipeline.yaml

NOTE - The following steps are not executed by the playbook above and must be run by the participant:

Finally, we need to patch our GitHub token and image pull secrets into the Tekton pipeline Service Account for our namespace:

$ ansible-playbook -i inventory "${YOUR_USERNAME}"-secrets.yaml \
  -e kubeconfig=/path/to/kubeconfig --ask-vault-pass

NOTE - Your workshop facilitator will provide you with the Ansible vault password you need to enter.

Now we can make a change to our pipeline run CRD. For this change, we will update the version number for our image (image-version in YOUR_USERNAME-sample-app-ci/30-pipeline-run.yaml):

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: gitops-workshop-pipeline-run-
  annotations:
    argocd.argoproj.io/hook: Sync
spec:
  pipelineRef:
    name: build-increment-commit
  workspaces:
    - name: gitops-workshop-repo-source
      persistentVolumeClaim:
        claimName: git-clone-output
  params:
    - name: repo-url
      value: https://github.com/darthlukan/gitops-workshop
    - name: git-branch
      value: master
    - name: image-name
      value: quay.io/sscaling/gitops-workshop
    - name: image-version
      value: v0.0.2
    - name: app-path
      value: sample-app
    - name: app-config-path
      value: sample-app-config
    - name: app-config-filename
      value: sample-app-deployment.yaml

Now we will commit this change to our git repo:

$ cd /path/to/gitops-workshop
$ git pull
$ git add .
$ git commit -m "changed image version"
$ git push

And our final step is to kick off a new PipelineRun, which will trigger a new image build and push that image to our desired image registry with the new version number we specified:

$ cd ../"${YOUR_USERNAME}"-sample-app-ci
$ oc create -f 30-pipeline-run.yaml -n "${YOUR_USERNAME}"-sample-app-ci

After one PipelineRun has been created, you can also choose to trigger a new run from the OCP console by navigating to Pipelines -> Pipeline Runs -> Rerun (on the last PipelineRun):

The following actions will then be triggered automatically:

• A new PipelineRun is created in OCP with a generated name following the pattern: gitops-workshop-pipeline-run-<random-string>

• The Tekton pipeline checks out our feature branch, builds a new image, and pushes it to our image repository

• Tekton commits a NEW change to our repository indicating the updated image has been created (updates image name in sample-app-deployment.yaml)

NOTE - It would be possible at this point to configure an app in ArgoCD that would watch the sample-app repository and automatically deploy the updated image. That is beyond the current scope of this workshop.

Considerations for production implementations

This section includes some best practices to note when considering the use of GitOps and/or related tools through production. As the focus of the workshop is on concepts and techniques, any mention of specific tools is meant to be illustrative and not prescriptive.

Security

• Follow the "Principle of Least Privilege". Do not grant broad access unless broad access is actually warranted. A best practice is to have engineers document the roles and permissions required for their workloads based on the functionality those roles and permissions provide.
• Operations should be audited. GitOps by nature provides traceability (who did what and when) but someone or some thing should periodically review traced actions to ensure correctness.
• All tools should be vetted by your security team(s) and tools prior to being used in production.
• Use SSO to set up users and roles, and then remove the admin account. ArgoCD and other GitOps tools have SSO functionality that should be used instead of "local users".
• Plan for and make use of Secrets Management before implementing GitOps or choosing your tools. Examples of tools known to work well in a GitOps setting are HashiCorp Vault, Helm Secrets, and Ansible Vault.

Configuration

• Separate application configuration from application source code. Practically, this means your application manifests "live" in their own repository (e.g. sample-app-config) and your application sources (e.g. sample-app) are also stored in their own repository.
• The application configuration repo(s) should be owned by the same team writing the application, not an external team who can be a bottleneck to deployment.
• Test your configuration changes before committing them.
• Configuration should not change due to external changes. For example, pin your modules to versions so that you don't introduce drift. This also makes it easier to control version changes with Continuous Integration tools.

Execution

• Document the steps necessary to deploy manually. With GitOps, these are primarily git tasks along with corresponding automation tools such as ansible, oc, and kubectl. Ensure the team understands these steps before introducing further automation via tools such as ArgoCD or FluxCD.
• Once the manual steps are well understood, documented, and automation tools implemented, automate as much as possible. This limits errors caused by typos or distractions common during manual execution of tasks.

Further Reading

For those that wish to learn more about GitOps, and the tools that are currently available to implement GitOps workflows quickly and efficiently, check out the following resources:

• ArgoCD - GitOps continuous delivery tool for Kubernetes

• Tekton - Continuous integration and delivery solution for Kubernetes

• OpenShift Pipelines - OCP specific Tekton integration

• Introduction to GitOps with OpenShift

• GitOps Overview - High level explanation of GitOps Philosophy

• 5 GitOps Best Practices

• Understanding GitOps with Advanced Cluster Management