Realm of Chaos - juju

WordPress Charm for k8s

2020-12-02T00:00:00+00:00

Canonical uses the WordPress blogging system for all our company blogs. Earlier this year I was tasked with updating our WordPress charm from a Services Framework Juju Charm on OpenStack to a Kubernetes based Operator Framework Charm.</p>

The WordPress Operator Charm is simple by design, the goal of the charm is to just provide the various configuration options for our WordPress Kubernetes image, which does all the heavy lifting. See the config.yaml</a> file for details on what is supported, one option that can be useful during testing is container_config</code>, which gives you the ability to pass through custom Kubernetes spec environment variables. For example, to enable debug level logging and ensuring you always have the latest image, you would set:</p>

microk8s.juju config wordpress container_config='WORDPRESS_DEBUG: "1" imagePullPolicy: "always"'</span></span></code></pre>
The image build process downloads the latest WordPress codebase and installs it behind an Apache web server, it then downloads the plugins all of our blogs depend on, such as, akismet anti-spam support, SSO with the openid teams plugin, and a variety of Canonical Open Source themes. By default the charm will use the current stable build here, however if you wish to customise the image you can fork the code and update the image</code> charm config option to point to the location of your custom image.</p>
To get started with the Operator Framework WordPress charm you will need a MySQL database running locally. As I write this post there is no Kubernetes MySQL charm, so deploy one to an IaaS model with juju deploy cs:mysql</code>. Initialise the database as follows:</p>
CREATE DATABASE wordpress CHARACTER SET utf8 COLLATE utf8_unicode_ci;</span></span>
CREATE USER 'wordpress'@'%' IDENTIFIED BY 'wordpress';</span></span>
GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'%';</span></span>
FLUSH PRIVILEGES;</span></span></code></pre>
Once the database is prepared we are now able to deploy the WordPress charm. The easiest way to get started with a local Kubernetes cluster is to have MicroK8s installed, reference the README</a> of the charm for details on how to get one setup. Deploy the charm as follows.</p>
Deploy the charm into your Kubernetes Juju model.</p>
microk8s.juju deploy cs:~wordpress-charmers/wordpress</code></p>
The charm requires Kubernetes TLS secrets to be pre-configured to ensure logins are kept secure. Create a self-signed certificate and upload it as a Kubernetes secret.</p>
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout server.key -out server.crt</code> microk8s.kubectl create secret tls -n wordpress tls-wordpress --cert=server.crt --key=server.key</code></p>
Tell the charm where the database is and provide some initial setup.</p>
DB_HOST=$IP_OF_YOUR_MYSQL_DATABASE</span></span>
microk8s.juju config wordpress db_host=$DB_HOST db_user=wordpress db_password=wordpress tls_secret_name=tls-wordpress \</span></span>
            initial_settings="user_name: admin</span></span>
            admin_email: devnull@example.com</span></span>
            weblog_title: Test Blog</span></span>
            blog_public: False"</span></span></code></pre>
From there you can test the site by updating your /etc/hosts</code> file and creating a static entry for the IP address of the Kubernetes ingress gateway.</p>
    App        Version                  Status   Scale  Charm      Store  Rev  OS          Address  </span></span>
    Messagewordpress  wordpress:bionic-stable  waiting      1  wordpress  local    0  kubernetes  10.152.183.140 </span></span></code></pre>
echo '10.152.183.140 myblog.example.com' | sudo tee -a /etc/hosts</code></p>
It will take about 5 to 10 minutes for Juju hooks to discover the site is live and perform the initial setup for you. Look for this line in the output of juju debug-log</code> to confirm.</p>
unit.wordpress/0.juju-log Wordpress configured and initialised</code></p>
This is due to issue #166</a> and will be fixed once Juju supports a Kubernetes pod ready hook.</p>
To retrieve the random admin password, run the following.</p>
microk8s kubectl exec -ti -n wordpress wordpress-operator-0 -- cat /root/initial.passwd</code></p>
You should now be able to browse to https://myblog.example.com/wp-admin</a>.</p>
We’ve been using this charm in production for five months now, but recently updated it to bring it up to date with a current version of the Operator Framework. We’d be interested in any feedback on the charm itself, either here or via bugs against the charm project on Launchpad</a>.</p>



Managing Multi Cloud Services With Juju
2018-10-24T00:00:00+00:00
Managing a service with deployments in multi-cloud environments can be a challenge in terms of troubleshooting and scalability due to the complexity of dealing with different public cloud providers. An effective way to manage services deployed cross-cloud is to use tools that allow you to define your service once and deploy anywhere: in the cloud, on bare metal, or locally inside containers. In this blog post I am going to describe how the Canonical SRE team has achieved this, the tools that we use and the way we apply them to manage the Ubuntu Archive Mirror service.</p>
Historical Archive Mirror Setup</h1>
</p>
It’s no secret that Ubuntu is a popular OS for running applications in the cloud. It was clear from the beginning that the increased adoption of Ubuntu in the cloud would cause a significant amount of stress on the master Ubuntu Archive Mirror service due to an increased rate of requests from Ubuntu instances running in public clouds. This was not a new problem for us, the Ubuntu Archive Mirror Network was originally conceived to protect the master Ubuntu Archives from getting over saturated with requests. The network architecture is a globally distributed system of web servers and caches that are operated by the community. The goal of this network is to improve response times for community users by directing traffic destined for the master Ubuntu Archive repositories to repositories geographically closer to the request’s origin. Without this distributed cache the millions of apt requests we receive every day would cause a denial of service to the master repository servers. Ubuntu updates would then be significantly degraded, which is unacceptable for any system running production grade services.</p>
The nice thing about the Ubuntu Archive Mirror architecture is how extensible it is. Within each cloud region we have a deployment of the Ubuntu Archive Mirror service. Each regional deployment keeps in sync with the master repositories. When an Ubuntu instance is started in the cloud its apt sources are configured to point to the deployment servicing the region of which the instance is currently running. We apply this pattern to every public and private cloud we manage. This includes AWS, GCE, Azure and OpenStack.</p>
Identifying the Challenges</h1>
After understanding our architectural requirements, the next step was to figure out how to manage each individual Ubuntu Archive deployment. A non exhaustive list of challenges we needed to address were:</p>

How do we manage the lifecycle of each Ubuntu Archive Mirror service?</li>
Can we be sure that each environment is consistent and that the user experience remains the same no matter which Archive Mirror environment you are working on?</li>
How can we test proposed changes before upgrading production services?</li>
How do we respond to incidents when things go wrong with any of our environments and how do we ensure each environment is isolated from each other to avoid unnecessary outages?</li>
</ul>
Our engineers can’t get bogged down with the semantics of each cloud. The best approach is to have a uniform interface that applies to any platform. This saves engineering time by making our playbooks more succinct and easier to digest, there is less room for error when there are minimal commands to run, a consistent UX reduces mean time to recovery as an engineer can be confident that the same troubleshooting methodology can be applied no matter which environment they are in, finally, environment specifications stored in version control allow us to keep track of changes, and easily test and rollback revisions that do not work.</p>
The Solution</h1>
Our Ubuntu Archive mirror services are modeled with Mojo which orchestrates deployments with Juju. “Automate once, deploy anywhere” is an approach we have taken in developing our environment automation solution. We define what our environments look like once and have the tooling take care of the boring details. Juju</p>
Juju is the component which drives each deployment. It is responsible for talking to the cloud APIs, installation and configuration of each application, provisioning cloud services, such as security, networking, storage, and user access control. Juju enables us to deploy and scale services quickly and efficiently to a variety of providers: public clouds, physical servers, OpenStack, and local containers. Juju is responsible for provisioning instances and installing the application (a charm in Juju context) onto them. It is also responsible for managing services, such as ensuring ACLs are configured, configuring advanced networking, and allocating storage.</p>
Deployment Control</h1>
</p>
All our Archive Mirror deployments are centrally managed. We have a deployment host from which all SREs connect to environments we manage. An internally developed tool allows SREs to search for the environment they need to connect to (such as “Azure Archive Mirror for Japan East region”), and then configures for them everything they need to connect to that environment in a consistent way.</p>
The Ubuntu Archive Mirror service itself is modeled as a Mojo specification. Mojo is a system of configuration and tools for verifying the success of Juju environment deployments. It allows us to define what an Archive Mirror is in a revision-controlled configuration repository, and a means of validating whether what we deployed is as we intended.</p>
To deploy a new environment we use Mojo to run a manifest file which tells it which operations (phases) to perform and in what order. The manifest file looks similar to this:</p>
collect</span></span>
deploy config=services</span></span>
deploy config=relations</span></span>
script config=post-deploy</span></span>
sleep config=10</span></span>
nagios-check skip-checks-extra="check_ksplice,check_total_procs"</span></span>
juju-check-wait</span></span>
verify config=verify-deploy</span></span></code></pre>
Mojo recognises different phases as different actions to perform. The three main phases to be concerned about are:</p>

collection

Mojo will download the charms (service modules) required as per the file specified in the collect call.</li>
</ul>
</li>
deployment

The deploy phase is fairly self explanatory, it takes the artefacts from the collection phase and deploys them into the Juju model.</li>
</ul>
</li>
verification

A script can be called once the deploy phase is complete. The script is configured to run some verification commands to make sure the deployment is in the desired state.</li>
</ul>
</li>
</ul>
We define a collect file that contains an inventory of each component, known as a charm. This file looks like the following:</p>
ubuntu-repository-cache lp:ubuntu-repository-cache;overwrite=True</span></span>
haproxy                 cs:haproxy</span></span>
nrpe                    cs:nrpe</span></span></code></pre>
The actual service and relation specifications detail how each charm is related and any configuration options, i.e. Squid cache size and HAProxy monitor settings. For a look at the specification please see the appendix at the end of the post. Mojo uses codetree to pull down the charms, from either the charmstore or version control (e.g. launchpad.net or github.com), to the host and copies them to a charm repository for deployment. A phased approach protects us from incorrectly configured specifications as Mojo will not proceed if a phase fails. At the end of this process we have a new validated environment deployed and ready for use.</p>
You can try this out for yourself, download the example mojo spec.</p>


Before we begin you will need to have Juju, Mojo and a Juju controller bootstrapped. You can find a tutorial on how to set this up here.</p>
</li>

Configure a Mojo project and workspace which we will use to build and test our archive mirror environment locally before it gets deployed into our cloud.</p>
export MOJO_STAGE=production export MOJO_WORKSPACE=archive-mirror export MOJO_PROJECT=mojo-archive-mirror-proj export MOJO_SERIES=xenial export MOJO_SPEC=/mojo/mojo-archive-mirror-spec/ export MOJO_LOCAL=</del>/mojo/LOCAL/mojo-archive-mirror-spec/</p>
juju bootstrap</p>
mkdir -p ${MOJO_SPEC}; mkdir -p ${MOJO_LOCAL}; (cd ~/mojo; curl https://admin.insights.ubuntu.com/wp-content/uploads/db69/mojo-archive-mirror-spec.tar.gz</a> | tar xv)</p>
mojo project-new -s $MOJO_SERIES -c lxd mojo workspace-new –project $MOJO_PROJECT -s $MOJO_SERIES $MOJO_SPEC $MOJO_WORKSPACE mojo run</p>
</li>
</ul>
Mojo has now orchestrated a deployment of our Archive Mirror environment. Below details the instances that were created and the applications installed onto them.</p>
$ juju status</span></span></code></pre>
We know our new Archive Mirror is functioning because the verify phase in our manifest has curl’d the HAProxy unit and downloaded the list of supported Ubuntu series.</p>
2018-07-16 06:53:34 [INFO] Running script verify-deploy</span></span>
2018-07-16 06:53:35 [INFO] /ubuntu/</span></span>
artful-backports/</span></span>
artful-proposed/</span></span>
artful-security/</span></span>
artful-updates/</span></span>
artful/</span></span>
bionic-backports/</span></span>
bionic-proposed/</span></span>
bionic-security/</span></span>
bionic-updates/</span></span>
bionic/</span></span>
[...]</span></span>
2018-07-16 06:53:35 [INFO] Completed script verify-deploy in 0s (0.31s)</span></span></code></pre>
We keep track of each Archive Mirror deployment that we manage with stages embedded in the root of the Mojo spec directory. These stages are nothing more than sub-directories that Mojo looks out for. Under each stage is the information used by each Mojo phase.</p>
The directory layout looks a little something like this:</p>
mojo-archive-mirror-spec # repository root.</span></span>
├── manifest # the manifest script, this is what mojo run is looking for.</span></span>
└── production</span></span>
├── collect # Contains the list of charms that the collect phase will download.</span></span>
├── haproxy-services.yaml</span></span>
├── relations # Specifies the contracts between each service.</span></span>
└── services # Details how each service should be configured.</span></span></code></pre>
Mojo uses this approach when sourcing local significant configuration. This comes in handy when you want to do something like configure a service with dummy credentials for the CI deployment. If we wanted to add a new stage, like ci to the list of deployments we manage, we only need to create a new directory and populate the required files:</p>
mojo-archive-mirror-spec</span></span>
├── manifest</span></span>
└── production</span></span>
├── collect</span></span>
[...]</span></span>
└── ci</span></span>
├── collect</span></span></code></pre>
Any changes made to the Archive Mirror specification are tested. Our testing environment is a mixture of technologies: Jenkins, Openstack, Mojo and Juju. The testing procedure is broken down as follows:</p>

Jenkins job pulls the latest Mojo specification from version control.</li>
A new juju model is created for the revision of the specification. The model is deployed into one of our private OpenStack clouds.</li>
The Jenkins script then executes a Mojo run which deploys the Archive Mirror service to the new Juju model.</li>
Jenkins monitors the progress and provides feedback on any errors.</li>
</ul>
This process gives us assurance that the changes we have made are deployable albeit only within an Openstack context.</p>
Closing Comments</h1>
So far our current solution has worked well. The time taken for an SRE to redeploy an Archive Mirror environment is under an hour. If an environment is sick and we start receiving alerts we can be confident that if all else fails, and troubleshooting is taking too long, we can destroy the environment and redeploy a fresh baseline. The process is as simple as failing over DNS, recreating the Juju model, running Mojo, and configuring DNS and monitoring.</p>

Realm of Chaos - juju

WordPress Charm for k8s

Managing Multi Cloud Services With Juju

Identifying the Challenges</h1> After understanding our architectural requirements, the next step was to figure out how to manage each individual Ubuntu Archive deployment. A non exhaustive list of challenges we needed to address were:</p>