TL/DR

This post explores running faasd in LXC containers.

  • As a consumer of Faasd on a linux workstation I wanted an options to run in native containers.
  • Faasd Developers can benefit from a native runtime environment, as well as cheap/fast sandbox environments provided by LXC.
  • OpenFaas function function developers can benefit from a tighter inner-loop that doesnt invole pushing docker images to a remote registry just to test their functions

Introduction

First lets discuss what exactly is faasd and what it can be used for. Faasd is a provider implementation of OpenFaas run as a systemd daemon. There are a handful of providers that each plug into OpenFaas which enable it to run in different environments. For example there is a provider for docker, kubernetes and even an in-memory provider for an ephemeral experience. Faasd is really geared for low resource boards like Raspberry Pi, where the overhead of running a docker daemon is too high (faasd interfaces directly with containerd using runc). So what does this mean? Well faasd makes a nice controller-manager for very small containerized processes.

At the time of this post, faasd runs purely as systemd processes, the components are:

  • Faasd/faas-cli - systemd daemon and go CLI for managing functions of http
  • Containerd/Runc - container runtime (all functions must be containerized)
  • CNI - I havn't dug too deep here but I assume this is for creating and cleaning up IP addresses as all functions must be network callable

Why LXC?

Okay cool so it sounds like the functions are already in containers, why bring LXC into the mix? First let's take a look at how the components are deployed. The current set up is quite simple, just a few binaries and some systemd templates. These are all glued together in a simple cloud-config. This is awesome, basically our only requirement is a fresh linux OS that can execute the script.

But what if we want to run faasd on an existing linux system? Of course the simplest option, we can run a Virtual Machine, but we will have to pay the penalty of virtualization.

confession: I do not actually know the overhead of running a VM these days. It's probable the host would never know the diffence since I understand it is quite low.

Another option would be to run the processes directly on the host. This gets super messy without config managment tools like ansible. Given the current deployment is a handful of bootstrap scripts it would be a total pain to track things down if something breaks or we need to upgrade.

Which brings us to LXC. Linux Containers are simply namespaced filesystems running on top of a shared linux kernel. Some advantages to running faasd in LXC (beyond the above mentioned):

  1. We can easily spin up n+ environments including different versions for testing
  2. Containers are extremely fast to boot once the initial file system is pulled down.
  3. When something breaks we can blow away the entire container with minimal risk to the host.
  4. LXC actually has a network daemon included (LXD) which makes managing containers over the wire very convenient. See Next steps for a thought experiment involving LXD.

Demo

Okay enough yapping let's look at some code..

Pre-requisites

You will need to install LXD and then run lxd init to set up the network and storage pool. I stuck with dir for the storage backend to keep things simple and lxdbr0 for the network (bridge to the host).

warning: don't use ZFS storage pool yet, containerd support is not quite there.

Creating a Container

The basic steps are:

  • create a LXC profile with the faasd cloud-init script and security.nesting enabled (since we are using containerd inside of linux namespaces)

    LXD_PROFILE=faasd
LXD_USERDATA="/tmp/cloud-config.yml"

wget -qO $LXD_USERDATA https://gist.githubusercontent.com/gabeduke/7ccb3f3147d79ac30e2187432808060c/raw/4028ea0658e9aa0a37f6ac44473a8092e3824406/cloud-init.yml
lxc profile set $LXD_PROFILE user.user-data - < $LXD_USERDATA
lxc profile set $LXD_PROFILE security.nesting=true

lxc launch ubuntu:18.04 faasd -p $LXD_PROFILE

  • Once the container is finished bootstrapping we can fetch the faasd password and connect to the OPENFAAS_GATEWAY over the network

    FAASD_PASSWORD=$(lxc exec faasd -- cat /var/lib/faasd/secrets/basic-auth-password)
FAASD_IP=$(lxc exec faasd -- /sbin/ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1)

echo $FAASD_PASSWORD | faas-cli login --password-stdin --gateway http://${FAASD_IP}:8080

  • There should now be a container running:

    » lxc list
+-------+---------+-----------------------+----------------------------------------------+-----------+-----------+
| NAME  |  STATE  |         IPV4          |                     IPV6                     |   TYPE    | SNAPSHOTS |
+-------+---------+-----------------------+----------------------------------------------+-----------+-----------+
| faasd | RUNNING | 10.62.0.1 (openfaas0) | fd42:ad54:4bbd:9f9:216:3eff:fe50:66ee (eth0) | CONTAINER | 0         |
|       |         | 10.225.76.18 (eth0)   |                                              |           |           |
+-------+---------+-----------------------+----------------------------------------------+-----------+-----------+

  • And OpenFaas should be addressable on the gateway IP:

    » faas store deploy figlet
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.

Deployed. 200 OK.
URL: http://faasd.local:8080/function/figlet

» faas list
Function                      	Invocations    	Replicas
figlet                        	0              	1

Here is the full convenience script (I am using hostess to update /etc/hosts with the new gateway IP):

#!/bin/bash 

# Set up the environment
export OPENFAAS_URL=http://faasd.local:8080

LXD_PROFILE=${1:-dukemon}
LXD_USERDATA="/tmp/cloud-config.yml"
LXD_CONTAINER=${2:-faasd}

if ! (lxc info $LXD_CONTAINER > /dev/null 2>&1)
then
	echo "ensure lxc profile"
	wget -qO $LXD_USERDATA https://gist.githubusercontent.com/gabeduke/7ccb3f3147d79ac30e2187432808060c/raw/4028ea0658e9aa0a37f6ac44473a8092e3824406/cloud-init.yml
	lxc profile set $LXD_PROFILE user.user-data - < $LXD_USERDATA
	lxc profile set $LXD_PROFILE security.nesting=true

	echo "launching LXC container ${LXD_CONTAINER}.."
	lxc launch ubuntu:18.04 $LXD_CONTAINER -p $LXD_PROFILE
fi

# Wait for cloud-init sequence to finish before moving on to the configuration steps
until (lxc exec $LXD_CONTAINER -- cat /var/lib/cloud/instance/boot-finished > /dev/null 2>&1)
do
	echo "Waiting for cloud-init complete signal.."
	sleep 5
done

# let's be safe and wait for the last thread to finish
wait

echo "fetch faasd password from $LXD_CONTAINER"
FAASD_PASSWORD=$(lxc exec $LXD_CONTAINER -- cat /var/lib/faasd/secrets/basic-auth-password)

FAASD_IP=$(lxc exec $LXD_CONTAINER -- /sbin/ip -o -4 addr list eth0 | awk '{print $4}' | cut -d/ -f1)
echo "faasd IP is $FAASD_IP"

if which hostess
then
	echo "Updating /etc/hosts"
	sudo hostess add "faasd.local" $FAASD_IP

	wait

	until (echo $FAASD_PASSWORD | faas-cli login --password-stdin > /dev/null 2>&1 && echo "login successful")
	do
		echo "Attempt faasd login.."
		sleep 5
	done
else
       	echo "install hostess to /usr/local/bin to automatically update /etc/hosts (https://github.com/cbednarski/hostess)"
fi

echo "init complete!"

Hot loading images

This is great we now have a container running with all of the faasd components and can easily add copies or swap out a new version. There is one last issue which is shortening the development loop for buildling functions. Containerd does not re-pull an image so once we publish a tag we cannot update it. It is common in development cycles to use a tag like latest to keep publishing images to. But we are running the containerd socket locally so why would we even want to push to a remote registry just to pull the same image back down to a different location on our machine? We can do better..

Let's go over the steps to hot load images directly into containerd:

  • When developing OpenFaas functions we can just build the images locally and save them to OCI compatiable images using docker

    faas build -f fn.yml
docker save [registry]/[image]:[tag] -o fn.tar

  • Now we can push the tarball into the LXC container and import it to the correct namespace:

    lxc file push fn.tar faasd/
lxc exec faasd -- ctr -n openfaas-fn images import /fn.tar

  • Now we can simply run faas deploy to restart the container with the updated image.

    faasd deploy

Next steps

I had a lot of fun playing around with and containerizing Faasd, on a closing note here are some more things to try:

  • Use LXD to create an autoscaling group of containers (need to ship container metrics and scale on capacity events)
  • Use distrobuilder to pack a sharable image for more of a docker or ISO like experience.