fluids.rocks

tsgxr: Loading HyLogger TSG data into xarray

Morgan Williams — Sat, 19 Sep 2026 00:00:00 +0000

For those working with HyLogger data in TSG format, I've put together an open source Python package tsgxr which allows reading these relatively arcane formats directly into Xarray. tsgxr is available via GitHub and also directly from PyPI.

Over the past few weeks I've restructured the package and added the ability to lazily-load data, meaning that even the largest files should now be able to be opened (if not loaded, necessarily) on even modest hardware.

Note: tsgxr leans on another package for some of the basic structure/loaders, pytsg, now managed by the Geological Survey of Western Australia but originally written by Ben @ FractalGeoAnalytics and Andrew Rodger (ex-CSIRO). The dependence on pytsg has lessened after writing the xarray loaders, and some bugfixes in tsgxr are yet to make it back to pytsg, but they're on the way.

HyLogger Data

HyLogger instruments are used principally for hyperspectral line-scanning of drillcore (typically at about 1cm resolution), from which spectral mineralogy is derived - allowing identification and unmixing of mineral signatures downhole, and later domaining along the drillhole trace ($\pm$ in 3D). The instrument generates a composite dataset including spectral images (for which there are typically two or three hyperspectral cubes - near-infrared, thermal-infrared and with the latest HyLogger instruments also mid-infrared; NIR, TIR and MIR for short), a lidar trace along the centre of the drillcore, and high resolution truecolor imagery of the samples.

The datasets are provided in TSG formats (partly due to the fact that they're processed through TSG), which include one .bip image per hyperspectral sensor, each paried with a .tsg file containing metadata/headers, a lidar trace in .dat and a .bip file containing a concatenated series of compressed JPEGs (i.e., a small JPEG bomb, if you're short on RAM). This latter item has caused issues previously, particularly for longer drillholes (e.g., for one dataset, 6GB of compressed JPEGs would expand to ~220GB), and as such was something I was looking to handle more cleanly.

HyLogger data is typically scanned by state geological surveys across Australia, and a large amoutn of this is provided openly via the National Virtual Core Library (NVCL), supported by AuScope and also available via the AuScope portal.

Recent `tsgxr` Updates

In the interest of keeping this brief and minimially outdated over time I'll just mention a few things which I think are useful below, and save the rest for the README or to be updated in the package directly.

Using an `xarray.DataTree` Structure

Newer versions of tsgxr use a xarray.DataTree structure, which enables each of the varied parts of the dataset to be represented (and stored/serialized if desired) within a single heirarchical dataset. Being in xarray, this also allows labelling the relevant physical/semantic dimensions of the dataset, assignment of the respective coordinates, and the inclusion of metadata as attributes. A short overview for an example (the STAVELY_17 hole as provided by the NVCL) is given below:

DT : xarray.DataTree = open_tsg(
    "./07e4dcac-5216-44a6-9a6b-0c4c1f7ce7d", index_coord="depth",
)
DT

<xarray.DataTree>
Group: /
├── Group: /NIR
│       Dimensions:                    (depth: 19754, wavelength: 531, feature: 25)
│       Coordinates:
│         * depth                      (depth) float32 79kB 0.004111 0.004112 ... 156.0
│           sample                     (depth) uint64 158kB 1 4 2 ... 23233 23276 23251
│           tray                       (depth) uint16 40kB ...
│           section                    (depth) uint8 20kB ...
│           hole                       (depth) object 158kB ...
│         * wavelength                 (wavelength) float64 4kB 380.0 384.0 ... 2.5e+03
│           band                       (wavelength) int64 4kB ...
│         * feature                    (feature) uint8 25B 0 1 2 3 4 ... 20 21 22 23 24
│       Data variables: (12/54)
│           Spectra                    (depth, wavelength) float32 42MB ...
│           Centres                    (depth, feature) float32 2MB ...
│           ...
│       Attributes: (12/34)
│           core_qual:                 ['Void', 'Rubble', 'Crack', 'Core']
│           TSA704_S Minerals:         ['Opal', 'Dickite', 'Kaolinite-PX', 'Kaolinite...
│           ...
├── Group: /TIR
│       Dimensions:                        (depth: 19754, wavelength: 341, feature: 25)
│       Coordinates:
│         * depth                          (depth) float32 79kB 0.004111 ... 156.0
│           sample                         (depth) uint64 158kB 1 4 2 ... 23276 23251
│           tray                           (depth) uint16 40kB ...
│           section                        (depth) uint8 20kB ...
│           hole                           (depth) object 158kB ...
│         * wavelength                     (wavelength) float64 3kB 6e+03 ... 1.45e+04
│           band                           (wavelength) int64 3kB ...
│         * feature                        (feature) uint8 25B 0 1 2 3 4 ... 21 22 23 24
│       Data variables: (12/47)
│           Spectra                        (depth, wavelength) float32 27MB ...
│           Centres                        (depth, feature) float32 2MB ...
│           ...
│       Attributes: (12/28)
│           TSA703_T Groups:           ['SILICA', 'K-FELDSPAR', 'PLAGIOCLASE', 'GARNE...
│           ...
├── Group: /Lidar
│       Dimensions:  (depth: 19754)
│       Coordinates:
│         * depth    (depth) float32 79kB 0.004111 0.004112 0.004114 ... 156.0 156.0
│           sample   (depth) uint64 158kB 1 4 2 5 0 3 ... 23204 23205 23233 23276 23251
│           tray     (depth) uint16 40kB ...
│           section  (depth) uint8 20kB ...
│           hole     (depth) object 158kB ...
│       Data variables:
│           Lidar    (depth) float32 79kB 92.44 75.37 75.98 77.04 ... 2.398 0.9316 58.43
└── Group: /Image
        Dimensions:  (depth: 2898500, channel: 3, width: 926)
        Coordinates:
          * depth    (depth) float32 12MB 0.004111 0.004177 0.004243 ... 156.0 156.0
            section  (depth) uint16 6MB 0 0 0 0 0 0 0 0 0 ... 49 49 49 49 49 49 49 49 49
            tray     (depth) uint16 6MB 0 0 0 0 0 0 0 0 ... 186 186 186 186 186 186 186
          * channel  (channel) int64 24B 0 1 2
          * width    (width) float64 7kB -0.03054 -0.03047 -0.0304 ... 0.03047 0.03054
        Data variables:
            Image    (depth, width, channel) uint8 8GB ...

Lazy-Loading

tsgxr now includes a series of backends for xarray which allow opening of components of TSG datasets without reading all of the contained data (i.e. only the metadata and some of the smaller components are read at open-time). Notably, when loading data, some limitations of indexing and lazy loading mean that more data than necesary might be loaded, as the depth-slicing is used as the principal axis (this is how the files are arranged; e.g., if slicing across the drillcore width for imagery, or across wavelengths for spectra - all of the data for that depth slice will be loaded, then subset). Some improvements to this may come in the future.

import xarray 

ds : xarray.Dataset = xarray.open_dataset(tsgfile, engine="tsg") # loading spectral .bip file lazily

ds : xarray.Dataset = xarray.open_dataset(crasfile, engine="lazycras") # loading truecolor image .bip lazyily

These interfaces are used internally within tsgxr, such that the key entry point open_tsg() now by-default lazy-loads the components for which this is an option (spectra, truecolor imagery).

Simpler Spectral Products

TSG has some limitations in how it runs its unmixing algorithms, such that for any given pixel along the line scan, only a few mineral species are unmixed by TSA (typically 2 or 3, depending on which spectral sensor is considered). These are provided in long-form in the TSG dataset, where paired records under e.g.(Min1, Grp1, Wt1), (Min2, Grp2, Wt2), represent series of values along the line scan identifying i) the mineral species unmixed (Min1 would be dominant), ii) the group assigned to that mineral (the more general classification, for which we can typically have more confidence) and iii) the unmixing fraction assigned to that mineral/group. tsgxr provides some options for turning this into a nice wide-form (but sparse) table more amenable to statistics and visualization.

Where collapse_products=True is used in the open_tsg() call, the spectral products/scalars will be reorganized into more useful tabluar forms (here e.g sTSAS_Grp, sTSAS_Min, ...) indexed by whichever coordinate used for spectra and the group/mineral as labelled under the respective systems used:

DT : xarray.DataTree = open_tsg(
    "./07e4dcac-5216-44a6-9a6b-0c4c1f7ce7d", index_coord="depth", collapse_products=True
)
DT['NIR'].ds : xarray.Dataset

<xarray.DatasetView> Size: 57MB
Dimensions:                    (depth: 19754, wavelength: 531, sTSASgroup: 7,
                                sTSASmineral: 15, sTSAVgroup: 2,
                                sTSAVmineral: 2, uTSASgroup: 7,
                                uTSASmineral: 15, uTSAVgroup: 2,
                                uTSAVmineral: 2, feature: 25)
Coordinates: (12/16)
  * depth                      (depth) float32 79kB 0.004111 0.004112 ... 156.0
    sample                     (depth) uint64 158kB 1 4 2 ... 23233 23276 23251
    tray                       (depth) uint16 40kB ...
    section                    (depth) uint8 20kB ...
    hole                       (depth) object 158kB ...
  * wavelength                 (wavelength) float64 4kB 380.0 384.0 ... 2.5e+03
    ...                         ...
  * sTSAVmineral               (sTSAVmineral) object 16B 'Goethite' 'Galvanis...
  * uTSASgroup                 (uTSASgroup) object 56B 'KAOLIN' ... 'CARBONATE'
  * uTSASmineral               (uTSASmineral) object 120B 'Kaolinite-PX' ... ...
  * uTSAVgroup                 (uTSAVgroup) object 16B 'OXIDE' 'NOTAROK'
  * uTSAVmineral               (uTSAVmineral) object 16B 'Goethite' 'Galvanis...
  * feature                    (feature) uint8 25B 0 1 2 3 4 ... 20 21 22 23 24
Data variables: (12/22)
    Spectra                    (depth, wavelength) float32 42MB ...
    sTSAS_Grp                  (depth, sTSASgroup) float64 1MB ...
    sTSAS_Min                  (depth, sTSASmineral) float64 2MB ...
    sTSAV_Grp                  (depth, sTSAVgroup) float64 316kB ...
    sTSAV_Min                  (depth, sTSAVmineral) float64 316kB ...
    uTSAS_Grp                  (depth, uTSASgroup) float64 1MB ...
    ...
Attributes: (12/34)
    core_qual:                 ['Void', 'Rubble', 'Crack', 'Core']
    TSA704_S Minerals:         ['Opal', 'Dickite', 'Kaolinite-PX', 'Kaolinite...
    TSA704_V Groups:           ['MISC-SILICATE', 'CARBONATE', 'SULPHATE', 'OX...
    ...

The functions achieving this are also available for use more generally: tsgxr.products.products_to_mineral_table(), tsgxr.products.products_to_group_table(). These will take a given spectral dataset, extract the relevant unmixing system, and return a nicely formatted table. They can also perform compositing via suppling a step argument (e.g. where depth is used as the primary index, step=0.5 will return a 50x downsampling to give a 50cm composite), and will return tables where the respective colormap as used within TSG is provided on the attribtes (mapping minerals or groups to hex-labelled colours).

Simple Access to Downhole Plotting

These nice mineralogy tables are also then amenable to simple downhole plotting, and tsgxr.products.plot_product_downhole() will produce stacked bar plots for a specified unmixing system, at a specific level (mineral or group), while optionally providing compositing as desired.

Hosting 'The Littlest JupyterHub'

Morgan Williams — Sat, 04 Apr 2026 00:00:00 +0000

Back in 2024 I ran a pyrolite workshop as part of the SGTSG conference (The Geological Society of Australia's Specialist Group in Tectonics and Structural Geology Biennual Meeting), where I used a hosted Jupyterlab instance to provide easy access to an environment with pyrolite installed for the attendees, in an attempt to maximise the use of time and minimise technical issues.

This proved a to be both technically fairly simple (in terms of getting a JupyterHub running) and logistically hard (in terms of getting infrastructure to run it on). Since I've wondered whether I can host such thing from a server (desktop running Proxmox) I own, and save both the hassle and the bill. I think I've figured out one way which will work, and this post covers it.

Getting a Large VPS is Hard

I had a bunch of headaches trying to set up a server in multiple of the major players (AWS, Azure, OVH, Digital Ocean). Azure even blocked my account for trying to do this, with no explanation as to why - not helpful a few days out from the workshop. The other major issue was that to access nodes with enough resources to handle 40-odd simultaneous users was nigh on impossible without having a previous record (and series of bills) with each of these providers. I think in the end I went back to AWS and settled for something subpar which would at least have a few folks running on the night. But the experience wasn't particularly fun.

Setting up The Littlest JupyterHub (TLJH)

As you'll find in other posts here, I use Proxmox as a host for many of my self-hosted services (small and not-to-small). Within this, I principally use Linux Containers (LXC) as my container solution. Typically, installing into an LXC will be pretty similar to installing on a Virtual Private Server (if you're e.g. renting a cloud server, as I did for the workshop originally) and especially similar to installing on any Linux host.

Here I'm going to start with a Debian 13 image (I think TLJH uses Debian 12 by default, which might explain one or two of the changes I needed), which we can quickly update after starting:

apt update && apt upgrade -y

Note: I don't use sudo here as I'm running as root for the majority of important tasks within the container, but on your device or in your container you might need it. Particularly for this one, you might need sudo -E.

We'll loosely follow the official guide for getting set up on your own server, which starts with installing some basic pre-requisites. I've had issues running the bootstrap.py install script on Debian 13 and Python 3.13, so have simplified this down to a few commands below. Here this is updated to include python3-venv, which for Python 3.13 as of writing this requires it:

apt install python3 python3-dev python3-venv git curl -y

The bootstrap.py script as per the installation instructions, with defaults, essentially reduces to creating a venv, cloning a repo and running the installer:

python3 -m venv /opt/tljh/hub
/opt/tljh/hub/bin/pip install --upgrade git+https://github.com/jupyterhub/the-littlest-jupyterhub.git
/opt/tljh/hub/bin/python -m tljh.installer --admin admin

When this is finished, you can visit your local IP address (found with, for example, ip a) and you should see the Jupyter login page. It'll probably warn you that this is being served over HTTP rather than HTTPS - and that's something we can deal with later. Note that by default, the Jupyterhub Service is running on ports 80 and 8443 (HTTP/HTTPS), but if you visit the IP address locally you should be mapped to the right place. You can login with the admin username we set above (here, admin), and choose a password to associate with this account on first login.

Setting up Default Authentication

Here we're going to modify the default authentication method (which allows anyone to sign up, using a username and password of their choice at first login). We'll instead stop this method from being able to create new users, so essentially whitelisting usernames.

This allows whitelisted users to create a password at first login, but it won't automatically create a user for any random person trying to access your server; this means you'll need to add a username for anyone wanting to access it (or a set of usernames which people can take one of, which is what I did for the workshop). This strikes a bit of a balance in terms of security - while the presence of e.g. an admin user is fairly guessable, the absence of others makes it harder to gain access. As more users are added, there's a bit of a security hole before they set a password, and if they set a bad password but it's manageable for e.g. workshops (the intended use case of exposing this, where you do).

We can modify the YAML configuration file at /opt/tljh/config/config.yaml to have an authenticator set as follows, then reload:

authenticator:
  type: firstuseauthenticator.FirstUseAuthenticator
  create_users: false

tljh-config reload

Default Environment Management

To add packages to the default environment (apt, pip or conda), you can log into the JupyterHub interface with your admin user, open a terminal and use e.g. conda install -c conda-forge <package_name> or pip install <package_name>.

For workshop-style work, consider using an environment.yml or requirements.txt you manage in git for this purpose, and use e.g. conda env update -n base -f environment.yml to update it.

Using `nbgitpuller`

If you're running a workshop with some standardised content you want everyone to have a copy of - their own copy - then nbgitpuller can come in handy. This assumes you have git installed (we do), and in this instance works through providing a specific URL to attendees, which itself includes instructions to pull the repository on login For example, in my pyrolite workshop I used this to pull a repository for each user on login, and also some extra parameters to i) open a specific file within that repository (here ./notebooks/00_overview.ipynb relative to the root of that repository), and ii) pull down the develop branch: https://{HUB_URL_OR_IP}/hub/user-redirect/git-pull?repo={REPOSITORY}&urlpath=lab%2Ftree%2F{REPOSITORY}%2Fnotebooks%2F00_overview.ipynb&branch=develop. nbgitpuller provides a link generator for you, which is pretty simple to use. Note this is a good way to distribute content, but in terms of environment and packages, see Default Enviroment Management to save doing this per-user for packages/software.

Overriding Default Jupyter Settings

For example, we can create a file called overrides.json, and add ruff as the default formatter:

{
  "jupyterlab_code_formatter:settings": {
    "preferences": {
      "default_formatter": {
        "python": "ruff"
      }
    }
  }
}

Caches for `matplotlib` and `numba`

Some packages use cache directories, and wil complain about these not existing for each user. You can set the environment variables for the spawner with an extra configuration file:

from pathlib import Path

def tmpdir(instance, key):
    return str(Path("~") / ".{}".format(key))

def mpl_usr_tmp(instance):
    return tmpdir(instance, "mpl")

def nb_usr_tmp(instance):
    return tmpdir(instance, "numba")

 
c.Spawner.environment.update( # noqa: F821 # type: ignore
    {"MPLCONFIGDIR": mpl_usr_tmp, "NUMBA_CACHE_DIR": nb_usr_tmp}
)

If you write this to a file at e.g. /opt/tljh/config/jupyterhub_config.d/usr_env_caches.py the environment variables should be loaded when the notebook server starts.

Adding Users

We've set up our server to only allow users which are explicitly added to a whitelist (i.e. already created). You can add these in bulk, separated by lines. To add users in the JupyterHub interface, you can you can go to https://<HUB_DOMAIN>/hub/admin#/add-users, and add them there.

Quickly Purging Users

If you need to quickly purge the users of your hub (e.g. I used this after the workshop), you can use this kind of setup where you obtain an API token configuration (at http://<HUB_DOMAIN_OR_IP>/hub/token):

import requests

token = '<API_TOKEN>'
api_url = 'https://<HUB_DOMAIN_OR_IP>/hub/api'

r = requests.get(api_url + '/users',
    headers={
             'Authorization': 'token %s' % token,
            "Content-Type": "application/json",
            }
    )

r.raise_for_status()
users = r.json()

matching_users = [u['name'] for u in users if not u['name'].startswith('admin')]
for u in matching_users:
    requests.delete(api_url + "/users/" + u,
         headers={
             'Authorization': 'token %s' % token,
            "Content-Type": "application/json",
            })

Adding a Tailscale Funnel for JupyterHub

To make this locally hosted app available to external users, we'll use Tailscale (and more specifically, Tailscale funnel). This only requires a Tailscale account, which is free. You'll end up with a publicly-accessible URL for your JupyterHub, which if you want you could map to a subdomain in DNS configuration for a domain you own.

If you're using an LXC, you'll need to mount /dev/net/tun to use Tailscale.

On Linux, you can install Tailscale with a basic install script:

curl -fsSL https://tailscale.com/install.sh | sh

To start a funnel providing access to JupyterHub on port 80, you can run:

tailscale funnel -bg  http://localhost:80

This will make the JupyterHub instance available on your tailnet; if you wanted, you could point the NDS record on your custom domain to this also.

Note: I did try doing this with Cloudflare tunnels, but was only half successful - I could get a tunnel to show parts of the JupyterHub interface (the home, admin and token pages) but it wouldn't load a notebook. I suspect there's either some part of JupyterHub running on different ports/protocols which I didn't see, or some setting I needed to enable to get it working. After an evening of headaches trying to make this work I switched to Tailscale and had it up after only a little fiddling.

Self-Hosted Discord Alternative: Haven

Morgan Williams — Thu, 02 Apr 2026 00:00:00 +0000

I'm not the gamer I once was, but I still manage the odd evening game. Now as before a voice chat option when gaming with friends is a necessity. While a few more modern games (and now Steam) have voice chat built into them, I generally prefer having an independent solution for this. I also mainly play old titles in any case.

I've used a variety of things for this in the past, including Ventrilo, the until-recently home of this in Discord, and I'm currently hosting Teamspeak. In the case of Discord - which I've ceased using - I tend not to use 90% of it's 'features' (many of which are to me more in the realms of social media) and instead I'm simply after some basic chat (+GIFs) and voice chat capability, screen sharing is a bonus. With Discord (among other things) moving to implement age verification, many others also seem to be intending to leave; this serves as a bit of a template for one option: host a server for your family/friends/community/group.

Looking for Open and Self-Hosted Alternatives

The first alternative I've looked at is Teamspeak, a free (but not open source) voice chat application, which as noted above I'm currently hosting on my Proxmox server. In general, this works fine for my typical use case over a local connection (playing a game with my partner, who's in another room). However, in order to invite other friends to this server, I'd need to open respective ports on my router in order for them to access the server (and probably would need a static IP). With a hosted virtual private server (VPS) this is probably something I'd go for (there are providers which specifically host Teamspeak servers if that's what you're after).

The second option I looked at for Teamspeak is using a kind of tunnel service (first Tailscale funnel and secondly Cloudflare tunnel; I use Tailscale already and have at least one domain managed in Cloudflare). However - this quickly runs into a roadblock which wasn't initially apparent (to my uninitiated self), in that Teamspeak communicates over UDP, TCP and HTTP - tunelling generally only works for HTTP/HTTPS traffic (at least without other work/conversion) - so that plan was dead in the water.

Mumble is another open source (one-up on Teamspeak) and fairly minimal text + voice chat option here, but similarly to Teamspeak it also uses UDP so doesn't fit the requirements for use via a tunnelling service.

So, looking for a voice chat application I could self host which communicates over HTTP/HTTPS, I found Haven. Generally it has the look and feel of a slightly less flashy version of Discord (this is a plus) while also retaining most of the functionality (and 100% of what I need from it). As a client, the application is accessible directly in your browser, but as a bonus for those who want it has both desktop and Android client applications.

Haven: Getting Started

Haven is MIT-licensed on GitHub, and the server component has multiple documented installation methods for Linux systems, including:

The the preferred method via Docker compose (this is arguably easier to update)
Running a server directly on a host

Accessing your Server: Networking Requirements

In terms of the networking setup to share your server with friends, Haven has multiple options. The first two don't require any manual intervention on your part, and will work with either installation method:

Use it locally, with IP addresses within your home network - works for who you live with and maybe your neighbour which has your WiFi password
Use the built-in Cloudflare or Localtunnel options - these will give you (likely randomly generated) externally accessible URLs which 'just work'. If your server is intended to be ephemeral, or you're happy to ship around a URL to friends whenever you want to jump into a game - this might work pretty well for you.

The third set of options does require some extra work, but will give you a persistent URL to access Haven at, and is possible to configure via a range of services:

With Cloudflare tunnel, if you have a domain with them
With Tailscale funnel, although arguably less straightforward to setup
With Localtunnel, if you pay for the non-free tier (from US$5/month).

We'll look at using a tunnel via Cloudflare below, and link it to the domain which this blog is hosted on.

While this is largely focused on using your own hardware (particularly if you have an old computer lying around and a need for these kinds of basic services) - but f you're happy to pay for it, a Virtual Private Server (VSP) is also a good option - you'll likely get relatively predictable monthly pricing, and this can be simpler and in some ways more secure than hosting something on your own machine. You won't necessarily need a domain (although you could map a domain to your VPS if you wanted) and while the networking setup likely requires a little configuration it's probably a well documented thing for whatever service you're looking at using. Note above the limitations of having simultaneous usage of HTTP/HTTPS, TCP and UDP in an application would be possible in this scenario (you could host Teamspeak or Mumble if voice chat is the core part you're after) - and you could also just use a provider which hosts specific services for you, as I'd mentioned above for Teamspeak.

Installation

In this instance I'm going to start with a Debian 12 LXC image, running on Proxmox. We can quickly update it after starting:

apt update && apt upgrade -y

Note: I don't use sudo here as I'm running as root for the majority of important tasks within the container, but on your device or in your container you might need it.

In either scenario, we'll use the git repository as a starting point:

git clone https://github.com/ancsemi/Haven.git
cd Haven

Option 1: Docker

Haven hosts a docker-compose.yml in their repository, and a pre-built image is available to save building the repository from source; this image is used by default.

To get setup with Docker, we can first install the Docker engine (if you don't already have Docker in your environment):

Docker Installation

We'll follow the Docker instructions on doing this in a Debian-based system using apt but I've added the (March 2026) commands for doing this below.

The first step is to remove any existing Docker components:

apt remove $(dpkg --get-selections docker.io docker-compose docker-doc podman-docker containerd runc | cut -f1)

After that we add Docker's GPG key:

apt install curl -y
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

And from here modify the apt sources to add the Docker repository:

tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

Now we can install the Docker packages:

apt update
apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

And check the status of Docker with (Ctrl+C /:q to exit):

systemctl status docker

From here, assuming we're in the Haven repository, we can use Docker compose to pull the images and start the container in detached mode:

docker compose up -d

I've added the default YAML compose file below for reference, if you're curious:

docker-compose.yml

# ── Haven Docker Compose ─────────────────────────────────
# Quick start:  docker compose up -d
# Stop:         docker compose down
# View logs:    docker compose logs -f haven
# Update:       docker compose pull && docker compose up -d --force-recreate
# Shell:        docker compose exec haven sh
# ─────────────────────────────────────────────────────────

services:
  haven:
    image: ghcr.io/ancsemi/haven:latest
    # build: .                    # Uncomment to build from source instead of using the pre-built image
    container_name: haven
    ports:
      - "${PORT:-3000}:${PORT:-3000}" # Main HTTPS port
      - "${REDIRECT_PORT:-3001}:${REDIRECT_PORT:-3001}" # HTTP → HTTPS redirect
    volumes:
      - haven_data:/data
      # ── Or use a local folder instead (good for NAS / Synology): ──
      # - ./haven-data:/data
    environment:
      - PORT=${PORT:-3000}
      - HOST=0.0.0.0
      # - SERVER_NAME=My Haven           # Name shown in the UI
      # - ADMIN_USERNAME=admin            # First account to register becomes admin
      # - GIPHY_API_KEY=                  # Optional: enable GIF search
      # - VAPID_EMAIL=mailto:you@example  # Optional: email for push notifications
      # - TURN_URL=turn:your-server:3478  # Optional: TURN relay for voice over internet
      # - TURN_SECRET=your-secret         # Shared secret (coturn --use-auth-secret)
    # Uncomment to load settings from a .env file:
    # env_file: .env
    restart: unless-stopped

volumes:
  haven_data:

After this completes, your service should now be running on https://<LOCAL_IP_ADDRESS>:3000. To check the IP address of your machine, you can use the following to list all of your network interfaces:

ip a

If you know you have e.g. an ethernet interface on eth0, you can use:

ip a show eth0

You can take the address (an IPv4 address will look something like 192.168.0.127) you can visit e.g. https://192.168.0.127:3000 to access your Haven server; note you'll probably have a 'Be careful. Something doesn’t look right.' kind of warning message the first time you access this, you can click 'Advanced' -> 'Proceed'. You should now see a login/register page for Haven. You can register with the username admin in order to have access to server admin functions (you can change the display name later). If you don't need a persistent tunnel for external access, you're now done, congrats! Have a look through setting up your server, and if you're going to use the built-in tunnel functionality, at least consider whitelisting the usernames who are able to register.

Updating

In terms of updating, it's typically going to look like:

docker compose pull
docker compose up -d --force-recreate

Option 2: Server Directly on the Host

If you're running this in a Linux Container (LXC), you already have one level of isolation on your host, and you don't necessarily need to use Docker. Similarly, if this e.g. the only thing running on a small machine, you could install it directly on the host.

To use the server, the main thing we'll need is

apt install nodejs npm

To run the server, from the repository we've cloned above, we can use the start scripts provided:

chmod +x start.sh && ./start.sh

This will run until killed, so any other things we want running will need to run another way.

Tunnel Setup

Once this is up and looks OK locally, we can add the route in Cloudflare by configuring your tunnel in the ... menu, and then clicking Add Route in the Routes diagram. You'll need to select your subdomain (I used hvn), and add the service URL (Haven uses https://localhost:3000).

We will need to create a tunnel on Cloudflare, and get a tunnel token. Once you've created your tunnel, you can extract the token from one of the installation options which Cloudflare gives you (copy paste and take the long text token at the end).

¹
One of the key things here which otherwise tripped me up was that the setup for Haven requires a configuration change in Cloudflare Zero Trust to turn off TLS verification (you might need to make an Org to get in; the basics of this are still free-tier). For Docker, I'd tried using --no-tls-verify in the run command and NO_TLS_VERIFY=true in the environment but neither helped, and I was relegated to the Cloudflare dashboard.
You'll need to select your tunnel in the page linked above, select the 'published application routes' tab, and then click on the '⋮' menu, select 'Edit', go down to 'Additional application settings', select 'TLS' and turn on 'No TLS Verify'. Save this, and with luck you can access your server via your subdomain added earlier (e.g. https://hvn.fluids.rocks).

If you want to see what this might look like with Tailscale funnel (would require a Tailscale account but otherwise free and doesn't require you to own a domain, let alone one in Cloudflare), let me know and I'll see what I can put together!

Tunnel on the Host

To use cloudflared with an installation of Haven on the host, we can pull down the binary from GitHub:

curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o /usr/local/bin/cloudflared
chmod +x /usr/local/bin/cloudflared

We can verify this is installed:

cloudflared --version

You can run the binary itself to create a tunnel (e.g. without a token this would create an anonymous/ randomized url), but we'll set it up as a service using our tunnel token:

cloudflared service install <TUNNEL_TOKEN>

Once installed, we can start the cloudflared service:

systemctl start cloudflared

If any of the configuration (IP addresses, the TLS settings etc) changes, it's best to restart the service:

systemctl restart cloudflared

From here, we would then run the server:

chmod +x start.sh && ./start.sh

Tunnel in Docker

If you want to use Docker but want a peristent tunnel URL, we can modify the docker-compose.yml file to add a cloudflared service which provides the tunnel to the networks. This requires:

A Cloudflare account
A domain managed by Cloudflare which you can attach this to (i.e., the persistent part)

Before we do anything else, we can quickly take down our Docker services:

docker compose down

We'll insert our tunnel token in place of <TUNNEL_TOKEN> in the below YAML file:

docker-compose-cloudflared.yml

# ── Haven Docker Compose ─────────────────────────────────
# Quick start:  docker compose up -f docker-compose-cloudflared.yml -d
# Stop:         docker compose down
# View logs:    docker compose logs -f haven
# Update:       docker compose -f docker-compose-cloudflared.yml pull && docker compose up -f docker-compose-cloudflared.yml -d --force-recreate
# Shell:        docker compose exec haven sh
# ─────────────────────────────────────────────────────────

services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    command: tunnel --no-autoupdate run
    environment:
      - TUNNEL_TOKEN=<TUNNEL_TOKEN>
    ports:
      - "${PORT:-3000}:${PORT:-3000}" # Main HTTPS port
      - "${REDIRECT_PORT:-3001}:${REDIRECT_PORT:-3001}" # HTTP → HTTPS redirect
  haven:
    image: ghcr.io/ancsemi/haven:latest
    # build: .                    # Uncomment to build from source instead of using the pre-built image
    container_name: haven
    #ports:
    #  - "${PORT:-3000}:${PORT:-3000}"    # Main HTTPS port
    #  - "${REDIRECT_PORT:-3001}:${REDIRECT_PORT:-3001}"  # HTTP → HTTPS redirect
    volumes:
      - haven_data:/data
      # ── Or use a local folder instead (good for NAS / Synology): ──
      # - ./haven-data:/data
    environment:
      - PORT=${PORT:-3000}
      - HOST=0.0.0.0
      # - FORCE_HTTP=true
      # - SERVER_NAME=My Haven           # Name shown in the UI
      # - ADMIN_USERNAME=admin            # First account to register becomes admin
      # - GIPHY_API_KEY=                  # Optional: enable GIF search
      # - VAPID_EMAIL=mailto:you@example  # Optional: email for push notifications
      # - TURN_URL=turn:your-server:3478  # Optional: TURN relay for voice over internet
      # - TURN_SECRET=your-secret         # Shared secret (coturn --use-auth-secret)
    # Uncomment to load settings from a .env file:
    # env_file: .env
    restart: unless-stopped
    network_mode: service:cloudflared
volumes:
  haven_data:

In this file we've added a bit of network configuration: for the cloudflared service we've mapped the ports 3000 and 3001 (used by Haven) and in the haven service used network_mode: service:cloudflared. The port mapping allows local IP access (if you don't need it, you can remove this), and the network_mode: service:cloudflared ensures that Haven can use our tunnel. Note that we're not mapping ports in the haven service here (the port mapping has just moved up).

There may be some secret sauce which means we don't need to do the no-tls-verify faff above¹, but I haven't found it yet (tried the PORT variable, but I had no luck with this so far) .

We can copy the contents of the file above, and paste it to a file called docker-compose-cloudflared.yml:

nano docker-compose-cloudflared.yml

From here, we can start up our modified services:

docker compose -f docker-compose-cloudflared.yml up -d --force-recreate

Before we go any further, if you've kept the port mapping it's best to check that you can access your service at your local IP address (e.g. https://192.168.0.127:3000, but using the IP address of your machine). If there's an issue already here, we might not succeed in the tunnel bit.

Whitelisting Users in Haven

By default, your Haven instance will allow registrations from anyone who accesses the server. This may not be what you're after. If you have a group of friends you're looking to host a server for, you can do a call around and get a list of usernames to whitelist in Haven, which marginally improves your security situation (random folks who find your server won't be able to sign up without using one of these usernames).

If you're logged in as the/an admin user, the configuration for this is currently under Settings -> Whitelist. You'll need to both i) turn on whitelisting with the 'enabled' toggle and ii) add the users you're allowing to sign up.

You can also use invite codes, and set channels to invite-only/private, if you're planning to use multiple for different purposes/allow different folk to do and see different things.

Zola + Codeberg Pages: Deploying a Blog

Morgan Williams — Sun, 29 Mar 2026 00:00:00 +0000

Codeberg Pages

Similar to GitHub pages (which this site was originally deployed on), Codeberg has 'Codeberg pages' which uses a simliar but open technology to deploy the content of a git repository as a webpage. Similar to GitHub pages which allows you to have one uncomplicated URL tied to your repository (e.g., {username}.github.io), Codeberg does the same (morganjwilliams.codeberg.page). It has a few prerequisites:

You'll need a public repository called pages (for the variant tied to your username; you can also set up Codeberg pages for other repositories).
Within this, you'll need a branch called pages.
Beyond that you'll need to setup a webhook in settings.

Moving to Zola

This site was originally built in Jekyll (a static site generator using Ruby), and I'm now using Zola (a static site generator built in Rust). Both build HTML sites from templated markdown, Zola's a bit quicker, and for the time being I find it less complex to work with.

Installing Zola

If you have Rust and cargo installed, you can build Zola from source (cargo install --locked --git https://github.com/getzola/zola), but there's a long list of binary/executable install methods on Zola's installation page. There are also prebuilt binaries in GitHub, should you wish to just use one (maybe do/don't add it to your git repo?).

Get a Theme

Like Jekyll, Zola has a decent number of themes you can use to get a leg up on styling your page (or, leave it to someone with the interest, and find something which fits your vibe). These inform both the structure and styling of the page, and while a bit of a learning curve, the main bits will likely be documented.

Looking to simplify a little from where I'd been (and the components I'd modified/made with little real understanding of what I was doing) with Jekyll, I went with serene.

Note: Themes in Zola are typically best managed as updatable git submodules; this isn't something everyone deals with every day, but it's something to consider in your build and deploy workflows.

Building Your Site

Zola has some simple commands for building your site. By default, zola build will build your site and put the contents in ./public/. zola serve will give you a live preview of content as you edit it on a local webserver. In terms of deployment, you'll typically want to run zola build and deploy the rendered content within ./public.

Without CI Workflows

Unlike GitHub, 'actions' in Codeberg are a little less plug and play, and building locally and just pushing rendered content is a viable (but repetitive) option. This is how I started, using a two-parallel-branch strategy in git - a main branch where 'structure' lives (what I build from) and a 'pages' branch where I push updates to content (HTML, mostly) which then gets rendered on Codeberg pages. I wrote a simple batch script to automate the 'build, switch branches, move things and stage changes on the pages branch' workflow.

Local Tools (of Choice/Habit)

I'm still on Windows for my main personal PC now, so it's good old cmd and/or Powershell for a terminal.
I use VS Code for work, and it's free, so in terms of editors, that's where it's at. I just find all the buttons to turn off anything referencing AI.
I use GitKraken as a git GUI to save my brain a little with some git management. But, having habitually run into authentication issues as 90% of the repositories I work on are private/not feeling the need to bake in credentials into this tool, I typically push to/pull from remotes from the command line. Also turned off all the AI features here.

Using CI Workflows

Codeberg does have hosted solutions for basic workflows accessible after enabling them in a few settings (alongside their 'Woodpecker CI' solution which requires approval). I've since set up self-hosted runners for this purpose, see the post covering that here.

The action I've set up for deploying the page currently uses a self-hosted runner, builds Zola from source, builds the site, and pushes the changes to the pages branch in a simiiar fashion to how I'd done it locally prior to this.

Note that the caching of Zola/cargo is a work in progress (it'll need some modifications to work...), but this should minimise the computational load until a new version of Zola is requested in the workflow. I've set up caching of the Zola cargo build, which makes the workflow about 6x faster; when a new Zola version is desired I can just bump an environment variable and the cache should refresh.

`serene` Reactions

I'm not tracking you when you visit here, and as it's a static site you'll not have a great deal of interactivity here, and I won't know you've been (feel free to ping me on Mastodon @metasomite though!), making it feel very 'shout into the ether'. To slightly remedy this and give an option for anonymously noting 'I was here', I've followed serene's guide to adding a basic reactions endpoint using a basic Cloudflare database and a hundred or so lines of Typescript defining the endpoint to allow each post to have reactions at the bottom. If I'm missing one you'd rather use, let me know...

Note: As I have cargo installed locally (will get to talking about Rust one day), I used fnm to install Node; note the shell setup step if you want to follow along, in my case using Powershell.

Using a Custom Domain

If you don't want to have {username}.codeberg.page as your domain, you can fork out for a custom domain from one of the many registrars. To use a custom domain to host your site on Codeberg Pages, at the root of your pages branch of your repository, you'll need a file called .domains containing your domain at the top.

Note that the workflow contained in the Zola instructions for getting set up on Codeberg Pages contains a step which would remove the .domains (git rm -r "*" || true) and return an associated error in your page when deployed; in my deployment workflow I've modified this to refer to specific filetypes/avoid the . files with git rm -r *.html *.json *.xml *.txt *.js *.css *.jpg *.png || true - YMMV.

You will also need to set up some custom DNS (I needed this variant).

Setting Up Self-Hosted Forgejo Runners

Morgan Williams — Sat, 28 Mar 2026 00:00:00 +0000

In an earlier post I mentioned that Codeberg actions was less 'plug and play' than GitHub actions, and it is. But it's not quite as difficult to get going as I'd thought, and regarding setting up self-hosted runners for Codeberg actions went from 'not today' to 'in a few days' pretty quickly.

First, you need to turn on actions in a Codeberg repository, at https://codeberg.org/{username}/{repo}/settings/units you'll need to select 'Actions' to show the tab in the repo and also configure runners.

Second, hosted runners are now a thing on Codeberg, but the resources are limited and I'd rather host my own, learn a bunch in the process, and let these be used by those who don't have another option, those who don't have the time to do what I've done here, and those who are otherwise doing better things for the world than publishing this blog.

Third, you can run self-hosted Forgejo Runners which are not publicly accessible/don't have a public/static IP address. This means it's much more convenient to do from a home lab scenario.

Fourth, I found a Proxmox helper script which enabled me to get up and running quickly (and subsequently enter a cycle of breaking things, figuring it out and starting again for a day), which gave me sufficient confidence to try and make it work. This is the debrief, and a place to add further notes as they arise.

Proxmox

I run all of my self-hosted services in Proxmox VE, an open-source virtualization platform with a web-based GUI which you can run for free. I use this to enable one machine to serve many purposes flexibly - and I run it from a desktop-format server sitting in my office, making good use of RAM purchased before /ai ruined the personal computing market.

Some of the information below is specific to using Proxmox, but beyond setting up a container to run things in, a lot of it will be transferrable (e.g. if you set this up in a VM, on another virtualization host, or set up the runner to run directly on a host like an old computer you have sitting around which you don't necessarily want to run anything else on).

You can create runners which accept workflows for different scopes (all repositories - at least of a Forgejo instance you run yourself, an organization, all repositories of a your user, or a specific repository), and this comes into play when the runner is registered with Codeberg (or an other Forgejo instance).

Proxmox Helper Scripts

One useful aspect of running services on Proxmox which makes some of it's technical features a little easier to handle is the Proxmox Community Helper Scripts page which provides pre-bakend solutions for getting common (and some less common) services up and running under Proxmox. It doesn't have everything, and it's worth checking what the setup scripts are actually doing, but generally this is a good way to try setting something up, and judge whether you want to put effort into doing it properly/maintaining it.

As of March 2026, there's a Proxmox Helper Script for Forgejo Runner, but it's noted as being 'under development. I have successfully used this to set up a runner, but will also go through the detail of setting it up in a LXC template below (this uses Debian but it could easily be adapted to similar distributions).

Setting Up a Forgejo Runner in a Debian LXC

In Proxmox, I typically use LXC's (Linux containers) to run services in. These provide a simple layer of isolation, but enable flexible usage of resources on my server as i) the kernel is shared and the templates are small, ii) resources allocated to the LXC aren't reserved specifically but instead provide more of a cap on usage (as opposed to fully fledged virtual machines).

Creating a Debian LXC

In Proxmox, creating an LXC from a template involves:

Selecting your Proxmox node and clicking 'Create CT' is a blue button on the top right, or you can right-click on your node and select 'Create CT'
Selecting a hostname (what your runner will be called, locally and in Codeberg), and adding a password (containers are by-default password protected in Proxmox)
Selecting your template (different templates can be stored on different storage volumes).

If the options you see here don't cover Debian (or whatever else you want to use), you'll need to browse to that storage volume in your Proxmox node, select 'CT Templates' from the menu, and browse 'Templates' for what you're after (this is a Proxmox-maintained list of templates). If you're after something not in this list, you can also manually upload your own from an external source.
Configuring template resources and options

For this exercise, I used the debian-13 template, with 4 cores and 8GB RAM. Make sure you configure the network to use DHCP/SLAAC unless you're happy to configure static IPs within your local network for this container. Don't run the container after creating it, instead modify the options to turn on nesting and keyctl (will be needed to run the Docker engine).

After you have an LXC, you'll want to open a console, and we'll get started.

Initial Setup

The first step in this is to quickly update your container, and add some basic dependencies.

apt update && apt upgrade -y
apt install curl jq gpg ca-certificates

Note: I don't use sudo here as I'm running as root for the majority of important tasks within the container, but on your device or in your container you might need it.

Install Docker

The next step in the workflow is to install the Docker engine. We'll follow the Docker instructions on doing this in a Debian-based system using apt but I've added the (March 2026) commands for doing this below.

The first step is to remove any existing Docker components:

apt remove $(dpkg --get-selections docker.io docker-compose docker-doc podman-docker containerd runc | cut -f1)

After that we add Docker's GPG key:

install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

And from here modify the apt sources to add the Docker repository:

tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

Now we can install the Docker packages:

apt update
apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

And check the status of Docker with (Ctrl+C /:q to exit):

systemctl status docker

Forgejo Runner Binary

We'll use a binary installation for the Forgejo Runner in our container; I've added the (March 2026) commands for this below.

The first part of this is just selecting the right binary for your system:

export ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
export RUNNER_VERSION=$(curl -X 'GET' https://data.forgejo.org/api/v1/repos/forgejo/runner/releases/latest | jq .name -r | cut -c 2-)
export FORGEJO_URL="https://code.forgejo.org/forgejo/runner/releases/download/v${RUNNER_VERSION}/forgejo-runner-${RUNNER_VERSION}-linux-${ARCH}"

We can then pull the runner binary from this URL, make it executable, and verify the GPG key:

wget -O forgejo-runner ${FORGEJO_URL} || curl -o forgejo-runner ${FORGEJO_URL}
chmod +x forgejo-runner
wget -O forgejo-runner.asc ${FORGEJO_URL}.asc || curl -o forgejo-runner.asc ${FORGEJO_URL}.asc
gpg --keyserver hkps://keys.openpgp.org --recv EB114F5E6C0DC2BCDD183550A4B61A2DC5923710
gpg --verify forgejo-runner.asc forgejo-runner && echo "✓ Verified" || echo "✗ Failed"

From here, we copy the binary into /usr/local/bin (we'll use this later).

cp forgejo-runner /usr/local/bin/forgejo-runner

The Forgejo Runner installation guide also suggests creating a user for the runner; as we're using Docker they'll need to be added to the docker group too:

useradd --create-home runner
usermod -aG docker runner

Register the Runner

Now we have a runner, we need to register it with Codeberg so we can use it in Codeberg Actions. The Forgejo Runner installation instructions suggest doing this with the runner user we created above, so we'll switch to that user and switch into their home directory.

su runner
cd ~

At this point, if you want to create a configuration file, it's a convenient time. You can create it with the forgejo-runner binary and edit it with nano:

forgejo-runner generate-config > config.yml
nano config.yml

The two things I edited here (as of this post) were the runner.labels (I'll use "rust-latest:docker://rust:latest" for Zola, and I'm sure I can find uses for the others) and the container.docker_host. The labels here are important in that they indicate what can be run on the runner. Note that the labels have both a friendly name (e.g. ubuntu-latest together with the identifier for the image to use, in this instance docker://node:20-bookworm).

    labels: [  
    "ubuntu-latest:docker://node:20-bookworm",
    "ubuntu-22.04:docker://node:20-bookworm",
    "python-latest:docker://python:latest", 
    "rust-latest:docker://rust:latest"
          ]
    ...
    docker_host: "automount"

Note: You can use a runner which isn't Docker-based, e.g. using Podman or LXC images; these involve a few different/extra steps see the installation guide for more.

At this point, we also need to get a registration token from Codeberg (or other Forgejo instance) in order to connect to it. Make sure you've added the 'Actions' unit (at at https://codeberg.org/{username}/{repo}/settings/units, if you don't have it), and after that you should be able to find the 'Create new runner' button at https://codeberg.org/{username}/{repo}/settings/actions/runners which will create a new runner specifically scoped for this repository (you can get a registration token scoped to your user, but you'll need to get it from https://codeberg.org/user/settings/actions/runners). Clicking this button will give you a registration token to copy, which you'll need below - alongside the instance URL (i.e. https://codeberg.org for me).

Then we can register the runner using this configuration, this will give you some prompts to reply to:

forgejo-runner register --config config.yml

For my setup, the prompt replies to this were (kept the hostname, already had the labels in config):

> https://codeberg.org
> {registration token}
>
>

Almost there!

Start the Runner, and Keep it Running

You can can run the daemon with your specific configuration as follows:

forgejo-runner daemon --config config.yml

However, to have this start automatically with your container, you can add it as a systemd service. You'll need do do this as root (or otherwise give runner sudo access). Here we change back to the root user, and open a new file to edit with nano; the Forgejo Runner installation instructions point to adding/copy-pasting in this file - note that I needed to edit line 7 which refers to the configuration (replace /home/runner/runner-config.yml with /home/runner/config.yml which we created above):

su root
nano /etc/systemd/system/forgejo-runner.service

My file, after the edit to keep it in line with the file naming above:

[Unit]
Description=Forgejo Runner
Documentation=https://forgejo.org/docs/latest/admin/actions/
After=docker.service

[Service]
ExecStart=/usr/local/bin/forgejo-runner daemon -c /home/runner/config.yml # I needed to edit this line
ExecReload=/bin/kill -s HUP $MAINPID

# This user and working directory must already exist
User=runner
WorkingDirectory=/home/runner
Restart=on-failure
TimeoutSec=0
RestartSec=10

[Install]
WantedBy=multi-user.target

From here we can reload the daemon, start the service to check it's working, and from there enable the systemd service for the Forgejo Runner.

systemctl daemon-reload
systemctl start forgejo-runner.service
systemctl enable forgejo-runner.service

Back to Codeberg

In Codeberg, you should see your new runner in the list at https://codeberg.org/{username}/{repository}/settings/actions/runners- and with luck it'll have a green 'idle' status alongside the name you gave it (the hostname if you left as the default, otherwise the name you configured for it in the section above).

Now you can create actions which are triggered by Codeberg but run on your self-hosted Forgejo Runner. In your repo, you can create YAML file defining your workflow (e.g. .forgejo/workflows/build.yaml), and indicate what you want it to run on, e.g.:

jobs:
  publish:
    runs-on: rust-latest

How This Site Was Built (Again)

Morgan Williams — Wed, 25 Mar 2026 00:00:00 +0000

I rebuilt the same site, ten years later!

I'll save on some of the details for now, but there are some notes below on what's changed.

Moving to Codeberg

The world has changed a lot in 10 years, and Microsoft acquired GitHub since this was initially built. I'm looking to move back into more open spaces, and Codeberg seems to suit.

Codeberg Pages and Zola

Instead of GitHub pages and Jekyll, I'm now using Codeberg Pages and Zola - you can see a post on that here.

Workflows: Self-Hosted Forgejo Runners

I've set up self-hosted runners for my build workflows, and put together a basic guide for it.

Building for Humans

A notable addition relative to the old site is that one implements the human.json protocol (you can find pages of other humans here). I've also added a page detailing how I don't use generative AI here (or elsewhere): /ai.

PhD Research

Morgan Williams — Fri, 01 Mar 2019 00:00:00 +0000

During my PhD at the Australian National University, I investigated how oceanic rocks change when interacting with fluids (water, mostly) throughout the lifetime of the oceanic plate, from its inception near mid-oceanic ridges to its destruction in subduction zones. The chemical changes (especially hydration/dehydration) that happen in these environments are the key to global chemical recycling and generation of new continental material, and ultimately the plate tectonic system. Exploring the what, when, how and why of these processes is central to understanding the evolution of our planet.

More specifically, I'm was working on 'fingerprinting' different processes and fluid compositions that rocks encounter, and provide a better framework to interpret rocks that aren't as well preserved. To get the chemical 'fingerprints', I sampled small parts of minerals (using lasers and ion beams) and measure the composition of the material. My core projects utilized the ratios of the light stable isotopes of boron (¹⁰B, ¹¹B) and oxygen (¹⁶O, ¹⁸O) within hydrous minerals. The relative isotope ratios in fluids and minerals change in response to both environmental variables (temperature, pH) and the initial fluid and rock isotope compositions; when minerals (or zones within minerals) are preserved, these isotope systems can be used as efficient tracers/proxies for events that happened millions of years ago.

In most metamorphic systems, a number of common rock-forming minerals grow in the presence of fluids. Growth can occur over relatively short time periods, and successive periods of mineral growth can result in zoned patterns (like tree rings). These preserving a time-sequence which allows the extraction of information on how systems evolve. By combining a suite of chemical systems, including different element and isotope systems, we can accrue further information and better constrain how (and sometimes even why) these geological systems change.

Investigating Geochemical Records of Fluids in the Oceanic Crust

My research focused on the use of in-situ analytical methods (principally SIMS and LA-ICP-MS) to investigate the geochemical systematics of fluid-rock interaction from seafloor hydrothermal systems, through subduction and during ultra-high-pressure metamorphism.

Lago di Cignana, NW Italy

To better constrain processes active in the uppermost subducting slab, I'm investigatedthe boron and oxygen isotope systematics in metasediments, metabasalts and serpentinties from the Lago di Cignana Unit and underlying Zermatt-Saas Ophiolite (NW Italian Alps). Here we hoped to constrain the timing and extent of fluid infiltration within the unit, and better understand the fluid-mediated interactions between common lithologies in the slab. For this purpose, I developed standard operating routines for boron isotope analysis by SHRIMP, with the assistance of the technical staff here at RSES.

The Atlantis Massif, 30°N, Mid Atlantic Ridge

As part of the International Ocean Discovery Program (IODP) Expedition 357 - Atlantis Massif: Serpentinization and Life and in collaboration with Daniela Rubatto and Mark Kendrick, I'm also worked on serpentinized ultramafic rocks within an active seafloor hydrothermal system adjacent to the Mid-Altantic Ridge. We constrained previously active fluid-rock interaction processes associated with the oceanic detachment fault using halogen, noble gas and in-situ oxygen isotope analysis.

pyrolite

Morgan Williams — Fri, 01 Mar 2019 00:00:00 +0000

I'm currently building an open-source python package for working with geochemical data, pyrolite (pip install pyrolite). It's currently in beta, and you can find the source on GitHub, and documentation on Read the Docs.

For a quick overview, see the poster I put together for the Dorothy Hill Women in Earth Sciences Symposium.

Serpentine Nights

Morgan Williams — Thu, 16 Mar 2017 00:00:00 +0000

Note This post is an excerpt from an article I co-wrote with Joëlle Ducommun for the European Association of Geochemistry (EAG), which provided financial support enabling us to attend. The original article can be found here. We thank the EAG for the financial support which enabled us to attend Serpentine Days, and also the organisers of the workshop for their efforts in putting together a very successful and thoroughly enjoyable meeting.

Group photo during the Serpentine Days field trip to the Lherz Massif, Southern France (Image Credit: Joëlle Ducommun).

This past September fellow ANU PhD student Joëlle and I had the privilege to attend Serpentine Days, a biannual workshop on serpentinite and serpentinization processes which is distinctly multidisciplinary. The workshop covers much of the active research on serpentinites, including everything from microbiological communities in near surface environments, through serpentinite dehydration at great depth within subduction zones and all the way out to investigations of the surface of mars. Serpentinization is one of the key processes controlling a number of global geochemical cycles, being the carrier of the bulk of bound water into subduction zones and with further relevance for carbon cycling. More recently, the discovery of microbial communities associated with serpentinite-hosted hydrothermal systems has spurred research into these environments as life-supporting habitats.

This year the Serpentine Days workshop was held at Sète. Located approximately half an hour south of Montpellier, France and is situated on the Mediterranean Sea, the town is famous for its seafood and as a seaside destination for all ages. Despite the lack of any observable serpentinite, it made for an idyllic location to host a workshop. Our venue and accommodation for the workshop was Le Lazaret, a holiday village just off the beach. Each day of the workshop presented opportunities to network and discuss with other researchers; for a few days we were able to live and breathe science, all the while enjoying the delights of southern France. A field trip to the Pyrenees followed the completion of the workshop, visiting the hallowed Lherz peridotite (of lherzolite fame) and nearby serpentinite locality near Bestiac to investigate records of mantle exhumation during the Cretaceous rifting event in the North Pyrenean Zone.

A few days in the French Pyrenees. a) Introduction to the field trip to the Lherzolite body led by Yves Lagabrielle, here pointing at the sedimentary breccias closely associated with the Ultramafic bodies in the Etang de Lherz area; b) collecting samples in the field, from left to right: Emily H.G. Cooperdock (University of Texas, Austin), Suzanne Picazo (University of Lausanne) and Joëlle Ducommun (Australian National University, Canberra). Image Credit: Joëlle Ducommun.

A Few Notes of My Own

During the workshop I presented a brief overview of my current research on serpentinites from the Atlantis Massif (which I’m completing as a science party member of IODP Expedition 357), and in particular how I’m investigating fluid-mobile/volatile elements and using in-situ techniques to provide greater constraints on the serpentinization process and evolution of the Massif. The novel aspects I discussed during this talk included microscale oxygen isotopic heterogeneities recording multistage serpentinization at generally elevated temperatures, and also atypically low serpentinite noble gas abundances. Some of the many talks I found intriguing covered the potential incorporation of sulfate within the serpentine structure (Baptiste Debret), the formation of organic films/gels at mineral interfaces (Bénédicte Menez), and unravelling alteration histories in ultramafic rocks using magnetite U-Th/He dating (Emily Cooperdock). In addition to hearing talks directly relevant to my own work, I was able to satisfy my inner child and interact with scientists studying extreme environments both here on Earth and on Mars. This led to some great conversations, particularly regarding the formation and evolution of early life. I was also grateful to have the opportunity to catch up with fellow Expedition 357 scientists and discuss research to date, and potential for collaboration both within and beyond the science party.

Field trip attendees take in views of the French Pyrenees on the way to the next field stop at Bestiac.

A Few Thoughts from Joëlle

The first international workshop I was privileged to attend was one of the most memorable experience I have had in my research journey so far. It was exactly what I needed at the time. Of course, I could start by mentioning that it was slightly intimidating to stand up in front of an expert audience, not feeling very much like a serpentinite expert myself, to present a research project on the quest of a noble gases and halogens surface signature from the Archaean. At the time, I had been working on my PhD project at the Australian National University for just over a year, and was starting to dig deeper into my main project focusing on the fate of volatiles during alteration of the oceanic crust. The workshop pushed me to take a step back to these remarkable Archaean samples that I had the opportunity to work with earlier on in my PhD. I am still astonished by the fact that these serpentinites, after surviving a 3.7 Ga year long history, sometimes still preserving their original textures, were eventually brought back all the way from Greenland to end up on my desk, offering an invaluable window into the early earth.

The workshop gave me the opportunity to gain experience in presenting my research, but mostly it helped me recover the enthusiasm and faith in science. As for many of my fellow PhD friends, I soon realized that during the ebb and flow of PhD life, it appears to be common to go through a dark tunnel. Doubts and questions take hold of the mind; we start to be sceptical about the journey we have embarked on and the relevance of the small details we are trying to understand. But being able to meet peers and colleagues, exchange ideas, friendships, good times and mostly lots of laughter, made me realized that science is very attractive when it is shared, that communicating and exchanging is what makes science, and that put together, the small details we are all tackling are part of a much broader and comprehensive story. At times, I admit, I have felt caught up in a system that lost its soul, a system that pushes students and academics to focus on personal objectives, to advance their careers, a system I had much trouble to identify with. The workshop was a perfect reminder that behind the aggressive and competitive academic life, there is still room for a science community, room to build something greater that goes far beyond each individual personal goal. I am very thankful to have been given the opportunity to meet whole hearted scientists passionate about their work but mostly about passing on their knowledge and their experiences, and I am very much looking forward to crossing their paths again.

How This Site Was Built (the first time)

Morgan Williams — Sat, 16 Apr 2016 00:00:00 +0000

You know that feeling when you find a website that just works... and doesn't look like it was made in the 1990's?

Don't get me wrong, I'm a 90's kid through and through, but you have to admit it wasn't exactly the height of design, and well-designed easy-to-navigate sites were virtually non-existent.

In an academic world we tend to ignore these aspects (or leave them to someone else) in lieu of simply presenting the required information - but the look and feel of the site is an aspect of communication that is too often disregarded. If we want people to read about science, we have to make it as easy as possible. Despite what you may think, this need not be particularly difficult, nor costly.

I've been curious about how it would be to create a net-worth personal webpage. As I usually do with these things, I gave it a shot. A dash of research, a pinch of frustration and a touch of tweaking here and there.. et voilà! Along the way I have aggregated a few tips from a variety of sources, which I've summarised below. Consider this a how-to for building a simple personal static website on a budget ($0-10/year).

How This Site Runs

Before You Start

Before you embark on this quest, I'd recommend getting a nice text editor to create and customize your site's files offline (I'm currently writing this in Atom). Another option I'd recommend would be Prose.io, which connects directly to GitHub and works quite well for small editing tasks, especially if you're on the go.

While it's not completely necessary for the walk-through below, it would also be a great idea to learn a bit of HTML, CSS, Javascript and Markdown. There are some great online courses you could try (I started on CodeAcademy and have also been through a bit of freeCodeCamp).

Hosting On GitHub

This site is currently hosted free on GitHub. GitHub is used primarily as a repository for code/software, and is great for developing projects that are constantly being changed and improved. GitHub builds on the distributed versioning control system Git and provides a platform to host your project in a single place, keep track of all the changes and versions and work both effectively and collaboratively with others. You can sign up for a free account on GitHub, after which you can start enjoying some of these benefits (perhaps even for your thesis!?). Make sure to check out the tutorial on the website if you're new to Git.

If you're a student or involved in education I'd encourage you to check out the GitHub education website. You can apply for a micro/personal account on GitHub (normally $7/month), with which you can create private repositories should you wish to keep the website files out of sight (e.g. draft posts etc). Their 'Education Pack' also includes a bunch of useful tools, and a nifty offer from namecheap for a free .me top-level domain for a year (here's your $0 website, folks).

One way you can use GitHub to host a simple website is via GitHub Pages, a straightfoward tool to host and publish sites on GitHub. Once you have the repository set up, you have two choices for creating your site:

With GitHub's Automatic Page Generator, you can can create a functional site in minutes.
You can build your own site from scratch/from a template.

I went for the second option, using a template of a site which I then customized slightly.

Building Your Site with Jekyll

One other great feature of GitHub is that it supports and encourages the use of Jekyll. In fact, Github Pages is powered by Jekyll. Jekyll is "a simple, blog aware, static site generator". It's most commonly used to build simple personal websites and blogs, but can be used to manage larger sites to great effect (e.g. GitHub Training). GitHub have also put together a quick overview of how to use Jekyll with GitHub Pages.

I find one of the most useful features to be the use of markdown to write posts and pages, rather than directly writing HTML. This saves me quite a bit of time, and simplifies the workflow for building the site. The ability to easily use relative links and liquid templating also makes life a bit easier - but a bit more on that later.

Installing Jekyll Locally

First thing's first - you'll need to install Jekyll and its dependencies. While having a Jekyll-built site on Github is great, occasionally your website builds (which happen automatically after any updates) will fail due to the odd mistype. During development of your site, I'd recommend you download Jekyll and build your sites from the terminal on your local computer. This is particularly important when you're having issues as you can catch the error messages and troubleshoot problems, and will allow you to work offline in case you live somewhere where internet (a connection, let alone speed) is not always guaranteed.

Ubuntu Installation

Firstly you'll want to install Ruby, and then use gem to install the ruby gems jekyll and bundler. Additionally, github-pages if you're likely to host on Github and rouge if you're likely to want syntax highlighting as seen below.

sudo apt-get update
sudo apt-get install ruby ruby-dev make gcc
sudo gem install jekyll bundler github-pages rouge

You should now create a project directory, and find an appropriate template if you choose that avenue: If you use a template and have a Gemfile.lock, you should be able to bundle install within your project directory to install required packages as specified by the template. Be aware that if you're using bundler you'll also be required to prefix calls to jekyll with bundle exec (e.g. bundle exec jekyll serve). To update the Gemfile.lock to check for recently updated dependencies, you can run bundle update.

Windows Installation

To install Jekyll on Windows, a relatively straightfoward method has been documented on David Burela's blog:

Download Chocolatey by entering the following into a command prompt with administrator access: @powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin
In a new command prompt with administrator access, Ruby can now be installed using the following: choco install ruby -y
Likewise, in a new command prompt with administrator access, Jekyll can be installed by entering the following: gem install jekyll

If you want to use jekyll plugins you'll likely need to install them, a few examples are: gem install jekyll-paginate jekyll-sitemap jekyll-gist jekyll-feed

Note: Another way to achieve this is using bundler (gem install bundler) and a Gemfile (if you're using a template and it comes with one!). You can bundle install in the folder containing the Gemfile, which will install the desired packages and dependencies. Note however that this puts you at the mercy of the designer, and may lead to problems if the site is used in a different way than originally intended (e.g. GitHub pages vs jekyll).

Using Templated Sites

I've built this site using a template built by Michael Rose. The easiest way to use a site template hosted on GitHub is to fork the repository, then rename it to the default .io site recognised by GitHub pages (USERNAME.github.io). Alternatively you can download the repository as a .zip file and work with it offline as described briefly above.

Note: You can only have one personal site - adding additional 'project pages' is possible, though (using a gh-pages subfolder).

Serving Up Your Page

Now that we have Jekyll installed and ready to go, you'll want to download a 'clone' of your GitHub repository. This is rather easy to do using the downloadable clients - you'll want to select clone and target your username.github.io repository. After a minute, you'll have a local copy of your site on your computer. You can make local changes - and 'save' them using Git commits, and push them to the server as well as download any updates to the remote repository using the 'sync' option. The internet has many Git tutorials if this seems confusing, and I'd encourage you to check them out to become confident using the versioning tools.

You can build your site (to the folder _site) with the jekyll serve command. This will create a local copy of your site, which is served to localhost:4000 - which you can simply use as a web address in your favourite browser. Any changes to your local files will be reflected in this dynamically-built site (they'll take a second or two to rebuild), which makes this a great option for developing offline before you go live.

Another feature that I'd recommend is using a _drafts folder (depending on how you set up you may have to create this manually). Here you can work on your posts prior to publishing them in _posts, but they can be served up with the rest of your site using jekyll serve --drafts.

Yet to Come

Customizing Your Page
- General Configuration
- Page layouts
Buying and Managing a Domain

A short list of those who provided useful material, who I wish to acknowledge here:

Michael Rose (Made Mistakes, Minimal Mistakes)
Tania Rascia (www.taniarascia.com)
Christopher Tarwater (ctarwater)
Dave Cole (DevelopmentSeed)
David Burela (BURELA'S HOUSE-O-BLOG)

IODP Expedition 357: Atlantis Massif

Morgan Williams — Sat, 16 Apr 2016 00:00:00 +0000

Over the past 6 months or so I've become involved with the International Oceanic Discovery Program's (IODP) expedition to the Atlantis Massif, on the Mid-Atlantic Ridge (MAR). In collaboration with the European Consortium for Ocean Research Drilling (ECORD), a 47-day drilling expedition aboard the Royal Research Vessel James Cook was conducted in late 2015. In January I travelled to Germany to participate in the onshore phase of the expedition, which was conducted at the Centre for Marine Environmental Sciences (MARUM) in Bremen.

Image Credit: V. Diekamp, MARUM

The Expedition

The Atlantis Massif is the surface expression of an oceanic detachment fault adjacent to the MAR at 30°N; the serpentinite-dominated footwall of the fault protrudes 4000m from the seafloor and at its summit hosts the Lost City Hydrothermal Field (LCHF). The LCHF is a low-temperature, high-pH hydrothermal vent system which hosts unique microbial communities and large carbonate chimneys. A number of expeditions have visited the site since its initial discovery in late 2000, seeking to understand the stark contrast to magmatically-driven hydrothermal systems (black smokers), the potential for abiogenetic processes and the actively serpentinizing system. Drilling on the Massif was conducted (IODP Expedition 304/305, see Hole U1309D, below) with the aim of investigating the interplay of magmatic and tectonic processes during exhumation, spurred by the newfound significance of oceanic core complexes as features of spreading centres globally. Expedition 357 aims to further investigate the exhumation of mantle rocks to the seafloor at slow-spreading ridges, the physiochemical systematics of serpentinization and the biosphere interaction within the active ultramafic-hosted hydrothermal system.

Bathymetry of the Atlantis Massif (Schoolmesters et al., 2012).

To investigate these processes, we needed to recover intact sequences of hydrothermally altered lithologies (e.g. serpentinites and talc-schist) interspersed with more robust magmatic rocks (basalts, dolerites and gabbros). Drilling in this material is difficult, and typically non-continuous cores would be recovered - most of the softer material abrades away, the surface lithologies are not preserved, and competence contrasts make recovering intact contacts difficult. One way to maximise the recovery is to use smaller diameter coring drills. For our expedition we utilized two seafloor rock drills (MeBo and Rock Drill 2, from MARUM, Germany and the British Geological Survey, UK respectively) to retrieve shallow continuous cores from a transect across the surface of the massif.

The Science Party

The science party for the expedition is led by Co-Chief Scientists Prof. Gretchen Früh-Green (ETH Zürich, Switzerland) and Dr. Beth Orcutt (Bigelow Laboratory for Ocean Sciences, Maine, USA), and is distinctly multinational and multidisciplinary. Notably, the expedition is the first to have a female-dominated science party and one of the first to have two female Co-Chiefs. The 31 scientists conducting research as part of the expedition are from 13 different countries and include PhD students, post-doctoral fellows and tenured professors.

The Onshore Party

My role in Bremen was as an igneous petrologist with the core description team. The lithologies recovered from the Massif were challenging to process and thoroughly describe during the onshore phase, but as the saying goes, ‘the best geologist is the one who has seen the most rocks’, and the opportunity to closely examine core was certainly a worthwhile one (even if I’m more of a geochemist!). Spending two weeks with the core description team was the highlight of my experience – both in the core lab and out in Bremen.

Snow in Bremen.

A few thousand metres of core in the MARUM core repository. Image Credit: Prof. Dominique Weis, UBC.

Working with a wide variety of scientists, and venturing beyond my local academic ‘bubble’ has been both a revealing and an encouraging experience. Beyond a set of valuable samples, my stay in Bremen built numerous friendships, resulted in current and potential collaborators and also a clearer, more confident outlook regarding my future career.

In the core lab. Image Credit: Dr. Javier Escartin, IPGP.

My igpet dream team - Nori Akizawa (Kyoto) and Kirsten Hesse (Geo-Zentrum Nordbayern). Image Credit: Prof. Dominique Weis, UBC.

So we have rocks, what next?

A wide range of post-cruise research has been proposed by the multidisciplinary science party to investigate the biological, geochemical, petrological and structural features of the recovered cores in the coming months. I’ll be working on a variety of serpentinite samples to constrain the evolution of serpentinization processes within the massif, principally using a novel combination of halogen ratio and in-situ oxygen isotope measurements.

fluids.rocks

tsgxr: Loading HyLogger TSG data into xarray

HyLogger Data

Recent tsgxr Updates

Using an xarray.DataTree Structure

Lazy-Loading

Simpler Spectral Products

Simple Access to Downhole Plotting

Hosting 'The Littlest JupyterHub'

Getting a Large VPS is Hard

Setting up The Littlest JupyterHub (TLJH)

Setting up Default Authentication

Default Environment Management

Using nbgitpuller

Overriding Default Jupyter Settings

Caches for matplotlib and numba

Adding Users

Quickly Purging Users

Adding a Tailscale Funnel for JupyterHub

Self-Hosted Discord Alternative: Haven

Looking for Open and Self-Hosted Alternatives

Haven: Getting Started

Accessing your Server: Networking Requirements

Installation

Option 1: Docker

Updating

Option 2: Server Directly on the Host

Tunnel Setup

Tunnel on the Host

Tunnel in Docker

Whitelisting Users in Haven

Zola + Codeberg Pages: Deploying a Blog

Codeberg Pages

Moving to Zola

Installing Zola

Get a Theme

Building Your Site

Without CI Workflows

Local Tools (of Choice/Habit)

Using CI Workflows

serene Reactions

Using a Custom Domain

Setting Up Self-Hosted Forgejo Runners

Proxmox

Proxmox Helper Scripts

Setting Up a Forgejo Runner in a Debian LXC

Creating a Debian LXC

Initial Setup

Install Docker

Forgejo Runner Binary

Register the Runner

Start the Runner, and Keep it Running

Back to Codeberg

How This Site Was Built (Again)

Moving to Codeberg

Codeberg Pages and Zola

Workflows: Self-Hosted Forgejo Runners

Building for Humans

PhD Research

Investigating Geochemical Records of Fluids in the Oceanic Crust

Lago di Cignana, NW Italy

The Atlantis Massif, 30°N, Mid Atlantic Ridge

pyrolite

Serpentine Nights

A Few Notes of My Own

A Few Thoughts from Joëlle

How This Site Was Built (the first time)

How This Site Runs

Before You Start

Hosting On GitHub

Building Your Site with Jekyll

Installing Jekyll Locally

Ubuntu Installation

Windows Installation

Using Templated Sites

Serving Up Your Page

Yet to Come

IODP Expedition 357: Atlantis Massif

The Expedition

The Science Party

Recent `tsgxr` Updates

Using an `xarray.DataTree` Structure

Using `nbgitpuller`

Caches for `matplotlib` and `numba`

`serene` Reactions