borntyping

Writing my own media catalogue, one year later

2025-09-25T00:00:00+01:00

It’s been over a year since I started writing my own system to track media. To quickly recap, it’s a small Python webapp that replaced a mix of spreadsheets, notion tables, and Goodreads lists that I used to track games, films and books; and attempted to unify it all into a single database.

Screenshot of Vancelle's homepage

I made a few big changes over that time:

Two or three rewrites of a lot of the internal codebase.
- I wrote my own micro Python-to-HTML library (inspired by hotmetal) that entirely replaced HTML templates. I was fully expecting to regret that later on, but it’s been nice enough to use that I’d happily use it again. It made it very easy to turn things into reusable components, and logic could use all the features of Python rather than the subset Jinja2 provides.
- The app still uses HTMX in a few places, but I used it less and less as I rewrote things. It was usually much easier to do things the traditional web-app way with forms and full page reloads, and I didn’t have much need for interactivity.
Replaced Bulma with Bootstrap. Bulma was very pretty, but I kept reaching for elements I was used to from Bootstrap, and there was a major version upgrade that seemed to be missing some features that put me off it. Looking back at old screenshots, it’s definitely not quite so nice looking now.
Added a lot of new categories for tracking where I’m at with media. I even ended up adding some meta-categories to group them up. It’s maybe ended up a bit overboard, but it seems to work for me and I don’t end up with too many items in the “now playing” groups (even when I have odd exceptions like games that can’t be completed). The meta-categories and categories I ended up with were Upcoming (upcoming, unreleased, undecided), Playing (playing, replaying, ongoing, infinite), Paused (Paused, Shelved), Completed (played, completed, abandoned), and Other (unsorted, reference, untouched).
I gave up on trying to do “REST” style URLs, and made different ways to view the same data entirely separate endpoints. Originally they had a toggle on the page to change a query parameter that selected a kanban-board style view or a simple table, and it ended up being much easier to maintain when I split the endpoints for them entirely, and shared some small bits of code for doing filtering.

There were quite a few changes I didn’t do, which I think are more interesting:

I never added support for other data sources. Even where I had multiple sources for the same type of media, I only ended up using one. (In this case I used Goodreads over OpenLibrary, even though the project was created to move my book lists out of Goodreads, because Goodreads had much better cover art images).
Lots of little buttons and features were broken and I never fixed them. There wasn’t much motivation to fix things when it was easy to work around them. I had some little buttons for things like marking I’d finished a book yesterday or today, and it was easy to work around them by manually inputting a date.
“Boring” work like adding indexes, debugging slow queries, and upgrading outdated libraries. The whole thing is running on a private network and I’m always going to be the only user, so I’m not very worried about maintained or security.

I stuck to my approach of writing an app that was entirely for myself. It was really nice to be able to entirely rewrite bits of it just because I wanted to, without worrying about breaking things or deadlines or the time spent. It ended up being a really relaxing project to work on at times.

If you want, you can take a look at the source at codeberg.org/borntyping/vancelle, but don’t expect it to come with instructions.

Using a static site generator with Codeberg pages

2025-09-01T00:00:00+01:00

I had a set of small sites with various build processes I wanted to port from GitHub Pages to Codeberg pages, and wanted a simple way to publish content to a branch. I didn’t want to have to use or learn a new CI system, so instead settled on adding some commands to a Justfile.

Create an empty pages branch to host the content:

git switch --orphan pages
git commit --allow-empty --message "initial commit"
git switch -

Configure dist/ (or the directory the build process outputs to) as a worktree with the pages branch.

git worktree add dist pages --no-checkout

Run the static site generator or build, outputting files to the dist/ directory.

Enter the dist/ directory—git commands will now operator on the pages branch—and commit and push the built files. You can run this every time the source changes.

cd dist
git add --all
git commit --message "dist"
git push

All done!

When working from a fresh git checkout, the worktree will need to be configured first:

git switch pages
git switch -
git worktree add dist pages --no-checkout

Can you use a Kubernetes secret from another namespace?

2024-08-20T00:00:00+01:00

Yes, and no:

A Pod can only reference a Secret in the same namespace¹.

This means a Pod manifest can only use Secrets from the same namespace, whether it’s used for environment variables, mounting files, or configuring credentials for a container image registry.

RBAC can allow a ServiceAccount access to Secrets in other namespaces².

This means an API request made from a Pod running in one namespace can access Secrets in another namespace.

See imagePullReference and envFrom in the Pod API spec. ↩
Which is how operators that manage secrets across multiple namespaces work. ↩

Writing my own media catalogue

2024-02-19T00:00:00+00:00

Screenshot of Vancelle's index

Recently, I had a lot of time away from work, and a set of related problems:

I use Goodreads to track all the books I read, but I couldn’t add non-traditional works like fics from Royal Road or Archive of Our Own to my lists.
- I looked at alternative sites like StoryGraph, but Goodread’s data exports don’t include all your data. Specifically, start dates and dates for multiple reads aren’t include in their CSV dump.
Video games are now spread across a lot of different storefronts. If a game wasn’t at the top of Steam’s recently played list, there’s a good chance I’d forget about it even if I was enjoying it.
There’s several sites out there for tracking films and tv I’ve watched, but I don’t want to spread my lists across multiple services. I’m also not really interested in my lists being public, or in writing reviews for other people to read—I just want to remember if I’ve already watched something!
The Notion spreadsheet I was using to fill some of these gaps was starting to creak. Manually entering names and looking up covers so my kanban board would look pretty was starting to get frustrating and time-consuming. I also couldn’t support entering multiple sets of dates for books I’d read more than once, or games I’d put down and come back to.

Screenshot of Vancelle's homepage

So, with these in mind, I set out to replace my spreadsheet with a full web application.

My data model went through lots of iterations, but I eventually settled on having “Works” (data about a work), “Remotes” (data about a work from an external source), and “Records” (a start and end date). Each work could have zero-or-more remotes and records. When displayed on screen, details about a piece of media were collected from the work and it’s remotes, with manually provided details from the work taking priority. This meant I could import data from multiple sources, and they could fill in gaps the other didn’t cover, and I could manually override any details I didn’t like (for instance, a lot of Steam games include ™ and ⓒ symbols in their titles).

This was also where I discovered my issues with Goodreads data exports. The first external source I added to my app was the ability to import data from my Goodreads account, both as that was the main service I wanted to replace and as the 700+ entries I had in there would be a great starting point for being able to build my app around real data. In the end, I actually ended up building a module that scraped HTML manually exported from Goodread’s infinitely scrolling books index so that I could include the dates I started reading a book.

Screenshot of Vancelle's 'Create a new work' screen

A long time ago I’d always have a cloud server around to run various applications on (irc bouncers, a personal website, etc) but for a long time now I’ve been using exclusively cloud services to meet my needs. I didn’t really want to go to all this effort only to fuss over the ongoing costs of a cloud server — not so much due to the price, but the regular checking of if it was still something I needed. Instead, I got a spare Raspberry Pi from a friend, plugged that in next to my desk, and hosted the app on that. I originally used Dokku for this, but once I had multiple applications running I moved it over to a somewhat-less-powerful docker compose service.

It’s been really nice to get back some control over my own data—I dropped both Goodreads and a paid Notion subscription due to this! The Raspberry Pi has also worked out great, and once I had it I immediately started finding other uses for it. Having an always-online device that can run Syncthing to mediate files between my different personal devices (which are often not on at the same time) has been great, and allowed Obsidian to partially replace both Google Drive and Notion for me while keeping data within my own network. Tailscale was also really useful here, as it meant I could easily provision TLS certificates for my devices and serve everything over HTTPS, and I could reach my application when I left the house without having to expose the Raspberry Pi to the open internet.

If you’re interested in the source code, you can find it at borntyping/vancelle on Codeberg. It’s named after a librarian from the Old Kingdom series by Garth Nix.

My essential tools

2023-06-14T00:00:00+01:00

A common theme across the tools I use frequently is that they all let me do work faster or with less repetition. I thought I’d list the tools I use a lot and wouldn’t want to go without when doing software development.

I’ve spent a lot of time installing, configuring, and managing my toolkit — probably more time than I’ve saved! However, even if I’ve not saved any time, I find it a lot easier to both focus and get things right when I can automate something. It’s very easy to make mistakes with repetitive tasks, especially if you have to remember a lot of details that aren’t relevant to your current goal.

Development

These are the tools I use constantly when working on code.

PyCharm Professional, Visual Studio Code, and Micro. I use PyCharm by default as JetBrains code intelligence tools greatly improve my productivity on most codebases. Just being able to quickly (and reliably) jump to definitions or usages is enough to make this my favorite editor. I use VSCode or Micro when I want to edit something quickly.
Git Cola. A very simple GUI for Git. I use it write commit messages and preview the changes that will be included in a commit.
Switchbox. My personal git toolbox, used to automate repetitive branch management tasks like rebasing and cleaning up merged branches.
Fedora. Has a very developer-friendly package ecosystem. Packages are up-to-date, maintained, and usually allow multiple versions of programming languages like Python to be installed side-by-side. Almost everything works out-of-the-box without me having to think about it.

Shell

I spend a lot of time working in a terminal, so it’s important to me that it’s nice to use. Several of the tools here simply make it easier to do something I could otherwise do by typing out a longer command.

Atuin. Store shell history in a sqlite database. I don’t use all of it’s features, but the interactive UI helps me find commands from my shell history much faster.
Direnv. Load .envrc and .env files in the current directory. This makes it really easy to manage environment variables for various projects, without making a mess of my global shell.
Starship. Shell prompt system. Keeps important information front-and-center when working in a terminal. I almost exactly replicated my previous zsh configuration, which was hand-built over years. Works in any shell, so I’m not as tied to zsh as I once was.
zoxide. Quickly jump to recent directories. It saves a lot of time not needing to type out full paths or remember where things are when working in a terminal.
ack. Quickly search codebases. Defaults to recursive search and ignoring directories like .git or node_modules, making it a lot easier to use than grep.
bat and exa. A cat replacement with syntax highlighting, and a ls replacement with slightly nicer output.
gh. Automate some common GitHub tasks. I mostly use this to clone repositories quickly. I use GitHub Desktop instead when I need to work with git repositories on Windows machines.

Configuration management

I generally prefer to have all the tools I need installed on my local development machine. I tend to avoid using Docker or Podman for local development, which helps avoid conflicts when using someone else’s workflow that does.

Ansible. I use Ansible to manage my development machines. It’s a lot more flexible than other approaches I’ve used or seen for managing dotfiles. It’s a nice way to abstract over all the different types of packages I need to install.
Poetry. Manage Python dependencies. There’s lots of options in this space, but Poetry works well for me.
pipx. Install Python executables in a dedicated space. Keeps the Python tools I use separate from my system installation of Python and separate from codebase-specific virtualenvs.
npx. Run NodeJS exectuables without installing them.

Cloud systems

These are tools that mostly make a different tool easier to use.

aws-vault. Manage AWS credentials. Integrates with your system keyring and issues temporary credentials when needed. Makes it a lot easier to manage lots of different AWS roles.
kubectx + kubens. Quickly switch between Kubernetes contexts and namespaces. I don’t know why this isn’t built into the kubectl CLI.
stern. Follow logs from multiple containers. If nothing else, not having to restart a kubectl logs command every time a deployment or pod restarts is very helpful, but stern also provides a nice interactive interface for reading logs.

Why I have separate personal and work GitHub accounts

2023-06-08T00:00:00+01:00

Permissions on a GitHub token are opt-in for an organisation, but you can’t opt out of granting permissions for the account the token belongs to. For instance, if an GitHub App or GitHub OAuth application at my place of work requests permissions to read private repositories (a very common permission needed for basic actions like cloning), the token issued at the end of the authentication flow will always have permissions to access my personal respositories. Depending on the settings of the organisation, this may be automatically granted, or may require going though the single-sign on process for that organisation before the token is granted permissions to read from private repositories in the organisation.

If I use the same account for both personal use of GitHub and work use of GitHub, that restricts how carefully I can control access to resources owned by my account. A particually relevant example of this was the recent CircleCI breach, which was the event that prompted me to start using two GitHub accounts. If my account was used purely for personal use, I could have happily removed the application from my account and not worried about it again. But, since my job required working with CircleCI I needed to stay authenticated to it and take the risk that any access tokens were still vulnerable after beeing rotated.

I also work a lot on developer tooling, which means issuing a lot of tokens to work on GitHub applications locally. These are subject to much the same problems, and importantly, end up on devices that you don’t own and may get used with unfinished or insecure code.

My GitHub account was created well over a decade ago now - it’s older than most of the companies I work for, and while the risks here are extremely low I’m a lot more comfortable taking the approach of having a less-valuable GitHub account dedicated to corporate use rather than using something that’s been a very public part of my online presense for a long time.

There have been some other advantages to this split. I use separate mobile phones for personal use and work use, and this means I can use the GitHub android app without needing to go though corportate SSO on a personal device, or getting work-related GitHub notifcations while I’m not working. It’s also nice to have separate GitHub contribution graphs for my own personal work and corportate work.

You can find me on GitHub at @borntyping and @borntyping-corporate.

Final Fantasy 14 microsites

2023-05-16T00:00:00+01:00

Recently I built two small microsites that I find helpful for playing Final Fantasy 14.

While working on a Backstage/React codebase at Kaluza I wanted to take a look at what else the 2023 JavaScript ecosystem had to offer, and especially wanted to experience JavaScript development “from scratch” in a much smaller codebase.

ffxiv-daily-quest-tracker

The first tool I built was a calculator for a series of long quest chains in FFXIV that can only be progressed a small amount each day. I wanted to be able to work out how long each quest chain would take me to finish - being able to visualise my progress kept me a lot more engaged and encouraged me to finish them.

I used Alpine.js for this, alongside Bootstrap. I was already pretty familiar with Bootstrap, and wanted to build something quite quickly so avoided learning a new CSS framework. Alpine.js seemed pretty similar to htmx, a library I’d had some success with previously for a professional project. I didn’t use it here as I wanted to be able to deploy this site to GitHub Pages and avoid the need for a server.

The JavaScript I wrote using Alpine.js was almost certainly not very idiomatic, but it made it very easy to provide some classes containing the structure of my data and then link that into my HTML using <template> tags. The two-way binding meant I didn’t have to do anything complex to link the fields where a user would provide information to my data model. When the values of those fields change the data model and all the HTML using it immediately updates, even though much of that was in a nested class structure — I didn’t need to structure my codebase to fit the conventions of the library like I would have done with React or Vue.

Alpine was very simple to use, and I didn’t need to use any build step or dependency management at all, so the end result was very easy to deploy directly to GitHub Pages. The code for the site is ~4 small files (index.html, index.css for styles, index.js for code and index.json for data). I’d very happily use Alpine.js again for a website.

ffxiv-signposts

The second site was a list of links to other people’s FFXIV sites that I either used or had been recommended. There’s a wonderful ecosystem of helpful tools people have built for this game, even though there’s not much support from the developers of the game for it (e.g. Destiny 2 provides a very full-featured API with access to information about most things in the game, which has been a great base for developers to build extensive and powerful interfaces to that data).

This site displayed a long list of cards that each had a link to the site and a list of tags. Each tag had a one-line description alongside it explaining how the site related to the tag. There were also some special tags for “complexity” (how much a player would need to know to use the site) and whether a site was an official site from the developers of the game. A pair of toolbars at the top allow selecting only the cards in a certain category, or tag within that category.

I used Vue.js for this, alongside Bulma. I wanted to use a framework closer to React that used a component model, so that I could understand the approach better by seeing different ways to implement it. I also wanted to try something different to Bootstrap UI when it came to CSS, as I’ve very rarely used anything else for a HTML project in a long time.

I was initially attracted to Vue.js for it’s promise of being able to work with or without a build step or server, and hoped to take the same approach I did using Alpine.js. That didn’t turn out to be very practical, as a lot of it’s syntax is only supported via a compilation step. I wanted to look at using Deno or Bun to manage dependencies and compilation, but I couldn’t find enough documentation to bootstrap a working build process — I couldn’t find any documentation on how to actually build a Vue.js project, only instructions on how to use create-vue to generate all the files I’d need to setup a project with their recommended approach. The development experience and build process using npm and Vite was actually pretty good, but it still frustrates me that I’ve no idea how it works.

Building components with Vue.js was okay. I didn’t like the templating language much compared to JSX/TSX, and it was frustrating that I’d need {brackets} in some places but not others. I loved that I could write completely normal CSS in a <style> tag and have it scoped to the component though, and that was a far nicer experience than the JSS approach I’d previously used in React. Adding animations around changes to the page (e.g. when switching tabs) was incredibly easy using <TransitionGroup> with very pleasing results.

I’m not sure I’ve come out with any strong opinion on using Vue.js in the future — in many ways it felt like a slightly different React, though I think for a good comparison I’d need to use both on similar sized projects. I made very little use of Vue.js’ features, especially reactivity which it silently got right without me needing to think about it.

Meanwhile Bulma was a surprisingly good experience. It’s a lot smaller in size and scope than Bootstrap, and doesn’t come with a full set of components or a JavaScript library. Given that, I expected to be writing far more CSS to extend what it provided, but I actually found myself writing far less CSS than I would in a comparable Bootstrap project. It covered every basic use case I had, and was extremely readable with verbose selectors like field is-grouped is-grouped-multiline. This library will probably become the first thing I reach for next time I need a CSS framework.

Cleaning up merged git branches

2023-03-31T00:00:00+01:00

Over the last few years (and jobs) I’ve collected a number of scripts and processes for working with git and centralised forges like GitHub and GitLab. switchbox is my latest attempt to collect the bits of that automation I’m currently using, and mostly does some fiddly work to discover branches that have been merged, rebased, or squashed into the main branch of a git repository.

The most useful bit from my previous tools was the ability to clean up branches and references I wasn’t using anymore. Teams at previous jobs had all enforced a standard way to apply merge requests — for instance, my team Datto enforced linear merge commits. My previous attempt at automating this cleanup, git-river, handled standard merge commits but no other cases.

However, my current job doesn’t yet enforce any standard repository configuration and repositories use a mix of rebasing, squashing, and merge commits. I wanted switchbox to handle any type of merge I could think of, so that I wouldn’t need to implement this next time I encountered a project that used a different approach to merging changes.

GitHub supports “commit merging”, “commit squashing”, and “commit rebasing”, which I used to name the three strategies I used to find merged branches in switchbox. It does cover a couple other cases beyond those though.

Merge commits are easy: git branch --list --merged $upstream will list every branch merged into $upstream. Calling git and parsing it’s output here is simple.
Rebased branches are a little more difficult: git cherry $upstream $branch will tell us if each commit in $branch has an equivalent commit in $upstream. This works for rebases and cherry-picked commits, which both take the contents of a commit and reapply it onto a different parent commit — the commit hash will change, but the contents should be the same.
Squashed commits are painful. We can’t compare branches or commits, so we’re left comparing diffs between commits.
1. The first thing to do is find a “merge base” - the common ancestor between our upstream branch and the branch that might have been squashed into it. We can do this with git merge-base $upstream $branch.
2. Then we build a diff between the merge base and our potentially-squashed branch with git diff $branch $merge_base.
3. Then we iterate through each commit in our upstream branch since the merge base. For each of these commits, we check if the the diff between that commit and it’s parent matches the diff we built in the previous step, with git diff $commit ^$commit. If it does, that means the commit is the result of squashing our branch.

Once we’ve found our merged branches from these three strategies, switchbox then deletes them. It runs git remote update before doing any of this, to update all branches from their remote equivalents.

Merging a subdirectory of a git repository, revisited

2020-07-24T00:00:00+01:00

My previous post on the topic used git’s builtin filter-branch subcommand, which has been superseded by the git-filter-repo project.

In the repository being merged, we create a branch named filtered that only contains the contents of the src/ directory, moved to src/other/ - the branch no longer contains files that were outside the src/ directory.

git switch -c filtered
git-filter-repo --path src/ --path-rename src/:src/other/

In the repository the subdirectory is being merged into:

git remote add other ../other-repository/
git fetch other
git merge -s ours --no-commit --allow-unrelated-histories other/filtered
git read-tree -mu other/filtered
git commit

Hiding in D&D

2018-07-09T00:00:00+01:00

I found the mechancies for hiding¹ in D&D 5E a little confusing to understand, as the rules are scattered in a few places in the book.

As an action (or a bonus action for a Rogue), a character can Hide². They make a Dexterity (Stealth) check, with the total becoming the DC for other creatures to notice you. A character can’t hide if they can be clearly seen by other creatures, but they don’t need to been in full cover.

The DM compares the passive Wisdom (Perception) of other creatures to see if they notice the character. Creatures can also use the Search³ action, making a Wisdom (Perception) check to discover the character. In either case, success on the creature’s check means the character is no longer hidden.

A character has advantage on attack rolls against creatures that can’t see them⁴ - for a Rogue, this means they can use Sneak Attack. Once the attack hits or misses, the character is no longer hidden.

Some addition clarifications come from the game designers, but are not in the books:

The rules for unseen say “you give away your location”, but this does mean the character is no longer hidden ⁵

A character can be hidden from a creature even if the creature knows where they are⁶.

A hidden character can lean out or step out of cover to make an attack without being discovered⁷. They are still discovered when the attack hits or misses.

A DM might call that hiding in the same spot results in disadvantage on the Dexterity (Stealth) check⁸.

Package versioning

2018-06-21T00:00:00+01:00

This post descibes my approach to versioning packaged applications, using version numbers that look like this: 3.2.0-2109.12272.

The first part (3.2.0) is a manually incremented version number. It’s incremented for changes that need to be communicated and talked about - “our new feature will be in version 4, which will be deployed next week”.

The second part (2109.12272) is an automatically incremented release number. It’s entirely managed by the CI/CD process that packages the software. In my case, that’s done using GitLab, and the number is composed from the pipeline and job identifiers.

Both of these numbers increase monotonically, so that a each time the CI/CD process runs it builds a package with a higher version and release number without a human needing to remember to manually increase it. The automatic release number also ensures that you never build a package with the same version number as an existing one, and so as long as you keep older packages around you can always downgrade to a previous version.

The approach works very neatly with packaging tools like RPM, which have seperate version and release fields. My build process uses a version.txt and release.txt file, the latter of which is created at build time. Both are included in the package name and in the package itself, allowing the application to read them at runtime to display it’s own version number.

You could quite easily remove the manually incremented version number and loose no technical capability. However, I find simple, easy to remember numbers much easier to communicate than the long digit sequences an automatic release number will provide. Asking someone to check if they are running “version 4 or above” is much simpler than asking if the version number is “12034 or above” - people remember small numbers much more easily. You can also still use familiar SemVer like semantics to describe the scale of change - “we’re upgrading from version 9034 to 9725” says very little about the changes involved, whereas “we’re upgading from version 4.1 to 5.3” implies multiple large changes, or “we’re upgading from version 4.1.0 to 4.1.1” implies a small fix. I find this is especially useful when communicating with non-technical people who use the application.

However, a downside to this is that humans will likely end up being very lax about changing the version number. This is fine for me - I only use it to commuicate major changes and not small patches - but is unlikely to work for all cases. It’s particually unsuited for libraries or APIs, where tracking API changes is important and strictly following SemVer may be important.

Managing version numbers in Python

The rest of this post goes describes an implementation of the above approach that I use for Python packages.

A release.txt file includes the manually updated version number, and an optional release.txt file includes the CI/CD pipeline and job number. The __version__ attribute in my Python module is set by reading from these files instead of being statically defined.

`init.py`

"""An example package."""

import example.version

__author__ = 'Sam Clements'
__version__ = example.version.read_version(__file__)

`version.py`

"""Read version numbers from an embedded text file."""

import pathlib


def read_version(relative_path):
    version_path = pathlib.Path(relative_path).with_name('version.txt')
    release_path = pathlib.Path(relative_path).with_name('release.txt')

    version = version_path.read_text().strip()

    if release_path.exists():
        release = release_path.read_text().strip()
        version = "{}-{}".format(version, release)

    return version

This is included in the Python package metadata by using the attr: directive in the setuptools setup.cfg file (a relatively recent feature that allows package metadata to be set by reading values from a Python module). For this to work properly, the __init__ file has to import as little as possible, so that it’s not including dependencies that may not be installed when you first build or install the package. For my projects, the __init__.py file normally only includes metadata and absolute essentials (e.g. exit handlers with no dependencies).

[metadata]
name = example
version = attr: example.__version__
...

With everything setup to build the application as a Python package, I also use this in a Makefile that builds an RPM with the Python package using fpm.

CI_PIPELINE_ID?=0
CI_JOB_ID?=0

NAME=$(shell python setup.py --name)
VERSION=$(shell cat odyssey/version.txt)
RELEASE=$(CI_PIPELINE_ID).$(CI_JOB_ID)

example/release.txt:
	echo "$(RELEASE)" > "$@"
	
$(NAME)-$(VERSION)-$(RELEASE).x86_64.rpm:
	fpm -s dir -t rpm \
	--name "$(NAME)" \
	--depends "python" \
	--version "$(VERSION)" \
	--iteration "$(RELEASE)" \
	--maintainer "Sam Clements" \
	.

Merging a subdirectory of a git repository

2016-06-29T00:00:00+01:00

In the repository being merged:

git checkout -b temp
git filter-branch --prune-empty --subdirectory-filter <subdirectory>/ temp

In the repository the subdirectory is being merged into:

git remote add other ../<other-repository>
git fetch other
git merge -s ours --no-commit --allow-unrelated-histories other/temp
git read-tree --prefix=<path> -u other/temp
git commit

Done!

The output of the merge should look something like this:

$ cd other-repository
$ git checkout -b temp
Switched to a new branch 'temp'
$ git filter-branch --prune-empty --subdirectory-filter other/ temp
Rewrite 82e8de6f33e9e4e797e0331aac1fc602f68f6bfa (13/13) (0 seconds passed, remaining 0 predicted)    
Ref 'refs/heads/temp' was rewritten
$ cd ../repository
$ git merge -s ours --no-commit --allow-unrelated-histories other/temp
Automatic merge went well; stopped before committing as requested
$ git read-tree --prefix=other -u other/temp
$ git commit
[feature/merge bc00b33] Merge remote-tracking branch 'other/temp' into feature/FTD-768-build

Interesting papers

2015-12-02T00:00:00+00:00

Holistic Configuration Management at Facebook

How Facebook manages and distributes configuration - from small JSON files to huge machine-learning datasets - at a massive scale.
The impact of syntax colouring on program comprehension

Advait Sarkar, 2015.

Study measuring the effects of syntax highlighting - notably that it is useful, and programmers spend less time focusing on keywords when using it.
Google Votes: A Liquid Democracy Experiment on a Corporate Social Network

Hardt, Steve and Lopes, Lia C. R., 2015.

An experimental voting system built and used inside Google that uses software to mix direct and representative democracy by allowing users to vote directly or entrust their vote to another user.
Scaling Agile at Spotify

Henrik Kniberg & Anders Ivarsson (2012).

Describes how Spotify has previously organised their engineers into tribes, squads, chapters and guilds; aiming to avoid layers of bureaucracy and other problems often associated with large numbers of employees.
Bypassing HTTP Strict Transport Security

Jose Selvi, 2014

Attacking NTP to force HSTS policies to expire.

Error handling in Rust

2015-02-06T00:00:00+00:00

A short cheatsheet for dealing with Return values in Rust.

When a Result is Ok<T>:

let result: Result<&str, &str> = Ok("Succeeded");

result.is_ok() == true;
result.is_err() == false;

result.ok() == Some("Succeeded");
result.err() == None;

let other_result: Result<&str, &str> = Ok("Other result");
result.and(other_result) == Ok("Other result");
result.or(other_result) == Ok("Succeeded");

fn example(string: &str) -> Result<&str, &str> { Ok("Example") }
result.and_then(example) == Ok("Example");
result.or_else(example) == Ok("Succeeded");

When a Result is Err<E>:

let result: Result<&str, &str> = Err("Failed");

result.is_ok() == false;
result.is_err() == true;

result.ok() == None;
result.err() == Some("Failed");

let other_result: Result<&str, &str> = Ok("Other result");
result.and(other_result) == Err("Failed");
result.or(other_result) == Ok("Other result");

fn example(string: &str) -> Result<&str, &str> { Ok("Example") }
result.and_then(example) == Err("Failed");
result.or_else(example) == Ok("Example");

When an Option is Some<T>:

let option: Option<&str> = Some("Example");

option.is_some() == true;
option.is_none() == false;

option.unwrap() == "Example";
option.unwrap_or("Other") == "Example";
option.expect("the world is ending") == "Example";

When an Option is None:

option.is_some() == false;
option.is_none() == true;

option.unwrap(); // panic!()
option.unwrap_or("Other") == "Other";
option.expect("the world is ending"); // panic!("the world is ending")

Backups are important

2014-11-06T00:00:00+00:00

Your laptop will die. Your hard drive will fail. You will lose your USB key. You’ll have a power cut halfway through typing up your essay. Accidents happen, and making sure you don’t lose work - especially assignments and essays - is important.

Any important work you do should be saved in multiple places. At a minimum, you should have at least two copies on physically separate devices. Preferably, you should have a third copy - this means that if one copy is destroyed or inaccessible, you still have two copies, and are still protected from a second device failing.

Geographical separation is also useful, and makes it much harder to lose work - this doesn’t need to mean keeping copies of your work in another country, but you should avoid keeping multiple copies in the same place. For example, if you have one copy on a laptop and another on a USB key, you may want to consider not keeping them in the same backpack - if the backpack was stolen, you would lose both copies of the work.

Work in progress

As well as making sure you have multiple copies of your files, you should make sure that you save changes to your files while you are working on them. Most editors will have a feature that automatically saves a copy of your work every few minutes - make sure you enable it. You don’t need to make a full backup every time you write a sentence, but making sure your work is saved regularly will save you from losing all the work you did that day when your word processor crashes or your laptop runs out of battery.

Online file storage

Various online services allow you to synchronise files and folders between online storage and multiple computers. This is one of the easiest ways to ensure you have backup copies of your work, and is extremely useful when working on something from multiple computers. You can also access your files though a web browser or mobile application, so you’re almost always able to access them.

Online storage services generally have a free plan, with paid plans available for additional features or extra space. This post doesn’t cover their paid features, and unless noted each service has applications for Linux, OSX and Windows, as well as Android and iOS phones.

Dropbox provides 2GB of storage (with extra space for using certain features or recruiting people), keeps a 30 day history of changes to files, and has applications for almost any device or operating system.
Google Drive provides 15GB of storage, and integrates with a set of office software that you can use from your browser or phone (word processing, spreadsheets, presentations). It’s missing a Linux application.

If you have existing services or devices you want a service to work with, Amazon and Apple also provide their own storage offerings: Amazon Cloud Drive provides 5GB of space, though doesn’t have much to show off aside from being available on Amazon Fire devices. iCloud is part of iOS and OSX, and you need an Apple device to use it, and there’s no Android client.

Version Control

Git can be used to incrementally record changes to your work and push/pull them to other repositories. Mercurial is a simpler and more user friendly alternative to Git, but both will need you to spend some time learning how version control systems work and how to use the tools they provide. For Computer Science students, basic knowledge of how to use Git and version control is essential, and it’s worth knowing how to use at least one version management system well as it’s almost essential for any software project. Various hosting services for Git allow you to store and publish repositories online.

GitHub is the most popular, and is used for many open source projects. You can publish an unlimited number of public repositories, but private repositories either require the Student Developer Pack (which provides 2 private GitHub repositories along with many other goodies), or a paid plan.
BitBucket provides unlimited public and private repositories. It’s similar to Github, but provides some different features and is aimed at companies and teams rather than individuals.

You can use Git and Mercurial on your own server with nothing more than SSH access, though there are also applications that will provide an interface similar to Github or Bitbucket. I’d suggest looking at GitLab or Gitorious if you plan on doing this.

Summary

Remember to regularly save work in progress.
Your work doesn’t exist until there are at least two copies.
It doesn’t count if those copies are in the same physical location.

Installing Pandoc on Windows

2014-04-15T00:00:00+01:00

Install pandoc using the Windows installer from here. Install pandoc globally using this command (using the path to the version of pandoc you downloaded):

msiexec /i pandoc-1.12.3.msi ALLUSERS=1

Install miktex from here.

Run these commands to check that pandoc and miktex are installed:

"C:\Program Files (x86)\Pandoc\pandoc.exe" --version
"C:\Program Files (x86)\MiKTeX 2.9\miktex\bin\pdflatex.exe" --version

Building a RPM for Supervisor

2014-04-09T00:00:00+01:00

Start with a skeleton specfile, which can be generated using rpmdev-newspec:

rpmdev-newspec supervisor.spec

The example specfile used to write this post can be found here.

Package metadata

The basic package metadata can be extracted from Supervisors existing package metadata (setup.py) and it’s website (supervisord.org).

Name: supervisor
Version: 3.0
Release: 1
Summary: A system for controlling process state under UNIX

Group: System Environment/Daemons
License: https://github.com/Supervisor/supervisor/blob/master/LICENSES.txt
URL: http://supervisord.org/

%description
Supervisor is a client/server system that allows its users to control a number
of processes on UNIX-like operating systems.

Defining requirements

Supervisor requires Python 2.4 or above (excluding Python 3.0), Setuptools (a packaging library), and meld3 (a library used inside Supervisor). In CentOS, these are packaged as python, python-setuptools and python-meld3, and so are added to the package requirements using their package names.

Requires: python > 2.4
Requires: python < 3
Requires: python-meld3 >= 0.6.5
Requires: python-setuptools

Sources

Supervisor is developed on GitHub, which provides source tarballs for repositories.

In the best case, the source url could be defined in the specfile, and spectool could be used to download the tarball into the SOURCES folder:

    Source0: https://github.com/Supervisor/supervisor/archive/3.0.tar.gz

    spectool --get-files --sourcedir supervisor.spec

Unfortunately, Github’s tarball URLs use a series of redirects that result in a different name for the file, and so the files downloaded by spectool don’t match the filenames rpmbuild expects.

Instead, I’ve used a Makefile that downloads the tarball (and ensures the filename is correct, which also makes it easy to checksum the sources before using them. The Makefile used to build the package is decribed later on in the article. For now, the specfile only has to know the name of the downloaded tarball:

Source0: supervisor-%{version}.tar.gz

Building from source

The package is built across three major steps: prep, build and install.

The preparation step is simple, and does nothing but extract the source tarball. The -n option describes the name of the tarball to extract, though %{name}-%{version} is the default anyway:

%prep
%setup -q -n %{name}-%{version}

The build step uses setuptools to build the package, and then calls it again to install the package into the ‘build root’:

%build
%{__python} setup.py build

%install
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT

Finally, the specfile has to define the files that will be included in the package:

%files
%defattr(-,root,root,-)
%doc CHANGES.txt COPYRIGHT.txt LICENSES.txt PLUGINS.rst README.rst
%{_bindir}/echo_supervisord_conf
%{_bindir}/pidproxy
%{_bindir}/supervisorctl
%{_bindir}/supervisord
%{python_sitelib}/*

This declares that the files should be owned by root, that Supervisors documentation should be placed in a automatic documentation directory, that each of the scripts Supervisor provides should be included, and that all of the Python modules that were installed should be included.

This should result in a directory tree like this:

└── usr
    ├── bin
    │   ├── echo_supervisord_conf
    │   ├── pidproxy
    │   ├── supervisorctl
    │   └── supervisord
    ├── lib
    │   └── python2.6
    │       └── site-packages
    │           ├── supervisor
    │           ├── supervisor-3.0-py2.6.egg-info
    │           └── supervisor-3.0-py2.6-nspkg.pth
    └── share
        └── doc
            └── supervisor-3.0
                ├── CHANGES.txt
                ├── COPYRIGHT.txt
                ├── LICENSES.txt
                ├── PLUGINS.rst
                └── README.rst

Including configuration

As well as Supervisor itself, including distribution specific information is useful, and one of the primary reasons to build an RPM for it. Firstly, an init script to mange the service is needed, and that requires default configuration so that Supervisor can be started.

Two additional sources are used, containing an init script and a Supervisor configuration file.

Source0: supervisor-%{version}.tar.gz
Source1: supervisor.init
Source2: supervisor.conf

The Supervisor configuration used is a simple as possible, only overriding the options needed to run Supervisor as a system service (by default, most of Supervisors files are placed in /tmp):

[unix_http_server]
file=/var/run/supervisor.sock

[supervisord]
pidfile=/var/run/supervisord.pid
logfile=/var/log/supervisor/supervisord.log
childlogdir=/var/log/supervisor

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

[include]
files = /etc/supervisor.d/*.conf

These files are added to the install step, along with the directories they need:

%install
mkdir -p %{buildroot}/%{_initrddir}
install -p -m 755 %{SOURCE1} %{buildroot}/%{_initrddir}/supervisord
mkdir -p %{buildroot}/%{_sysconfdir}
mkdir -p %{buildroot}/%{_sysconfdir}/supervisord.d
install -p -m 644 %{SOURCE2} %{buildroot}/%{_sysconfdir}/supervisord.conf
mkdir -p %{buildroot}/%{_localstatedir}/log/%{name}
%{__python} setup.py install --skip-build --root $R

Finally, the package needs to be configured to add the supervisord service to chkconfig using the post-install step:

%post
/sbin/chkconfig --add %{name}d || :

And to then remove it in the pre-uninstall step:

%preun
if [ $1 = 0 ]; then
    /sbin/service supervisord stop >/dev/null 2>&1 || :
    /sbin/chkconfig --del %{name}d || :
fi

The $1 = 0 statement checks that the package is being removed.

With these changes, the specfile should be complete, and the directory tree should now include files in /etc and /var:

.
├── etc
│   ├── rc.d
│   │   └── init.d
│   │       └── supervisord
│   ├── supervisord.conf
│   └── supervisord.d
├── usr
│   ├── bin
│   │   ├── echo_supervisord_conf
│   │   ├── pidproxy
│   │   ├── supervisorctl
│   │   └── supervisord
│   ├── lib
│   │   └── python2.6
│   │       └── site-packages
│   │           ├── supervisor
│   │           ├── supervisor-3.0-py2.6.egg-info
│   │           └── supervisor-3.0-py2.6-nspkg.pth
│   └── share
│       └── doc
│           └── supervisor-3.0
└── var
    └── log
        └── supervisor

Automating the build with make

Since rpmbuild requires all sources to be placed in ~/rpmbuild/SOURCES, I’ve used a makefile to automate the collecting the sources, and building the RPM:

specfile=supervisor.spec

version=$(shell grep Version ${specfile} | awk '{ print $$2 }')
release=$(shell grep Release ${specfile} | awk '{ print $$2 }')

package=supervisor-${version}-${release}.x86_64.rpm
rpmbuild_package=${HOME}/rpmbuild/RPMS/x86_64/${package}

tarball=supervisor-${version}.tar.gz
tarball_url=https://github.com/Supervisor/supervisor/archive/${version}.tar.gz

${package}: ${rpmbuild_package}
    cp ${rpmbuild_package} ${package}

${rpmbuild_package}: sources
    rpmbuild -ba ${specfile}

sources: ${tarball}
    spectool --list-files ${specfile} | awk '{ print $$2 }' | xargs -i cp "{}" ${HOME}/rpmbuild/SOURCES/

${tarball}:
    wget ${tarball_url} --output-document ${tarball}
    md5sum -c sources

.PHONY: sources

The version and release are read from the specfile, the used to work out the package name. spectool is used to list the sources required and copy them into rpmbuilds directories, and wget is used to fetch the source tarball due to the previously mentioned issues with redirects.

A simple filesystem watcher

2014-02-14T00:00:00+00:00

watch-fs is a small, simple tool that watches a directory and runs commands when files change.

Lots of similar tools already exist - nodemon, my own spotter, inotifywait. Each of them have been to complex to use quickly, or in the case of nodemon, are geared towards a specific use case.

watch-fs aims to work as simply as possible out of the box, with any non-essential features being optional.

watch-fs "echo File changed"

In this example, watch-fs will echo File changed any time a file changes in the current directory. It will ignore further file changes for a short delay once the command finished. This is so that editors (and commands) that change multiple files don’t result in the command running several times.

Commands can use the name and path of the file that changed:

watch-fs "echo File '{name}' changed at path '{path}'"

A few arguments control optional features:

watch-fs --first --clear --verbose "rspec"
watch-fs -fcv "rspec"

--first runs the command before waiting for changes.
--clear runs clear before running the command, which keeps the terminal free of output from multiple runs of the command.
--verbose prints the command before running it, which is useful for separating the output or showing what was run when using --clear.

Advanced Tox usage

2014-01-29T00:00:00+00:00

This article explains the Tox configuration used by several of my Python projects, which both tests the module on multiple versions of python and runs testing-related tools for style guides, static code checking and coverage reports.

The tox.ini used here is taken from Supermann, and the source can be found here.

Running pytest to find and run tests

py.test is used to discover, run and write tests. It uses tox.ini as a configuration file by default, and the two tools work very well in combination. The first two sections tell Tox to use the default Python 2.6 and 2.7 environments, to install pytest and mock as test dependancies, and to uses py.test to run the modules tests.

[tox]
envlist=py26,py27

[testenv]
commands=py.test supermann
deps=
    pytest
    mock

[pytest]
addopts=-qq --strict --tb=short

Running flake8 to check code

flake8 wraps several Python tools for static checking of code - pep8 for style, and pyflakes for static error checking. This configures it to use tox.ini for it’s configuration - reducing the number of configuration files in the repo.

[testenv:py26-flake8]
commands=flake8 --config=tox.ini supermann
basepython=python2.6
deps=flake8

[flake8]
exclude=supermann/riemann/riemann_pb2.py
max-complexity=10

Running coverage to show untouched code

Coverage is by far the most verbose tool configured in my tox.ini, but the HTML reports it generates are very useful for working out which parts of my code aren’t being run by my tests. This does several new things with tox: it runs two commands (one to generate the data, and another to render a HTML report), and uses the dependencies from the existing testenv configuration.

Coverage itself is told to use tox.ini for it’s configuration, and is configured to ingnore various parts of the codebase (the tests folder, and lines of code that should never run). The coverage files are placed in .tox to keep the repository tidy while working.

[testenv:py26-coverage]
basepython=python2.6
commands=
    coverage run --rcfile tox.ini --source supermann -m py.test
    coverage html --rcfile tox.ini
deps=
    {[testenv]deps}
    coverage

[run]
data_file=.tox/py26-coverage/data
omit=supermann/tests/*

[report]
exclude_lines=
    def __repr__
    raise NotImplementedError
    class NullHandler

[html]
title=Supermann coverage report
directory=.tox/py26-coverage

Tox and Travis CI

2013-09-02T00:00:00+01:00

Until recently, my approach to building python projects that used Tox on Travis CI was simply to have two separate configuration files, each duplicating the others settings.

This would normally work okay, with both Tox and Travis defining a set of python runtimes to use and a command to run on them. However, when it came to the more complex tox.ini file I’m using for python-dice, recreating this in the Travis configuration would have difficult to do, and require a lot of maintenance when the Tox settings changed.

The solution? Using Travis to run Tox, using it’s matrix feature and the TOXENV environment variable to create a different build for each of several Tox environments, like so:

:::yaml
language: python
env:
- TOXENV=py26
- TOXENV=py27
- TOXENV=py32
- TOXENV=py33
- TOXENV=pypy
install:
- pip install tox
script:
- tox

Playing with the Github Timeline data

2013-05-22T00:00:00+01:00

I came across ‘The Open Source Report Card’ today, which included some interesting results on my report. It claimed I’d contributed to repositories using VimL and Go, neither of which I could remember doing anything in relation to.

Looking into how it worked led to the Github Archive, which is available as a dataset on Google BigQuery. Playing with this eventually led to finding the problem - the ‘contributions’ graph included all of a users events. This included ‘star’ (previously ‘follow’) events, so it was counting repositories I’d starred as repositories that I had contributed to.

A more accurate result was achieved with the following query:

    /* Count the number of my events by language excluding stars/follows */

    SELECT repository_language, count(repository_language) as n,
    FROM [githubarchive:github.timeline]
    WHERE actor="borntyping" AND type != "WatchEvent"
    GROUP BY repository_language
    ORDER BY n DESC

The Github Archive documentation seemed to be devoid of simpler example queries, so this is how I reached the above one, starting from one adapted from the primary example on githubarchive.org.

    /* List all of my repositories, counting the number of events */

    SELECT repository_name, repository_owner, count(repository_name) as events,
    FROM [githubarchive:github.timeline]
    WHERE repository_owner="borntyping"
    GROUP BY repository_name, repository_owner
    ORDER BY events DESC

    /* List all of my repositories, counting the number of events, and sorted by language */

    SELECT repository_name, repository_owner, repository_language, count(repository_name) as events,
    FROM [githubarchive:github.timeline]
    WHERE actor="borntyping"
    GROUP BY repository_name, repository_owner, repository_language
    ORDER BY repository_language, events DESC

    /* Count the number of my events by language */

    SELECT repository_language, count(repository_language) as repository_language_count,
    FROM [githubarchive:github.timeline]
    WHERE actor="borntyping"
    GROUP BY repository_language
    ORDER BY repository_language_count DESC

Spotter at Show and Tell

2013-05-04T00:00:00+01:00

My second talk at a BCS Mid Wales Show and Tell event, this time on Spotter, a command line tool I wrote. The slides are available here.

ZSH startup time

2013-03-10T00:00:00+00:00

I’ve had issues with zsh starting up and running very slowly for a while. I’d assumed the problem simply lay with the number of plugins and other scripts run by oh-my-zsh, but a bit of research has found two solutions that have worked very well for me (your mileage may vary):

Removing the github plugin. I don’t think I was using it much, if at all, and removing it stoped the noticable delay each time the prompt was shown (Source)
Adding skip_global_compinit=1 to .zshenv. This dropped the startup time of zsh from ~0.85 seconds to ~0.15 seconds. This was benchmarked using /usr/bin/time zsh -i -c exit (Source)

Argumented Testing

2012-11-25T00:00:00+00:00

I’ve finally gotten round to ensuring diceroll has a set of unit tests, and along the way I’ve ended up creating a very small, but useful testing library, called argumented (as it lets you agument tests with arguments!).

The library provides a few functions for ‘packing’ argument sets to be passed into a function or unit test, and a decorator for unpacking a class using these functions and replacing said functions with a set of functions that each call one of the given argument sets.

You can read more about it here.

Aleph, version 2.0.0

2012-05-29T00:00:00+01:00

Version 2 of the Aleph library has been released - a large jump in version numbers, but justified by a rewrite of most of the library. The library now follows the reactor pattern - inspired in part by a wish to make the library more compatible with a user oriented interface, as opposed to a bot that requires no (local) user input to run.

Rewriting it to use event handlers and a reactor allowed the use of things I was not able to attempt before - such as having an event handler for STDIN, so that user can interact with the program locally. User input is currently very simple - all it does is exit on any STDIN input event - but has potential for far more interesting applications.

The use of event handlers mean that individual client objects now handle each message, not the core object. This both sandboxes clients so that they can’t access data from the other clients, and would let you run multiple client handlers using completely different code for each (not that I can actually think of a use for that…).

There’s very little in the way of other new features, though the next few versions should introduce more in order to better support strategy.

Aleph, version 1.2.0

2012-04-22T00:00:00+01:00

Aleph 1.2.0 brings mostly internal changes to the way the Message class works and collection of server and channel meta-data such as channel topics and the server MOTD.

Previously, message subclasses would look like this, and the messages would be created by checking the messages module to find a class with the name of the irc command.

:::python
class JOIN (Message):
    ...

However, when it came to add classes for numeric server replies, I found two problems with this approach: that python class and function names cannot start with a number, and that even if I could do that, naming classes with 3-digit numbers would be ugly and confusing.

The solution to this was to use decorators, allowing message subclasses to be defined like this:

:::python
@Message.Reply(332)
class RPL_TOPIC (Message):
    ...

The decorator adds the class to a dictionary of classes stored inside the Message class, and allows the code to access the class for a numeric reply directly, as well as allowing the class to keep a human-readable (read: slightly more human-readable, because RFC 1459 doesn’t define very clear names for the server replies) name.

I also adapted this to work for ‘normal’ message classes as well, so that the message creation code was not looking in two places (Message.NUMERIC_REPLIES and globals()), and added a second decorator for server commands.

:::python
@Message.Command
class JOIN (Message):
    ...

This also resulted in a cleaner module, as the IRC command and reply classes are added to Message.MESSAGES, which is then the only location searched for classes to use for an IRC message (instead of searching globals() for a class).

Aleph, version 1.1.0

2012-02-07T00:00:00+00:00

Aleph version 1.1 brings the CallbackBot class, which is now the default instance of Aleph.

It’s the first version that allows a bot to be made without subclassing any of the core classes and can be instantiated using only the configuration.

The ‘callbacks’ config item should be a list or other iterable of tuples containing (message, function), where message is a string or list of strings that name Message types - when one of these message types is generated by an incoming message, the function is called with the Message instance.

Example script:

:::python
#!/usr/bin/python

import aleph

bot = aleph.Aleph({
    'nick' : 'AlephExampleBot',

    'servers' : [{
        'id': 'AFNet',
        'address': 'irc.aberwiki.org',
        'channels': ['#aleph']
    }],

    'callbacks' : [
        ('PRIVATE', lambda message: message.reply('Oh my!')),
        ('PRIVMSG', lambda message: message.reply('Hello!')),
    ],
}).run()

Aleph, version 1.0.0

2012-02-01T00:00:00+00:00

A published, working version of the Aleph IRC library is now available. While there are plenty of features not yet implemented, it should now be possible to use it to implement a simple (or complex) IRC bot.

An example of a bot that logs all incoming messages:

:::python
#!/usr/bin/python

from aleph.core import Core
from aleph import logger

class AlephTestBot (Core):
    def input (self, message):
        # Log the message
        logger.log(message, 6, message.command)

bot = Aleph({
    'nick' : 'AlephExampleBot',

    'servers' : [{
        'id': 'IrcNet',
        'address': 'server',
        'channels': ['#aleph']
    }],
}).run()

Features I hope to add in the near future include:

A callback interface, so functions can be registered to message types - this will become the default instance of an Aleph bot and will be significantly easier to build a bot with.
Better understanding of the irc protocol on the part of the IrcServer class, including things such as reading the MOTD and other data.
Documentation that does not rely on looking at the source code.

Future Shock

2011-11-02T00:00:00+00:00

“Wake up.”

…

“Wake up.”

…

“I don’t think she’s survived.”

My eyes blink open, taking in the blur of blue, pink and black above me. The fuzzy shapes slowly merge into a face and body, black hair and pale blue clothing. The face seems a little surprised, although happy.

“She’s alive. 21st century cyro was better than I thought, maybe. Hey. Can you hear me?”

I nod.

“Good. So you can understand what I’m saying? If I remember my history lessons, you should be hearing English or Mandarin right now.”

It take a moment for me to realize the sound of the blur’s words aren’t coming from their mouth, but almost from the inside of my ear. I close my eyes to better concentrate on the person’s words.

“I can hear English”, I reply. “But whats happening to my hearing?”

“There’s a ryth-yk-is-tchn in your ear. Umm, I don’t think it’ll handle that word actually. I think the closest concept you may have is a ‘babel fish’. I speak nyth-en-Yth, and it translates it into late 21st century English so you can understand.”

“Right -“ I start thinking. “You talk in a diffrent language, and imply that 21st century English isn’t recent. So, I’m not in the 21st century, am I? What century am I in?”

“The 29th.”

“Ah. This isn’t going to be very simple, is it?”

“I’m afraid not. A couple of the others we woke from cyro-sleep… went uthr-ni-kr - umm, ‘time and culture shock’. The good news is, so far, that you seem to be handling it a lot better than they did.”

“Right. I’m getting up.” I reopen my eyes, and this time they quickly adjust. In front of me is a dark haired girl, wearing a pair of thin rimmed glasses and a light blue shirt. Beyond her, a small white box of a room, brightly lit but with no visible light source. I’m too busy wondering why I’ve just woken up from 8 centuries of cyro-sleep to take in more information about the room. I push forward, realising that all I can do is strain against the straps around my torso and limbs. “Can you take these off me?”

“I think so. You seem sane enough.”

She has something that looks like a ridgid peice of paper in her arms, upon which she uses her finger to click on it. The small chamber that contains me beeps, and the straps fold back - I realise that I’m in a chamber thats probably older than me - small patches of rust and streaks down the side indicating leaks. I hate to think about my own condition. One step forward, and another. And a third. After that, my legs decide that they don’t like having to move and ungracefully dump me on the floor in a messy pile of limbs.

The girl puts her paper in - I think that it just became part of the wall when she put it there - and leans down to pick me up. I’m lifted like a feather, and gently placed into something thats somewhere between a chair and a coccon. A plain white room, with this chair, a cyro chamber against one wall, a blue line that I assume marks the door on the opposite wall - I’m beginning to wonder if I’m inside a really advanced sci-fi sensie. I try and think about leaving, and the world stays solid around me. Maybe this is reality.

I look at the sheet that she placed on the wall. It’s covered in unreadable symbols, with the exception of an image - I think it might be me - and a phrase in english.

‘Helen Valentine’. I think that’s my name.

What do I mean, ‘I think it’s my name’ - of course it is. It’s hardly something you forget. I don’t think my brain is quite working. My thoughts are disconnected, I can’t plan or think. I just want to curl up into a ball and sleep — the ‘chair’ pulls up beneath me. Maybe I should have seen that coming. I mean, psychic chairs — who wouldn’t have guessed?

Infinitum: Sara

2011-04-22T00:00:00+01:00

I retired to Mesa-1:Luma after eight years in the Marines. I piloted a dropship, a Model 4 Deminutio - “Sara”. Got shot to pieces, and me with it. I was deemed unfit for duty, and poor Sara was consigned to the robofac scrapheap - well, I couldn’t let that happen, so I brought her off the TA with me retirement money, and got her refitted.

Got rid of all the weaponry, and the fancy systems - most of them were dead, anyways - and made her a bit more comfy, and stocked it with civilian grade flightsuits. And with that, I set up on Luma, flying people into orbit and letting them make the jump back to sea.

Luma’s a great planet for this kinda sport. Small, with as dense atmosphere, low gravity, and it’s almost entirely sea,. Man made islands and structures making up most of the land mass - it’s a really popular holiday resort, with plenty of people passing though to and from the TA Core systems.

Not many people get to try jumping out a dropship, not without several years of military service, and so I pull a pretty good business. Brilliantly relaxing - all I do is fly up, watch my passengers jump out, and land again.

And the sights, they never get old. You watch a group of brightly coloured specks jump down to sea as the sun falls below the horizon, and you tell me it’s not something you couldn’t do forever.

Infinitum: Warp

2011-02-22T00:00:00+00:00

Our drives are amazing machines. They can pull a ship though a node, between two points anywhere in the universe.

Not a small ship, like the shuttles and rockets that used to take people and equipment from Earth into orbit - these are ships that dwarf towns. The smallest ships we can fit the drives on are two and a half miles long, 10 billion tons of mass - it’s not something you’d ever want to stand in front of.

But when you look at the fleet, those ships come under the categories ‘small’ and ‘expendable’. Each Arc ship holds hundreds of these, in the hangar bays of numerous larger ships. This fleet is the result of years of the Arc’s factories working from the raw material of a universe.

Many say the Arcs are humanities greatest achievement. And who am I to disagree? I work on the engines of the Callista, the last Arc to leave Sol. It’s jump engine is almost as long as the ship itself, a core running though the heart of the ship. It’s surrounded by fusion plants that cluster round it like cubs to their mother. It’s a drive that pulls 10 trillion tons of metal, plastic, biology, gas and water though two completely separate points in space.

It requires enough power that almost every other major ship system shuts down to achieve it. For a moment, the ship and the space around it plunges into darkness, and when it lights up again the stars around it are completely different, and the secondary engines kick in. Those engines are an achievement in their own right, larger than cites and several miles long, there to push the Arcs though space where there are no jumpnodes to guide us.

Those engines are a beautiful sight to behold. Five perfect blue flames silently roaring into the beyond, shoving a chunk of metal though space.

We’ve only found a few nodes from which we could see the same stars as another. Our ships may be an absurd size and weight - but the universe is able to look down on them and laugh at our feeble attempts. For all we know, it goes on forever.

Infinitum: Beginnings

2011-02-22T00:00:00+00:00

The solar system where we begun is Sol. Nine planets, an asteroid belt, lots of moons, and a single inhabitable planet - Earth. Earth, the third rock from the Sun: humanities home. It’s where we began… and we’re never going back. It’s a place we’ll never see again, lost to time and already but a memory for most. And what happened to Terra that we could never go back? Twenty-five years after the first Arc Ship, Sol’s one and only jumpnode closed. We were lucky. A few decades late, and humanity may have never reached the stars.

We left on the Arcs. Humanities greatest achievement, in our thousands of years of history. Ships larger than cities, carrying humanities population in hibernation, waiting to arrive on new planets and to go forth and multiply.

I was one of them, you know. I was only twenty-five when I left Earth on the Dementer. Such a long time ago… But I was one of the first, one of the first billion humans to reach the heavens, boldly going where no man had gone before. heh …You kids wouldn’t remember that, but there was a time when we would dream of the heavens, thinking of what it would be like to go to space, to find aliens and make war and peace with them. But those were very different times.

So our Arcs plummeted though space, punching though the ‘nodes with the finesse of a drunken American. And then we found the inhabitable planets, and started to find Life. Oh, that was a shock to some people! Animal and plant life, with even more variation than Earth ever had. And so we did what humanity does best. Claim, colonize, spread, advance. Some would say infect. Sentient life is brilliantly effective at expanding.

And the Arc’s helped us to do that very efficiently. Able to cross systems in days, and jump across almost infinite amounts of our universe with the aid of the jumpnodes. We found ourselves in sections of the universe that we had never seen, that were untouched by man - for a short time, anyway.

But like always, we were still followed by disaster. The third arc, the Arc Ship Titania, may have been humanities greatest ever loss. It came out of it’s second jump and was hit by a small moon. Nothing special, no last rally against a honorable enemy, nothing they could do to save themselves from destruction. Titania’s still out there, somewhere. Near the Origin, theres a system with a few small planets and moons - one of them has a beautiful sparkling atmosphere. We call it Splendor.

Before Titania hit, Splendor had no atmosphere. Now, it has a sky of twisted metal and plastic, debris from the Arc. It’s the biggest human graveyard in history.

And a few years after that, we lost the Eidolon. They decided that they could not agree with our limits on weaponry, AI and bio-modification, the technologys that made us leave Earth behind. They forgot what those things had done to humanity before. The forgot the destruction that they caused. We no longer know what Eidolon’s goals are. They advance far more aggressively than we did, tearing planets for resources and continuing to live in space. We can only hope they never turn on us.

Callista was the fourth Arc to leave Sol. They were careful, not wanting to fall to disaster. And they didn’t - but they did discover disaster. Their scientists saw what our jumpdrives were doing to the nodes. We saw the fluctuation in the nodes far too late, and realized that our drives made them unstable. Having something punch though you at almost instantaneous speed will damage anything. We have far better technology now, and can traverse the jumps far more safely. But we didn’t have this before many of the nodes closed from our older, more destructive drives.

And thats the second reason we can’t ever see Sol again. We lost it.

Even if the node reopened, we have no idea where it could be. The few systems that make up the Origin? We can’t ever get back to them either. I say ‘we’ can never get back to Sol, and the Origin. But… not all of us left. The final Arc Ship, Absenti. It never left. Neither did any of those who stayed on Earth and the moon colonies, the crazies that thought we could ever renew Earth, after what we did to it. We may never know what happened to them now. Maybe they’re actually terraforming Earth. Maybe they could have found another jumpnode, and left the system to another place. Or they all died. Who knows?

borntyping

Writing my own media catalogue, one year later

Using a static site generator with Codeberg pages

Can you use a Kubernetes secret from another namespace?

Writing my own media catalogue

My essential tools

Development

Shell

Configuration management

Cloud systems

Why I have separate personal and work GitHub accounts

Final Fantasy 14 microsites

ffxiv-daily-quest-tracker

ffxiv-signposts

Cleaning up merged git branches

Merging a subdirectory of a git repository, revisited

Hiding in D&D

Package versioning

Managing version numbers in Python

__init.py__

version.py

Merging a subdirectory of a git repository

Interesting papers

Error handling in Rust

Backups are important

Work in progress

Online file storage

Version Control

Summary

Installing Pandoc on Windows

Building a RPM for Supervisor

Package metadata

Defining requirements

Sources

Building from source

Including configuration

Automating the build with make

A simple filesystem watcher

Advanced Tox usage

Running pytest to find and run tests

Running flake8 to check code

Running coverage to show untouched code

Tox and Travis CI

Playing with the Github Timeline data

Spotter at Show and Tell

ZSH startup time

Argumented Testing

Aleph, version 2.0.0

Aleph, version 1.2.0

Aleph, version 1.1.0

Aleph, version 1.0.0

Future Shock

Infinitum: Sara

Infinitum: Warp

Infinitum: Beginnings

`init.py`

`version.py`