You know how recipes often have a reputation for rambling sections about the origin of those recipes? Well, let’s just say that I finally understand why.

Vent is my biggest project ever, at time of writing (over 4k LOC of Rust, not including the SQL or the templates 😲 which is pretty damn huge for me) and has been my baby for the last 8 months. It’s finally proper time that I actually show it off and explain the backend (this definitely isn’t a disguised way for me to re-familiarise myself with the whole thing 😉).

You know the saying Vexation breeds innovation? Maybe not? Well, regardless it’s definitely true for this project. I’d just come into possession of a monstrous spreadsheet for managing various events, as well as a OneDrive folder where the photos got dumped. The old spreadsheet was exclusively for managing the details of the events and had no facilities for managing who was attending the event - this was dealt with by a series of newsletters that died out as people got bored and, in the late stages, teams messages which people were told to react with a 👍 to. The photos weren’t organised at all, were a pain to manage and nobody could see them unless you just asked and then they had to be manually dug out.

No more! To fix this, we go to a self-hosted 100% custom solution with Rust using axum and postgres.

  • Jack, did you need to do this? God no.
  • Jack, was anyone asking you to do this? Also, no.
  • Weren’t there any prebuilt solutions? Alas, I couldn’t bring myself to look when I saw the potential for such an exciting project.

The plan for the rest of this is to go over the project as it is now, then to go over the creation and some of my interesting hurdles.

Overview#

Technical Details#

As of right now, the project basically just acts as an access-managed front-end for a postgres database serving GET requests using liquid, and handles all interactive elements via POST requests. For CSS, I’m using Bootstrap, which feels 2015-esque but is the easiest way for me to get some vaguely-competent looking CSS. I like doing things myself rather than using pre-built solutions, but I’m willing to limit this project to one new main thing - the sentence above.

If you want a version to see just about what I’m waffling on about - feel free to have a gander over one instance over at Knot. If you want my help to deploy your own instance - feel free to ping me an email here.

If you’re wondering where I’m going to mention what Javascript framework I’m using, the answer is none. I’m a systems dude who’s trying to learn, and I’ll be honest - this is as much a project to learn the ropes of postgres, deployment, back-ends and the web as it is a project to manage events. I’ve tried lots of the java/type-script-y frameworks, and they frankly don’t really appeal to me. I’ve become pretty damn hooked on the Rust programming language due to the package management, documentation, tooling and unique features like discriminated unions which are missing from loads of other languages so I didn’t want to really use any. I’ve tried (trust me I’ve really actually tried), but I can’t quite get javascript to fit inside my head - I had the most success with Svelte but it just doesn’t work for me ;(.

User Story#

So, the main way this project works is through events and peoples’ links to events. An event has a number of properties (that came from the original spreadsheet - one of the project requirements was to be able to import stuff from that relatively easy), and then a bunch of links. They all have links to the people participating, the people looking managing the event, and the photos that have been added. Participant-Event links also can be marked as verified - this is useful for making sure people have actually attended the event. A manager-level user has to do that after the event, and in my experience the photo comes in very useful for this.

Finally, there’s a rewards system where after completing a certain number of events, you can be eligible for different rewards. There’s also a system in place for two different entry points each of which have their own requirements for the rewards.

The access control works pretty sensibly imho - for example, someone with manager-level permissions can add and remove whomever they please from events to satisfy reality, but a participant-level user can only add and remove themselves. I won’t go into insane detail - just know that there’s 4 levels (Participant, Manager, Admin, Dev), where the admins can add & remove users, and the dev gets stuff that the admin wouldn’t find as interesting like reloading partial templates.

The Story#

Here’s the story of my application - a few items have been moved around to better tell the story, but it’s mostly accurate and if you want the 100% accurate version then feel free to read the Commit History.

Beginning#

This project started with the beginners guide for the axum crate (for this was my first axum project which slowly expanded in capability as I learnt more about axum & async rust), where you get started with a basic GET/POST which worked well for me. I then added a liquid based templating system inspired by Amos Wenger (specifically this one iirc), who’s writings I cannot recommend enough. I then slowly by surely added the other things like participants and photos. This part of the project was probably the second most fun, where I could move mountains with little effort - partially because I hadn’t done much yet so any change was big, and partially because I was free to look at all kinds of different approaches before I got entrenched in a pretty specific style.

I also had a few interesting constraints to work within:

  • I wanted to be able to run the whole thing on just the one tiny VPS (a Linode Nanode which has 1GB of RAM, 1 vCPUs and 25GB of storage) I rent whilst still keeping some space for other stuff (like this blog!). This was mainly an attempt to reduce both vendor lock-in as well as costs.
  • I wanted this to keep backwards compatibility with the existing spreadsheets - that meant making sure that I had lots of import/export tools.
  • I wanted this to be easily used by lots of more non-technical users - a JSON feed would never be acceptable and I needed to put lots of work in to get something looking halfway decent (a challenge even with the aid of Bootstrap).

Authentication#

I then took it to the person who actually had final say on adoption on this system and he said that I needed to follow all relevant guidelines that he had to, which in this case included making sure that you couldn’t see photos without logging in. My initial thoughts were something along the lines of balls. i don’t really want to do this, because the more user data i store the more impact my screwups have, but I wanted adoption so I just nodded and said yes. Turns out I was still right, just not in the way I thought I was.

At this point I was pretty happy with the general axum ecosystem, and I came across two main crates that seem to be used for logging in - axum-auth & axum-login. axum-auth seemed to have more downloads but was just one rust file to check headers and get them. axum-login seemed to have more features and even drumroll please - Session Management, although it had fewer downloads. Generally, I like writing things myself (as has already been mentioned, often for the learning opportunity), but if I’ve learnt one thing in my years of Tom Scott/Computerphile viewership - it’s that you leave password management to other people, preferably open source people with code that’s been explored & checked over. That having been said, it still wasn’t the easiest to work with - there were a few annoyances like having to rewrite the PostgresStore for sessions to work with my system, as well as getting passwords to work right. I started with the 0.6 version, which has recently been updated with more breaking changes than I’ve had hot dinners (future update coming - issue here). I then used bcrypt to store the hashed passwords.

One fun solution to a problem is signing up. Since I know my entire userbase in advance, I can just import them all via CSV with blank passwords set. My original solution was to just set the password to whatever the user entered on first login, before I realised that I’d have a sea of password resets as people logged into other peoples’ accounts for shits & giggles. I eventually ended up going with emailed magic links to set the password.

Eventually I got all that working, with the access control coming not long after. As I alluded to earlier, there was still a pain point. This was just managing the access control levels across the whole application - I have checks in a whole lot of places to make sure someone’s not trying to be sneaky with direct POST requests to the server and making sure that no PII leaks.

The Modern Web#

At this point though it was just something running on my machine, or just via direct HTTP from my registrar (originally Hover, but I transferred to Cloudflare for cheaper renewal fees) to my VPS (Linode seems to give a decent compromise between cost, configuration & ease of use). I needed HTTPS if I wanted people to send passwords through here - there’s nothing particularly important, and I’ve got the password hygiene not to use the same password everywhere but I can’t say that for all my users. Luckily, one of my very good friends (HandyHat) had experience here, and walked me through the entire setup of Caddy for Cloudflare using Reverse DNS to get HTTPS certificates.

You might’ve noticed that I’ve put all of my eggs into one basket - if Cloudflare goes down then I’m down. I understand the concern, but I also trust that if Cloudflare goes down then the one instance I run of Vent personally will be of little concerns.

I also chose to use a VPS here, rather than a Docker Container running on a Droplet or something - this is for multiple reasons. First is that I’m definitely not just using the VPS for this so I save the more projects I use. I also find it insanely convenient to quickly fix small issues in things like SQL queries in production without needing some complicated deployment process - I just ssh in, edit the relevant file with micro, then build the project and restart it in a tmux session I keep around.

Adoption#

Eventually, I started some user groups on the system and when nothing went wrong (apart from some misspelt usernames which were easily fixed) I went for the full rollout. Whilst some people still don’t mark themselves as planning to attend, it’s now the de-facto system and those people probably check when and where events are on the main page. I’m really happy with how it went.

No first plan survives contact with the enemy though, so I had to rapidly issue a few updates like publicly marking who uploads photos, and improving my logging infrastructure which at that point just forwarded whatever domain specific error it got from whatever caused the error via thiserror & the tracing ecosystem. You don’t want to know how long its taken to tune the logs 😔.

Fun Technical Things#

You didn’t think I’d finish here, did you? Nah - I’ve got a few fun things which I’ll add to as I remember from the making of this project.

Threads (not of the Zuck variety)#

axum has a State extractor you can use with all of your methods, and whilst from what I’ve seen most people don’t use it as a state manager and just use it as various components I’ve got it all tricked out. For example, I publish an ICS but don’t want the faff of re-making the file for every request (which definitely isn’t what I was doing for a while which caused event duplication issues 😉) so I have one persistent file that gets updated. How do I know when to update it? Every time an event gets updated in such a way that would affect the calendar, I call a function on the State which sends a blank message through a channel to the calendar updater.

I’ve also got threads to delete old sessions (hopefully no longer needed in axum-login 0.7?), and send emails to users.

Templating#

I’ve mentioned that I use liquid templating, but given no more insight than that. Currently, the partials get cached but the actual templates don’t (there’s an issue and it’s next on my list after axum-login 0.7). The main reason for this is that liquid templates are entirely synchronous and the compiler is created in an environment with no way to access async APIs other than caching them beforehand 🤷, so if ever anyone looks at my code and gets confused then that’s why.

If you look over my templates, I’ve also got a couple of invariants (like people always being ordered by group in the view all page when logged in) that allow me to commit some horrendous crimes that make me giggle every time I look at them, and sigh every time I have to think about doing something with that part.

Having now got experience with liquid, I can say that it probably wasn’t the right choice - it can be annoyingly limited to work with (something like handlebars might’ve been better in that respect), but I’m also happy to accept that it’s good enough and that maybe everything doesn’t always need to be perfect.

Gotchas for future me#

These are just a few pointers for people ever embarking on similar projects:

  • You need to setup a page rule to turn off SSL for DOMAIN/.well_known/acme_challenge/* because that prevents Caddy from setting up the certificate.
  • Linode blocks email-related ports (presumably for spam/DDOS reasons?) and doesn’t ever say clearly (just in some random blog post) so you need to contact support to get them unblocked.
  • If you’re tee-ing JSON logs into a file and then serving that file, you might need to parse that file and reverse all the elements because otherwise all the most recent logs (which are probably the important ones) are right at the bottom.
  • axum::debug_handler compiles to nothing in release mode - cover your project in them because all they’ll do is slightly increase compile times and they’ll massively help in errors.
  • Use DBeaver - it’s insanely useful for quickly editing database records and doing other database-adjacent things like complete migrations.
  • People aren’t joking about backups - you wouldn’t believe how easy it is to completely accidentally wipe something important from (or just wipe the whole) production database. It doesn’t matter how many flags you’ve setup in DBeaver.
  • Hostinger can shut off servers with too high activity - make sure to double check that doesn’t happen to you, because I didn’t get emailed about it or realise until I checked wanting to see the events and saw that production was down because Docker was being screwy.