Search for a command to run...
Infrastructure, DevOps, and why they're tedious
I'm a professional Software Developer and Developer Relations Manager.
My first love will always be Ruby but I'm writing a lot of JavaScript these days. I specialise in Automated Testing and work at Sauce Labs.
I truely believe that crafting Software is how I can best make the world better for its inhabitants, and I love educating and mentoring folks to who'd like to do the same.
No comments yet. Be the first to comment.
The mid-ness of Surprising Simplicity
In Tokyo, I fell in love with a Robot
Or, a Gentlehacker's Guide to Releasing a Component as a Library
How ADHD folks can start their year off right.
Many, many thousands of Developers have worked on data and infrastructure solutions for Northwind Traders, the speciality foods import/export group. In fact, I'd suggest that it's the most widely re-implemented system of all time (second only to devs writing ToDo apps).
Why is that? Because Northwind Traders is the fictitious company Microsoft uses for examples and education of its database products, starting way back in the Microsoft Access days. Modelling Northwind's customer/product/invoice relationships was my first exposure to normalisation, linking, and calculations.
I was thinking about Northwind today because I have another data modelling task that relies on Office-esque tooling. Microsoft Access might be long gone, but I'd argue the replacement was there all along: Spreadsheets. Simple, powerful, and basic tabular data is managed waaaaaay easier than by spinning up a CRUD instance.
That's why I used it to build what is effectively a CSV generator using Google Sheets and now, reasons which are good but not interesting., I've built an app on top of it and I need to put it somewhere.
Crucially, I need to get a SvelteKit/Sheets API/MongoDB app online, and I want to do that without having to worry about infrastructure, deployment config, or complicated environmental syncing. So, I decided to check out another North-named company, Northflank.
Northflank is a comprehensive developer platform designed for scale. It provides a slick, unified interface for all your services and projects, optimized for DevOps and fully configurable. Northflank's goal is to make managing complex cloud native applications simple, accommodating any stack, providing API/CLI/UI access, and generally providing your entire team with the chance to use your tools, instead of spending all your time supporting them.
I signed up for Northflank, pootled around the docs, and realised I only needed to do three things:
Link my repo to a new "Service" inside a "Project"
Add a MongoDB instance
Do some basic environment configuration
When setting up a new Service, I was prompted to create a project first; Since I've got exactly one user (me) and my needs aren't that complex, I happily fit within Northflank's free tier.
Then I had to do the hard thing and name it, along with choosing a colour for the icon, which is a neat touch.
[port]--[service]--[project]--[entity-dns-id].code.runRight now (September '23) Northflank supports six regions in Europe, US and SE-Asia. I was offered a choice of US Central or Europe West for my project, which I'm guessing is a free tier limitation. Australia? Never heard of it.
Next up, my app code. Northflank calls user-based code "Services", and choosing to add a new one lets me grab code from a Git repo or a Docker registry.
I could also have chosen to add background or async tasks as a "Job", databases and message queues as "Addons", or create a group of encrypted secrets... And if I wasn't sure what to do, Northflank provides several one-click deployments.
I grabbed my web app's service "frontend" because I am very creative. I allowed the Northflank app access to Github, then proceeded to stare at "Build Options" for a while.
I don't have a Dockerfile, but the "Buildpack" option implies it needs to be a Heroku buildpack, and I also don't have one of those. I am a lazy developer, though, so I decided to see if that option would work.
I left everything else default and clicked "create service", and Northflank got on with the hard work of building and deploying.
Then it fell down.
Clicking the failed build shows us our build log, so we're able to check out what happened, and what happened is that there's no configured MongoDB instance.
2023-09-12T02:02:13.866831708Z stderr F RollupError: "NF_MONGODB_MONGO_SRV" is not exported by "$env/static/private", imported by "src/lib/db.ts".
2023-09-12T02:02:13.866758961Z stderr F error during build:
2023-09-12T02:02:13.866751237Z stderr F 4: await client.connect()
2023-09-12T02:02:13.866723785Z stderr F 3: const client = new MongoClient(NF_MONGODB_MONGO_SRV)
2023-09-12T02:02:13.866701994Z stderr F ^
2023-09-12T02:02:13.866616414Z stderr F 2: import { NF_MONGODB_MONGO_SRV } from '$env/static/private'
2023-09-12T02:02:13.866607016Z stderr F 1: import { MongoClient } from "mongodb"
Which makes sense; I'm not on my local machine and I've not created a Mongo database in my Northflank project yet. Let's do that.
Adding a MongoDB instance is as simple as clicking "Create New", choosing "Addon", and filling in the blanks. My options included Postgres, MySQL, Redis, and MongoDB (along with MinIO and RabbitMQ if you're so inclined). I clicked MongoDB and added a name (as well as a name for the database within Mongo itself), and that was it. A few minutes wait and Mongo was provisioned.
However, there's a catch. Those variables are name-spaced to the Mongo instance itself; they're not yet available to other services in the project. Let's fix that, shall we?
Clicking the "Link to secret groups" button created me a new secret group. Secret groups are exactly what they sound like; a group of secrets that can be bound to services. It's nice to be able to control which services can see which secrets, instead of giving access to everything.
Northflank also allows you to control whether your secrets are available during buildtime, runtime, or both. That's a neat touch!
Northflank automatically namespaces your variables as NF_[addon_name]_[variable_name], which is a little lengthy for my tastes, so I just aliased this back to MONGO_SRV :
Northflank helpfully applied this group to all services and jobs, so I should now be able to re-run my frontend build and get...
Ahh.
2023-09-12T04:16:19.940548529Z stderr F Error [MongoServerSelectionError]: read ECONNRESET
I see.
This took some documentation diving to figure out. It turns out that your build environment is isolated from your run environment. My app is trying to connect to Mongo during the build process (for... reasons) and can't do so, because I've not made it publically available.
I can do so by going into the Mongo addon network settings and choosing "Publically accessible", which I don't love as a solution because I'd rather it be categorically impossible to contact my Mongo instance from outside the project itself, but what can you do.
Adding the variable caused the service to rebuild, and this time, it succeeded. Northflank has graciously given us a domain name and as such...
Superb.
One last piece; Because Northflank is designed for CI/CD workflows and multi-environment teams, it would be helpful to be able to access cloud resources from my local machine, for debugging, remediation, and general development.
Handily, they provide a CLI which can do just that. A little npm i -g @northflank/cli and a touch of northflank login, and I'm able to fully control my setup with the CLI.
From there, you create a context, a set of related permissions, options and resources. Contexts empower things such as providing a 3rd party tool with limited control to restart services, or separating your team and personal projects.
I want to spin up my project locally but connect to the cloud Mongo instance. I can do this with the public URL but, if I had not had to enable public access for the build step, the northflank tool would let me run sudo northflank forward addon --projectId ankiedit --addonId mongo to forward traffic to the private networking address. (Conveniently, the command to do this is pre-filled in connection settings, so you don't even have to remember what the CLI flags are.)
Northflank is quite nice. I appreciate the balance it strikes between control and convenience; The unified, team-accessible control panel is very powerful and having internal logging should go some way to providing in-app remediation. They've sunk a lot of effort into ensuring that Northflank works with your tools instead of against them, with multiple means of interaction, ample monitoring and useful in-built options.
Also, free tiers are a rare beast now, and sometimes you just want to put a wee app online, without paying $5 for a whole server or being invoiced every month for a single-digit amount of cents.
There are caveats. Free tiers have a habit of becoming "Free" trials (or simply getting cancelled), and I wouldn't use them to deploy anything you must have online unless you're prepared to pay at some point. The richness of options is occasionally confusing in a manner the documentation doesn't solve particularly well, and it's easy to leave things default because you just don't notice them.
Finally, the inability to allow access to private resources during the build step is a bit wonky; it's quite common to front-load assets, branding or content processing at build time, so it would be nice to be able to do so without giving up the security of internal-only networking.
All in all, using Northflank was fun, and impressive, and I think it's worth checking out.