"Even non-developers can use!"

"Even your Mum could do!"

"Can even be interpreted by end-users"

I call this Surprising Simplicity and it's dreadful.

Using "non" frames developers as being a special class of people, which makes your readers feel good... But it implies that people who aren't in that class aren't as worthwhile. You're typically writing as part of the in-group, the developers in question. Don't make fun of those who are not du monde; That's no more then simple bullying.

You also want to avoid the universal truth vibes given off by "Even". This form is reliant on the assumption that the group described would typically be incapaple of making use of your product. You're positioning these folks as incapable so that It feels like a win that these folks can make use of your product. If that's not true, that's kinda gross. If it is true, is that an inherent property to the problem, or to the solution? What's the difference? The latter tends to be an assumed reality; something that products make true by assuming it's true. (A much more visceral (and harmful) example of this is explained by the video ASSUME THAT I CAN released for World Down Syndrome Day 2024).

The "Mum" example also includes bonus of propagating some sexist and ageist stereotypes about women, older people, and parents. Best to skip this one entirely.

What should you do instead?

Describe things in terms of the power it affords people:

"Gives end-users flexible reporting, and developers the fine-grained controls you need"

This isn't just better for inclusivity, it's better, period. People read tech writing because they have a problem & want to not. As an author, you're aiming to be convincing in your ability to make said problem go away. By presenting your solution as a tool they can use, it invests the reader with feelings of agency and capability. A focus on surprising simplicity doesn't do this; Instead, it shifts the focus to other, apparently less capable people.

Use explicit archetypes, not implicit stereotypes

"Simpler then programming a VCR"

"Easy enough for your 'bad-with-computers' friend"

(NB: A VCR was a device for recording & playing back TV shows; like a bad-quality CD for video.)

(NB: CDs were a flat plastic disc which encoded music by laser etching; Like a offline version of Limewire.)

(NB: Limewire was...)

Archetypes help scaffold our thinking. This is why personas are useful for product decisions, user stories and marketing strategies. They're just as useful in tech writing, with two caveats.

Firstly, people tend to use stereotypes instead of archetypes. What's the difference? Let us compare

"Self-Service customers are not skilled at statistics, have short result timeframes and a tendency to escalate support tickets."

with

"Karen wants to speak to the manager again because she doesn't understand regressions."

Archetypes have their relevant traits, skills and behaviours explicitly defined. They are (or at least should be) formed from emotionally neutral observations. Stereotypes, on the other hand, are only indexical of their traits; That is, the presence of the stereotype implies the traits. Typically they do so by creating a caricature, over-emphasizing certain behaviours and ascribing them to a certain kind of person, often in ways that reflect existing prejudices and relies on emotional implications.

Secondly, when referring to archetypes by name, you risk over-reliance on internal information. I have often experienced folksrefers to a market segment or persona by their companies' internal name, to the confusion of external parties. A mid-market customer probably doesn't know they're mid-market; a vendor might not know that you use architect in a way they use business analyst. This one is easy to avoid; Make the traits and behaviours specific instead of relying on names. (We're back to indexicality.)

I think you're being too picky about semantics.

Then I invite you to consider: These forms are #techwriting#cliches. Your copy will resonate more strongly and read more fluently without them, and your audience will walk away thinking about your product, not your writing.

Additionally, we are living in the algocene, an epoch where vaugely understood advertising junkie information systems shuttle human attention around like hummingbees doing a speedrun, yet our ability to find accurate information appears worse then ever before. It is entirely conceivable that a simple "Ahoy Computer: How do I {x}" will lead one of your non-developer, four-kids-and-a-dog customers to your writing, leaving them with the gross sense that they're lesser beings; and your product with one less user.

Liked this? Hated it? Comment down below!

See the original vent over onMastodonandTwitter.

As such, it won't surprise that researchers have conducted significant research into the relative efficacy of learning practices, and reached some tentative conclusions. One such conclusion is that it can be quite a boon to have a source of both textual and audio examples of comprehensible input.

Tatoeba is one such boon, being a large, community driven database of cross-language sentences, with their respective translations and, quite often, spoken examples. Sadly, only quite often; examples are added by kind and generous community members, so may be absent for any given phrase.

We shan't let that stop us! Let us make use of ElevenLabs.io's splendid AI text to speech faculties to build a browser extension that generates missing audio on demand. Given ElevenLabs' excellent articulation, cadence, and emotional inflection, along with the ability to stream audio via HTTP, our challenge lies primarily in wiring our apparatus together.

The Setup

ElevenLabs Access

Get an API Key

You shall need an API key and a chosen voice. Easily obtained; Sign up for their free plan thusly. (This is an affiliate link, as the GentleHacker is not above securing additional funds, should his recommendations prove useful.)

Make note of the API key, which you may find in your profile; The button to the bottom left.

Choose a Voice

ElevenLabs has a wealth of voices to choose from. Should you wish to use a community-created voice, you can peruse the voice library. Once chosen, click "Add to VoiceLab".

That done, open the Voice Lab and click the "ID" losenge to copy it to your clipboard.

Alternatively, if you'd prefer an official voice, that is... less simple. Those are found in the Speech Synthesis section. Once you've found a voice you enjoy, you can either use the /getVoices method of the API to find the ID, or you can open the developer tools, generate a speech sample, and grab the voice_id from the path of the POST made to /api.elevenlabs.io/v1/text-to-speech/voice_id_is_this_bit .

This is a little inelegant, but I was unable to find a superior method. We prevail.

(Oh, and by the way; I recommend choosing the Eleven Multilingual v2 model; the documentation claims it is of slightly lower quality but the non-English output is more verisimilar to reality.)

The Web Extension

Web Extensions are little more then a bundle of webpages, along with guidance to the browser about how they ought be used. There's an ersatz standard. The browsers somewhat follow it.

As such we could hand-write some HTML, some Javascript, a smidgen of CSS and a skerrick of JSON and have ourselves a functional extension.

We could. But no.

It is 2024 as I pen this missive, and I enjoy the conveniences of Svelte, and Vite, and TypeScript far too stridently to hand-carve my extension. Some brief research revealed vite-plugin-web-extension, by Aaron Linker. A brief npm create vite-plugin-web-extension and it provided the shell of a functional extension, with the ability to use Svelte components and rely on Vite's hot module reloading. Delightful.

I also installed tailwind-css, along with flowbyte-svelte, (a component library unfamiliar to me; I deemed this a good opportunity to play), and the webextension-polyfill, to provide missing web extension methods in Chrome

npx svelte-add@latest tailwindcss
npm i -D flowbite-svelte flowbite webextension-polyfill

Fetch and Playback of Speech

But first, a brief detour for configuration

As one might imagine, ElevenLabs requires that one identify oneself when making API requests. We require some means by which our user can provide said API key, and the Options UI, combined with the Web Extension storage API, seems the perfect way to do so.

In our manifest.json, we can specify the URL to a page to present as part of the extensions options. We also need to request access to the storage API for the domains we intend to make use of it upon.

Storing our API Key

Options.html loads our Options component, which, despite appearance, is a fairly simple affair. Here's the meat of it:

When the user enters their key and clicks "Save", we take the following steps:

1. Get the key from the DOM

2. Validate the key by requesting this user's details from the API, providing the key via the xi-api-key header

3. If successful, generate a truncated version of the key, and store both the full and truncated versions in local storage as eleven_labs_key and truncated_eleven_labs_key respectively.

This approach provides the user with two conveniences. Firstly, it allows the user immediate knowledge of whether their key is operational (and thus, that any bugs are due to the extension). Secondly, displaying the truncated key allows users to check they've already saved the correct key.

Retrieving our API Key

When we wish to use our API Key, we can retrieve it from storage. However, we also need to keep our code abreast of changes to said key, in case our user should replace the key whilst our other scripts are already loaded. Let us make use of storage.onChanged.addListener to do just that.

Now, we may fetch our speech

Selecting a Generation Method

The ElevenLabs API provides three methods for generating speech.

Firstly, the Text-to-Speech API operates in a traditional HTTP fashion, generating all audio data then returning it once complete.

Secondly, the Text-to-Speech Streaming API begins streaming audio data as soon as it's available, allowing for faster playback (especially for longer audio passages). This is what we shall make use of.

Thirdly, the Websockets API provides real-time audio responses over a, well, WebSocket. It is intended for situations where the input is not entirely available up front, or where word-to-audio concordance is required.

Calling the API

Let us package the code for retrieving and playing audio data in elevenAPI.ts. We start with data retrievall; as fetch in modern browsers can provide a readableStream, it is eminently suitable for our purposes.

There are three parameters of note. We provide optimize_streaming_latency on line 9, which (according to the documentation) provides us with 75% of the maximum latency optimization, at the cost of "some" quality. I consider this fair; Language is not always heard in ideal situations!

The output_format on line 10 indicates how our audio shall be encoded. We choose PCM, as this is the closest to what the Web Audio API consumes. More on this topic later. Should we care to, ElevenLabs could also furnish us with MP3 or μ-law, all at a range of sample rates.

Lastly, we choose the eleven_multilingual_v2 model on line 20, as this model supports the widest range of languages; ElevenLabs will determines which language is in use automatically, which is rather handy!

Note also, the retrieval of our apiKey on line 5. Because we must await the storage API, we are unable to load the key when this module is imported, due to the harsh realities of Top Level Await. To avoid calling the storage API needlessly, we only do so if apiKey is not already defined, using the ternary operator.

Playback

The Vexation of Audio Formats

The question of playing out audio back brings up a slight quibble. A vexation. One might call it inconvenience, perhaps even a botherance. Were we using the non-streaming API, we'd convert our data into a blob URL and provide it to a HTML5 <audio> tag. We, brave hackers, opted for streaming instead and, as such, will be receiving a series of chunks of audio in lieu of one complete file.

I apologize in advance, gentle reader, for the number of times you are about to read the word chunk.

We rely on the auspices of the Web Audio API to make use of our chunks. It furnishes us with the ability to play individual buffers of audio, albeit with one futher peevishment. Recall when I mentioned that the PCM data returned by the API is the closest to that required by the Web Audio API? Well, closest here won't do. The Web Audio API requires PCM to be encoded as:

(signed) 32-bit linear PCM with a nominal range between -1 and +1, that is, a 32-bit floating point buffer, with each sample between -1.0 and 1.0

Whereas the Elevenlabs API returns signed, 16-bit integer PCM, whose values are not between -1 and +1.

Compounding our difficulty, the ReadableStream is Uint8, an 8-bit unsigned integer TypedArray, meaning we shall have to first convert it to Int16, signed 16-bit integer and thence to Float32, signed 32-bit float.

Converting the response to Float32

Firstly, our type signature and read loop. Our function will take the incoming stream, which is a ReadableStreamDefaultReader. Because we generate this within a try/catch block in doFetch, there's chance it will be undefined.

We are also providing a Function called fn, which takes a Float32Array and can return anything. This function will be called once for each decoded chunk, allowing us flexibility in how we manage the decoded data.

Once the reader indicates that our stream is complete, we exit the loop.

Our data processing is managed in the section marked // Omitted for brevity, which is supplied below.

As we read each chunk, we encounter another vexation of streaming. Our 16-bit data is encoded as an array of 8-bit responses, meaning each 2-byte value is split into two 1-byte values. Furthermore, there is no guarantee that each chunk contains an even number of bytes.

This necessitates we keep track whether we have a leftover byte or not, every time we process a chunk. If so, the next chunk needs to have said byte appended at the beginning. Frustratingly, it's not possible to add elements to a TypedArray, so we must resort to using the spread operator to create a new array containing the old one.

Once we've dealt with vexatiously odd chunk lengths, we need to deal with our signing issue.

The DataView we've created on line 16 can see all of the data in the chunk's underlying buffer, and allows us to call getInt16 to read any two bytes, in any position in said buffer, as though they were a signed, 16-bit integer.

Because we're consuming two bytes to generate each value, the resulting Float32Array only needs to hold half as many elements. We can then convert the Int16 values to Float32 values (Which we will discuss shortly) and that's that.

Why the Divide?

You may be wondering, why half as many, instead of one quarter?

Think of the conversion thusly. Say you download a 10x10 image from an API, which returns an array containing each pixel's red, blue, and green value, one after the other, until a full 300 element array is in your possession.

Once you've fetched your array, you convert it into the corresponding hex representation; You now have one hundred elements where previously you had three. Say you now need to feed this image into one which supports hex with opacity. You don't have opacity values, so you convert your values and default to "full opacity" (ff). Each individual value now has more data, but you do not have more values.

When we converted our audio data, we were doing something similar. To go fromUInt8 to Int16, we required two unsigned bytes to combine and decode into one, two-byte long signed sample. We then have to convert each sample to Float32 to please our Web Audio API master. Because signed 16-bit values range from -32768 to 32767, and we need a range between +1 and -1, we can convert our Int16 values to Float32 by dividing by 32768. Each sample has more (empty) data but the number of samples remains the same.

Using the Web Audio API for Playback

The Web Audio API api possesses a helper method called decodeAudioData(), which plays back an AudioBuffer obtained from the likes of fetch... Alas, it requires the full file be ready, and so is not suitable here. We shall have to make our own pudding, as it were.

All online Web Audio processing takes place within an AudioContext, which represents a graph of audio processing nodes. Should one be not averse, one can use the Web Audio API to perform a substantial array of audio generation and processing. Handily, we are spared such complex manipulations.

Our playback handler needs merely create an AudioContext and return a function which will do naught but play back each chunk at the appropriate time. Said function can then be passed to processStream().

As processStream() receives each chunk, it requests a single-channel AudioBuffer of the appropriate length and sample rate from the AudioContext. (The 1 in the call to createBuffer() indicates how many channels the buffer should possess).

The chunk's data is then copied to the first channel therein, and the entire buffer encapsulated within an AudioBufferSource (also kindly provided by the AudioContext). That AudioBufferSource is then connected to the AudioContext's destination, which, by default, is the system audio output. Then, it's scheduled to start.

Timing in the Web Audio API is interesting, as it's driven by the API itself. Each AudioContext has a currentTime, a Double measured in seconds. Each source can be started at any given positive time, also in seconds, and if the requisite time has already started, the source begins immediately.

This furnishes us with an extremely simple means of scheduling, as well as providing an optional delay. We can simply start each sample at 0 + delay + already_queued_samples. This delay also acts as a simplified cache; delay the start of playback by a short period to allow extra audio to be streamed.

Some Interface Niceties

Connecting doFetch(), playHandler() and processStream() is a straightforward process; we invoke doFetch() with our desired text and voice, pass the streamReader to playHandler(), then pass the returned function to processStream(). Our audio now streams without further intervention.

After adding some nice defaults, I also added a method to return the entire audio data, as well as both stream and return the audio. These are left as an exercise for the reader.

Enhancing Tatoeba.org

Finangling the Front End

Our work thus far is, sadly, of no import if we do not provide our user a means of requesting a sentence be generated. Let us now turn our attention to the same.

Web Extensions allow one to inject scripts into the body of webpages. They are called content scripts and may be injected either programmatically, or by specifying the pages in advance. We return to the manifest and instruct our extension to insert src/addMissingVoices.ts on every page of the tatoeba.org domain:

The Lists Page

I chose to restrict my initial version to Tatoeba's lists feature, whereby users can save lists of sentences. Taking a look, we see the greyed-out, muted volume icon indicates the absence of a spoken example.

We are thus presented with two challenges:

1. Extract all sentences within a list for which no example exists

2. Provide a user interface action allowing them to generate said example on demand

Let's tackle each in turn.

Whither Missing Examples?

In days of yore, a developer might reach for old friend JQuery when interrogating the DOM. We wish to keep our extention's code light and fast, so we shall rely on built in facilities whereever possible.

By peering through the DOM using developer tools, I determined each sentence is contained within a element whose class is sentence-and-translations. By finding all such elements, I could determine whether an audio example was missing by checking whether it contained an element whose aria-label attribute was volume-off.

For each relevant sentence, I would need to collect the actual text in question (by finding a ng-if attribute set to !vm.sentence.furigana, oddly). I also choose to store the sentence ID (a element with an ng-href attribute, contained within md-subheader-content), as my future self might make use of it for caching purposes.

Armed with my CSS Selectors, I set off to achieve the above.

(GentleHacker sidenote: Whenever I can use map I feel like a Wizard. This may express deep secrets about my psyche and I don't wish to dwell on it).

Shiny new buttons

Now we know what's missing, we can begin to compensate for said absence. This comprises constructing new buttons, adding a eventListener for click, and inserting them into the appropriate place in the DOM.

Now, you might be asking yourself, why use browser.runtime.sendMessage to call the API, instead of calling it directly via elevenAPI.streamAudio()? That is an excellent question and the answer, as it so often does, comes back to CORS.

Whereas previously, content scripts were afforded an exemption from labouring under the harsh realities of Cross-Origin-Resource-Sharing, modern browsers consider such practices... distasteful. Thankfully, we have a workaround: background scripts.

Background Scripts: The Intermediary

Background scripts load in the background, as you might imagine, and have access to the full gamut of WebExtension APIs. They can either persist, or load in response to certain events; The latter, non-persistent scripts will eventually be the only kind.

As such, it behoves us to write our script in a non-persistent manner, which is not at all arduous. It merely requires that listeners are top-level, and we avoid relying on global variables.

To whit:

Our manifest contains some new syntax! One advantage of vite-plugin-web-extension is that it allows us to define manifest.json in terms of the browser we're targetting. On Firefox, background scripts are simply called scripts, whereas on Chrome, they're called service_worker. Thankfully, it's trivial to account for both.

Speaking of trivial, background.ts isn't particularly impressive. It polyfills the browser APIs for that rascal Chrome, then adds a listener for messages and, if one is received, fires off streamAudio.

Triumph!

Click one of those buttons, gentle reader, and ElevenLabs will spin into action, generating you a lovely audio example. Marvellous.

Thank You, Gentle Reader

We have accomplished what we set out to do! We can access the ElevenLabs API and generate audio where missing, and we've done so in a such a way that we can easily extend it further.

Were I to spend more time working on this extension, I'd consider things such as:

• Using IndexedDB to cache responses

• Pulling down the full list of voices as part of the options, and allowing users to choose them in-app, on a per-language basis

• Adding a translation challenge mode, where the listener hears a random sentence, then a pause, and must translate it themselves before hearing the "Correct" translation.

• A universally available popup, allowing users to check selected text on any webpage to determine whether it's available on Tatoeba.org and, if so, hear a foreign translation in the language of their choice.

But these are tasks for another day.

Thank you again, gracious friend, for your time and attention. If you found this helpful I would sincerely appreciate a follow, like or comment. Sharing this article with your peers helps me greatly, as does following me on Mastodon, at https://tech.lgbt/@TheGentlehacker.

Ta-tah!

One of the glories of the Hypertextual medium is the ability to enrich one's messages with links. Indeed, Hyperlinks (to give their proper address) to websites, images, video and such are the very fabric of the Internet, although not without drawbacks.

Sometimes, links may lead to content a reader is already familiar with, or does not wish to consume, and as such it is not surprising that one might wish to provide a preview of the object to which a link pertains.

Sadly, this is not a native faculty possessed by The Internet at large, which may surprise given the common appearance of previews in apps such as Slack:

How, then, are link previews generated?

You might suppose the process involves using a headless browser to take screenshots, or interrogating the HTML to guess the details. Thankfully, this is not the case; often, the required information is supplied via the Open Graph Protocol which

enables any web page to become a rich object in a social graph.

A simple, dependency-free protocol defined under the Open Web Foundation Agreement, Version 0.9, the Open Graph Protocol (henceforth the OGP) allows page authors to mark up specific metadata to help other sites provide information about the linked content.

This, I'm sure you'll agree, is much easier then finangeling a headless browser. Let us make use of it!

The makeup of the Metadata

The OGP is a fairly simple affair, relying on <meta> tags embedded into the <head> tag of a response. Which is appropriate, as this is indeed what said tag is for.

Open Graph describes resources as objects and provides some common properties from which to obtain information. Each property name starts with og: to indicate it belongs to the Open Graph namespace. Four properties are required, and seven are considered optional (but generally recommended).

Required Properties

og:title has the object's title as it should be displayed when embedded.
og:type details what type of object this page contains (such as website or image).
og:image is a URL to an image which should be used as part of the preview.
og:url provides the canonical URL for the object in question.

This is what the Required Properties might look like for a site detailing Professor Eliza Williams' research into Difference Engine Temperature Compensators:

Optional Properties

Provisions are also made for additional information; Knowledge is Power after all.

og:description is a one-to-two sentence description of the object in question.
og:site_name gives the name of the larger website an object belongs to, if any.
og:locale indicates which linguistic locale the tags are marked up in.
og:locale_alternate offers an array of other locales available.
og:determiner supplies the word which should fall before this object's title in a sentence ("the", "an" et al).

og:audio provides a URL to an audio file to 'accompany' this object.
og:video advises a URL to a video file that 'complements' the object.

(Mostly straightforward, although you might be curious why audio files accompany but video files complement, and what, precisely, those verbs are intended to convey. I am also curious; if you know, I beseech you to leave a comment.)

Professor Eliza is a Modern Woman, so her site includes all of the above:

Further Types of Object

The website type is perhaps the most common object embedded with the OGP, but the protocol includes many other object. These objects, in turn, have their own structured properties, namespaced as og:type_of_object. For instance, the image type makes the dimensions of an image available as og:image:width and og:image:height.

The full list is available at the source; We shaln't discuss it here in the interests of brevity.

So what shall we build?

Let us build a link preview component for SvelteKit which takes a URL and renders for us a lovely preview of the associated content, using the OGP metadata embedded therein.

For ease of use, we shall use TypeScript to help ensure correct data management. The completed component should be pleasing to the eye and behave well different resolutions.

We shall, at first, restrict ourselves to displaying only the basic metadata; Whilst the support for extra content types is robust, we do not, for the first attempt, need the extra complexity.

We shall also make an assumption which, while unkind, is not unfair. Developers are only human and may, at times, make mistakes, causing them to omit some or all of the required fields. As such, our component will not assume that the provided metadata is correct.

Our Requirements are thus:

• Display the basic image along with the title, and provide a link to the canonical URL

• Exhibit a responsive, fetching design

• Allow for incorrect metadata

• Provide developer friendly interface, inc. Typescript.

In future entries, we shall endevour to provide more useful default behaviour; enrich our support for divers content types, and package our component as an installable library which can be distributed at will. Such fancies, however, shall have to wait; For now, our target is a component that will look like the following when used in situ:

That looks rather fetching, don't you think?

Getting Ready.

To begin, I've created a basic SvelteKit site posessing an assortment of pages, each professing a different kind of OpenGraph object. The site uses images from UnSplash as preview images and is otherwise fairly unremarkable.

Having said that, the index will serve as a gallery showing how the finished component will render for diverse types, so it may prove of some interest. You can find it at https://ogcomponentdemo.gentlehacker.codes/ and the source lives at https://github.com/DylanLacey/OGComponentDemo.

Onwards!

In my next missive, we shall implement the basis of our component, and briefly discuss the correct architecture of SvelteKit applications.

Secondly, don't buy that new planner/productivity tool/todo app. Seriously. We both know it won't help; They're not for you. They're for the NTs with their boring, functional executives.

You're a Sorcerer Bard; you can't equip Paladin gear. You need different tools and different spells for your adventure. Herein I shall do my best to identify some. With (some) citations! Lovely.

The TL;DR

1. Take your pills; Eat; Exercise; Sleep.

2. Body Doubling

3. Pomodoro

4. Digital todo schedules, not lists

5. Get a small success in first thing

6. Break tasks into tiny, well defined tasks, as needed

7. Be pessimistic in time estimates

8. Make it your own

Potions and Training

Sourcery requires mental focus. Bards need stamina. You need to drink your Potion of Focus (be it Ritalin, Dex, or something else) and make sure you eat, sleep, and exercise, before you're ready to set off on your quest.

It's Dangerous to Go Alone

Arrange a body double; Ideally someone who will do the work of organising said doubling!

Is it efficacious? Despite being widely practised, significant research is yet to be conducted. Dobrowolski believes that body doubling introduces external pressure which, conversely, helps keep the ADHD learner engaged (Dobrowolski, 2022). Eagle et al found that

many people were unfamiliar with the term but had intuitively been using the strategy (Eagle et al, 2023)

Which suggests we're participating because we've found it helpful. Clever us!

Short Rests

I started a linguistics degree in 2021 (because that's a sane thing for a Software Engineer employed full time to do) and part of the course on how to be successful at language learning made us try the Pomodoro technique. I hated it. It also worked really well.

Is it efficatious? One of the best ways to get us going is strong incentives (Dovis et al, 2021). Pomodoro provides two such incentives. Firstly, they function as a deadline, providing urgency (found to be helpful in a work environment by Lasky et al, 2016). Secondly, you get a reward every 5 minutes, when you can do whateverthefuck you want! The method includes built in termination times, so you don't hyperfocus into oblivion.

Use Specters, not Scribes

You will forget the planner. I assure you. But, you must still keep track of every tedious task and need in some fashion, and that fashion should be digitally, with the spirits of the cloud.

You cannot lose the cloud.

Furthermore, you should avoid having todo lists and try having a todo schedule. If something is important enough to do, it is important enough to schedule time for. If it isn't, you probably won't remember it anyway.

Please don't take that pointed tone with me. I don't say it to be cruel.

Long Rests

We suffer greatly when forced to estimate exactly how long it will take to ride from Candlekeep to Waterdeep, or how the amount of time we need to mend that damnable gauntlet that keeps snagging. Be pessimistic. Double estimates. Record how many Pomodoros it takes to do something, so you'll have a good idea for next time.

Stab the Goblin. Squishy, Squishy Goblins.

Start each day with a simple, easy to accomplish task, so you gather XP at the first step. If it's something you enjoy, it's like getting some loot, as well.

Perhaps you don't have a simple, easy to accomplish task? I very much doubt that, but in that case, what is the smallest possible thing you could do to progress a larger task? You might not be ready to slay the Wyvern, but you can ask at the tavern if anyone's lost any sheep, lately. Turn the bigger task into a pile of Goblins as you need too, rather than all at once.

Write your own spells

You may, of course, need more systematic structure. Conjure it yourself! Any system with rules from outside will make you feel stifled and stuck.

Try to construct the most minimal system you can, and feel free to change it up. We get bored easily, and productivity systems can be dreadfully dreary. Your system should be friviolous, and rewarding.

A grand experiment

This year, I am trying something new myself, that being themes. I have no evidence this will work, but I shall attempt to tackle shame by committing to anything that isn't for the furthering of my themes. I am still free to dabble and drift, but shall only be focusing, when concerned, on two areas:

1. Career - I shall launch the app I have been working on, on and off, for two years now.

2. Health - Heavy things, the repetitive lifting and lowering of said.

Good luck on your quest!

References

Dobrowolski, S. (2022). Attention Is Not the Solution: Experiences of University Students With ADHD With Remote Learning (Bachelor's thesis, University of Twente).

Dovis, S., Van der Oord, S., Wiers, R.W. et al. Can Motivation Normalize Working Memory and Task Persistence in Children with Attention-Deficit/Hyperactivity Disorder? The Effects of Money and Computer-Gaming. J Abnorm Child Psychol 40, 669–681 (2012). https://doi.org/10.1007/s10802-011-9601-8

Eagle, T., Baltaxe-Admony, L. B., & Ringland, K. E. (2023, October). Proposing Body Doubling as a Continuum of Space/Time and Mutuality: An Investigation with Neurodivergent Participants. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1-4).

Lasky, A. K., Weisner, T. S., Jensen, P. S., Hinshaw, S. P., Hechtman, L., Arnold, L. E., ... & Swanson, J. M. (2016). ADHD in context: Young adults’ reports of the impact of occupational environment on the manifestation of ADHD. Social Science & Medicine, 161, 160-168.

Why is that? Because Northwind Traders is the fictitious company Microsoft uses for examples and education of its database products, starting way back in the Microsoft Access days. Modelling Northwind's customer/product/invoice relationships was my first exposure to normalisation, linking, and calculations.

Yes, and?

I was thinking about Northwind today because I have another data modelling task that relies on Office-esque tooling. Microsoft Access might be long gone, but I'd argue the replacement was there all along: Spreadsheets. Simple, powerful, and basic tabular data is managed waaaaaay easier than by spinning up a CRUD instance.

That's why I used it to build what is effectively a CSV generator using Google Sheets and now, reasons which are good but not interesting., I've built an app on top of it and I need to put it somewhere.

Crucially, I need to get a SvelteKit/Sheets API/MongoDB app online, and I want to do that without having to worry about infrastructure, deployment config, or complicated environmental syncing. So, I decided to check out another North-named company, Northflank.

What is Northflank?

Northflank is a comprehensive developer platform designed for scale. It provides a slick, unified interface for all your services and projects, optimized for DevOps and fully configurable. Northflank's goal is to make managing complex cloud native applications simple, accommodating any stack, providing API/CLI/UI access, and generally providing your entire team with the chance to use your tools, instead of spending all your time supporting them.

Getting Started

I signed up for Northflank, pootled around the docs, and realised I only needed to do three things:

1. Link my repo to a new "Service" inside a "Project"

2. Add a MongoDB instance

3. Do some basic environment configuration

Project Setup

New Project

When setting up a new Service, I was prompted to create a project first; Since I've got exactly one user (me) and my needs aren't that complex, I happily fit within Northflank's free tier.

Then I had to do the hard thing and name it, along with choosing a colour for the icon, which is a neat touch.

Right now (September '23) Northflank supports six regions in Europe, US and SE-Asia. I was offered a choice of US Central or Europe West for my project, which I'm guessing is a free tier limitation. Australia? Never heard of it.

New Service

Next up, my app code. Northflank calls user-based code "Services", and choosing to add a new one lets me grab code from a Git repo or a Docker registry.

I could also have chosen to add background or async tasks as a "Job", databases and message queues as "Addons", or create a group of encrypted secrets... And if I wasn't sure what to do, Northflank provides several one-click deployments.

I grabbed my web app's service "frontend" because I am very creative. I allowed the Northflank app access to Github, then proceeded to stare at "Build Options" for a while.

I don't have a Dockerfile, but the "Buildpack" option implies it needs to be a Heroku buildpack, and I also don't have one of those. I am a lazy developer, though, so I decided to see if that option would work.

I left everything else default and clicked "create service", and Northflank got on with the hard work of building and deploying.

Then it fell down.

Clicking the failed build shows us our build log, so we're able to check out what happened, and what happened is that there's no configured MongoDB instance.

2023-09-12T02:02:13.866831708Z stderr F RollupError: "NF_MONGODB_MONGO_SRV" is not exported by "$env/static/private", imported by "src/lib/db.ts".
2023-09-12T02:02:13.866758961Z stderr F error during build:
2023-09-12T02:02:13.866751237Z stderr F 4: await client.connect()
2023-09-12T02:02:13.866723785Z stderr F 3: const client = new MongoClient(NF_MONGODB_MONGO_SRV)
2023-09-12T02:02:13.866701994Z stderr F             ^
2023-09-12T02:02:13.866616414Z stderr F 2: import { NF_MONGODB_MONGO_SRV } from '$env/static/private'
2023-09-12T02:02:13.866607016Z stderr F 1: import { MongoClient } from "mongodb"

Which makes sense; I'm not on my local machine and I've not created a Mongo database in my Northflank project yet. Let's do that.

Adding MongoDB

Adding a MongoDB instance is as simple as clicking "Create New", choosing "Addon", and filling in the blanks. My options included Postgres, MySQL, Redis, and MongoDB (along with MinIO and RabbitMQ if you're so inclined). I clicked MongoDB and added a name (as well as a name for the database within Mongo itself), and that was it. A few minutes wait and Mongo was provisioned.

However, there's a catch. Those variables are name-spaced to the Mongo instance itself; they're not yet available to other services in the project. Let's fix that, shall we?

Environment Configuration

Add a Secret Group

Clicking the "Link to secret groups" button created me a new secret group. Secret groups are exactly what they sound like; a group of secrets that can be bound to services. It's nice to be able to control which services can see which secrets, instead of giving access to everything.

Northflank also allows you to control whether your secrets are available during buildtime, runtime, or both. That's a neat touch!

Northflank automatically namespaces your variables as NF_[addon_name]_[variable_name], which is a little lengthy for my tastes, so I just aliased this back to MONGO_SRV :

Rebuild using Secrets

Northflank helpfully applied this group to all services and jobs, so I should now be able to re-run my frontend build and get...

Ahh.

2023-09-12T04:16:19.940548529Z stderr F Error [MongoServerSelectionError]: read ECONNRESET

I see.

This took some documentation diving to figure out. It turns out that your build environment is isolated from your run environment. My app is trying to connect to Mongo during the build process (for... reasons) and can't do so, because I've not made it publically available.

I can do so by going into the Mongo addon network settings and choosing "Publically accessible", which I don't love as a solution because I'd rather it be categorically impossible to contact my Mongo instance from outside the project itself, but what can you do.

Adding the variable caused the service to rebuild, and this time, it succeeded. Northflank has graciously given us a domain name and as such...

Superb.

Connect to our Dev Environment

One last piece; Because Northflank is designed for CI/CD workflows and multi-environment teams, it would be helpful to be able to access cloud resources from my local machine, for debugging, remediation, and general development.

Handily, they provide a CLI which can do just that. A little npm i -g @northflank/cli and a touch of northflank login, and I'm able to fully control my setup with the CLI.

From there, you create a context, a set of related permissions, options and resources. Contexts empower things such as providing a 3rd party tool with limited control to restart services, or separating your team and personal projects.

I want to spin up my project locally but connect to the cloud Mongo instance. I can do this with the public URL but, if I had not had to enable public access for the build step, the northflank tool would let me run sudo northflank forward addon --projectId ankiedit --addonId mongo to forward traffic to the private networking address. (Conveniently, the command to do this is pre-filled in connection settings, so you don't even have to remember what the CLI flags are.)

Thoughts

Northflank is quite nice. I appreciate the balance it strikes between control and convenience; The unified, team-accessible control panel is very powerful and having internal logging should go some way to providing in-app remediation. They've sunk a lot of effort into ensuring that Northflank works with your tools instead of against them, with multiple means of interaction, ample monitoring and useful in-built options.

Also, free tiers are a rare beast now, and sometimes you just want to put a wee app online, without paying $5 for a whole server or being invoiced every month for a single-digit amount of cents.

There are caveats. Free tiers have a habit of becoming "Free" trials (or simply getting cancelled), and I wouldn't use them to deploy anything you must have online unless you're prepared to pay at some point. The richness of options is occasionally confusing in a manner the documentation doesn't solve particularly well, and it's easy to leave things default because you just don't notice them.

Finally, the inability to allow access to private resources during the build step is a bit wonky; it's quite common to front-load assets, branding or content processing at build time, so it would be nice to be able to do so without giving up the security of internal-only networking.

All in all, using Northflank was fun, and impressive, and I think it's worth checking out.

Well, the second is still imminent. I shall be travelling to San Francisco to present at QConf, 'pon the topic: "Are Programming Languages... actually Languages?". A brief exploration of Second Language Acquisition, and what it can teach us about upskilling our teams. I am Quite Excited About It.

Nevertheless! Onward and upward.

For Developers

Be Productive

Abi Noda has summarised a paper from Meyer et al detailing what makes for a Good Developer Workday. Good days are those with an appropriate amount of collaboration, where the developer feels they were productive and added value, and most of all:

Good Days go as planned.

Store Things in the User's Browser

There are many, unique and wonderful APIs in hidden in the depths of Chromium and Friends. One of those is IndexedDB, a client-side API for storing structured data on the client side. This can be a most efficacious way of storing data such as, say, search indexes, document drafts, or infrequently-updated reference data.

The GentleHacker suggests perusing the Web.Dev introduction, should you wish to learn more.

Or perhaps not!

Randel Degges concurs, partly because he believes that one of the alternatives, that of LocalStorage, is always a bad idea. You can read his take here; it's excellent nuanced.

Be secure

It is a sad fact that the internet is awash with rapscallions, vagabonds and charlatans of the highest water. As such, the clever girls and boys who work in technology have come up with a multiplicity of techniques for ensuring the safety and security of you and your customers.

One such technique is Subresource Integrity, which allows you to specify a cryptographic hash for each resource you retrieve, keeping you safe from MITM attacks on CDNs and such.

How clever.

With Thanks

Should you have enjoyed this edition of The Week That Was, I would greatly appreciate you sharing with your fellows, or a Subscription or Follow over on My Web Log.

Should you wish to find previous entries, you shall find them all collected here.

As you may have read in the previous entry in this series, our Session Management API shall use DynamoDB for data storage, and, specifically, will not be using Redis. The reasons for this are many, but the most pertinent one is cost. For more on this, consult your nearest Cloud Architect.

(Yes, this is Step One. The previous post is The Introduction. Should you wish to follow along with the code as a whole, it's over on GitHub.)

Let the Dynamo spin!

Create a new Table

My SST Stack configurations already has a DynamoDB table called MarukiTable; let's add another one, called MarukiSessionTable:

This table has a standard partition and sort key, along with a Global Secondary Index (w/ requisite keys), along with an expiry column. That column is being marked as the timeToLiveAttribute. DynamoDB allows you to configure a column whose value represents the expiry time (in Unix epoch time) of that item. DynamoDB will automatically run background processes to identify and remove expired items, at no cost.

(Because erased data is written to DynamoDB Streams, this feature is useful for a myriad of things, including removing subscriptions, censoring data after specific times, and billing.)

Provide table access to functions & site

I'll also need to give my functions access to my new table, as well as making its name available to my Static Site (Which I'm doing with sst-env, which is not germane to our discussion):

Why am I creating a separate table for sessions, instead of keeping things in the erstwhile Single Table realm? Paranoia, Dear Reader, paranoia. Whilst DynamoDB won't erase anything whose TTL exceeds 5 years in the past (and thus my data is safe even should blank entries evaluate to 0), I simply don't like the risk. It would be all too easy for me to, with a moment's inattention, write erroneous code whose execution would add contemporary expiry dates to my entire table. I shan't risk it.

Define Session Data Layer

I am using the delightful DynamoDBToolbox to work with my session data, along with DayJS for time management and ulid for ... ulids. Let's install those now, by adding them to our package.json:

In my functions directory, I have a data subdirectory, where we shall now scribe our table definition:

The only notable thing here is the table name; I like to provide an obviously mistaken fallback option. That can help short-circuit debugging when you accidentally omit environment variables; It also stops TypeScript complaining that undefined is not string.

Next, we need to create a session Entity. We'll need a unique partition key, along with a way to store the authorized party (the principal) and what mechanism authenticated them. We additionally need a timeToLive value so that automatic session expiry takes place, and it might be worthwhile storing the originating IP of the authentication event. Finally, everyone loves a created/modified timestamp, so we'll add those as well:

DynamoDBToolbox is providing us automatic creation and modified timestamps, as well as helping us fill in some of our data. We're prefixing the session & principal IDs so the data makes a bit more sense when we look at it "raw", lines 21-29. On line 32, we set a default value for expiry, using dayjs to set a value one hour in the future, in the required Unix Epoch Time Format. Finally, we make the authorization mechanism and origin required values.

Add a Global Secondary Index

One thing we're missing is the ability to look up all sessions for a specific principal. DynamoDB stores data such that data of a specific nature is stored in a specific partition, inside of which individual rows are sorted (hence the key names). Ideologically, DynamoDB expects that users (in this case, our application) know what nature of data (and thus which partition and thus, partition ID) they wish to retrieve data from. As such, almost every operation expects you to provide partition key, with the exception of Scan. The latter scans and returns every item in the table sequentially, optionally filtering items after retrieval but before responding. Scan is not the fastest operation. Generally speaking, don't use Scan.

Scan is not the fastest operation. Generally speaking, don't use Scan.

Currently, we can retrieve a session if we know the ID (and we should, since validating IDs is this code's entire balliwick), but finding all sessions for a user would require a Scan, as we don't know the session ID (and thus partition key) on which to query.

We can solve this by adding data into our Global Secondary Index (GSI) fields. GSI's contain a subset of attributes from the table and support the Query operation. They have their own, unique partition key and sort key, which we can instruct DynamoDBToolbox to populate:

We're setting these attributes to hidden because they're not unique; They're simply duplicating data on the table. Each has a prefix, again, to make the raw data clearer, and then we're setting their value with a function.

By querying the table's GSI for Principal|some_principal_id, we can retrieve all sessions for a principal at once, allowing us to (for instance) show them to the user, or bulk invalidate them.

(We could also consider making an index that uses authorizationOrigin as the partition key. That way, we could easily see every authorization request from a specific IP address.)

And that's that.

We're ready to write the more finicky bits; Those actively managing sessions. But firstly, dear reader, let us take a break. We shall resume in Step Two!

(Should you enjoy this content, or wish to be notified of the publication of updates, I humbly request you provide me with a Follow and a Like. Of such actions are Viral Content made, and I wish only to help as broadly as I might.)

I'm currently building a tool to help developers track and share what they're working on, provisionally called Maruki. Because it's a tool for developers, I want it to have both a full-featured API and a front end. I also want to avoid as much infrastructure management as I can, and as such, I'm building a Serverless architecture, using the lovely SST toolkit.

Maruki uses Lambda to access a DynamoDB powered data-layer (with what I hope is a good first attempt at Single Table Design), along with a Sveltekit powered front-end. Working in Sveltekit has been lovely, with one slight caveat.

Session Management in Sveltekit has been the very dickens.

Because it's relatively new, Sveltekit is undergoing rapid changes. I had just got a naïve implementation of session management working when this update was released, necessitating major changes throughout the app, including session management. This, coupled with the paucity of documentation (owing to the new changes) has meant that my experience of Session Management in Sveltekit has been the very dickens.

Still, I have the basic "Send Login Get Cookie Nom Nom Nom" flow working, and now need to actually authenticate users & construct unique sessions.

The problem we're solving

I need to support both API and website access, so I might as well have a unified mechanism. I want it to be serverless, scalable, and as hands-off as possible.

For now, users will have a username and password, but in the future I'd like to allow alternative access methods. Users are already identified in my system with a unique ID that's separate from their email and username. I'd also like to have a non-password authentication system for my API.

Sessions should expire at some point, and I'd like to be able to force-expire them myself.

I'd also like to have things be well tested and monitored, because Knowing is Half the Battle, or so I hear.

How we're solving it

Design

Let's implement traditional, server-side sessions, using API Gateway Lambda Authorizers to enforce access requirements and DynamoDB for session storage.

Website users will log in with their username and password, and receive an associated cookie for access. Sessions will expire every week or so. Reasons for the "Or So" are detailed below.

API users will generate a request token using their API Key. That token will expire every hour (or so). This helps prevent replay attacks, and also saves us a bit of compute (by virtue of only running our access-key computation once per generation).

Components

API Gateway

API Gateway is Amazon's solutions for scalable, controllable API proxying. We're making use of it's ability to dispatch HTTP requests to an underlying Lambda function.

Jeff Bezos and Investors tour API Gateway

API Gateway Lambda Authorizers

Built into API Gateway is the ability to add a Lambda function as an Authorizer to any route. This function receives either a bearer token or some configurable subset of the request being made, and needs to return an IAM Policy in return.

Using an authorizer function gives me a few things. By denying access before even hitting my functions, it provides me with a level of abuse protection. I can implement complicated access patterns because I'm returning an IAM Policy, and have the flexibility to use whatever authentication mechanism I want. Plus, the result is automatically cached.

Lambda

AWS Lambda is "a compute service that lets you run code without provisioning or managing servers". Basically, we upload stand-alone pieces of code which, when needed, are executed in a controlled environment, then terminated. There is no additional management required. It's wonderful.

DynamoDB

Yes, I could use Redis as an in-memory Session Store, but the pricing doesn't make sense; at the lowest cost I'd be paying nearly 40c/hr to keep Amazon MemoryDB online, which is enough to pay for 1.5 million DynamoDB reads or a Gb/Month of storage. Even using Elasticache would cost us 1.6¢/hr. It's not a fair comparison because you have to consider all usage costs.... But even a back of envelope calculation puts DynamoDB far ahead of the alternatives.

But what about the speed I hear you ask! In practical terms, I've observed DynamoDB response times for my authentication logic are consistently below 20ms. Additionally, I feel that tiny delays in areas like security and "calculation" activities make users feel more reassured.

Typescript

I profess no special TypeScript knowledge, nor do I claim to be demonstrating the "right" way to achieve any particular goal. I am merely a dilettante! However, I do find type checking helps keep some of Javascript's more egregious peccadillos in check.

Next up

I think our plan is sufficient reading for now! Should you wish to follow the rest of this series, please be so kind as to Follow me here, or over on Twitter.

(Ruby Layout) provides the rendering model and formatting controls related to the display of ruby annotation. Ruby annotation is a form of interlinear annotation, consisting of short runs of text alongside the base text

That clarified nothing

I know; This one requires a bit of background. I promise you'll at least learn something, and you may find a new technique for enhancing your CSS Designs with:

• Pronunciation guides
• Short, snarky comments
• Translations
• Synonyms

So I'm a fool -- Pandemic Hobbies edition

When the Pandy got its hooks in, many people took up a new hobby, mostly baking Sourdough. I already baked Sourdough because I'm insufferable, so I went to the next logical step: acquiring a Linguistics degree.

Well, I must clarify; acquiring a Diploma of Japanese I subsequently decided to expand into a full three-year Linguistics degree majoring in Japanese, while still working full time.

Japanese is hard

This has been ... stressful. Japanese is considered one of the most difficult languages for English speakers to learn, primarily because of their writing system.

Japanese writing

Japanese has an alphabet called Hiragana, which is a syllabary, that is, a set of characters tell you how to pronounce a word. Once you can read Hiragana, you can pronounce any word in the Japanese language. Compare this to English, where the pronunciation of "live" depends on whether you mean "not pre-recorded" or "not die".

(Thank you Wikipedia)

Japanese also has Katakana, which is basically a 1:1 equivalent for Hiragana, used for writing foreign loan words (amongst other things).

(Despite being called syllabaries these two scripts technically don't express syllables but instead mora, which is a timing-based unit that isn't germane to this discussion. If you want to learn more, leave a comment and I'll write another article!)

Japanese also has a logographic (picture based) writing system called Kanji which they kind-of borrowed wholesale from China. Actually, they're complicated enough to deserve their own H1.

Why are Kanji so hard?

1. Because there's so many

Seriously, tens of thousands

One of the definitive dictionaries, the Dai Kan-wa Jiten, contains over 50'000 characters.

Lies. No-one could learn that many.

You're right. Japanese students learn Kanji all the way through graduating high-school, gradually getting through the 2136 jōyō kanji, the official Government list of characters required for functional literacy in Japanese.

But wait, there's more!

Wikipedia points out that there are roughly 1000 additional, commonly used characters (including those for names) as well as some specialised industry characters.

2. Because they're not always read the same way

Sound Readings & Meaning Readings

Because Kanji were borrowed from China, they already had a meaning and a pronunciation in Chinese. The Japanese had their own words for most of the meanings, so the characters could be read with the Japanese word for the Chinese reading. However, the Japanese also decided to import the pronunciations, adjusted for their syllable structure.

Just to spice things up further, the Japanese also didn't import meanings wholesale; Some characters had their meanings adjusted.

For instance, the character 田 is Middle Chinese for Field, pronounced /den/. In Japanese, it specifically means Rice Paddy, not field, and so not only can it be pronounced den, it can also be pronounced ta.

But wait, there's still more!

More readings, that is. See, languages aren't static things; they change. As China's pronunciation of their characters changed, Japan imported the new pronunciations... and kept some of the old ones!

Not only that, but the meanings of characters differed as well, depending on which region of Japan you're in. A character might mean "Shovel" in one region, and "Rake" in another, and over time, both meanings spread nationwide... Which led to Japanese having multiple sound readings and multiple meaning readings for each character.

So how do you tell readings apart?

Context, mainly. Let's look at an example! This is the Kanji for Electrical Appliances:

Those are the kanji for Electricity, Child, Manufacture and Goods.

The kanji for Child shows up in lots of words, pronounced in several ways:

As does the Kanji for Goods:

That is stunningly unhelpful

Yes!

There are some general rules that can help, which I'm not going to cover here. Roughly speaking, there's one primary reading of each type which is commonly used. The structure of the word (Kanji alone, Multiple Kanji with no Kana, etc) can suggest which readings to use. And then it comes down to knowing the word and recognising it in context (Just like you do for English!)

So what does this have to do with CSS?

Japanese Children aren't fully literate until they graduate Highschool

Remember when I said that Japanese study Kanji right up until graduation? That means that, functionally speaking, for the first 18 years of their life the Japanese can't read all of the characters they're likely to encounter.

In fact, given the large number of Kanji, the existence of technical vocabulary and rare characters and such, it's unlikely that most adults in Japan know every Kanji, even if we restrict ourselves to just those used in the last 100 years.

Why this isn't a problem

The Japanese speak... Japanese

This gives them a huge advantage over a foreign leaner. Even if a Japanese child doesn't know how a Kanji is read, they can use their language knowledge and the other Kanji in a sentence to guess.

Take Denshiseihin, the "Electrical Goods" I referenced before. If a Japanese child knows the kanji for Electricity (Which is also used in Phone and Light), the Kanji for Child can be pronounced as "shi" in "boushi" (Meaning Hat) and Goods can be pronounced "hin" in "shokuhin" (Foodstuffs), and has heard the word "Denshiseihin" before, they can make an educated guess that the unknown character is "sei*".

They Cheat with Furigana

If you've read any Japanese, you might have noticed that some Kanji words are printed with Hiragana above them. This is called Furigana and indicates to the reader how a word is pronounced:

These are common in children's texts, and also appear in Newspapers and print where an average adult reader might not be familiar with a character.

This is a lot

You're right, it is a lot, and I've still not come to the point. Yikes.

Tell you what; Subscribe to me here on Hashnode, and you'll get notified when Part 2 comes out, and I finally explain what all of this has to do with CSS. It'll be good, I promise.

Rhetorical question; Tagging is great; my favourite organisation system, in fact. Test systems with tagging allow you the flexibility to organize your tests according to what code they're testing, while running tests according to what kind of test you're after.

Tagging & Cypress

Unfortunately, Cypress lacks native support for tagging, and this makes me very sad face emoji.

So, I'm going to use saucectl to retrofit tagging functionality onto our Cypress tests, whether you're running locally in Docker mode, or remotely against Sauce Labs' cloud.

Wait, what's saucectl?

Let's go to the README!

The saucectl command line interface orchestrates the tests in your framework, providing rich parallelization, test history filtering, and analytics in Sauce Labs.

Basically, saucectl is an amplifier for your tests. It drags your Cypress, Playwright, TestCafe or Puppeteer kicking and screaming onto the Sauce Labs cloud, giving you delicious Sauce features (like analytics) along with all the goodies your framework has to offer. Saucectl can also run your tests with Docker, giving you a quick and reliable test environment. It's a great solution for using Cypress with CI.

(Disclosure: I work for Sauce Labs. I still really like saucectl.)

Our Approach

In config.yml (the config file for saucectl), you can specify individual suites see here which let you provide individual config options to Cypress.

By passing the testFiles option to Cypress, you can control which individual files it runs. We're going to combine these two, to give us individually named test suites, each of which runs tests with a specific set of tags.

Can't you do that with Cypress alone?

Yes, technically, but I prefer this approach for... let's call them ideological reasons.

While you could have individual config files, and call cypress run --config-file somesuite.json for each one, it's messy. If you make changes to one suite file, you have to make it across all of them, which is easy to forget.

With this solution, all your Cypress config stays as is, and only the tests being executed are changed. Even better, control of which tests run is maintained in the infrastructure configuration, and infrastructure is likely to have the most significant impact over what 'flavours' of test you want to run.

How we're implementing tagging

This solution makes use of globbing, a way of selecting a group of files. It's pretty common; Common enough to have a Wikipedia page, and you've likely seen at least one glob before.

We're going to alter our tests so each test suite is configured to run a set of globs corresponding to our tagged tests.

Example

Suite Files

Say this is our test suite. Each file contains tests that should only run on a specific browser (or combination of browsers). We've named them so each filename includes the relevant tags, separated by underscores, like _chromeOnly or _ieOnly. These could just as easily be _ui or _FrenchLanguage or anything you want to tag by.

specs/
  ui/
    login_chromeOnly_spec.js
    login_firefoxOnly_spec.js
  accounting/
    jsMathBugFix_ieOnly_spec.js
    chromiumCantCount_edgeOnly_chromeOnly_spec.js

Config.yml

Here's the suites component of our config.yml. Each suite has a unique glob for testFiles which finds all tests in your specs folder which include the respective tag/s.

  suites:
    - name: "Where There's Smoke There's"
      browser: "firefox"
      platformName: "Windows 10"
      config:
        testFiles: ["specs/**/*firefoxOnly*_spec.js"]

    - name: "Cream on"
      browser: "chrome"
      platformName: "Windows 10"
      config:
        testFiles: ["specs/**/*chromeOnly*_spec.js"]

    - name: "Livin' on the"
      browser: "edge"
      platformName: "Windows 10"
      config:
        testFiles: ["specs/**/*edgeOnly*_spec.js", "specs/**/*ieOnly*_spec.js"]

How it works

Each glob starts with specs/**/*. This glob means "start in the specs folder (specs/), then look in any subfolder(**), for any characters at all(*) followed by an underscore(``)".

Let's look at how the first glob continues. After specs/**/* we have firefoxOnly*_spec.js. This means "_find anything containing the phrase firefoxOnly, followed by zero or more of any character(*), followed by _spec.js_". For our tests, that only matches specs/ui/login_firefoxOnly_spec.js.

The second glob is specs/**/*chromeOnly*_spec.js, so it matches every file in every subfolder of specs which contains chromeOnly, followed by zero or more of any character, followed by _spec.js. In our case, that's specs/ui/login_chromeOnly_spec.js AND specs/accounting/chromiumCantCount_edgeOnly_chromeOnly_spec.js

Did you notice?

The testFiles parameter doesn't take a glob directly; it takes an array of globs. That lets us pass in multiple entries, which is how suite three works. We're able to pass in two entries, one for any test tagged edgeOnly and one for any tagged ieOnly. It will match any file in either of those blobs; in this case specs/accounting/jsMathBugFix_ieOnly_spec.js and chromiumCantCount_edgeOnly_chromeOnly_spec.js.

Additionally, test files can have multiple tags. This is really useful when you want to run some tests in multiple scenarios; Say you have a limited set of functional tests that are also used to ensure deploys go well; You could tag them _functional_postDeploy_spec.js and they'd run when using either the _functional tag or the _postDeploy one.

And that's it!

A small change to how you name and select tests, and you're left with a more flexible testing system. And of course, you can still run a default suite with no tags, and make sure you're running everything.

Happy Testing!

Credits

Photo by Angèle Kamp on Unsplash