Frontend Ramblings feed

Shell function to show line in file with context

Wed, 02 Jul 2025 00:00:00 GMT

Shell function to show line in file with context

Today I wrote a shell function to display lines in a file, with optionally some surrounding lines for context. This can come in handy when looking at output like stack traces including files and line numbers and you want to see what's in that file on that line.

Built-in IDE terminals often feature clickable links, this line function is for when you don't have that luxury.

Usage

$ line
Usage: line <file> <line_number> [lines_around=0]
       line <file:line_number[:column]>
       cat <file> | line <line_number> [lines_around=0]

The following examples are all equivalent and show line number 10 of file.txt with 2 extra lines before and after:

$ line file.txt 10 2
$ line file.txt:10:5 2
$ cat file.txt | line 10 2

The default context is 0 lines.

Script

Add this function somewhere in your $HOME/.bash_profile (or equivalent):

line() {
  local FILE LINE_NUMBER LINES_AROUND=0
  local NAME="${FUNCNAME[0]}"

  if [[ ! -t 0 ]]; then
    LINE_NUMBER=$1
    LINES_AROUND=${2:-$LINES_AROUND}
  elif [[ $1 =~ ^([^:]+):([0-9]+)(:[0-9]+)?$ ]]; then
    FILE="${BASH_REMATCH[1]}"
    LINE_NUMBER="${BASH_REMATCH[2]}"
    LINES_AROUND=${2:-$LINES_AROUND}
  else
    FILE=$1
    LINE_NUMBER=$2
    LINES_AROUND=${3:-$LINES_AROUND}
  fi

  if [[ -t 0 && -z "$FILE" || -z "$LINE_NUMBER" ]]; then
    echo "Usage: ${NAME} <file> <line_number> [lines_around=0]
       ${NAME} <file:line_number[:column]>
       cat <file> | ${NAME} <line_number> [lines_around=0]"
    return 1
  fi

  if [[ -t 0 && -n "$FILE" && ! -f "$FILE" ]]; then
    echo "${NAME}: $FILE: No such file or directory"
    return 1
  fi

  sed -n "`expr $LINE_NUMBER - $LINES_AROUND`,`expr $LINE_NUMBER + $LINES_AROUND`p" ${FILE}
}

TIL

What I found interesting:

Use -t 0 to check if stdin (file descriptor 0) is connected to a terminal, and if negative it means non-interactive and thus piped (or redirected, tbh I haven't dug into that)
BASH_REMATCH is the equivalent of, for example, this in JavaScript:
```
const BASH_REMATCH = str.match(regex);
```
FUNCNAME[0] holds the function name of itself
That sed always seems to have a new trick up its sleeve. The result in the script with sed + expr above is, for example, sed -n "8,12p" which shows lines 8-12 of the input.

In closing

Obviously I've added line to my dotfiles!

Beyond Git aliases: git clone + cd

Thu, 19 Jun 2025 00:00:00 GMT

Beyond Git aliases: git clone + cd

When a Git alias doesn't provide enough scripting powers, here's a clean way to go beyond them.

This is a little technique I cobbled together trying to clone a GitHub repo and cd into the created directory in a single go, but couldn't find an easy way using a Git alias alone.

The first step is optional. For a clean approach, create an alias to hook into later. In your global Git config:

[alias]
	c = clone

To cd into the latest modified directory we'll create a little function. This is useful on its own so let's store it separately:

cdl() {
  cd "$(ls -dt */ | head -n 1)"
}

Now to bring it all together, add this function somewhere in the .bash_profile (or what have you):

function git() {
  if [ $1 = "c" ];
  then
    local url="$2";
    [[ "$url" =~ ^[^:]+/.+$ ]] && url="git@github.com:${url}.git"
    command git clone "$url" "${@:3}" && cdl
  else
    command git $@
  fi
}

This only affects git c and leaves everything else alone, including git clone.

From now on, git c webpro/venz will clone the repository and cd into it right away. Any valid Git URL will still work as expected.

Privileged, but not a clown

Fri, 28 Jun 2024 00:00:00 GMT

Privileged, but not a clown

This is a story in 3 parts about OSS and me: how it started, how we got here, and how it's going.

Part 1. How it started

View Source

Soon after the first "View Source" in Internet Explorer 5.0, and copy-pasting a snippet from a <SCRIPT> tag into some text editor, I've started using free and open source software.

PHP, MySQL, phpMyAdmin, JSLint, Prototype, jQuery and Lodash come to mind. Using and understanding this free software taught me a lot about programming, and allowed me to start a career as a developer.

Enthusiasm

Remember Apache Ant, Closure Compiler or YUI Compiler? It's what I've used to bundle and minify CSS, JS and HTML, using XML! This little tool became Jaguarundi 13 years ago, and now I'm sitting here realizing I've always had this relentless enthusiasm for developer tooling.

Movement

Around 2009, projects like jQuery and Lodash got me into both freelancing and open source. I remember working for Backbase at the time, but also looking at this OSS movement. I wanted to use it, be part of it, build cool things with it! Alongside heroes like John Resig (jQuery) and John-David Dalton (Lodash), many developers in the OSS community were doing their part. Some genius work got pushed and it was all available right there in the open! I've always enjoyed solving problems with code, and more and more browser APIs and libraries became available to do just that.

Being a freelance frontend developer getting to use the latest & greatest OSS in my work. This was (and still is!) great, nothing to complain.

Open

I love coding, I love sharing. I'm coding in the open since 2010. As a self-taught developer I've learned a ton from reading other people's source code. From pushing code and receiving honest feedback about it. I still like to think some of it has once been helpful or inspirational to someone in one way or another.

Act of kindness

There's a little story I'd like to share here. Kerim Satirli, an early adopter of release-it, reached out to me about 6 years ago. He wanted to send me a gift as an appreciation of my work, and sent me two delicious bottles of beer! I'll never forget this wonderful act of kindness. I mean, sending swag and all is cool, but this hit different. Sometimes we almost forget there are actually human beings on both ends. Thanks, Kerim!

Win

And just this morning, Jonas Felix reminds me of his feature request for reveal-md earlier this year. Support for Mermaid diagrams in Reveal.js slides is useful to other users of reveal-md as well, and they've sponsored my work on this. A win for everyone, thanks Jonas!

Privilege

Pushing code. Honestly, I didn't mean to earn anything with any of this. My projects aren't extraordinary, I'm a developer in a safe and rich country, and so many other developers deserve at least the same. Being involved in OSS is simply a great way to keep learning and stay hirable with an open profile, which is an invaluable privilege in itself.

Part 2. How we got here

Return the favor

Let's imagine for a moment you're a developer working in a commercial setting. You're adding a valuable open-source package to package.json. The pull request gets merged, and you or someone else on the team returns the favor to the creator of the dependency on behalf of the company. Sounds fair enough, right?

In contrast with more traditional, commercial software, this transaction usually doesn't happen, while no-one is accountable. Businesses love permissive licenses, this is why open-source became so popular in the first place. No matter how you look at it, exploitation it is. Feels like an odd thing to say, knowing eventually there's always humans on both ends. That's why we need to normalize companies paying for OSS.

Does your company have a budget for FOSS?

Yes, that's "budget" and "FOSS" in the same sentence. If the software is of value in a commercial setting and creators or maintainers indicate support is welcome, then by all means, please provide it if you can.

Like myself, many of you probably prefer support from companies over that from fellow developers. Because the latter is mostly a redistribution of hard-earned income (let alone tax-over-tax deductions).

Sustainable

There's something sadly ironic about developers trying to get hired elsewhere, just to sustain their work on OSS. If we talk about sustainable open source, this isn't it. Some of those brilliant minds are potentially much more effective and happy when working on their own projects used by many others, compared to being an IC for a single company.

Initiatives

Lots of awesome people and companies are improving the situation around open-source funding. By financially supporting people and projects, hiring developers to let them work on (their own) open source, and initiatives like OpenCollective, Tidelift, Polar, Tea, and so on. I'm glad to see the needle is moving.

Incentives

Some metrics in use by such initiatives are based on the number of downloads or the number of dependencies. SMART perhaps, but such metrics can be abused and don't necessarily represent the actual value of a project. Sometimes popular projects become even counter-productive to the OSS community as a whole. We need different incentives to better distribute funds.

Bills

The number of dependent projects, GitHub stars, npm downloads, contributors and so on are useful metrics. They're good to get a sense of usage and popularity. Grateful people publicly and privately praising your work and contributing to your work is important and a great motivation overall. These types of input might often be fulfilling for quite some time. But eventually, the harsh reality is that none of this pays the bills.

AS-IS

Now let's imagine for a moment you're a developer and excited your work is on GitHub. It's open, it's visible, and it seems useful to others. The moment other people start using your work an interesting dynamic starts to happen. Feedback, bug reports and feature requests begin to trickle in. Open-source licenses come with an "AS-IS" clause and no SLAs, so remember: you don't owe anything to anyone. Yet chances are you feel a sense of connection and responsibility.

Trust

In fact, trust is essential to OSS and most developers tend to build a lot of trust working in the open. In many ways: by being responsive to questions and feedback, by adding automated test suites, by not breaking things unexpectedly, and so on. This requires time and energy. And some of of that feedback might be ungrateful and negative, consuming even more energy.

Space

Dealing with negativity is another interesting dynamic. How do you handle it? Sometimes you feel the urge to address negative feedback immediately and get over with it. Other times, you should get a good night's rest first and avoid responding in a way you might regret. I try to remind myself such decisions are made in the space between stimulus and response, and understanding I'm in control here makes all the difference.

Dealing positively even with the most unconstructive negative feedback is a superpower. It's just another data point highlighting a potential weakness in the project. Why return negativity immediately, if you can find the space for a positive response instead?

You are not your code

As a developer, I think this is one of the best principles to live by. Bad code will get better. The more important question is: does it solve a real problem? If it's useful to you, then that's great already. If it's your ambition to reach more people and if it solves a problem they're having, then that's fantastic. You put in the work and hopefully you'll reap the rewards.

Only you get to decide what the rewards actually mean to you

Stories

Rewards can come in many shapes or forms. But whatever constitutes low rewards to you is what fuels burnout. Here are some recent stories I've read about burnout and exploitation in the OSS community:

Unfortunately this is just a tiny tip of the iceberg. I'm grateful Artem, Anthony, David and Michael shared their stories. So many projects that their creator poured their energy in, but for one reason or another didn't get the recognition they deserved or the support they needed.

Part 3. How it's going

To be honest, I'm not even sure what I'm getting at with all of this. Is it idealism? Ignorance? Profit-impaired? I'm still not sure what to call it. Whatever it is, my vision on things slowly started to change. I feel like I can and want to contribute and share in a more sustainable way than I'm used to, and I guess that's what I'm trying to figure out these days.

Stability

My most-downloaded project is Release It! Over a decade old and tons of cool projects use it daily to publish their packages. There are plenty of alternatives, but apparently its stability and feature set is still appreciated by plenty of people. So I'll happily keep maintaining it, it's not going anywhere.

Knip

The thing you might know I can hardly contain my excitement about these days is Knip. I love working on Knip, because it solves a real problem I have in projects myself. Finding dead code and unused dependencies in an automated fashion is an invaluable tool in my belt. Another reason I enjoy working on it is that it's a right-sized project for me with still many elements I can learn from and improve. This can't possibly go without saying Knip has really great contributors and I'm receiving financial support from wonderful backers.

Reward

The greatest reward I can get is fellow developers sharing they got rid of dead code in their codebase, and how they've added Knip to their CI workflow. I will never get enough of those little red blocks on GitHub for deleted code (specially when done using Knip). It's that relentless enthusiasm over developer tooling, I guess!

Cycle

At some point Knip will have a successor, or become obsolete. That's the cycle, and that's how it should be. Until then, my goal is to keep raising the bar and just keep doing my part.

Trying

Principles to live by and stories about burnout in the OSS community are important to share and have taught me a lot.

I try to only pour in the amount of energy I can afford to lose

Fortunate enough to not have experienced a burnout myself, I feel like this is going well. I won't let anyone trick, threat or nerd-snipe me into anything more. Unless it's a critical issue, that fix can usually wait.

There are days I couldn't care less about silly edge cases. Other days, bugfixes arrive in your node_modules faster than you can say "Node.js is actually pretty damn good".

Clown

As with so many things, there's a tipping point where the law of diminishing returns kicks in, and sometimes I can't help but feel like a clown.

Since I believe the value of my contributions to OSS is too far off from what I'm receiving in monthly sponsorships, I'll sure be looking for more funding in one way or another. Or rethink what the heck I'm doing here.

Care

Coding is great, the OSS community is great. I can't imagine not sharing and learning, and I want to keep doing my part. Rest assured I'll be taking care of myself along the way. Writing this article was a great step!

Using subpath imports & path aliases

Sun, 25 Feb 2024 00:00:00 GMT

Subpath imports and TypeScript path aliases are useful and convenient features, especially in large codebases. Both are pretty widely supported across runtimes and bundlers for the web. However, the situation is different in more "vanilla" setups when using the TypeScript compiler (tsc) directly.

This article is about using subpath imports and path aliases with tsc. Specifically, we're going to discuss two pitfalls when compiling to JavaScript for a runtime like Node.js:

Subpath imports are less well-known, and not fully supported in IDEs before TypeScript v5.4.
TypeScript path aliases are not supported by Node.js.

tl/dr; See the recommendations and closing note at the end.

Subpath imports

Subpath imports are configured in package.json, They're a runtime and dependency-free option to use aliases. Here's an example import with a hash specifier:

import { add } from '#utils/calc.js';

Internal subpath imports are configured in package.json like so:

{
  "name": "my-lib",
  "version": "1.0.0",
  "imports": {
    "#utils/*.js": "./lib/utils/*.js",
    "#sub/*.js": "./lib/sub/path/*.js"
  }
}

Using *.js in subpath imports configuration is essentially the same as **/*.js in glob patterns, so it recurses into subdirectories.

Make sure to check out the Node.js → subpath imports documentation for more features, such as conditional exports.

Problem

Support for subpath imports in package.json has been in TypeScript since v4.5, so tsc compiles them just fine. But the TypeScript Language Server did not fully catch up until v5.4.

Solution (option 1)

Upgrade TypeScript to v5.4.0+ and use only a single subpath "imports" configuration in package.json:

{
  "imports": {
    "#utils/*.js": "./dist/utils/*.js",
    "#sub/*.js": "./dist/sub/path/*.js"
  },
  "scripts": {
    "build": "tsc"
  },
  "dependencies": {
    "typescript": "5.4.0"
  }
}

(Install typescript@beta until latest is 5.4.0 or higher.)

TypeScript will resolve paths properly and prioritize the aliases with auto-import suggestions in your IDE. Here's an example of how that looks like:

Pros:

Supported natively by Node.js (since v12.19.0/v14.6.0) and fully supported in TypeScript since v5.4.0.
Subpath imports can make use of conditional exports.

Cons:

Syntax is restricted to what subpath imports support:
- The #hash specifier syntax must be used (not @ or ~).
- The "#/*" alias is invalid, but as short as e.g. "#@/*" is valid.

If you have path aliases configured in tsconfig.json you'd need to replace them with subpath imports across your codebase.

If this is not an option for you, let's discuss some alternatives.

TypeScript path aliases

Path aliases are a similar feature to subpath imports. Here's an example configuration for the TypeScript compiler:

{
  "compilerOptions": {
    "paths": {
      "~/utils/*": ["./src/utils/*"],
      "~/sub/*": ["./src/sub/path/*"]
    }
  }
}

Problem

The TypeScript compiler (tsc) does not rewrite import specifiers, so they're still the same when compiled to JavaScript:

import { add } from '~/utils/calc.js';

However, this syntax is not supported during runtime in Node.js, resulting in an error:

$ node index.js
Error [ERR_MODULE_NOT_FOUND]: Cannot find package '~' imported from [...]/index.js

Solution

We have some options here:

Switch to subpath imports with TypeScript v5.4.0+
Build time resolution
Runtime resolution

Option 2: Build time resolution

You can use path aliases and tsc-alias to convert them after the fact to relative paths in the output that tsc generates:

{
  "scripts": {
    "build": "tsc && tsc-alias"
  },
  "dependencies": {
    "tsc-alias": "1.8.8",
    "typescript": "5.3.3"
  }
}

{
  "compilerOptions": {
    "paths": {
      "~/*": ["./src/*"],
      "~such/path/*": ["./src/much/wow/*"]
    }
  }
}

Pros:

Use path aliases as supported by TypeScript.
No duplicate configuration.
No performance hit during runtime.

Cons:

Requires a dependency (e.g. tsc-alias).

Option 3: Runtime resolution

Other solutions work at runtime. A popular option is tsconfig-paths.

After compilation with tsc you can use a dependency like tsconfig-paths as a loader to convert the import paths during runtime:

node -r tsconfig-paths/register main.js

Pros:

Use path aliases as supported by TypeScript.
No duplicate configuration.

Cons:

Requires a dependency (e.g. tsconfig-paths).
Requires injection of a loader via command line or in code.
Small(?) performance hit during runtime.

Recommendations

1. Relative paths

Your safest bet is to use no subpath imports or path aliases at all.

2. Subpath imports

Second best is to use only subpath imports (option 1), if supported by other tooling in your project such as TypeScript, test runners and code linters. The Node.js and Bun runtimes do support it.

3. Path aliases + build time resolution

And if that's not an option yet, I'd recommend to use path aliases with build time resolution (option 2). This is fairly well supported across tooling today. There's no runtime performance hit, and no risk of running the code in an environment that has no support.

Check out the documentation of your tooling to see what's supported.

Closing Note

Subpath imports are perhaps less well known and less used today compared to TypeScript path aliases, but likely to become even more of a standard in the future. So subpath imports are generally recommended over path aliases going forward, especially considering support in TypeScript v5.4 has fully caught up.

Resources

The State of Benchmarking in Node.js

Thu, 21 Dec 2023 00:00:00 GMT

The State of Benchmarking in Node.js

Benchmarking becomes more important as we build more and more applications and tooling for runtimes like Node.js and Bun. This article is about macro and micro benchmarking, and explores options we can use today. The article includes code examples and a CodeSandbox to try and implement in your own applications.

Macro benchmarking
Micro benchmarking
Conclusion

Macro benchmarking

Benchmarking a part of your application code is an important scenario. While running a real-world application, how many times are expensive functions called and how much time is spent in each? These are essential metrics for any CPU intensive code such as bundlers, compilers, linters, formatters, and so on.

Not many of those tools are using the node:perf_hooks module, while much of this native module is available since Node.js v8.5.0 (released over 6 years ago). This included performance.now(), performance.timerify() and PerformanceObserver, and the built-in module has been improved and extended ever since. This module allows to integrate all sorts of performance timings right into your application.

Ecosystem

There aren't that many libraries or runners on top of the Node.js built-ins. I sure hope I'm missing something, but here are some data points at the time of writing:

1 package found for npmjs.com/search?q=timerify
4 packages found for npmjs.com/search?q=PerformanceObserver
32 hits for npmjs.com/search?q=perf_hooks
a few hits for twitter.com/search?q=perf_hooks

I think there's room for tooling in this space to reduce boilerplate and improve accessibility. For instance, it would help a lot if we could import a utility to wrap functions in any application, and render or return metrics about the wrapped functions as needed.

Example: PerformanceObserver for functions

Let's look at an example which logs each recorded function invocation with a PerformanceObserver instance:

const fnObserver = new PerformanceObserver(items => {
  items.getEntries().forEach(entry => {
    console.log(entry);
  });
  fnObserver.disconnect();
});
fnObserver.observe({ entryTypes: ['function'] });

function myFunctionUnderTest() {
  // Such intensive, very cpu, much wow
}

const wrapped = performance.timerify(myFunctionUnderTest);

wrapped();
wrapped();
wrapped();

This will log three PerformanceEntry objects and one of the properties is duration:

$ node observer.mjs
PerformanceNodeEntry {
  name: 'myFunctionUnderTest',
  entryType: 'function',
  startTime: 20.211291000247,
  duration: 0.02987500000745058,
  detail: []
}
PerformanceNodeEntry {
  name: 'myFunctionUnderTest',
  entryType: 'function',
  startTime: 20.426791000179946,
  duration: 0.0017919996753335,
  detail: []
}
PerformanceNodeEntry {
  name: 'myFunctionUnderTest',
  entryType: 'function',
  startTime: 20.432208000682294,
  duration: 0.0006669992581009865,
  detail: []
}

Now we have a basis to record the number of calls to a function and the duration of each call.

Note: this article focuses on the function performance entry type. Other valid entryTypes include mark, measure, http, net and dns. See the Node.js docs on perf_hooks for more details and examples.

Example: timerify application code

Expanding on this idea, I was hoping there would be utilities to make this easier and more accessible, but unfortunately I didn't find much.

Since Node.js provides the building blocks, I created this Performance.js class in Knip last year. I've been meaning to turn this into a separate module to be published, but haven't got around doing this.

For this article I created a modified version to try and play with. The code is in this CodeSandbox.

Run the demo from the terminal inside the sandbox:

node index.js --performance

Here's the gist of it, again with the PerformanceObserver class and performance.timerify() function as the main building blocks:

import { performance, PerformanceObserver } from 'node:perf_hooks';
import EasyTable from 'easy-table';
import { parseArgs } from 'node:util';

export const timerify = fn => (isEnabled ? performance.timerify(fn) : fn);

export class Performance {
  constructor(isEnabled) {
    if (isEnabled) {
      this.startTime = performance.now();

      this.fnObserver = new PerformanceObserver(items => {
        items.getEntries().forEach(entry => this.entries.push(entry));
      });
      this.fnObserver.observe({ entryTypes: ['function'] });
    }
  }

  getTable() {
    const entriesByName = this.entries;
    const table = new EasyTable();
    // ..build table..
    return table.toString().trim();
  }

  getTotalTime() {
    return this.endTime - this.startTime;
  }

  async finalize() {
    this.endTime = performance.now();
  }
}

And here's how to use it in any real-world application:

import { setTimeout } from 'node:timers/promises';

const fnA = setTimeout;
const fnB = setTimeout;

const wrap = fn => (isEnabled ? timerify(fn) : fn);
const wrappedA = wrap(fnA); // (1) Wrap functions
const wrappedB = wrap(fnB); // to get metrics when called

async function myApplication() {
  await Promise.all([wrappedA(100), wrappedA(200), wrappedA(300)]);
  await wrappedB(500);
}

// (2) Installs PerformanceObserver#observe({ entryTypes: ['function'] }) to observe functions
const perfObserver = new Performance(isEnabled);

await myApplication();

await perfObserver.finalize();
console.log(perfObserver.getTable());
console.log('Total running time:', prettyMs(perfObserver.getTotalTime()));

perfObserver.reset();

After running this, here's some example output:

$ node performance.mjs --performance
Name  size  min     max     median  sum
----  ----  ------  ------  ------  ------
fnA      3  101.18  300.59  200.70  602.47
fnB      1  502.24  502.24  502.24  502.24
Total running time: 804ms

The functions are only wrapped when using the --performance flag. Without the flag, the functions are not wrapped and there is no overhead.

Micro benchmarking

Benchmarking arbitrary code in isolation is important too. Sometimes you want to benchmark and compare two or more ways to do the same thing. Paste some code, let it ramble for a bit and see results. There's plenty of options available to do this in a browser, but what about Node.js and other runtimes?

We have options console.time() and performance.now(), but there's some boilerplate and ceremony involved to get results.

And we shouldn't have to worry about things like process isolation, state resets between runs, external conditions, turbulence, and aggregating numbers to yield statistically significant results.

For some more serious benchmarking, we'll need something better.

Ecosystem

Node.js was pretty close to having a built-in node:benchmark module. In November 2023, a pull request to add an experimental node:benchmark module to Node.js core was opened. And closed, after an interesting debate.

This leaves us with a diverse set of packages for micro benchmarking in Node.js:

node-bench - the effort that led to this PR, currently in active development, looking for feedback and ideas; aims to be the foundation of node:benchmark
Benchmark.js - still good and widely used
isitfast - not production-ready yet, but innovative and promising
cronometro - runs tests in isolated worker threads
Tinybench - also works in the browser (like Benchmark.js)
mitata - fast and accurate (used by Bun and Deno)

If you need something production-ready today, Benchmark.js is a good choice. It's battle-tested and versatile. However, its latest release was 6 years ago and the repository has been archived (as of 2024-04-14).

The other options are all worth checking out. Consult the overview table and benchmarks-comparisons that Vinicius Lourenço put together for more details.

For the record, Deno has a built-in benchmark runner.

A CLI for such tools would be great. Have some code in a file and let a CLI tool import and benchmark it. Much like aforementioned tools, but move the API from runtime to CLI.

Pitfalls

Before we continue, here's the mandatory warning to not forget about the pitfalls of micro benchmarking:

Running code in isolation means missing real-world context and different compiler optimizations. For various reasons, the same code may have different performance characteristics when running in the context of a real-world application.
Micro benchmarking is often associated with premature optimization. Don't lose sight of the big picture!

Example: string concatenation

Let's look at an example. We want to know which function is the fastest, and by how much. The following functions do the same thing:

function join(strings) {
  return strings.join('');
}

function concat(strings) {
  return ''.concat(...strings);
}

Benchmark.js

Benchmark.js is great and battle-tested software, despite the fact its last publish was early 2017 when it was tested on Node.js version 10 and 11.

Let's create a test suite to benchmark and compare three string concatenation alternatives:

import Benchmark from 'benchmark';

const strings = ['aa', 'bb', 'cc', 'dd', 'ee', 'ff', 'gg', 'hh'];

function plus(strings) {
  let result = '';
  for (const str of strings) result += str;
  return result;
}

function join(strings) {
  return strings.join('');
}

function concat(strings) {
  return ''.concat(...strings);
}

const suite = new Benchmark.Suite();

suite
  .add('plus', function () {
    plus(strings);
  })
  .add('join', function () {
    join(strings);
  })
  .add('concat', function () {
    concat(strings);
  })
  .on('cycle', function (event) {
    console.log(String(event.target));
  })
  .on('complete', function () {
    console.log('Fastest is ' + this.filter('fastest').map('name'));
  })
  .run();

Running this on my machine gives the following output:

$ node benchmark.mjs
plus x 20,171,223 ops/sec ±0.41% (94 runs sampled)
join x 10,288,969 ops/sec ±0.19% (101 runs sampled)
concat x 17,782,613 ops/sec ±0.18% (98 runs sampled)
Fastest is plus

Clear output. All options are fast, but we have a winner.

Tinybench

Tinybench is the new kid on the block. You can use it stand-alone, and it also comes shipped with Vitest.

The API of Tinybench is similar to Benchmark.js:

import { Bench } from 'tinybench';

const suite = new Bench();

suite
  .add('plus', function () {
    plus(strings);
  })
  .add('join', function () {
    join(strings);
  })
  .add('concat', function () {
    concat(strings);
  });

suite.addEventListener('complete', function () {
  console.table(suite.table());
});

suite.run();

Running this gives the following output:

$ node tinybench.mjs
┌─────────┬───────────┬──────────────┬────────────────────┬──────────┬─────────┐
│ (index) │ Task Name │   ops/sec    │ Average Time (ns)  │  Margin  │ Samples │
├─────────┼───────────┼──────────────┼────────────────────┼──────────┼─────────┤
│    0    │  'plus'   │ '13,188,219' │  75.8252486995182  │ '±0.61%' │ 6594110 │
│    1    │  'join'   │ '7,958,935'  │ 125.64493939618565 │ '±0.54%' │ 3979468 │
│    2    │ 'concat'  │ '11,681,195' │ 85.60767506752819  │ '±0.91%' │ 5840598 │
└─────────┴───────────┴──────────────┴────────────────────┴──────────┴─────────┘

A CLI, maybe?

Wouldn't it be convenient if we could just export our functions from a file:

const strings = ['aa', 'bb', 'cc', 'dd', 'ee', 'ff', 'gg', 'hh'];

export function plus(strings) {
  let result = '';
  for (const str of strings) result += str;
  return result;
}

export function join(strings) {
  return strings.join('');
}

export function concat(strings) {
  return ''.concat(...strings);
}

And point our imaginary bench CLI tool at this file:

$ bench string-concat.js
plus x 20,171,223 ops/sec ±0.41% (94 runs sampled)
join x 10,288,969 ops/sec ±0.19% (101 runs sampled)
concat x 17,782,613 ops/sec ±0.18% (98 runs sampled)
Fastest is plus

And, maybe, one day:

$ node --bench string-concat.js

Conclusion

Although the building blocks are there, I think especially in the area of macro optimizations there's room for tooling to make our lives easier.

When it comes to micro benchmarking, it feels a bit odd to recommend a package last updated in 2017 (Benchmark.js). Let's watch this space!

This concludes my perspective on the current state of benchmarking in Node.js, at the end of 2023. Do you agree?

Versioned documentation with Starlight & Vercel

Wed, 20 Dec 2023 00:00:00 GMT

Versioned documentation with Starlight & Vercel

Introduction

Let's say in our project we're working on a new major version. This may come with new features or even breaking changes. Users that did not upgrade yet reading documentation for the new version might get confused. This is why we want to serve a separate version of the documentation along with each major version of our project.

UPDATE (2024-06): The website in this example has been migrated from Vercel to Netlify. They offer "branch deploys" so knip.dev has v3.knip.dev and v4.knip.dev based on the same repository. Easy-peasy!

UPDATE (2025-03): The website in this example no longer needs versioned documentation. This might change in the future.

A Temporary Solution

Using Starlight and Vercel, this quick guide shows how I did it in my own project: serve a separate version of the documentation at its own path. We're assuming the documentation is deployed from the main branch (still at v1) and we're working in the v2 branch to prepare the next major release. We're going to deploy this version branch and make it accessible at the /v2 path of a domain we already own.

This solution requires a separate Vercel project for each major version.

Hopefully the Starlight team will deliver a built-in solution, but until then this approach might work for you if you need it.

The Plan

This guide assumes we have our site (example.org) running, and want to add the next version at example.org/v2.

We start by setting up a new versioned project, make sure it works, and then integrate the main project with it. It's a short story in three parts:

Set base and outDir options for Starlight in the versioned branch.
Create a new project in Vercel and deploy this versioned branch.
Configure rewrites for Vercel in the main branch.

Configure the versioned branch

Create or switch to the version branch, v2 in our example.

In your Astro configuration, set both the base and the outDir to match the version in the current branch:

export default defineConfig({
  site: 'https://example.org',
  base: '/v2',
  outDir: './dist/v2',
  trailingSlash: 'never',
});

This example has trailing slashes removed, which makes sense with Vercel's cleanUrls option we'll set later on. The example site is using the default static output.

Now is the time to verify a local astro dev serves the site at the /v2 path properly and links are working fine. Push your version branch to your Git remote. Make sure not to merge these changes to the main branch.

Create a new project in Vercel

Create a new project in the Vercel control panel and connect it with your Git repository. It makes sense to name the project after the version, something like "example-org-v2".

Go to Settings → Git → Production Branch to set the version branch:

In the screenshots we see "v4", because I was at v3 in the main branch.

No need to buy or connect a new domain, versioned docs can be served from a free Vercel subdomain such as example-org-v2.vercel.app. No worries: users won't see this, they will only see the main domain you've already set up.

Go to Settings → Domains, edit and double-check it does not have redirects configured and that the Git branch is v2:

Verify the site is deployed and working at the Vercel domain and version path (our example would run at https://example-org-v2.vercel.app/v2).

Need to trigger a deployment? Go to Settings → Git → Deploy Hooks:

You can copy the link and use curl in a shell to trigger a deployment:

curl https://api.vercel.com/v1/integrations/deploy/prj_rKtw..

Once your versioned site is running smoothly we're ready for the final step!

Configure rewrites for Vercel

Switch to the main branch and create a vercel.json file in the same folder as astro.config.mjs (if it isn't already there). Add two items to the rewrites array as follows:

{
  "cleanUrls": true,
  "rewrites": [
    {
      "source": "/v2/:path*",
      "destination": "https://example-org-v2.vercel.app/v2/:path*"
    },
    {
      "source": "/v1/:path*",
      "destination": "/:path*"
    }
  ]
}

Modify the domain and the paths to match your situation. In this example, v1 is still the default version. You can basically swap them around once v2 is the new default in the main branch. You'll want to omit the cleanUrls if you prefer trailing slashes.

Push this in the main branch to the remote and wait for the deployment to finish. Verify everything works as expected: example.org , example.org/v1 and example.org/v2.

Awesome! 🎉

We've completed all steps and this process can be repeated for new versions in the future.

Navigation

What's left to do is to show the user the current version of the documentation and a way to navigate to a different version. There's multiple ways to do this, so this is left as an exercise to the reader.

Feel free to check out my basic solution using a dropdown in a custom Header.astro component with minimal configuration to keep track of available versions. The repo contains the astro.config.ts and vercel.json configuration files in the main branch too. This solution is currently running at https://knip.dev.

Good luck and have a great day!

How to use a compiled bin in a TypeScript monorepo with pnpm

Thu, 14 Sep 2023 00:00:00 GMT

How to use a compiled bin in a TypeScript monorepo with pnpm

Today's scrap has a very long title and is about pnpm workspaces that contain a compiled executable in a TypeScript monorepo.

Problem

When running pnpm install in a monorepo, the local bin file of a workspace may not exist yet. This happens when that file needs to be generated first (e.g. when using TypeScript). Then pnpm is unable to link the missing file. This also results in errors when trying to execute the bin from another workspace.

tl/dr; Make sure the referenced file in the bin field of package.json exists, and import the generated file from there.

Solution

So how to safely use a compiled bin? Let's assume this situation:

The entry script for the CLI tool is at src/cli.ts
This source file is compiled to lib/cli.js
Compiled by the build script that runs tsc

Here are some relevant bits in the package.json file of the workspace that wants to expose the bin:

{
  "name": "@org/my-cli-tool",
  "bin": {
    "my-command": "bin/my-command.js"
  },
  "scripts": {
    "build": "tsc",
    "prepack": "pnpm run build"
  },
  "files": ["bin", "lib"]
}

Use "type": "module" to publish as ESM in package.json. Import the generated file from bin/my-command.js:

#!/usr/bin/env node
import '../lib/cli.js';

Publishing as CommonJS? Then use require:

#!/usr/bin/env node
require('../lib/cli.js');

Make sure to include the shebang (that first line starting with #!), or consumers of your package will see errors like this:

bin/my-command: line 1: syntax error near unexpected token `'../lib/index.js''

Publishing

In case the package is supposed to be published, use the prepack (or prepublishOnly) script and make sure to include both the bin and lib folders in the files field (like in the example above).

A note about `postinstall` scripts

Using a postinstall script to create the file works since pnpm v8.6.6, but postinstall scripts should be avoided when possible:

Can perform malicious acts (security scanners don't like them)
Can be disabled by the consumer using --ignore-scripts
Can be disabled if the consumer uses pnpm.onlyBuiltDependencies

Bun does not execute arbitrary lifecycle scripts for installed dependencies.

That's why this little guide doesn't promote it, and this scrap got longer than I wanted!

Additional notes

This scrap is based on this GitHub comment in the pnpm repository.
I've seen and tried workarounds to (mkdir and) touch the file from postinstall scripts, but that's flaky at best and not portable.
The same issue might occur when using npm, Bun and/or Yarn. True or not, it's better to be safe than sorry.
If you are using only JavaScript (or JavaScript with TypeScript in JSDoc) then you can target the src/cli.js file directly from the bin field.

Using OpenAI with JavaScript

Wed, 03 May 2023 00:00:00 GMT

Using OpenAI with JavaScript

When trying to find my way around in the buzzing lands of OpenAI and vector databases, the dots were not always easy to connect. In this guide I'm sharing what I've learned during my journey to make yours even better. You might find a trick or a treat!

Most of OpenAI tooling and examples is based on Python, but this guide uses JavaScript exclusively.

We'll begin with a brief explanation of some core concepts, before diving into more and more code. Towards the finish we'll discuss some strategies for token management and maintaining a conversation.

Overview

Here are the topics we will be discussing:

OpenAI endpoints
Key concepts
Ingestion
Query
User Interface
Conversation
Tokens
Parameters
Markdown & code blocks
Next steps
Closing remarks

OpenAI endpoints

In this guide, we will work with two OpenAI REST endpoints.

Chat Completions

POST https://api.openai.com/v1/chat/completions

The Create chat completion endpoint generates a human-like text completion for a provided prompt. We'll use it to start and keep the conversation going between the end-user and OpenAI's Large Language Models (LLMs) such as GPT-3.5 and GPT-4.

Create Embeddings

POST https://api.openai.com/v1/embeddings

With the embeddings endpoint, we can create embeddings from plain text. We will use these embeddings to store and query a vector database. Embeddings? Vector database? No worries, we have you covered.

The `openai` package

We're going to use these endpoints directly, and not OpenAI's npm package. This package targets Node.js, but eventually you might want to deploy your own endpoint on an environment without Node.js, such as a serverless or edge platform like Cloudlare Workers, Netlify Edge or Deno. Now that fetch is ubiquitous I think the REST APIs are just as easy to use without any dependencies. I like being "closer to the metal" and stay flexible.

Key concepts

We've already introduced a few concepts that may be new to you. Let's discuss embeddings, vector databases and prompts briefly before diving into any code.

If you're familiar with them, feel free to skip straight to ingestion.

Embeddings

Vector embeddings are numerical representations of textual data in a high-dimensional space. They are generated using large language models (LLMs). Embeddings allow for efficient storage and search of content that is semantically related to a user's query. Semantically similar text is mapped close together in the vector space, and we can find relevant content using a vector embedding created from user input.

For comparison, a lexical or "full text" search looks for literal matches of the query words and phrases, without understanding the overall meaning of the query.

Vector databases

Why do we need a vector database? Can't we just query OpenAI and get a response?

Yes, we can use the ChatGPT UI or even the OpenAI chat completions endpoint directly. However, the response will be limited to what the OpenAI models are trained on. The response may not be up-to-date, accurate, or specific enough for your needs.

What if you want to have OpenAI generate responses based solely on your own domain-specific content? For users to "chat with your content". Sounds interesting! But how to go about this?

Unlike ChatGPT, the OpenAI APIs are not storing any of your content and they do not store state or a session of the conversation(s). This is where vector databases come in. Adding a vector database in the mix has interesting advantages:

Store and maintain domain-specific knowledge.
Support semantic search across your content.
Control your own data and keep it up-to-date and relevant.
Reduce the number of calls to OpenAI.
Store the user's conversational history.

Setting up a vector database might be easier than you think. I've been trying out managed solutions like Pinecone and Supabase without any issues. There are more options though, and I don't feel like I'm in a position to recommend one over another. I do like that I can use Pinecone without dependencies using only fetch and their REST API.

Prompts

A prompt is the textual input we send to the chat completions endpoint to have it generate a relevant "completion". You could say a prompt is a question, and a completion is an answer.

Prompts are plain text and we can provide extra details and information to improve the results. The more context we provide, the better the response will be.

Requests to the chat completions endpoint are essentially stateless: not your content, no session, no state. The challenge is to optimize and include the right information with each request. We'll be discussing prompts throughout this guide, and ways to optimize them.

Ingestion

Armed with this knowledge, let's begin building a chat application with a vector database.

We'll need to get content into this database. Content is stored as vector embeddings, and we can create those from textual content by using the embeddings endpoint.

Metadata

Before creating the database table or index, it's important to consider what we will do with the results of semantic search queries.

Vector embeddings are a compressed representation of semantics for efficient storage and querying. It's not possible to translate them back to the original text. This is the reason we need to store the original text along with the embeddings in the database.

The text can be stored as metadata and can include more useful things to display in the application, such as document or section titles and URL's to link back to the original source.

Tools

There are tools that can help with this. I have seen a few solutions that offer easy content ingestion, but you don't have much freedom such as to choose where the content will be stored:

7-docs

As I wanted to start out with command-line tools and learn more about the OpenAI APIs, embeddings and vector database, I decided to develop a tool myself.

This work ended up as 7-docs and comes with the 7d command-line tool to ingest content from plain text, Markdown and PDF files into a vector database. It ingests content from local files, GitHub repositories and also HTML from public websites. Currently it supports "upserting" vectors into Pinecone indexes and Supabase tables.

To get an idea what ingestion using 7d looks like, here are some examples that demonstrate how to ingest Markdown files:

7d ingest --files '*.md' --db pinecone --namespace my-docs

7d ingest --source github --repo reactjs/react.dev \
  --files 'src/content/reference/react/*.md' \
  --db supabase \
  --namespace react

Query

When the embeddings and metadata are in the database, we can query it. We'll look at some example code to implement this 4-step strategy:

Create a vector embedding from the user's textual input.
Query the database with this vector for related chunks of content.
Build the prompt from the search results and the user's input.
Ask the model to generate a chat completion based on this prompt.

The next examples show working code, but contain no error handling or optimizations. Just plain JavaScript without dependencies.

(Don't want to implement this yourself, or just want to see examples? Visit 7-docs for available demos and starterkits to hit the ground running.)

1. Create a vector embedding

The first function we'll need creates a vector embedding based on the user's input:

export const createEmbeddings = async ({ token, model, input }) => {
  const response = await fetch('https://api.openai.com/v1/embeddings', {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${token}`,
    },
    method: 'POST',
    body: JSON.stringify({ input, model }),
  });

  const { error, data, usage } = await response.json();

  return data;
};

This function can be called like this:

const vector = await createEmbeddings({
  token: '[OPENAI_API_TOKEN]',
  model: 'text-embedding-ada-002',
  input: 'What is an embedding?',
});

2. Query the database

In the second step we are going to query the database with the vector embedding we just created. Below is an example that queries a Pinecone index for vectors with related content using fetch. The rows returned from this query are mapped to the metadata that's stored with the vector in the same row. We need this metadata in the next step.

export const query = async ({ token, vector, namespace }) => {
  const response = await fetch('https://[my-index].pinecone.io/query', {
    headers: {
      'Content-Type': 'application/json',
      'Api-Key': token,
    },
    method: 'POST',
    body: JSON.stringify({
      vector,
      namespace,
      topK: 10,
      includeMetadata: true,
    }),
  });

  const data = await response.json();
  return data.matches.map(match => match.metadata);
};

This query function can be invoked with the vector we received from createEmbeddings() like so:

const metadata = await query({
  token: '[PINECONE_API_KEY]',
  vector: vector, //  Here's the vector we received from `createEmbeddings()`
  namespace: 'my-knowledge-base',
});

3. Build the prompt

The third step builds the prompt. There are multiple ways to go about this and the content of the template probably requires customization on your end, but here is an example:

const template = `Answer the question as truthfully and accurately as possible using the provided context.
If the answer is not contained within the text below, say "Sorry, I don't have that information.".

Context: {CONTEXT}

Question: {QUERY}

Answer: `;

const getPrompt = (context, query) => {
  return template.replace('{CONTEXT}', context).replace('{QUERY}', query);
};

And here is how we can create the prompt with context from the metadata returned from the database query:

// Create a concatenated string from search results metadata
const context = metadata.map(metadata => metadata.content).join(' ');

// Build the complete prompt including the context and the question
const prompt = getPrompt(context, 'What is an embedding?');

Later in this guide, we will also look at example code to maintain a conversation instead of merely asking one-shot questions.

4. Generate chat completion

We are ready for the last step: ask the model for a chat completion with our prompt. Here's an example function to call this endpoint:

export const chatCompletions = async ({ token, body }) => {
  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(body),
  });

  return response;
};

And here's how to make the request with the prompt:

const messages = [];
messages.push({
  role: 'user',
  content: prompt, // This is the `prompt` we received from `getPrompt()`
});

const response = await chatCompletions({
  token: '[OPENAI_API_TOKEN]',
  body: {
    model: 'gpt-3.5-turbo',
    messages,
  },
});

const data = await response.json();
const text = data.choices[0].message.content;

The text contains the human-readable answer from OpenAI.

Excellent, this is the essence of generating chat completions based on your own vector database. Now, how do we combine these four steps and integrate them into a user interface? You can create a function that abstracts this away, or use the @7-docs/edge package to do this for you. Keep reading to see an example.

In the next part of this guide, we will explore a UI component featuring a basic form for users to submit their queries. This component will also render the streaming response generated by the function in the next section.

User Interface

Let's put our 4-step strategy into action and build function and form.

(Don't want to implement this yourself, or just want to see examples? Visit 7-docs for available demos and starterkits to hit the ground running.)

Function

The /api/completion endpoint will listen to incoming requests and respond using all of the query logic from the previous section.

We're going to use the @7-docs/edge package, which abstracts away the 4-step strategy and some boring boilerplate. We need to pass the OPENAI_API_KEY and a query function from a database adapter, Pinecone in this example. We pass it to getCompletionHandler so it can query the database when it needs to. We would pass a different function if we wanted to used a different type of database (like Supabase or Milvus).

Let's bring this together in a serverless or edge function handler in just a few lines of code:

import { getCompletionHandler, pinecone } from '@7-docs/edge';
import { createClient } from '@supabase/supabase-js';

const OPENAI_API_KEY = process.env.OPENAI_API_KEY;
const PINECONE_URL = process.env.PINECONE_URL;
const PINECONE_API_KEY = process.env.PINECONE_API_KEY;
const namespace = 'my-knowledge-base';

const query: QueryFn = (vector: number[]) =>
  pinecone.query({
    url: PINECONE_URL,
    token: PINECONE_API_KEY,
    vector,
    namespace,
  });

export default getCompletionHandler({ OPENAI_API_KEY, query });

This pattern can be used anywhere from traditional servers to edge functions, since there are no dependencies on modules only available in Node.js.

Form

Now we still need a UI component to render an input field, send the input to the /api/completion endpoint, and render the streaming response.

This minimal example uses a little React and JSX for an easy read, but it could just as well be plain JavaScript or any other framework.

import { useState } from 'react';

export default function Page() {
  const [query, setQuery] = useState('');
  const [outputStream, setOutputStream] = useState('');

  function startStream(query) {
    const searchParams = new URLSearchParams();
    searchParams.set('query', encodeURIComponent(query));
    searchParams.set('embedding_model', 'text-embedding-ada-002');
    searchParams.set('completion_model', 'gpt-3.5-turbo');
    const url = '/api/completion?' + searchParams.toString();

    const source = new EventSource(url);
    source.addEventListener('message', event => {
      if (event.data.trim() === '[DONE]') {
        source.close();
      } else {
        const data = JSON.parse(event.data);
        const text = data.choices[0].delta.content;
        if (text) setOutputStream(v => v + text);
      }
    });
  }

  const onSubmit = event => {
    if (event) event.preventDefault();
    startStream(query);
  };

  return (
    <>
      <form onSubmit={onSubmit}>
        <label>
          How can I help you?
          <input
            type="search"
            value={query}
            onChange={event => setQuery(event.target.value)}
          />
        </label>
        <input type="submit" value="Send" />
      </form>

      <div>{outputStream}</div>
    </>
  );
}

Now all the components in a "chat with your content" have come together:

Ingest content as vector embeddings into a database
Create a function to query the content using the 4-step strategy
Build a UI to accept user input and render a streaming response

The following sections will build on to make everything even more interesting!

Conversation

To start a chat, we've seen how to build a basic prompt. This is good enough for one-shot questions, but we need more to build a meaningful conversation. The chat completions endpoint accepts an array of messages, so a pattern to fill this array could look like this:

Add a system message that explains the model (i.e. the assistant) how to behave and respond.
Add the conversation history with user and assistant messages.
Add the user prompt, containing the context and the query.

Here is an example building on the initial prompt example that extends the messages array to build the conversation:

// Create a concatenated string from search results metadata from step 2: query the database
const context = metadata.map(metadata => metadata.content).join(' ');

const system = `Answer the question as truthfully as possible using the provided context.
If the answer is not contained within the text below, say "Sorry, I don't have that information.".`;

// In a real application, the conversation `history` can be sent
// with every request from the client, or by using some kind of storage.
const history = [
  ['What is an embedding?', 'An embedding is...'],
  ['Can you give an example?', 'Here is an example...'],
];

const prompt = getPrompt(context, 'Can I restore the original text?');

const messages = [];

messages.push({
  role: 'system',
  content: system,
});

history.forEach(([question, answer]) => {
  messages.push({
    role: 'user',
    content: question,
  });

  messages.push({
    role: 'assistant',
    content: answer,
  });
});

messages.push({
  role: 'user',
  content: prompt,
});

const response = await chatCompletions({
  token: '[OPENAI_API_TOKEN]',
  model: 'gpt-3.5-turbo',
  messages,
});

The actual history can come from the client. For instance, this could be stored in UI component state, or browser session storage. In that case, it will need to be sent with every request to the function. Other ways of storing and retrieving the conversation history is outside the scope of this guide.

See the starter kits for examples to handle this in the user interface in tandem with the @7-docs/edge package.

Tokens

Tokens (not characters) are the unit used by OpenAI for limits and usage. There are limits to the number of tokens that can be sent to and received from the API endpoints with each request.

Embeddings

The maximum number of input tokens to create embeddings with the text-embedding-ada-002 model is 8191.

The price is $ 0.0004 per 1k tokens, which comes down to a maximum of $ 0.0032 per request when sending 8k tokens. That's roughly 6.000 words that can be sent at once to create vector embeddings. We can send as many requests as we want.

During content ingestion you may need this endpoint for a short period in bursts, depending on the amount of content. Remember that we also need it to create an embedding from the user's input to query the vector database. Depending on the user's input this request is usually smaller, but may occur frequently for a longer period depending on application traffic.

Chat completions

For the chat completions endpoint, the max_tokens value represents the number of tokens the model is allowed to use when generating the completion. The models have their own limit (context length) and pricing:

Model	Context Length	$/1k prompt	$/1k completion
gpt-3.5-turbo	4.096	$ 0.002	$ 0.002
gpt-4	8.192	$ 0.03	$ 0.06
gpt-4-32k	32.768	$ 0.06	$ 0.12

The sum of the tokens for the prompt plus the max_tokens for completion cannot exceed the model's context length. For gpt-3.5-turbo this means:

num_tokens(prompt) + max_tokens <= 4096

To see what this means in practice, we'll discuss tokenization first and then look at an example calculation.

Tokenization

The number of tokens for a given text can be calculated using a tokenizer (such as GPT-3-Encoder). Tokenization can be slow on larger chunks, and npm packages for Node.js may not work in other environments such as the browser or Deno.

The alternative is to make an estimate: use 4 characters per token or 0.75 words per token. That's 75 words per 100 tokens. This is a very rough estimate for the English language and varies per language. You should probably also add a small safety margin to stay within the limits and prevent erors.

OpenAI provides an online Tokenizer. For Python there's tiktoken.

Example

Let's say you're using the gpt-3.5-turbo model. If you want to preserve 25% for the completion, use max_tokens: 1024. The rest of the model's context can be occupied by the prompt. That's 3072 tokens (4096-1024), which comes down to an estimated 2304 words (3072*0.75) or 12.288 characters (3072*4).

The length of the prompt is the combined length of all content in the messages (i.e. the combined messages of the system, user and assistant roles in Conversation).

If the prompt has the maximum length and the model would use all completion tokens, using 4096 tokens would cost $ 0.008 (4*$0.002).

Using the gpt-4 model, the same roundtrip would cost $ 0.15 (3*$0.03 for the prompt + 1*$0.06 for the completion).

Strategies

To optimize for your end-user, you'll need to find the right balance between input (prompt) and output (completion).

When adding context and conversation history to the chat completion request it may become a challenge to keep everything within the model's limit. More context and more conversation history (input) means less room for the completion (output).

There are a few ways I can think of to help mitigate this:

Limit the number of messages to keep in the conversation history.
Truncate or leave out previous answers from the assistant.
Send some sort of summary of the conversation history. That would likely require additional effort and requests.
Use a solution like GPTCache to cache query results.
Some form of "compression" could work in certain cases. An example using GPT-4 can be found at gpt4_compression.md.

Another thing to consider is the amount of context to send with the prompt. This context comes from the semantic search results when querying the vector database. You may want to create smaller vector embeddings during ingestion to eventually have more options and wiggle room when building the context for the chat completion. On the other hand, including smaller but more varied pieces of context may result in less "focused" completions.

Overall, I think what matters most is to not lose the first and last question throughout the conversation. Keep in mind that the model does not store state or session.

Usage

When using OpenAI endpoints, the token usage for the request is included in the response (with separate prompt_tokens and completion_tokens). Unfortunately, usage is not included for streaming chat completion responses (stream: true).

Parameters

A quick overview of some common parameters you may want to tweak for better chat completions.

`temperature`

The temperature parameter is a number between 0 and 2 (default: 1). A low number like 0.2 makes the output more focused and deterministic. You want this when the output should be generated based on the context sent within the prompt. A higher value like 0.8 makes the output more random.

`presence_penalty` and `frequency_penalty`

A number between -2 and 2 to decrease or increase the presence and frequency of tokens. The default value is 0 and this is fine for most situations. If you want to reduce repetition, try numbers between 0.1 and 1. Negative numbers increase the likelihood of repetition.

`name`

As we've seen when creating the messages array, each message is assigned a role (system, user or assistant). You can make the conversation more personal and send a name with each message.

Markdown & code blocks

If you ingest Markdown content, you likely also want the completion to include Markdown and code blocks when relevant. Here's a list of things to remember during ingestion and building the client application:

Ingestion

Don't strip out code blocks from the Markdown during ingestion.
Try to prevent splitting text in the middle of code blocks.

Client

Include something like "Use Markdown" and "Try to include a code example in language-specific fenced code blocks" in the prompt, ideally in the system message.
Use a Markdown renderer (e.g. react-markdown).
Use a syntax highlighter (e.g. react-syntax-highlighter).

Next steps

After figuring out how to connect the dots, it's exciting to tinker and continue the journey to improve the user experience. Here are a few pointers that may inspire you:

Consider the integration of the conversation in the user interface, as well as the place and the role of the chat box.
Keep refining the prompt to better align with your content and your target audience.
Improve chat completions by further tweaking the parameters, vector embedding sizes, and context in the prompt.
Empowere users with more control by providing affordances to adjust the prompt or by incorporating multiple prompts.
Combine multiple sources of content, such as searching a database with source code or a table with more generic content.
Generate multiple chat completions in a single response.
Use the Moderations endpoint to make sure the input text does not violate OpenAI's content policy.
Last but not least, listen to your customers. What are their needs?

Closing remarks

We've explored many aspects of using OpenAI with JavaScript to create useful applications. We've covered everything from ingesting content to building a user interface with your own serverless or edge function. Hopefully, this guide is helpful in your own journey. Good luck!

I would love to hear about your thoughts and what you are building, please share with me on Bluesky!

Special thanks goes out to Enis Bayramoğlu for a great review.

Using Git bisect to divide & conquer

Mon, 19 Dec 2022 00:00:00 GMT

Using Git bisect to divide & conquer

When you have a series of commits and want to find where a bug or a change of behavior was introduced, git bisect is your friend. With a command that understands what is "bad" and what is "good", this process can be fully automated. For instance, use npm test and report back the first commit where the tests fail.

Here's how to start the process:

git bisect start
git bisect bad HEAD
git bisect good v5.1.0

Usually the HEAD is a bad commit, and v5.1.0 is a tag or commit you are sure is good. Create a file to run the commands. Here's an arbitrary example:

npm run build
npm test

Make this file executable (chmod +x bisect.sh), and run with it:

git bisect run ./bisect.sh

In case there is no script to automate this, then you can do this manually. Just say git bisect good or bad, and Git will check out the next commit for you, you verify whether it's good or bad, and so on. Git uses a binary search algorithm to do this efficiently.

Note that this technique is often used to find which changeset introduced a bug, but other ideas include finding a performance regression, the output of some program changes, etcetera. You can even use different terms (instead of "bad" and "good") to support this:

git bisect start --term-old fast --term-new slow

When you are done, or made a mistake marking good or bad commits, the process has to be reset:

git bisect reset

See the git bisect documentation for more details.

Handling errors in Azure pipelines

Fri, 09 Sep 2022 00:00:00 GMT

Handling errors in Azure pipelines

Put mildly, Azure pipelines don't always behave as expected. Sometimes a pipeline does not fail when it should. This scrap shows a few solutions to make pipelines fail for script and task errors.

We're going to look at three cases and ways to make pipelines fail:

Fail on errors written to stderr in scripts (failOnStderr)
Fail on errors written to stderr in tasks (failOnStandardError)
Fail on script errors (set -e)

And we'll also look at a way to make a pipeline continue, even if there are errors.

An unpleasant surprise

Let's dive straight into our topic and take a look at an example script task that tries to tag a pipeline run:

- script: |
    az pipelines runs tag add --run-id $(Build.BuildId) --tags my-container
    echo "Tagged build for my-container"
  displayName: Tag successful build

In this example the az command fails due to some missing extension. This results in output like this:

Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /agent/_work/_temp/3ecc72e6-92f7-4de6-96c3-35ae602c7620.sh
ERROR: The command requires the extension azure-devops. Unable to prompt for extension install confirmation as no tty available. Run 'az config set extension.use_dynamic_install=yes_without_prompt' to allow installing extensions without prompt.
Tagged build for my-container

Finishing: Tag successful build

The command fails and prints an ERROR (to stderr). But both the task and the pipeline still succeed:

Why does this not make the task fail? It's because the az command does not exit with a non-zero code.

This is often not the desired behavior. Fortunately, when we want to fail the pipeline we do have some options:

Use the failOnStderr task option
Or use set -e inside the script

Let's look what happens when either of these are used.

Fail on errors written to stderr in scripts

Here we can add failOnStderr as a task configuration option:

- script: |
    az pipelines runs tag add --run-id $(Build.BuildId) --tags my-container
    echo "Tagged build for my-container"
  displayName: Tag successful build
  failOnStderr: true

This will execute the whole script, but make the task fail, since the az command prints the error to stderr:

Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /agent/_work/_temp/3ecc72e6-92f7-4de6-96c3-35ae602c7620.sh
ERROR: The command requires the extension azure-devops. Unable to prompt for extension install confirmation as no tty available. Run 'az config set extension.use_dynamic_install=yes_without_prompt' to allow installing extensions without prompt.
Tagged build for my-container
##[error]Bash wrote one or more lines to the standard error stream.
##[error]ERROR: The command requires the extension azure-devops. Unable to prompt for extension install confirmation as no tty available. Run 'az config set extension.use_dynamic_install=yes_without_prompt' to allow installing extensions without prompt.

Finishing: Tag successful build

The pipeline fails:

Fail on errors written to stderr in tasks

When we want to do the same for a task (as opposed to a script), this requires a different setting. For tasks the failOnStandardError option needs to be set as part of the inputs:

- task: AzureCLI@2
  displayName: Deploy my-container
  inputs:
    failOnStandardError: true

Alright, so we have:

Script → failOnStderr
Task → failOnStandardError

And we still have another option left: set -e

Fail on script errors

To make the script fail on errors, use set -e at the start of the script:

- script: |
    set -e
    az pipelines runs tag add --run-id $(Build.BuildId) --tags my-container
    echo "Tagged build for my-container"
  displayName: Tag successful build

This will fail the script immediately:

Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /agent/_work/_temp/a64b21a0-0a8e-4e6b-a0b4-271980ef4d05.sh
ERROR: The command requires the extension azure-devops. Unable to prompt for extension install confirmation as no tty available. Run 'az config set extension.use_dynamic_install=yes_without_prompt' to allow installing extensions without prompt.
##[error]Bash exited with code '2'.
Finishing: Tag successful build

And again, the pipeline fails as intended:

But notice the difference in behavior: the "Tagged build for my-container" message is not printed here.

Depending on the use case one or the other is the better choice, although I think in general failing immediately is the better option.

Continue on error

Last but not least, sometimes the pipeline should continue even if the task failed. For this use case, there is continueOnError to the rescue:

- script: |
    set -e
    az pipelines runs tag add --run-id $(Build.BuildId) --tags my-container
    echo "Tagged build for my-container"
  continueOnError: true
  displayName: Tag successful build

This will result in a green pipeline, but also a warning sign for the stage with the failed task:

Compare this to the initial situation where everything is naively green. At least now we can see something is off.

Azure DevOps documentation links

Command Line task covers failOnStderr
Azure CLI task covers failOnStandardError
Task types & usage covers continueOnError

The value of abstractions

Fri, 19 Aug 2022 00:00:00 GMT

The value of abstractions

In software systems, maintenance quickly becomes harder as more components are added. Given a well-designed system, when a component deteriorates, it should be possible to refactor or replace it without major impact on other component in the system. Following this principle, components should separate concerns clearly with well-designed interfaces.

This article discusses some considerations in the process of designing a complex system and its components.

Minimize the cost of change

Over time, implementations and underlying dependencies will change, but the interfaces should remain stable. Changes to a component's implementation should have minimal impact on other components in the system.

An interface should depend on the code that calls it, not its implementation.

Let's take an example component "C" in our system. The deeper and more frequently C is integrated into the system, the more important its interface becomes. When considering the cost of C, it may help to ask ourselves questions like this:

Is it hard to refactor without affecting other components?
Is it hard to replace within the structure?
Is it hard to replace its underlying dependencies?
Is it hard to build another feature with an alternative of C?

Answering "yes" means more coupling of other components towards C. This means the project is harder to maintain, increasing the cost of C.

This may also indicate a leaky abstraction, if C fails to encapsulate and hide its underlying implementation details.

In short: there's value in having the right abstraction for C.

The more expensive it is to refactor or replace a component in the system, the more value it has to design an interface to abstract the implementation away.

One of the hardest part of system design is to work out the modularity of the system, and seeing what components benefit the most from an abstraction.

Is a component large and complex, perhaps resembling more of a framework? If you know upfront a potential refactoring is hardly feasible then an abstraction is likely not worth it. In that case, maybe you need to take a step back and reconsider the modularity of the system as a whole. Is it possible to increase its modularity, and lower the cost of changes that will inevitably be necessary over time? What are the trade-offs when going all-in on the framework?

On the other hand, excessive component fragmentation leads to an over-engineered system with too many moving parts and interrelationships.

This balancing act between under- and over-engineering can be a tough cookie. When the overall structure is largely in place, the process of interface design comes down to iteration until all questions above are answered with "no".

Perhaps ironically, this article itself is abstract, and the actual process of software design itself may feel distant or overrated. Yet taking the time to think and design before and during ambitious projects pays off in the long run as it will reduce maintenance complexity and facilitate system evolution.

Enforce module boundaries

Going from less abstract to more concrete at some point means moving into lower level module boundaries. Here, it becomes essential to enforce and guard module boundaries. Even the best interfaces and abstractions may go unnoticed by fellow developers. How to prevent boundaries from being crossed? Depending on the technology stack of choice, tooling might be available to guard certain layers of modularity. At the level of code and configuration, linters may prove very effective.

Examples: ESLint & Nx

Within the JavaScript and Node.js ecosystem, a great example of such a linter is ESLint. Relevant rules include the built-in no-restricted-imports and Nx's @nrwl/nx/enforce-module-boundaries rule.

They help to enforce module boundaries and prevent direct imports of underlying modules or dependencies, and suggest to use the provided abstraction instead.

When properly configured, tools like this effectively encourage developers to think about the system and its components, and consider the usage and value of abstractions.

Using CSS Grid to Stack Elements

Mon, 23 May 2022 00:00:00 GMT

import './styles.css';

To stack elements using CSS, we previously had to turn to absolute positioning and z-index tricks. Yet with CSS Grid, there's a new way to do this. To show what I mean, we're going to build this example component:

:::section

:::div{.grid}

::div{.back}

:::div{.front}

What A Badge

:::

First, we need a few HTML elements:

<div class="grid">
  <div class="back" />
  <div class="front">What A Badge</div>
</div>

The .back element could be an image or something more interesting. The following style declarations show how to stack the .back and .front elements on top of each other:

.grid {
  display: grid;
  grid-template-rows: auto min-content 16px;
}

.back {
  grid-area: 1 / 1 / 4 / 2;
}

.front {
  grid-area: 2 / 1 / 3 / 2;
  margin: 0 -20px;
}

The trick is to use overlapping grid-area values for the elements you want to stack. Use grid-template-rows (or grid-template-columns) to lay out the elements. Additionally, you can use (negative) margin to position the stacked element relative to the grid for even more flexibility.

The stacking order follows the order in the DOM: the last element will be on top of the previous element(s).

The min-content value for the second row of the grid ensures the row takes the height of the stacked .front element, while the auto value for the first row makes it occupy the rest of the available space.

This idea is certainly not new. For instance, it was presented in How to Stack Elements in CSS. This scrap presents the same idea in a more focused way, and with a different example use case.

Support for CSS Grid is currently great across browsers, so you can go ahead and use all of this today!

The JavaScript block statement

Fri, 06 May 2022 00:00:00 GMT

The JavaScript block statement

In the first scrap of this blog I'd like to make a case for a great little feature that I rarely see used in the wild.

Want to organize your code a little bit better?
Have a hard time coming up with another name for the same variable?
Want to run the same code multiple times in the console or a REPL?

Use blocks! Let's take a bogus unit test:

function test() {
  const type = 'some';

  const thing = getThing(type, 1);
  assert.equal(thing, 1);

  // Ehm... how to call this `thing` now...?
  const thing2 = getThing(type, 2);
  assert.equal(thing2, 2);
}

Specifically for unit tests you could argue that this particular case should be separated in two unit tests, but sometimes that's just not what you want. You can use blocks instead:

function test() {
  const type = 'some';

  {
    const thing = getThing(type, 1);
    assert.equal(thing, 1);
  }

  {
    const thing = getThing(type, 2);
    assert.equal(thing, 2);
  }
}

A pair of curly braces are the delimiters of the blocks, and define a new scope for let, const and function declarations.

Neat!

How to add search to your static site

Sat, 30 Apr 2022 00:00:00 GMT

How to add search to your static site

Static websites are popular nowadays. There are many static site generators, but not all have search built-in. Recently I've added a static search option to a few websites, including the one you're reading. In this article I would like to share how I did this, as it might take less efforts than you think!

Re-search

When searching to find a good library for a static full-text search, I came across popular solutions such as Lunr.js, FlexSearch and Fuse.js. To my surprise, these libraries are not very well maintained anymore, while they all have quite some open issues. To me, this does not relate to the popularity of static websites in general.

I've used Fuse.js before to implement a simple but fast search engine (on lejan.com.br), but this was over a year ago, and I'm always looking for better options. I've tried them all again, and each still has at least a few minor glitches. Another reason is that both markdown-rambler and this website itself use ES Modules and other modern JavaScript features, which usually improves developer experience, maintenance and/or performance.

MiniSearch

Later, I was lucky enough to find MiniSearch when looking for "search index" in the npm registry. This package happens to be easy to use, while performance and file size feel good. To be honest, I didn't do an actual file size and performance comparison between the various options, but this MiniSearch blogpost has a good overview.

The fuzzy search and prefix search are optional, and work very well for the websites where I integrated it. This always requires a bit of fine-tuning, depending on the size and density of documents. I did not try the auto-suggestion feature yet. Overall a pleasant experience, much like the alternatives. What stood out initially, was simply better search results.

In my experience so far, the contents of Markdown documents can be used just fine as input for the index. The minimal syntax of Markdown does not seem to negatively impact the index. This makes solutions like MiniSearch great for static websites, as they are often powered by Markdown files.

Getting started

So, how to integrate MiniSearch in your static website? There are roughly four steps here:

Build the index
Serve the index with the rest of the static site
Connect the index with a DOM element (such as an input field)
Search and render results

Building the index

Here's a fragment from markdown-rambler, but the concept can be applied in any JavaScript build system. The idea is to map existing files or pages to be indexed to an object containing the data necessary for both the index and the fields to eventually be displayed in the search results.

The result (documents) can be provided to a MiniSearch instance using minisearch.addAll(documents), which creates and returns the actual index. The final step here is to store this JSON file to disk:

const documents = files
  .filter(files => file.type === 'article')
  .map((file, index) => ({
    id: index,
    title: file.data.meta.title,
    description: file.data.meta.description,
    pathname: file.data.meta.pathname,
    content: file.data.markdown,
  }));

const miniSearch = new MiniSearch({
  fields: ['title', 'content'],
  storeFields: ['title', 'pathname'],
});

await miniSearch.addAllAsync(documents);

await fs.writeFile('search-index.json', JSON.stringify(miniSearch.toJSON()));

This example uses the title and content fields of each document to index the full-text search. The title and pathname (storeFields) will be available to render the search results later.

Serving the index

Next step is to make sure the index is served with the rest of the static website. This could be the root of the "dist" or "public" folder.

Connecting the search component

Depending on the requirements and the type of (static) website, the next part could be implemented in many ways. Here I'm going to try and keep it very concise. Let's add search.js to the static site with the following snippet. This will attach an event listener to an existing <input type="search"> in the DOM:

(async () => {
  await import('https://cdn.jsdelivr.net/npm/minisearch@4.0.3/dist/umd/index.min.js');
  const searchIndex = await fetch('/search-index.json').then(response => response.text() );
  const index = MiniSearch.loadJSON(searchIndex, { fields: ['title', 'content'] });

  const input = document.querySelector('input[type=search]');

  const search = query => {
    const results = index.search(query, { prefix: true, fuzzy: 0.3 });
    console.log(results);
  };
  input.addEventListener('input', event => {
    search(event.target.value);
  });
})();

Here we already have the basics of our search component, in only a few lines of code. Note that the indexed fields title and content should be provided again when loading the index. For brevity, this example logs the search results in the browser console. Combined with very little styling, this is everything this website uses for the static search.

Searching and rendering results

How to render search results depends on the type of static website or which framework is being used. For the sake of completeness, here's a minimal example using vanilla JavaScript. This extends the search function from the previous example:

const container = document.createElement('div');
container.setAttribute('id', 'search-results');

const search = query => {
  if (query.length > 1) {
    const results = index.search(query, { prefix: true, fuzzy: 0.3 });
    const list = document.createElement('ol');
    results.slice(0, 10).forEach(result => {
      const item = document.createElement('li');
      const link = document.createElement('a');
      link.setAttribute('href', result.pathname);
      link.appendChild(document.createTextNode(result.title));
      item.appendChild(link);
      list.append(item);
    });
    container.replaceChildren(list);
    input.after(container);
  } else {
    container.parentNode.removeChild(container);
  }
};

This is roughly the code used on this website, and appends a container element to the DOM as a sibling of the input element. This way, the search results can be rendered relative to this input field.

The search results (with the title and pathname fields we stored in the index before) are appended the container element as an ordered list. Ordered, since MiniSearch provides the results sorted by relevance score.

Final notes

React

If you are using React, you might be interested in react-minisearch, providing React integration for MiniSearch.

Index size

The search index is a relatively large static asset, as it includes both the index and the data to show in the search results. Loading this file on page load, as shown above, could degrade the performance of your website. It does not block the main render thread as it uses a dynamic import, but for larger websites this may impact overall performance. One way to mitigate this is to only load the index when the user actually uses the search, for instance on the focus event of the input field. To get an idea of the file size, currently the index of this website with its first 11 articles, the size is 113Kb uncompressed, and 20Kb gzipped. This is generally not really an issue, but definitely something to keep an eye on when a website is large or growing. After the first load, the browser will cache the static search index on subsequent page loads.

Multiple search indices

Depending of the site contents, another interesting feature might be to create multiple indices. This would be straight-forward following the steps in this article.

Using Nx Affected in Azure Pipelines

Thu, 17 Mar 2022 00:00:00 GMT

Using Nx Affected in Azure Pipelines

When trying to combine the concepts of an Affected Nx projects and building and deploying them in Azure Pipelines, there is no plugin or anything readily available to do so. Since it wasn't trivial to find and compose all the bits and pieces together, I decided to write this down. Maybe it'll help you, or maybe you can help me improve it.

One Step Further

By default, many solutions use the diff between HEAD and HEAD~1 to calculate the affected projects. Such as Nrwl's own Example of setting up distributed Azure build for Nx workspace.

Although this may work well, I think this isn't always optimal. Mostly because the latest run(s) may have failed, which Nx isn't aware of. This may require to manually re-run a pipeline, or it may take an unknown amount of time before the container will re-build.

However, Azure has an API to set and list pipeline run tags. We can use the Azure CLI to add a tag for each pipeline run having a successful Nx project build.

The main steps in this guide include:

Find the latest successful build and the corresponding SHA-1.
Use this SHA-1 as the --base for the nx affected command.
Store the affected Nx project names in output variables.
Use these output variables to conditionally execute the corresponding jobs or stages to build and deploy Nx projects.
After a successful build, tag the current pipeline run (for step #1 in the next run).

Find The Latest Successful Build

When the list of pipeline runs is filtered by the Nx project's tag and sorted by time, we need only the latest result and we can further simplify the output by returning only the SHA-1 (sourceVersion) in the most concise TSV format:

az pipelines runs list
  --branch main
  --pipeline-ids $(System.DefinitionId)
  --tags "my-app"
  --query-order FinishTimeDesc
  --query '[].[sourceVersion]'
  --top 1
  --out tsv

This will return the SHA-1 associated to the latest pipeline run tagged with my-app. This is the run we are looking for, as we have tagged it only after a successful build. Now, nx affected can determine whether this Nx project is currently affected or not compared with the latest SHA-1 a successful build for this Nx project was made from.

Later we will see how to set this tag for a successful build.

Write It Down For Later

To set an output variable for use in a later stage in Azure pipelines, we need use the task.setvariable logging command (Azure docs: Set variables in scripts). This writes the value AFFECTED to the output variable BUILD_MY_APP:

echo "##vso[task.setvariable variable=BUILD_MY_APP;isOutput=true]AFFECTED"

Putting It Together

With the above ingredients, we can write a script to write the output variables. Initially I wrote a Bash script is-affected.sh as that made sense at the time. Here's the gist:


is-affected() {
  local SHA=$(az pipelines runs list --branch main --pipeline-ids $(System.DefinitionId) --tags "$2" --query-order FinishTimeDesc --query '[].[sourceVersion]' --top 1 --out tsv)
  local WRITE_VARIABLE="##vso[task.setvariable variable=$3;isOutput=true]";
  local AFFECTED=$(npx nx print-affected --type=${1} --select=projects --plain --base=$SHA --head=HEAD)
  if [[ "$AFFECTED" == *"$2"* ]]; then
    echo "${WRITE_VARIABLE}AFFECTED"
    echo "##[warning]$2 is affected (base: $SHA)"
  fi
}

As I think Bash scripts are not very robust and not easy to maintain, I ported this to a Node.js script is-affected.js with JSDoc/TypeScript annotations. The idea stays the same, and with both scripts the output looks like this:

##[warning]my-app is NOT affected (base: 62ed6e5d1dd73564a088be879a47634456a07676)
##[warning]container5 is affected (base: 62ed6e5d1dd73564a088be879a47634456a07676)
##[warning]some-lib was not previously tagged

When system diagnostics are enabled, also the other echo commands that actually set the variables are printed.

To see this script in perspective, here's an example "Prepare" stage:

stages:
  - stage: Prepare
    pool:
      vmImage: ubuntu-latest
    jobs:
      - job: Determine_Affected
        displayName: Determine Affected Nx Projects
        steps:
          - task: NodeTool@0
            displayName: Use Node.js v16.17.1
            inputs:
              versionSpec: 16.17.1

          - script: npm install nx
            displayName: Install Nx

          # Required for `az pipelines runs`
          - bash: |
              az config set extension.use_dynamic_install=yes_without_prompt
              az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project="$(System.TeamProject)"
            displayName: Set default Azure DevOps organization and project

          - bash: |
              node is-affected.js --pipelineId $(System.DefinitionId) --app my-app --app container5 --lib some-lib
            name: AffectedNxProjects
            displayName: Determine affected Nx projects
            env:
              AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

Conditional Builds

The condition for the job in another (build) stage is based on the variable that was written with the Bash script in an earlier stage. The pattern to read it:

stageDependencies.[[STAGE]].[[JOB]].outputs['[[STEP_NAME]].BUILD_MY-APP']

Also see Azure docs to Set a variable for future stages.

When this variable has a value of AFFECTED or TAG_NOT_FOUND the condition will evaluate to true and the job to build the Nx project will run. For brevity, here is only the relevant part:

jobs:
  - job: BUILD_MY_APP
    displayName: Build my-app
    condition: |
      in(stageDependencies.[[STAGE]].[[JOB]].outputs['[[STEP]].BUILD_MY-APP'], 'AFFECTED', 'TAG_NOT_FOUND')
    steps:
      - task: ...build container...

We can move the build job(s) to a template and reuse it with nxProjectName as a parameter. Here's an example how to do that:

- stage: Build
  dependsOn: Prepare
  jobs:
    - template: build-container.yaml
      parameters:
        nxProjectName: my-app
    - template: build-container.yaml
      parameters:
        nxProjectName: container5
    - template: build-container.yaml
      parameters:
        nxProjectName: some-lib

And then in the build-container.yaml template:

parameters:
  - name: nxProjectName
    type: string

jobs:
  - job: BUILD_${{ upper(replace(parameters.nxProjectName, '-', '_')) }}
    displayName: Build ${{ parameters.nxProjectName }}
    condition: |
      in(stageDependencies.Prepare.Determine_Affected.outputs['AffectedNxProjects.BUILD_${{ upper(replace(parameters.nxProjectName, '-', '_')) }}'], 'AFFECTED', 'TAG_NOT_FOUND')
    steps:
      - task: ...build container...

Note that stages referred to in stageDependencies must be part of the dependsOn option (Prepare in this example). Otherwise, the value will silently resolve to Null without warning.

Tag Successful Builds

This task should follow the build step(s) from the job above. We can tag successful build runs with the name of the Nx project ([nx-project-name]):

az pipelines runs tag add --run-id $(Build.BuildId) --tags [nx-project-name]

Now this command from the Bash script should find the latest tag for this Nx project in the next pipeline run:

az pipelines runs list --tags "[nx-project-name]" [...]

Again, you may need to set the organization and project first. Here's a complete step:

- script: |
    az config set extension.use_dynamic_install=yes_without_prompt
    az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project="$(System.TeamProject)"
    az pipelines runs tag add --run-id $(Build.BuildId) --tags ${{ parameters.nxProjectName }}
    echo "##vso[task.setvariable variable=DEPLOY_${{ upper(replace(parameters.nxProjectName, '-', '_')) }};isOutput=true]true"
    echo "##[warning]Tagged build for ${{ parameters.nxProjectName }} (BUILD_${{ upper(replace(parameters.nxProjectName, '-', '_')) }})"
  condition: succeeded()
  displayName: Tag successful build
  name: TagBuild
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

This writes the DEPLOY_MY_APP variable. In a later (deployment) stage, the same idea can be applied to read this variable and conditionally deploy the build to any environment. An example chunk of the "production" stage:

- stage: Production
  dependsOn: Build
  condition: succeeded('Build')
  jobs:
    - template: deploy-container.yaml
      parameters:
        nxProjectName: my-app
        environment: production

And then in deploy-container.yaml:

parameters:
  - name: nxProjectName
    displayName: The Nx project key
    type: string
  - name: environment
    displayName: Environment
    type: string
    values:
      - test
      - staging
      - production

jobs:
  - deployment: |
      Deploy_${{ replace(parameters.nxProjectName, '-', '_') }}_${{ parameters.environment }}
    displayName:
      Deploy ${{ parameters.nxProjectName }} to ${{ parameters.environment }}
    environment: ${{ parameters.environment }}
    condition: |
      eq(stageDependencies.Build.BUILD_${{ upper(replace(parameters.nxProjectName, '-', '_'))}}.outputs['TagBuild.DEPLOY_${{ upper(replace(parameters.nxProjectName, '-', '_')) }}'], 'true')
    strategy:
      runOnce:
        deploy:
          steps:
            - ...deploy container?...

Again, stages referred to in stageDependencies must be part of the dependsOn option (Build in this case).

I'll update this article as I find improvements. Hopefully this guide has been of some help or inspiration.

How to build a great theme toggle switch

Sat, 12 Mar 2022 00:00:00 GMT

How to build a great theme toggle switch

Today, "dark mode" is everywhere. Personally I love to use it wherever I can. This guide shows how to build your own accessible switch to toggle dark and light mode on your own website, and offer your visitors their preference. This website has a switch at the top right corner, which serves as an example.

A great solution ticks the following boxes:

Using only CSS, apply the default theme based on the setting of the operating system (OS) setting automatically.
When JavaScript is enabled, progressively enhances by showing a switch to override this default theme, which will be stored for subsequent visits.
Never shows flashing styles from one theme to another during page load.

Let's get going

So that's what we're after. Our solution depends on the prefers-color-scheme media query, reflecting the OS setting. Perhaps 10 steps sounds like a lot of work, but I promise each is small and fun!

If you want to quickly see the final solution, feel free to scroll to the of this page and find how to put it all together.

The Foundation
Prepare The Switch
Add The Switch
Activate The Switch
Hide The Switch
Remember The Switch
Check The Switch
Sync The Switch
Extra: Swapping Stylesheets
Putting It All Together

The foundation

The stylesheet should contain the theme-related variables and the media query to override them for the other theme. This way, the stylesheet automatically responds to changes in the OS setting. Let's use dark as the default theme, and override the variables when the OS setting is light:

:root {
  --bg-color: rgb(42, 42, 42);
  --font-color: rgb(250, 250, 250);
}

@media (prefers-color-scheme: light) {
  :root {
    --bg-color: rgb(250, 250, 250);
    --font-color: rgb(82, 82, 82);
  }
}

html {
  background-color: var(--bg-color);
  color: var(--font-color);
}

With only CSS, our styles with media queries respond properly to the OS setting. You can see this in action by opening this website and changing the OS setting. In macOS, this can be found in "System Preferences" and then "General":

Prepare the switch

We are going to need a switch for the user to override the default theme. First, we need two classes, matching our themes (.dark and .light):

:root,
html.dark {
  --bg-color: rgb(42, 42, 42);
  --font-color: rgb(250, 250, 250);
}

@media (prefers-color-scheme: light) {
  :root {
    --bg-color: rgb(250, 250, 250);
    --font-color: rgb(82, 82, 82);
  }
}

html.light {
  --bg-color: rgb(250, 250, 250);
  --font-color: rgb(82, 82, 82);
}

html {
  background-color: var(--bg-color);
  color: var(--font-color);
}

The styles for the "light" theme, unfortunately, are duplicated. This is required to override a "dark" OS setting, while the user prefers "light" on this website. To my knowledge, it is currently not possible to define these variables only once (e.g. by combining the media query with the html.light selector in CSS).

Add the switch

The UI element to switch the theme could be as simple or as fancy as you please. Let's take this website's switch as an example:

<label class="theme-switch">
  <button id="theme-toggle" role="switch" aria-checked="false"></button>
</label>

It could also be a checkbox as it has two states: checked or unchecked. Feel free to borrow the markup and styles from this website's switch (a slight variation of what's in this article), or find your own. There's plenty of great looking switches out there.

Activate the switch

When the user switches the toggle, the theme should follow suit. Let's make this happen by adding an event listener to our input element:

const toggle = document.querySelector('#theme-toggle');
const classList = document.documentElement.classList;

toggle.addEventListener('click', () => {
  const isChecked = toggle.getAttribute('aria-checked') !== 'true';
  const theme = isChecked ? 'light' : 'dark';
  classList.remove(toggle.checked ? 'dark' : 'light');
  classList.add(theme);
  toggle.setAttribute('aria-checked', isChecked);
});

This will swap the light and dark classes on the <html> tag when the user uses the <input> element. This will set the values of the corresponding CSS variables, effectively applying the theme. Now we have a functional theme switch! Yet there's a few more things we can do to make it even better.

Hide the switch

Without JavaScript, the switch can't do anything. So let's hide the switch, and only show it when JavaScript is enabled:

.theme-switch {
  display: none;
}

.js .theme-switch {
  display: flex;
}

We can inform CSS that JavaScript is enabled with only one line of JavaScript:

document.documentElement.classList.add('js');

Remember the switch

Using the switch, visitors can override the default theme. To also remember this setting for returning visitors, we can use JavaScript and localStorage. Let's write the theme value to localStorage when the user toggles the switch:

toggle.addEventListener('click', () => {
  localStorage.setItem('theme', toggle.checked ? 'light' : 'dark');
});

When the user comes back to visit your website later, we can read from localStorage and apply the theme by adding it as a class to the <html> element:

const theme = localStorage.getItem('theme');
if (theme) document.documentElement.classList.add(theme);

Ideally, we place this as an inline <script> tag just before the stylesheets containing the theme variables. This will make sure we will not see a flash of styling changes when the theme in localStorage does not match the user's OS theme setting.

Check the switch

Now, we have a remaining issue. Since the <input> is initially unchecked, it may initially not match the OS setting. So we need to check the checkbox to keep things in check:

const prefersLight = matchMedia('(prefers-color-scheme: light)');
const classList = document.documentElement.classList;
if (prefersLight.matches || classList.contains('light')) {
  document.querySelector('#theme-switch').checked = true;
}

This script is ideally executed before showing the switch, so before adding the js class to the <html> element.

Sync the switch

A fancy feature is to also sync the switch when the OS setting is changed. We can listen to changes to the media query, and switch the toggle, unless the theme was explicitly overridden and stored in localStorage:

const toggle = document.querySelector('#theme-switch');
const preferDark = window.matchMedia('(prefers-color-scheme: dark)');
preferDark.addEventListener('change', event => {
  if (!localStorage.getItem('theme')) {
    toggle.checked = !event.matches;
  }
});

You can see this in action by changing the OS setting, and find the theme and the switch have been toggled accordingly.

Extra: swapping stylesheets

In addition to applying theme styles based on media queries or classes, we can also swap entire stylesheets to match the theme. This website swaps the stylesheet related to syntax highlighting. There are multiple ways to achieve this. We can extend the event listener from above, and find the related stylesheet element to update its href:

const highlightSheet = document.querySelector('link[href*=hljs]');
const highlightSheets = {
  light: '/css/hljs.github.min.css',
  dark: '/css/hljs.github-dark-dimmed.min.css',
};

toggle.addEventListener('click', () => {
  const theme = toggle.checked ? 'light' : 'dark';
  if (highlightSheet) highlightSheet.href = highlightSheets[theme];
});

Putting it all together

Let's put all the bits and pieces together.

When we look at how the browser executes things, this is what we need:

Read the theme from localStorage and apply this class to <html>
Load the stylesheet containing the media query and CSS variables
Render a hidden toggle switch
Load the JavaScript containing:
1. Event handler for toggle switches
2. Event handler for OS setting changes
3. Toggle the switch to match the theme initially
4. Show the switch by adding the js class to <html>

Alternatively, as this page serves as a working example, we can "view source" into these three elements:

High in the <head> is an inline <script> tag (to read and apply the stored theme).
There is a <link> to stylesheet.css containing the styles.
The <body> has theme-switch.js for the rest of the functionality.

Migrate from getInitialProps to getServerSideProps in Next.js

Thu, 10 Mar 2022 00:00:00 GMT

Migrate from getInitialProps to getServerSideProps in Next.js

Pages in Next.js can use either getInitialProps or getServerSideProps to fetch data. This article will not repeat their documentation, but instead list relevant differences, and shows example code to migrate from one to another. This may provide some guidance when choosing or migrating between both methods.

Advantages of getServerSideProps:

No need to implement isomorphic code.
Has resolvedUrl (no need to mangle with req.originalUrl and asPath).
Types can be inferred using InferGetServerSidePropsType.
Features like notFound and redirect are available.
Preview Mode is available.

On the other hand, it has a few downsides as well:

The basePath is not available.
Return value must be serializable to JSON.
Requires and exposes the /_next/data endpoint.
This /_next/data path includes a .json extension, which may result in unexpected caching (in another layer, like a CDN or API Gateway).

An example Page.tsx using getInitialProps:

interface PageProps {
  content: unknown;
  statusCode: number;
  error?: Error;
  path: string;
}

const Page: NextPage<PageProps> = ({ content, statusCode, error, path }) => {
  return <div>{content}</div>;
};

Page.getInitialProps = async ({ req, res, asPath }): Promise<PageProps> => {
  // @ts-ignore We can count on `originalUrl`
  const resolvedUrl = req ? req.originalUrl : asPath;

  const { content, statusCode, error } = await fetchContent({ resolvedUrl });

  return { content, statusCode, error };
};

export default Page;

The same Page.tsx refactored to use getServerSideProps:

type PageProps = InferGetServerSidePropsType<typeof getServerSideProps>;

const Page: NextPage<PageProps> = ({ content, statusCode, error }) => {
  return <div>{content}</div>;
};

export const getServerSideProps: GetServerSideProps = async ({
  req,
  res,
  resolvedUrl,
}) => {
  const { content, statusCode, error } = await fetchContent({ resolvedUrl });

  return {
    notFound: statusCode === 404,
    props: {
      content,
      statusCode,
      hasError: boolean(error),
    },
  };
};

export default Page;

Notice the differences:

No need to type PageProps separately.
Use resolvedUrl directly (no isomorphic logic).
Move the return value to props.
hasError is serializable (error is not).
getServerSideProps is a separate export (not a member of Page).

Introducing the terminal to developers

Mon, 18 Sep 2017 00:00:00 GMT

Introducing the terminal to developers

The article I wish I had read when I had to open the terminal for the first time.

Introduction

Being a developer can be quite overwhelming these days. Getting familiar with a codebase and the framework(s) and libraries it uses is not the whole story. There is also a real demand of additional skills to get your job done, such as using Git, package managers, and build tooling.

Many of these tools are to be used in the terminal, which may be new and a bit frightening. But don't worry, we've all been there. This article provides an overview, along with practical examples, to get you up to speed with the terminal and some essential tooling.

Why do I need the terminal?

For certain tasks you can get away without using the terminal at all. For example, there are great GUI tools for working with Git. However, getting familiar with tools like Git from the terminal gives you more power and flexibility. In the end, a GUI is a graphical shell in front of a command-line tool. Often limited by screen estate or a minimalistic design, a GUI may feature only a subset of the underlying command-line interface. Being "closer to the metal", it can also help you to get out of trouble in case a GUI is stuck or messed up.

Frameworks and libraries for programming languages (such as JavaScript or PHP) come and go, but knowing your way around in the terminal is a skill you can use always and everywhere.

What actually is a terminal?

A terminal is text-based, and serves as the command-line interface (CLI) you can type your commands in. A shell takes these commands, and tells the operating system to execute them.

On macOS, the default terminal application is named "Terminal", and the default shell is Bash. You will also find a "terminal" in Linux distributions like Debian and Ubuntu. On Windows 10, you can install a Linux environment.

For the rest of this article, I'll be assuming you are using Bash or a similar shell.

Opening a terminal

On macOS, you can use Spotlight (⌘+Space), and type "Terminal" to find and open it.
On Linux, search for the "terminal" app and open it. You can also try the Ctrl+Alt+T combo.
On Windows 10, start the "Bash on Ubuntu on Windows" application.

The first thing you will always get is a prompt. It's what "prompts" you to type a command. The prompt might include information such as the computer and/or user name, and usually ends with a $. Behind the $ is the cursor, where you can type commands.

Last login: Sun Sep 17 21:20:17 on ttys000
lars ~ $

If your terminal has a white background color, but you prefer to have a black background color instead: this article contains instructions on how to do this.

Finding your way

In an application like Finder or File Explorer you can navigate your files and folders. In the terminal we can do the same with commands like ls and cd (to list files and change directories, respectively). For example:

Last login: Sun Sep 17 21:20:17 on ttys000
$ cd Documents
$ ls
tutorial.pdf Keynote.pdf

The last line is the default output of ls. You can pass it extra arguments (or "options") to change its behavior. For instance, there is -a to show all files (including hidden ones), and -l to show one file per line:

$ ls -a -l
total 1160
drwx------   6 lars staff     204 Sep 12 19:52  .
drwxr-xr-x  64 lars staff    2176 Sep 12 22:29  ..
-rw-r--r--   7 lars staff      42 Sep 12 19:52  .DS_Store
-rw-r--r--   7 lars staff   81238 Sep 12 19:52  tutorial.pdf
-rw-r--r--   1 lars staff   11086 Jul  5 22:35  Keynote.pdf

The arguments can also be combined into one: ls -al means the same.

Here are some essential commands to manage files and directories:

ls
cat    # Show file content
cd     # Change directory
mv     # Move or rename a file (or dir)
cp     # Copy file (or dir)
mkdir  # Make directory
rm     # Remove file (or dir)
pwd    # Output the current directory

If you want more information about any of these commands, you could either use the --help argument, or perhaps google the command. Both ways are not ideal: the former is correct and complete, but often hard to read. And the latter takes more time and needs you to switch context between the terminal and a web browser. Maybe we can improve on this. We all like readable documentation, right?

Package managers

Let's use this as a an opportunity to install a package manager, and then install a package with it. In this case, one for simplified documentation.

In a nutshell, package managers are an essential tool to install system-wide (or "global") software, or local dependencies within a project. You may have heard of Homebrew, npm, Maven, RubyGems, or apt-get. In this article, we'll be looking at Homebrew and npm.

On macOS, Homebrew is "the missing package manager". Find it at brew.sh.
Most Linux distributions (including Linux on Windows), come with a pre-installed package manager, such as apt-get or yum.

Are you working with JavaScript? You need npm (the package manager for JavaScript), which comes with Node.js. You can install Node.js via many package managers. This will also make npm available on your system.

Now we can install a package named "tldr" with either one of them. Using npm:

$ npm install --global tdlr

Or, with Hombrew:

$ brew install tldr

Now you can try for instance tdlr cd to see what I mean with readable documentation:

$ tldr cd

  cd

  Change the current working directory.
  More information: https://man.archlinux.org/man/cd.n.

  - Go to the given directory:
    cd path/to/directory

  - Go to home directory of current user:
    cd

  - Go up to the parent of the current directory:
    cd ..

  - Go to the previously chosen directory:
    cd -

We'll dive into npm in a minute. However, if you have questions or got stuck installing a package manager, then you might want to check out one of the following links:

Git

Git is a program that many projects use for version control. Basically, it allows people to work on code together, while keeping track of changes. Everybody should not be editing the same files directly, and sometimes changes need to be reverted.

Collaborating with Git involves cloning ("downloading") the code repository of a project, making changes to fix bugs or develop new features, and then pushing back ("uploading") the changes. It is common to create a separate branch first, so others can review the changes before they are merged back into the master branch.

Let's install Git (with a package manager, obviously)! Here are some options for various operating systems:

For macOS with Homebrew: brew install git
For Linux (e.g. Debian, Ubuntu): apt-get install git
For Windows: use either the Git for Windows installer or Chocolatey: choco install git

Below is a screenshot after running a few commands as an example:

$ mkdir my-first-repository
$ cd my-first-repository/
$ git init
Initialized empty Git repository in /Users/lars/Projects/my-first-repository/.git/
$ git checkout -b cool-feature
Switched to a new branch 'cool-feature'
$ echo "Some content" > somefile.txt
$ ls
somefile.txt
$ git add somefile.txt
$ git status
On branch cool-feature

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
	new file:   somefile.txt

$ git commit -m "Add some file"
[cool-feature (root-commit) b706c43] Add some file
 1 file changed, 1 insertion(+)
 create mode 100644 somefile.txt
$

For more information on working with Git, here are a two excellent tutorials:

npm

npm is the package manager for JavaScript. Even though Node.js itself is a server-side application runtime, and npm was originally built for Node modules, npm proves to be a great dependency manager for JavaScript in general. Next to JavaScript modules, the npm repository contains many tools built on top of Node.js, such as Grunt, Webpack, Babel, UglifyJS, and many more.

To manage JavaScript dependencies with npm, a project has a package.json file. It contains meta data about the project, and always includes a project name and version. Here's an example:

{
  "name": "my-awesome-project",
  "version": "1.2.6",
  "description": "My awesome project!",
  "license": "MIT",
  "repository": "git@github.com:webpro/my-awesome-project.git",
  "dependencies": {
    "lodash": "4.17.4",
    "react": "15.6.1"
  },
  "devDependencies": {
    "mocha": "3.5.3"
  }
}

Let's clone this project with Git, and install its dependencies:

$ cd Projects/
$ git clone git@github.com:webpro/my-awesome-project.git
Cloning into 'my-awesome-project'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
$ cd my-awesome-project/
$ ls -a
.  ..  .git  package.json
$ npm install
added 56 packages, and audited 57 packages in 4s
[...]
$ ls node_modules/
asap		      fbjs		 js-tokens		 loose-envify	   react
balanced-match	      fs.realpath	 json3			 minimatch	   react-is
brace-expansion       glob		 lodash			 minimist	   safer-buffer
browser-stdout	      graceful-readlink  lodash._baseassign	 mkdirp		   setimmediate
commander	      growl		 lodash._basecopy	 mocha		   supports-color
concat-map	      has-flag		 lodash._basecreate	 ms		   ua-parser-js
core-js		      he		 lodash._getnative	 node-fetch	   whatwg-fetch
create-react-class    iconv-lite	 lodash._isiterateecall  object-assign	   wrappy
debug		      inflight		 lodash.create		 once
diff		      inherits		 lodash.isarguments	 path-is-absolute
encoding	      is-stream		 lodash.isarray		 promise
escape-string-regexp  isomorphic-fetch	 lodash.keys		 prop-types
$

The command npm install installed the dependencies lodash and react (and their dependencies) locally in the node_modules directory.

Note that this installs the dependencies locally to the project. This is very different from installing packages globally, as we did with the tldr package (using the --global or -g argument). In general, local packages are dependencies for a single project and global packages are used from the command line.

If you want to learn more, I can recommend this article about npm dependencies and scripts: Using npm dependencies in npm scripts.

Tips & tricks

In this section I'd like to present a few tips and tricks to make your life in the terminal easier right away.

Change the terminal's theme

In macOS and some Linux distributions, the default background color of the terminal application is white, and the window might be slightly transparent. Yet many people prefer a black background in the terminal (like the screenshots in this article).

Here's you can change this for macOS: from the Terminal.app, go to Preferences (⌘,), go to Profiles, and select the desired profile (e.g. the "Pro" theme). Make sure to press the "Default" button to store this for later sessions as well. In the same screen, you can go into the "Background Color" modal and set opacity to 100% to remove the transparency.

Shortcuts & commands you should know

The ^ (caret) below represents the "Control" key:

Shortcut	Description
`↑`	Show the previous command
`↓`	Show the next command
`^a`	Move the cursor to the start of the line
`^e`	Move the cursor to the end of the line
`⇥`	(tab) Auto-complete commands, and directory and file names
`!!`	Run the previous command again
`^l`	Clear the screen (or `clear`)
`^c`	Cancel the current process (if it hangs or becomes unusable)
`^d`	Exit the current terminal (or `exit`)

Aliases and functions

Over time, you will see that you are using the same commands and parameters over and over again. This is where aliases and functions come in. An alias can be used as an abbreviation for a more complex command. For example:

alias ll="ls -lA --color"

Now you can use ll as an alias and it will execute the ls command including the extra arguments. You can also still add extra arguments to ll. Use alias without arguments to get a list of all active aliases.

Functions can contain logic, and a main difference with aliases is that you can pass it arguments. Here's a trivial example:

say_hi () {
  echo Hello, $1
}

You can invoke this function with say_hi John, and it would print Hello, John:

$ say_hi John
Hello, John

This is a very short and superficial introduction to this topic. If you are interested in learning more, please check out the following resources:

Wrapping up

After reading this article I hope you feel slightly more comfortable to use the terminal. I've compiled a short list of great resources to get some directions to learn more. What's next for you?

If you feel there's something important missing in this article, feel free to let me know (in a comment here, or on Bluesky). Thanks for reading!

Why and how I’m using SVG sprites over fonts for icons

Tue, 24 Mar 2015 00:00:00 GMT

Why and how I'm using SVG sprites over fonts for icons

In a recent project, I've been doing some research and testing to find the best solution for icons and small images in a web application.

The requirements included support for customization of both the background and the font color of the application. Obviously, crisp images and performance are important as well.

After trying a bit and reading up on some related articles, I came to the conclusion I wanted to go for SVG sprites, instead of an icon font. Some of the main advantages of SVG over an icon font include:

Slightly more control when styling SVG elements, since fonts are text.
Fonts might be less crisp due to anti-aliasing, or off by half a pixel.
Less trickery to make it work cross-browser.
It's easier to change SVG shapes.

It's a bummer that, given the requirements, we couldn't use SVG sprites in CSS as background images. The reason being that you can't change the fill color dynamically when the background image is set.

So, we need to resort to inline SVG images. One way is to simply use inline <svg> elements in the HTML, but this means duplication of potentially quite some SVG images in the page. Fortunately, there is a great way to reuse shapes from a single SVG file across the page!

The single SVG sprite file (say, "defs.svg") looks like this:

<svg display="none" width="0" height="0" version="1.1" xmlns="http://www.w3.org/2000/svg">
    <defs>
        <symbol id="icon-delete" viewBox="0 0 1024 1024">
            <title>delete</title>
            <path class="path1" d="M810.667 273.707l..."></path>
        </symbol>
        <symbol id="icon-info" viewBox="0 0 1024 1024">
            <title>info</title>
            <path class="path1" d="M448 304c0-26.4..."></path>
            <path class="path2" d="M640 768h-256v..."></path>
            <path class="path3" d="M512 0c-282.77..."></path>
        </symbol>
        <symbol id="icon-arrow-left" viewBox="0 0 1024 1024">
            <title>arrow-left</title>
            <path class="path1" d="M1024 512c0-282.752..."></path>
        </symbol>
    </defs>
</svg>

Note that I've cut off some path definitions for brevity.

You can use a service like the IcoMoon App and/or create custom icons using e.g. Illustrator. Then paste the SVG shapes (<path>, <polygon>, <rect>, <circle>, etc.) into a <symbol> as shown in this SVG sprite.

Now in the HTML, use <svg> images like so:

<svg role="img" title="delete">
    <use href="defs.svg#icon-delete"></use>
</svg>

This results in the browser downloading the file once, and using a cached instance afterwards. We need only a little bit of styling as a basis:

svg {
  background-color: transparent;
  fill: currentColor;
  width: 24px;
  height: 24px;
}

Now we are able to set font and background colors in any way we want!

The good thing is that this works great in most browsers. We only need to inject SVG for Everybody into our page to support Internet Explorer. I've tried it down to IE9, but there's even support for IE6–8. The script is only 1KB minified, and leaves the other browsers unharmed.

The maintenance process isn't perfect, as we need to manually edit the sprite file, but I'm still happy with it. It shouldn't be hard to write a script that concatenates a bunch of SVG files into one sprite, though. See for example svg-sprite-generator.

Happy styling!

Resources:

Managing your dotfiles

Wed, 22 Oct 2014 00:00:00 GMT

Managing your dotfiles

Once you've started out enjoying dotfiles, you may wonder about the best way to organize and manage them. Do you want to keep it small and simple? Or do you want to manage as many packages, applications and their settings as possible? Or are you an administrator and need to orchestrate many systems?

Here are some questions and pointers to consider during your dotfiles journey.

Where to store dotfiles?

Many prefer to store them in a public repository, such as GitHub or Bitbucket. This way, you make them accessible for others to get inspired or steal from. Good repositories can also easily be forked and customized to fit your particular needs. Alternatively, you can store your dotfiles in a private repository or a personal cloud storage, such as Dropbox or Google Drive.

How to install dotfiles?

You can copy and sync your dotfiles to their designated location, or create symlinks. Another option is to use one of the many tools for this. Some frameworks have this built-in. Many repositories include an installation script to ease this process.

Start from scratch or framework?

You may want to be in full control and like to know exactly what's going on in your system. Then you can start from scratch, and borrow and steal bits and pieces you like from others. On the other hand, you can put your trust in a community behind a large framework. This allows you to make a head start and quickly have all the goodness installed (including many sensible default settings).

Which shell should I use?

It's a good idea to make up your mind regarding the shell you want to use. Most commands and packages run fine on common shells, but not all. The more you customize, the more likely it is to run into compatibility issues. For example, Bash and Zsh are popular shells. My advice is to pick one, and make yourself at home.

Need to orchestrate multiple setups and/or machines?

Depending on the environment, you might be better of with more robust solutions for configuration management like Puppet, Chef, or Ansible.

Getting started with dotfiles

Wed, 16 Jul 2014 00:00:00 GMT

Getting started with dotfiles

You're the king of your castle!

tl/dr; You can set up a new system using dotfiles and an installation script in minutes. It's not hard to create your own repository, and you'll learn a ton along the road. This is truly more about the journey than the destination!

Introduction

Dotfiles are used to customize your system. The "dotfiles" name is derived from the configuration files in Unix-like systems that start with a dot (e.g. .bash_profile and .gitconfig). For normal users, this indicates these are not regular documents, and by default are hidden in directory listings. For power users, however, they are a core tool belt.

There is a large dotfiles community. And with it comes a large number of repositories and registries containing many organized dotfiles, advanced installation scripts, dotfile managers, and mashups of things people collect in their own repositories.

This article will try to give an introduction to dotfiles in general, by means of creating a basic dotfiles repository with an installation script. It is only meant to provide some inspiration, some pointers to what is possible and where to look for when creating your own.

Note that this writeup has a focus on Linux and macOS based systems.

Automate all the things!

Ideally, you store your personal files not on your machine only. If you have your files on either local drives (e.g. USB drive, NAS) or in the cloud (Dropbox, Google Docs, iCloud, etc., etc.), you save yourself from the risks of machine theft, damage, or hardware failure.

Now your documents, photos, etc. are kind of safe. Still, if you ever have to setup a system, you need to install every single application again. I can't count the times I needed to find the application's download page, download, install. Next. Next. Again. You forgot one. One more. And I did not even mention the plethora of system preferences and other configurations, which I usually can't remember when I need them. Again, I need to search.

So, how awesome is it that we can automate all this? You may not realize it, but most system tools, applications and settings can be installed in an automated fashion. I don't know about you, but this is like music to my ears!

Today, I could literally throw my laptop out of the window, buy a new one, and be up and running in a matter of minutes (not hours!).

Without breaking a sweat (apart from the $$$).

Getting started

It's pretty simple to get started. You need to organize your dotfiles in some directory. You could do this practically anywhere, like a USB drive or something. Since version control is great, a hosted git repository like GitHub is a great option to store your dotfiles.

An example dotfiles repository

For this example, I'm just going to use a subset of my own dotfiles repo.

Structure

Below is the structure of my dotfiles repo. It's also what we'll use in our walk-through below.

.
├── git
│ ├── .gitconfig
│ └── .gitignore_global
├── install.sh
├── osxdefaults.sh
├── runcom
│ ├── .bash_profile
│ └── .inputrc
├── system
├── .alias
├── .env
├── .function
├── .path
└── .prompt

The dotfiles

We'll be taking a look at the following example dotfiles:

.bash_profile
.inputrc
.alias
.functions
.env
.prompt

Startup script

In a Bash shell, this file (or .profile) in your home directory is loaded first. What to put in the .bash_profile and other dotfiles is truly worth a book alone, but we're going to give it a quick shot here anyway. I like to use a small .bash_profile that links to several others that have a dedicated purpose, i.e. one file for the aliases, one for the functions, etc. Here's an example of how you can include (or actually "source" or execute) all files in a folder:

for DOTFILE in `find /Users/lars/Projects/.dotfiles`
do
    [ -f "$DOTFILE" ] && source "$DOTFILE"
done

Full examples include my own .bash_profile, Mathias's .bash_profile. Some people like to put most of their startup configuration in one file. This is perfectly fine, as long as you keep it sane and dense.

If you want to dive into startup scripts a bit more, Peter Ward explains about Shell startup scripts, and here's another about startup script loading order.

Keybindings

The behavior of line input editing and keybindings is stored in a .inputrc file. Here's an excerpt of my own:

set completion-ignore-case on
# List all matches in case multiple possible completions are possible
set show-all-if-ambiguous on
# Flip through autocompletion matches with Shift-Tab.
"\e[Z": menu-complete
# Filtered history search
"\e[A": history-search-backward
"\e[B": history-search-forward

Full example: my .inputrc.

Aliases

Aliases allow you to define shortcuts for commands, to add default arguments, and/or to abbreviate longer one-liners. Here are some examples:

alias l="ls -la"       # List in long format, include dotfiles
alias ld="ls -ld */"   # List in long format, only directories
alias ..="cd .."
alias ...="cd ../.."
alias ....="cd ../../.."

# Recursively remove .DS_Store files
alias cleanupds="find . -type f -name '*.DS_Store' -ls -delete"

Full examples: my .alias, Mathias's .aliases

Functions

Commands that are too complex for an alias (and perhaps too small for a stand-alone script) can be defined in a function. Functions can take arguments, making them more powerful.

# Create a new directory and enter it
function mk() {
  mkdir -p "$@" && cd "$@"
}
# Open man page as PDF
function manpdf() {
  man -t "${1}" | open -f -a /Applications/Preview.app/
}

Full example: Mathias's .functions.

Environment variables

Environment variables can go in another dotfile:

export PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$DOTFILES_DIR/bin"
export EDITOR="subl -w"
export CLICOLOR=1
export LSCOLORS=gxfxcxdxbxegedabagacad
# Tell grep to highlight matches
export GREP_OPTIONS='-color=auto'
# Case-insensitive globbing (used in pathname expansion)
shopt -s nocaseglob
# Autocorrect typos in path names when using `cd`
shopt -s cdspell

Full example: my .env Mathias's .exports.

Prompt

A custom prompt can be convenient. You could, for example, show where you are in the directory tree, and/or which git branch you're currently working with. There's plenty of options here, but personally I'd like to keep this a bit easy on the eyes. Here's my prompt:

lars ~/Projects/blog main ❯

Examples: my .prompt, Color Bash Prompt, Sexy Bash Prompt, How to Customize your Bash Prompt

Other dotfiles

Many packages store their settings in a dotfile, e.g.:

.gitconfig for Git
.vimrc for Vim

Because these are basically simple text files, they are perfect to store in your dotfiles repo!

Installing the dotfiles

To "activate" the dotfiles, you can either copy or symlink them from the home directory. Otherwise they're just sitting there being useless.

Beware you probably already have a .bash_profile and .gitconfig in the user folder. So please be careful here. With great power comes great responsibility. Probably it's best to backup important files before you're moving them around.

Let's assume you have the relevant dotfiles together in ~/.dotfiles. You can create a symlink from here to the directory where they are expected (usually your home directory):

ln -sv "~/.dotfiles/runcom/.bash_profile" ~
ln -sv "~/.dotfiles/runcom/.inputrc" ~
ln -sv "~/.dotfiles/git/.gitconfig" ~

We already have the core of a dotfiles setup.

Installation script

You may want to have an installation script to automate symlinking the dotfiles in the repo to your home directory. But there's more we can put in a script that we run once to install a new system. See this Makefile for an example. Also make sure to check out other people's scripts for more ideas and inspiration.

To install the dotfiles on a new system, we can do so easily by cloning the repo:

$ git clone https://github.com/webpro/dotfiles.git
$ cd dotfiles
$

Then do the symlinking (either manually or with a script), et voilà! Now I would like to show you some more neat things you can do in your dotfiles.

Homebrew and Homebrew Cask

Let's install my favourite combo for package management in macOS, Homebrew and Homebrew Cask:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This opens up a giant repository of system tools you can install from the command line. Here's a short sample:

$ brew install node
$ brew install git
$ brew install wget

And how about macOS applications? Thanks to Homebrew Cask we have the power to install GUI applications in macOS from the command line:

$ brew install --cask atom
$ brew install --cask dropbox
$ brew install --cask firefox
$ brew install --cask google-chrome
$ brew install --cask spotify
$ brew install --cask sublime-text3
$ brew install --cask virtualbox
$ brew install --cask vlc

Please note that you can install multiple applications with a single command:

$ brew cask install alfred dash flux mou

macOS defaults

Many, many macOS settings can be set from the command line. Here's just a small sample to get an idea:

# Finder: show hidden files by default
defaults write com.apple.finder AppleShowAllFiles -bool true
# Automatically hide and show the Dock
defaults write com.apple.dock autohide -bool true
# Save screenshots to the desktop
defaults write com.apple.screencapture location -string "$HOME/Desktop"
# Save screenshots in PNG format (other options: BMP, GIF, JPG, PDF, TIFF)
defaults write com.apple.screencapture type -string "png"
# Display full POSIX path as Finder window title
defaults write com.apple.finder _FXShowPosixPathInTitle -bool true
# Disable the sound effects on boot
sudo nvram SystemAudioVolume=" "

We must credit Mathias Bynens here for creating and maintaining an awesome collection of macOS defaults in his dotfiles.

To apply the macOS defaults you've stored in e.g. macosdefaults.sh:

$ source macosdefaults.sh

This line is a perfect candidate to include in your installation script.

Updating your system

It's fine to run the installer script again, e.g. to fix some symlinks or update packages (it should be idempotent). But it's better and faster to run a couple of update commands separately. Here are some example commands to put in an alias or function to update macOS, Homebrew, npm, and Ruby packages:

# Update App Store apps
sudo softwareupdate -i -a

# Update Homebrew (Cask) & packages
brew update
brew upgrade

# Update npm & packages
npm install npm -g
npm update -g

# Update Ruby & gems
sudo gem update -system
sudo gem update

Next steps

I have always enjoyed to wade through the various existing dotfiles repos and find real gems out there. Sometimes it takes real effort to make something work the way you want it to, but eventually it makes your dotfiles really yours.

You might have missed tools like Zsh, Vim, and many more. Well, my apologies for that, but you would never reach the end of this article.

In any case, there are plenty of great resources and dotfiles covering these as well. My curated awesome-dotfiles list might be a good start.

If you have nice ideas to share or want to collaborate, feel free to send me a message or open a PR!

Getting gulpy

Mon, 05 May 2014 00:00:00 GMT

Getting gulpy

Advanced tips for using gulp.js

After getting excited about gulp.js, at some point you need more than the shiny but basic examples. This post discusses some common pitfalls when using gulp.js, plugins and streams in a more advanced and custom way.

Basic tasks

In a basic setup, gulp has a nice syntax to use streams and plugins to transform your source files:

gulp.task('scripts', function () {
  return gulp
    .src('./src/**/*.js')
    .pipe(uglify())
    .pipe(concat('all.min.js'))
    .pipe(gulp.dest('build/'));
});

This works just fine in many cases, but once you need something more tailored, you may soon face tricky situations. This post addresses some of them.

Incompatible streams?

When using gulp you may have run into the issue of "incompatible streams". This mostly has to do with the difference of regular streams versus vinyl file objects, and gulp plugins that use libraries supporting only buffers (and not streams).

For example, you can't pipe a regular Node stream directly to gulp and/or gulp plugins. Let's take a read stream, transform the contents using gulp-uglify and gulp-rename, and finally write the result to disk with gulp.dest(). Consider this (erroneous) example:

var uglify = require('gulp-uglify');
var rename = require('gulp-rename');

gulp.task('bundle', function () {
  return fs
    .createReadStream('app.js')
    .pipe(uglify())
    .pipe(rename('bundle.min.js'))
    .pipe(gulp.dest('dist/'));
});

Why can't we pipe a read stream to a gulp plugin? Gulp is the streaming build system after all, right? Yes, but the example above ignores the fact that gulp plugins expect Vinyl file objects. You can't just pipe a read stream to a function (plugin) that expects vinyl file object(s).

The vinyl file object

Gulp uses vinyl-fs, from which it inherits the gulp.src() and gulp.dest() methods. Vinyl-fs uses the vinyl file object, its "virtual file format". If we want to use gulp and/or gulp plugins with a regular read stream, we need to convert the read stream to vinyl first.

A great option is to use vinyl-source-stream, which does exactly that:

var source = require('vinyl-source-stream');
var marked = require('gulp-marked');

fs.createReadStream('*.md')
  .pipe(source())
  .pipe(marked())
  .pipe(gulp.dest('dist/'));

The next example starts with a Browserified bundle and eventually converts this to a vinyl stream.

var browserify = require('browserify');
var uglify = require('gulp-uglify');
var source = require('vinyl-source-stream');

gulp.task('bundle', function () {
  return browserify('./src/app.js')
    .bundle()
    .pipe(source('bundle.min.js'))
    .pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

Great. Note that we don't need to use gulp-rename anymore, since vinyl-source-stream creates a vinyl file instance with the specified filename (which gulp.dest will use to write the bundle).

gulp.dest

This gulp method creates a write stream, and is really convenient. It reuses the file names from the read stream, and creates directories (using mkdirp) as necessary. After writing, you can continue piping the stream (e.g. to also gzip the data and write the result to other files).

Streams and buffers

Since you're interested in using gulp, this post simply assumes you have some basic knowledge of streams. Vinyl works with virtual files containing either a buffer or a stream (or null). With a regular read stream you can listen to emitted chunks of data:

fs.createReadStream('/usr/share/dict/words').on('data', function(chunk) {
    console.log('Read %d bytes of data', chunk.length);
});

> Read 65536 bytes of data
> Read 65536 bytes of data
> Read 65536 bytes of data
> Read 65536 bytes of data
> ...

In contrast, gulp.src() emits buffered vinyl file objects back to the stream. This means you won't get chunks, but (virtual) files with buffered contents. The vinyl file format has a contents property representing a buffer or a stream, and gulp is using buffers by default:

gulp.src('/usr/share/dict/words').on('data', function(file) {
    console.log('Read %d bytes of data', file.contents.length);
});

> Read 2493109 bytes of data

This clearly shows the data is buffered before the file gets emitted to the stream as a whole.

Gulp uses buffers by default

Although in general it's recommended to stream the data, many plugins have underlying libraries that work with buffers. Sometimes this is simply necessary for transformations that require the source contents as a whole. Consider for instance text-based replacements with regular expressions. You would run the risk of matching patterns being in separate chunks, failing to find those matches. Likewise, tools like UglifyJS and the Traceur compiler need complete files as their input (or at least syntactically complete strings of JavaScript).

This is why gulp is using buffered streams by default, since they're just easier to work with.

The downside of using buffered content is that they are inefficient for large files. The file is read completely, before it is emitted back to the stream. The question is, for which file sizes does this really hurt performance? For regular text files such as JavaScript, CSS, templates, etcetera there's likely just minimal overhead in using buffers.

In any case, you can tell gulp to pass on a stream for contents if you set the buffer option to false. Here's a contrived example:

gulp.src('/usr/share/dict/words', {buffer: false}).on('data', function(file) {
    var stream = file.contents;
    stream.on('data', function(chunk) {
        console.log('Read %d bytes of data', chunk.length);
    });
});

> Read 65536 bytes of data
> Read 65536 bytes of data
> Read 65536 bytes of data
> Read 65536 bytes of data
> ...

From streams to buffers

Depending on the desired input (and output) stream, and depending on the gulp plugin, you may need to switch from streams to buffers (or vice versa). As said, most plugins work with buffers (although some of them also support streams). Examples include gulp-uglify and gulp-traceur. You can do the conversion to buffers using gulp-buffer:

var source = require('vinyl-source-stream');
var buffer = require('gulp-buffer');
var uglify = require('gulp-uglify');

fs.createReadStream('./src/app.js')
  .pipe(source('app.min.js'))
  .pipe(buffer())
  .pipe(uglify())
  .pipe(gulp.dest('dist/'));

Or, another contrived example:

var buffer = require('gulp-buffer');
var traceur = require('gulp-traceur');

gulp
  .src('app.js', { buffer: false })
  .pipe(buffer())
  .pipe(traceur())
  .pipe(gulp.dest('dist/'));

From buffers to streams

You can also "streamify" the output of a plugin working with buffers (back) to a read stream by using gulp-streamify or gulp-stream. Then plugins that work (only) with streams can be used before and after the buffer-based plugin:

var wrap = require('gulp-wrap');
var streamify = require('gulp-streamify');
var uglify = require('gulp-uglify');
var gzip = require('gulp-gzip');

gulp
  .src('app.js', { buffer: false })
  .pipe(wrap('(function(){<%= contents %>}());'))
  .pipe(streamify(uglify()))
  .pipe(gulp.dest('build'))
  .pipe(gzip())
  .pipe(gulp.dest('build'));

You don't need a plugin for everything

Although there are many plugins out there that are very useful and convenient, some tasks and transformations can easily be done without Yet Another Plugin™. Plugins do cause some overhead in that they make you depending on an extra npm module, a plugin interface, (unresponsive?) maintainer, etc. If it's very easy to do the task at hand without a plugin, or to directly use the original module, then in most cases I would recommend to do so. It's important to understand the concepts I've described above to make the right decision in your situation. Let's take a look at some examples.

vinyl-source-stream

In our examples above we've already seen an example of using Browserify directly instead of the (blacklisted) gulp-browserify plugin. The key here is to use vinyl-source-stream (or similar) to allow for regular read streams as input to Vinyl plugins.

Textual transformations

Another example is string-based transformations. Here is a very basic plugin to use directly with vinyl buffers:

function modify(modifier) {
  return through2.obj(function (file, encoding, done) {
    var content = modifier(String(file.contents));
    file.contents = new Buffer(content);
    this.push(file);
    done();
  });
}

You could use this plugin like this:

gulp.task('modify', function () {
  return gulp
    .src('app.js')
    .pipe(modify(version))
    .pipe(modify(swapStuff))
    .pipe(gulp.dest('build'));
});

function version(data) {
  return data.replace(/\_\_VERSION\_\_/, pkg.version);
}

function swapStuff(data) {
  return data.replace(/(\\w+)\\s(\\w+)/, '$2, $1');
}

The plugin is unfinished and doesn't even deal with streams. However, it shows it's possibly easy to create new transformations using some basic functions. The through2 library is a great wrapper to Node streams and enables transform functions as shown above.

Task orchestration

In case you need some custom or dynamic tasks to run, it's useful to know that gulp is using the Orchestrator module. The gulp.add method is Orchestrator.add (actually all methods are inherited from the Orchestrator module). But, why would you need this?

You don't want to clutter the list of gulp tasks with "private" tasks (i.e. not exposing them to the CLI tool).
You need more dynamic and/or reusable sub-tasks.

Closing thoughts

Please note that gulp itself (or Grunt) itself is not always the best tool for the job. If, for instance, you just need to concatenate and uglify a couple of Javascript files, or you need to compile some SASS files, you may want to consider using Makefiles or npm run and get a lot done from the command line. Less dependencies and less configuration can be truly liberating.

Read up on Task automation with npm run to learn more. Just make sure you define clearly what you need on a scale of "build customization", and what would be the best tool(s) for the job.

However, I think gulp is a great build system that I love to use and really introduced to me the power of streams in Node.js.

Hope this helps! If you have any feedback or additional tips, please let me know in the comments or on Bluesky.

The $ object demystified

Fri, 24 Jan 2014 00:00:00 GMT

The $ object demystified

Wrap Like An Egyptian

Let's take a quick look at querySelector-based libraries such as jQuery and Zepto. You're probably familiar with their syntax:

var $items = $('.items');

Once you've queried some elements, there's a lot you can do with those elements, such as adding classes (e.g. $el.addClass('active')), insert other elements, add event listeners, and so on.

Elements vs. API

The elements being returned from the the call to $(selector) represent an array of matching DOM elements, while the API methods that come with them are properties to an object. To combine them, it might seem ideal if any array of elements would have its prototype set to the API object. The API prototype object could then be shared across each wrapped object, which would be very efficient. However, we can't just set the prototype of an array (and it's not a good idea to extend that prototype directly with a bunch of mostly unrelated methods). So how could this wrapping of things be implemented?

Implementation options

This leaves us with a couple of less optimal options. For example:

Use the array and assign all members of the API as properties to the array.
Use the array and set its __proto__ member to the API object.
Use Object.create(), and assign all DOM elements as indexed members to the object.
Use a constructor and use the API object as its prototype. Assign all DOM elements as indexed members to the object.

Here's a basic, untested implementation of each:

Array with iteration over API methods

function $(selector) {
  var collection = document.querySelectorAll(selector),
    wrapped = [].slice.call(collection);
  for (var method in MyAPI) {
    wrapped[method] = MyAPI[method];
  }
  return wrapped;
}
var $myCollection = $('.items');

Array with __proto__

function $(selector) {
  var collection = document.querySelectorAll(selector),
    wrapped = [].slice.call(collection);
  wrapped.__proto___ = MyAPI;
  return wrapped;
}
var $myCollection = $('.items');

Object.create with iteration over elements

function $(selector) {
  var collection = document.querySelectorAll(selector),
    wrapped = Object.create(MyAPI);
  for (var i = 0, l = collection.length; i < l; i++) {
    wrapped[i] = collection[i];
  }
  return wrapped;
}
var $myCollection = $('.items');

Constructor with iteration over elements

function $(selector) {
  var collection = document.querySelectorAll(selector);
  for (var i = 0, l = collection.length; i < l; i++) {
    this[i] = collection[i];
  }
}
$.prototype = MyAPI;
var $myCollection = new $('.items');

Each of the options require an iteration over either the elements or the API members. That's exactly why they're less optimal options. Depending on the length of either the elements or the API, this might end up expensive. But that's not even mentioning that it's generally considered bad practice to either augment an object with array members, or vice-versa.

jQuery and Zepto

How are the big guys doing it? Basically, jQuery follows strategy #4, while Zepto uses the __proto__ (#2).

`Object.proto`

Let's consider the __proto__ strategy for a moment. Since an array is also an object in JavaScript, it makes sense to use Object.prototype.__proto__ (or ES6's upcoming Object.setPrototypeOf). And it actually works in most browsers, except for Internet Explorer IE10 and below. Another downside is that it isn't fast, especially when combined with the obligatory Array conversion (Array.slice or iteration). Because in more real-world scenario, array-like collections such as NodeList and ElementList should be converted to static collections, as having live NodeLists might lead to unexpected behavior. So you'd still need the iteration.

Performance

During a bit of isolated benchmarking, this gives interesting and wildly varying results across browsers and number of elements. Actually setting the __proto__ itself makes the strategy to be performing slightly worse than the others.

Wrapping up

In most situations I would go with the constructor approach, while making an iteration over the array of elements (#4). This is a safe option with regards to browser support, works everywhere today and tomorrow, and in my benchmarking came out performing very well across browsers. jQuery essentially does the same thing, and it's also what I ended up doing myself in DOMtastic.

Feel free to check out the DOMtastic project if you'd like to see code, run benchmarks, and/or see their results.

Related resources

Bubbling events in detached DOM trees

Tue, 21 Jan 2014 00:00:00 GMT

Bubbling events in detached DOM trees

Here's a quick post on the topic. Sometimes we need events to still work in a detached DOM tree. Even though the end-user can't really interact with detached trees, DOM elements in that tree can still listen to other events and react to them. This might also be efficient performance-wise, since changes in detached trees don't trigger repaints.

Detached DOM trees

We'll talk about how to support events in detached DOM trees, and how to do this in a performant way. First, a detached DOM tree is an HTML fragment that's not in the current document (e.g. you won't find it in the Element Inspector of your debugger), but it's still referenced in memory. Use cases where they come in useful include:

Fragments that were just rendered with a template engine, and ready to be inserted to the DOM.
Fragments that are attached and detached to minimize repaints while their DOM structure is modified.
Fragments that act as fixtures or sandboxes during tests.

In any of these situations, it can be very helpful if events would be able to bubble up, even though it's not attached to the document yet.

What most libraries do is either not support this at all, or not in the most optimal way. For example, Zepto does not support it, and jQuery does something similar to what I usually see:

while (element.parentNode) {
  element.dispatchEvent(event);
  element = element.parentNode;
}

This might work for either attached or detached DOM trees: just dispatch the event on each ancestor of the targeted element (often by calling the "trigger" method).

However, wouldn't it be better if we let the browser do all the work, and just let the event bubble up the tree (while dispatching only a single event without traversing the tree)?

Detect support

Here's a way to detect if a browser supports bubbling events in detached DOM trees:

var isEventBubblingInDetachedTree = (function (global) {
  var isBubbling = false;
  var doc = global.document;
  if (doc) {
    var parent = doc.createElement('div'),
      child = parent.cloneNode();
    parent.appendChild(child);
    parent.addEventListener('e', function () {
      isBubbling = true;
    });
    child.dispatchEvent(new CustomEvent('e', { bubbles: true }));
  }
  return isBubbling;
})(this);

In a browser that supports this, dispatching the event is enough to have the event bubble up. Currently, at least in IE10, IE11 and Firefox you can take advantage of this.

In other browsers, you still need to dispatch the event on each element in the ancestor chain. Here's a snippet to demonstrate what this might look like:

if (
  !params.bubbles ||
  isEventBubblingInDetachedTree ||
  isAttachedToDocument(element)
) {
  element.dispatchEvent(event);
} else {
  triggerForPath(element, type, params);
}

I think this code is quite self-explanatory. See DOMtastic's event implementation for an extended example.

Wrapping up

Bubbling events might not be your biggest (performance) issue, but I think it's good to know how to deal with them anyway. Including situations where you're not using jQuery to handle this for you.

My takeaways from “Clean Code”

Tue, 28 May 2013 00:00:00 GMT

My takeaways from "Clean Code"

To write clean code, you must first write dirty code and then clean it.

With pleasure I have been reading Clean Code by Robert C. Martin. The book is a nice read, with short chapters. However, just reading the book has no value. You will need to recognize the "smells and heuristics" in your day to day work, and act on them. This requires labor and dedication, which will gradually enhance your level of experience. The power of this book, at least to me, lies in defining and describing many heuristics, making them easier to recognize.

The book is full of takeaways, and below is a small selection from the book that drew my attention most.

Flag arguments are ugly

Perhaps the only exception is for specific setters that directly set the value of an object property (flag?) itself. But I have to agree that flags implicitly mean that the method is probably doing too much (e.g. there is no Command Query Separation).

Minimize the number of arguments

I had seen the term of dyadic functions before, but the term "dyadic" is hardly used in programming conversations. I also think that when you do use the term, you still have to explain what it means! Let alone "dyads" and "triads"...

Anyway, it is always a good programming advice to minimize the number of arguments. Zero or one argument is easiest to understand and maintain.

Have no side effects

Very valuable advice. Simple to understand, and shouldn't be too hard to implement. Probably sometimes side effects happen when writing a method at first, but during refactoring such "lies" should be taken care of, and removed.

Avoid output arguments

Output arguments are objects that the method operates on, and then returns. An example:

extendWall(h);

Does this function extend "h" with a wall? Or is the wall extended with "h"? And what would it return? It's more clear to use "this" as the output argument:

h.extendWall();

Command Query Separation

Functions should either do something or answer something. That's practical and clear advice.

Comments are fails

Truth can only be found in one place: the code.

The author has a clear opinion on comments. He considers every single comment written as a failure, because the code apparently isn't expressive enough. One reason, which is hard to deny, is that comments are often badly maintained. Investing time in proper and descriptive naming in the code is a rewarding practice. Still, I think it's fine to explain the "why" of code where code alone simply is not expressive enough to easily understand what's going on. But the takeaway here to me is that "the only truth is in the code".

The purpose of formatting

Your style and discipline survives, even though your code does not.

I'm a big advocate of clear coding standards, but I didn't take it as far as this. But ultimately, I think this is true. Maintainability and extensibility are always top priority, more so than some implementation details. Still, conventions alone will take you nowhere.

Don't pass null

Simply do not return or pass null. It's better to use "empty" versions of the type that's being expected, e.g. an empty array, string or object. This way, the receiving code doesn't have to check the type. Unless you are writing some public, robust API. But internally, it saves a lot of exception handling to minimize such usage of null values.

Learning tests are better than free

Writing (unit) tests are an absolutely smart way to learn and exercise a (new) interface. It gives a feel about something you need to learn anyway. Tests from both simple and exercising production code can serve as documentation along the way.

Tests enable the -ilities

It's the tests that keep our code flexible, maintainable, and reusable [...] Because tests enable change.

Think about that for a while, and probably you will appreciate tests even more.

Getting clean via emergent design

Any design is considered "simple" if it:

Runs all the tests
Contains no duplication
Expresses the intent of the programmer
Minimizes the number of classes and methods

Making a system testable motivates (or forces) to implement established programming principles, which leads to better designs. Then, the rest follows with incremental refactorings which can be done because of the tests. The takeaway for me here is that tests both motivate and catalyse refactoring to better designs.

Conclusion

There are many more principles, patterns, and practices in the book. This list summarizes what stood out for me most. I think any serious programmer will pick up something useful from reading this book. Highly recommended!

Frontend Ramblings feed

Shell function to show line in file with context

Shell function to show line in file with context

Usage

Script

TIL

In closing

Beyond Git aliases: git clone + cd

Beyond Git aliases: git clone + cd

Privileged, but not a clown

Privileged, but not a clown

Part 1. How it started

View Source

Enthusiasm

Movement

Open

Act of kindness

Win

Privilege

Part 2. How we got here

Return the favor

Sustainable

Initiatives

Incentives

Bills

AS-IS

Trust

Space

Stories

Part 3. How it's going

Stability

Knip

Reward

Cycle

Trying

Clown

Care

Using subpath imports & path aliases

Subpath imports

Problem

Solution (option 1)

TypeScript path aliases

Problem

Solution

Option 2: Build time resolution

Option 3: Runtime resolution

Recommendations

1. Relative paths

2. Subpath imports

3. Path aliases + build time resolution

Closing Note

Resources

The State of Benchmarking in Node.js

The State of Benchmarking in Node.js

Contents

Macro benchmarking

Ecosystem

Example: PerformanceObserver for functions

Example: timerify application code

Micro benchmarking

Ecosystem

Pitfalls

Example: string concatenation

Benchmark.js

Tinybench

A CLI, maybe?

Conclusion

Versioned documentation with Starlight & Vercel

Versioned documentation with Starlight & Vercel

Introduction

A Temporary Solution

The Plan

Configure the versioned branch

Create a new project in Vercel

Configure rewrites for Vercel

Navigation

How to use a compiled bin in a TypeScript monorepo with pnpm

How to use a compiled bin in a TypeScript monorepo with pnpm

Problem

Solution

A note about `postinstall` scripts

The `openai` package

`temperature`

`presence_penalty` and `frequency_penalty`

`name`