Unshorten (expand) short URLs with Node.js

Expand short URLs programmatically in Node.js with tall library

Short URLs have been an invaluable tool for social media marketing for so many years and we are now used to seeing them everywhere. Most of the credit probably goes to URL shorteners services like Bit.ly, Goo.gl, YOURLS and Rebrandly that popularised the concept and made easy for everyone to start creating short URLs.

When working with URLs in some automation scenarios like analytics, information crawling, data retrieval, etc. it can be important to resolve (or “unshorten” or “expand”) short URLs, which means retrieving the original long URL.

In this article, we are going to see how short URLs work and how we can “expand” them into their original URL.

How short URLs work

A short URLs is a regular URL that most of the time results very short by following a very simple format:

http://<domain>/<id>

The shorter the domain and the id (often called also slashtag), the shorter will, of course, be the URL (e.g. http://loige.link/b).

What happens behind the scene of an URL shortener service is that there is a big database table that contains a map of all the existing short URLs and the related full-length URLs.

A URL shortener service lookup table might look like the following:

DomainIdUrl
loige.linkbhttps://www.nodejsdesignpatterns.com/
loige.linkghhttps://github.com/lmammino
rebrand.lyluciano-twhttps://twitter.com/loige

Note that in this example the table supports multiple domains and that the pair domain-id must be unique in the table.

URL shorteners have to expose a web server which will be reached through all the domains mapped (in fact, most of the URL shortener services — like Rebrandly — supports different domains or even the possibility to associate your own branded domain).

For every request that satisfies the short URL format, domain and id are extracted and a query is executed against the database to search for the corresponding long URL. Finally, if a match is found, the web server responds with a redirect message, sending the browser of the user to the corresponding long URL.

Resolving short URLs

URL shorteners need to talk with browsers so they have to work as any other web server adopting the HTTP standard. The following diagram illustrates how the HTTP protocol is used when a user clicks a short URL:

How short URL services work

  1. A user clicks on a short URL and the browser issues a GET request to that URL.
  2. The request hits the short URL service server which executes a look-up on its database to find the associated long URL. The long URL is returned as a redirect response to the user browser which means an HTTP 301 Moved Permanently response using the header Location to specify the new (long) URL.
  3. The browser automatically follows the redirect response issuing a new GET request to the long URL displaying the content of that URL to the user.

This is what happens when a user through a browser deals with a short URL, everything happens behind the scenes and the user won’t even notice that there was an exchange with a short URL server.

Simple enough, right?

Resolving short URLs in Node.js

Now that we know how short URL services work, it shouldn’t be hard to create a Node.js script that is capable of resolving any type of short URL.

var request = require('request')

var uri = 'http://bit.ly/2fR0xVj'
request(
  {
    uri: uri,
    followRedirect: false,
  },
  function (err, httpResponse) {
    if (err) {
      return console.error(err)
    }
    console.log(httpResponse.headers.location || uri)
  },
)

For simplicity, in this script, we are using the request module to perform the HTTP request. By default, this library follows redirects so we need to explicitly disable this behaviour by specifying the option followRedirect: false. By doing so we have an opportunity to catch a redirect response in our callback and so we can read its Location header which will represent the expanded URL.

In case we are not dealing with a short URL, this script will print back the original URL passed to the function.

A Promise-based, dependency free, Node.js library for expanding short URLs

You can easily adapt the code above to work with your specific use case, but if you want a ready-made and easy to use URL-unshortener NPM library you can trust a library I recently published called tall.

tall is a dependency-free, promise-based library that allows you to easily resolve a short URL and get its corresponding long URL.

To understand how easy to use tall is, just check out this ES2015 sample usage:

import { tall } from 'tall'

tall('http://www.loige.link/codemotion-rome-2017')
  .then((unshortenedUrl) => console.log('Tall url', unshortenedUrl))
  .catch((err) => console.error('AAAW 👻', err))

More examples are available on the official repository.

Bonus: Resolving short URLs with CURL

Of course, you can also use the evergreen CURL to issue the HTTP request for a short URL and then read the Location header to retrieve the long URL:

curl --head "http://bit.ly/2fR0xVj"

This will output something like this:

CURL unshorten short URL output

As you can see the Location header reports our long URL.

If you are a bash and Unix lover you are probably already thinking to pipe this output to grep and other unix command line tools to extract only the long URL out of the full output. If you are curious about this kind of approach you can read a similar article I previously wrote titled “Extracting data from Wikipedia using curl, grep, cut and other shell commands”.

Recap

In this simple article, we learned how short URL services work and thus how we can use them in a programmatic way if we need to know the associated long URL.

I hope this was interesting for you and that you had fun reading the article and the code examples.

As usual, I’d love to have your opinion and know if you plan to use this knowledge or the tall library in one of your next projects. Feel more than welcome to let me know this in the comments!

See you next time!

Sharing is caring!

If you got value from this article, please consider sharing it with your friends and colleagues.

Found a typo or something that can be improved?

In the spirit of Open Source, you can contribute to this article by submitting a PR on GitHub.

You might also like

Cover picture for a blog post titled Introducing Gulp cozy - Manage your gulp tasks in a cozier way

Introducing Gulp cozy - Manage your gulp tasks in a cozier way

Gulp-cozy is an experimental NPM package that allows you to separate Gulp tasks into small modules inside a dedicated folder, making them easier to maintain. It brings Node.js modularity principles into your Gulp workflow.

Calendar Icon

Cover picture for a blog post titled Migrating from Gatsby to Astro

Migrating from Gatsby to Astro

This article discuss the reason why I wanted to migrate this blog from Gatsby to Astro and the process I followed to do it. Plus a bunch of interesting and quirky bugs that I had to troubleshoot and fix along the way.

Calendar Icon

Cover picture for a blog post titled Middy 1.0.0 is here

Middy 1.0.0 is here

The middleware framework Middy reached version 1.0, bringing middleware capabilities to AWS Lambda. This allows cleaner handler code by extracting cross-cutting concerns into reusable middleware.

Calendar Icon

Cover picture for a blog post titled Lean NPM packages

Lean NPM packages

Learn how to configure NPM packages to publish only the files needed by users, avoiding bloating node_modules folders.

Calendar Icon

Cover picture for a blog post titled Fastify and Preact for quick web app prototyping

Fastify and Preact for quick web app prototyping

This article shows how to quickly build web app prototypes using Fastify for the backend API and Preact for the frontend UI. It also covers how to dockerize the app for easy sharing. Key points are the plugin architecture of Fastify, the lightweight nature of Preact, and the use of htm for defining UI without transpilation.

Calendar Icon

Cover picture for a blog post titled JavaScript iterator patterns

JavaScript iterator patterns

This article explores different ways to create iterators and iterable values in Javascript for dynamic sequence generation, specifically using functions, iterators, iterables and generators. It provides code examples for implementing the Fibonacci sequence with each approach.

Calendar Icon