NodeJS: migrate from deprecated Request package to Got

OMG!!! The Request package is deprecated… some tips beyond the docs and migration guide when moving to Got.

Many of you may use the quite famous Request library in numerous projects as I also did. Unfortunately there’s a problem…

There’s also an issue on GitHub describing in detail why this decision was made. At the end, the only conclusion is: time to change and migrate to another http-client library.

Possible alternatives and decision for Got

Luckily there are numerous good alternatives available to migrate to, e.g. node-fetch, superagent, axios etc. A quite comprehensive comparsion that may help to choose one of them is provided in the documentation of Got.

Here are the focal points why my decision was made to migrate to Got:

Basic migration steps

Luckily migrating from request to Got isn’t that difficult – except for a few little things. The original developers of Got provide a very helpful and quite complete migration guide.

Following this tutorial most of the work should be done and you should have Got up & running.

However neither the migration guide nor the quite comprehensive documentation contain examples for two – which I think – very, very common tasks when using a http-client library. Let’s have a look on that.

Authorization Header with Bearer token

When creating an web API today you always have to deal with security. One common pattern to control and restrict access to your API is the use of a bearer token, e.g. by using jsonwebtoken. Normally the token is sent in the HTTP authorization header of a request, like so:

Authorization: Bearer YOUR_TOKEN_HERE

Now when looking into Got’s documentation it becomes absolutely clear that should be no problem but at the end there isn’t any helpful example on how to achieve this.

Setting the Authorization header to the desired bearer token value could be achieved by using Got’s hooks feature to modify HTTP headers before a request is made. Here is an example on how to do that.

const got = require('got');

// get url and token
const url = 'YOUR_URL_HERE';
const token = 'YOUR_TOKEN_HERE';

got.get(url), {
  responseType: 'json',
  hooks: {
    beforeRequest: [
      options => {
        options.headers['Authorization'] = 'Bearer ' + token;
      }
    ]
  }
}).then(response => {

}).catch(error => {
        
});

Easy, right? The only annoying thing here is that you don’t want to write the entire hooks block for every request you make. To mitigate this I simply extracted a simple helper module.

module.exports.getHooks = function (token) {
  return {
    beforeRequest: [
      opts => {
        opts.headers['Authorization'] = 'Bearer ' + token;
      }
    ]
  };
};

Having that in place you can simplify the code for every request, like so:

const got = require('got');
const hooks = require('../utils/got-hooks');

// get url and token
const url = 'YOUR_URL_HERE';
const token = 'YOUR_TOKEN_HERE';

got.get(url, {
  responseType: 'json',
  hooks: hooks.getHooks(token)
}).then(response => {
  // ...handle result...
}).catch(error => {
  // ...handle error...
});

That looks good. So let’s go on with the second little issue…

Downloading a file using streams and the original file name

Quite often you may want to download a file with your http-client library. To do so, using streams is a very efficient and straight forward way. Got fully supports streams and there are quite some examples on how to down-stream a file using this library. But there’s one little issue: all examples require you to know the name of the downloaded file before making the request.. uhhh!

Normally it’s working a bit different. If you request a download from an API, the response would contain a HTTP header which gives you the filename, like so:

content-disposition:'attachment; filename="IMG_2358.JPG"'

Obviously this header field is not known before firing the request. So how can we kind of “intercept” this HTTP header to know where to stream the request body to?

The answer is to handle the response event of the download stream. There, all the response headers are available but the “streaming” of the file hasn’t started yet. Here’s how to do that…

const got = require('got');
const stream = require('stream');
const fs = require('fs');
const { promisify } = require('util');
const pipeline = promisify(stream.pipeline);

// instantiate the download stream - use options to set authorization header etc. if needed 
let downStream = got.stream('https://example.com/download');

downStream.on('response', response => {
  // response header will typically include the filename in content-disposition, like
  // content-disposition:'attachment; filename="IMG_2358.JPG"'
  let cd = response.headers['content-disposition'];
  let filename = decodeURIComponent(cd.substring(cd.indexOf('=') + 2, cd.length - 1));
  let outStream = fs.createWriteStream(filename);
  pipeline(downStream, outStream)
      .then(() => { /* ...handle success... */ })
      .catch((err) => { /* ...handle error... */ });
});
downStream.on('error', err => {
  // ...handle error...
});

Please not that also the error event should be catched, e.g. to handle refused connections because the target host isn’t reachable.

With this setup you can download any file using streams and save it under its original filename provided by the response header.

Of course you can combine this with the Bearer token utility we developed before, if the download needs authorization.

let downStream = got.stream('https://example.com/download', { hooks: hooks.getHooks(token) });

The retry-trap

After migrating a middleware component which now uses Got to call other microservices behind the scenes, I realized that in some cases the performance of frontend AJAX requests to that middleware have lost dramatically in performance. Response tooks seconds instead of milliseconds as it was before when the request library was used… uhh, why that?

Investigating a bit further it became clear that the poor performance is directly connected to some HTTP response codes returned from the called microservices, especially faulty ones like 500. Digging into the middleware’s log I saw that the slow-down was caused because in that scenarios three instead of one request where made by Got (with some time in between). At the end this results in quite a “waiting period” before the middleware could respond to the UI.

Digging deeper into the very comprehensive, but out of my perspective not so clearly structured, documentation of Got, the problem was found…

And below on options.retry…

Ahh ok – this means that Got has some defined HTTP response codes and error codes on which it will automatically retry requests for two times unless you disable that feature.

I think automatically retrying requests is a nice feature but if it is the best way to turn it on by default… don’t think so, but anyway. At least the solution to our problem is now clear.

got.get(url, {
  retry: 0
}).then(response => {
  // ...handle result...
}).catch(error => {
  // ...handle error...
});

And here we go… all responses will come back without any retry-delay, also the 500s.

Hope this helps you a bit on migrating to Got.

Useful links