Problems with a traditional MongoDB installation on Fedora

Trying to install a MongoDB community server locally on a newer version of Fedora (in my case MongoDB 5 on Fedora 36) could be a mess. In the good old days using a repo with DNF was the best option…

$ cat /etc/yum.repos.d/mongodb-org-5.0.repo
[mongodb-org-5.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/5.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-5.0.asc
$ dnf install mongodb-org
Last metadata expiration check: 0:10:51 ago on Tue Nov 22 21:14:41 2022.
Error: 
 Problem: conflicting requests
  ...
  - nothing provides /usr/libexec/platform-python needed by mongodb-org-database-tools-extra-5.0.0-1.el8.x86_64
  (try to add '--skip-broken' to skip uninstallable packages)

Since needed dependencies have been dropped in newer Fedora releases (platform-python at least), this doesn’t work any more out-of-the-box. Falling back to a manual rpm install, this also is not working straight forward since MongoDB is relying on older crypto libraries not shipped with current Fedora releases…

$ rpm -ihv mongodb-org-server-5.0.14-1.el8.x86_64.rpm 
warning: mongodb-org-server-5.0.14-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID e2c63c11: NOKEY
error: Failed dependencies:
        libcrypto.so.1.1()(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libssl.so.1.1()(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libssl.so.1.1(OPENSSL_1_1_0)(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libssl.so.1.1(OPENSSL_1_1_1)(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64 

You could now investigate hours on how to solve this problems manually, but at the end of the day keep in mind that MongoDB also clearly states that they don’t provide package support for Fedora. So this is a good starting point to think about a more stable & standard alternative, which could be running MongoDB as a Docker container…

Migration goals

Assuming you already have a locally running MongoDB in an older version of Fedora (or any other Linux distribution), this guide focusses on how to:

Moving from local MongoDB installation to dockerized version

Step 1: Backup existing MongoDB data and configuration

First we’ll need to locate and save all of the current data.

Step 2: Prepare the new environment

To run MongoDB using docker with data and configuration hosted outside the container for flexibility reasons we should first create an appropriate place for that. To do so, switch to root and create a new user mongod and two directories data and conf under /var/db to separate all from existing users and home directories.

$ useradd -M mongod
$ usermod -L mongod
$ cd /var/db
$ mkdir mongo
$ mkdir mongo/data
$ mkdir mongo/conf

The created user mongod has no home directory and is also not permitted to log-in using an password as both is not needed in our scenario. Having this in place, copy the saved MongoDB data from step 1 to the new directories. Finally, change the ownership of all files to the mongod user.

Assuming you have stored the data on an USB stick mounted as /run/myuser/usbstick, like so:

$ cp -R /run/myuser/usbstick/mongo/data /var/db/mongo/data
$ cp /run/myuser/usbstick/mongo/conf /var/db/mongo/conf
$ chown -R mongod:mongod /var/db/mongo

Note: For executing the next steps, you can switch back from root to your normal user.

Step 3: Run MongoDB as a Docker container

First, we should get the appropriate offical image from Docker Hub. To avoid version incompatibilities I recommend start using the same version of MongoDB you had in your local installation before. In my case it was 5.0, so I choosed the image tagged with 5.0.13.

$ docker pull mongo:5.0.13

After downloading, the last thing to do before starting it up is to figure out the id of the mongod user which is the owner of all data in the filesystem.

$ id -u mongod
1005

Now we can start up the MongoDB container with docker run like so:

$ docker run \
-p 27017:27017 \
--user 1005 \ 
-v /var/db/mongo/data:/data/db \
-v /var/db/mongo/conf:/etc/mongo \
mongo:5.0.13 \
--config /etc/mongo/mongod.conf 
...
{
  "t":{"$date":"2022-11-23T20:02:25.878+00:00"},
  "s":"I",
  "c":"NETWORK",
  "id":23016,   
  "ctx":"listener",
  "msg":"Waiting for connections",
  "attr":{"port":27017,"ssl":"off"}
}

Awesome! MongoDB is now running and waiting for connections on port 27017. All databases, collections, users & roles are there like they were before in the local installation.

Note that this runs MongoDB directly in the current shell. To stop it, simply press CTRL-C. If you rather want it to run in the background, add the -d option to the docker run command.

For a better understanding, let’s break down the command issued above.

Troubleshooting

I cannot connect to the MongoDB container also it started up without any error

If everything seems to be running fine without any errors but you are still not able to connect to MongoDB via localhost:27017 it is most likely that your configuration prohibits it.

In that case, please check the configuration in mongod.conf (or the file you use) under /var/db/mongo/conf and there the IP’s allowed to bind to MongoDB. Allowing 0.0.0.0 should solve it and be sufficient in the Docker scenario.

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0

The container won’t start because of denied permissions when acessing the local filesystem

If you try to start up the MongoDB container like described above and get an error saying that permissions on the filesystem are denied like so…

dpkg: warning: failed to open configuration file '/data/db/.dpkg.cfg' for reading: Permission denied
{
  "t":{"$date":"2022-11-23T21:09:43.425Z"},
  "s":"F",  
  "c":"CONTROL",  
  "id":6384300, 
  "ctx":"-",
  "msg":"Writing fatal message",
  "attr":{"message":"terminate() called. An exception is active; attempting to gather more information\n"}
}
{
  "t":{"$date":"2022-11-23T21:09:43.425Z"},
  "s":"F",  
  "c":"CONTROL",  
  "id":6384300, 
  "ctx":"-",
  "msg":"Writing fatal message",
  "attr":{"message":"std::exception::what(): boost::filesystem::status: Permission denied: \"/etc/mongo/mongod.conf\"\nActual exception type: boost::filesystem::filesystem_error\n\n"}
}

Then you should first ensure that the ownership of the local filesystem is correctly set to the mongod user. For that, execute as root:

$ chown -R mongod:mongod /var/db/mongo

If that doesn’t solve the problem, it is very likely that you have SELinux enabled with an enforcing policy which prohibits docker from accessing the files. You can check the status with sestatus.

$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

In simple words, the Current mode: enforcing is telling you that SELinux is running in a “strict-mode” where you explicitly have to allow Docker to access the local filesystem.

There are three options to solve this:

1. Set SELinux to permissive mode by executing setenforce 0 as root. Note that this change is temporary and has to be re-executed after a reboot.

$ setenforce 0
$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

2. Modify the docker run command an add the :Z option (please note it is a capital Z) to the volume mappings, like so.

$ docker run \
-p 27017:27017 \
--user 1005 \ 
-v /var/db/mongo/data:/data/db:Z \
-v /var/db/mongo/conf:/etc/mongo:Z \
mongo:5.0.13 \
--config /etc/mongo/mongod.conf 

3. Relabel the needed directories and files to set the right SELinux context for Docker as decsribed in this very good article.

Option 3 “relabeling” should be the best solution, but to be honest since it is a very complex process I personally decided to go for the :Z addition in the volume mappings and didn’t investigate the relabeling further. In the Docker docs it is stated to better not use this option with /home and /usr directories, which in fact we are not doing here.

Have a great time with your dockerized MongoDB

Useful links

Many of you may use the quite famous Request library in numerous projects as I also did. Unfortunately there’s a problem…

There’s also an issue on GitHub describing in detail why this decision was made. At the end, the only conclusion is: time to change and migrate to another http-client library.

Possible alternatives and decision for Got

Luckily there are numerous good alternatives available to migrate to, e.g. node-fetch, superagent, axios etc. A quite comprehensive comparsion that may help to choose one of them is provided in the documentation of Got.

Here are the focal points why my decision was made to migrate to Got:

Basic migration steps

Luckily migrating from request to Got isn’t that difficult – except for a few little things. The original developers of Got provide a very helpful and quite complete migration guide.

Following this tutorial most of the work should be done and you should have Got up & running.

However neither the migration guide nor the quite comprehensive documentation contain examples for two – which I think – very, very common tasks when using a http-client library. Let’s have a look on that.

Authorization Header with Bearer token

When creating an web API today you always have to deal with security. One common pattern to control and restrict access to your API is the use of a bearer token, e.g. by using jsonwebtoken. Normally the token is sent in the HTTP authorization header of a request, like so:

Authorization: Bearer YOUR_TOKEN_HERE

Now when looking into Got’s documentation it becomes absolutely clear that should be no problem but at the end there isn’t any helpful example on how to achieve this.

Setting the Authorization header to the desired bearer token value could be achieved by using Got’s hooks feature to modify HTTP headers before a request is made. Here is an example on how to do that.

const got = require('got');

// get url and token
const url = 'YOUR_URL_HERE';
const token = 'YOUR_TOKEN_HERE';

got.get(url), {
  responseType: 'json',
  hooks: {
    beforeRequest: [
      options => {
        options.headers['Authorization'] = 'Bearer ' + token;
      }
    ]
  }
}).then(response => {

}).catch(error => {
        
});

Easy, right? The only annoying thing here is that you don’t want to write the entire hooks block for every request you make. To mitigate this I simply extracted a simple helper module.

module.exports.getHooks = function (token) {
  return {
    beforeRequest: [
      opts => {
        opts.headers['Authorization'] = 'Bearer ' + token;
      }
    ]
  };
};

Having that in place you can simplify the code for every request, like so:

const got = require('got');
const hooks = require('../utils/got-hooks');

// get url and token
const url = 'YOUR_URL_HERE';
const token = 'YOUR_TOKEN_HERE';

got.get(url, {
  responseType: 'json',
  hooks: hooks.getHooks(token)
}).then(response => {
  // ...handle result...
}).catch(error => {
  // ...handle error...
});

That looks good. So let’s go on with the second little issue…

Downloading a file using streams and the original file name

Quite often you may want to download a file with your http-client library. To do so, using streams is a very efficient and straight forward way. Got fully supports streams and there are quite some examples on how to down-stream a file using this library. But there’s one little issue: all examples require you to know the name of the downloaded file before making the request.. uhhh!

Normally it’s working a bit different. If you request a download from an API, the response would contain a HTTP header which gives you the filename, like so:

content-disposition:'attachment; filename="IMG_2358.JPG"'

Obviously this header field is not known before firing the request. So how can we kind of “intercept” this HTTP header to know where to stream the request body to?

The answer is to handle the response event of the download stream. There, all the response headers are available but the “streaming” of the file hasn’t started yet. Here’s how to do that…

const got = require('got');
const stream = require('stream');
const fs = require('fs');
const { promisify } = require('util');
const pipeline = promisify(stream.pipeline);

// instantiate the download stream - use options to set authorization header etc. if needed 
let downStream = got.stream('https://example.com/download');

downStream.on('response', response => {
  // response header will typically include the filename in content-disposition, like
  // content-disposition:'attachment; filename="IMG_2358.JPG"'
  let cd = response.headers['content-disposition'];
  let filename = decodeURIComponent(cd.substring(cd.indexOf('=') + 2, cd.length - 1));
  let outStream = fs.createWriteStream(filename);
  pipeline(downStream, outStream)
      .then(() => { /* ...handle success... */ })
      .catch((err) => { /* ...handle error... */ });
});
downStream.on('error', err => {
  // ...handle error...
});

Please not that also the error event should be catched, e.g. to handle refused connections because the target host isn’t reachable.

With this setup you can download any file using streams and save it under its original filename provided by the response header.

Of course you can combine this with the Bearer token utility we developed before, if the download needs authorization.

let downStream = got.stream('https://example.com/download', { hooks: hooks.getHooks(token) });

The retry-trap

After migrating a middleware component which now uses Got to call other microservices behind the scenes, I realized that in some cases the performance of frontend AJAX requests to that middleware have lost dramatically in performance. Response tooks seconds instead of milliseconds as it was before when the request library was used… uhh, why that?

Investigating a bit further it became clear that the poor performance is directly connected to some HTTP response codes returned from the called microservices, especially faulty ones like 500. Digging into the middleware’s log I saw that the slow-down was caused because in that scenarios three instead of one request where made by Got (with some time in between). At the end this results in quite a “waiting period” before the middleware could respond to the UI.

Digging deeper into the very comprehensive, but out of my perspective not so clearly structured, documentation of Got, the problem was found…

And below on options.retry…

Ahh ok – this means that Got has some defined HTTP response codes and error codes on which it will automatically retry requests for two times unless you disable that feature.

I think automatically retrying requests is a nice feature but if it is the best way to turn it on by default… don’t think so, but anyway. At least the solution to our problem is now clear.

got.get(url, {
  retry: 0
}).then(response => {
  // ...handle result...
}).catch(error => {
  // ...handle error...
});

And here we go… all responses will come back without any retry-delay, also the 500s.

Hope this helps you a bit on migrating to Got.

Useful links