Linux – tsmx https://tsmx.net pragmatic IT Fri, 31 May 2024 20:43:56 +0000 en-US hourly 1 https://wordpress.org/?v=6.6.1 https://tsmx.net/wp-content/uploads/2020/09/cropped-tsmx-klein_transparent-2-32x32.png Linux – tsmx https://tsmx.net 32 32 Running PostgreSQL with Docker on Linux using local persistent data storage https://tsmx.net/postgresql-docker-local-persistent-storage/ Tue, 28 May 2024 20:05:09 +0000 https://tsmx.net/?p=2745 Read more]]> A quick guide demonstrating how to get PostgreSQL up & running in minutes under Linux using Docker. We’ll create local persistent data storage using a Docker volume and connect from localhost with psql.

Preparing local data storage

By default, running PostgreSQL with Docker would store all the data within the container, meaning it would not survive a rebuild of the container. To overcome that, let’s create a local directory that will be mounted as a Docker volume where PostgreSQL can save all of the data to be persistent regardless of container rebuilds etc.

As a best practice, let’s first create a new user called postgres to manage the permissions for all local directories mounted as a volume by the PostgreSQL Docker container. To do so, run the following commands with root privileges.

$ useradd -M postgres
$ usermod -L postgres

This creates a new user postgres having no home directory and a locked password preventing unwanted use. Next, we create a directory (I prefer something under /var/db) for the PostgreSQL data volume and assign it to the newly created user.

$ cd /var/db
$ mkdir postgres
$ chown -R postgres:postgres /var/db/postgres

That’s it for the local preparation. Now let’s move on to start PostgreSQL with Docker. All following steps can be done without having root privileges.

Running PostgreSQL with Docker

To run PostgreSQL with docker, we’ll first download the offical Postgres docker image.

$ docker pull postgres:16.2

Before starting the database, we need to figure out the user ID of the postgres user for passing it to Docker. This will ensure the running container has sufficient rights to access the local data directory mounted as a volume.

$ id -u postgres                                
1002

Now we are ready to run Postgres with Docker using the following command:

$ docker run -d \
  --name mypostgres \
  --user 1002 \
  -e POSTGRES_PASSWORD=Test123$ \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v /var/db/postgres:/var/lib/postgresql/data:Z \
  -p 5432:5432 \
  postgres:16.2

Let’s break down the options of the docker run command in detail.

docker run optionDescription
-dRun the container in background mode.
--name mypostgres[optional] Name the container mypostgres.
--user 1002Set the user ID the container is running with to ensure appropriate rights for accessing the mounted volumes.
-e POSTGRES_PASSWORD=Test123$Set the environment variable for the Postgres admin password.
-e PGDATA=/var/lib/postgresql/data/pgdataSet the environment variable for the Postgres data directory.
-v /var/db/postgres:/var/lib/postgresql/data:ZMount local directory /var/db/postgres as a volume to container directory /var/lib/postgresql/data for persistent data storage on your localhost.
-p 5432:5432Map the Postgres container port 5432 to local port 5432.
postgres:16.2The container image to run.

Note here that the PGDATA variable must be set to something different then /var/lib/postgresql to support mounting to a local persistent directory. For details refere to the corresponding Postgres docker image documentation.

Also you may need the “:Z” option for the volume mounts depending on your SELinux configuration. For an in-depth explanation refer to the Docker MongoDB guide.

Now let’s check if the container is up & running.

$ docker ps | grep postgres
b020097a3a6f   postgres:16.2         "docker-entrypoint.sā€¦"   2 days ago      Up 9 seconds    0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   mypostgres

Looks good – we’re now ready to connect and use the PostgreSQL Docker instance.

Connecting from localhost with psql

Having the container running, let’s connect with the standard Postgres CLI tool psql. If you haven’t installed it already, you should opt-in for a local installation. This is done by installing the postgresl package using dnf with admin rights. If you are on another Linux distribution than Fedora, use your respective package manager.

$ dnf install postgresql

After psql is installed, run the following command as a normal user to connect to the Postgres container from localhost.

$ psql -h localhost -p 5432 -U postgres
Password for user postgres:  # enter the password specified in the docker run command

postgres=#

That’s already it – you’re connected to PostgreSQL running with Docker šŸ™‚

If you want to use other database systems with Docker under Linux, also check out the guides for MongoDB and Oracle.

Useful links

]]>
Moving MongoDB from local installation to Docker – Fedora migration guide https://tsmx.net/local-mongodb-to-docker-fedora-migration-guide/ Wed, 23 Nov 2022 22:12:56 +0000 https://tsmx.net/?p=1830 Read more]]> Complete guide for moving an existing local MongoDB installation to newer versions of Fedora using Docker. Also useful if you are looking for getting a MongoDB container up & running on Linux with locally hosted data.

Problems with a traditional MongoDB installation on Fedora

Trying to install a MongoDB community server locally on a newer version of Fedora (in my case MongoDB 5 on Fedora 36) could be a mess. In the good old days using a repo with DNF was the best option…

$ cat /etc/yum.repos.d/mongodb-org-5.0.repo
[mongodb-org-5.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/5.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-5.0.asc
$ dnf install mongodb-org
Last metadata expiration check: 0:10:51 ago on Tue Nov 22 21:14:41 2022.
Error: 
 Problem: conflicting requests
  ...
  - nothing provides /usr/libexec/platform-python needed by mongodb-org-database-tools-extra-5.0.0-1.el8.x86_64
  (try to add '--skip-broken' to skip uninstallable packages)

Since needed dependencies have been dropped in newer Fedora releases (platform-python at least), this doesn’t work any more out-of-the-box. Falling back to a manual rpm install, this also is not working straight forward since MongoDB is relying on older crypto libraries not shipped with current Fedora releases…

$ rpm -ihv mongodb-org-server-5.0.14-1.el8.x86_64.rpm 
warning: mongodb-org-server-5.0.14-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID e2c63c11: NOKEY
error: Failed dependencies:
        libcrypto.so.1.1()(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libssl.so.1.1()(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libssl.so.1.1(OPENSSL_1_1_0)(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64
        libssl.so.1.1(OPENSSL_1_1_1)(64bit) is needed by mongodb-org-server-5.0.14-1.el8.x86_64 

You could now investigate hours on how to solve this problems manually, but at the end of the day keep in mind that MongoDB also clearly states that they don’t provide package support for Fedora. So this is a good starting point to think about a more stable & standard alternative, which could be running MongoDB as a Docker container…

Migration goals

Assuming you already have a locally running MongoDB in an older version of Fedora (or any other Linux distribution), this guide focusses on how to:

  • Running MongoDB community server as a Docker container on Fedora or another Linux system
  • Migrating all data and permissions (users/groups) to the Docker version without the need of dumping or other heavy lifting
  • Showing how MongoDB’s data, meta data and configuration can be held outside the Docker container to gain maximum flexibility

Moving from local MongoDB installation to dockerized version

Step 1: Backup existing MongoDB data and configuration

First we’ll need to locate and save all of the current data.

  • Database data: Locate the path where MongoDB is storing all of its data. Normally this is /var/lib/mongo on a Fedora system or another dbPath provided at startup via the --dbPath option or in the storage section of the configuration file which usually is /etc/mongod.conf.
    After you have located the path, backup it entirely including subfolders (e.g. copy to a USB Stick, your NAS or whatever).
  • Configuration data: If your current MongoDB is using a configuration file – normally located at /etc/mongod.conf – or another file provided via the --config parameter then also save this file.

Step 2: Prepare the new environment

To run MongoDB using docker with data and configuration hosted outside the container for flexibility reasons we should first create an appropriate place for that. To do so, switch to root and create a new user mongod and two directories data and conf under /var/db to separate all from existing users and home directories.

$ useradd -M mongod
$ usermod -L mongod
$ cd /var/db
$ mkdir mongo
$ mkdir mongo/data
$ mkdir mongo/conf

The created user mongod has no home directory and is also not permitted to log-in using an password as both is not needed in our scenario. Having this in place, copy the saved MongoDB data from step 1 to the new directories. Finally, change the ownership of all files to the mongod user.

Assuming you have stored the data on an USB stick mounted as /run/myuser/usbstick, like so:

$ cp -R /run/myuser/usbstick/mongo/data /var/db/mongo/data
$ cp /run/myuser/usbstick/mongo/conf /var/db/mongo/conf
$ chown -R mongod:mongod /var/db/mongo

Note: For executing the next steps, you can switch back from root to your normal user.

Step 3: Run MongoDB as a Docker container

First, we should get the appropriate offical image from Docker Hub. To avoid version incompatibilities I recommend start using the same version of MongoDB you had in your local installation before. In my case it was 5.0, so I choosed the image tagged with 5.0.13.

$ docker pull mongo:5.0.13

After downloading, the last thing to do before starting it up is to figure out the id of the mongod user which is the owner of all data in the filesystem.

$ id -u mongod
1005

Now we can start up the MongoDB container with docker run like so:

$ docker run \
-p 27017:27017 \
--user 1005 \ 
-v /var/db/mongo/data:/data/db \
-v /var/db/mongo/conf:/etc/mongo \
mongo:5.0.13 \
--config /etc/mongo/mongod.conf 
...
{
  "t":{"$date":"2022-11-23T20:02:25.878+00:00"},
  "s":"I",
  "c":"NETWORK",
  "id":23016,   
  "ctx":"listener",
  "msg":"Waiting for connections",
  "attr":{"port":27017,"ssl":"off"}
}

Awesome! MongoDB is now running and waiting for connections on port 27017. All databases, collections, users & roles are there like they were before in the local installation.

Note that this runs MongoDB directly in the current shell. To stop it, simply press CTRL-C. If you rather want it to run in the background, add the -d option to the docker run command.

For a better understanding, let’s break down the command issued above.

Command expressionExplanation
docker runTell Docker to start a new container. See the docker run docs for all details.
-p 27017:27017Exposes MongoDB’s standard port 27017 from the container to your local system. Enables you to connect via localhost:27017 to the MongoDB container like it was with a local installation.
--user 1005Tell Docker to run the container as user with id 1005 (= mongod). This is essential since the MongoDB image will depend on that the owner of the local data/config files are identical. Otherwise it’ll try to chown them which would fail.
-v /var/db/mongo/data:/data/dbMaps the local directory /var/db/mongo/data to /data/db in the container. This is the MongoDB image default place for database data inside the container. Can be changed with the dbPath parameter in the storage section of the config file.
-v /var/db/mongo/conf:/etc/mongoMaps the local directory /var/db/mongo/conf to /etc/mongo in the container. This is the MongoDB image default place for the configuration file inside the container. Can be changed with the --config parameter – see below.
mongo:5.0.13Tell Docker to use the image mongo with tag 5.0.13 when running the container.
--config /etc/mongo/mongod.confTell MongoDB to use the specified configuration file within the container.

Troubleshooting

I cannot connect to the MongoDB container also it started up without any error

If everything seems to be running fine without any errors but you are still not able to connect to MongoDB via localhost:27017 it is most likely that your configuration prohibits it.

In that case, please check the configuration in mongod.conf (or the file you use) under /var/db/mongo/conf and there the IP’s allowed to bind to MongoDB. Allowing 0.0.0.0 should solve it and be sufficient in the Docker scenario.

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0

The container won’t start because of denied permissions when acessing the local filesystem

If you try to start up the MongoDB container like described above and get an error saying that permissions on the filesystem are denied like so…

dpkg: warning: failed to open configuration file '/data/db/.dpkg.cfg' for reading: Permission denied
{
  "t":{"$date":"2022-11-23T21:09:43.425Z"},
  "s":"F",  
  "c":"CONTROL",  
  "id":6384300, 
  "ctx":"-",
  "msg":"Writing fatal message",
  "attr":{"message":"terminate() called. An exception is active; attempting to gather more information\n"}
}
{
  "t":{"$date":"2022-11-23T21:09:43.425Z"},
  "s":"F",  
  "c":"CONTROL",  
  "id":6384300, 
  "ctx":"-",
  "msg":"Writing fatal message",
  "attr":{"message":"std::exception::what(): boost::filesystem::status: Permission denied: \"/etc/mongo/mongod.conf\"\nActual exception type: boost::filesystem::filesystem_error\n\n"}
}

Then you should first ensure that the ownership of the local filesystem is correctly set to the mongod user. For that, execute as root:

$ chown -R mongod:mongod /var/db/mongo

If that doesn’t solve the problem, it is very likely that you have SELinux enabled with an enforcing policy which prohibits docker from accessing the files. You can check the status with sestatus.

$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

In simple words, the Current mode: enforcing is telling you that SELinux is running in a “strict-mode” where you explicitly have to allow Docker to access the local filesystem.

There are three options to solve this:

1. Set SELinux to permissive mode by executing setenforce 0 as root. Note that this change is temporary and has to be re-executed after a reboot.

$ setenforce 0
$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

2. Modify the docker run command an add the :Z option (please note it is a capital Z) to the volume mappings, like so.

$ docker run \
-p 27017:27017 \
--user 1005 \ 
-v /var/db/mongo/data:/data/db:Z \
-v /var/db/mongo/conf:/etc/mongo:Z \
mongo:5.0.13 \
--config /etc/mongo/mongod.conf 

3. Relabel the needed directories and files to set the right SELinux context for Docker as decsribed in this very good article.

Option 3 “relabeling” should be the best solution, but to be honest since it is a very complex process I personally decided to go for the :Z addition in the volume mappings and didn’t investigate the relabeling further. In the Docker docs it is stated to better not use this option with /home and /usr directories, which in fact we are not doing here.

Have a great time with your dockerized MongoDB šŸ™‚

Useful links

]]>