PostgreSQL – tsmx https://tsmx.net pragmatic IT Wed, 19 Jun 2024 05:30:40 +0000 en-US hourly 1 https://wordpress.org/?v=6.6.1 https://tsmx.net/wp-content/uploads/2020/09/cropped-tsmx-klein_transparent-2-32x32.png PostgreSQL – tsmx https://tsmx.net 32 32 Using SQL Developer with PostgreSQL https://tsmx.net/using-sqldeveloper-with-postgresql/ Tue, 18 Jun 2024 20:36:51 +0000 https://tsmx.net/?p=2928 Read more]]> Quick guide on how to connect to a PostgreSQL database using Oracle SQL Developer.

Also it’s primary usage is for Oracle DB, SQL Developer also a good tool for managing other databases like PostgreSQL and you can benefit from known UX/UI. I assume you already have a local installation of SQL Developer and a PostgreSQL up & running at localhost. If you don’t have a local PostgreSQL DB, have look on this article how to achieve this in minutes using Docker.

Preparing SQL Developer to connect to PostgreSQL

To enable SQL Developer to connect to a Postgres DB, get the offical PostgreSQL JDBC driver first. Save the downloaded jar file in an appropriate folder like /opt/postgres and set the permissions so that SQLDeveloper is able to read the file.

In SQL Developer navigate to Tools --> Preferences and there to Database --> Third Party JDBC Drivers. Click on Add Entry and search for the Postgres JDBC jar.

sqldeveloper-postgres-driver

After restarting SQL Developer you have PostgreSQL available in the database type dropdown for a new connection.

sqldeveloper-postgres-new-connection

Create a connection to the PostgreSQL database

Having that, create a new PostgreSQL connection to localhost port 5432 with user postgres and the default password postgres. If you have started PostgreSQL using Docker, provide the password set in the POSTGRES_PASSWORD variable of the Docker run command.

sqldeveloper-postgres-new-connection-2

After saving and connect you are ready to use your PostgreSQL DB in SQL Developer.

sqldeveloper-postgres-connected

Connecting when username does not equal the database name

By default, the connection in SQL Developer is made to a PostgreSQL database named exactly like the user.

Supposing you have a user testuser (without having an own database named the same) and want to connect to a database called testdb. After entering the credentials for that user neither the Choose Database dropdown was populated nor did the button itself worked for me.

The trick – found in this article on StackOverflow – when connecting to a database with a different name is to add the databse name after a slash to the hostname and finally add a question mark at the end, like so: localhost/testdb?.

sqldeveloper-postgres-different-dbname

That’s it. Have fun using SQL Developer with your PostgreSQL DB šŸ™‚

Useful links

]]>
COPY data between localhost and PostgreSQL running with Docker https://tsmx.net/copy-data-postgresql-docker/ Tue, 04 Jun 2024 20:17:16 +0000 https://tsmx.net/?p=2936 Read more]]> A short article showing how to easily use the COPY command to move data between localhost and a PostgreSQL database running with Docker.

With the COPY command PostgreSQL povides an easy way to move data between the database and the local file system. In this article we show how to use this command when PostgreSQL is running with Docker. This is especially useful if you want to move large amounts of data, e.g. populating a database with mass-data.

Mounting a data transfer volume to the PostgreSQL container

In this article I assume you are familiar on how to run PostgreSQL with Docker using a separate user to manage permissions for local mounted volumes. In our example the user is named postgres and has ID 1002.

To be able to transfer data between PostgreSQL in Docker and the host file system, we’ll need a local directory mounted as a volume to the database container. In this article we’ll use /tmp/postgres for that. Any other directory would be good too as long as the PostgreSQL Docker container can access it.

First, we’ll create the local directory with appropriate permissions. This may require root privileges on your system.

$ cd /tmp
$ mkdir postgres
$ chown postgres:postgres postgres/

Having that, let’s mount the new directory as a volume to the PostgreSQL container. For that we’ll add the option -v /tmp/postgres:/tmp/postgres:Z to the Docker run command – for details refer to the guide on running PostgreSQL with Docker. This maps /tmp/postgres from the Docker container to our locally created /tmp/postgres directory.

Note we make use of the “:Z” option for the mounted volume here to overcome potential issues with SELinux. You might not need that depending on you SELinux configuration. For more details also refer to the corresponding section in our MongoDB docker guide.

The final command to launch PostgreSQL with local storage (recommended) and the mounted data transfer directory for COPY’ing data is:

$ docker run -d \
  --name mypostgres \
  --user 1002 \
  -e POSTGRES_PASSWORD=Test123$ \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v /var/db/postgres:/var/lib/postgresql/data:Z \
  -v /tmp/postgres:/tmp/postgres:Z \
  -p 5432:5432 \
  postgres:16.2

That’s it – you now have a PostgreSQL database running in Docker with directory /tmp/postgres mounted to localhost, ready for transferring data.

Extract data to localhost with COPY TO

First, we’ll use COPY to transfer data from a PostgreSQL table to localhost. For that we connect to the database running with Docker and create a simple table tasks having two columns. Then we insert some rows and use COPY to extract the data to a local CSV file.

$ psql -h localhost -p 5432 -U postgres          
Password for user postgres: 

postgres=# CREATE TABLE tasks (id bigint PRIMARY KEY, description varchar(100));
CREATE TABLE
postgres=# INSERT INTO tasks VALUES (1, 'My first Task');
INSERT 0 1
postgres=# INSERT INTO tasks VALUES (2, 'Task Nr. 2');
INSERT 0 1
postgres=# COPY tasks TO '/tmp/postgres/export.csv' DELIMITER ',' CSV HEADER;
COPY 2
postgres=#

Back on localhost after logging out from PostgreSQL we can verify that the data has been written to export.csv in the directory /tmp/postgres of the local file system.

postgres=# quit
$ cat /tmp/postgres/export.csv 
id,description
1,My first Task
2,Task Nr. 2

Brilliant, that’s working as expected.

Insert data from localhost with COPY FROM

Next, we’ll go the other way round and insert data from a local CSV file into a table. For that we place a file import.csv in the transfer directory on localhost.

$ cat /tmp/postgres/import.csv 
id,description
100,Task-100
200,Task-200
300,Task-300

Having that, we connect back again to our PostgreSQL database, truncate the tasks table and populate it with the data from the import file using the COPY command.

$ postgres psql -h localhost -p 5432 -U postgres
Password for user postgres: 

postgres=# TRUNCATE TABLE tasks;
TRUNCATE TABLE
postgres=# COPY tasks FROM '/tmp/postgres/import.csv' DELIMITER ',' CSV HEADER;
COPY 3
postgres=# SELECT * FROM tasks;
 id  | description 
-----+-------------
 100 | Task-100
 200 | Task-200
 300 | Task-300
(3 rows)

postgres=# 

That’s already it. We’ve successfully moved data between the local file system and our dockerized PostgreSQL back and forth using the COPY command.

Useful links

]]>
Running PostgreSQL with Docker on Linux using local persistent data storage https://tsmx.net/postgresql-docker-local-persistent-storage/ Tue, 28 May 2024 20:05:09 +0000 https://tsmx.net/?p=2745 Read more]]> A quick guide demonstrating how to get PostgreSQL up & running in minutes under Linux using Docker. We’ll create local persistent data storage using a Docker volume and connect from localhost with psql.

Preparing local data storage

By default, running PostgreSQL with Docker would store all the data within the container, meaning it would not survive a rebuild of the container. To overcome that, let’s create a local directory that will be mounted as a Docker volume where PostgreSQL can save all of the data to be persistent regardless of container rebuilds etc.

As a best practice, let’s first create a new user called postgres to manage the permissions for all local directories mounted as a volume by the PostgreSQL Docker container. To do so, run the following commands with root privileges.

$ useradd -M postgres
$ usermod -L postgres

This creates a new user postgres having no home directory and a locked password preventing unwanted use. Next, we create a directory (I prefer something under /var/db) for the PostgreSQL data volume and assign it to the newly created user.

$ cd /var/db
$ mkdir postgres
$ chown -R postgres:postgres /var/db/postgres

That’s it for the local preparation. Now let’s move on to start PostgreSQL with Docker. All following steps can be done without having root privileges.

Running PostgreSQL with Docker

To run PostgreSQL with docker, we’ll first download the offical Postgres docker image.

$ docker pull postgres:16.2

Before starting the database, we need to figure out the user ID of the postgres user for passing it to Docker. This will ensure the running container has sufficient rights to access the local data directory mounted as a volume.

$ id -u postgres                                
1002

Now we are ready to run Postgres with Docker using the following command:

$ docker run -d \
  --name mypostgres \
  --user 1002 \
  -e POSTGRES_PASSWORD=Test123$ \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v /var/db/postgres:/var/lib/postgresql/data:Z \
  -p 5432:5432 \
  postgres:16.2

Let’s break down the options of the docker run command in detail.

docker run optionDescription
-dRun the container in background mode.
--name mypostgres[optional] Name the container mypostgres.
--user 1002Set the user ID the container is running with to ensure appropriate rights for accessing the mounted volumes.
-e POSTGRES_PASSWORD=Test123$Set the environment variable for the Postgres admin password.
-e PGDATA=/var/lib/postgresql/data/pgdataSet the environment variable for the Postgres data directory.
-v /var/db/postgres:/var/lib/postgresql/data:ZMount local directory /var/db/postgres as a volume to container directory /var/lib/postgresql/data for persistent data storage on your localhost.
-p 5432:5432Map the Postgres container port 5432 to local port 5432.
postgres:16.2The container image to run.

Note here that the PGDATA variable must be set to something different then /var/lib/postgresql to support mounting to a local persistent directory. For details refere to the corresponding Postgres docker image documentation.

Also you may need the “:Z” option for the volume mounts depending on your SELinux configuration. For an in-depth explanation refer to the Docker MongoDB guide.

Now let’s check if the container is up & running.

$ docker ps | grep postgres
b020097a3a6f   postgres:16.2         "docker-entrypoint.sā€¦"   2 days ago      Up 9 seconds    0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   mypostgres

Looks good – we’re now ready to connect and use the PostgreSQL Docker instance.

Connecting from localhost with psql

Having the container running, let’s connect with the standard Postgres CLI tool psql. If you haven’t installed it already, you should opt-in for a local installation. This is done by installing the postgresl package using dnf with admin rights. If you are on another Linux distribution than Fedora, use your respective package manager.

$ dnf install postgresql

After psql is installed, run the following command as a normal user to connect to the Postgres container from localhost.

$ psql -h localhost -p 5432 -U postgres
Password for user postgres:  # enter the password specified in the docker run command

postgres=#

That’s already it – you’re connected to PostgreSQL running with Docker šŸ™‚

If you want to use other database systems with Docker under Linux, also check out the guides for MongoDB and Oracle.

Useful links

]]>