JSON – tsmx https://tsmx.net pragmatic IT Thu, 05 Sep 2024 19:08:08 +0000 en-US hourly 1 https://wordpress.org/?v=6.6.1 https://tsmx.net/wp-content/uploads/2020/09/cropped-tsmx-klein_transparent-2-32x32.png JSON – tsmx https://tsmx.net 32 32 Migrating eslintrc.json to eslint.config.js in a CommonJS project https://tsmx.net/migrating-eslintrc-to-flat-config-in-commonjs/ Mon, 19 Aug 2024 20:37:10 +0000 https://tsmx.net/?p=3036 Read more]]> A practical end-to-end guide for migrating an existing .eslintrc.json config (ESLint v8 and before) to the new flat file config eslint.config.js (ESLint v9 and above) in a CommonJS Node.js project including linting of unit-tests with Jest.

This article comes along with a public GitHub example repository enabling you to comprehend everything and easily switch between before/after migration state.

Starting point: existing eslintrc configuration

In this migration guide we’ll use a very standard ESLint configuration set which should cover basic linting for the vast majority of your projects.

  • Configure ESLint for use with Node.js and CommonJS using a specified ECMA version
  • Ensure a proper linting of Jest tests
  • Use a predefined set of linting rules as a starting point
  • Ensure right linting of basics like indention, unused variables, use of globals, semicolons and quote signs used

So far, the usual way to configure ESLint in Node.js was to place an .eslintrc.json file in the root folder of the project. The below .eslintrc.json covers all the mentioned points and serves as the basis for this guide.

{
    "env": {
        "node": true,
        "commonjs": true,
        "es6": true,
        "jest": true
    },
    "extends": "eslint:recommended",
    "globals": {
        "Atomics": "readonly",
        "SharedArrayBuffer": "readonly"
    },
    "parserOptions": {
        "ecmaVersion": 2018
    },
    "rules": {
        "indent": [
            "error",
            4,
            {
                "SwitchCase": 1
            }
        ],
        "quotes": [
            "error",
            "single"
        ],
        "semi": [
            "error",
            "always"
        ],
        "no-unused-vars": [
            2,
            {
                "args": "after-used",
                "argsIgnorePattern": "^_"
            }
        ]
    }
}

Additionally, you might have a .eslintignore file placed in the root folder of the project to exclude files and paths from linting, e.g. to exclude the two directories conf and coverage – like so:

conf/
coverage/

Errors after upgrading ESLint to v9

Having this configuration in place you’ll notice that your environment, in this case VSCode, is coming up with an error after upgrading to ESLint v9. The highlighting of linting errors and warnings also isn’t working any more.

eslint-config-error-vscode

Having a look in the ESLint output quickly gives you the reason why.

[Info  - 20:51:44] ESLint server is starting.
[Info  - 20:51:44] ESLint server running in node v20.14.0
[Info  - 20:51:44] ESLint server is running.
[Info  - 20:51:46] ESLint library loaded from: /home/tsmx/projects/weather-tools/node_modules/eslint/lib/api.js
(node:4117) ESLintIgnoreWarning: The ".eslintignore" file is no longer supported. Switch to using the "ignores" property in "eslint.config.js": https://eslint.org/docs/latest/use/configure/migration-guide#ignoring-files
(Use `code --trace-warnings ...` to show where the warning was created)
[Error - 20:51:46] Calculating config file for file:///home/tsmx/projects/weather-tools/weather-tools.js) failed.
Error: Could not find config file.

Starting with ESLint v9.0.0, the default configuration was changed to flat file and eslintrc.json as well as eslintignore became deprecated. Although it’s possible to continue using eslintrc.json, it’s recommended to switch to the new file format being future-proof.

Migrating to the new flat file configuration

For a CommonJS project, the new flat file configuration is a normal JavaScript file called eslint.config.js which is placed in the root folder and simply exports an array of ESLint configuration objects via module.exports.

Installing needed dev dependencies

The flat file config doesn’t contain an env section anymore that allows you to specify ESLint is running in Node.js and enabling Jest features for correct linting of unit-test files. Also, the recommended ruleset has been outsourced to an own module.

To include all these features in the new ESLint v9 configuration, you’ll need to install the following dependencies in your project.

  • @eslint/js for using the recommended ruleset as a basis
  • eslint-plugin-jest to enable proper linting of Jest test files
  • globals to make ESLint aware of common global variables for Node.js and Jest avoiding they are marked as undefined

As these dependencies are only used for ESLint, you should install them – like ESLint itself – as dev dependencies in your Node.js project.

# npm install @eslint/js eslint-plugin-jest globals --save-dev

Creating eslint.config.js

Next, in the root folder of your Node.js project, create an eslint.config.js file with the following contents. This will lead to an almost identical, yet more customizable, linting behaviour as the old .eslintrc.json did.

const { configs } = require('@eslint/js');
const jest = require('eslint-plugin-jest');
const globals = require('globals');

module.exports = [
    configs.recommended,
    {
        languageOptions: {
            ecmaVersion: 2018,
            sourceType: 'commonjs',
            globals: { 
                ...globals.node, 
                ...globals.jest, 
                Atomics: 'readonly', 
                SharedArrayBuffer: 'readonly' 
            }
        },
        rules: {
            semi: 'error',
            quotes: ['error', 'single'],
            indent: ['error', 4, { 'SwitchCase': 1 }],
            'no-unused-vars':
                [
                    'warn',
                    {
                        'varsIgnorePattern': '^_',
                        'args': 'after-used',
                        'argsIgnorePattern': '^_'
                    }
                ]
        },
        ignores: ['conf/', 'coverage/']
    },
    {
        languageOptions: {
            globals: { ...globals.jest }
        },
        files: ['test/*.test.js'],
        ...jest.configs['flat/recommended'],
        rules: {
            ...jest.configs['flat/recommended'].rules
        }
    }
];

Thats’s already it. Linting now should work again as expected and you can safely delete the old .eslintrc.json as well as .eslintignore in your project.

Breakdown of the new flat file configuration

As noted before, the flat file configuration is simply an exported array of ESLint configuration objects. Based on our eslintrc.json we want to migrate, this array will have three entries.

Part 1: Importing recommended ESLint ruleset

First element of the configuration array should be the recommended ruleset that is delivered by the @eslint/js package. This line is the replacement for the "extends": "eslint:recommended" entry in the old eslintrc.

configs.recommended

Part 2: Custom rules for normal JavaScript code files and files to be ignored

Next object in the configuration array holds all our own custom rules and properties for normal JavaScript code files as well as the patterns of all files/folders that shout be ignored bei ESLint.

{
    languageOptions: {
        ecmaVersion: 2018,
        sourceType: 'commonjs',
        globals: { 
            ...globals.node, 
            ...globals.jest, 
            Atomics: 'readonly', 
            SharedArrayBuffer: 'readonly' 
         }
    },
    rules: {
        semi: 'error',
        quotes: ['error', 'single'],
        indent: ['error', 4, { 'SwitchCase': 1 }],
        'no-unused-vars':
            [
                'warn',
                {
                    'varsIgnorePattern': '^_',
                    'args': 'after-used',
                    'argsIgnorePattern': '^_'
                }
            ]
    },
    ignores: ['conf/', 'coverage/']
}

This section is quite self-explanatory when compared to the old eslintrc configuration. The key differences are:

  • There is no env section anymore, most of that configuration is now located under languageOptions.
  • Note that in the globals object all node and jest globals where added explicitly by using the corresponding arrays provided in the globals-package. This ensures that all common Node.js globals like process and Jest globals like expect are not treated as undefined variables. The latter makes sense if you create some kind of test-utils files which use Jest commands but are not unit-test files themselves. See the example GitHub repository for such an example (/tests/test-utils.js).
  • There is now an ignore property that takes an array of files/folders to be ignored by ESLint. The syntax of the entries is the same as it was in .eslintignore which is now obsolete. For more details see ignoring files.

The linting rules themselves are quite unchanged in the new configuration style.

Part 3: Rules for linting Jest test files

The last needed configuration object is for correct linting of Jest tests. There is no "jest": true option anymore which was very simple. Instead, we’ll need to import the eslint-plugin-jest package and use recommended rules out of it. In the example, all Jest test files are located in the projects folder test/ and have the common extension .test.js.

The resulting configuration object for our eslint.config.js is:

{
    languageOptions: {
        globals: { ...globals.jest }
    },
    files: ['test/*.test.js'],
    ...jest.configs['flat/recommended'],
    rules: {
        ...jest.configs['flat/recommended'].rules
    }
}

This ensures a proper linting of all Jest tests located under test/. If you have test files located in other/additional locations, simply add them to the files property.

Note: If you use Node.js globals like process in your Jest tests, you should add ...globals.node to the globals property. This prevents ESLint from reporting those globals as undefined variables.

Example project on GitHub

To see a practical working example of a migration before and after, clone the eslintrc-to-flatfile GitHub repository and browse through the branches eslint-v8 and eslint-v9. The code files contain several example errors in formatting, quoting etc. that should be highlighted as linting errors or warnings. For details see the code comments.

# clone the example project
git clone https://github.com/tsmx/eslintrc-to-flatfile.git

# check out/switch to the original ESLint v8 branch using eslintrc.json
git checkout eslint-v8
npm install

# check out/switch to the migrated ESLint v9 branch using new eslint.config.js
git checkout eslint-v9
npm install

Useful links

]]>
secure-config – easy and secure NodeJS configuration management https://tsmx.net/secure-config/ Thu, 27 Aug 2020 21:16:45 +0000 https://tsmx.net/?p=16 Read more]]> A convenient npm package to handle multi-environment NodeJS configurations with encrypted secrets and HMAC validation. Works with CommonJS and ESM/ECMAScript.

With secure-config we provide a lean and secure configuration management for NodeJS.

  • Lean: following the KISS principle – minimalistic and focused feature set, easy to use
  • Secure: state-of-the-art encryption and data integrity for sensible configuration data

Just follow some basic guidelines like a common naming convention and you are in the game!

Note: This documentation refers to the latest version of secure-config, for docs on older versions please look here.

Benefits

  • Easy integration with just a few lines of code. Works with CommonJS as well as ESM/ECMAScript.
  • Strong AES-256-CBC encryption. Secured configuration is still a valid, legible JSON.
  • HMAC validation of your configuration to ensure data integrity.
  • Configuration management for different environments (dev, test, prod…)
  • No need to use 3rd party secret stores, no vendor or platform lock-in.
  • Works perfectly on-premise, with Docker, cloud platforms like Google AppEngine etc.

Encryption and HMAC validation of your configurations are totally optional. You can also use secure-config just to manage plain, unencrypted JSON configurations for different environments. Also you can combine different security features for the different environments, e.g. encryption-only for TEST and encryption plus HMAC validation for PROD.

Usage

Suppose you have the following JSON configuration file config.json with secret information about your database connection…

{ 
  "database": {
    "host": "127.0.0.1", 
    "user": "MySecretDbUser",
    "pass": "MySecretDbPass" 
  } 
} 

Step 1 – Install secure-config-tool [optional]

[tsmx@localhost ]$ npm i -g @tsmx/secure-config-tool

Note: In the explanation I will use the provided tool secure-config-tool. You can do all that without the tool as it is NodeJS crypto standard, but it’s a lot more easier so…

Step 2 – Get a secret key and export it.

[tsmx@localhost ]$ secure-config-tool genkey
df9ed9002b...
[tsmx@localhost ]$ export CONFIG_ENCRYPTION_KEY=df9ed9002b...

Step 3 – Encrypt your configuration JSON values and generate a new, secure configuration file.

[tsmx@localhost ]$ secure-config-tool create config.json > conf/config.json
[tsmx@localhost ]$ cat conf/config.json
{ 
  "database": {
    "host": "127.0.0.1", 
    "user": "ENCRYPTED|50ceed2f97223100fbdf842ecbd4541f|df9ed9002bfc956eb14b1d2f8d960a11",
    "pass": "ENCRYPTED|8fbf6ded36bcb15bd4734b3dc78f2890|7463b2ea8ed2c8d71272ac2e41761a35" 
  },
  "__hmac": "3023eb8cf76894c0d5c7f893819916d876f98f781f8944b77e87257ef77c1adf"
}

The generated file should be in the conf/ subfolder of your app. For details see naming conventions.

Step 4 – Use your configuration in the code.

// CommonJS
const conf = require('@tsmx/secure-config')();

// ESM
import secureConfig from '@tsmx/secure-config';
const conf = secureConfig();

function MyFunc() {
  let dbHost = conf.database.host; // = '127.0.0.1'
  let dbUser = conf.database.user; // = 'MySecretDbUser'
  let dbPass = conf.database.pass; // = 'MySecretDbPass'
  //...
}

To change the default behaviour or make use of more advanced features like HMAC validation you can pass custom options to the function returned by require/import.

Step 5 – Run your app using the new encrypted configuration.

[tsmx@localhost ]$ export CONFIG_ENCRYPTION_KEY=df9ed9002b...
[tsmx@localhost ]$ node app.js

File name and directory conventions

You can have multiple configuration files for different environments or stages. They are distinguished by the environment variable NODE_ENV. The basic configuration file name is config.json if this environment variable is not present. If it is present, a configuration file with the name config-[NODE_ENV].json is used. An exception will be thrown if no configuration file is found.

To change the default configuration file name or loading multiple configuration files you can pass the prefix option.

By default, all configuration files are expected to be located in a conf/ directory of the current running app, meaning a direct subdirectory of the current working directory (CWD/conf/). To overwrite this behaviour, you can pass the directory option.

Example project structure

Let’s assume you are working on a typical project where you are developing locally, using automated tests and finally will deploy to a production environment. From a configuration perspective, you’ll have these three stages…

Development stage

  • NODE_ENV: not set
  • Configuration file: conf/config.json

Production stage

  • NODE_ENV: production
  • Configuration file: conf/config-production.json

Test stage, e.g. Jest

  • NODE_ENV: test
  • Configuration file: conf/config-test.json

The final project structure would look like that:

path-to-your-app/
├── conf/
│       ├── config.json
│       ├── config-production.json
│       └── config-test.json
├── app.js
└── package.json

Note: it is a very common practice that NODE_ENV is set to test when running automated tests. Some test libraries like Jest will do that automatically.

Custom options

To retrieve a configuration using all default values and without advanced features, you simply invoke a function after the require/import statement without any argument (empty set of parenthesis after require or simple method call after import).

// CommonJS
const conf = require('@tsmx/secure-config')();

// ESM
import secureConfig from '@tsmx/secure-config';
const conf = secureConfig();

To make use of the more advanced features and customize default values, you can pass an options object to this function call.

const confOptions = {
  keyVariable: 'CUSTOM_CONFIG_KEY',
  hmacValidation: true, 
  hmacProperty: '_signature',
  prefix: 'myconf'
}

// CommonJS
const conf = require('@tsmx/secure-config')(confOptions);

// ESM
import secureConfig from '@tsmx/secure-config';
const conf = secureConfig(confOptions);

The following options are available.

keyVariable

Type: String

Default: CONFIG_ENCRYPTION_KEY

The name of the environment variable containing the key for decrypting configuration values and validating the HMAC. See also options on how to pass the key.

hmacValidation

Type: Boolean 

Default: false

Specifies if the loaded configuration should be validated against a given HMAC. If set to true, secure-config will validate the HMAC of the decrypted configuration content against a given HMAC using the current key. If the validation fails, an exception will be thrown. If it succeeds, the decrypted configuration will be returned.

The given HMAC is retrieved from a configuration file property with the name of hmacProperty, e.g.:

{
  "database": {
    "host": "127.0.0.1",
    "user": "ENCRYPTED|50ceed2f97223100fbdf842ecbd4541f|df9ed9002bfc956eb14b1d2f8d960a11",
    "pass": "ENCRYPTED|8fbf6ded36bcb15bd4734b3dc78f2890|7463b2ea8ed2c8d71272ac2e41761a35"
  },
  "__hmac": "3023eb8cf76894c0d5c7f893819916d876f98f781f8944b77e87257ef77c1adf"
}

Enabling this option is recommended for production environments as it adds more security to your configuration management ensuring the loaded configuration is safe against tampering. Unwanted modifications of any – even unencrypted – entries in your configuration would cause the HMAC validation to fail and throw the error HMAC validation failed.

Please ensure that your stored configuration files have an appropriate HMAC property before enabling this option. Otherwise loading the configuration would always fail. secure-config-tool adds the HMAC by default when creating secured configuration files.

To get more information on how the HMAC creation & validation works internally, please refer to the package object-hmac which is used for that. The HMAC value is created out of the entire configuration data before encryption / after decryption. Also take a look on the short description how secure-config is working under the hood.

hmacProperty

Type: String 

Default: __hmac

The name of the HMAC property in a configuration file to be validated against. Only used when hmacValidation is set tor true.

Example configuration file using a custom HMAC property name:

{
  "database": {
    "host": "127.0.0.1",
    "user": "ENCRYPTED|50ceed2f97223100fbdf842ecbd4541f|df9ed9002bfc956eb14b1d2f8d960a11",
    "pass": "ENCRYPTED|8fbf6ded36bcb15bd4734b3dc78f2890|7463b2ea8ed2c8d71272ac2e41761a35"
  },
  "_signature": "3023eb8cf76894c0d5c7f893819916d876f98f781f8944b77e87257ef77c1adf"
}

Loading the configuration with HMAC validation enabled:

const confOptions = {
    hmacValidation: true, 
    hmacProperty: '_signature'
}

const conf = require('@tsmx/secure-config')(confOptions);

directory

Type: String

Default: ./conf/

Use this parameter to change the directory where the configuration files should be loaded from.

E.g. if the files are located under /var/myapp/configurations:

const confOptions = {
    directory: '/var/myapp/configurations'
}
const conf = require('@tsmx/secure-config')(confOptions);

This option can be combined with the prefix option to control the configuration filenames within the directory. Naming conventions according to NODE_ENV are applied as normal.

Hint: Setting a relative path within the current running app or an unit-test can easily be achieved by using path.join with process.cwd. E.g. if the files are located in ./test/configurations.

const confOptions = {
    directory: path.join(process.cwd(), 'test/configurations')
}

prefix

Type: String 

Default: config

Use this parameter to change the default file name pattern from config-[NODE_ENV].json to [prefix]-[NODE_ENV].json for loading files with deviating names or additional ones. The value of NODE_ENV will be evaluated as described in the naming conventions.

To load multiple configurations, use the following pattern in your code.

const secureConf = require('@tsmx/secure-config');
const config = secureConf();
const myconf = secureConf({ prefix: 'myconf', keyVariable: 'MYCONF_KEY' });

This example will load the default config.json using the the key from environment variable CONFIG_ENCRYPTION_KEY as well as the additional myconf.json using the key from MYCONF_KEY. Note that different configurations should use different encryption keys.

Depending on the value of NODE_ENV the following configuration files will be loaded in this example.

Value of NODE_ENVVariableFilename
not setconfigconf/config.json
myconfconf/myconf.json
productionconfigconf/config-production.json
myconfconf/myconf-production.json
testconfigconf/config-test.json
myconfconf/myconf-test.json

Injecting the decryption key

The key for decrypting the encrypted values is derived from an environment variable named CONFIG_ENCRYPTION_KEY. You can use the keyVariable option to change the name if you need to.

You can set the environment variable whatever way is most suitable. Here are some examples for common use-cases.

Set/export directly in the command line.

export CONFIG_ENCRYPTION_KEY=0123456789qwertzuiopasdfghjkly

Set the key in your launch.json configuration for developing/debugging.

... 
    "env": { "CONFIG_ENCRYPTION_KEY": "0123456789qwertzuiopasdfghjklyxc" },
...

Best and most secure way of passing the decryption key to a cloud function is using GCP Secret Manager. Create a new secret for the key (e.g. CONFIG_KEY) there and pass it as an an environment variable using the --set-secrets option.

gcloud functions deploy my-cloud-function \
--source=. \
--entry-point=myFunction \
--trigger-http \
--set-secrets=CONFIG_ENCRYPTION_KEY=projects/100374066341/secrets/CONFIG_KEY:latest

Replace the number in the key reference path with your GCP project number. For more detailed instructions take a look at the article about secure configuration management for a cloud function.

Set the key in an environment block in app.yml.

... 
env_variables:
  CONFIG_ENCRYPTION_KEY: "0123456789qwertzuiopasdfghjklyxc"
...

Note: Make sure to not include the productive app.yml deployment descriptor in your code repository to not expose the production key. In general only your deployment managers should have access to this file as it also contains other sensitive and cost-relevant information like instance sizes and scaling options.

Pass the key to the docker run command.

docker run --env CONFIG_ENCRYPTION_KEY=0123456789qwertzuiopasdfghjklyxc MYIMAGE

There’s a complete docker test example available on GitHub.

For testing with Jest I recommend to create a test key and set it globally for all tests in the jest.config.js.

process.env['CONFIG_ENCRYPTION_KEY'] = '0123456789qwertzuiopasdfghjklyxc';
module.exports = { testEnvironment: 'node' };

For more details also have a look at how to configure Jest.

If your NodeJS app using secure-config should run as a systemd service, set the key in the [Service] section of your service file.

...
[Service]

Environment=CONFIG_ENCRYPTION_KEY=0123456789qwertzuiopasdfghjklyxc
WorkingDirectory=/path/to/your/app
...

Note: You should also set the WorkingDirectory in the service configuration so that secure-config can find the configuration files in the conf/ subfolder correctly.

The key length must be 32 bytes! The value set in CONFIG_ENCRYPTION_KEY has to be:

  • a string of 32 characters length, or
  • a hexadecimal value of 64 characters length (= 32 bytes)

Otherwise an error will be thrown.

Examples of valid key strings:

  • 32 byte string: MySecretConfigurationKey-123$%&/
  • 32 byte hex value: 9af7d400be4705147dc724db25bfd2513aa11d6013d7bf7bdb2bfe050593bd0f

Different keys for each configuration environment are strongly recommended.

Working with secure-config-tool

For an easy handling of secure configuration files it is recommended to use the accompanying secure-config-tool. It is an easy command line tool covering all necessary workflows.

To install the tool, simply run the following command with sufficent rights:

[tsmx@localhost ]$ npm i -g @tsmx/secure-config-tool

Please note that using secure-config-tool is not a must. It is absolutely possible to do everything on your own. For details refer to the implementation available in the GitHub repo. All functions used in the tool are either NodeJs standard or open-source.

Obtaining a secret key

First thing you have to do when securing your configuration files is to get a key for encryption and/or HMAC generation. For that, secure-config-tool offers the genkey option.

[tsmx@localhost ]$ secure-config-tool genkey
a0e9a2d6447adf308cabeb64a075f6f3a45041be956c3f7a94de8a28c0df695e

This function uses the NodeJS standard function randomBytes to generate a cryptographically strong 32 bit long hex key.

Keep the generated key secret. Separate keys for different environments are strongly recommended.

For every operation dealing with encryption/decryption or HMAC’s, the key must be present in an environment variable CONFIG_ENCRYPTION_KEY.

[tsmx@localhost ]$ export CONFIG_ENCRYPTION_KEY=a0e9a2d6447adf308cabeb64a075f6f3a45041be956c3f7a94de8a28c0df695e

As a best practice you should save the generated key into a file and export it directly from there.

[tsmx@localhost ]$ secure-config-tool genkey > prod-key.txt
[tsmx@localhost ]$ export CONFIG_ENCRYPTION_KEY=$(cat prod-key.txt)

Initial creation of a secure configuration file

To easily create a secured configuration out of an existing JSON file, secure-config-tool provides the create option. Using the default settings, this will take your file, encrypt the values of properties matching the terms (user, pass, token) and add a HMAC.

[tsmx@localhost ]$ cat config-plain.json
{
  "database": {
    "host": "localhost",
    "username": "secret-db-user",
    "password": "Test123",
    "port": 1521
  },
  "settings": {
    "https": true,
    "attempts": 5
  }
}
[tsmx@localhost ]$ secure-config-tool create config-plain.json > config-production.json
[tsmx@localhost ]$ cat config-production.json
{
  "database": {
    "host": "localhost",
    "username": "ENCRYPTED|143456e2d2e401bde5d63224a88d44bb|312d01644cc65161f77ce2ed47458a60",
    "password": "ENCRYPTED|2bca8b3b444e9061e57f4e288ef8aecd|99453a3e55cf621f492c474c8a1c0c81",
    "port": 1521
  },
  "settings": {
    "https": true,
    "attempts": 5
  },
  "__hmac": "9d68c2082f2e5fc61268a8171872ae1c15bf15b6a6ab0f82e27e40f19cc156bd"
}

Now you’re ready to use the secured configuration file. For more details on how to customize the file generation, please refer to the docs for create.

Adding new encrypted properties

As you work on your project, you will have to add further sensible properties to your configuration file. Let’s assume you want wo add a property schema with a value of secret-db-schema to the database section in the example above.

For that, use secure-config-tool’s encrypt function for generating the encrypted value and put it into the config file.

[tsmx@localhost ]$ secure-config-tool encrypt "secret-db-schema"
ENCRYPTED|56c1cd4685e1405607b28a84fa083881|bf052def52a7544cfe9b50badd9c907c2b7b399fade2d86b3b9715baa79e6efb
[tsmx@localhost ]$ vi config-production.json # copy and paste the new property value
[tsmx@localhost ]$ cat config-production.json
{
  "database": {
    "host": "localhost",
    "username": "ENCRYPTED|143456e2d2e401bde5d63224a88d44bb|312d01644cc65161f77ce2ed47458a60",
    "password": "ENCRYPTED|2bca8b3b444e9061e57f4e288ef8aecd|99453a3e55cf621f492c474c8a1c0c81",
    "schema": "ENCRYPTED|56c1cd4685e1405607b28a84fa083881|bf052def52a7544cfe9b50badd9c907c2b7b399fade2d86b3b9715baa79e6efb",
    "port": 1521
  },
  "settings": {
    "https": true,
    "attempts": 5
  },
  "__hmac": "9d68c2082f2e5fc61268a8171872ae1c15bf15b6a6ab0f82e27e40f19cc156bd"
}

The new encrypted property is now ready to use.

Please note that after any change to a configuration file, an eventually present HMAC property – like in our example above – will be invalid until it is re-calculated. To do so, just read on…

Updating the HMAC

If you make use of the HMAC validation, you must be aware of that any change of the content of a configuration file will change it’s HMAC value and therefore invalidate any existing HMAC property.

For re-calculation of the HMAC to make it valid again, simply use the update-hmac function.

[tsmx@localhost ]$ secure-config-tool update-hmac -o config-production.json

This command will take your configuration file and re-calculate the HMAC property. Any other value, especially encrypted values, will stay untouched.

Please also refer to the update-hmac docs and the description of how the HMAC validation is working.

Test an existing configuration file

With the test function, secure-config-tool offers an easy way to test if a configuration file is valid with regards to:

  • decrypting the content
  • validating the HMAC

To test a file, simply call…

[tsmx@localhost ]$ secure-config-tool test config-production.json
Decryption: PASSED
HMAC:       PASSED

If you are using a wrong key or encrypted entries are malformed, you”ll get an error.

[tsmx@localhost ]$ secure-config-tool test config-production.json
Decryption failed. Please check that the right key is used and the encrypted secret is valid and has the form "ENCRYPTED|IV|DATA"
See the docs under: https://github.com/tsmx/secure-config
Decryption: FAILED
HMAC:       not tested

If you get this error, please check if you are using the right key and if all encrypted entries are valid according to the specification. If necessary, you can re-create them with the create or encrypt function.

For more details, please refer to the test docs.

To check a single encrypted entry, just copy it and use the decrypt function. It will print out the decrypted value.

[tsmx@localhost ]$ secure-config-tool decrypt "ENCRYPTED|143456e2d2e401bde5d63224a88d44bb|312d01644cc65161f77ce2ed47458a60"
secret-db-user

Encrypted configuration entries

Specification

Encrypted configuration entries must always have the form: ENCRYPTED | IV | DATA.

PartDescription
ENCRYPTEDThe prefix ENCRYPTED used to identify configuration values that must be decrypted.
IVThe ciphers initialization vector (IV) that was used for encryption. Hexadecimal value.
DATAThe AES-256-CBC encrypted value. Hexadecimal value.

Example: "ENCRYPTED|aebc07dd97af3f857cb585b4c956661b|ea18ce1feaa5b8cf4ecb471b9b4401da"

Generation

In general, it is recommended to use secure-config-tool to create and maintain your configuration files. Have a look here for some typical workflows.

Anyways, it’s totally fine to use the standard crypto functions from NodeJS. With the following snippet you can create valid encrypted entries for your configuration files:

const crypto = require('crypto');
const algorithm = 'aes-256-cbc';

function encrypt(value) { 
  let iv = crypto.randomBytes(16); 
  let key = Buffer.from('YOUR_KEY_HERE'); 
  let cipher = crypto.createCipheriv(algorithm, key, iv); 
  let encrypted = cipher.update(value); 
  encrypted = Buffer.concat([encrypted, cipher.final()]); 
  return 'ENCRYPTED|' + iv.toString('hex') + '|' + encrypted.toString('hex');
}

Under the hood

This is how secure-config works: when importing and invoking the package via require('@tsmx/secure-config')() it…

  1. Retrieves the decryption key out of the environment variable CONFIG_ENCRYPTION_KEY
  2. Loads the applicable JSON configuration file.
  3. Recursively iterates the loaded JSON and decrypts all encrypted entries that were found.
  4. Optionally, checks the HMAC of the decrypted configuration object.
  5. Returns a simple JSON object with all secrets decrypted.

Important notes / good to know:

  • Decryption is only done in-memory. Decrypted values are never persisted anywhere.
  • The CONFIG_ENCRYPTION_KEY environment variable is always required, even if the config file doesn’t contain encrypted secrets.
  • Recursion depth is not limited, encrypted entries can be used at every object level in the JSON config file.
  • HMAC validation is optional and can be used without any encryption, e.g. if you don’t have sensitive data but want to ensure data integrity anyways.
  • Recursion depth is not limited, encrypted entries can be used at every object level in the JSON config file.
  • At the time of writing only decryption of simple values is supported (no arrays or complete sub-objects. but properties of sub-objects). This covers most of the use-cases as sensitive data like passwords are normally all plain, nonstructured values.

Sample projects

GitHub

To get familiar with the use of secure-config I provided a secure-config-test project on Github.

Docker Hub

For trying out with Docker and Kubernetes a public docker image is also available on Docker-Hub.

Unit tests

The package contains a comprehensive set of unit tests with a very high coverage of the code. To run them, install or clone the package and run the tests via npm:

npm run test

To output the code coverage run:

npm run test-coverage

Also check out the current coverage stats at Coveralls.

Changelog

Version 2.1.0 (13.12.2023)

  • Support for encrypted properties of objects in arrays added, e.g.
    { configArray: [ { key: 'ENCRYPTED|...' }, { key: 'ENCRYPTED|... ' } ] }

Version 2.2.0 (29.03.2024)

  • Support for loading multiple configurations with new option prefix added.

Version 2.3.0 (05.09.2024)

  • Support for custom configuration file path with new option directory added.

Older versions

This article refers to the current version of secure-config. Documentations for older versions can be found here.

Useful Links

]]>