User-configurable settings in Symfony applications with jbtronics/settings-bundle (Part 3): Migrations and environment variables

In the first two parts of this series, the basic concepts of the settings-bundle were introduced and how to use it to create nice user-configurable settings in Symfony applications.
In this part, you will learn how to version your settings and migrate between them. Additionally, you will learn how to combine environment variables with settings.

Versioning and migration

Over time you application will evolve and so will your settings. This means that over time new parameters will be added to settings, old ones will be removed and existing ones will be changed. To handle this, the settings-bundle provides a versioning and migration mechanism, which takes care of most of the work for you.

Let's assume you have a simple settings class like this:

namespace App\Settings;

use Jbtronics\SettingsBundle\Settings\Settings;
use Jbtronics\SettingsBundle\Settings\SettingsParameter;

#[Settings]
class TestSettings {

    #[SettingsParameter]
    public string $email = 'test@invalid';

    #[SettingsParameter]
    public int $baz = 42;
}

These settings were already used for some time in your application and users already saved their customizations to it. If you just want to add a new parameter to the settings, you can do this by simply adding a new property to the class, and it will work fine. The new parameter will be initialized with the default value and users can change it as they like:

#[Settings]
class TestSettings {

    #[SettingsParameter]
    public string $email = 'test@invalid';

    #[SettingsParameter]
    public int $baz = 42;

    #[SettingsParameter]
    public bool $qux = true;
}

Removing a parameter works similarly. If you remove a property from the class, the settings-bundle will ignore existing values for it, and delete it the next time the settings are saved.

However, what is more tricky, is if you want to rename a field or, even more complex, change its type or how data is exactly saved. To not lose existing customizations of users, you have to specify how to convert between the different representations of the settings. The settings-bundle can support you with this by providing a framework for migrations.

Let's assume you want to change your settings class in a way, that you now can have multiple email addresses. Also, you want to change the indexing of the baz parameter, so that it not start at 0, but at 1, meaning that all existing values should be incremented by 1. In the end your settings class should look like this:

namespace App\Settings;

use Jbtronics\SettingsBundle\Settings\Settings;
use Jbtronics\SettingsBundle\Settings\SettingsParameter;

#[Settings(version: self::VERSION, migrationService: TestSettingsMigration::class)]
class TestSettings {

    public const VERSION = 1;

    #[SettingsParameter(type: ArrayType::class, options: ['type' => StringType::class])]
    public array $email = ['test@invalid'];

    #[SettingsParameter]
    //Now with different indexing
    public int $baz = 43;
}

The test settings class now have the new intended structure and can be used in the application. However, the settings-bundle will not know how to convert the existing data to the new structure. This is where migrations
come into play.

You can see that the settings attribute now have the version option and the migrationService option specified:

The version option specifies the most recent schema version of the settings and is just a integer (greater zero), which is incremented every time you change the structure of the settings class. You can start with 1 and increment it every time you change the structure of the settings class. You can put the version number directly into the attribute, or you can define a constant for it, as shown in the example, which has the advantage that you can retrieve the current version easily from outside the class.

The second new thing is the migrationService option. This specifies the service class, which actually performs the data migration. The migrationService must implement the SettingsMigrationInterface, which specifies a migrate function that is responsible for performing migration between two given versions of the data.

In most cases you want to step-wise migrations between the versions (meaning you migrate 1 -> 2, then 2 -> 3 and so on, instead of 1 -> 3 directly to avoid code duplication). In this situation, it is easier to extend the SettingsMigration class. Using this abstract class, your migration service might look like this:

namespace App\Settings\Migrations;

use Jbtronics\SettingsBundle\Migrations\SettingsMigration;

class TestSettingsMigration extends SettingsMigration  {

    /**
     * This method is called automatically by the migration class and handles 
     * migration of version 0 (non versioned settings) to version 1.
     */
    public function migrateToVersion1(array $data, SettingsMetadata $metadata): array
    {

        /*
         * $data contains the old settings data, in the normalized form (in the way it was saved in the database)
         * Each key is the parameter name (not necessarily the property name) 
         * 
         * In the end we must return the new data in the normalized form, which is later then passed to 
         * the parameter type converters.
         */

        //If the email parameter was set, convert it to an array
        if (isset($data['email'])) {
            $data['email'] = [$data['email']];
        }

        //Increment the baz parameter, if it was set
        if (isset($data['baz'])) {
            $data['baz']++;
        }

        //Return the new data
        return $data;
    }

    /**
     * This method is called, to handle migration from version 1 to version 2.
     */
    public function migrateToVersion2(array $data, SettingsMetadata $metadata): array
    {
        //Perform some more migrations...

        return $data;
    }

}

The migration service contains various methods in the form migrateToVersionXX(), which are called automatically by the class if the settings are migrated from version XX-1 to version XX. The method receives the data in the normalized form and the metadata of the settings class and must return the data in the normalized form, which is then passed to the parameter type converters. If you want to specify explicitly which functions are called for which version, you can override the resolveStepHandler method, which returns the closure to use for a given version.

As the existing data had no version yet, it is assumed it was version 0. Therefore, when encountering these data settings-bundle will call the migrateToVersion1 handler to migrate from 0 to the most recent version 1.

The old data from the storage is passed to the migration method (as $data) and you have to convert it to the new form how it can be saved to storage and how the parameter type conversions can understand it. Each parameter is stored in the $data array with the parameter name as key. You can then modify the data as you like and return it in the end.

Please note that the $data array is in the normalized form, meaning that you only have simple datatypes like strings, integers, arrays and so on. If you want to like to work with the denormalized form (like objects, etc.) you might find the getAsPHPValue() and setAsPHPValue() methods available in the SettingsClass (or in the PHPValueConverterTrait) useful. Or you call the ParameterTypes you need directly.

The settings-bundle stores the version of the data in the storage provider, so that it is automatically known what version the data has and what migrations to perform. The migrations are automatically performed when trying to retrieve settings data (by getting the settings from the SettingsManager or calling a property of a lazy settings class). By default, the migrated data is written back to the storage after the migration, so that the migration only has to be performed once for each setting, even if the settings are not explicitly written back to the storage.

Environment variables

Environment variables are one of the classic possibilities to configure a Symfony application. They allow you for an easy configuration approach in automatic deployed applications, containers, etc. via a more or less unified interface. So they are pretty ideal for server administrators, who want to configure an application without touching the code. However, the big disadvantage of environment variables is, that they are not user-configurable, as users (even those intended as admin users) can not change them without direct access to the server.

To retain the advantages of environment variables, while also allowing users to configure the applications via the settings-bundle, the bundle can map environment variables to settings class parameters.

This is done via the envVar option on the SettingsParameter attribute:

#[Settings]
class TestSettings {

    #[SettingsParameter(envVar: 'APP_EMAIL')]
    public string $email = 'test@invalid';

    #[SettingsParameter(envVar: 'int:APP_BAZ', envVarMode: EnvVarMode::OVERWRITE)]
    public int $baz = 42;

    #[SettingsParameter(envVar: 'bool:APP_TEST_SETTINGS_QUX', envVarMode: EnvVarMode::OVERWRITE_PERSIST)]
    public bool $qux = true;
}

The envVar option specifies the environment variable to map to the parameter. If it does not exist, nothing happens. However, if it exists, the bundle will retrieve the value of the environment variable and set it as the value of the parameter. By default, the "raw" environment variable contains just a string. If you have another simple data type (like an integer or a boolean), you can use one of Symfony's env var processors to convert the string value of the env variable to the desired type (e.g. int:APP_BAZ, which converts the content of APP_BAZ to an int).

The environment variable handling happens transparently in the background, meaning that you can use the settings class as usual, and you (almost) do not have to care about the environment variables when using the settings.

Environment variable handling modes

The envVarMode option specifies how the environment variable should be handled. If no mode is specified, the mode EnvVarMode::INITIAL is used. In this mode the environment variable is only used to initialize the parameter. That means if the parameter is used the first time, instead of the default value in the code, the value of the environment variable is used. Users can change this value as they like, and the environment variable will not affect the parameter anymore. This mode allows a server administrator to set useful initial defaults via environment variables (e.g. while deploying the container), but users can change them completely later.

However, in some cases, you might want the server admin to enforce a certain value via environment variables and forbid users to change them via WebUI. For these cases, you can use the EnvVarMode::OVERWRITE and EnvVarMode::OVERWRITE_PERSIST mode. In this mode, the environment variable will always overwrite a parameter value, no matter what was set as a value before by users. This means that freshly retrieved settings will always have the value of the environment variable, even if the user changed it before. The OVERWRITE_PERSIST mode additionally writes the value back to the storage, so that the value is still set even after the env variable is removed (however users can then change the value again).

If a parameter is overwritten by an environment variable, its form field will be disabled in the default generated WebUI, so that users can see that the value is enforced by the environment variable and can not be changed via the WebUI.

A limitation of this system is that you can still change the value of a settings parameter in your code, even if it is overwritten by an environment variable. The changes will also be used in other parts of the application during the request. It is just that these changes do not get persisted, meaning that if you reload the settings from the storage, the value of the environment variable will be used again. If you try to change settings parameters via direct access in you code, you might want to check if the parameter is overwritten by an environment variable (by using the isEnvVarOverwritten method of the SettingsManager), and if so, you might want to disable the possibility to change the parameter in your code.

Environment variables mapper

For many constellations, the type conversion via the env var processor works fine. However, in some cases where you have more complex parameter types, you need a more complex conversion logic. For these cases, you can use the envVarMapper option of the SettingsParameter attribute. This option specifies a callable, which is called with the value of the environment variable and must return the value to set as the parameter value:

class TestSettings {

  #[SettingsParameter(envVar: 'string:ENV_VAR3', envVarMapper: [self::class, 'mapDateTimeEnv'])
  private ?\DateTime $dateTimeParam = null;

  public static function mapDateTimeEnv(?string $value): ?\DateTime
  {
    return $value ? new \DateTime($value) : null;
  }
}

The $value parameter passed, is the value retrieved from the environment variable, with env var processors applied, meaning that it not necessarily has to be a string.

Conclusion

You can see that jbtronics/settings-bundle can support you with handling changes in the schema of settings, and how to map environment variables to settings parameters. This allows you to have a flexible configuration system, which can be used by users and server administrators alike.

As always you can find more information in the bundle documentation.