Configuration Management

Johan Olsson

• 6 min read
Configuration Management
Image by Freepik

As promised, I will delve into configuration management and how to provision configurations to the Azure App Configuration Store resource. The Azure resource provides a user interface, retention policies, secret storage, and various ways to provision configurations. However, when it comes to environmental configuration, cost management, sharing configurations among multiple apps, and more, the Azure App Configuration Resource and configuration provisioning may feel limited. While the Microsoft documentation for App Configuration is good and suitable for basic usage, for more complex scenarios, we need to create our own system for configuration creation.

In this blog post, I aim to accomplish a few things in my solution:

1. Versioning alongside the code: Storing all configuration values in a Git source repository enables versioning alongside the code.

2. Ability to share configuration: Sharing configurations among applications is necessary to propagate new configurations for all applications simultaneously. For example, the configuration for an endpoint address should have one source of truth so that all client applications can utilize the same address.

3. Link Secrets from a Azure Key Vault: It is necessary to protect and manage secrets, certificates, and more. The process of adding a secret should be as easy as adding a configuration value. However, as I described in a previous post, secrets always involve some additional steps, such as inputting the secret somewhere (e.g., GitHub Secrets or Azure Key Vault, in my case).

4. Developer help for writing new configurations: To maintain and develop new application configurations effectively, it is essential to provide developers with assistance. This can be achieved by implementing a JSON Schema for validation and providing IntelliSense support in VSCode.

5. Tooling: The aforementioned objectives are made possible through the use of custom design scripts and harnessing the diversity of infrastructure and configuration resources.

Implementation

First, we need to structure the files and folders for our configurations in Git. The following is an example structure:

+---.vscode
|       settings.json
\---src
    \---configuration
        +---configurations
        |   +---environments
        |   |   \---dev
        |   |       +---catalog
        |   |       |   |   appsettings.json
        |   |       |   \---inventory-items
        |   |       |           appsettings.json
        |   |       +---gateway
        |   |       |       appsettings.json
        |   |       \---order
        |   |               appsettings.json
        |   +---globals
        |   |       scopes-schema-definitions.json
        |   +---iac
        |   |       env-configuration.json
        |   |       keyvalues.bicep
        |   |       main.bicep
        |   |       main.bicepparam
        |   \---scripts
        |       \---configuration-squirrel
        |               hoard-configuration.ps1
        +---iac
        |       main.bicep
        |       main.bicepparam
        \---schemas
                config-schema.json

Although this structure may seem comprehensive, the files are lightweight, and each folder and file serves a specific purpose:

  • The configuration folder contains everything related to configuration.
    • The configurations folder is for environmental configurations.
      • The environments folder contains nested environments (in this example, only the dev environment is present).
        • The [dev] folder (can have any name) serves as a container for all application configurations for a given environment. This structure also supports nesting.
      • The globals folder contains a file with defined JSON Schema definitions for the allowed scopes (more on scopes later).
      • The iac folder at the inner level is used for deploying the configurations themselves.
      • The scripts folder contains PowerShell tools for gathering the configuration, for example, for use in GitHub Actions.
    • The iac folder at the outer level is used for deploying the Azure App Configuration Stores.
    • The schemas folder contains the JSON Schema files that help enforce the style of the configuration files in the dev folder.

Versioning

✅ All the configuration will be checked in git source control.

Share Configuration

If we examine a configuration file, it contains properties of Scopes and optionally AdditionalScopes at the config node level. These Scopes are also enforced by the JSON Schema definitions located in the globals folder.

//src/configuration/configurations/dev/catalog/appsettings.json
{
  "Scopes": ["catalog-service"],
  "catalog:service:endpoint": {
    "Value": "https://localhost:44302",
    "AdditionalScopes": ["api-gateway"]
  },
  "catalog:datastore:cosmosdb": "https://catalog.documents.azure.com:443/"
}

The Scopes serve as marker names that will be part of the final configuration stored in Azure App Configuration as separate labels. Each Scopes value will be a separate configuration in Azure App Configuration.

In the above example, the node catalog:service:endpoint would be spread out over two scopes: catalog-service and api-gateway. The catalog:datastore:cosmosdb would only have the catalog-service scope.

This feature will empower developers to share configurations for multiple applications (scopes).

Secrets can simply be linked using the Azure Key Vault URI, with or without the version identifier.

//src/configuration/configurations/dev/catalog/appsettings.json
{
  "Scopes": ["catalog-service"],
  "catalog:service:endpoint": {
    "Value": {
      "Endpoint": "https://localhost:44302",
      "ApiKey": "https://kv-dist-cfg-nonprod.vault.azure.net/secrets/catalog-service-apikey-dev"
    },
    "ContentType": "application/json",
    "AdditionalScopes": ["api-gateway"]
  },
  "catalog:datastore:cosmosdb": "https://catalog.documents.azure.com:443/"
}

By using the https://kv-dist-cfg-nonprod.vault.azure.net/secrets/catalog-service-apikey-dev link, we effectively create an Azure App Configuration Azure Key Vault link configuration resource. The Azure Key Vault linking even works if the secret has not yet been created in Azure Key Vault.

Note: This requires that the consuming application or client has access to the Key Vault via RBAC or Access Policies.


The linking process is simple, but the process of handling the secrets and putting them in the Key Vault must be defined as a DevOps process within the organization or development team.

The above two statements should be considered on a case-by-case basis.

Developer help for writing new configurations

With the JSON Schemas in place, developers get IntelliSense and error checking, which helps them in the process of creating and managing configurations on a daily basis.

Gif Animation of JSON Schema Help

With the JSON Schema, we can enforce the schema by failing at build or pull request time, thus notifying the developer of any issues.

Tooling

The tooling I've created for this solution is a PowerShell script that scans files, uses folder and JSON file conventions, and produces a configuration JSON file that can be used for deployment by the iac module.

The script takes the path to the src/configuration/configurations/environments folder as input and generates a JSON file that is compatible with the iac configuration deployment.

Show Powershell script 
param (
    [Parameter(Mandatory = $true)]
    [string]$sourceEnvironmentPath
)

function Get-PropertyValueOrDefault {
    param (
        [PSCustomObject]$Object,
        [string]$PropertyName,
        $DefaultValue
    )

    $hasProperty = [bool]($Object.PSObject.Properties.Name -match $PropertyName)
    if ($hasProperty) {
        return $Object.$PropertyName
    }
    else {
        return $DefaultValue
    }
}

function Get-VaultValues {
    param (
        [PSCustomObject]$Object,
        [string]$Environment,
        [array]$Scopes,
        [string]$Path = '',
        [object]$Tags
    )

    $vaultValues = @()
    $value = $Object.Value
    $localScopes = ($Scopes + (Get-PropertyValueOrDefault -Object $Object -PropertyName "AdditionalScopes" -DefaultValue @())) | Get-Unique

    foreach ($property in $value.PSObject.Properties) {
        if ($property.Value -is [String] -and $property.Value -match '\.vault\.azure\.net') {
            foreach ($propertyScope in $localScopes) {
                $config = [PSCustomObject]@{
                    Name        = if ($Path) { "$($Path):$($property.Name)" } else { $property.Name }
                    Value       = $property.Value
                    ContentType = "application/vnd.microsoft.appconfig.keyvaultref+json;charset=utf-8"
                    Label       = "$Environment-$propertyScope"
                    Tags        = $Tags
                }
                $vaultValues += $config
            }
            # Remove the property from the source object
            $value.PSObject.Properties.Remove($property.Name)
        }
        elseif ($property.Value -is [PSCustomObject]) {
            $vaultValues += Get-VaultValues -Object $property -Environment $Environment -Scopes $Scopes -Path "$($Path):$($property.Name)" -Tags $Tags
        }
    }

    return $vaultValues
}

function Get-ConfigurationNodes {
    param (
        [string]$PropertyName,
        [object]$PropertyValue,
        [string]$Environment,
        [array]$Scopes,
        [object]$Tags
    )

    $configurationValues = @()

    $localScopes = $Scopes + (Get-PropertyValueOrDefault -Object $PropertyValue -PropertyName "AdditionalScopes" -DefaultValue @())
    $contentType = Get-PropertyValueOrDefault -Object $PropertyValue -PropertyName "ContentType" -DefaultValue ""
    $value = Get-PropertyValueOrDefault -Object $PropertyValue -PropertyName "Value" -DefaultValue $PropertyValue

    if ($PropertyValue.Value -is [PSCustomObject]) {
        $configurationValues += Get-VaultValues -Object $PropertyValue -Environment $Environment -Scopes $localScopes -Path $PropertyName -Tags $Tags
    }
    if ($value -is [PSCustomObject]) {
        $value = "$($value | ConvertTo-Json -Compress -Depth 100)"
    }

    foreach ($propertyScope in $localScopes) {
        $config = [PSCustomObject]@{
            Name        = $PropertyName
            Value       = $value
            ContentType = if ($value -match ".vault.azure.net") { "application/vnd.microsoft.appconfig.keyvaultref+json;charset=utf-8" } else { $contentType }
            Label       = "$Environment-$propertyScope"
            Tags        = $Tags
        }
        $configurationValues += $config
    }

    return $configurationValues
}

function Get-EnvironmentConfiguration {
    param (
        [object]$environment
    )

    $transformedConfigs = @()
    Get-ChildItem $environment.FullName -Recurse -Filter "*.json" `
    | Select-Object -ExpandProperty FullName
    | ForEach-Object {
        $sourceJsonPath = $_

        $sourceJson = Get-Content -Path $sourceJsonPath -Raw
        if ($null -eq $sourceJson) {
            Write-Host "Could not parse contents of file: '$sourceJsonPath'"
        }
        else {
            $sourceObject = ConvertFrom-Json $sourceJson
            $scopes = $sourceObject.Scopes

            foreach ($property in $sourceObject.PSObject.Properties) {
                $propertyName = $property.Name
                $propertyValue = $property.Value
                if ($propertyName -eq "Scopes") {
                    continue
                }
                $transformedConfigs += Get-ConfigurationNodes -PropertyName $propertyName -PropertyValue $propertyValue -Environment $environment.Name -Scopes $scopes -Tags $propertyValue.Tags
            }
        }
    }

    $transformedObject = [PSCustomObject]@{
        Configs = $transformedConfigs
    }
    return $transformedObject
}

$environments = Get-ChildItem $sourceEnvironmentPath | Select-Object -Property Name, FullName
$environments | ForEach-Object {
    $environmentConfigurations = Get-EnvironmentConfiguration $_

    $transformedJson = $environmentConfigurations | ConvertTo-Json -Depth 10
    $transformedJson | Out-File "./$($_.Name)-result.json"
}

Let's examine the input and output for a single file:

Yellow indicates the scopes. Notice how the configuration values have been spread out into multiple labels.
Orange indicates the environment and is prefixed to the Label property.
Red indicates Azure Key Vault secret links.
Green indicates generic configuration. If the value is of type application/json, we stringify and escape the JSON for Azure App Configuration to interpret it properly.

Results in Azure App Configuration

Wrapping up

All the code for this post is located in the GitHub repository.
I will continue develop this repository along with the blog posts about configuration.

Next we will look at different ways of solving Configuration Validation.