Variables Processing

To reduce and prevent duplications, Monteur offers its variables system for dynamic processing in its config files. It's unique and clean!

The Problem

During the first CI Job design and creation, the team bumped into a big problem when:

  1. There is a need to dynamically formulate a field. A good case study would be to formulate a package name for a given project/app version. Constantly modifying many Package API recipe because of the version changes is not maintainable as there are too many duplications.

  2. Too many dynamic values duplications. Likewise, a lot of project/app info can be made available in a single app data file and query from there.

  3. Specific CI Job customization needs. Some CI Job recipes, due to the reliance on interfacting with third-party program, can have its own specific parameters and inputs processing.

Hence, we need a technical solution to solve all these problems.

The Solution

To solve the problem above, Monteur facilitates variables formatting for the following purposes to applicable data fields:

  1. Comply to Don't Repeat Yourself principle.

  2. Permits the ability to dynamically create configurations data for adaptive environment.

  3. Permits an algorithmic approach to simplify all CI Job's recipe whenever applicable.

A very good example of its usage is the Commands Execution Unit where dynamic formatting is heavily used to formulate its final command instruction. Here is an example where the Location field is dynamically filled with the value from .WorkingDir: 
1
2
3
4
5
6
7
8

[[CMD]]
Name = 'Get Publish Branch First Commit for Cleaning'
Condition = 'all-all'
Type = 'command'
Location = '{{- .WorkingDir -}}'
Source = 'git rev-list --max-parents=0 --abbrev-commit HEAD'
Target = ''
Save = 'FirstCommitID'

Variables Processing function is available since Monteur version v0.0.1.

Data Structure

The variable processing function deploys 2 different types of variables: the plain variables and the formattable variables. Although both have different roles and purposes, they share the same operating mechanics which are a common and unique list of KEY:VALUE data.

The KEY from the variable is used as the ID for the formattable variables is strictly a string data type.

The VALUE however, can be any data types.

Monteur always maintain a SINGLE common and uniquely identified list of variables and carries out the "create or overwrite" policy. Hence, any duplicated variables entry having the same KEY shall overwrites its existing value.

Plain Variables Definition

The first type is the plain variables. These variables are straight-up basic values meant to be used directly or for formatting. They are parsed as it is.

The data structure usually looks something as such: 

1
2
3

[Variables]
Version = '1.17.3'
BaseURL = 'https://golang.org/dl/'

  1. The table indicator is [Variables].

  2. Sub-definitions (e.g. [Variables.MyDef]) are strictly prohibited.

Formattable Variables Definition

The second type is the formattable variables. These variables can be formatted with existing available variables before being parsed into the variables list.

The data structure is usually looks something as such: 

1
2
3
4
5

[Variables]
Version = '1.17.3'

[FMTVariables]
Payload = 'myapp-{{- .Version -}}.tar.gz'

  1. The table indicator is [FMTVariables].

  2. Sub-definitions (e.g. [FMTVariables.MyDef]) are strictly prohibited.

  3. Due to the KEY:VALUE list nature, there is no guarentee that the items are parsed in a consistent and orderly manner. Therefore, ONLY create formattable variables independent of one another.

  4. The above example shall create or overwrite the variable Payload with the Value as myapp-1.17.3.tar.gz, as Version = '1.17.3' was supplied as a plain variable right above.

Variables Formatting

It's quite easy to use variable formatting. Consider the following fields as our case study: 

1
2
3
4

Subject1 = "Name  {{ .Version }}  .tar.gz"
Subject2 = "Name  {{- .Version }}  .tar.gz"
Subject3 = "Name  {{ .Version -}}  .tar.gz"
Subject4 = "Name  {{- .Version -}}  .tar.gz"
The first thing to consider is the whitespace cleaning which are shown in all 4 subject fields. Then, given as example that we have a plain variables table as such: 
1
2

[Variables]
Version = '1.0.0'
There is a KEY called Version pointing to its string value 1.0.0. Hence, the above case study yields: 
1
2
3
4

Subject1 = "Name  1.0.0  .tar.gz"
Subject2 = "Name1.0.0  .tar.gz"
Subject3 = "Name  1.0.0.tar.gz"
Subject4 = "Name1.0.0.tar.gz"
You should notice that the existence of the hyphen (-) right after/before the opening/closing template braces instructs the removal of whitespaces like space " ", new line (\n), return carriage (\r), and tab (\t) in that direction.

Variables Formatting Availability

Not all data fields have variable formatting function. One good case is the plain variables above where it makes no sense to faclitate variable formatting.

Therefore, you need to consult the component's documentation to check the variable formatting's availability.

Parsing Mechanism

To achieve the solution presented earlier, Monteur maintains multi-layer overwriting policy for parsing all variables across different data sources. Here, we will briefly walkthrough each key activities in sequences.

Level 1: Parsing Main Source

Monteur starts off by parsing all plain variables ONLY from the main configuration files located in .configs/monteur/workspace.toml by default. This will serve as the foundation of the variables list.

All variables here are visible globally across all CI Jobs.

Level 2: Parsing CI Configuration Source

Monteur then parse the CI Job's common configuration plain and formattable variables. All formattable variables are processed before being saved into the variable list. The data source are located in your CI Job recipe file usually located in .configs/monteur/[CI-JOB]/config.toml.

Any new variables created at this level ONLY visible to all the recipes in this CI Job while any existing global variables shall remain globally visible.

Also, any conflicting variables (same KEY) shall be overwritten by the new ones.

Level 3: Parsing Recipe Source

Monteur then parse the recipe-level plain and formattable variables. All formattable variables are processed before being saved into the variable list. The data source are located in your CI Job recipe file usually located in .configs/monteur/[CI-JOB]/jobs/[recipe].toml.

Any new variables created at this level ONLY visible to THIS recipe while any existing variables retains its status quo visibility.

Also, any conflicting variables (same KEY) shall be overwritten by the new ones.

Reserved Variables

In common cases, Monteur reserved and use a set of pre-defined variables for its CI Job operations, made visibly available via the variables list. Hence, you should avoid these pre-defined KEY when creating your own variables!

Among them are:

Epilogue

That's all for Monteur's Variable Processing Function. If you have any question, please feel free to raise your question at our Issues Section.