Setup Meta Processing

Monteur has a special metadata processing function for keeping its Setup CI Job sane and maintainable. Here's its working mechanics and specifications!

The Problem

It was started with supporting development on various operating system and CPU archictures. Due to this, many software and tool packages are organized in its own list of distributions, naming conventions, and release strategies.

Among known problems are:

  1. There are simply too many platform variants and archiving formats offered by 3rd-party software for great cross-platform experiences.

  2. There are no consistent executable scripts (e.g. shell vs batch) that allows unilingual scripting.

  3. There are various combinations and permutations of methods to make dependencies available for a development.

  4. All the problems above are specific to Setup API and not seen in other jobs.

The Solution

Since Monteur wants its dependencies management to be managable via ONE (1) interface without too many duplications, this setup meta processing function is created to solve all the problems above. To solve the problem above, Monteur uses this Setup Meta Processing function to solve those Setup CI Job specific problems above for any amount of the supplied variants in a managable manner. The idea is to:

  1. Keep the dependency artifact sourcing from supplier maintainable using the overwrite policy.

  2. Facilitates various type of built-in sourcing methods to meet various sourcing needs.

  3. Acts as a command intepreter whenever possible for ensuring performance consistencies across different platforms specifically dealing with Setup CI Job.

By doing so, we kept the development, maintenance, and using Setup CI Job experiences seamlessly easy to perform.

Current Deployment

Currently, this function is deployed in the following CI Jobs:

  1. Setup since Monteur version v0.0.1.

Data Structure

This function is specifically built into the Setup CI Job. Hence, its data structure is scattered across various parts and sections of the CI Job recipe. In this specification, all its settings are specified in each sub-sections by following the CI Job's recipe's data file from top to bottom.

Sourcing Method Type

This function offers a number of sourcing method types that are offered in its own catalog section offered later in this page. Each type has its own explaination of its requirements and the how-to in the recipe file. The configuration is set via Metadata.Type. Here is a code snippet of the recipe file: 

1
2
3
4
5

[Metadata]
...
Type = 'https-download'

...

Recipe Sourcing Section

This function has a very special method to list all the recipe's sourcing data complying to Don't Repeat Yourself policy. To do that, the function employs the overwriting policy allowing one to first write a generic settings and then overwrite the needed fields with specific data.

Depending on the sourcing method type above, some types might not need all the fields. Hence, please refer to your selected sourcing method type's specification on how to construct this section.

Here is the full list of data fields along with its dummy data for specification's referencing purposes: 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

[Sources.{OS}-{ARCH}]
Format = 'tar.gz'
URL = '{{- .BaseURL -}}{{- .Archive -}}'
Archive = 'go{{- .Version -}}.{{- .ComputeSystem -}}.{{- .Format -}}'
Method = 'GET'

[Sources.{OS}-{ARCH}.Checksum]
Type = 'sha256'
Format = 'hex'
Value = '...'

[Sources.{OS}-{ARCH}.Headers]
"User-Agent" = 'Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148'
"auth-token" = "{{- GetSecret "Github.Token" -}}"

  • Sources.{OS}-{ARCH}

    1. The Monteur Platform ID for this sourcing data to be effective. Example: [Sources.all-all] and [Sources.linux-amd64].

    2. COMPULSORY - this setting is used for uniquely grouping and structuring source data between multiple groups of data fields. It is also used for facilitating cross-platform supports like listing all the sources here instead of spinning multiple recipe files.

    3. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Format

    1. Originally designed to specify the source format.

    2. COMPULSORY / OPTIONAL - Its definition is in the selected sourcing method type's specification.

    3. Supported built-in format are raw, tar.gz, and zip.

    4. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.URL

    1. Originally designed to specify the source URL for networking sourcing type.

    2. COMPULSORY / OPTIONAL - Its definition is in the selected sourcing method type's specification.

    3. Depending on the sourcing method type, some can implement Monteur's built-in functions like unpack https download.

    4. Variables formatting is available for this field.

    5. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Archives

    1. Originally designed to define the downloaded archived filename pattern.

    2. COMPULSORY / OPTIONAL - Its definition is in the selected sourcing method type's specification.

    3. Depending on the sourcing method type, some can implement Monteur's built-in functions like unpack https download.

    4. Variables formatting is available for this field.

    5. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Method

    1. Originally designed to define the HTTPS method for HTTPS download type.

    2. COMPULSORY / OPTIONAL - Its definition is in the selected sourcing method type's specification.

    3. Depending on the sourcing method type, some can implement Monteur's built-in functions like unpack https download.

    4. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Checksum

    1. Perform checksum integrity checking using Monteur's built-in checksum function.

    2. COMPULSORY / OPTIONAL - Its definition is in the selected sourcing method type's specification.

    3. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Checksum.Type

    1. Defines the checksum algorithm type. Supported algorithms are: md5, sha256, sha512, and sha512->sha256.

    2. COMPULSORY - if Sources.{OS}-{ARCH}.Checksum is enabled.

    3. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Checksum.Format

    1. Defines the format of the checksum value. Supported values are: hex, base64, and base64-url.

    2. COMPULSORY - if Sources.{OS}-{ARCH}.Checksum is enabled.

    3. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Checksum.Value

    1. Defines the checksum value.

    2. COMPULSORY - if Sources.{OS}-{ARCH}.Checksum is enabled.

    3. This field was added since Monteur version v0.0.1.

  • Sources.{OS}-{ARCH}.Headers

    1. Originally designed for HTTPS sourcing method type to append or overwrite the HTTPS request headers.

    2. OPTIONAL - made available in case there is a need for customization like setting authentication token.

    3. The final value is a list of KEY:VALUE headers where the KEY is the header identifer while VALUE is the value of the header. Both KEY and VALUE are as it is.

    4. Variables formatting is available for the VALUE of the field.

    5. This field was added since Monteur version v0.0.1.

User Startup Configurations

This function also offers a standard methods of starting up the project repository after setup completions. This allows the user to startup the repository for any tasks in a uniformed manner.

From Monteur version v0.0.1, this function uses the operating system's command line scripting method to source all configurations. A full list of its data structure is shown below: 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

[Config]
linux = """
#!/bin/bash

_name="hugo"

case $1 in
--stop)
	;;
*)
	if [ ! -z "$(type -p "$_name")" ]; then
		1>&2 printf "[ DONE  ] '$_name' is available.\\n"
        else
		1>&2 printf "[ ERROR ] '$_name' is missing.\\n"
	fi
	;;
esac
"""
windows = """
...COMING SOON...
"""
darwin = """
...COMING SOON...
"""
...

  • Config.{OS}

    1. The {OS} MUST be the Monteur Platform ID like linux and windows.

    2. COMPULSORY - the value MUST be the operating system's terminal scripts like shell or powershell scripts.

    3. Whenever applicable, Montuer shall never set these scripts to be executables.

    4. This field was added since Monteur version v0.0.1.

Available Sourcing Methods

Monteur shall continuously develop and add new sourcing methods (Metadata.Type) from time to time. Here are the available sourcing methods at your disposal: