Package Meta Processing

Monteur has a special metadata processing function for keeping its CI Job sane and maintainable. The inner mechanics are explained here!

The Problem

It was all started with a very messy mathematical problem. Say a build variant linux-amd64 is required to be packaged into 3 different distribution channels:

.appimage
.deb
.tar.gz

That's a total of 1 x 3 = 3 SAME CI job recipe files.

Now, try to deploy a full list of build variants for the same list of channels above, let's say:

linux-arm64
linux-arm
darwin-amd64
darwin-arm64
windows-amd64
windows-arm64

That would be 6 x 3 = 18 SAME CI job recipe files! If we scale things in a mathematical way, that's a total of m x n duplicated CI job recipe files for a project supporting m number of build variants for n of distribution channels.

That is A LOT of duplications which is extra ordinary scary and unmaintainable! This means that we need a solution to scale the CI Job not just horizontally across each different distribution channel but also vertically across each build variant.

The Solution

To solve the problem above, Monteur uses this Package Meta Processing approach to facilitate the vertically scaling for any amount of build variants. The idea is to:

Keep the CI Job recipe specific to the distribution channel (means scaling horizontally).
Each recipe lists out all the build variant and package them repeatedly based on the recipe (means scaling vertically).

By doing so, we kept all vertically scaling vector into 1 single CI Job recipe while letting the recipe creator to focus on the distribution channel's requirements.

Current Deployment

Currently, this function is deployed in the following CI Jobs:

Package since Monteur version v0.0.1.
Release since Monteur version v0.0.1.
Prepare since Monteur version v0.0.2.

Built-In Packagers

Monteur offers a list of built-in packagers to provider cross-platform supports as far as possible for applicable CI Job like Package CI Job. To select a packager in the recipe, simply set Metadata.Type as shown below:

1
2
3


[Metadata]
...
Type = 'deb-manual'

Currently, these are the available packagers:

manual
1. EXTERNAL - Perform the packaging manually via the recipe's CEU commands.
2. Available since Monteur version v0.0.1.
deb-manual
1. EXTERNAL - Perform the .deb packaging manually using third-party software like debuild via the recipe's CEU commands.
2. Available since Monteur version v0.0.1.
targz
1. BUILT-IN - Perform the .tar.gz packaging using built-in targz function.
2. Available since Monteur version v0.0.1.
zip
1. BUILT-IN - Perform the .zip packaging using built-in zip function.
2. Available since Monteur version v0.0.1.

Data Structure

To operate this function, Monteur uses a single unified data structure. Please keep in mind that not all fields are used in a CI Job recipe. You should consult the CI Job's documentation for specific fields' availability and purpose.

The full data structure is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[Packages.(ID)]
OS = [ 'linux' ]
Arch = [ 'amd64' ]
Name = '{{- .PkgName -}}-{{- .PkgVersion -}}-{{- .PkgOS -}}-{{- .PkgArch -}}'
Distribution = [
        'stable',
]
Changelog = '{{- .DataDir -}}/debian/changelog-{{- .PkgArch -}}'
BuildSource = false
Source = '{{- .PackageDir -}}/targz/{{- .PkgName -}}-{{- .PkgVersion -}}-{{- .PkgOS -}}-{{- .PkgArch -}}.tar.gz'
Target = '{{- .ReleaseDir -}}/archives'

[Packages.(ID).Files]
'{{- .PackageDir -}}/monteur' = '{{- .BuildDir -}}/{{- .PkgOS -}}-{{- .PkgArch -}}'
'{{- .PackageDir -}}/License.pdf' = '{{- .LicensePath -}}'

Package.(ID)
1. Label of the package. the (ID) can be anything as long as period (.) is not used.
2. (ID) data map is used to uniquely sort each package to a single entry. That being said, any same (ID) data structure comes later will overwrite its existing entry.
3. Recommends to use Platform Identification IDs as (ID) like linux-amd64 for consistency and mental sanity sake.
  Example: [Packages.linux-amd64]
4. Available since Monteur version v0.0.1.
Package.(ID).OS
1. List of supported operating system. The FIRST value is given priority for any single value filling.
2. Can be used as .PkgOS (first value) in Variables Formatting for supported fields.
3. Available since Monteur version v0.0.1.
Package.(ID).Arch
1. List of supported CPU architecture. The FIRST value is given priority for any single value filling.
2. Can be used as .PkgArch (first value) in Variables Formatting for supported fields.
3. Available since Monteur version v0.0.1.
Package.(ID).Name
1. Naming pattern for the package's filename without the file extension.
2. Variables Formatting function is available for formulating the pattern dynamically.
3. Depending on the packaging program, this field can be overwritten (e.g. debuild has its own strict naming pattern.
4. Available since Monteur version v0.0.1.
Package.(ID).Distribution
1. List of distribution series in a distribution channel. Example: stable, unstable, experimental, debian, ubuntu, bullseye, and etc.
2. When in doubt, stick to: stable, unstable, and experimental.
3. Introduced and used by Release API.
4. Available since Monteur version v0.0.1.
Package.(ID).Changelog
1. States the location of the changelog data file.
2. Variables Formatting function is available for formulating the pathing dynamically.
3. Introduced and used by Package API.
4. Available since Monteur version v0.0.1.
Package.(ID).BuildSource
1. Decision to package source codes depending on the packager program.
2. Value can be either true or false.
3. Introduced and used by Package API.
4. Available since Monteur version v0.0.1.
Package.(ID).Source
1. Package filepath.
2. Variables Formatting function is available for formulating the pathing dynamically.
3. Introduced and used by Release API.
4. Available since Monteur version v0.0.1.
Package.(ID).Target
1. Release destination directory.
2. Variables Formatting function is available for formulating the pathing dynamically.
3. Introduced and used by Release API.
4. Available since Monteur version v0.0.1.
Package.(ID).Files
1. The list of files for packaging in a KEY:VALUE pattern. The KEY is the destination while the VALUE is the source of the file.
2. Variables Formatting function is available for formulating the pathing dynamically for both KEY and VALUE accordingly.
3. For safety reason, files are copied over instead of move.
4. Should the destination has a different filename, the copied file shall be renamed accordingly.
5. Example, for '{{- .PackageDir -}}/License.pdf' = '{{- .LicensePath -}}', a file located in .LicensePath shall be copied into .PackageDir/ directory and renamed as License.pdf.
6. Introduced and used by Package API.
7. Available since Monteur version v0.0.1.

Epilogue

That's all for Monteur's Package Meta Processing function. If you have any question, please feel free to raise your question at our Issues Section.