linci configuration

How to write build configs

Writing build scripts is a case of creating a text file in /var/linci/config with a specific name file extension. If you are in the linci shell create build myproj will create a file /var/linci/config/myproj.build based on a template, and open it in vi, or whatever editor you have defined in the EDITOR environment variable. Vim syntax highlighting is provided, contributions for other editors are more than welcome.

This is what a linci build configuration looks like

linci> edit myproj

description: Example build
environment:
init: p4-checkout.sh ssl:vbox:1424 //depot/myproj/... 
clean: rm -rf *

targets::

sync: p4-update.sh
test: test/compile-bash.sh
deb: sudo deploy/builddeb.sh
install: sudo deploy/installdeb.sh
publish: sudo -E /var/linci/config/lib/publish.sh myproj-*.x86_64.deb 

Lines before targets:: must be executed individually e.g.

linci> build myproj clean

All the lines after targets:: are executed in order when you call

linci> build myproj

Syntax

If you use vi the following syntax rules are validated in the editor.

File names

The file extensions should be .build for build files and .job for job files, everything else is ignored.

File names are very restricted letters from a to z, numbers 0 to 9 and the dash character. While Linux OS allows filename with any UTF-8 character in them linci does not. Linci follows sanename guidelines. You are strongly discouraged from using uppercase letters but you may, if you insist; the regexp of file names is as follows. ^[a-zA-Z][a-zA-Z0-9-]*$

Commands and escaping

You are strongly encouraged to keep linci commands down to simply defining a script to run and its arguments. Multiple lines are not supported. Apart from that, technically, you could put a lot of weirdo bash code into a single command line and it might work since linci copies each command to a temporary shell script and runs it. No variable expansion is performed and there is no modification of the command at all when the temporary script is created. That applies to the current version, presuming a bash execution environment is not a wise idea.

Comments

Lines starting with a hash # and blank line are ignored when parsing build files.

Targets

Each target line should be a target name, a colon, and a command. The target name should be all lowercase letters from a-z, no numbers or capitals or other funny characters (^[a-z]+$).
Warning: Target name syntax restrictions are not validated at time of writing, but if you try things may break subtly and will break in future versions of the code when validation is implemented.
Whitespace is significant, it is not permitted before the target name or between the target name and the colon, after the colon any whitespace is sent unadjusted to bash, bash ignores leading whitespace.

Passing arguments to targets

No arguments are passed to the targets. The target itself can be a command followed by arguments. If you need to support running the same build with different runtime arguments use environment variables. e.g.

linci> FOO=bar build myproj

Encoding

File should be text files. Line endings should be \n, use system encoding, i.e. UTF-8. Commands will be sent direct to bash unchanged, bash expects UTF-8 on all modern Linux systems.

Current directory

Each command is executed from the workspace's current directory, ala Make. It is possible to call each target individually with linci scripts, if CD were maintained this would be complicated. This is also done as a deliberate annoyance to discourage trying to use build files as mini scripts by writing lots of targets.

If you are interested in best practices for what to put in linci build files and what to put in scripts in the source tree or elsewhere, read the following sections.

What goes where

When writing config files for linci its important to make the distinction between what you can do and what you should do. The linci config files allow you to run any commands; that gives you enough rope to hang yourself.

linci's responsibilities are as follows.

Check out the code

First step in a build is to check out the code and this has to be linci's responsibility. Script to do an initial checkout and to update to latest version should be stored with the linci code here /usr/share/linci/bin/vcs.

It would have been possible to store all the rest of the build instructions inside the source repository, other CI tools go this route. Linci takes the stance that some configuration related to linci's responsibilities are best stored on the server, outside the source code tree. This facilitates setting up configurations that don't necessarily perform the whole CI flow, and allows you have have different builds for the same source. e.g. development on a branch might require linci to perform all of the normal tasks, but instead of publishing direct to production, it should publish to a QA environment. With that in mind linci defines a location /var/linci/config for build configuration files, the syntax is defined below, and a location for linci specific scripts. When creating linci configuration they should contain a simple list of what to do from the above responsibilities. Linci config files should not contain too many details about how to perform any of those jobs. The build configs in /var/linci/config/ should be backed up or under version control.

Linci build configs define what tasks linci should execute, the definition of how tasks should be performed should be stored in one of two locations, the source tree or on the linci server. The following sections aim to help deciding in which of the two places to put scripts. It may seem like it does not matter, if you have ever done a migration from one build environment to another, you'll know it does.

Build processes

Scripts that define how to build the code should almost always be in the source tree. Typically this is a Makefile, maven poms, ant scripts etc. this is pretty standard practice. However its worth noting that as CI tools have got more features often build instructions end up in the CI tool. Linci aims to avoid that anti-pattern, hence the requirement that the build task be a one-liner.

Development processes

If the task is to be performed by developers working on the same code tree outside linci, scripts defining how to perform these tasks should be in the source repository. e.g. scripts that define how to run unit tests (if thats not integrated in the build of has some setup thats more than a one-liner). Linci targets should always be a one-liner. Its possible to write in a config file

pre-test: scripts/prepare.sh
do-test: scripts/run-unit-tests.sh
Thats bad form, since this is encapsulating how to run tests in linci. Presuming the tests will also be run outside linci the two steps to the process should be defined outside linci. In this case in a trivial script in ./scripts/

Publishing / Reporting processes

Presuming that you never publish code straight from your laptop and you follow a CI flow that ensures tests are run before a release. It may make sense that scripts to handle publishing should be in linci itself, these may be complicated or at least something more than a one-liner. If you have a standard process shared across many projects it makes sense to puts these scripts in linci. The correct location is /var/linci/config/lib. These scripts should be versioned and backed up along with linci configs. There is no reason you can't store publishing scripts in the source control along side the code, but if it makes sense to separate these scripts from the code /var/linci/config/lib is the place to put the scripts to make them available to linci. Similarly if you have reporting scripts that are only relevant in the CI server, e.g. some script that monitors the builds and calculates who developer of the month is, the lib dir is the place for these scripts. The idea is to avoid littering the source tree with linci specific code. If you change the CI tool it should not require changes to the source code.

Deployment processes

Its not so clear cut as to where to store scripts that define deployment processes. Potentially this is a CI only concern and the scripts should live in /var/linci/config/lib. This is OK provided you version lib carefully and don't loose track of which deployment process goes with which version of the source. The alternative approach of defining the deployment environment requirements in AWS build scripts or a Vagrantfile makes a lot of sense.


Hopefully, after reading all that, you understand why we did not just use Make.