If you successfully added a Gitlab project and pushed our Yesod code there, you might notice that some builds are being executed on your runner. That’s because the project already contains a .gitlab-ci.yml file. As you can see, it’s pretty much empty – just some prints for sake of checking whether the runner is configured properly.
Since it is, now is the time to adjust our pipeline to a more complex scenario. Obviously, there are dozens of pipelines used for many cases. Here I want to present one, quite simple deployment routine. We won’t be using all the steps yet (since we only have unit tests now), but they will come later during development (if not on this blog, then during your own coding sessions).
I propose a four-step pipeline, in .gitlab-ci.yml coded as:
dev-testing is the part executed by developers on their local machines – this usually boils down to some linter, compilation and unit test execution. I treat “unit tests” as tests which do not require any particular binary file available on build server (for example a separate database instance or some set of services). For these reasons, tests that use only sqlite are fine for me in this phase. Of course, feel free to disagree, I’m not going to argue about it. This phase goes first (before the “official” build phase), because – for compiled languages like Haskell – a separate build is required for test cases, and that’s kind of a commit sanity check. This stage should only fail if the developer didn’t run proper scripts before commit (ideally never), or if he’s not required to (e.g. for a really small project).
Next stage, packaging, is a phase that should never fail. It consists of building a whole deployment package, resolving dependencies, constructing RPM, DEB, Docker image or whatever deployment system do you use and pushing it to test-package repository (not necessarily – it may be sometimes passed as an artifact between builds).
Third stage, integration-testing is arguably the most important piece of the whole pipeline. It is needed to verify whether all the pieces fit together. They require full environment set up, including databases, servers, security rules, routing rules etc. I’m a big fan of performing this phase automatically, but many real-world projects require manual attention. If you have such a project, the best advice I can give you is – run whatever is reasonable here, and publish internally if it passes. Then handle the passed scenarios to your testers and add another layer of testing-publishing (possibly using a tool dedicated for release management). This stage will fail often – mostly due to bugs in either your code or your scripts (which are also your code) – there will be races, data overrides and environment misalignments. Be prepared. Still, it’s the purpose of this stage – things that will fail here, most probably would fail on production otherwise, so it’s still good!
The last stage, publishing is simple and should never fail – it should simply connect to release repository and put the new package there. It might be an input point for your Ops people to take it and deploy, it might be an input point for the testers. This stage should be executed only for your release branches (not ones hidden in developer repositories) and is the end of the automated road – next step has to be initated by a human being, be it deployment to production or further testing. This job should also make a proper version tag on the repository (this may be done in packaging as well, but I prefer to have less versions).
Of course, all stages may additionally fail for a number of reasons – invalid server configuration, network outage, out of memory exception, misconfiguration etc. I didn’t mention them earlier, because they aren’t really related to (most of) the code you create and will occur pretty much at random. However, remember my warning: while they might seem random, you should investigate them the first time you encounter any of them. Later on they will only become more and more annoying, and in the end you’ll either spend your most-important-time-just-before-release-oh-my to solve them or ignore the testing stage (which is bad).
A few more words about the choice of tooling: I tend to agree that Gitlab CI might not be the best Continous Deployment platform ever, especially due to limited release management capabilities and tight connection to automated-everything (I like it, but most projects require some manual testing). Perhaps a choice of Jenkins or Electric Flow would be better, but would require significantly more attention – first of all, installing and configuring a separate service and second – managing integration. Configuring Gitlab CI only takes a few lines of YAML, but for Jenkins it’s not that easy anymore!
Now, after we’ve managed to design the pipeline, let us create an example jobs for it.
dev-testing is easy – it should simply run stack setup && stack test (we have to linters for now).
preparing-package is a little trickier:
preparing-package:
stage: packaging
script:
- stack setup
- stack install --local-bin-path build
artifacts:
paths:
- build/
expire_in: 1 hr
cache:
- .stack-work
first, we need to install the package to build directory (otherwise it would remain in a hash-based location or be installed to local system – which is not what we want), then define the artifacts (whole build directory) and it’s expiration (1 hour – should be enough for us). The cache option is useful to speed up compilations – workspace is not fully cleared between builds. Note that this might be dangerous, if your tools don’t deal well with such “leftovers”. However, clean installation of GHC and all packages takes about a year, so caching is required (of course, you may also set up your own package server with a cache for the used ones, if your company is a tad bigger).
Rest of the stages is just printing for now – we have no integration tests, and installing Apt repository or Hackage server seems to be a bit of an overkill right now. I also hate polluting the public space (public Hackage) with dozens of packages, so I won’t do nothing right now there (I might reconsider later on, of course!).
If you download the code from GitHub, you will see that it doesn’t work in Gitlab. Apparently, stack is not installed in our Runner container! This requires quite a few commands, but luckily, they are listed in Stack GitHub installation manual.
For Ubuntu Server 16.04 this goes as following:
# add repository key sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 575159689BEFB442 # add repository echo 'deb http://download.fpcomplete.com/ubuntu xenial main'|sudo tee /etc/apt/sources.list.d/fpco.list # update index and install stack sudo apt-get update && sudo apt-get install stack -y
Manual configuration management and tool installation is not the best practice ever, but it’s often good enough, as long as the project is relatively small (or you have dedicated people for managing your servers). We might consider changing this to some configuration management tool later on, when dependencies get more complex.
Aaand, that’s it! First pipeline builds should already successfully leave the Gitlab area. Congratulations!
Next post, promised a long time ago – GHCJS instead of jQuery in our app – comes soon.
Stay tuned!
codemore code
~~~~