In my last blog post, I wrote about structuring the code of an Arduino library, and I also mentioned how important it is to write tests and proper, detailed documentation for it. Today I will build on those foundations and explain the concept of continuous integration with Arduino in mind. As a tool, we’ll use Travis CI, which is for open-source projects free of charge.
Team collaboration in git
Imagine a situation where you work on a project with two other developers. In our case, an Arduino library. You’re using git to store your code, and every developer has its copy.
You’re adding support for user authentication; your colleague writes code to read data from the GSM module, and another colleague works on input data validation. Now, as you implement the feature, your local codebase naturally diverges from the files on the server — a single source of truth. For simplicity, let’s assume you all work on the same branch. If you don’t know what it is, read here.
At some point in time, you’ll push your changes to the central repository. Your colleagues will try to do the same. Ideally, you all have worked on different parts of the library, and you didn’t modify the same section of code as your two colleagues. In reality, this is rare, and you need to deal with the issues.
Git is just a tool. You’re the one responsible for resolving conflicting code changes.
Git provides several safeguards to merge the code from all sources correctly:
- If you changed the files, which were not modified on the server while you were working on the change, your change gets accepted (this behavior is configurable)
- If you changed the files, which somebody else updated on the server in the meantime, your code gets rejected. You need to fetch the latest changes and resolve merge conflicts.
- When you fetch the changes from the server, if git can figure out how to merge changed files, it will do it automatically. Otherwise, it will stop, and you need to merge the changes manually. Sometimes it’s hard. You may not be familiar with the whole codebase, and your conflict resolution may not be correct. Git will be, however, happy to accept your decision.
There may be other rules, but those are the most important. Let me emphasize one thing: git is just a tool. You’re the one responsible for resolving conflicting code changes.
Multiple branches and pull requests
Let’s expand the previous example a little, to be more realistic. Usually, your git repository has one main branch called
master in older repos). Developers working on a new feature will create a new feature branch from the main, e.g.,
feature/authentication. Similarly, if they work on a bugfix, they will create a bugfix branch. Multiple git development models exist to provide some guidelines, like GitFlow. Today I won’t go into details. I just want you to know that realistically a git repository may have a lot of branches, which get merged eventually into the main one.
A common way how to merge changes from the derived branches into the main one is through pull requests. These are not an integral part of git itself. Instead, various software tools added support for them on top of git. In GitHub, they are called pull requests, GitLab calls them merge requests, etc. The point is, you create a pull request, select a source branch from where to bring the changes, select a destination branch, where the changes should be merged and hit on the Create pull request button.
Other developers have a chance to take a look at your changes, comment them and, finally, merge them into the main branch. As long as between source and destination branch are no conflicting changes, git will happily accept the request and merge the code.
Continuous integration. Why do we need it?
If you count on merging the code solely on git, you have a problem. Git provides a guarantee that all conflicting changes were resolved, but it doesn’t guarantee that the code changes are correct and that the code still works properly.
If there’s a problem, you’ll eventually find out from your users sooner or later. We’ll probably all agree that it’s best to prevent these situations. Someone from lead developers could do occasional manual testing, but he may still miss something. Manual testing is tedious and time-consuming. Instead, we should focus on automating things. And that’s what continuous integration tools provide for us.
Continuous integration (CI) tools can help us with the following:
- They will automatically build your code after each change in any branch. If the machine isn’t able to build your code, you’ll get notified by an e-mail.
- They will optionally run your tests to make sure the code still does what it should do.
- They will generate documentation from the source code and publish it to a webpage. It’s sometimes called continuous deployment.
Imagine a virtual machine, which will be run just to build your code. You are responsible for specifying the environment, e.g., which operating system to use, what tools to install to run the code, what version of dependency libraries to use, etc. It’s up to you what things you want to automate. All commands which you can put into the command line, you can also put into the automation script.
Also, a very common thing to do is to build (and test) your code in multiple environments. One CI will make your library in Windows 10 with PlatformIO; another will use the Arduino toolchain under Linux. In the case of Arduino, you can also produce binaries for several different hardware platforms to make sure the library builds fine on Arduino Uno, Mega, or Leonardo. If the build process fails for any of the platforms, you’ll get an e-mail on failure, and you’ll need to fix your code to be compatible or drop support for the hardware platform. Either way, without the CI, you most likely wouldn’t know that your library doesn’t work on Leonardo, because you manually test only on Uno.
Today I will explain basic principles on Travis CI. This tool is free for all open-source projects. There are other CI solutions, e.g., Bamboo from Atlassian, or Jenkins. Also, GitLab provides its own CI, and recently GitHub introduced GitHub Actions. Most of the tools work very similarly. Once you get the gist for Travis, you’ll understand the rest.
Travis CI works with GitHub only. If you host your code someplace else, check the alternative solutions. Adding support for Travis CI into our repository is very easy. First, go to their webpage and sign up with your GitHub account. Travis will also ask you for authorization to access your repositories, just follow the guide.
After you’ve authorized Travis to access your public repositories, you’re still not done. Travis CI has no clue what to do once you make some changes to your code. For that purpose, you need to add a configuration file in the root of your repository. It’s called
.travis.yml and supports many options. Ideally, check the documentation for Travis CI to fine-tune the configuration.
In my last blog post, I explained the use of the ArduinoCI library to automate the building and testing of your Arduino library. Let’s elaborate on that and integrate ArduinoCI into Travis so that each commit of yours will trigger them. You can find the full configuration file here.
1sudo: false2language: ruby3dist: bionic4git:5 depth: false6 quiet: true7# Blacklist8branches:9 except:10 - gh-pages11before_install:12 - sudo apt-get install -y doxygen graphviz13script:14 - bundle install15 - bundle exec arduino_ci_remote.rb16 - doxygen doc/Doxyfile17deploy:18 provider: pages19 skip-cleanup: true20 github-token: $GITHUB_TOKEN21 keep-history: false22 local-dir: doc/html23 verbose: true24 on:25 branch: master
As you can see, I specified Ubuntu 18.04 Bionic Beaver as my target platform. Also, I requested the presence of the Ruby programming language. That’s because the author of ArduinoCI wrote it in Ruby.
before_install hook. There I ask to install additional dependencies. In my case, they are used to generate Doxygen documentation after successfully building the project.
script section, I put step-by-step commands I want to be executed. You can put there almost anything, even run external bash scripts. In my case, Travis will do the following:
- Install dependencies necessary to run ArduinoCI test framework
- Run ArduinoCI — It will try to build the code for several Arduino platforms and run unit tests if there are any
- If ArduinoCI finished successfully, Travis generates documentation for the project
deploy hook is intriguing. I use GitHub Pages to host my documentation online.
GitHub expects you to create a special repository branch called
gh-pages. When you put HTML and CSS files inside the root of the
gh-pages branch, you can browse the generated documentation online on a unique web address. For example, here is the documentation for my project. Check the GitHub Pages documentation for the details.
What I would like you to notice is that this deploy hook will automatically push your generated documentation into this particular GitHub Pages branch. Your documentation gets updated after each commit, which didn’t cause a failure in previous steps.
When you authorized Travis to access your repositories, you gave it read-only access. If you want from Travis to push code into your repository, you need to approve it explicitly. In theory, you could provide your username and password, but this is very insecure. Instead, GitHub allows you to generate access tokens. They are bound to your account, and you define the scope of what is permitted. Your credentials were not exposed, and you can revoke the token at any time from the GitHub Admin page.
When you have all this automation in place, there’s one very convenient feature for you to use. Travis CI provides for each repository a unique link. It points to a picture, which tells us that our build has passed. In the case that the build fails, it will update the image to show failure instead. The path doesn’t change.
Such an image is called a badge and is very common on GitHub. You can put it into your Readme file. Path to the badge picture doesn’t change, but the content reflects the current status of your repository. You know at a glance that there’s an issue.
Adding a badge into your Readme is a matter of just one line:
Continuous integration tools are very convenient. Even if you don’t have unit tests for your code, they add many benefits.
At a minimum, I would strongly suggest using the CI tools to perform automated builds for several target Arduino platforms. You’ll have at least some guarantee that your library will work on them. You can get this set up within minutes, and the benefits are enormous. For open-source projects, this should be a no-brainer. All other projects require a paid CI service — in these cases, enabling CI should be at least a subject of discussion. You can always set up your server for running your own CI server to lower your cost.