We’ve been lax in implementing CI in the latest project I’m working on. Now that decision has caught up with me and its time to implement something. I mainly held off on CI because I couldn’t bring myself to have to setup another Jenkins instance. I’m a long time user and fan of Jenkins, I even developed the Perforce plugin for it back when it was called Hudson.

But these days, I no longer want to manage servers, be they physical or virtual. Life is just too short. Instead, I’d like to use one of the new CI services that has cropped up in recent years. For something as important as CI, I’m not going to make a blind choice. Especially, when we’ve been spoiled with how customizable Jenkins is. So let’s get down to reviewing some build tools…

I tested three CI services, what I considered to be good bets basedon my loosely based requirements:
– I don’t want to scratch out my eyes when looking at the UI
– I don’t want to spend a crazy amount of time installing/configuring/modifying my project to work with a CI service
– I want it to be reasonably fast
– I want it to be reasonably inexpensive
– I want it to be hosted
– I want it to integrate with everything in my toolchain

My top three to test (and it was after only about an hour of research, did I miss any?) were:

  • Codeship (100 builds/month free, $49/month cheapest paid plan)
  • Snap CI ($30/month cheapest paid plan)
  • Travis CI ($130/month cheapest paid plan)

(Travis CI actually fails the inexpensive test at $130/m starting price. But with how widely used it is, I kept it in the list.)

The use case I went through was quick enough to give me an indication of, “Should I do a deep dive and try this product out for awhile?” This is not meant to be an exhaustive review and comparison of the tools. More of a first impression, if anything.

CodeShip

Codeship worked out of the box. I signed in with GitHub, selected my repository, and the first build started. The UI is alright. The build list screen is quite ugly and messy.

Pros:

  • UI is fairly recent
  • Worked out of the box for rails/rspec/postgres, no config needed.
  • Handles deploys to Heroku
  • Got a response from support on my question about reports in 50 minutes
  • Reasonable way to use the system for free: 100 builds for free per month

Cons:

  • Build results UI is terrible
  • No test reports generated! No details about why a build failed!

Snap-CI

Snap also worked out of the box for me. The process was the same as Cadetship. Sign in with GitHub, select my repository and a “pipeline” was configured for me. Rails/rspec/postgres. The first build was finished quickly.

Pros:

  • Seems like the pipeline and stages with auto/manual triggers are very flexible.
  • UI is quite thought out. However, they could do with bumping up their font size.

Cons:

  • 26 hours to get a response to my question on reports.
  • Doesn’t seem to rerun a pull-request when new commits are added to it? LOOKS LIKE IT DOES ACTUALLY but maybe there is a delay. The first time I tried this, I didn’t see any new build. But the pull-request with new commits did create a new build.
  • It won’t build when there is a new commit on a non-master branch. Only a new pull-request. Not entirely a problem, but
    something to be aware of.
  • A Pipeline automatically created for a build request did not match pipeline configured for project. It was an out of date version. You must edit the configuration for pull requests and “sync from master configuration.” Takes a bit to figure that out.
  • New builds for the pull request aren’t propagated to a higher level in UI. You have to drill down to pull requests to see results.
  • No color in log output. A small thing, but coupled with the next issue, a PITA.
  • No test reports generated! No details about why a build failed!

Travis-CI

Pros

  • UI is good. Although maybe overly simple.

Cons

  • Fracking Price Man. $130 to play ball.
  • 25 hours to get a response to my question on reports.
  • No automated setup. Must create a config file and hope you get it right. It took a good hour to read the docs and and figure out everything I needed to get to where I was within minutes with the other two.
  • Seems to run a build twice if the first one fails. I noticed this randomly a few times.
  • Along with no automated setup, I dislike how it has separate config stored in source code for things like ruby versions, postgres details, build steps, etc.
  • It doesn’t seem to cache gems, but I could be wrong. Builds took much longer than Snap-CI or Codeship

After reviewing these three, I’m very disappointed. I just can’t believe that a CI system doesn’t allow for generating, reviewing, and reasoning about the results of a build.

With each, the only thing a build does is execute commands. It does nothing intelligent with the results beyond pass something on to the next build phase or uploading a file or deploying a file. Because there is no intelligence around the results of a build phase:

  • I can run source code analysis, but I can’t display the results or reason about the results.
  • I can run test coverage, but I can’t display the results or reason about the results.
  • I can have a build fail from failed tests, but I don’t know what tests or how many failed. I have to take the initiative to parse the build log to see what broke.

What the Front Door?!

It seems each of them can rely on 3rd party services to provide for things like test code coverage and code analysis. But all that does for me is increase cost, complexity, and latency of the entire CI/CD system. Not to mention, now, I’m trusting another 3rd party with the confidentiality of my code.

After all these years, is self hosted Jenkins still the CI leader?

What am I missing?

About the Author:

Learned something? Great! Need help on your development project? I'm available for hire:

  • Ruby on Rails
  • iOS Development
  • System Architecture & Performance

Get in touch:

Discussion

  1. Vishal Naik says:

    Hi Mike,

    Regret the delay in getting back on that ticket about displaying build results on Snap. We are usually more prompt than that :)

    You can configure automatic branch tracking on Snap so that new branches are picked up based on the pattern. https://docs.snap-ci.com/working-with-branches/automatic-branch-tracking/

    For existing branches, you can create a clone on the pipeline of any branch https://docs.snap-ci.com/working-with-branches/cloned-pipelines/

    Snap will also attempt to merge the changes to the master and then run selected stages so that you have better confidence on the changes in the branch.

    We have noted all the other feedback you mentioned. The one about reporting test results is certainly something we have on our roadmap.

    Do checkout our blog on how Snap is different from other hosted services
    https://blog.snap-ci.com/blog/2014/07/22/why-snapci-and-travisci-are-not-the-same-thing/

    This one is from a user who wanted to move away from Jenkins and compared a bunch of hosted CI services before going with Snap.
    https://medium.com/continuous-delivery/continuous-delivery-3a4a55baa58a

    Happy to talk to you if you have any questions about Snap. Just send us an email or mention us on our twitter handle @snap_ci

    Thanks.
    Vishal

Leave a Comment