Arc Forumnew | comments | leaders | submitlogin
What direction should Anarki's continuous integration/deployment take?
2 points by rocketnia 15 days ago | 8 comments
Travis CI was acquired by a private equity company a couple of years ago. This November, they announced a new pricing structure that had many people believing they're shutting down their "free for open source" option. On November 24, they blogged again saying that they'll keep the service free for open-source projects.

I've watched a few projects respond to these developments by switching from Travis CI to GitHub Actions, especially in the Racket community. The readme of Greg Hendershott's travis-racket project (which we currently depend on for Anarki) has been recommending people switch to Bogdanp's GitHub Action setup-racket. Another option is Jack Firth's GitHub Action racket-package-ci-action. There might be a few more like these out there.

Even if Travis CI is staying around for the foreseeable future, it's understandable that people would migrate away from it in response to the uncertainty. And as for Anarki, we face not only the uncertainty of whether Travis CI will keep being viable, but whether the travis-racket repo will keep being viable too.

I think there are a number of options we could take:

- Stay on Travis CI. Keep using travis-racket, possibly forking it if there comes a need for that in the future. (This is easiest for now.)

- Move to GitHub Actions, following the Racket community's lead and benefiting from their attentive maintenance. (This is probably easiest into the future, but it basically does mean coupling our repository even further to GitHub.)

- Look into using GitLab specifically for its CI platform.

- Look into using CircleCI.

- Look into building a system of our own or a set of scripts that makes it more feasible to crank the tests manually.

I doubt this decision needs to be made with any urgency, but now seems like a good time to start up a discussion about it.

---

For background:

The Anarki repo currently uses Travis CI for continuous integration (CI) testing and continuous deployment.

The Travis CI script runs a few tasks on each commit and pull request:

- It runs the `raco` commands for installing, compiling, and testing the repository as a Racket library. (Anarki is a Racket library that can be installed with `raco install anarki` and used with `(require anarki)` and `#lang anarki`.) The `raco test` command corresponds to the tests run by the Racket package catalog, so a failure here corresponds to a "failing tests" status on the catalog.

- It runs the build-web-help.arc script which builds an HTML documentation from docstrings.

- - That in turn runs tests.arc, the primary unit test entrypoint in the codebase. This runs various .arc.t files, it runs the tests derived from conanite's Rainbow, and it runs the tests from zck's unit-test.arc. (Note that a failure in these tests is only reported in Travis CI and not in the Racket package catalog. We might want to hook it in to the catalog's testing flow, but I think they have a strict limit of 3 minutes, and our Arc tests might push that limit.)

- Finally, it deploys that documentation by automatically committing it and pushing it to the gh-pages branch. To do the commit, it uses the an access token for GitHub user @rocketniabot, which is entered into the Travis CI settings so that it's not committed to the repo.

Everything but the deployment is run for a few different versions of Racket. That way, we stay aware of our compatibility level.

---

The benefits of having a CI script at all, as I see it, are:

- It means we can pretty much test on just one version of Racket locally, relying on the CI script to catch when something breaks on another version.

- It means most people who contribute to Anarki don't have to worry about testing the rarely used Anarki-as-a-Racket-library functionality; they can just test using tests.arc as the readme recommends. If something about Anarki-as-a-Racket-library breaks, the CI script catches it.

- It means if someone updates a docstring, they don't have to make a second commit to the gh-pages branch just to update the HTML documentation.

I'm probably the main person driving most of the complexity here. I introduced the ability to install Anarki as a Racket library (and promptly never used it...). I introduced the docstring-based HTML documentation page and its automatic deployment, too.

Regrettably, it is rather complex in its current state. I've often seen people run across CI failures related to specific versions of Racket or to the behavior of Anarki as a Racket library and not know what to do to fix them.

If it's impossible to get maintainers fluent in fixing these issues, we should probably be willing to remove some of these for the future maintainability of the language.

However, I am hopeful that we can do a lot better with explaining what's going on with these and helping maintainers navigate them. For instance, as a starting point, the readme could update from this:

  # start in the anarki directory
  $ ./arc.sh
  arc> (load "tests.arc")
To this:

  # start in the anarki directory
  $ raco pkg remove anarki
  $ raco pkg install --deps search-auto --name anarki
  $ raco setup --check-pkg-deps anarki
  $ raco test --drdr --package anarki
  $ ./arc.sh -n build-web-help.arc
Kind of ludicrous, right? It confronts new users with a lot of information, and that's why I never pushed for it before. Still, allowing this complexity to go undocumented may have caused more confusion than clarity in the long run.

Maybe after a bit of massaging, we could simplify this to:

  # runs the core language tests (the default)
  $ ./arc.sh -n tests.arc
  
  # does the following:
  #   - does a test installation and `raco test` of the Racket package (racket)
  #   - does a dry run of documentation building (doc-dry-run)
  # (The doc-dry-run target also runs the core language tests.)
  $ ./arc.sh -n tests.arc racket doc-dry-run
  
  # builds documentation, first deleting anything already in the build-gh-pages/ directory
  # (This also runs the core language tests.)
  $ ./arc.sh -n tests.arc doc-full-build
  
  # deletes anything in the build-gh-pages/ directory, which is where docs are generated
  $ ./arc.sh -n tests.arc doc-clean
  
  # uninstalls any Racket package left behind by a failed test install
  $ ./arc.sh -n tests.arc racket-clean
Then if someone's work keeps breaking the HTML docs or the Racket library, they can simply learn to add "doc-dry-run" or "racket" to their test command when they're working on that part of the system.

With that in place, the scope of the CI script would simplify to:

- Run the `racket` and `doc-full-build` targets on a few different versions of Racket.

- Deploy the documentation by pushing it to the gh-pages branch, using a GitHub access token entered into the CI settings for authorization.

What do you think? I'm not sure if I actually have the time to follow through on all of this right at the moment. And, again, I don't think this is necessarily urgent; I just figure having a discussion about it would be good.





2 points by akkartik 15 days ago | link

Thanks for raising these questions! A few random thoughts that immediately occur to me:

My general bias is that CI infrastructure is uneconomic for this project[1], given the amount of effort I'm willing to put into Anarki at this point. But I have absolutely no objection to others continuing to maintain it.

I'd be fine with having a more complex set of instructions for contributors than newcomers. I'm totally willing to perform those commands for myself when I make changes to Anarki.

I'd also be ok with undoing some of this complexity if you want to do that, though I don't mean to push for that in any way.

I think it's also not super onerous to require people to manually update the docs when making changes. That way we can get rid of gh-pages, and configure Github to just rebuild docs from the trunk branch. That eliminates one use case for CI.

Your final proposal is also fine, though as you mentioned someone would have to implement it.. :)

[1] As context, I also plan to remove CI from the project I _do_ actively work on.

reply

2 points by rocketnia 15 days ago | link

I appreciate your frankness, lol. Anarki maintenance happens whenever we feel like it. :-p I have been feeling an inkling of an Arc urge lately, hence this post, but I have other priorities right now that this is basically procrastination from.

What do you mean by getting GitHub to build docs from the trunk branch? That's what that part of the CI script is there to simulate; I think there's still no actual way to configure GitHub Pages to build from anything but a `gh-pages` branch or a `<username>.github.io` repository.

reply

2 points by akkartik 15 days ago | link

A few years ago GitHub added the ability to specify a branch for a project. See the GitHub pages section of https://github.com/arclanguage/anarki/settings

I have a site at http://akkartik.github.io/mu but it's served directly from the main branch. There's no gh-pages branch at https://github.com/akkartik/mu

reply

2 points by rocketnia 13 days ago | link

Oh, I see, you're right. I checked that settings page, but the big blank "social preview" image made me think I was at the end of the page, so I forgot I could scroll down. That's interesting.

reply

2 points by krapp 14 days ago | link

>I think it's also not super onerous to require people to manually update the docs when making changes.

It really isn't. I need to add documentation for a lot of things.

reply

2 points by krapp 14 days ago | link

This is a bit off topic but what happened to the projects tab for the repo? I had a vaguely organized project for the forum up there.

reply

2 points by akkartik 14 days ago | link

Oh sorry, I cleaned up some tabs because I saw tabs overflowing into a menu on the right. I'll restore it.

reply

3 points by krapp 13 days ago | link

Ok thanks.

Also, hey everybody, there's a project tab for the forum on the repo. Feel free to add to it, or whatever.

I'll get around to working on it again sooner or later.

reply