2016-09-17 | 2069 words | Description of work done to automate Perl 6 releases
Deconfusion note: if you're just a regular Perl 6 user, you likely use and only ever heard of Rakudo Star, which is a distribution that includes the Rakudo Perl 6 Compiler, some modules, and the docs. This post details a release of that compiler only, which gets released more often than Rakudo Star. So please don't think there's a new release, if Star is all you use.
If you follow what I write, you know that last month I messed up a Perl 6 release and vowed to improve the process, by making a release robot. Well, I didn't lie!
* Zoffix dissapears for a couple of weeks
<Zoffix> Can be reached via twitter if needed
I popped in some relaxing music and got cracking. Today, I'll talk about the goodies I've made, which touch a much broader scope than just releasing Perl 6.
Let the gift unwrapping begin!
Perl6.Fail: R6 Is The New RT
First thing I needed was a web app where a release manager could view new bug tickets since last release and mark them as release-blockers, if needed. The app needs to keep track of what tickets were already reviewed, so the release manager can spend just a couple of minutes of their time every few days, instead of cramming everything into a single sitting on release date.
I named that app R6.
The six fits the Perl 6 theme, and 6 is above T
on the keyboard, which I found apropos, since my app is better than
stock RT we currently use 😜.
With the name for the bug app in place, I went to hunt for neat domain names to
host it on and nearly immediately found the perfectest one:
perl6.fail.
Helping release managers is the smaller side of the utility of the app and it aims to address some of the major pain points with RT (or rather with our particular instance of RT that we also have to share with Perl 5).
Viewing Tags
The RT version has overly complicated interface when it comes to trying to find a ticket tagged with a particular tag. Worse still, some tags are just special codes in ticket subject lines, while others use the actual tag interface. Annoying!
This is one of the first things I solved in R6. The home page lists all available tags, along with ticket counts for each tag. Simply clicking the tag will show the tickets just for that tag.
Simple. Just the way it's supposed to be.
Searching and Decent Editor
Once I get more tuits, I plan to also add full-text ticket search and a decent Markdown editor for ticket replies.
Release Managers
If you log in as a release manager, you get extra bells and whistles in the interface that let you mark tickets as was the original plan, as well as mark commits as reviewed, when updating changelog.
API
The release robot has to fetch info from this app to know whether all tickets
and commits have been reviewed. As such,
adding .json to the URL
of most pages will cause the app provide output in JSON format, rather than
plain ol' HTML.
Speaking of robots...
Buggable
Some of you have already seen my prototype bot buggable. It
was... well, let's just say the name suits it.
<buggable> ☠ [travis build above] One job failed but NOT due to the timeout.
<lizmat> "No output has been received in the last 10m0s, this potentially
indicates a stalled build or something wrong with the build itself."
<lizmat> sounds like timeout to me
<buggable> geekosaur, Found 2 tickets tagged with STAR.
Details: http://bug.perl6.party/1470861574.html
<geekosaur> o.O
<geekosaur> buggable appears to have a math problem
First, I fixed the Travis bogus-failure watcher and taught it to handle missing build logs and GitHub connectivity issues:
<buggable> [travis build above] ✓ All failures are due to timeout (0),
missing build log (1), or GitHub connectivity (0)
As for tags and tickets, I completely threw away the old backend that used RT API and its own database. Instead, the bot now uses the API of the R6 app I made.
<Zoffix> buggable, tags
<buggable> Zoffix, Total: 439; BUG: 189; UNTAGGED: 58; LTA: 54; RFC: 40;
CONC: 20; SEGV: 18; JVM: 14; UNI: 13; PERF: 12; TESTNEEDED: 12; POD: 9;
PRECOMP: 9; @LARRY: 7; NATIVECALL: 7; NYI: 6; IO: 5; REGEX: 5; BUILD: 3;
MATH: 3; REPL: 3; GLR: 2; OSX: 2; WEIRD: 2; BOOTSTRAP: 1; CONFIGURE: 1;
OO: 1; RT: 1; SITE: 1; SPEC: 1; STAR: 1; TESTCOMMITTED: 1;
See http://perl6.fail/ for details
<Zoffix> buggable, tag bug,rfc
<buggable> Zoffix, There are 2 tickets tagged with BUG, RFC;
See http://perl6.fail/t/BUG,RFC for details
Not only all the issues are gone, it now picks up the proper RT tags too and not just the ones in the subject line. You can specify multiple tags with a comma, to find tickets with a combination of tags.
Once ticket search is implemented in R6, buggable will be able to search for
tickets, which can be handy in-chat when trying to decide whether something
was or needs to be rakudobugged.
NeuralAnomaly
Finally, I'm proud to present the star of the hour and the newest member of the
release team: NeuralAnomaly. It graciously made the release of the Rakudo
Perl 6 Compiler this month, successfully replacing me at the job:
<Zoffix> NeuralAnomaly, yo
<NeuralAnomaly> Zoffix, hey
<Zoffix> NeuralAnomaly, it's time
<NeuralAnomaly> Zoffix, Oh boy! Really?! We're doing a realease‽‽ YEY!
<Zoffix> Yup
<Zoffix> NeuralAnomaly, cut the release
<NeuralAnomaly> Zoffix, Will do! If you're feeling particularly naughty, you can watch me at http://perl6.fail/release/progress or go look at some cats http://www.lolcats.com/
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ Prep done
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ nqp tests OK
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ nqp release tarball tests OK
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ nqp release DONE
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ Rakudo stresstest (master) OK
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ Rakudo stresstest (6.c-errata) OK
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ Rakudo release DONE
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ Post: upload tarballs to rakudo.org
<NeuralAnomaly> Zoffix, 🎺🎺🎺📯📯📯📯📯📯🌈🌈🌈📦📦📦
<NeuralAnomaly> Zoffix, The release of Rakudo #103 2016.09 has now been completed
<NeuralAnomaly> Zoffix, 🎺🎺🎺📯📯📯📯📯📯🌈🌈🌈📦📦📦
* NeuralAnomaly celebrates with an appropriate amount of fun
The Tech
The code for all of the deliverables is available on GitHub, in R6 repo, Buggable repo, and NeuralAnomaly repo.
R6
The R6 app uses RT's REST API to ask the running RT instance for updated tickets every 10 minutes.
I chose Perl 5 as the weapon of choice, utilizing the talents of
the Mojolicious web framework and
DBIx::Class ORM.
I attempted to
use RT::Client::REST for API
interfacing, but found the module oddly designed and requiring too many
requests to obtain information I needed. So I implemented the
relevant portions of the RT's REST API interface myself.
For user accounts, I stole borrowed, RT::Client::REST authentication
mechanism, although at the time of this writing, no user-account features
are available to users (and perl6.fail still runs on plain HTTP). And
Will "Coke" Coleda was very helpful in pointing out a relevant portion of the
API that let me save making a huge number of requests. And current
implementation makes
just a single request every 10 minutes, asking only for updated tickets since
last request.
Buggable
The bot uses my very own IRC::Client Perl 6 module
and is pretty thin
and isn't much to look at. Using JSON::Fast and HTTP::UserAgent
modules it accesses R6 using its JSON endpoints to fetch
the tag info and perform ticket searches.
Travis features use Travis API. Since the stuff I use does not require authentication, this is nothing more than fetching data from an endpoint and decoding JSON. I found JSONViewer.Stack.Hu helpful when figuring out what bits of data I wanted to keep.
NeuralAnomaly
When I planned this bot, I suspected developing it would be somewhat difficult,
with lots of thinking... In reality, writing code for it
turned out to be super easy.
Popping ssh into Proc::Async
was child's play, and the Proc bailed out on non-zero exit codes, which made
it super easy for me to abort failing stages of the process. I basically
ended up with Perl-6-super-charged bash scripts... quite literally:
However, when it came to giving gpg and git tag the passphrase for the
key, that... is worth its own section.
Won't You Take My Passphrase Please
The major roadblock was trying to pass
the GPG passphrase to the gpg (which was easy) and to the git when signing
the tag (which got annoying quick).
Avoiding idiotic solutions that tell you to write your passphrase
into world-readable files, I went
to enable the gpg-agent by installing gnupg-agent, uncommenting
use-agent in ~/.gnupg/gpg.conf,
and running eval $(gpg-agent --daemon --sh)
That did the trick with starting the agent, but git tag was now outright
choking when attempting to sign, telling me 'gpg: cancelled by user',
even though
I did naught.
After several hours of trying things and getting a helping hand from mst, the solution was nowhere in sight:
However, after installing a program called unbuffer
(sudo apt-get install expect-dev), reading its help page, and
inserting an appropriate amount of sleeps, I ended up with these
chunks of code in my release bot that seemed to do the trick when passing
them to commands requiring keys:
constant $with-github-credentials is export
= "(sleep 6; echo -e '$github-user\\n';"
~ " sleep 6; echo -e '$github-pass\\n'; sleep 12) | unbuffer -p";
constant $with-gpg-passphrase is export
= "(sleep 6; echo '$gpg-keyphrase'; sleep 12) | unbuffer -p";
...
gpg --batch --no-tty --passphrase-fd 0 -b \\
--armor rakudo-$rakudo-ver.tar.gz ||
\{ echo '$na-fail Rakudo: Sign the tarball'; exit 1; \}
$gpg-keyphrase
...
$with-github-credentials git push
Echoing like that is not ideal, but this is running in a single-user VM, so it'll do for now.
The remaining bits of the scripting were easy.
Step Right Up
After completing full release scripts for NQP and Rakudo and ensuring they work, it was time to break them. If a single spectest test fails in final tarball testing due to being floppy, you don't want to repeat the whole process from scratch. So I broke up release scripts into bite-size pieces and made the bot able to run individual pieces on command.
That had a positive unintended effect:
<Zoffix> NeuralAnomaly, run pre r-clone r-build r-p5 r-stress r-stress-v6c
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ Prep done
<NeuralAnomaly> Zoffix, ♥♥♥♥♥♥ Rakudo stresstest (master) OK
<NeuralAnomaly> ♥♥♥♥♥♥ Rakudo stresstest (6.c-errata) OK
<NeuralAnomaly> ♥♥♥ All Done! ♥♥♥
<Zoffix> :D
<Zoffix> So the release bot doubles as a stresstester too :)
Testing
The most difficult annoying phase was testing. Not only
does the full build runs for about 40 minutes even on my 24-core VM, but I
had to delete and then re-clone both NQP and Rakudo repos, because their state
had to be pristine.
After countless hours and hundreds of Rakudos built, I can now recite the build process with my eyes closed :)
The Future
All of these goodies burnt me out a bit—stuff started to feel like "work" rather than enjoyment. It wasn't the code, but the testing. There's nothing quite like 40-minute test sessions, where a single typo sends you to square one. So I plan to take the next month or so "off," working on other things in Rakudo. But eventually, I plan to improve both R6 and the release robot.
Currently, the bot does not send email announcements and I've not even checked whether Wikipedia has an API to make automated updates. R6 also has a ton of improvement potential by allowing to view and comment on existing tickets, as well as provide useful search features.
Conclusion
This month, Perl 6's release cycle received an awesome upgrade: release steps got abstracted into commands to be issued to an IRC bot. More to it, release managers no longer have to cram all of their work into a single day, but are now free to spread it out throughout the month.
The new website perl6.fail makes it easy to locate tickets tagged with a particular tag and will be eventually expanded into a more usable bug tracker interface.
More than getting just the release robot, the Perl 6 development team now also has a spectest robot that will regularly "practice" doing releases, spotting any of the issues much sooner than previously was possible.
Things got better.
-Ofun