Archive for the ‘Raku’ Category

Guarding dynamics

August 14, 2020 Leave a comment

Dynamic variables are a nice way to get the benefits of global variables without taking the drawbacks. They pass information up the call tree without forcing a declaration upon the caller. However, dynvars share a burden with exceptions. The callee knows how to do what the caller might not expect.

use Module::By::Author::A;
use Module::By::Author::B;

my $*dynvar = 42;

sub-from-A(); # expects $*dynvar to be Int
sub-from-B(); # expects $*dynvar to be IO(Str)
sub-from-C(); # expects $*dynvar to be IO::Handle

In this example sub-from-B() will silently fail until it tries to open the file named “42”. While sub-from-C() will try to coerce 42 to become a file handle and throw. So there lies a danger in dynvars expected to be set by independent modules. Worse, the behaviour might suddenly pop up after any zef --install. Raku is a dynamic language that will try its best to convert values automatically and fail at runtime. It is good advice to support the compiler by failing early.

I want to use dynvars in a few modules to provide semi-global flags to remove boilerplate. The following seems to provide protection against unintentional dynvar-spill over.

class Switch is Mu {
    has $.name;
    method gist { $.name }
    method Str { die('invalid coersion') }

constant on = :name<on>;
constant off = :name<off>;

sub s() {
    # put $*d; # this will die
    say $*d;
    dd $*d;
my $*d = off;
# OUTPUT: off
#         Switch $*d = => "off")

I derive from Mu to get rid of all the coercers from Cool and overload Str to take that one out of the loop as well. Using say for anything but debugging is a bug anyway, so I might support it properly with .gist. Since I create a type with the class I can then go and protect my subs and methods with a whereception.

sub dyn-var-typecheck(Mu:U \T, $name) {
    sub {
        DYNAMIC::($name) ~~ T || die(„$name is set to an unexpected value“)

constant &dyn-s-typecheck = dyn-var-typecheck(Switch, ‚$*d‘);

sub s($? where dyn-s-typecheck) { }
# OUTPUT: $*d is set to an unexpected value
#         in sub  at /home/dex/projects/blog/shielded-dynvars.raku line 6
#         in sub s at /home/dex/projects/blog/shielded-dynvars.raku line 22
#         in block <unit> at /home/dex/projects/blog/shielded-dynvars.raku line 29

Using the DYNAMIC::($name) in a where clause will slow down any call on MMD. So pulling the check into the sub might be reasonable.

With this measure I feel better to add $*always-capture-stderr to Shell::Piping to get rid of :stderr(Capture) on pretty much any pipe I start. And I feel a lot better when adding $*debug in everywhere.

Raku was not designed with type checks on dynvars in mind. It was designed to be a Perl. That gives us the flexibility to fix problems as they occur.

Categories: Raku


August 9, 2020 1 comment

I have a sub that takes a file and tries to guard itself against a file that does not exist. Where clauses don’t make good error messages.

sub run-test(IO() $file where .e & .f) { };
# Constraint type check failed in binding to parameter '$file'; expected anonymous constraint to be met but got IO::Path ("not-th...)

The signature of that sub is quite expressive. Often we don’t have time to read code to hunt down mistakes. That’s why bad error messages are LTA. We can extend any where clause with a die or fail to provide a better message.

sub run-test(IO() $file where .e && .f || fail("file $_ not found")) { };
# file not-there.txt not found

We can also throw exceptions of cause. With just one parameter the signature has gotten quite long already. Also, when working with many files we will write the same code over and over again. Since we don’t code in Java we would rather not to.

my &it-is-a-file = -> IO() $_ {
    .e && .f || fail (
sub run-test(IO::Path $file where &it-is-a-file) { }

That’s much better. Since exceptions are classes we can reuse code amongst them. Like coloured output and checking for broken symlinks.

class X::Whereception is Exception is export {
    has $.stale-symlink is rw;
    method is-dangling-symlink {
        $.stale-symlink = do with $.path { .IO.l & !.IO.e };

class X::IO::FileNotFound is X::Whereception is export {
    has $.path;
    method message {
        RED $.is-dangling-symlink ?? „The file ⟨$.path⟩ is a dangling symlink.“ !! „The file ⟨$.path⟩ was not found.“

I have added a few to Shell::Piping. Suggestions what else to check for are very welcome.

Categories: Raku

Dropin replacement

August 1, 2020 1 comment

Today I learned that whereis can take multiple commands to look for in $PATH.

$ whereis not-there raku-test-all zef
raku-test-all: /home/dex/bin/raku-test-all
zef: /usr/local/src/rakudo/install/share/perl6/site/bin/zef

The results are always in the requested order. So we can use a shell spell to find a candidate to be used as our tester in a Makefile.

TESTER := $(shell whereis raku-test-all zef | cut -d ' ' -f 2 -s | head -n 1)

        zef --depsonly install .

test: install-deps
        $(TESTER) --verbose test .

        zef install .

all: test

push: test
        git push

I had to do a few changes to raku-test-all so it mimics the interface of zef. The idea behind this Makefile is to be able to hit F6 and have all tests run and then push to github. Our current ecosystem simply distributes links to github repos. As a result pushing without testing can lead to somebody else cloning a broken repo (given the timing is bad enough). As you might imagine any speed up to testing is very welcome. Travis is quite slow. I think I should work in that area next.

Categories: Raku, shell

Wrapping Exceptions

August 1, 2020 1 comment

As stated before I would like to reduce simple error handling in Shell::Piping. The following is just way to long for putting a single line into a terminal.

    when X::Shell::CommandNotFound {
        when .cmd ~~ ‚commonmarker‘ {
            put ‚Please install commonmarker with `gem install commonmarker`.‘;
            exit 2;
        default {

We have a conditional in line 3 and a print statement in line 4. The rest is boilderplate. The shortest syntax I came up with looks like the following.

X::Shell::CommandNotFound.refine({.cmd eq ‚commonmarker‘}, {„Please install ⟨{.cmd}⟩ in ⟨{.path}⟩.“});
X::Shell::CommandNotFound.refine({.cmd eq ‚raku‘}, {„Please run `apt install rakudo`.“});

I want to modify the type object behind CommandNotFound because the module will create instances of them and throw them before I can intercept them. Of cause I could catch them. But the whole excercise is done to get rid of the CATCH block. Adding a method to any exception both inside a module, and thus at compile time, and at runtime would be nice. We can do so with a trait.

multi sub trait_mod:<is>(Exception:U $ex, :$refineable) {
    $ex.HOW.add_method($ex, ‚refine‘, my method (&cond, &message) {
        $ex.WHO::<@refinements>.append: &cond, &message;
        state $wrapped;
        $ex.HOW.find_method($ex, ‚message‘).wrap(my method {
            my @refinements := self.WHO::<@refinements>;
            for @refinements -> &cond, &message {
                if cond(self) {
                    return message(self);
        }) unless $wrapped;
        $wrapped = True;

A trait is a fancy sub that takes type objects or other stat-ish (Raku is a dynamic language without static things) objects and does stuff with it. That can be wrapping – a MOP operation in disguise – or actual MOP-calls. This redefining trait adds a class attribute named @refinements via autovification on the package-part of the type. This is hidden behind the pseudo method .WHO. Then it adds a method that takes a pair of Callables and stores them in @refinements. The type object’s original .message method is wrapped unless already done so. This is controlled by a state variable.

The wrapped .message will call the first callable and expects a Bool in return. If that one is True is will call the 2nd callable. Both are called with an exception instance.

The trait does not .^compose the type object it is called on. This is (for now) needed because of the following case.

class X::Shell::CommandNotFound is Exception is refineable { ... }

The trait is actually called on Exception. That works because the added method .refine is not shadowed by anything. We can therefore use this method to wait for it to be called to finish what we need to do after compile time. The wrapper has a fall through via nextsame if no condition has matched.

I don’t think I got all corner cases covered yet. That will need further testing. The user facing syntax can stay the way it is because traits are deboilerplaters. The trait itself is likely going to get more complex. But that is fine. The burden is supposed to be on the side of the implementer.

Categories: Raku

Concurrent dogfood

July 31, 2020 1 comment

I’m using zef --verbose test . in a Makefile triggered by the F2 key to run test in Raku projects. While zef is very nice it’s also very slow not fast yet. It could be a good bit faster if it could run tests in parallel. Since I split up tests in individual files running them concurrently isn’t all that hard. The biggest hurdle is to collect all the outputs and show them without mixing up lines. With Shell::Piping that is very easy indeed.

constant NL = $?NL;
my &RED = { "\e[31m$_\e[0m" };
my &BOLD = { "\e[1m$_\e[0m" };

&RED = &BOLD = { $_ } unless $*OUT.t;

sub run-test(IO::Path $file where { .e & .f }) {
    my @out;
    my @err;
    my $failed;
    px«raku -Ilib $file» |» @out :stderr(@err) :done({$failed =});

    („Testing: {$file}“.&BOLD, @out, @err».&RED).flat.join(NL);

I shell out to raku and let it run a single test file. The streams of STDOUT and STDERR end up in Arrays. These are then merged in the right order with some colouring for good measure. Now I have to get list of files and run run-test in parallel.

.put for dir(‚t/‘).grep(*.extension eq ‚t‘).sort.hyper(:batch(1), :degree(12)).map(*.&run-test);

The outputs are .put out in the right order thanks to the magic of .hyper. With a single raku process the tests need 11.3s. With 12 threads it’s down to 3s. I shall change the binding to F2 in vim at once!

The whole script can be found here and Shell::Piping here. The latter will land in the ecosystem shortly.

Categories: Raku

Dogfood time!

July 31, 2020 1 comment

Shell::Piping has now all the features I had on my list. So it is time to use it. I still need to work on documentation in the form of It would be nice to render it to HTML without pushing to github. As it turns out there is a ruby gem that takes a fancy Markdown file and outputs a HTML-fragment. I already had a script that embeds HTML into an HTML page with some effort to make it printable.

#! /usr/bin/env raku

use v6;

put q:to/EOH/;
      body {
        margin: auto; margin-top: 4em; max-width: 80em;

      @media print {
        @page { margin: 4em; }
        p { break-inside: avoid; break-before: avoid; }
        h3 { break-after: avoid; break-before: avoid; }
        h2 { break-after: avoid-page; break-before: auto; }
put $*ARGFILES.slurp;
put ‚  </body>‘;
put ‚</html>‘;

Since this script is taking input from STDIN by the virtue of $*ARGFILES it lends itself to be part of a unix pipe. Starting such a pipe by hand is way to much work. Writing to creates all the data needed to decide that and how to create a README.html.

#! /usr/bin/env raku

use v6;
use Shell::Piping;

react whenever Supply.merge(, ‚.‘ {
    say "something changed";
    for dir(‚.‘).grep(*.ends-with('.md')) {
        my $src = .IO;
        my $dst = $src.extension(‚html‘);
        if $src.modified > (try $dst.modified // 0) {
            my @html;
#`[MARK]    px«commonmarker $src» |» px<html-container> |» $dst;
            say ‚spurted!‘;

The MARKed line saves me about a dozen lines of code not counting error handling. Most errors will produce error messages by having exceptions thrown by Shell::Pipe. Since we can’t easily have dependencies across language borders, it would be nice to remind my future self what is needed.

    when X::Shell::CommandNotFound {
        when .cmd ~~ ‚commonmarker‘ {
            put ‚Please install commonmarker with `gem install commonmarker`.‘;
            exit 2;
        default {

I’m specialising an exception so to speak. There is the X::Shell::CommandNotFound type object and a conditional. If that conditional is not met we use the already provided exception. Otherwise we replace the message with a better one. I believe that is a patten worth thinking about. There may be more boilerplate to remove.

Categories: Raku

Augmenting with Exitcode

July 29, 2020 1 comment

In my last post I found a nice way to match against an Exitcode. I wanted to extend that to matching against STDERR if exitcode is non-zero. I already got a way to capture all error streams of a pipe.

px<find /tmp» |» px<your-script-here> |» @a :stderr(Capture);

I’m using Capture (the type object) in the same way as we use * or Whatever. It just indicated that magic stuff should happen. That magic boils down to sticking all STDERR streams into a 2-dimensional array. If I want to handle errors I might want to match against the exitcode, the name of the shell-command and of parts of it’s output to STDERR. A syntax like the following would be nice.

my $ex = :STDERR(<abc def ghi>), :exitint(42), :command<find>;
given $ex {
    when ‚find‘ & 42 & /def\s(\S+)/ {
        note „find terminated with 42 and $0“;

As it turns out getting the match against Str, Numeric and Regex in a Junction is easily done. All we need to do is augmenting Regex.

augment class Regex {
    multi method ACCEPTS(Regex:D: Shell::Piping::Exitcode:D $ex) {

augment class Int {
    multi method ACCEPTS(Int:D: Shell::Piping::Exitcode:D $ex) {

augment class Str {
    multi method ACCEPTS(Str:D: Shell::Piping::Exitcode:D $ex) {

This only works for matching. I don’t get (\S+) to capture into $0 that way. We know and love that Str.match does do that – seemingly with ease. Let’s steal codelearn from it!

proto method match(|) { $/ := nqp::getlexcaller('$/'); {*} }

So the first thing .match is doing is to bind it’s local $/ to the caller’s one. Thus any changes to the local version will actually change the caller’s. I tried to mimic that and it didn’t work. Neither the nqp-way nor the slightly cleaner Raku way.

augment class Regex {
    multi method ACCEPTS(Regex:D: Shell::Piping::Exitcode:D $ex) {
         CALLER::<$/> := $/;


At least not in the given/when block. A simply say $ex ~~ def\s(\S+); in the global scope did work just fine. I even got an error message that \$ does not exist in OUTER. Given that it should exist in every block by definition, that was rather strange.

We can inspect the lexical scope of the caller with the following construct.

say CALLER::.keys;

This will output the lexicals and how many stack frames there are. And indeed, given/when does introduce an additional stack frame that contains $_ but not $/ (and a few other bits and bobs). Since CALLER is a Stash, what is turn halve a Hash we can use :exists as usual and add another caller.

augment class Regex {
    multi method ACCEPTS(Regex:D: Shell::Piping::Exitcode:D $ex) {
        CALLER::<$/>:exists ?? (CALLER::<$/> := $/) !! (CALLER::CALLER::<$/> := $/);


Now it works as envisioned.

However – actually HOWEVER – I am augmenting a buildin class. That is risky. I am not alone. We got an ircbot that can grep in the source codes of the ecosystem.

20:01 < gfldex> greppable6: augment class
20:01 < greppable6> gfldex, 49 lines, 26 modules:

When we write modules we introduce scopes where our custom names reside in. The user of a module can decide to import those names into a scope under the users control. When we augment we introduce a new name into a globalish scope. What happens if two modules have the same idea? If the injected method is actually a multi then it will most likely work. But it does not have to. When two or more multi candidates have the same precedence, the first one found will win. By using augment on a non-multi we got a chance to get an error message. If we add a method via MOP we wont. If I want to allow smart match in a when statement or block, I need to provide the ability to have my custom class on the LHS of ~~. So there is no way around augment. I would even go so far as to say that this is the reason augment was added to the design of the language. It gets worse when we consider Raku to get less young and the language version keeps increasing. A module with sloppy tests might collide with a new method added with a language release. Should we enforce a use statement with a language version in a compunit with an augment statement?

This is really bothering me. We can of cause use META6 and add the field augments:"Cool,Int,Regex". That way zef would have a chance to spot collisions and provide a warning. Sadly, there is no way to enforce this (becaue of EVAL). I will spend some more time thinking about this and might start a problem solving issue.

Categories: Raku

Handling Failure

July 27, 2020 1 comment

After some back and forth I have found a practical way to handle error conditions in Shell::Piping. The practical way is to have more then one way. A process does have an exitinteger (called a code, because it can be quite cryptic) and text output to STDERR to indicate something went wrong. Sometimes we need sloppy error handling, sometimes we need to look into the textual output and react to it.

I found a really nice way to use a Junction and Rakus type system to remove some boilerplate from error handling. Combining both allows us to create a flexible type.

class Exitcode {
    has $.value;
    has $.command;
    method Numeric { $.value }
    method Str { $.command }

So this class produces objects that are both a number and a text. What is actually looked at depends on who is looking. We can use infix:<~~> to make the decision which comparison operator to use.

say $ex ~~ 42 && $ex ~~ ‚find‘; # OUTPUT: True

That’s still quite wordy. We can use a Junction because it binds tighter then ~~.

say $ex ~~ 42 & ‚find‘; # OUTPUT: True

Now we can CATCH an Exception and narrow done the command in a pipe that failed easily.

    when X::Shell::NonZeroExitcode {
        given .exitcode {
            when 42 & ‚find‘ {
                warn ‚Oh now! We found the answer!‘;

Not all users of a module might like to use Exceptions. So we use a construct in a Shell::Pipe object to create a Failure to return from .sink. If the method Shell::Pipe.exitcode is called, we assume the user is dealing with them by hand. We can then call .handled to “abort” the Exception. This has to be easy or it might get skipped. Hence the unusual usage of the coercer methods in the class Exitcode.

Categories: Raku

Mimicing quotes

July 24, 2020 2 comments

My quest to shorten code is coming along nicely. One thing that is still quite wordy is object creation. To fill a pipe with life we need instances of Proc::Async.

my $find = </usr/bin/find /tmp>;

As I showed earlier operators are deboilerplaters. We will need that argument list that is currently going to .new. It would be nice to reduce anything else to just two letters. Let’s define a sub prefix to do so.

sub prefix:<px>(List:D \l) is looser(&infix:<,>) { say ‚ <‘, l.Str, ‚>‘; };

px</usr/bin/find /tmp>; # OUTPUT </usr/bin/find /tmp>
px ‚/usr/bin/find‘, ‚/tmp‘; # OUTPUT </usr/bin/find /tmp>

By using is looser(&infix:<,>) we tell the compiler to create a List and then call the sub called px with a funny syntax. By doing so we mimic a quote construct that is used to implement qx and are able to leave a space out. There is a catch though.

px«/usr/bin/find /tmp»;

# ===SORRY!=== Error while compiling /home/dex/projects /raku/lib/raku-shell-piping/EVAL_0
# Two terms in a row
# at /home/dex/projects/raku/lib/raku-shell-piping/EVAL_0:1
# ------> px«/usr/⏏bin/find /tmp;
#     expecting any of:
#         infix
#         infix stopper
#         statement end
#         statement modifier
#         statement modifier loop

The grammar seams to be confused here. Since there is an alternative to list quotes with interpolation, I shall bugrepot and soldier on.

My nagging via R#3799 has been fruitful and provided a solution for this case.

class C {
    has $.state is rw;

my \px =;

multi postcircumfix:<{ }>(C:D, $s, Bool :$adverb = False) { $s;

multi postcircumfix:<{ }>(C:D, @a, Bool :$adverb = False) { @a;

multi infix:<|»>(Proc::Async:D $l, Proc::Async:D $r, :$different-adverb = "non-given") {
    dd $l;
    dd $r;

my $a = 42;
px<ls 1 2 3 $a>;
px«ls $a»;
px«ls $a»:adverb;

px{'foo' ~ 41.succ};

px«ls $a»:adverb |» (px«sort»:adverb) :different-adverb(42);

This is providing all I need to mimic qx in most of it various forms. The instance of C is used as a placeholder for the compiler to hold onto. It can have state that is global to all calls to postcircumfix:<{ }>(C:D, ...). This might came in handy later.

Shell::Piping has caused 3 bug reports so far. I do feel like I’m the first to tread this swamp. If I don’t make it, please hug my friends and delete my browser history.

Categories: Raku


July 18, 2020 2 comments

I agree with Damian that envy is clearly a virtue. We should add being boastful to that list. What good does it that we can make easy things easy without much efford, if we never tell anybody? Hence this blog.

While working on Shell::Piping I realised that many languages use operators or operator overloading to get rid of plenty of boilerplate. Please allow me to boastfully illustrate.

my $find ='/usr/bin/find', '/tmp');
my $sort ='/usr/bin/sort', :w);

$find.stdout.lines.tap: -> $l {
    once await $sort.ready;
    $sort.write: „$l\n“.encode if $l ~~ /a/;
my $handle-sort-output = start { put $sort.stdout.lines.join(„\n“); }
my $sort-started = $sort.start;
    await $find.start;
    CATCH { default { } }

await $handle-sort-output;

So we basically find /tmp | grep "a" | sort. Sort is a bit unusual as it waits for its STDIN to be closed before it actually starts to do anything. We don’t use grep but do the filtering ourselves. If we wouldn’t we could just shell-out and save us the bother. I found a way to do the same with a little less code.

$find |> -> $l { $l ~~ /a/ ?? $l !! Nil } |> $sort :quiet;

Looking at the operators of Raku that pattern can be found all over the place. Especially the hyper operators replace a loop and/or a chain of method calls with a single expression. The same goes for subscripts.

my %a = flat (1..12) Z <January February March April May June July August September October November December>;
say %a<3 5 7>;
# OUTPUT: (March May July)

Here the postcircumfix operator iterates over 3 5 7 and for each element calls .AT-KEY on %a. The result is returned as a list of the produced values. If we would do that by hand, we would be fairly quickly by 6 lines of code.

An instance of Proc::Async can only run once. When shell scripting in Raku that might become a burden. I need a way to declare a routine (or something that behaves like one) that will create objects from the same set of arguments. My goal was the following.

Shell::<&ls> = px<ls -l>;
Shell::<&grep> = px<grep>;
Shell::ls |> Shell::grep('a');

This would be quite easy to achieve if I didn’t step on a bug. So for now there is one extra space needed in px <grep>. While I was on it I added some error checking in px. It makes little sense to try to start a shell command if the file in question is not there nor executable.

Simple error handling however was easy to add because I rely on an infix to build the pipe. In contrast to postcircumfix they are well supported by Rakudo.

multi sub handle-stderr(|) { };
multi sub handle-stderr(0, $line) { say „ERR stream find: $line“ };
$find |> $errorer |> $sort :done({ say .exitcode if .exitcode }) :stderr(&handle-stderr);

The adverb :stderr registers a callback with the Shell::Pipe that is called with lines of any STDERR of the members of the pipe. As a first argument that callback receives the position of the command that produced that line. By using a multi we can offload the selection of the correct handler to the compiler. A single | in a signature declares a default candidate that will catch all otherwise unhandled outputs to STDERR. The operator doesn’t really do much here.

my multi infix:«|>»(Shell::Pipe:D $pipe where $pipe.pipees.tail ~~ Shell::Pipe::BlockContainer, Proc::Async:D $in, :&done? = Code, :$stderr? = CodeOrChannel, Bool :$quiet    ?) {
     $pipe.done = &done;
     $pipe.stderr = $stder;
     $pipe.quiet = $quiet;

It just sets a public attribute of the Shell::Pipe object to the provided callback. So the pattern that I’m using here is actually quite simple. Use an infix to turn two operands into an intermediary type. If more infix are used on that type, add to its state. Adverbs are used to trigger optional behaviour. The compiler will then call .sink on that intermediary to set anything in motion. As the first example in this blog post shows that motion can actually be quite elaborate. Yet we can hide that behind very cursory syntax by defining a custom operator.

I managed to implement anything I need to start turning the whole thing into a module. Automatic testing of that module will be a bit challenging. Luckily Raku is well suited to test crashing as jnthn kindly pointed out.

Judging by the weeklies we are producing more blog posts then ever. I welcome that move. We have no reason to be modest about Raku. Go forth and show off!

Categories: Raku