Tripping over variables

I was wondering where lizmat gets the info for changed modules from. She kindly answered with a link. I learned that updates to modules only show up, when we put them on CPAN. Since most modules are hosted on github, changing a module there does not mean that the world will be informed. I believe a better way to do that would be to fetch the ecosystems (we got two) once a week and check if version in any META6.json has changed.

Anyway, the reason I started this post is the documentation for FixedInt. It reads:

One major caveat to be aware of when using this module. The class instance may not be instantiated in a $ sigiled variable.

Raku $ sigiled scalar variables do not implement a STORE method, but instead do direct assignment; and there doesn’t seem to be any easy way to override that behaviour.

An implication of that is that classes that do implement a STORE method can not be held in a $ sigiled variable. (Well, they can, they just won’t work correctly. The first time you try to store a new value, the entire class instance will be overwritten and disappear.)

That is not true.

class Changing {
    has $!var handles <Str gist raku> is default(Nil);
    method STORE(\v) { note 'storing'; $!var = v }
    method FETCH { note 'fetching'; $!var }
}

constant term:<$a> := Changing.new;

$a = 42;
put $a;
# OUTPUT: storing
          42

The problem here is that the docs talk about variables while Raku don’t got any. It got containers with mutable content and values which are immutable. The language also got symbols that we can actually point at in source code. (Values we can point at in source code are called literals.) In the example above I created a symbol that looks like a variable but is a “reference” to a value of type Changing. The assignment operator can not be overloaded so we can protect immutable values. We can implement the method STORE instead. In fact we must, because there is no container in-between the symbol $a and the instance of Changing. (We get X::Assignment::RO if we try to assign without a STORE.) Since Rakudo does not recognise Changing as a container, it will refuse to call FETCH.

Thundergnat wrote a neat module with very little effort. Quite useful to do calculations with integers of fixed bit size.

my \fixedint = FixedInt.new(:8bit);

say fixedint; # 0

say fixedint -= 12;   # 244
say fixedint.signed;  # -12
say fixedint.bin;     # 0b11110100
say fixedint.hex;     # 0xF4

He achieved all that in just 36 lines of code. The trick is to force the user to bind and thus avoid the creation of a container while using STORE and FETCH to change the object in place. I doubt this is thread safe. Also the user of the module loses the ability to use some forms of quote interpolation and data dumper functions/modules will have less to display.

my \i = Int.new(10);
my $i = Int.new(10);

dd i;
dd $i;
# OUTPUT: 10
          Int $i = 10

We don’t have to define many operators to make custom types work because of plenty of meta-programming that is done in CORE. Many of those constructs assume immutable values. Autothreading is planned and will make the use of ». “interesting”. Thundergnat did not specify a language version for his module. The module itself is not hard to make safe. But – acutally BUT – this will change the interface for the user.

The flexibility of the language bites us here. Even thought the docs explain the difference between different sigils nobody is forced to read it. Also, nobody is forced to stick use v6.d at the beginning of a module. Please do so or the compiler wont be able to help you in the future. While naming immutable constructs quite often, the docs don’t explain why we use them. Concurrency and thus threading is very easy to add to a program. Testing it is hard.

I don’t have a solution to those problems but I’m pretty sure we need one or they will haunt us the next 100 years.

Exceptionally colourful

STDERR is often (ab)used for printing debug or status information. This can create clutter which in turn hides the important stuff. I want to print the essential stuff in exceptions in red unless a dynvar or environment variable is set.

class Explode is Exception {
    method message {
        put "$*dynvar is bad‼";
    }
}

sub e() {
    await start {
        Explode.new.throw;
    }
    CATCH { default { put .message } }
}

my $*dynvar = 'foo';
e();

# OUTPUT: foo is bad‼

We can access a dynvar inside the method of an exception from within an exception handler. In Shell::Piping error handling is a bit more involved. The biggest issue is fail because the enclosed exception is thrown by some routine in CORE about two steps down the call tree seen from the implicit or explicit MAIN sub. The dynvar is simply not there at this point in time. Luckily instances of Exception tend not to be long lived so we can get away with capturing the state of a dynvar at object creation. A good place to do so is TWEAK.

sub infix:<///>(\a, \b) is raw {
    my $dyn-name = a.VAR.name;
    my $has-outer-dynvar = CALLER::CALLERS::{$dyn-name}:exists;
    CALLER::{$dyn-name} = $has-outer-dynvar ?? CALLER::CALLERS::{$dyn-name} !! b
}

role Exception::Colored is Exception is export {
    has $.color;
    submethod TWEAK {
        my $*colored-exceptions /// on;
        $!color = $*colored-exceptions ~~ on && $env-color ?? 31 !! 0;
    }
    method RED($str) {
        $*ERR.t ?? ("\e[" ~ $.color ~ 'm' ~ $str ~ "\e[0m") !! $str
    }
}

Now I can use $.RED in .message of any exception that is Exception::Colored.

To have a look at the full stack was very helpful to figure out why the dynvar wasn’t there in some cases. For such cases I have a context sensitive binding in my .vimrc.

nmap <F1> :w<CR>:!raku -I ./lib %<CR>
imap <F1> <esc>:w<CR>:!raku --ll-exception -I ./lib %<CR>

In insert mode F1 will write the file and run Rakudo with an additional parameter. This results in a full stack trace.

foo failed
   at SETTING::src/core.c/Exception.pm6:62  (/usr/local/src/rakudo/install/share/perl6/runtime/CORE.c.setting.moarvm:throw)
 from SETTING::src/core.c/Failure.pm6:56  (/usr/local/src/rakudo/install/share/perl6/runtime/CORE.c.setting.moarvm:throw)
 from SETTING::src/core.c/Failure.pm6:111  (/usr/local/src/rakudo/install/share/perl6/runtime/CORE.c.setting.moarvm:sink)
 from /home/dex/tmp/tmp-2.raku:56  (<ephemeral file>:<unit>)
 from /home/dex/tmp/tmp-2.raku:1  (<ephemeral file>:<unit-outer>)
 from gen/moar/stage2/NQPHLL.nqp:1948  (/usr/local/src/rakudo/install/share/nqp/lib/NQPHLL.moarvm:eval)
 from gen/moar/stage2/NQPHLL.nqp:2153  (/usr/local/src/rakudo/install/share/nqp/lib/NQPHLL.moarvm:evalfiles)
 from gen/moar/stage2/NQPHLL.nqp:2113  (/usr/local/src/rakudo/install/share/nqp/lib/NQPHLL.moarvm:command_eval)
 from gen/moar/Compiler.nqp:60  (/usr/local/src/rakudo/install/share/perl6/lib/Perl6/Compiler.moarvm:command_eval)
 from gen/moar/stage2/NQPHLL.nqp:2038  (/usr/local/src/rakudo/install/share/nqp/lib/NQPHLL.moarvm:command_line)
 from gen/moar/rakudo.nqp:116  (/usr/local/src/rakudo/install/share/perl6/runtime/perl6.moarvm:MAIN)
 from gen/moar/rakudo.nqp:1  (/usr/local/src/rakudo/install/share/perl6/runtime/perl6.moarvm:<mainline>)
 from <unknown>:1  (/usr/local/src/rakudo/install/share/perl6/runtime/perl6.moarvm:<main>)
 from <unknown>:1  (/usr/local/src/rakudo/install/share/perl6/runtime/perl6.moarvm:<entry>)

As you can see there are quite a few things called before your script will be executed. Luckily Rakudo is implementing Raku in Raku so we have a chance to see what is going on.

Defined or dynvar

While adding dynvars to Shell::Piping to reduce the risk of finger injury I made a typo lizmat kindly corrected. She suggested to use the defined-or operator to test if a given dynamic variable is declared.

($*FOO // '') eq 'yes'

This is not equivalent to test if a dynvar was declared down the call tree. For that we need to check CALLERS.

say CALLERS::<$*colored-exceptions>:exists;
dd CALLERS::<$*colored-exceptions>;
# OUTPUT: False
#         Nil

In case the dynvar is declared we get a different result.

sub dyn-receiver {
    say CALLERS::<$*colored-exceptions>:exists;
    dd CALLERS::<$*colored-exceptions>;
}
my $*colored-exceptions;
dyn-receiver();
# OUTPUT : True
#          Any $*colored-exceptions = Any

For a module author that means we can have somebody sneak some undefined value into a dynvar we use, that has a type we don’t expect. Composebility is not the same thing as correctness. If we want do deal with this situation properly we need to check if the caller declared the dynvar and use a proper default value if they don’t.

class Switch {
    has $.name;
    method gist { $.name }
    method Str { die('invalid coersion') }
    method Bool { die('invalid coersion') }
}

constant on is export := Switch.new: :name<on>;
constant off is export := Switch.new: :name<off>;

sub dyn-receiver {
    my $*colored-exceptions = CALLERS::<$*colored-exceptions>:exists 
        ?? CALLERS::<$*colored-exceptions>
        !! off
}

In this example there are just two possible values but if there are more and they can be undefined we need to be more careful. However, this is quite a bit of typing. Can we use a deboilerplater here?

sub infix:<///>(\a, \b) is raw {
    my $dyn-name = a.VAR.name;
    my $has-outer-dynvar = CALLER::CALLERS::{$dyn-name}:exists;
    CALLER::{$dyn-name} = $has-outer-dynvar ?? CALLER::CALLERS::{$dyn-name} !! b
}

sub c {
    my $*colored-exceptions /// Int;
    dd $*colored-exceptions;
}

sub d {
    my $*colored-exceptions = Str;
    c();
}

c();
d();
# OUTPUT: Int $*colored-exceptions = Int
          Str $*colored-exceptions = Str

This operator takes two bound arguments. If we call it with a dynvar a contains the container that is the dynvar. We can query the name of that container and us it to check if down the call tree the dynvar was already declared. If so we use its value and assign it directly into the dynvar declared in c. Otherwise we assign b to the dynvar. In both cases we might return something naughty so we better do so raw.

Poking across the stack is risky. This could be done better with proper macros. I am quite sure we can do so after Christmas*.

*) For any value greater then Christmas last year.

Guarding dynamics

Dynamic variables are a nice way to get the benefits of global variables without taking the drawbacks. They pass information up the call tree without forcing a declaration upon the caller. However, dynvars share a burden with exceptions. The callee knows how to do what the caller might not expect.

use Module::By::Author::A;
use Module::By::Author::B;

my $*dynvar = 42;

sub-from-A(); # expects $*dynvar to be Int
sub-from-B(); # expects $*dynvar to be IO(Str)
sub-from-C(); # expects $*dynvar to be IO::Handle

In this example sub-from-B() will silently fail until it tries to open the file named “42”. While sub-from-C() will try to coerce 42 to become a file handle and throw. So there lies a danger in dynvars expected to be set by independent modules. Worse, the behaviour might suddenly pop up after any zef --install. Raku is a dynamic language that will try its best to convert values automatically and fail at runtime. It is good advice to support the compiler by failing early.

I want to use dynvars in a few modules to provide semi-global flags to remove boilerplate. The following seems to provide protection against unintentional dynvar-spill over.

class Switch is Mu {
    has $.name;
    method gist { $.name }
    method Str { die('invalid coersion') }
}

constant on = Switch.new: :name<on>;
constant off = Switch.new: :name<off>;

sub s() {
    # put $*d; # this will die
    say $*d;
    dd $*d;
}
my $*d = off;
s();
# OUTPUT: off
#         Switch $*d = Switch.new(name => "off")

I derive from Mu to get rid of all the coercers from Cool and overload Str to take that one out of the loop as well. Using say for anything but debugging is a bug anyway, so I might support it properly with .gist. Since I create a type with the class I can then go and protect my subs and methods with a whereception.

sub dyn-var-typecheck(Mu:U \T, $name) {
    sub {
        DYNAMIC::($name) ~~ T || die(„$name is set to an unexpected value“)
    }
}

constant &dyn-s-typecheck = dyn-var-typecheck(Switch, ‚$*d‘);

sub s($? where dyn-s-typecheck) { }
# OUTPUT: $*d is set to an unexpected value
#         in sub  at /home/dex/projects/blog/shielded-dynvars.raku line 6
#         in sub s at /home/dex/projects/blog/shielded-dynvars.raku line 22
#         in block <unit> at /home/dex/projects/blog/shielded-dynvars.raku line 29

Using the DYNAMIC::($name) in a where clause will slow down any call on MMD. So pulling the check into the sub might be reasonable.

With this measure I feel better to add $*always-capture-stderr to Shell::Piping to get rid of :stderr(Capture) on pretty much any pipe I start. And I feel a lot better when adding $*debug in everywhere.

Raku was not designed with type checks on dynvars in mind. It was designed to be a Perl. That gives us the flexibility to fix problems as they occur.

Whereceptions

I have a sub that takes a file and tries to guard itself against a file that does not exist. Where clauses don’t make good error messages.

sub run-test(IO() $file where .e & .f) { };
run-test('not-there.txt');
# OUTPUT:
# Constraint type check failed in binding to parameter '$file'; expected anonymous constraint to be met but got IO::Path (IO::Path.new("not-th...)

The signature of that sub is quite expressive. Often we don’t have time to read code to hunt down mistakes. That’s why bad error messages are LTA. We can extend any where clause with a die or fail to provide a better message.

sub run-test(IO() $file where .e && .f || fail("file $_ not found")) { };
run-test('not-there.txt');
# OUTPUT:
# file not-there.txt not found

We can also throw exceptions of cause. With just one parameter the signature has gotten quite long already. Also, when working with many files we will write the same code over and over again. Since we don’t code in Java we would rather not to.

my &it-is-a-file = -> IO() $_ {
    .e && .f || fail (X::IO::FileNotFound.new(:path(.Str)))
}
sub run-test(IO::Path $file where &it-is-a-file) { }

That’s much better. Since exceptions are classes we can reuse code amongst them. Like coloured output and checking for broken symlinks.

class X::Whereception is Exception is export {
    has $.stale-symlink is rw;
    method is-dangling-symlink {
        $.stale-symlink = do with $.path { .IO.l & !.IO.e };
    }
}

class X::IO::FileNotFound is X::Whereception is export {
    has $.path;
    method message {
        RED $.is-dangling-symlink ?? „The file ⟨$.path⟩ is a dangling symlink.“ !! „The file ⟨$.path⟩ was not found.“
    }
}

I have added a few to Shell::Piping. Suggestions what else to check for are very welcome.

Dropin replacement

Today I learned that whereis can take multiple commands to look for in $PATH.

$ whereis not-there raku-test-all zef
not-there:
raku-test-all: /home/dex/bin/raku-test-all
zef: /usr/local/src/rakudo/install/share/perl6/site/bin/zef

The results are always in the requested order. So we can use a shell spell to find a candidate to be used as our tester in a Makefile.

TESTER := $(shell whereis raku-test-all zef | cut -d ' ' -f 2 -s | head -n 1)

install-deps:
        zef --depsonly install .

test: install-deps
        $(TESTER) --verbose test .

install:
        zef install .

all: test

push: test
        git push

I had to do a few changes to raku-test-all so it mimics the interface of zef. The idea behind this Makefile is to be able to hit F6 and have all tests run and then push to github. Our current ecosystem simply distributes links to github repos. As a result pushing without testing can lead to somebody else cloning a broken repo (given the timing is bad enough). As you might imagine any speed up to testing is very welcome. Travis is quite slow. I think I should work in that area next.

Wrapping Exceptions

As stated before I would like to reduce simple error handling in Shell::Piping. The following is just way to long for putting a single line into a terminal.

CATCH {
    when X::Shell::CommandNotFound {
        when .cmd ~~ ‚commonmarker‘ {
            put ‚Please install commonmarker with `gem install commonmarker`.‘;
            exit 2;
        }
        default {
            .rethrow;
        }
    }
}

We have a conditional in line 3 and a print statement in line 4. The rest is boilderplate. The shortest syntax I came up with looks like the following.

X::Shell::CommandNotFound.refine({.cmd eq ‚commonmarker‘}, {„Please install ⟨{.cmd}⟩ in ⟨{.path}⟩.“});
X::Shell::CommandNotFound.refine({.cmd eq ‚raku‘}, {„Please run `apt install rakudo`.“});

I want to modify the type object behind CommandNotFound because the module will create instances of them and throw them before I can intercept them. Of cause I could catch them. But the whole excercise is done to get rid of the CATCH block. Adding a method to any exception both inside a module, and thus at compile time, and at runtime would be nice. We can do so with a trait.

multi sub trait_mod:<is>(Exception:U $ex, :$refineable) {
    $ex.HOW.add_method($ex, ‚refine‘, my method (&cond, &message) {
        $ex.WHO::<@refinements>.append: &cond, &message;
        state $wrapped;
        $ex.HOW.find_method($ex, ‚message‘).wrap(my method {
            my @refinements := self.WHO::<@refinements>;
            for @refinements -> &cond, &message {
                if cond(self) {
                    return message(self);
                }
            }
            nextsame
        }) unless $wrapped;
        $wrapped = True;
    });
}

A trait is a fancy sub that takes type objects or other stat-ish (Raku is a dynamic language without static things) objects and does stuff with it. That can be wrapping – a MOP operation in disguise – or actual MOP-calls. This redefining trait adds a class attribute named @refinements via autovification on the package-part of the type. This is hidden behind the pseudo method .WHO. Then it adds a method that takes a pair of Callables and stores them in @refinements. The type object’s original .message method is wrapped unless already done so. This is controlled by a state variable.

The wrapped .message will call the first callable and expects a Bool in return. If that one is True is will call the 2nd callable. Both are called with an exception instance.

The trait does not .^compose the type object it is called on. This is (for now) needed because of the following case.

class X::Shell::CommandNotFound is Exception is refineable { ... }

The trait is actually called on Exception. That works because the added method .refine is not shadowed by anything. We can therefore use this method to wait for it to be called to finish what we need to do after compile time. The wrapper has a fall through via nextsame if no condition has matched.

I don’t think I got all corner cases covered yet. That will need further testing. The user facing syntax can stay the way it is because traits are deboilerplaters. The trait itself is likely going to get more complex. But that is fine. The burden is supposed to be on the side of the implementer.