We can do more

DataKinds proceeds with his educational translations of the way of the Python to the way of the butterfly. His quest is to show how to do stuff in Raku that is done in Python. As always TIMTOWTDI applies.

Looking at the examples one can see that Raku lacks nothing Python can provide. That’s nice to know but why would you want to switch if you can’t do more? So lets explore what Raku can do better.

For a Positional container we want to switch each even element with each uneven element. Flip element 1 with 2, then 3 with 4 and so on. We also want to provide a sensible error message for the case where this is not possible. Rakudo is providing and using the X::Parameter namespace. So that’s a natural place for our custom exception.

class X::Parameter::UnevenElementList is Exception {
    method message {
        'Positional with uneven elements in where clause.'
    }
}

Now we need a sub that takes one Positional in a positional parameter and does a type check on it. We shall move that type check into an aptly named sub. We don’t need to reduce readability if we don’t have to. Using a Callable container in a where clause is not in the docs. I will check Roast later and file an issue if needed.

sub swap-each-pair(@a is raw where &check-for-even-elements ) {
    sub check-for-even-elements {
        .elems %% 2
        || X::Parameter::UnevenElementList.new.throw
    }

    for @a <-> $a, $b {
        ($a, $b) = ($b, $a)
    }
}

my @l = 1..6;
@l.&swap-each-pair;
say @l;

I put the where clause sub into the main sub to not pollute the global namespace of my script. In a module we could stick it outside and reuse it. The argument Array is marked as is raw. That’s a bit redundant because @-sigiled arguments are call-by-reference anyway. By doing so I make it clear that this sub is meant to be a mutator. If you are a Haskell lubber, you may want to look away now. The <-> pointer is applying is rw to all arguments. That allows us to modify elements of @a with a destructuring assignment.

Using a good name for the where clause sub and throwing an exception we get a helpful error message with a stacktrace. We must do so or fear the LTA tag.

Positional with uneven elements in where clause.
  in sub check-for-even-elements at /home/dex/tmp/tmp.raku line 28
  in sub swap-each-pair at /home/dex/tmp/tmp.raku line 26
  in block <unit> at /home/dex/tmp/tmp.raku line 38

Both languages are late bound and don’t have strict type checks. In a large code base that can bite you. In that case it can be very helpful to do type and value checks on strategic points and provide error messages that actually help with debugging.

Raku got lots of features that make the life of module authors easier who want to make the life of module users easier. One might think that the language designers have written CPAN modules before.

A Raku riddle

While thinking about DataKinds blog post I came up with a most peculiar riddle. It goes as follows:

Name something that when removed is still defined but never existed.

The answer is an Array element whereby the Array got default values.

my @a is default('fill') = 1,2,3;
say @a[5]:exists; # False
@a = @a[5]:delete;
say @a[5].defined; # True
say @a[5]:exists; # False

How many elements got such an Array and when will iteration end?

say @a.elems; # 3
.print for @a; # 123

It has 3 elements and we can iterate over them. What makes perfect sense. However, if we judge by definedness for looping it will be infinite.

while @a[$++] -> $_ { .say } # will not stop

Is this an infinite list then? It can not run out of elements so if we believe the docs about infix:<Z> it should never stop the zipping.

use v6;

my @a is default('‽') = 1,2,3;
my @b = <a b c d e>;

say @a Z @b;
# OUTPUT: ((1 a) (2 b) (3 c))

I tried to convince Z to behave as desired to no avail. In the process I learned how to set the default value of a container long after its declaration.

my @a := Array.new;
{
    use nqp;
    my $descriptor = nqp::getattr(@a.VAR, Array, '$!descriptor');
    my $a = $descriptor.set_default('‽');
}
say @a[1];
# OUTPUT: ‽

Rakudo does a lot with iterators. Since they are an implementation detail, we can’t easily build our own. That’s not really a problem because the zip operator is clever enough to DWIM.

my @a = 1,2,3;
my @b = <a b c d e>;
say ((|@a, |('‽' xx *)) Z (|@b, |('‽' xx *)))[^(@a.elems max @b.elems)];
# OUTPUT: ((1 a) (2 b) (3 c) (‽ d) (‽ e))

I believe that is what DataKinds was looking for.

wat-freeness

Sadly I can’t attend the conference in the cloud because I lend my wings to an angel and the sucker didn’t bring them back. But there is nothing – not even a plague – that can stop me from writing a blog post.

Conferences tend to be a time of reflection and mug throwing. I aim to do so with this post. Most of my work on Raku went into the docs and this blog (please note the recursive reflection). This was a very selfish act. Without a documentation that does not have holes, learning a language can be quite tricky. The blog helped me a great deal to do the thinking that was required for hole plugging. It is a great tool for contemplation and creating awareness of new but not young language. I can strongly recommend to use a blog for both. The hits on this particular blog went up by about 63% relative to last year. If we can keep up that grown it will take only 54 years for the interest in Raku to exceed the population of the earth.

In my last post I struggled to understand the nature of a not-bug. It took a good nights sleep to figure out why. The reason seems to be that I didn’t have a mental category to put that thing it. To my delight wat turns into culture. In the context of programming a wat is not undefined behaviour. It is not a violation of syntax or semantics. It it an emergent interaction between different subsystems of a language implementation that violates the principle of least surprise. An LTA is a subset of wat. Let’s have a bit of code.

class C { has $.a = s; }
sub s {}
# OUTPUT: ===SORRY!===
#         Null regex not allowed
#         ------> class C { has $.a = s; ⏏}; sub s {}
#         Couldn't find terminator ;
#         ------> class C { has $.a = s; ⏏}; sub s {}
#             expecting any of:
#                 ;

wat.

If we rename the sub we can make this work.

class C { has $.a = a; };
sub a { 42 }
say C.new.a;
# OUTPUT: 42

What happens here is that the Raku grammar is looking for s/// instead of making a mental note to check for the existence of a sub called s at the end of the compilation unit. At first I thought it to be an LTA. So wats can be really deceiving. As such any language implementor should avoid them where ever possible. With Raku we aim to do so.

Do we tell newcomers? Or to phrase this question differently: Do we still suck at marketing? Let’s have a look at raku.org.

The Raku Programming Language

Hi, my name is Camelia. I’m the spokesbug for Raku. Raku intends to carry forward the high ideals of the Perl community. Raku has been developed by a team of dedicated and enthusiastic volunteers, and continues to be developed. You can help too. The only requirement is that you know how to be nice to all kinds of people (and butterflies). Go to #raku (irc.freenode.net) and someone will be glad to help you get started.

This is astoundingly bad. The heading doesn’t fit to the text right underneath it. It explains Camelia not what Raku is. Granted Camelia, although quite pretty, desperately needs explaining. The author of that paragraph is clearly a language implementor addressing other language implementors. The high ideals of the Perl community are not named nor are the high ideals of the language implementors. Fishing for grants is quite important, just not at the front page to the programming language. With this front page we manage to miss the audience in the very first lines one who is (rightfully) interested in the language will read. This needs to change.

Good marketing provides a benefit to all parties involved. We need to find out what we want the world to know about Raku and put it on that page. Much work goes into the weekly news. Big thanks to lizmat and her occasional replacements. Yet, why is this very helpful content not on raku.org? We link to irc-logs. We are bloody coders that have a really nice language with a really nice web framework. Do we use it to provide a community we want to grow with a hub to get what they need? I think we should.

Raku was meant as a rewrite of the language by the community and of the community. The language and its compiler are in a good place already. If we want it to become a replacement for all dynamic languages, we need to up our game in the marketing department. Changing the name of the language was the easy part of this endeavour.

I hope you, the kind reader of this blog, can make something good of my ramblings. Maybe even take some of it to a conference. In fact, if you read this blog post while attending the conference of strange times, I will consider myself hugged.

It felt like a bug

IO::Path provides quite a few bits of information about a file. What is missing is it’s true type. File endings can be deceiving. On operation systems that sport file we can get a clear picture. Shelling to get the mimetype with file -b -i is a bit slow so I tried to add a .hyper.map on a list of filenames and mixed in a role to IO::Path. I stepped on something that felt like a bug but wasn’t. Following is the golfed code to illustrade the probem.

my @list = 1..10;
my @results;
@results = @list[^24].hyper(:degree(2), :batch(1)).map({
    my $type = run('/bin/ls', :out).out.slurp.chars;
    $_ but role Mime { has $.mime = $type }
});

Rakudo seems to try to form a closure with the role but frees the data that is stored somewhere. This is only noticible when threading slows things down. It is a rare case where the syntax is valid but the semantics are wrong. So the grammar that is Raku can parse it but the compiler can`t do anything usefull with that match. It somehow speaks highly of the compiler to handle this case without crashing. Yet, a warning would be nice. As jnthn pointed out, this is not the right way to use a role.

Of cause there is a proper way to handle such a case. We need to define a role with a parameter that we can use as a default to our attribute.

sub get-filetype(IO() $_){ run('/usr/bin/file', '-b', '-i', .basename, :out).out.slurp }
role Mime[$type] { has $.mime = $type }
@results = @list[^24].hyper(:degree(2), :batch(1)).map({
    my $type = run('/bin/ls', :out).out.slurp.chars;
    $_ but Mime[$type]
});

I did not manage to use an anonymous parametric role. We can define one, store it in scalar but can’t instantiate it. Using the subscript [] on a scalar (and many other things) will tell the grammar to assume we want to work with a Positional.

my $R = role :: [$t] {};
dd $R;
dd $R[42];
# OUTOUT: <anon|10> $R = <anon|10>
#         Could not instantiate role '<anon|10>':
#         Too few positionals passed; expected 2 arguments but got 1

There is a way to use the MOP to do so. I did find the right method in Metamodel::ParametricRoleHOW but failed to figure out how to use it. (If you do know how to use .^parameterize please provide a code snippet.)

Not really closure forming roles seem to be a case where the grammar can handle it but the compiler doesn’t. Which leaves the question how one is supposed to document a thing that should not exist. It is a trap and we have a section for those. But this is placed pretty far from the description of roles. The first two pull requests are written and more will need to follow.

I’m glad that the docs are written in a language that supports hyperlinks. It would be pretty difficult to describe the network that is Raku in a linear thing like a book.

Swarms of Exceptions

According to Larry lazyness is a virtue. Let’s see if Rakudo agrees.

sub virtue { }; lazy virtue;

# OUTPUT: Nil

A few posts ago I lamented about the lack of Exceptions thrown by run, shell and qx{}. I since changed my mind. Neither of those constructs have any idea what OS they are running on or what a shell command even is. The programmer does (hubris).

This left me impatient. The compiler refuses to create an exception for me when Proc.exitcode is non-zero. In Raku that’s a solvable problem.

my $Exception = Metamodel::ClassHOW.new_type(:name($cmd-name));
$Exception.HOW.add_parent($Exception, Exception);
$Exception.HOW.add_attribute($Exception, Attribute.new(:name<$.exitcode>, :type(Int), :package($Exception), :has_accessor));
$Exception.HOW.add_method($Exception, 'message', my method { 'Shell command ⟨' ~ self.^name ~ '⟩ finished with non-zero exitcode of ' ~ self.exitcode ~ '.' } );

$Exception.HOW.compose($Exception);

We use the MOP to create a type at runtime and fill it with attributes and the message method that all Exceptions need. To be able to use this new type in the rest of the program, we need to have that runtime to happen before the rest of the program is compiled. To do so we can use a BEGIN phaser. We can then stuff the new type into an appropriate package.

BEGIN {
    package X::Shell {}
    X::Shell::<ls> := $Exception;
}

As we have access to OUR at BEGIN-time we can also add subs to the script’s scope that can be used in the rest of the program or add our shell commands to a dedicated package to avoid conflicts.

Shell::«$cmd-name» := sub (|c) {
    my $proc = run $cmd-name, |c, :out, :err;
    my $e = X::Shell::«$cmd-name»;
    $e.new(:exitcode($proc.exitcode), :stderr($proc.err.slurp)).throw if $proc.exitcode != 0;

    $proc.out.slurp;
}

Since we put exceptions into X::Shell we can handle exceptions for all shell commands or just a specific one.

Shell::ls('/home/not there', '-l');

CATCH {
    when X::Shell::ls { warn .stderr, .backtrace }
    when X::Shell { warn .^name, ': ', .message; .resume }
}
# OUTPUT: ls: cannot access '/home/not there': No such file or directory
#
#      in block  at exceptional-run.raku line 48

By capturing STDERR and stuffing it into an exception I can choose to expose the error message issued by the shell command to the user of my program and show a stracktrace if I like. The latter can help greatly with debugging. Since I’m really good at writing bugs that will come in handy.

The whole example script can be found here.

Dynamic languages come with a noticeable speed penalty. When we execute them. When we write them we can progress much faster.

The Topic is the topic

I lamented in my last post that we got a lack of beginners material. The reason I gave is that writing such guides is hard. Which leaves the question why it is that hard. While plugging holes in the docs I found that many features of the language pop up in many places. Raku is intertwined with itself. For a beginner you would want to start with something simple and than build on that until you reach maximum complexity. What could be more simple then then humble if statement? A lot, actually. And the reason is the Topic. This shall be the topic of this post.

As one can expect the Topic shows up in Raku in many places. The uninitiated might even think it apears like magic. So lets add a magic variable to a Callable.

my $magic = 42;
sub s( $hmm? is raw = OUTER::<$magic>) { $magic = $magic == 42 ?? 'answer' !! 'nope' }
s;
say $magic;

# OUTPUT: answer

That’s exactly how a Block gets handed over the Topic.

my &block = {;}
say &block.signature;

# OUTPUT: (;; $_? is raw = OUTER::<$_>)

Many language elements in Raku set the topic by default. If does so only when we ask for it.

if 42 { $_.say }
if 42 -> $_ { $_.say }

# OUTPUT: (Any)
#         42

How does the compiler know the difference?

my &pointy = -> $_ {;};
say &pointy.signature;

# OUTPUT: ($_)

The explicit Parameter in a pointy block is not optional. We can use arity and count to tell them apart.

say [&block.signature.arity, &pointy.signature.arity];
say [&block.signature.count, &pointy.signature.count];

# OUTPUT: [0 1]
#         [1 1]

There is another way how to change the arity of a block.

&block = { $^a.say };
say [&block.signature, &block.signature.arity];
if 42 { say $^a }

# OUTPUT: [($a) 1]
#         42

Still not conviced the compiler doesn’t cheat?

if 42 { say $^_ }

# OUTPUT:
# Redeclaration of symbol '$^_' as a placeholder parameter
# at /home/dex/projects/blog/topic-is-topic.raku:69
# ------> if 42 { say $^_⏏ }

Which doesn’t mean the compiler wont cheat. It’s the compiler after all. But this is one of those places in Raku where things fall into place. By having a language called Signature to describe parameters of Callables and access to the scope no magic is required to implement magic variables.

This blog post was sparked by a hole in the docs. By writing this post I spotted another one. We forgot the mention that the literal for an empty block is {;}. All we would need to do to find all holes is going through Roast and see if anything mentioned there is also in the docs. Sadly I got other things to do in the next 900 days or so.

But enough moaning. Let’s end with a positive conclusion. Raku may look like magic. It isn’t. It’s just very clever indeed.

Replacement similarities

In my last post I stated that hits on posts of my blog went up by 20% compared to last year. The reason I checked is clearly vanity. The reason I checked on that day is an article about the increasing popularity of Rust. As a spiritual exercise I turned the different reasons given by programmers who ditch the language they use into a check list.

[X] genuinely new features
[X] rule of least surprice (mentioned often)
[ ] no footguns
[x] language designers listen to community
[x] complete tooling (mentioned often)
[X] central repo for libraries/modules
[ ] ideomatic code is fast
[X] FFI interface
[x] many platforms support
[X] memory sefety/type safety/data race safety
[ ] fast/low memory footprint (mentioned often)
[X] backward compatible to itself
[ ] proper error handling
[ ] does not force OOP

The items are from the article and apply to Rust. The ticks are where I see Raku excels to. A capital X is a strong match for us. The domain of the two languages are quite different. I strongly doubt anybody would like to write code for a microcontroller in Raku. Yet what programmers care about forms a sizeable overlap.

Another aspect of the Rust article is the general tenor of the participants of that questionary. All users come from a different language and where driven by pain to reach safer grounds. That means all of them are replacing $language with Rust and like it that way. The success of Rust lies as much in getting things right as in other languages getting things wrong. That was pretty much the goal for Perl 6 in the year 2000. We can tick that box too.

The core team of Rust sees a very strong open community but little resources for beginners. The same is still true for us. Maybe because writing beginners guides is really really hard. We have the same weakness. There are plenty of books for Raku already. I never really liked that idea. I can understand why a contributor with an academic background would want to write a book. You aren’t really an academic if you don’t. I can also see why academics don’t really have a problem with paywalls. And yes, books are paywalls. I used to climb that wall too. There is Stroustrup’s The C++ Programming Language in a bookshelf right behind me. Yet I doubt that a paywall is going to get us more users. A beginners guide must be online and to be reached by one click from raku.org.

There are also differences. Rust was meant to replace C/C++ as a tool to make security critical software at Mozilla. When Raku was release we didn’t really want it to replace anything. Not because there are plenty of dynamic languages out there that need replacing real quick in a hurry. Anybody here who loves PHP? Fancy another outtake of Node.js because a module gets pulled? No, we didn’t want to replace things because we want to be nice. The whole point of marketing is to be nice to the folk you want to reach and be nasty to your competitors. And it’s not like those Rust folk are actually good at marketing. When I read rust I think of old oil barrels, decaying machines and another unwelcome reason to wash my hands. Names seam not to matter all that much for programmers.

At the beginning of 2018 I gave a very brief motivational speech. I still stand by what I wrote back then. Yet I’m not happy with the success we had. Maybe it’s time to stop learning from the mistakes of others and have a look of what swiftly growing language communities are doing right.

Down the error rabbit hole

Good error messages are useful because they tell us what we don’t have to check when something went wrong. (see: LTA) I came to that conclusion by trying to replace Bash with Raku. Lets have a look at some innocent piece of code.

my $p = Proc.new;
$p.spawn(<does-not-exist>);
$p.sink;

CATCH {
    default { say .^name, ‚: ‘, .message; .resume }
}

# OUTPUT: «X::Proc::Unsuccessful: The spawned command 'does-not-exist' exited unsuccessfully (exit code: 1, signal: 0)␤»

This error message is wrong. A command that is to be run by exec and derivates can only produce a non-zero exit code if the executable or script is executable, for the OS user accessible, a normal file and not a directory. (There are more but those vary from OS to OS.) As the name of the command suggests it does not exist and therefor can’t produce an exitcode of 1. This bit me shelling out to brtfs to create a snapshot. Because brtfs is not a valid command. It’s a typo. When I finally realised that btrfs works much better 15 minutes had passed. Many unix commands produce different error codes depending on the arguments provided. So this kind of typo can be really confusing when heavy shell scripting is required.

The example above is the only way for Proc::* to produce an exception. That’s troublesome because the language user is not meant to create a Proc object by hand. The subs run, shell and qx{} will never use it in sink context. Instead they will evaluate to False when checked for errors. One can mimic shell scripting by chaining commands with && that way as done in many shell languages. The $shell however will check errno and state ‘command not found’ to aid its users. Rakudo does not do so. We can change that with a few lines of code.

class X::Proc::CommandNotFound is X::Proc::Unsuccessful {
    method message {
        my $symlink = $.proc.command[0].IO.l ?? ' (symlink)' !! '';
        "The command '{$.proc.command[0]}'$symlink was not found."
    }
}

sub use-proc-fatal {
    &run.wrap(-> |c {
        my Proc:D $ret := callsame;
        if $ret.exitcode > 0 {
            with $ret.command[0].IO {
                X::Proc::CommandNotFound.new(:proc($ret)).throw unless .e;
           }
            X::Proc::Unsuccessful.new(:proc($ret)).throw if $ret.exitcode > 0 || $ret.signal > 0;
        }
        $ret
    });
}
use-proc-fatal;
say ‚### run does-not-exist‘;

run 'does-not-exist';
CATCH {
    default { say .^name, ‚: ‘, .message; }
}

run does-not-exist
X::Proc::CommandNotFound: The command 'does-not-exist' was not found.

This is not a proper solution. Firstly Proc should never throw an exception. If at all Raku should fail because that composes better with //. In shell scripting it is quite normal that the execution of a command doesn’t work. Error handling is just part of the game. Since it is so common there should be more and better exceptions. I have a script where I tested a few more cases with a few more exceptions. Having those returned as Failure from run and shell would make qx stop failing silently.

In my backup script I have to wrap each command in a sub and throw exceptions to take advantage of CATCH and LEAVE blocks for error handling and cleanup (there are temporary directories/files and btrfs snapshots). I really wished to have a use Proc::Fatal in the core to help with this task. I believe that needs to be in core because errno is hidden inside Proc inside Proc::Async inside nqp::spawnprocasync inside MVM_proc_spawn_async where I stopped and climbed back up the rabbit hole. In fact I never found the bloody exec*. Also Bash stops being fun after about 50 lines of code. Easy things should be easy.

I’m quite sure I missed a lot of error modes. So this requires futher thought. If you have such please share. If you write a blog post about Raku right now you can expect 500 hits a week. That is 20% up from last year. We are better then the stock market! In a world where money can be printed but jobs can not that’s a real feat.