A picky caller

I have got myself a paradoxical harddrive by combining a fast ssd and a sizeable disk by setting up a a bcache on my linux box. Now I got a harddrive that is really fast with small files. There are a few stats bcache is providing via sysfs. To watch them one requires to read a few text files. A well suited task for slurp. I ended up with a bunch of pointy blocks that look like this:

-> $cache {
    slurp("/sys/fs/bcache/$cache/cache_available_percent").chomp ~ '%'
}

I put them together with a name into an Array to be able to loop over them. As it turned out there are two groups of pointies. The one group needs the name of the bcache-block device which holds the actualy filesystem. The other group gets the UUID of the cache group. I need to group the output of the two groups so I get:

bcache0:
         dirty data: 36.0k
ebc67019-9d50-4042-8080-b173e2ba802f:
         hit ratio: 62% 66% 50% 0%
         bypassed: 7.0G 2.4G 65.1M 9.1M
         cache available: 94%

I could have split up the array but figured that I can check the signature of the pointy instead to select what is being output.

for bcache-devs() -> $dev {
  with $dev {
    say BOLD $dev, ':';
    for @stats -> $name, &f {
      next unless &f.signature.params».name eq '$dev';          
      put "\t", $name, ': ', .&f
    }
  }
}

If the name of the positional doesn’t fit I just skip the output.

Next I tried to match the signature by adding subsets of Str to the signature of the pointes. Sadly matching a signature literal like so doesn’t work in this case.

subset Dev of Str;
say 'match' if &f.signature ~~ :(Dev $dev);

If I would define my one classes that would certainly work. It seems the sloppy matching of subsets is working as intended. A bit of a shame because subsets are so easy to set up. For my example just matching the parameter name is fine because it saves time when typing the pointies.

Nonetheless it’s really neat that the caller can has a say if it likes the signature of the callee in Perl 6.

Iterating past the finish

A while ago the question was raised how to augment Any. As it turns out the augmenting part is working but the newly added method is not propagated to children of buildin types. One can force the propagation by calling .compose on all type objects. Getting a list of all parents is done with .^mro and the check is done with ∈.

augment class Cool { method HTML { HTML.new(self) } }
if Cool ∈ T.HOW.mro(T) { T.HOW.compose(T); }

I stepped on a Dragon

The tricky part is to get all predefined classes. Mostly because there is a lot of stuff in CORE:: that doesn’t even implement the interfaces to call methods. We can call .DEFINITE because that’s not a method. So we can weed out all predefined objects and are left with type objects and stuff that’s leaking from NQP into Perl 6-land. Those beasties don’t implement .mro so by guarding with try we can bild a list of all Perl 6 type objects. Those type objects contain IterationEnd. Hence we can’t trust for or anything else that is using iterators to loop over a list. There is also Slip in the list. We can help that by using binding everywhere.

my @a = CORE::.values;
my @types;
for 0..^@a.elems -> $i {
  my \T := @a[$i];
  try @types[$++] := T if not T.DEFINITE;
}

for 0..^@types.elems -> $i {
  my \T := @types[$i];
  try if Cool ∈ T.HOW.mro(T) {
    T.HOW.compose(T);
  }
}

And there we have it. All children of Cool have been re-.composed.

It’s no magic!

There are a few things I learned. Firstly much of the magic behind Perl 6 are just type checks. Anything that deals with iteration of lists or similar constructs is checking for Slip or IterationEnd and branching out to deal with their special nature.

And secondly there are a lot of interfaces leaking into spec-land that have no business there. I’m worried that might bite us later because any useful interface will be abused by programmers sooner or later. I would prefer the inner workings of Rakudo to be well hidden.

I found a way to deal with agumenting build in types so it can’t be long before the core devs fix that bug.

Deconstructing Simple Grammars

Last year I wrote an egg timer that was parsing command line arguments similar to GNU sleep. I was happy with the stringent form of the parser as follows.

my Seconds $to-wait = @timicles»\
    .split(/<number>/, :v)\
    .map(-> [$,Rat(Any) $count, Str(Any) $unit] --> Seconds { %unit-multipliers{$unit} * $count })\
    .sum;

It does a few simple things and does them one after another. A grammar with an action class would be overkill. I wasn’t happy with using splits ability to return the needle with the parts. It certainly does not improve readability.

After quite a few iterations (and stepping on a bug), I came up with a way to use Str.match instead. If I convert each Match-object into a Hash I can use deconstruction in a signature of a pointy block.

my Seconds $to-wait = @timicles»\
    .match(/<number> <suffix>+/)».hash\ # the +-quatifier is a workaround
    .map(-> % ( Rat(Any) :$number, Str(Any) :$suffix ) { %unit-multipliers{$suffix} * $number })\
    .sum;

Instead of using positionals I can use named arguments that correspond to the named regexes inside the match arguments.

Even in such a small pice of code things fall into place. Hyper-method-calls get rid of simple loops. The well crafted buildin types allow signature deconstruction to actually work without loads of temporary variables. It’s almost as certain language designers where aiming to make a most elegant language.

Expensive Egg-Timers

If you use a CLI you might have done something along the line.

sleep 1m 30s; do-the-next-thing

I have a script called OK that will display a short text in a hopeful green and morse code O-K via the PC speaker. By doing so I turn my computer into an expensive egg-timer.

As of late I found myself waiting for longer periods of time and was missing a count-down so I could estimate how much more time I can waste playing computer games. The result is a program called count-down.

Since I wanted to mimic the behaviour of sleep as closely as possible I had a peek into its source-code. That made me realise how lucky I am to be allowed to use Perl 6. If I strip all the extra bits a count-down needs I’m at 33 lines of code compared to 154 lines of GNU sleep. The boilerplate I have is mostly for readability. Like defining a subset called Seconds and a Rexex called number.

Errors in the arguments to the script will be cought by the where clause in MAINs signature. Since there are no further multi candidates for MAIN that might interfere, the usage message will be displayed automatically if arguments are not recognized. Pretty much all lines in the C implementation deal with argument handling and the fact that they can’t trust their arguments until the last bit of handling is done. With a proper signature a Perl 6 Routine can fully trust its arguments and no further error handling is needed. Compared to the C version (that does a lot less) the code can be read linear from top to bottom and is much more expressive. After changing a few identifiers I didn’t feel the need for comments anymore. Even some unclear code like the splitting on numbers and keeping the values, becomes clear in the next lines where I sum up a list of seconds.

Now I can comfortably count down the rest of a year that was made much better by a made better Perl 6. I wish you all a happy 2018.

Racing Rakudo

In many racing sports telemetry plays a big role in getting faster.  Thanks to a torrent of commits by lizmat you can use telemetry now too!

perl6 -e 'use Telemetry; snapper(½); my @a = (‚aaaa‘..‚zzzz‘).pick(1000); say @a.sort.[*-1 / 2];'
zyzl
Telemetry Report of Process #30304 (2017-11-05T17:24:38Z)
No supervisor thread has been running
Number of Snapshots: 31
Initial Size:        93684 Kbytes
Total Time:          14.74 seconds
Total CPU Usage:     15.08 seconds

wallclock  util%  max-rss  gw      gtc  tw      ttc  aw      atc
   500951  53.81     8424
   500557  51.92     9240
   548677  52.15    12376
   506068  52.51      196
   500380  51.94     8976
   506552  51.74     9240
   500517  52.45     9240
   500482  52.33     9504
   506813  51.67     6864
   502634  51.63
   500520  51.78     6072
   500539  52.13     7128
   503437  52.29     7920
   500419  52.45     8976
   500544  51.89     8712
   500550  49.92     6864
   602948  49.71     8712
   500548  50.33
   500545  49.92      320
   500518  49.92
   500530  49.92
   500529  49.91
   500507  49.92
   506886  50.07
   500510  49.93     1848
   500488  49.93
   500511  49.93
   508389  49.94
   508510  51.27      264
    27636  58.33
--------- ------ -------- --- -------- --- -------- --- --------
 14738710  51.16   130876

Legend:
wallclock  Number of microseconds elapsed
    util%  Percentage of CPU utilization (0..100%)
  max-rss  Maximum resident set size (in Kbytes)
       gw  The number of general worker threads
      gtc  The number of tasks completed in general worker threads
       tw  The number of timer threads
      ttc  The number of tasks completed in timer threads
       aw  The number of affinity threads
      atc  The number of tasks completed in affinity threads

The snapper function takes an interval at which data is collected. On termination of the program the table above is shown.

The module comes with plenty of subs to collect the same data at hand and file your own report. What may be sensible in long running processes. Or you call the reporter sub by hand every now and then.

use Telemetry;

react {
    snapper;
    whenever Supply.interval(60) {
        say report;
    }
}

If the terminal wont cut it you can use http to fetch telemetry data.

Documentation isn’t finished nor is the module. So stay tuning for more data.

There Is More Than One Way At The Same Time

The Perl 6 Rosattacode section for parallel calculations is terribly outdated and missing all the goodies that where added or fixed in the last few weeks. With this post I would like to propose an updated version for Rosettacode. If you believe that I missed something feel free to comment below. Please keep in mind that Rosettacode is for showing off, not for being comprehensive.

use v6.d.PREVIEW;

 

Perl 6 provides parallel execution of code via threads. There are low level custructs that start a thread or safely pause execution.

my $t1 = Thread.start({ say [+] 1..10_000_000 });
my $t2 = Thread.start({ say [*] 1..10_000 });
$t1.finish;
$t2.finish;

my $l = Lock.new;
$l.lock;
$t1 = Thread.start: { $l.lock; say 'got the lock'; $l.unlock };
sleep 2; $l.unlock;

$t1.finish;

When processing lists, one can use a highlevel Iterator created by the methods hyper and race. The latter may return values out of order. Those Iterators will distribute the elements of the list to worker threads that are in turn assigned to OS level threads by Rakudos ThreadPoolScheduler. The whole construct will block until the last element is processed.

my @values = 1..100;

sub postfix:<!> (Int $n) { [*] 1..$n }

say [+] @values.hyper.map( -> $i { print '.' if $i %% 100; $i!.chars });

For for-lovers there are the race for and hyper for keyword for distributing work over threads in the same way as their respective methods forms.

race for 1..100 {
    say .Str; # There be out of order dragons!
}

my @a = do hyper for 1..100 {
   .Int! # Here be thread dragons!
}

say [+] @a;

Perl 6 sports constructs that follow the reactive programming model. One can spin up many worker threads and use threadsafe Channels or Supplys to move values from one thread to another. A react-block can combine those streams of values, process them and react to conditions like cleaning up after a worker thread is done producing values or dealing with errors. The latter is done by bottling up Exception-objects into Failure-objects that keep track of where the error first occured and where it was used instead of a proper value.

my \pipe = Supplier::Preserving.new;

start {
    for $*HOME {
        pipe.emit: .IO if .f & .ends-with('.txt');

        say „Looking in ⟨{.Str}⟩ for files that end in ".txt"“ if .IO.d;
        .IO.dir()».&?BLOCK when .IO.d;

        CATCH {
            default {
                note .^name, ': ', .Str;
                pipe.emit: Failure.new(.item);
            }
        }
    }
    pipe.done;
}

react {
    whenever pipe.Supply {
        say „Checking ⟨{.Str}⟩ for "Rosetta".“;
        say „I found Rosetta in ⟨{.Str}⟩“ if try .open.slurp.contains('Rosetta');
        LAST {
            say ‚Done looking for files.‘;
            done;
        }
        CATCH {
            default {
                note .^name, ': ', .Str;
            }
        }
    }
    whenever Promise.in(60*10) {
        say „I gave up to find Rosetta after 10 minutes.“;
        pipe.done;
        done;
    }
}

Many build-in objects will return a Supply or a Promise. The latter will return a single value or just convey an event like a timeout. In the example above we used a Promise in that fashion. Below we shell out to find and process its output line by line. This could be used in a react block if there are many different types of events to process. Here we just tap into a stream of values and process them one by one. Since we don’t got a react block to provide a blocking event loop, we wait for find to finish with await and process it’s exitcode. Anything inside the block given to .tap will run in its own thread.

my $find = Proc::Async.new('find', $*HOME, '-iname', '*.txt');
$find.stdout.lines.tap: {
    say „Looking for "Rosetta" in ⟨$_⟩“;
    say „Found "Rosetta" in ⟨$_⟩“ if try .open.slurp.contains('Rosetta');
};

await $find.start.then: {
    say „find finished with exitcode: “, .result.exitcode;
};

Having operators process values in parallel via threads or vector units is yet to be done. Both hyper operators and Junctions are candidates for autothreading. If you use them today please keep in mind side effects may provide foot guns in the future.

It’s Classes All The Way Down

While building a cache for a web api that spits out JSON I found myself walking over the same data twice to fix a lack of proper typing. The JSON knows only about strings even though most of the fields are integers and timestamps. I’m fixing the types after parsing the JSON with JSON::Fast by coercively .map-ing .

@stations.=hyper.map: { # Here be heisendragons!
    .<lastchangetime> = .<lastchangetime>
        ?? DateTime.new(.<lastchangetime>.subst(' ', 'T') ~ 'Z', :formatter(&ISO8601))
        !! DateTime;
    .<clickcount> = .<clickcount>.Int;
    .<lastcheckok> = .<lastcheckok>.Int.Bool;

    (note "$_/$stations-count processed" if $_ %% 1000) with $++;

    .Hash
};

The hyper helps a lot to speed things up but will put a lot of stress on the CPU cache. There must be a better way to do that.

Then lizmat showed where Rakudo shows its guts.

m: grammar A { token a { }; rule a { } }
OUTPUT: «5===SORRY!5=== Error while compiling <tmp>␤Package 'A' already has a regex 'a' 
(did you mean to declare a multi-method?)␤

Tokens are regex or maybe methods. But if tokens are methods then grammars must be classes. And that allows us to subclass a grammar.

grammar WWW::Radiobrowser::JSON is JSON {
    token TOP {\s* <top-array> \s* }
    rule top-array      { '[' ~ ']' <station-list> }
    rule station-list   { <station> * % ',' }
    rule station        { '{' ~ '}' <attribute-list> }
    rule attribute-list { <attribute> * % ',' }

    token year { \d+ } token month { \d ** 2 } token day { \d ** 2 } token hour { \d ** 2 } token minute { \d ** 2 } token second { \d ** 2}
    token date { <year> '-' <month> '-' <day> ' ' <hour> ':' <minute> ':' <second> }

    token bool { <value:true> || <value:false> }

    token empty-string { '""' }

    token number { <value:number> }

    proto token attribute { * }
    token attribute:sym<clickcount> { '"clickcount"' \s* ':' \s* '"' <number> '"' }
    token attribute:sym<lastchangetime> { '"lastchangetime"' \s* ':' \s* '"' <date> '"' }
    token attribute:sym<lastcheckok> { '"lastcheckok"' \s* ':' \s* '"' <bool> '"' }
}

Here we overload some tokens and forward calls to tokens that got a different name in the parent grammar. The action class follows suit.

class WWW::Radiobrowser::JSON::Actions is JSON::Actions {
    method TOP($/) {
        make $<top-array>.made;
    }
    method top-array($/) {
        make $<station-list>.made.item;
    }
    method station-list($/) {
        make $<station>.hyper.map(*.made).flat; # Here be heisendragons!
    }
    method station($/) {
        make $<attribute-list>.made.hash.item;
    }
    method attribute-list($/) {
        make $<attribute>».made.flat;
    }
    method date($_) { .make: DateTime.new(.<year>.Int, .<month>.Int, .<day>.Int, .<hour>.Int, .<minute>.Int, .<second>.Num) }
    method bool($_) { .make: .<value>.made ?? Bool::True !! Bool::False }
    method empty-string($_) { .make: Str }

    method attribute:sym<clickcount>($/) { make 'clickcount' => $/<number>.Int; }
    method attribute:sym<lastchangetime>($/) { make 'lastchangetime' => $/<date>.made; }
    method attribute:sym<lastcheckok>($/) { make 'lastcheckok' => $/<bool>.made; }
}

In case you wonder how to call a method with such a funky name, use the quoting version of postfix:<.>.

class C { method m:sym<s>{} }
C.new.'m:sym<s>'()

I truncated the examples above. The full source can be found here. The .hyper-Version is still quite a bit faster but also heisenbuggy. In fact .hyper may not work at all when executed to fast after a program starts or when used in a recursive Routine. This is mostly due to the grammer being one of the oldest parts of Rakudo with the least amount of work to make it fast. That is a solvable problem. I’m looking forward to Grammar All The Things.

If you got grammars please don’t hide them. Somebody might need them to be classy.

 

The Siege Can Continue

A wise internet spaceship pirate once wrote: “Whining gets you stuff. That’s why humans are at the top of the food chain.”. My Whining got me a fix that put an end to segfaults on long running scripts that react to http requests.

So I continued to add missing bits to my golfed httpd and found another good use for term:<>.

constant term:<HTTP-HEADER-404> = "HTTP/1.1 404 Not Found", "Content-Type: text/plain; charset=UTF-8", "Content-Encoding: UTF-8", "";

Without term:<> the compiler would think I wanted to substract 400 from a http header.

If you got some time to spare please write a whiny blog post about a crash that is bugging you – works like a charm.

Goto the Last Fifo

I always admired the elegance of the sysfs. You take a text and write it to a file to talk to a function running in kernel space. As soon as you know how to work with files, you can change the systems behaviour and reuse access control mechanism without any special tools. It’s very easy to script and dump (parts) of the systems state.

Linux and friends come with fifos that serve the same purpose. Create a fifo, set access rights and start reading from that pseudo-file. Very easy to do in Perl 5.

my $fifo; open($fifo, "+);
while (<$fifo>) {
    do-things-with $_;
} 

Rakudo doesn’t really know about fifos yet, as a result it doesn’t block on a read of a fifo that don’t got data anymore. After a bit of fiddeling I found a way around that problem.

      1 use v6.c;
      2
      3 # `mkfifo radio-fifo-in`
      4 # `echo "foo" > radio-fifo-in`
      5 # `echo "foo^D" > radio-fifo-in`
      6
      7 my $fifo-in = open(„radio-fifo-in“, :r);
      8
      9 LABEL: loop {
     10     react {
     11         whenever supply { .emit for $fifo-in.lines } {
     12             say .Str;
     13             last LABEL if /‚^D‘/;
     14         }
     15     }
     16 }

I learned that whenever reacts to last and will teach the docs about it later today. Luckily Perl 6 got labels so we can tell last where to goto.

UPDATE: scovit found a short expression that gets very close to the behaviour of Perl 5.

my $fifo = open("radio-fifo-in", :r);
while defined $_ = $fifo.get { .say }

We Need to Siege Moar

Alas, my attempt to avoid bugs by moving with little load was not as fruitful as I hoped.

Siegeing my tiny httpd lasted for about an hour and then the gates gave way. With dmesg the culprit was swiftly revealed.

[4361268.087988] moar[12491]: segfault at 3c8b ip 00007f86b426868b sp 00007f86aaff86b0 error 4 in libmoar.so[7f86b407b000+55d000]

A had the suspicion that close to 70000 tests will only get us the failure modes that are cought within minutes. Looks like there is quite some squashing left to do for the moar team.