Expensive Egg-Timers
If you use a CLI you might have done something along the line.
sleep 1m 30s; do-the-next-thing
I have a script called OK that will display a short text in a hopeful green and morse code O-K via the PC speaker. By doing so I turn my computer into an expensive egg-timer.
As of late I found myself waiting for longer periods of time and was missing a count-down so I could estimate how much more time I can waste playing computer games. The result is a program called count-down.
Since I wanted to mimic the behaviour of sleep as closely as possible I had a peek into its source-code. That made me realise how lucky I am to be allowed to use Perl 6. If I strip all the extra bits a count-down needs I’m at 33 lines of code compared to 154 lines of GNU sleep. The boilerplate I have is mostly for readability. Like defining a subset called Seconds and a Rexex called number.
Errors in the arguments to the script will be cought by the where clause in MAINs signature. Since there are no further multi candidates for MAIN that might interfere, the usage message will be displayed automatically if arguments are not recognized. Pretty much all lines in the C implementation deal with argument handling and the fact that they can’t trust their arguments until the last bit of handling is done. With a proper signature a Perl 6 Routine can fully trust its arguments and no further error handling is needed. Compared to the C version (that does a lot less) the code can be read linear from top to bottom and is much more expressive. After changing a few identifiers I didn’t feel the need for comments anymore. Even some unclear code like the splitting on numbers and keeping the values, becomes clear in the next lines where I sum up a list of seconds.
Now I can comfortably count down the rest of a year that was made much better by a made better Perl 6. I wish you all a happy 2018.
Racing Rakudo
In many racing sports telemetry plays a big role in getting faster. Thanks to a torrent of commits by lizmat you can use telemetry now too!
perl6 -e 'use Telemetry; snapper(½); my @a = (‚aaaa‘..‚zzzz‘).pick(1000); say @a.sort.[*-1 / 2];'
zyzl
Telemetry Report of Process #30304 (2017-11-05T17:24:38Z)
No supervisor thread has been running
Number of Snapshots: 31
Initial Size: 93684 Kbytes
Total Time: 14.74 seconds
Total CPU Usage: 15.08 seconds
wallclock util% max-rss gw gtc tw ttc aw atc
500951 53.81 8424
500557 51.92 9240
548677 52.15 12376
506068 52.51 196
500380 51.94 8976
506552 51.74 9240
500517 52.45 9240
500482 52.33 9504
506813 51.67 6864
502634 51.63
500520 51.78 6072
500539 52.13 7128
503437 52.29 7920
500419 52.45 8976
500544 51.89 8712
500550 49.92 6864
602948 49.71 8712
500548 50.33
500545 49.92 320
500518 49.92
500530 49.92
500529 49.91
500507 49.92
506886 50.07
500510 49.93 1848
500488 49.93
500511 49.93
508389 49.94
508510 51.27 264
27636 58.33
--------- ------ -------- --- -------- --- -------- --- --------
14738710 51.16 130876
Legend:
wallclock Number of microseconds elapsed
util% Percentage of CPU utilization (0..100%)
max-rss Maximum resident set size (in Kbytes)
gw The number of general worker threads
gtc The number of tasks completed in general worker threads
tw The number of timer threads
ttc The number of tasks completed in timer threads
aw The number of affinity threads
atc The number of tasks completed in affinity threads
The snapper function takes an interval at which data is collected. On termination of the program the table above is shown.
The module comes with plenty of subs to collect the same data at hand and file your own report. What may be sensible in long running processes. Or you call the reporter sub by hand every now and then.
use Telemetry;
react {
snapper;
whenever Supply.interval(60) {
say report;
}
}
If the terminal wont cut it you can use http to fetch telemetry data.
Documentation isn’t finished nor is the module. So stay tuning for more data.
There Is More Than One Way At The Same Time
The Perl 6 Rosattacode section for parallel calculations is terribly outdated and missing all the goodies that where added or fixed in the last few weeks. With this post I would like to propose an updated version for Rosettacode. If you believe that I missed something feel free to comment below. Please keep in mind that Rosettacode is for showing off, not for being comprehensive.
use v6.d.PREVIEW;
Perl 6 provides parallel execution of code via threads. There are low level custructs that start a thread or safely pause execution.
my $t1 = Thread.start({ say [+] 1..10_000_000 });
my $t2 = Thread.start({ say [*] 1..10_000 });
$t1.finish;
$t2.finish;
my $l = Lock.new;
$l.lock;
$t1 = Thread.start: { $l.lock; say 'got the lock'; $l.unlock };
sleep 2; $l.unlock;
$t1.finish;
When processing lists, one can use a highlevel Iterator created by the methods hyper and race. The latter may return values out of order. Those Iterators will distribute the elements of the list to worker threads that are in turn assigned to OS level threads by Rakudos ThreadPoolScheduler. The whole construct will block until the last element is processed.
my @values = 1..100;
sub postfix:<!> (Int $n) { [*] 1..$n }
say [+] @values.hyper.map( -> $i { print '.' if $i %% 100; $i!.chars });
For for-lovers there are the race for and hyper for keyword for distributing work over threads in the same way as their respective methods forms.
race for 1..100 {
say .Str; # There be out of order dragons!
}
my @a = do hyper for 1..100 {
.Int! # Here be thread dragons!
}
say [+] @a;
Perl 6 sports constructs that follow the reactive programming model. One can spin up many worker threads and use threadsafe Channels or Supplys to move values from one thread to another. A react-block can combine those streams of values, process them and react to conditions like cleaning up after a worker thread is done producing values or dealing with errors. The latter is done by bottling up Exception-objects into Failure-objects that keep track of where the error first occured and where it was used instead of a proper value.
my \pipe = Supplier::Preserving.new;
start {
for $*HOME {
pipe.emit: .IO if .f & .ends-with('.txt');
say „Looking in ⟨{.Str}⟩ for files that end in ".txt"“ if .IO.d;
.IO.dir()».&?BLOCK when .IO.d;
CATCH {
default {
note .^name, ': ', .Str;
pipe.emit: Failure.new(.item);
}
}
}
pipe.done;
}
react {
whenever pipe.Supply {
say „Checking ⟨{.Str}⟩ for "Rosetta".“;
say „I found Rosetta in ⟨{.Str}⟩“ if try .open.slurp.contains('Rosetta');
LAST {
say ‚Done looking for files.‘;
done;
}
CATCH {
default {
note .^name, ': ', .Str;
}
}
}
whenever Promise.in(60*10) {
say „I gave up to find Rosetta after 10 minutes.“;
pipe.done;
done;
}
}
Many build-in objects will return a Supply or a Promise. The latter will return a single value or just convey an event like a timeout. In the example above we used a Promise in that fashion. Below we shell out to find and process its output line by line. This could be used in a react block if there are many different types of events to process. Here we just tap into a stream of values and process them one by one. Since we don’t got a react block to provide a blocking event loop, we wait for find to finish with await and process it’s exitcode. Anything inside the block given to .tap will run in its own thread.
my $find = Proc::Async.new('find', $*HOME, '-iname', '*.txt');
$find.stdout.lines.tap: {
say „Looking for "Rosetta" in ⟨$_⟩“;
say „Found "Rosetta" in ⟨$_⟩“ if try .open.slurp.contains('Rosetta');
};
await $find.start.then: {
say „find finished with exitcode: “, .result.exitcode;
};
Having operators process values in parallel via threads or vector units is yet to be done. Both hyper operators and Junctions are candidates for autothreading. If you use them today please keep in mind side effects may provide foot guns in the future.
It’s Classes All The Way Down
While building a cache for a web api that spits out JSON I found myself walking over the same data twice to fix a lack of proper typing. The JSON knows only about strings even though most of the fields are integers and timestamps. I’m fixing the types after parsing the JSON with JSON::Fast by coercively .map-ing .
@stations.=hyper.map: { # Here be heisendragons!
.<lastchangetime> = .<lastchangetime>
?? DateTime.new(.<lastchangetime>.subst(' ', 'T') ~ 'Z', :formatter(&ISO8601))
!! DateTime;
.<clickcount> = .<clickcount>.Int;
.<lastcheckok> = .<lastcheckok>.Int.Bool;
(note "$_/$stations-count processed" if $_ %% 1000) with $++;
.Hash
};
The hyper helps a lot to speed things up but will put a lot of stress on the CPU cache. There must be a better way to do that.
Then lizmat showed where Rakudo shows its guts.
m: grammar A { token a { }; rule a { } }
OUTPUT: «5===SORRY!5=== Error while compiling <tmp>Package 'A' already has a regex 'a'
(did you mean to declare a multi-method?)
Tokens are regex or maybe methods. But if tokens are methods then grammars must be classes. And that allows us to subclass a grammar.
grammar WWW::Radiobrowser::JSON is JSON {
token TOP {\s* <top-array> \s* }
rule top-array { '[' ~ ']' <station-list> }
rule station-list { <station> * % ',' }
rule station { '{' ~ '}' <attribute-list> }
rule attribute-list { <attribute> * % ',' }
token year { \d+ } token month { \d ** 2 } token day { \d ** 2 } token hour { \d ** 2 } token minute { \d ** 2 } token second { \d ** 2}
token date { <year> '-' <month> '-' <day> ' ' <hour> ':' <minute> ':' <second> }
token bool { <value:true> || <value:false> }
token empty-string { '""' }
token number { <value:number> }
proto token attribute { * }
token attribute:sym<clickcount> { '"clickcount"' \s* ':' \s* '"' <number> '"' }
token attribute:sym<lastchangetime> { '"lastchangetime"' \s* ':' \s* '"' <date> '"' }
token attribute:sym<lastcheckok> { '"lastcheckok"' \s* ':' \s* '"' <bool> '"' }
}
Here we overload some tokens and forward calls to tokens that got a different name in the parent grammar. The action class follows suit.
class WWW::Radiobrowser::JSON::Actions is JSON::Actions {
method TOP($/) {
make $<top-array>.made;
}
method top-array($/) {
make $<station-list>.made.item;
}
method station-list($/) {
make $<station>.hyper.map(*.made).flat; # Here be heisendragons!
}
method station($/) {
make $<attribute-list>.made.hash.item;
}
method attribute-list($/) {
make $<attribute>».made.flat;
}
method date($_) { .make: DateTime.new(.<year>.Int, .<month>.Int, .<day>.Int, .<hour>.Int, .<minute>.Int, .<second>.Num) }
method bool($_) { .make: .<value>.made ?? Bool::True !! Bool::False }
method empty-string($_) { .make: Str }
method attribute:sym<clickcount>($/) { make 'clickcount' => $/<number>.Int; }
method attribute:sym<lastchangetime>($/) { make 'lastchangetime' => $/<date>.made; }
method attribute:sym<lastcheckok>($/) { make 'lastcheckok' => $/<bool>.made; }
}
In case you wonder how to call a method with such a funky name, use the quoting version of postfix:<.>.
class C { method m:sym<s>{} }
C.new.'m:sym<s>'()
I truncated the examples above. The full source can be found here. The .hyper-Version is still quite a bit faster but also heisenbuggy. In fact .hyper may not work at all when executed to fast after a program starts or when used in a recursive Routine. This is mostly due to the grammer being one of the oldest parts of Rakudo with the least amount of work to make it fast. That is a solvable problem. I’m looking forward to Grammar All The Things.
If you got grammars please don’t hide them. Somebody might need them to be classy.
The Siege Can Continue
A wise internet spaceship pirate once wrote: “Whining gets you stuff. That’s why humans are at the top of the food chain.”. My Whining got me a fix that put an end to segfaults on long running scripts that react to http requests.
So I continued to add missing bits to my golfed httpd and found another good use for term:<>.
constant term:<HTTP-HEADER-404> = "HTTP/1.1 404 Not Found", "Content-Type: text/plain; charset=UTF-8", "Content-Encoding: UTF-8", "";
Without term:<> the compiler would think I wanted to substract 400 from a http header.
If you got some time to spare please write a whiny blog post about a crash that is bugging you – works like a charm.
Goto the Last Fifo
I always admired the elegance of the sysfs. You take a text and write it to a file to talk to a function running in kernel space. As soon as you know how to work with files, you can change the systems behaviour and reuse access control mechanism without any special tools. It’s very easy to script and dump (parts) of the systems state.
Linux and friends come with fifos that serve the same purpose. Create a fifo, set access rights and start reading from that pseudo-file. Very easy to do in Perl 5.
my $fifo; open($fifo, "+);
while (<$fifo>) {
do-things-with $_;
}
Rakudo doesn’t really know about fifos yet, as a result it doesn’t block on a read of a fifo that don’t got data anymore. After a bit of fiddeling I found a way around that problem.
1 use v6.c;
2
3 # `mkfifo radio-fifo-in`
4 # `echo "foo" > radio-fifo-in`
5 # `echo "foo^D" > radio-fifo-in`
6
7 my $fifo-in = open(„radio-fifo-in“, :r);
8
9 LABEL: loop {
10 react {
11 whenever supply { .emit for $fifo-in.lines } {
12 say .Str;
13 last LABEL if /‚^D‘/;
14 }
15 }
16 }
I learned that whenever reacts to last and will teach the docs about it later today. Luckily Perl 6 got labels so we can tell last where to goto.
UPDATE: scovit found a short expression that gets very close to the behaviour of Perl 5.
my $fifo = open("radio-fifo-in", :r);
while defined $_ = $fifo.get { .say }
We Need to Siege Moar
Alas, my attempt to avoid bugs by moving with little load was not as fruitful as I hoped.
Siegeing my tiny httpd lasted for about an hour and then the gates gave way. With dmesg the culprit was swiftly revealed.
[4361268.087988] moar[12491]: segfault at 3c8b ip 00007f86b426868b sp 00007f86aaff86b0 error 4 in libmoar.so[7f86b407b000+55d000]
A had the suspicion that close to 70000 tests will only get us the failure modes that are cought within minutes. Looks like there is quite some squashing left to do for the moar team.
