Releasing for virtuous programmers

Today I released META6::bin with the following command.

meta6 --release

Since I forgot to commit the changes to README.md I had to release again with.

meta6 --release --version=+

As a result there are now some friendly tarballs at https://github.com/gfldex/raku-meta6-bin/releases/. To get there I had to force travis to install Test::META from HEAD. If you wish to release on github with ease, you may have to run the following commands.

zef install --/test https://github.com/jonathanstowe/Test-META.git
zef install META6::bin

It should work with already existing META6.json files out of the box but likely need the setup instructions at the bottom of the README.

As soon as RabidGravy found the time to make a proper release of Test::META (shouldn’t take more then 10 seconds) I will set the proper dependencies.

Some oddities in the response of the github API (<tarball_url> doesn’t point at the tarball) gave me a good reason to have another look at Codeberg. A lot was added in the last two years. The project is run by a non-profit from Berlin and as such not based in a country that likes to punish people because they don’t like a government. If we want to provide a programming language for a global community we can’t rely on Trumpland anymore. The whole software stack is FOSS and the API looks really nice. I didn’t miss any features that are on github but I didn’t really look for projects that are handled by an organisation.

I aim to add support for Codeberg to META6::bin and shall report here before Christmas.

List breaks the chain

While watching RaycatWhoDat liking Raku, I realised that a List of Matches is not a MatchList.

say 'a1'.match(/ \d /).replace-with('#');
say 'a1b2'.match(/ \d /, :g).replace-with('#');
# OUTPUT: a#
#         No such method 'replace-with' for invocant of type 'List'
            in block <unit> at /home/dex/tmp/tmp-2.raku line 8

The 2nd call of replace-with will fail because .match with :g returns a List of Match. Of course we can play the easy game and just use subst which does the right thing when called with :g. This wouldn’t make a good blog post though.

To make replace-with work with a List we can use a where-clause. Any Match will have a copy of the original Str but not to the original Regex so we actually have to build a list of Str of everything that was not matched. This can be done by using the indices stored in .from and .to.

multi sub replace-with(\l where (.all ~~ Match), \r --> Str) {
    my $orig := l.head.orig;
    my @unmatched;

    @unmatched.push: $orig.substr(0, l.head.from);
    for ^(l.elems - 1) -> $idx {
        @unmatched.push: $orig.substr(l[$idx].to, l[$idx+1].from - l[$idx].to);
    }

    @unmatched.push: $orig.substr(l.tail.to);

    (@unmatched Z (|(r xx l.elems), |'')).flat.join;
}

say 'a1vvvv2dd3e'.match(/ \d /, :g).&replace-with('#');
# OUTPUT: a#vvvv#dd#e

If the original string does not end with a match, the list of matches will be one short to be just zipped in. That’s why I have to extend the list of replacements by an empty string before feeding it to Z.

So if subst is doing it right why bother with .replace-with? Because sometimes we have to use $/.

if 'a1bb2ccc3e' ~~ m:g/ \d / {
    say $/.&replace-with('#');
}

Often we could change the code but when a routine from a module returns Match or a list thereof, we are out of luck. For completeness we need a few more multies.

multi sub replace-with(Match \m, \r --> Str) {
    m.replace-with(r);
}

multi sub replace-with(Match \m, &r --> Str) {
    m.replace-with(r(m));
}

multi sub replace-with(\l where (.all ~~ Match), &r) {
    my $orig := l.head.orig;
    my @unmatched;

    @unmatched.push: $orig.substr(0, l.head.from);
    for ^(l.elems - 1) -> $idx {
        @unmatched.push: $orig.substr(l[$idx].to, l[$idx+1].from - l[$idx].to);
    }

    @unmatched.push: $orig.substr(l.tail.to);

    (@unmatched Z (|l.map(&r), |'')).flat.join;
}

Even if the problem is solvable it still bugs me. We have :g in many places in Raku to provide quite a lot of DWIM. In some places that concept breaks and almost all of them have to do with lists. Often ».comes to the rescue. When we actually have to work on the list and not the individual elements, even that doesn’t work. The methods on strings just work because in this case we deliberately avoid breaking the list of characters apart.

If you follow this blog you know that I’m leaning strongly towards operators. Sadly ., .? and ». are not real infixes which we can overload. Nor can we declare infixes that start with a . or we could introduce an operator that turns it’s LHS to a list and then does a dispatch to multis that can handle lists of a certain type.

Without that we need to change .match to return a subclass of List that got the method .replace-with. We could stick it into Cool but that is a crowded place already.

We don’t really have a nice way to augment the return values of builtin methods. So this will have to be fixed in CORE.

Releasing on github

In my last post I lamented the lack of testing metadata. Just a few days later it got in my way when I played with creating releases on github. My normal workflow on github is to commit changes and push them to trigger travis. When travis is fine I bump the version field in META6.json so the ecosystem and zef can pick up the changes. And there is a hidden trap. If anybody clones the repo via zef just before I bump the version, there will be a mismatch between code and version. CPAN doesn’t got that problem because there is always a static tarball per version. With releases we can get the same on github.

It’s a fairly straight forward process.

• build a tag-name from the github-project-name and a version string
• generate the URL the tarball will get in the end (based on tag-name) and use that as source-url in the META6.json
• commit the new META6.json locally
• create a git tag locally
• push the local commit with the changed META6.json to github
• push the git tag to github
• use the github API to create the release, which in turn creates the tarball

This is so simple that I immediately automated that stuff with META6::bin so I can mess it up. (Not released yet, see below.)

The result is an URL like so: https://github.com/gfldex/raku-release-test/archive/raku-release-test-0.0.19.tar.gz. When we feed that to zef it will check if the version is not already installed and then proceed to test and install.

And there is a catch. Even though zef is fine with the URL, Test::META will complain because it doesn’t end in .git and fail the test. This in turn will stop zef from installing the module. We added that check to make sure zef always gets a proper link to a clone-able repo for modules hosted on github. This assumption is clearly wrong and needs fixing. I will send a PR soon.

Having releases on github (other gitish repo-hosting sites will have similar facilities or will get them) can get us one step closer to a proper RPAN. Once I got my first module into the ecosystem this way I will provide an update here.

How does lizmat know?

I didn’t know so I asked her.

15:25 <gfldex> How do you gather info for "Updated Raku Modules"?
17:40 <lizmat> https://twitter.com/raku_cpan_new
17:59 <gfldex> thx
23:06 <lizmat> well volunteered  :-)

That’s what you get for being nosey. So off I went into the land of mostly undocumented infrastructure.

The objective is simple. Generate two lists of modules where the first contains all modules that are newly added to the ecosystem and the second got all updated modules. For both the timespan of interest is Monday of this week until Monday of last week. Currently we got two collections of META-files. Our ecosystem and CPAN. The latter does not know about META6 and that sucks. But we will manage. Conveniently both lists are provided by ugexe at github. Since there are commits we can travel back in time and get a view of the ecosystem from when we need it. To do so we first need to get a list of commits.

sub github-get-remote-commits($owner, $repo, :$since, :$until) is export(:GIT) {
    my $page = 1;
    my @response;
    loop {
        my $commits-url = $since && $until ?? „https://api.github.com/repos/$owner/$repo/commits?since=$since&until=$until&per_page=100&page=$page“ !! „https://api.github.com/repos/$owner/$repo/commits“;
        my $curl = Proc::Async::Timeout.new('curl', '--silent', '-X', 'GET', $commits-url);
        my $github-response;
        $curl.stdout.tap: { $github-response ~= .Str };

        await my $p = $curl.start: :$timeout;
        @response.append: from-json($github-response);

        last unless from-json($github-response)[0].<commit>;
        $page++;
    }

    if @response.flat.grep(*.<message>) && @response.flat.hash.<message>.starts-with('API rate limit exceeded') {
        dd @response.flat;
        die „github hourly rate limit hit.“;
    }

    @response.flat
}

my @ecosystems-commits = github-get-remote-commits(‚ugexe‘, ‚Perl6-ecosystems‘, :since($old), :until($young));

Now we can get a whole bunch of ex-json which was compiled of the META6.json and *.meta files. Both file formats are not compatible. The auth field of a CPAN module will differ from the auth of the upstream META6.json, there is no authors field and no URL to the upstream repo. Not pretty but fixable because tar is awesome.

my @meta6;
px«curl -s $source-url» |» px<tar -xz -O --no-wildcards-match-slash --wildcards */META6.json> |» @meta6;

my $meta6 = @meta6.join.chomp.&from-json;

(Well, GNU tar is awesome. BSD tar doesn’t sport --no-wildcards-match-slash and there is one module with two META6.json-files. I think I can get around this with a 2 pass run.)

This works nicely for all but one module. For some reason a Perl 5 module sneaked into the list of Raku modules on CPAN. It’s all just parsed JSON so we can filter those out.

my @ecosystems = fetch-ecosystem(:commit($youngest-commit)).grep(*.<perl>.?starts-with('6'));

Some modules don’t contain an auth field, some got an empty name. Others don’t got the authors field set. We don’t enforce proper meta data even though it’s very easy to add quality control. Just use Test::META in your tests. Here is an example.

I can’t let lizmat down though and github knows almost all authors.

sub github-realname(Str:D $handle) {
    my @github-response;

    my $url = 'https://api.github.com/users:' ~ $handle;
    px«curl -s -X GET $url» |» @github-response;

    @github-response.join.&from-json.<name>
}

If there is more then one author they wont show up with this hack. I can’t win them all. I’m not the only one who suffers here. On modules.raku.org at least one module shows up twice with the same author. My guess is that happens when a module is published both in our ecosystem and on CPAN. I don’t know what zef does if you try to nail a module down by author and version with such ambiguity.

I added basic html support and am now able to give you a preview of next weeks new modules.

New Modules

• Pod::Weave by Daniel Sockwell.
• Pod::Literate by Daniel Sockwell.
• System::Stats::NETUsage by Ramiro Encinas Alarza.

Updated Modules

• Font::FreeType by David Warring.
• Font::FreeType by David Warring.
• Math::Libgsl::Elementary by Fernando Santagata.
• Readline by Jeffrey Goff <drforr@pobox.com>; Daniel Lathrop <dwl@foo.ist>.
• Auth::SCRAM by Marcel Timmerman.
• Email::MIME by github:retupmoca; Rod Taylor <rbt@cpan.org>.
• Router::Right by gitlab:pheix.

If your module is in the list and your name looks funny, you may want to have a look into the META6.json of you project.

Yesterday we had a discussion about where to publish modules. I will not use CPAN with the wrong language. Don’t get me wrong. I like CPAN. You can tie an aircraft carrier to it and it wont move. But it’s a Comprehensive Perl Archive Network. It’s no wonder it doesn’t like our metadata.

Kudos to tony-o for taking on a sizeable task. I hope my lamentation is helpful in this regard.

The script can be found here. I plan to turn it into a more general module to query the ecosystem. Given I spend the better part of a week on a 246 lines file the module might take a while.