Autorotating logs

While translating my backup script from bash to Raku I did a peek into /var/log/and was shocked to find 650MB of stuff I never looked at. Unsurprisingly /var/log/journal was the largest offender . Sadly I must say that systemd is very consistent in convincing me to be not very good software. The oldest offender was from 2008. Debian is very consistent in convincing me to keep it around. Even though it does not come with logrotate for xrdp.log. How hard can it be to build in rotating and compressing logs into your own software?

That turned out to be a 60-line-problem.

sub open-logfile(IO() $log-file-path = IO::Path, :$flush = False, :$max-size = 2**20 * 50, :$max-backlog = 7 --> Routine) {
    my $log-channel = Channel.new;
    my $log-promise = start {
        my $log-handle = $log-file-path ?? open $log-file-path, :a !! $*ERR;
        my $log-file-dir = $log-file-path.dirname;
        my $log-file-basename = $log-file-path.basename;

        sub compress-file($path) {
            my $gz = run <gzip -9>, $path;
            warn „could not compress $path“ unless $gz;
        }

        my $log-file-length = $log-file-path ?? $log-file-path.s !! 0;
        sub rotate-log {
            return without $log-file-path;
            note ‚rotating logfile‘;
            $log-handle.close;

            my @siblings = dir($log-file-dir).grep(*.basename.starts-with($log-file-basename));
            for @siblings».Str.sort.skip.head($max-backlog - 1).reverse -> $path {
                my $n = $path.match(/log '.' (\d+)/)[0].Int + 1;
                compress-file($path) if $n == 2;
                my $new-name = $log-file-path ~ '.' ~ ($n > 1 ?? $n ~ ‚.gz‘ !! $n);
                ($n == 2 ?? $path ~ ‚.gz‘ !! $path).IO.rename($new-name);
            }

            $log-file-path.rename($log-file-path ~ '.1');
            $log-handle = open $log-file-path, :a;
            $log-file-length = 0;
        }

        react whenever $log-channel -> Str() $line {
            my $utf8 = ($line ~ "\n").encode;
            $log-file-length += $utf8.bytes;
            rotate-log if $log-file-length > $max-size;
            $log-handle.write: $utf8;
            $log-handle.flush if $flush;
        }
    }

    my $last-message = "";
    my int $dupe-counter = 0;
    sub ($message --> Nil) {
        my $now = now.DateTime;
        my $timestamp = $now.yyyy-mm-dd ~ ' ' ~ $now.hh-mm-ss;

        if $message eq $last-message {
            $dupe-counter++;
        } else {
            if $dupe-counter > 0 {
                $log-channel.send: $timestamp ~ ' Last message repeated ' ~ $dupe-counter ~ ' times.';
            } else {
                $log-channel.send: $timestamp ~ ' ' ~ $message;
            }
            $dupe-counter = 0;
        }

        $last-message = $message;
    } but role :: { method close { $log-channel.close; await $log-promise } }
}

Without .flush after each line performance is good. With it it’s pretty bad given that we don’t have access to fancy linux flags for open and as such don’t really flush all that hard.

Since rotating logs and compressing them might take some time I put that actual writing to the file into its own thread. So usage becomes a bit indirect and we have to tell the logger to finish writing and close the file. If we don’t we can lose data. I’m not sure if the latter is a bug.

my &log = open-logfile(‚/tmp/test.log‘, :max-size(2**20));

my $lore = ‚Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.‘;

loop {
    log(100.rand > 50 ?? $lore !! 42.rand);
    last if (now - ENTER now) > 60 * 60 * 2;
}

&log.await;

After much deliberation I have come to the conclusion that renaming Perl 6 to Raku was a mistake. It should have been named Exceedingly Simple.