Watching new arrivals

Any boring recurring task must be as easy as possible or it will be neglected. I’m quite sure this is why we invented computers. Backups are kinda borring. In fact you want to avoid any form of excitement when it comes to backups. So they must be as easy as possible. I have a script that is triggered by a udev rule when a new device is added. This is working fine when a single disk is plugged in. (This works very well.) I got a usb hub with a few usb sticks that form a btrfs raid5 for making quick backups of my $home whenever I switch the usb hub on. This does not work fine in some cases. Getting a bash script to check if a drive in a set is missing ain’t fun. Mostly because only proper languages come with Set. We do have a proper language.

On linux it is fairly easy to find out if a drive was plugged in. All we need to do is to watch for new files to pop up in /dev/disk/by-id/. We can also learn if new partitions where found. The directory looks like the following.

$ ls -1 /dev/disk/by-id/
ata-CT120BX500SSD1_1902E16BC135
ata-CT120BX500SSD1_1902E16BC2AA
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG-part1
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG-part2
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG-part3
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG-part1
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG-part2
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG-part3
usb-SanDisk_Ultra_USB_3.0_4C530001160708110455-0:0
usb-SanDisk_Ultra_USB_3.0_4C530001190708111070-0:0
usb-SanDisk_Ultra_USB_3.0_4C530001220708110370-0:0
usb-SanDisk_Ultra_USB_3.0_4C530001280708111064-0:0
wwn-0x50000398dc60029a
wwn-0x50000398dc60029a-part1
wwn-0x50000398dc60029a-part2
wwn-0x50000398dc60029a-part3
wwn-0x50000398ebb01681
wwn-0x50000398ebb01681-part1
wwn-0x50000398ebb01681-part2
wwn-0x50000398ebb01681-part3

If we look for anything that doesn’t end in '-part' \d+ we got a drive. We can also tell where it’s plugged in by checking the prefix.

sub scan-drive-ids(--> Set) {
    my Set $ret;
    for '/dev/disk/by-id/'.IO.dir.grep(!*.IO.basename.match(/'part' \d+ $/)) {
        $ret ∪= .basename.Str;
        CATCH { default { warn .message } }
    }

    $ret
}

my %last-seen := scan-drive-ids;

Sets don’t got an append method. We can substitude that with ∪=. Now we got a lovely Set of drives in %last-seen that are already there. We now need to wait for new files to pop up and apply set theory to them.

react {
    whenever IO::Notification.watch-path('/dev/disk/by-id/') {
        my %just-seen := scan-drive-ids;
        my %new-drives := %just-seen ∖ %last-seen;
        my %old-drives := %last-seen ∩ %just-seen;
        my %removed-drives := %last-seen ∖ %just-seen;
        %last-seen := %just-seen;

        # say ‚old drives: ‘, %old-drives.keys.sort;
        say ‚new drives: ‘, %new-drives.keys.sort || '∅';
        say ‚removed drives: ‘, %removed-drives.keys.sort || '∅';
    }
}

By binding the Sets to an Associative container we get for and other buildins to behave. If we want to take action if certain disks are added we need to define Sets that contain the right file names.

my %usb-backup-set = Set(<usb-SanDisk_Ultra_USB_3.0_4C530001160708110455-0:0 usb-SanDisk_Ultra_USB_3.0_4C530001190708111070-0:0 usb-SanDisk_Ultra_USB_3.0_4C530001220708110370-0:0 usb-SanDisk_Ultra_USB_3.0_4C530001280708111064-0:0>);

my %root-backup-disk = Set(<ata-TOSHIBA_DT01ACA200_8443D04GS>);

my $delayed-check := Channel.new;
my Promise $timeout-promise;

react {
    whenever IO::Notification.watch-path('/dev/disk/by-id/') {
        my %just-seen := scan-drive-ids;
        my %new-drives := %just-seen ∖ %last-seen;
        my %old-drives := %last-seen ∩ %just-seen;
        my %removed-drives := %last-seen ∖ %just-seen;
        %last-seen := %just-seen;

        # say ‚old drives: ‘, %old-drives.keys.sort;
        say ‚new drives: ‘, %new-drives.keys.sort || '∅';
        say ‚removed drives: ‘, %removed-drives.keys.sort || '∅';

        if %usb-backup-set ∩ %new-drives {
            $timeout-promise = Promise.in(5).then: {
                $delayed-check.send: True;
                $timeout-promise = Nil;
            } without $timeout-promise;
        }

        if %root-backup-disk ∩ %new-drives {
            sleep 2;
            backup-root-and-home-to-disk(%root-backup-disk);
        }

        say '';
    }
    whenever $delayed-check {
        my %just-seen := scan-drive-ids;
        if %usb-backup-set ⊆ %just-seen {
            backup-home-to-usb(%usb-backup-set);
        } elsif %usb-backup-set ∩ %just-seen {
            warn 'drive missing in usb set: ' ~ (%usb-backup-set ∖ (%usb-backup-set ∩ %just-seen)).keys;
            reset-usb-hub;
        }
    }
}

I use the $delayed-check whenever-block to handle the case when one of the usb sticks refuses to come online. The vendorid and deviceid of the usb hub are hardcoded. Please note that state and start don’t mix well.

sub reset-usb-hub(--> True) {
    state $reset-attempt = 0;
    if $reset-attempt++ {
        say ‚already reset, doing nothing‘;
        $reset-attempt = 0;
    } else {
        say ‚Resetting usb hub.‘;
        my $usb_modeswitch = run <usb_modeswitch -v 0x2109 -p 0x0813 --reset-usb>;
        fail ‚resetting usb hub failed‘ unless $usb_modeswitch;
    }
}

The entire script can be found here. I believe the example of watch-path could use a modified version of this script. If you read it you can tell where Sets are used simply by spotting set operators. Making Raku a operator oriented language was a good idea. Thank you Larry.

While turning my backup script from Bash to Raku I had some more findings about shell scripting in a proper language. I shall report about them here in the next few weeks.