swraid/mdcheck: add mdcheck timer for swraid device scrub #204713

mfenniak · 2022-12-06T00:27:46Z

Description of changes

Adds new options hardware.raid.swraid.monitor which, when enabled, ports a systemd timer from the mdadm package to NixOS that runs regularly scheduled scrubs on swraid md* devices.

To support the new NixOS module, this PR adjusts the mdadm package to: (a) package the mdcheck script, (b) remove the upstream systemd units (mdstart_, mdmonitor) that are not functional in NixOS.

Questions/uncertainties with this (raised in https://discourse.nixos.org/t/nixos-mdadm-mdcheck/23716):

I’ve named the configuration added hardware.raid.swraid.monitor – is this a good naming scheme?
enable defaults to false – this maintains compatibility with existing NixOS systems, but, has some risks of being an unsafe configuration since regular raid scrubbing is a somewhat standard safety practice. Should this be true, false, or maybe true-if-stateVersion>="23.05" so that it becomes the default on new systems?
I removed some existing systemd units that are packaged by the mdadm package, but aren’t usable because they don’t subst in the right mdcheck/mdadm paths, and instead recreated the units via systemd.timers and systemd.services to allow configurability. Is this a good direction, or, would it be better to use the upstream unit files and tweak them?
This doesn’t run the mdmonitor systemd service which has a couple of functionalities; (a) it emails or runs a program on a device event, (b) it swaps spare devices between raid arrays if appropriate on the event of a failure. I didn’t want to include it because getting the functionality (b) requires enabling the functionality (a), which would require explicit configuration for “what to do”. Is leaving this out pretty reasonable? It could be enabled pretty easily by another module, or enhanced in the future.

Things done

nixos-discourse · 2022-12-06T00:32:16Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-mdadm-mdcheck/23716/4

peterhoeg · 2022-12-06T01:18:48Z

Feel free to use anything from here:

{ config, lib, pkgs, ... }:

let
  cfg = config.toupstream.hardware.raid;

  inherit (lib) mkEnableOption mkIf mkMerge mkOption optional optionalAttrs types;
  inherit (import ../../../common/lib.nix { inherit config lib pkgs; }) mkBoolOption;

  diskPath = e:
    "/dev/disk/by-${e.idBy}/${e.name}";

  mdadmConf = ''
    MAILADDR = ${cfg.monitor.recipient}
    MAILFROM = no-reply@${config.networking.hostName}

  ''
  + "DEVICE " + lib.concatMapStringsSep " "
    (e: ''${diskPath e}${cfg.partitionSuffix}'')
    cfg.drives
  + "\n\n"
  + lib.concatMapStringsSep "\n"
    (e: "ARRAY /dev/${e.device} " + lib.concatStringsSep " " (lib.mapAttrsToList (k: v: "${k}=${v}") {
      metadata = e.version;
      UUID = e.uuid;
      name = config.networking.hostName + ":" + e.name;
    }))
    cfg.arrays;

  commonServiceConfig = {
    PrivateTmp = true;
    ProtectSystem = "strict";
    ProtectHome = "tmpfs";
  };

  enable = builtins.length cfg.drives > 0;
  doMonitor = builtins.length cfg.arrays > 0;

  arrayPath = array:
    "/sys/block/${array.device}/md";

in
{
  meta.maintainers = with lib.maintainers; [ peterhoeg ];

  options.toupstream.hardware.raid = {
    partitionSuffix = mkOption {
      description = "Partition suffix";
      type = types.str;
      default = "-part1";
    };

    earlyInit = mkOption {
      description = "Set up the RAID in initrd to ensure that the RAID is assembled with the proper name";
      type = types.bool;
      default = true;
    };

    openFirewall = mkBoolOption "Open the firewall for services that can be read remotely." true;

    temperature = {
      enable = mkBoolOption "Drive temperatures" true;

      preferHddtemp = mkBoolOption "Prefer hddtemp instead of drivetemp" false;
    };

    optimize = {
      enable = mkEnableOption "optimize software RAID";

      stripeCache = mkOption {
        description = "Device stripe cache size";
        type = types.ints.positive;
        default = 4096;
      };

      readAhead = mkOption {
        description = "Device read-ahead";
        type = types.ints.positive;
        default = 65535;
      };

      speed = mkOption {
        description = "Maximum resync/rebuild speed";
        type = types.ints.positive;
        default = 500000;
      };
    };

    drives = mkOption {
      description = "List of drives for software RAID";
      default = [ ];
      type = types.listOf
        (types.submodule {
          options = {
            name = mkOption {
              description = "Drive name without the full path.";
              type = types.str;
            };

            idBy = mkOption {
              description = "The method to identify the drive.";
              type = types.enum [ "path" "wwn" ];
              default = "path";
            };
          };
        });
    };

    monitor = {
      recipient = mkOption {
        description = "Notification recipient";
        type = types.str;
      };
    };

    scrub = {
      enable = mkEnableOption "automatic scrubbing";

      startAt = mkOption {
        description = "When to scrub";
        type = types.str;
        default = "weekly";
      };
    };

    arrays = mkOption {
      description = "software RAID arrays";
      default = [ ];
      type = types.listOf
        (types.submodule {
          options = {
            device = mkOption {
              description = "Device";
              type = types.str;
            };

            uuid = mkOption {
              description = "Device UUID";
              type = types.str;
            };

            name = mkOption {
              description = "Device name";
              type = types.str;
            };

            version = mkOption {
              description = "Version";
              type = types.enum [ "0.9" "1.2" ];
              default = "1.2";
            };
          };
        });
    };
  };

  config = mkMerge [
    (mkIf enable {
      hardware = {
        sata.timeout = {
          deciSeconds = 70;
          drives = map (e: { name = e.name; idBy = "path"; }) cfg.drives;
        };
      };

      systemd.services.mdmonitor.enable = false;
    })

    (mkIf doMonitor {
      boot.initrd.mdadmConf = mkIf cfg.earlyInit mdadmConf;
      environment.etc."mdadm/mdadm.conf".text = mdadmConf;
      systemd.services.mdadm = {
        description = "Monitor RAID";
        wantedBy = [ "multi-user.target" ];
        serviceConfig = commonServiceConfig // {
          Type = "exec";
          ExecStart = "${pkgs.mdadm}/bin/mdadm --monitor --scan";
        };
      };
    })

    (mkIf cfg.optimize.enable {
      boot.kernel.sysctl = {
        "dev.raid.speed_limit_max" = cfg.optimize.speed;
      };

      systemd.services = lib.listToAttrs (map
        (e: lib.nameValuePair
          "optimize-${e.device}"
          {
            description = "Optimize software RAID speed";
            wantedBy = [ "multi-user.target" ];
            unitConfig.ConditionPathExists = [
              "/dev/${e.device}"
              (arrayPath e)
            ];
            script = ''

              ###
              # /dev/${e.device}
              ###

              # read-ahead
              ${lib.getBin pkgs.util-linux}/bin/blockdev \
                --setra ${toString cfg.optimize.readAhead} \
                /dev/${e.device}

              # stripe cache size
              echo ${toString cfg.optimize.stripeCache} > \
                ${arrayPath e}/stripe_cache_size

              # use all resources for recovery
              echo max > \
                ${arrayPath e}/sync_max
            '';
            serviceConfig = commonServiceConfig // {
              Type = "oneshot";
              PrivateNetwork = true;
            };
          })
        cfg.arrays);
    })

    (mkIf cfg.scrub.enable {
      systemd.services = lib.listToAttrs (map
        (e: lib.nameValuePair
          "scrub-${e.device}"
          {
            description = "Scrub ${e.device}";
            inherit (cfg.scrub) startAt;
            unitConfig.ConditionPathExists = [
              "/dev/${e.device}"
              (arrayPath e)
            ];
            serviceConfig = commonServiceConfig // {
              Type = "oneshot";
              ExecStart = "${pkgs.runtimeShell} -c 'echo scrub > ${arrayPath e}/sync_action'";
              PrivateNetwork = true;
            };
          })
        cfg.arrays);
    })

    (mkIf cfg.temperature.enable {
      boot.kernelModules = optional (!cfg.temperature.preferHddtemp) "drivetemp";

      hardware.sensor.hddtemp = {
        enable = cfg.temperature.preferHddtemp;
        drives = map diskPath cfg.drives;
      };

      networking.firewall.allowedTCPPorts = [
      ]
      ++ optional (cfg.openFirewall && cfg.temperature.preferHddtemp) 7634;
    })
  ];
}

ElvishJerricco · 2022-12-06T03:34:59Z

I'm not a fan of the name hardware.raid.swraid.monitor if only because it expands to "hardware raid software raid" :P Something like services.mdmonitor or services.mdcheck would probably be better.

It's also probably better to use the upstream service units and patch in the correct paths. You can still do something like systemd.services.mdcheck_start.environment.MDADM_CHECK_DURATION = cfg.checkDuration, because NixOS will add that as a drop-in unit, which should override the env variable from the upstream unit.

aanderse · 2022-12-06T21:06:17Z

It's also probably better to use the upstream service units and patch in the correct paths. You can still do something like systemd.services.mdcheck_start.environment.MDADM_CHECK_DURATION = cfg.checkDuration, because NixOS will add that as a drop-in unit, which should override the env variable from the upstream unit.

👍

mfenniak · 2022-12-07T02:12:02Z

@ElvishJerricco @aanderse Thanks for the feedback! I wasn't aware of the drop-in unit concept -- I've adjusted the implementation to now use the upstream systemd units (patched), and renamed the module to services.mdcheck.

nixos-discourse · 2022-12-13T00:38:25Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1544

aanderse · 2022-12-13T03:10:20Z