Script for Nightly SnapRAID Syncs and Scrubs

UPDATE: I have an updated script available here.

I needed a simple, reliable way to run diffs, syncs, and scrubs on my SnapRAID array nightly without me manually having to run commands each time. Β This SnapRAID script does that, and will stop if there are issues, and it provides a nightly email alert of the status of the array. Β Let’s take a look at getting this setup.

You will likely not want to use this as is, because your parity files are likely named differently than mine. Also, I use Mutt to send out system emails in lieu of the /usr/bin/mail like the default script uses. If you want to use Mutt, you just need to install it. I have updated my updated script again for 6.0.

apt-get install mutt

I normally make a scripts directory for my root user, and put all scripts there.

sudo -i
mkdir /root/scripts
nano /root/scripts/snapraid_diff_n_sync.sh

Here is the revised script… again. I have added rudimentary scrubbing capabilities to the script. I had it coded for incremental and full scrubbing, but after getting it working, the idea seemed stupid to me, so I rolled back to a weekly whole array scrub. Here’s the revised version. Let me know if you run into any issues or have ideas for improvements in the comments.

#!/bin/bash
#######################################################################
# This is a helper script that keeps snapraid parity info in sync with
# your data and optionally verifies the parity info. Here's how it works:
#   1) It first calls diff to figure out if the parity info is out of sync.
#   2) If parity info is out of sync, AND the number of deleted files exceed
#      X (configurable), it triggers an alert email and stops. (In case of
#      accidental deletions, you have the opportunity to recover them from
#      the existing parity info)
#   3) If partiy info is out of sync, AND the number of deleted files exceed X
#      AND it has reached/exceeded Y (configurable) number of warnings, force
#      a sync. (Useful when you get a false alarm above and you can't be bothered
#      to login and do a manual sync. Note the risk is if its not a false alarm
#      and you can't access the box before Y number of times the job is run  to
#      fix the issue... Well I hope you have other backups...)
#   4) If parity info is out of sync BUT the number of deleted files did NOT
#      exceed X, it calls sync to update the parity info.
#   5) If the parity info is in sync (either because nothing changed or after it
#      has successfully completed the sync job, it runs the scrub command to
#      validate the integrity of the data (both the files and the parity info).
#      Note that each run of the scrub command will validate only a (configurable)
#      portion of parity info to avoid having a long running job and affecting
#      the performance of the box.
#   6) Once all jobs are completed, it sends an email with the output to user
#      (if configured).
#

#
# CHANGELOG
# ---------
# 23/10/2011 Initial release
# 04/01/2015 Updated script to handle changes in SnapRAID v7.0
#            Added scrub job as an optional task (after diff and sync)
# 06/01/2015 Made the script more robust by adding checks to make sure preceding
#            jobs completed as expected before continuing with the subsequent jobs.
#            Made emailing output to user optional.
# 24/01/2015 Inserted a sed step to clean up crlf (aka dos/unix formatting issue)
#            in sync & scrub outputs.
#            Detect sync and scrub job failures and highlight to user via warning
#            subject line in email to user.
# 25/01/2015 Added option to reduce progress report output in email (default is 2 -
#            report only in 10% intervals).
# 26/01/2015 For terse = 2 setting, removed lines for 1-8% from output
# 05/02/2015 Added logic to perform forced sync after X number of warnings
#            Cleaned up formatting in script file (changed tabs to spaces)
#            Made consistent the use of [ in the test statements
# 08/02/2015 Added warning number to the email subject line so that it is easier to
#            tell how many warnings have been issued so far
# 04/03/2015 Corrected Scrub job status check (i.e. added check for text "Nothing
#            to do") to avoid sending false warning email
# 27/10/2015 Corrected Sync job status check (i.e. added check for text "Nothing to
#            do") to avoid sending false warning email
# 29/10/2015 Fixed a bug with the job status check not detecting the right strings
#
#######################################################################
# Expand PATH for smartctl
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

## USER DEFINED SETTINGS ##
# address where the output of the jobs will be emailed to.
# comment it out to disable email output
EMAIL_ADDRESS="user@gmail.com"

# Set the threshold of deleted files to stop the sync job from running.
# NOTE that depending on how active your filesystem is being used, a low
# number here may result in your parity info being out of sync often and/or
# you having to do lots of manual sync.
DEL_THRESHOLD=50

# Set number of warnings before we force a sync job.
# This option comes in handy when you cannot be bothered to manually
# start a sync job when DEL_THRESHOLD is breached due to false alarm.
# Set to 0 to ALWAYS force a sync (i.e. ignore the delete threshold above)
# Set to -1 to NEVER force a sync (i.e. need to manual sync if delete threshold is breached)
SYNC_WARN_THRESHOLD=3

# Set percentage of array to scrub if it is in sync.
# i.e. 0 to disable and 100 to scrub the full array in one go
# WARNING - depending on size of your array, setting to 100 will take a very long time!
SCRUB_PERCENT=3
SCRUB_AGE=10

# Set the option to log SMART info. 1 to enable, any other values to disable
SMART_LOG=1

# this script will log its actions to a file at this location
LOG_FILE="/tmp/snapRAID.log"
# location of the snapraid binary
SNAPRAID_BIN="/usr/local/bin/snapraid"
# location of the mail program binary
MAIL_BIN="/usr/bin/mutt"

# how much progress output do we want to keep in email
# Default is 2 which means report progress in 10% intervals
# Set to 1 to report progress in 1% intervals
# Set to 0 to report everything
TERSE=2

## INTERNAL TEMP VARS ##
EMAIL_SUBJECT_PREFIX="[`hostname`] SnapRAID - "
TMP_OUTPUT="/tmp/snapRAID.out"
SYNC_WARN_FILE="/tmp/snapRAID.warnCount"
SYNC_WARN_COUNT=""

# auto determine names of content and parity files
#CONTENT_FILE=`cat /etc/snapraid.conf | grep snapraid.content | head -n 1 | cut -d " " -f2`
#PARITY_FILE=`cat /etc/snapraid.conf | grep snapraid.parity | head -n 1 | cut -d " " -f2`
CONTENT_FILE=`grep -v '^$\|^\s*\#' /etc/snapraid.conf | grep snapraid.content | head -n 1 | cut -d " " -f2`
PARITY_FILE=`grep -v '^$\|^\s*\#' /etc/snapraid.conf | grep snapraid.parity | head -n 1 | cut -d " " -f2`

# redirect all stdout to log file (leave stderr alone thou)
exec >> $LOG_FILE

# timestamp the job
echo "[`date`] SnapRAID Job started."
echo "SnapRAID DIFF Job started on `date`" > $TMP_OUTPUT
echo "----------------------------------------" >> $TMP_OUTPUT

#TODO - mount and unmount parity disk on demand!

#sanity check first to make sure we can access the content and parity files
if [ ! -e $CONTENT_FILE ]; then
  echo "[`date`] ERROR - Content file ($CONTENT_FILE) not found!"
  echo "ERROR - Content file ($CONTENT_FILE) not found!" >> $TMP_OUTPUT
  exit 1;
fi

if [ ! -e $PARITY_FILE ]; then
  echo "[`date`] ERROR - Parity file ($PARITY_FILE) not found!"
  echo "ERROR - Parity file ($PARITY_FILE) not found!" >> $TMP_OUTPUT
  exit 1;
fi

# run the snapraid DIFF command
echo "[`date`] Running DIFF Command."
$SNAPRAID_BIN diff >> $TMP_OUTPUT
# wait for the above cmd to finish
wait

echo "----------------------------------------" >> $TMP_OUTPUT
echo "SnapRAID DIFF Job finished on `date`" >> $TMP_OUTPUT
JOBS_DONE="DIFF"

DEL_COUNT=$(grep -w '^ \{1,\}[0-9]* removed$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
ADD_COUNT=$(grep -w '^ \{1,\}[0-9]* added$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
MOVE_COUNT=$(grep -w '^ \{1,\}[0-9]* moved$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
COPY_COUNT=$(grep -w '^ \{1,\}[0-9]* copied$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
UPDATE_COUNT=$(grep -w '^ \{1,\}[0-9]* updated$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)

# sanity check to make sure that we were able to get our counts from the output of the DIFF job
if [ -z "$DEL_COUNT" -o -z "$ADD_COUNT" -o -z "$MOVE_COUNT" -o -z "$COPY_COUNT" -o -z "$UPDATE_COUNT" ]; then
  # failed to get one or more of the count values, lets report to user and exit with error code
  echo "[`date`] ERROR - failed to get one or more count values. Unable to proceed. Exiting script."
  if [ $EMAIL_ADDRESS ]; then
    $MAIL_BIN -s "$EMAIL_SUBJECT_PREFIX WARNING - Unable to proceed with SYNC/SCRUB job(s). Check DIFF job output inside." "$EMAIL_ADDRESS" < $TMP_OUTPUT
  fi
  exit 1;
fi

echo "SUMMARY of changes - Added [$ADD_COUNT] - Deleted [$DEL_COUNT] - Moved [$MOVE_COUNT] - Copied [$COPY_COUNT] - Updated [$UPDATE_COUNT]" >> $TMP_OUTPUT

# check if the conditions to run SYNC are met
# CHK 1 - if files have changed
if [ $DEL_COUNT -gt 0 -o $ADD_COUNT -gt 0 -o $MOVE_COUNT -gt 0 -o $COPY_COUNT -gt 0 -o $UPDATE_COUNT -gt 0 ]; then
  # CHK 1 - YES, files have changed
  # CHK 2 - if number of deleted files exceed DEL_THRESHOLD
  if [ $DEL_COUNT -lt $DEL_THRESHOLD ]; then
    # CHK 2 - NO, delete threshold not reached, lets run the sync job
    echo "Deleted files ($DEL_COUNT) did not exceed threshold ($DEL_THRESHOLD), proceeding with sync job." >> $TMP_OUTPUT
    echo "[`date`] Changes detected [A-$ADD_COUNT,D-$DEL_COUNT,M-$MOVE_COUNT,C-$COPY_COUNT,U-$UPDATE_COUNT] and deleted files ($DEL_COUNT) is below threshold ($DEL_THRESHOLD). Running SYNC Command."
    DO_SYNC=1
  else
    #CHK 2 - YES, delete threshold breached! print warning message to both outputs
    echo "Number of deleted files ($DEL_COUNT) exceeded threshold ($DEL_THRESHOLD)." >> $TMP_OUTPUT
    echo "[`date`] WARNING - Deleted files ($DEL_COUNT) exceeded threshold ($DEL_THRESHOLD). Check $TMP_OUTPUT for details."
    # CHK 3 - if forced sync is set
    if [ $SYNC_WARN_THRESHOLD -gt -1 ]; then
      # CHK 3 - YES
      echo "Forced sync is enabled." >> $TMP_OUTPUT
      echo "[`date`] Forced sync is enabled."
      # CHK 4 - if number of warnings has exceeded threshold
      SYNC_WARN_COUNT=$(sed 'q;/^[0-9][0-9]*$/!d' $SYNC_WARN_FILE 2>/dev/null)
      SYNC_WARN_COUNT=${SYNC_WARN_COUNT:-0} #value is zero if file does not exist or does not contain what we are expecting
      if [ $SYNC_WARN_COUNT -ge $SYNC_WARN_THRESHOLD ]; then
        # CHK 5 - YES, lets force a sync job. Do not need to remove warning marker here as it is automatically removed when the sync job is run by this script
        echo "Number of warning(s) ($SYNC_WARN_COUNT) has reached/exceeded threshold ($SYNC_WARN_THRESHOLD). Forcing a sync job to run." >> $TMP_OUTPUT
        echo "[`date`] Number of warning(s) ($SYNC_WARN_COUNT) has reached/exceeded threshold ($SYNC_WARN_THRESHOLD). Forcing a sync job to run."
        DO_SYNC=1
      else
        # CHK 4 - NO, so let's increment the warning count and skip the sync job
        ((SYNC_WARN_COUNT += 1))
        echo $SYNC_WARN_COUNT > $SYNC_WARN_FILE
        echo "$((SYNC_WARN_THRESHOLD - SYNC_WARN_COUNT)) warning(s) till forced sync. NOT proceeding with sync job." >> $TMP_OUTPUT
        echo "[`date`] $((SYNC_WARN_THRESHOLD - SYNC_WARN_COUNT)) warning(s) till forced sync. NOT proceeding with sync job."
        DO_SYNC=0
      fi
    else
      # CHK 3 - NO, so let's skip SYNC
      echo "Forced sync is not enabled. NOT proceeding with sync job. Please run sync manually if this is not an error condition." >> $TMP_OUTPUT
      echo "[`date`] Forced sync is not enabled. Check $TMP_OUTPUT for details. NOT proceeding with sync job."
      DO_SYNC=0
    fi
  fi
else
  # CHK 1 - NO, so let's skip SYNC
  echo "[`date`] No change detected. Not running SYNC job."
  DO_SYNC=0
fi

# Now run sync if conditions are met
if [ $DO_SYNC -eq 1 ]; then
  echo "SnapRAID SYNC Job started on `date`" >> $TMP_OUTPUT
  echo "----------------------------------------" >> $TMP_OUTPUT
  $SNAPRAID_BIN sync | sed -e 's/\r/\n/g' >> $TMP_OUTPUT
  #wait for the job to finish
  wait
  echo "----------------------------------------" >> $TMP_OUTPUT
  echo "SnapRAID SYNC Job finished on `date`" >> $TMP_OUTPUT
  JOBS_DONE="$JOBS_DONE + SYNC"
  # insert SYNC marker to 'Everything OK' or 'Nothing to do' string to differentiate it from SCRUB job later
  sed -i 's/^Everything OK/SYNC_JOB--Everything OK/g;s/^Nothing to do/SYNC_JOB--Nothing to do/g' $TMP_OUTPUT
  # Remove any warning flags if set previously. This is done in this step to take care of scenarios when user has manually synced or restored deleted files and we will have missed it in the checks above.
  if [ -e $SYNC_WARN_FILE ]; then
    rm $SYNC_WARN_FILE
  fi
  $SNAPRAID_BIN scrub -p new
fi

# Moving onto scrub now. Check if user has enabled scrub
if [ $SCRUB_PERCENT -gt 0 ]; then
  # YES, first let's check if delete threshold has been breached and we have not forced a sync.
  if [ $DEL_COUNT -gt $DEL_THRESHOLD -a $DO_SYNC -eq 0 ]; then
    # YES, parity is out of sync so let's not run scrub job
    echo "[`date`] Scrub job cancelled as parity info is out of sync (deleted files threshold has been breached)."
  else
    # NO, delete threshold has not been breached OR we forced a sync, but we have one last test -
    # let's make sure if sync ran, it completed successfully (by checking for our marker text "SYNC_JOB--" in the output).
    if [ $DO_SYNC -eq 1 -a -z "$(grep -w "SYNC_JOB-" $TMP_OUTPUT)" ]; then
      # Sync ran but did not complete successfully so lets not run scrub to be safe
      echo "[`date`] WARNING - check output of SYNC job. Could not detect marker . Not proceeding with SCRUB job."
      echo "WARNING - check output of SYNC job. Could not detect marker . Not proceeding with SCRUB job." >> $TMP_OUTPUT
    else
      # Everything ok - let's run the scrub job!
      echo "[`date`] Running SCRUB Command."
      echo "SnapRAID SCRUB Job started on `date`" >> $TMP_OUTPUT
      echo "----------------------------------------" >> $TMP_OUTPUT
      $SNAPRAID_BIN scrub -p $SCRUB_PERCENT -o $SCRUB_AGE | sed -e 's/\r/\n/g' >> $TMP_OUTPUT
      #wait for the job to finish
      wait
      echo "----------------------------------------" >> $TMP_OUTPUT
      echo "SnapRAID SCRUB Job finished on `date`" >> $TMP_OUTPUT
      JOBS_DONE="$JOBS_DONE + SCRUB"
      # insert SCRUB marker to 'Everything OK' or 'Nothing to do' string to differentiate it from SYNC job above
      sed -i 's/^Everything OK/SCRUB_JOB--Everything OK/g;s/^Nothing to do/SCRUB_JOB--Nothing to do/g' $TMP_OUTPUT
    fi
  fi
else
  echo "[`date`] Scrub job is not scheduled. Not running SCRUB job."
fi

# Moving onto logging SMART info if enabled
if [ $SMART_LOG -eq 1 ]; then
  $SNAPRAID_BIN smart >> $TMP_OUTPUT
  wait
fi

echo "Spinning down disks..." >> $TMP_OUTPUT
$SNAPRAID_BIN down

# all jobs done, let's send output to user if configured
if [ $EMAIL_ADDRESS ]; then
  echo "[`date`] Email address is set. Sending email report to <$EMAIL_ADDRESS>"
  # check if deleted count exceeded threshold
  if [ $DEL_COUNT -gt $DEL_THRESHOLD -a $DO_SYNC -eq 0 ]; then
    # YES, lets inform user with an appropriate subject line
    $MAIL_BIN -s "$EMAIL_SUBJECT_PREFIX WARNING $SYNC_WARN_COUNT - Number of deleted files ($DEL_COUNT) exceeded threshold ($DEL_THRESHOLD)" "$EMAIL_ADDRESS" < $TMP_OUTPUT
  elif [ -z "${JOBS_DONE##*"SYNC"*}" -a -z "$(grep -w "SYNC_JOB-" $TMP_OUTPUT)" ]; then
    # Sync ran but did not complete successfully so lets warn the user
    $MAIL_BIN -s "$EMAIL_SUBJECT_PREFIX WARNING - SYNC job ran but did not complete successfully" "$EMAIL_ADDRESS" < $TMP_OUTPUT
  elif [ -z "${JOBS_DONE##*"SCRUB"*}" -a -z "$(grep -w "SCRUB_JOB-" $TMP_OUTPUT)" ]; then
    # Scrub ran but did not complete successfully so lets warn the user
    $MAIL_BIN -s "$EMAIL_SUBJECT_PREFIX WARNING - SCRUB job ran but did not complete successfully" "$EMAIL_ADDRESS" < $TMP_OUTPUT
  else
    # OPTIONALLY, let's reduce the amount of status lines in output.
    if [ $TERSE -gt 1 ]; then
      # Report progress in interval of tens %
      sed -i '$!N; /^\([0-9]\).*\n\1.*$/!P; D'  $TMP_OUTPUT
      sed -i '/^[1-8]%.*$/d'  $TMP_OUTPUT
    elif [ $TERSE -gt 0 ]; then
      # Report progress in interval of ones %
      sed -i '$!N; /^\([0-9]*\)%.*\n\1.*$/!P; D'  $TMP_OUTPUT
    fi
    $MAIL_BIN -s "$EMAIL_SUBJECT_PREFIX INFO - $JOBS_DONE Jobs COMPLETED" "$EMAIL_ADDRESS" < $TMP_OUTPUT
  fi
fi

echo "[`date`] All jobs ended."

exit 0;

When you are done, make it executable.

chmod +x /root/scripts/snapraid_diff_n_sync.sh

Finally, add it to the root crontab to run every night at 11:30pm.

crontab -e

and paste...

# Run a SnapRAID diff and then sync
30 23 * * * /root/scripts/snapraid_diff_n_sync.sh

Zack

I love learning new things and trying out the latest technology.

You may also like...

39 Responses

  1. lockheed says:

    Hi. I wondered if you could help me out. I tried using this script, after adopting it to my arch box, but after it run for few minutes, it quits with the following:

    # /root/scripts/snapraid_diff_n_sync.sh
    WARNING! UUID is unsupported for disks: 'd1', 'd2', 'd3', 'd4'. Move operations won't be optimal.
    Failed to readlink '/dev/block/0:69'.
    Failed to resolve device '0:69'.
    Smart is unsupported in this platform.
    Failed to readlink '/dev/block/0:69'.
    Failed to resolve device '0:69'.
    Spindown is unsupported in this platform.
    Error sending message, child exited 127 (Exec error.).
    

    I doubt it has anything to do with the fact that my data partitions are on BTRFS, while parity disk is ext4.

    • Zack says:

      What output do you get when you when snapraid diff or snapraid sync. Also, if this is a brand new snapraid array, you will want to run a full sync in a tmux session first before you try to use this script via cronjob.

  2. Syphonx says:

    Hey, was wondering if you could help with the following error I get when the script runs:

    [Sat Sep 17 04:30:01 SAST 2016] Running DIFF Command.
    [Sat Sep 17 04:30:01 SAST 2016] ERROR – failed to get one or more count values. Unable to proceed. Exiting script.

    I can run the diff command fine manually, there are usually a couple hundred changes every night due to Sonarr updating NFO’s. I can also run sync fine manually so not sure why the script is failing.

    Diff Command:

    WARNING! Inodes are not persistent for disks: ‘d2’, ‘d3’, ‘d4’, ‘d5’, ‘d6’. Move operations won’t be optimal.

    169121 equal
    6 added
    0 removed
    453 updated
    0 moved
    0 copied
    0 restored
    There are differences!

    Anything I need to edit in this part of the script?

    # run the snapraid DIFF command
    echo "[`date`] Running DIFF Command."
    $SNAPRAID_BIN diff >> $TMP_OUTPUT
    # wait for the above cmd to finish
    wait
    
    echo "----------------------------------------" >> $TMP_OUTPUT
    echo "SnapRAID DIFF Job finished on `date`" >> $TMP_OUTPUT
    JOBS_DONE="DIFF"
    
    DEL_COUNT=$(grep -w '^ \{1,\}[0-9]* removed$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
    ADD_COUNT=$(grep -w '^ \{1,\}[0-9]* added$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
    MOVE_COUNT=$(grep -w '^ \{1,\}[0-9]* moved$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
    COPY_COUNT=$(grep -w '^ \{1,\}[0-9]* copied$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
    UPDATE_COUNT=$(grep -w '^ \{1,\}[0-9]* updated$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
    
    # sanity check to make sure that we were able to get our counts from the output of the DIFF job
    if [ -z "$DEL_COUNT" -o -z "$ADD_COUNT" -o -z "$MOVE_COUNT" -o -z "$COPY_COUNT" -o -z "$UPDATE_COUNT" ]; then
      # failed to get one or more of the count values, lets report to user and exit with error code
      echo "[`date`] ERROR - failed to get one or more count values. Unable to proceed. Exiting script."
      if [ $EMAIL_ADDRESS ]; then
        $MAIL_BIN -s "$EMAIL_SUBJECT_PREFIX WARNING - Unable to proceed with SYNC/SCRUB job(s). Check DIFF job output inside." "$EMAIL_ADDRE$
      fi
      exit 1;
    fi
    
  3. codgedodger says:

    Amazing script! Thanks Zack, one question though. It will email me perfectly, but in the email with all the information laid out nicely (thanks for that) I have this stated at the end. Now granted this is a new snapraid array and doesn’t have much data on it yet but just curious what it means by this:

    —————————————-
    SnapRAID SYNC Job finished on Mon Oct 3 23:33:00 EDT 2016
    WARNING – check output of SYNC job. Could not detect marker . Not proceeding with SCRUB job.

  4. codgedodger says:

    root@ubuntu14:~# snapraid scrub
    Self test…
    Loading state from /var/snapraid.content…
    Using 8 MiB of memory for the FileSystem.
    Initializing…
    Scrubbing…
    Using 112 MiB of memory for 32 blocks of IO cache.
    Nothing to do
    Saving state to /var/snapraid.content…
    Saving state to /media/data/disk1/snapraid.content…
    Saving state to /media/data/disk2/snapraid.content…
    Verifying /var/snapraid.content…
    Verifying /media/data/disk1/snapraid.content…
    Verifying /media/data/disk2/snapraid.content…
    root@ubuntu14:~#

    • codgedodger says:

      When snapraid sync was ran it would come with this error

      WARNING! All the files previously present in disk ‘d2’ at dir ‘/media/data/disk2/’
      are now missing or rewritten!

      Just ran snapraid –force-empty sync there’s nothing it could harm since this whole 85TB array only has 300gb worth of content on it which is easily replaced if worst comes to worst. I’m letting it start the sync right now. Might be awhile. lol

  5. mtompkins says:

    Recommend a minor change:

    #CONTENT_FILE=`cat /etc/snapraid.conf | grep snapraid.content | head -n 1 | cut -d " " -f2`
    #PARITY_FILE=`cat /etc/snapraid.conf | grep snapraid.parity | head -n 1 | cut -d " " -f2`
    CONTENT_FILE=`grep -v '^$\|^\s*\#' /etc/snapraid.conf | grep snapraid.content | head -n 1 | cut -d " " -f2`
    PARITY_FILE=`grep -v '^$\|^\s*\#' /etc/snapraid.conf | grep snapraid.parity | head -n 1 | cut -d " " -f2`
    

    This will ignore any comment lines that may exist. When I make a change in a config file often I comment out the original during testing until I’m sure of the expected result. This will allow your search to continue to work during that interim phase.

    • Zack says:

      Thanks! I actually have an even more involved version of this on my local copy, because I’m using SnapRAID v11 Beta’s split parity functionality like this.

      PARITY_FILE1=`cat /etc/snapraid.conf | grep "^[^#;]" | grep parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 1`
      PARITY_FILE2=`cat /etc/snapraid.conf | grep "^[^#;]" | grep parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 2`
      PARITY_FILE3=`cat /etc/snapraid.conf | grep "^[^#;]" | grep 2-parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 1`
      PARITY_FILE4=`cat /etc/snapraid.conf | grep "^[^#;]" | grep 2-parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 2`
      

      I would agree that yours is a useful addition, so I will update the tutorial. Thanks!

      • mtompkins says:

        Thanks for that – when we all move to v11 it will definitely come in handy.

        I thought I would also share my version of the script which evolved from your core.
        Major differences (that come to mind):
        – Parsed and created functions for manageability
        – Added programmatic management of services
        — As an example, I run Sophos AV on-access scanning. This greatly impacts the snapraid processes as files are accessed. I’ve added a routine to stop & start the av (which can easily be extended to any systemctl services). The trap (next bullet) restores the service(s) on a clean exit or if CTRL-C
        – Added a trap for ‘cleanup’ (restoring services)
        – Moved to markdown / HTML email (and have put in a request for snapraid to export graphs and tables in that format)

        Feel free to incorporate anything you like. URL: https://gist.github.com/mtompkins/91cf0b8be36064c237da3f39ff5cc49d

        • Zack says:

          Thanks for the post! I have really intended to break out this script into functions to make it more readable. This has been the case where it works for me, so I haven’t fiddled with it πŸ™‚ There are still a few more pieces in your script that I would like to “functionize”, but this is awesome, and a good examples. I have already extended my local version to manage pausing/unpausing certain Docker containers while the script starts and stops, so this is another great inclusion on your part. The idea of integrating a markdown output into SnapRAID is also great (I saw your post on the forums).

        • Zack says:

          After looking at this script more. This is very well done! I added only one function to pull that out of the main function. Also, I changed the snapraid bin path to point to /usr/local/bin/snapraid like the default installed version. I also replaced mailx with mutt again and removed the markdown support on my local version (I just need simple, plain text here).

          function get_counts() {
                  DEL_COUNT=$(grep -w '^ \{1,\}[0-9]* removed$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
                  ADD_COUNT=$(grep -w '^ \{1,\}[0-9]* added$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
                  MOVE_COUNT=$(grep -w '^ \{1,\}[0-9]* moved$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
                  COPY_COUNT=$(grep -w '^ \{1,\}[0-9]* copied$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
                  UPDATE_COUNT=$(grep -w '^ \{1,\}[0-9]* updated$' $TMP_OUTPUT | sed 's/^ *//g' | cut -d ' ' -f1)
          }
          

          Finally, I changed the search for the parity files to create an array, and then iterate through it to test the existence of all parity files. This could be simplified further, but it would make it more difficult to read and understand.

          PARITY_FILES[0]=`cat /etc/snapraid.conf | grep "^[^#;]" | grep parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 1`
          PARITY_FILES[1]=`cat /etc/snapraid.conf | grep "^[^#;]" | grep parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 2`
          PARITY_FILES[2]=`cat /etc/snapraid.conf | grep "^[^#;]" | grep 2-parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 1`
          PARITY_FILES[3]=`cat /etc/snapraid.conf | grep "^[^#;]" | grep 2-parity | head -n 1 | cut -d " " -f 2 | cut -d "," -f 2`
          
          echo "Testing that all parity files are present..."
          for i in "${PARITY_FILES[@]}"
          do
            if [ ! -e $i ]; then
              echo "[`date`] ERROR - Parity file ($i) not found!"
              echo "ERROR - Parity file ($i) not found!" >> $TMP_OUTPUT
              exit 1;
            fi 
          done
          
          echo "All parity files found."
          
        • Zack says:

          I have posted a revised version of your script, with credit, as a new post. I have included all of the modifications I made to support dual parity and the things I mentioned previously here.

          http://zackreed.me/snapraid-split-parity-sync-script/

          • mtompkins says:

            I really liked the old script but think your new version is a significant improvement in readability and management. Kudos to your efforts all around. Thanks also for the credit.

            I’m sure the community will appreciate the changes.

            [Aside: I also found myself borrowing from your drive standby script which works well in concert with snapraid – thx again!]

          • Zack says:

            Thanks for all of updates with the new script. It works very well πŸ™‚

  6. rebels1405 says:

    This blog has been awesome! I can’t thank you enough for taking the time to put this together. It has made it so much easier for someone with very little linux experience to put together a nice server. I do have one question that I hope you can help me with…I had my server set up and everything was working well with the automated nightly sync and e-mails that I would get each night. Due to to a networking issue, I had to set everything back up from scratch, the snapraid drives all did amazing and they picked back up like nothing had even happened. I can’t tell you how relieved I was that I didn’t lose anything even with a complete new OS set up from scratch. Here is my question: I followed all of the steps again, but for some reason, none of my cron jobs are running anymore. I followed your instructions again, and I had the same issue last time where they did’t work, but I stumbled on the answer somewhere else, but I can’t seem to figure it out again. Any ideas on what I can try to get my scripts running each night? I was logged in a root when I created the /root/scripts and when I did the crontab -e setup. I can run the scripts manually and I get the e-mail and everything runs, but they are not running automatically. Any ideas? Thanks for everything!

    • Zack says:

      Hello, sorry for the delayed response. Have you set the script as executable?

      chmod +x /root/scripts/snapraid_diff_n_sync.sh

      Also, you may want to expand your available paths in your crontab. Try to add this above the minute, hour, day, etc line…

      PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
      
      # m h  dom mon dow   command
      
  7. phuriousgeorge says:

    I see you’re stopping nzbget and sonarr. Would it be possible to add a user-configurable array of services to stop during a sync? I’m working on adding specific functions for other docker services I have running, but thought it might be good to be able to easily configure later on.

    Again, great work, I absolutely love this blog and habe shared it amongst similar enthusiasts!

  8. lockheed says:

    Great script, but I am having a problem. I need to rise a threshold level of removed files, because it does not work as is:
    SnapRAID DIFF Job finished on Sun 15 Jan 05:56:59 CET 2017
    SUMMARY of changes – Added [12] – Deleted [143] – Moved [0] – Copied [0] – Updated [0]
    Number of deleted files (143) exceeded threshold (50).
    Forced sync is enabled.
    1 warning(s) till forced sync. NOT proceeding with sync job.

    How can I do that?

    • Zack says:

      You just need to modify the script and increase the value in this line to be greater than 143.

      DEL_THRESHOLD=50
      
      DEL_THRESHOLD=150
      
    • Zack says:

      Or, you can just manually run the sync by hand one time if you have deleted a bunch of files. This is probably safer than modifying the script just for one run.

      snapraid diff
      snapraid sync
      
  9. Dulanic says:

    Should the “service” idea be clarified? I had to dig into the code to determine if it meant a true service or a container.

    • Zack says:

      Hello, thanks for the comment. I have the new version of this script < ahref="http://zackreed.me/snapraid-split-parity-sync-script/" target="_blank">here that mentions these are for Docker containers. Although, I would love to create scripts that covers ever scenario, these scripts are just things I use myself that I publish for others when I’m not working my real job or with my wife and kids πŸ™‚

      If you have ideas, I’ll happily listen, but offering code yourself would be much more helpful. In both scripts, managing systemd services could be easily accomplished by removing the Docker components and adding calls to /usr/sbin/service.

  10. farrinux says:

    Zack

    I have a small bug with “Script for Nightly SnapRAID Syncs and Scrubs” It’s with the email. I use sSMTP to send mail. The script works except for how it treats sSMTP. sSMTP requires a “To:” a “From” and then the “Subject” when sending. When I run the script it does not set the email address correctly. Even though the email variable is set correctly. I’ve included the error message below. (See Last Line)
    “WARNING! Content files on the same disk: ‘/var/snapraid.content’ and ‘/var/snapraid.2content’.
    WARNING! Content files on the same disk: ‘/sraid1/snapraid.content’ and ‘/sraid1/snapraid.2content’.
    WARNING! Content files on the same disk: ‘/sraid2/snapraid.content’ and ‘/sraid2/snapraid.2content’.
    WARNING! Content files on the same disk: ‘/media1/snapraid.content’ and ‘/media1/snapraid.2content’.
    ssmtp: RCPT TO: (553 5.1.2 COMPLETED@gmail.com> is not a valid RFC-5321 address. n16sm4110560ywh.105 – gsmtp)”
    The “COMPLETED” on the first part of the email address is being pulled From “$EMAIL_SUBJECT_PREFIX INFO – $JOBS_DONE Jobs COMPLETED” I think the problem is with the order of the “subject ” and “Email Address” but not sure.
    One last thing, as you can see above from the errors, that I got warnings about the “Content” files. I set snapraid up to have 2 content files but I put both content files on several different disks, is this allowable? ie. each disk has a content 1 and content2 file.

  11. mon0 says:

    sorry, forgot to add the error
    Oct 16 07:15:48 nas sSMTP[30416]: RCPT TO: (501 5.1.3 Bad recipient address syntax)

    My previous snapraid install was on 18.04 and everything was running smooth.

  12. skyforce3 says:

    Hei Bro is this script work with standart dual parity, not split parity

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.