Announcement

Collapse
No announcement yet.

Server automation and script tricks - share your favs...

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Server automation and script tricks - share your favs...

    Some of us have self-built and programmed servers at home or at our disposal. One of the benefits to do-it-yourself servers is the lack of limits to what you can do with it.

    I was updating a little script on mine sever and thought, maybe if I share it;
    • someone else will find it useful
    • I might get suggestions on how to make it better
    • I might get some new ideas from others


    So here's my first contribution to this thread:

    Notification of needed server package updates

    One of my daily tasks is to check my server and see if it needs any package updates. I prefer not to fully automate the update process completely to avoid system breakage - I want my finger on the trigger so I can see what happens. My habit was to log into my server via ssh, run "sudo apt update && sudo apt list --upgradable" and see what needed upgrading. I got tired of doing this every morning, so I wrote this little bash script an added it as a root cronjob. Now every morning at 8 am my server lets me know what if anything is upgradable.

    Here's the script:
    Code:
    [FONT=monospace][COLOR=#000000]#!/bin/bash[/COLOR]
    #
    apt update >/dev/null 2>&1
    apt list --upgradable > upgrade.list
    if [[ "$(cat /root/upgrade.list)" != "Listing..." ]]
    then ssh office "DISPLAY=:0 notify-send -i package-install -t 0 'Server has updates waiting:' '`sed 's/\/.*//' /root/upgrade.list`'"
    else ssh office "DISPLAY=:0 notify-send -i package-install -t 0 'Server is up-to-date:' 'There are no pending updates.'"
    fi
    [/FONT]
    Each line of the code does this:
    1. run "apt update" to freshen the package lists
    2. generate a list of pending updates and save it
    3. compare the saved list to an empty list (no updates in the list)
    4. if the list is non-empty, send the list ( with some trimming) to my office computer via notify-send
    5. if the list is empty, tell me that too
    6. done


    Here's what the notification looks like;
    Click image for larger version

Name:	Screenshot_20171004_092103.png
Views:	1
Size:	17.9 KB
ID:	649237

    I set the messages to not time out so anytime I get to my office, the message is waiting. I can review the list at my leisure and run any needed update when I'm ready.

    ssh has to be set up both ways (server-to-office and office-to-server) to enable the message sending. I use a secure key method to allow ssh access so no passwords are stored in plain text.

    I run this as root which could be considered insecure. If I were more cautious, I would enable the needed commands in sudoer for my server username and do it all that way. Either way, you'd have to know the password(s) to gain access to my desktop so I'm not bothering at this point.

    If it matters, my server is currently on Ubuntu Server 14.04 and desktop KDEneon user. I do have one package - Plex Media Server - that's not from a ppa and wouldn't bring down the entire server if it failed, so it updates itself automatically via another script and cronjob.

    I'm interested in comments and seeing if anyone else has something to contribute...

    Please Read Me

    #2
    Here's one I wrote today to automatically make daily snapshots and weekly backups of my server install. It's brand new so lightly tested and very specific to my setup. For example, I use two drive bootable partitions as backups for my server install, so there's several duplicated commands to cover making both backups and cleaning up the environment. The next step for this one is to automatically edit the backups after sending them so they are bootable as-is with no manual edits. If you follow my other btrfs posts, you may know that UUIDs would need to be edited to boot to a transmitted btrfs backup...

    Code:
    [FONT=monospace][COLOR=#000000]#!/bin/bash[/COLOR]
    #
    # set variables
    distro="@Ubuntu_Server_14_04"
    source_mount="/mnt/sde2/"
    backup1_mount="/mnt/sdb2/"
    backup2_mount="/mnt/sdd2/"
    source_device="/dev/sde2/"
    backup1_device="/dev/sdb2/"
    backup2_device="/dev/sdd2/"
    removeday=_`date +%y%m%d --date="-2 day"`
    backupremoveday=_`date +%y%m%d --date="-7 day"`
    today=_`date +%y%m%d`
    source="$source_mount""$distro"
    oldsnap="$source_mount""$distro""$removeday"
    oldbackup1snap="$backup1_mount""$distro"_ro"$backupremoveday"
    oldbackup2snap="$backup2_mount""$distro"_ro"$backupremoveday"
    newsnap="$source_mount""$distro""$today"
    backupsnap="$source_mount""$distro"_ro"$today"
    
    # verify all devices mounted and mount if not.
    if mountpoint $source_mount
    then
        :
    else  
        mount $source_device $source_mount
    fi
    
    if mountpoint $backup1_mount
    then
        :
    else  
        mount $backup1_device $backup1_mount
    fi
    
    if mountpoint $backup2_mount
    then
        :
    else  
        mount $backup2_device $backup2_mount
    fi
    
    # Take new snapshot. If sunday, then take read-only snapshot and send to backups
    if [ `date +%w` = 7 ]
    then
        btrfs su sn -r "$source" "$backupsnap"
        btrfs send "$backupsnap" | btrfs receive "$backup1_mount"
        btrfs send "$backupsnap" | btrfs receive "$backup2_mount"
        if [ -d $oldbackupsnap1 ]
        then  
            btrfs su de -c $oldbackup1snap
        fi
        if [ -d $oldbackup2snap ]
        then  
            btrfs su de -c $oldbackup2snap
        fi
        ssh office "DISPLAY=:0 notify-send -i package-install -t 0 'Server backup:' 'Backup made and sent. Old backups removed.'"
    else
        btrfs su sn "$source" "$newsnap"
        if [ -d $oldsnap ]
        then  
            btrfs su de -c $oldsnap
        fi
        ssh office "DISPLAY=:0 notify-send -i package-install -t 0 'Server snapshot:' 'Daily snapshot made. Old snapshot removed.'"
    fi
    
    exit 0
    [/FONT]
    I set it up to make a regular snapshot every day and keep the last two snapshots. On Sundays, it will (should) take a read-only snapshot and send it to my two backup file systems on other drives.

    [#]btrfs[/#]
    Last edited by oshunluvr; Oct 06, 2017, 06:08 AM. Reason: fix naming inconsistencies, discovered typos

    Please Read Me

    Comment


      #3
      Here's what I added to the above script in an attempt to make the backups bootable:

      Code:
      # If Sunday, make backup snaps bootable and remove old backups, saving one previous
      source_UUID=`lsblk -no UUID "$source_device"`
      backup1_UUID=`lsblk -no UUID "$backup1_device"`
      backup2_UUID=`lsblk -no UUID "$backup2_device"`
      backup1_fstab="$backup1_mount"/etc/fstab
      backup2_fstab="$backup2_mount"/etc/fstab
      backup1_grubcfg="$backup1_mount"/boot/grub/grub.cfg
      backup2_grubcfg="$backup2_mount"/boot/grub/grub.cfg
      
      
      if [ `date +%w` = 7 ]
      then
          mv "$backup1_mount""$distro" "$backup1_mount""$distro".old
          mv "$backup2_mount""$distro" "$backup2_mount""$distro".old
          btrfs su sn "$backup1_mount""$backupsnap" "$backup1_mount""$distro"
          btrfs su sn "$backup2_mount""$backupsnap" "$backup2_mount""$distro"
          btrfs su de -c btrfs su sn "$backup1_mount""$backupsnap"
          btrfs su de -c btrfs su sn "$backup2_mount""$backupsnap"
      # attempt to make backup bootable
          sed -i "s@$source_UUID@$backup1_UUID@" "$backup1_fstab"
          sed -i "s@$source_UUID@$backup2_UUID@" "$backup2_fstab"
          sed -i "s@$source_UUID@$backup_1UUID@" "$backup1_grubcfg"
          sed -i "s@$source_UUID@$backup_2UUID@" "$backup2_grubcfg"
      fi
      I haven't tested this yet.
      Last edited by oshunluvr; Oct 04, 2017, 02:23 PM. Reason: fixed naming inconsistencies

      Please Read Me

      Comment


        #4
        Interesting scripts.

        Your choice of making the Sunday snapshots ro and the weekday snapshots rw is to avoid the necessity of having to convert a ro snapshot to a rw snapshot that can be booted?
        "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
        – John F. Kennedy, February 26, 1962.

        Comment


          #5
          Originally posted by GreyGeek View Post
          Your choice of making the Sunday snapshots ro and the weekday snapshots rw is to avoid the necessity of having to convert a ro snapshot to a rw snapshot that can be booted?
          I figure the only need to make an RO snapshot is to send it as a backup and I don't really have a need to make a backup of the server OS more than that. I rarely do much to it other than updates. I figure a snapshot will save me from a failed update and a backup will save me from a failed drive. I upgrade almost daily so the snapshot frequency seemed adequate enough but the backup wasn't required as often. Really, it's just my personal level of fault tolerance for the install.

          The data is all in RAID1 configuration on 4 hard drives so I don't back that up. I realize "RAID1 isn't a backup" but I'm the only user that does much with any of the data except those of my users that watch the videos and access the music and look at the family photos. Personal data in is the /var/www/nextcloud folder and that's backed up with the install.

          Please Read Me

          Comment


            #6
            I kept two of my three HDs in a RAID1 configuration but switched back to two HDs without RAID because RAID1, in effect, made two HDs have the space of one. I, too, keep ro snapshots for send&receive to my 3rd HD and for my backups on /mnt/snapshots as well, because a snapshot of an ro as an rw @ or @home works great if I have to rollback or replace a drive.
            "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
            – John F. Kennedy, February 26, 1962.

            Comment


              #7
              Previous post edited due to typo found in script:

              I put "sn de -c" instead of "su de -c" in three places. I guess that the danger of using the short-hand version of the commands.


              I also amended my original so I'm keeping 2 backups rather than just one.

              Please Read Me

              Comment


                #8
                OK, I rearranged and shortened the autosnapshot script and added remarks. Still mostly untested;

                autosnapshot
                Code:
                **[I][COLOR=#000000]code removed**[/COLOR][/I][FONT=monospace]
                [/FONT]
                I also increased the number of saved snapshots and backups by one each and corrected a couple errors I found.

                [#]btrfs[/#]
                Last edited by oshunluvr; Oct 30, 2017, 07:58 AM.

                Please Read Me

                Comment


                  #9
                  # Verify it's backup day;
                  if [ `date +%w` = 7 ]
                  Oops! Sunday is 0, not 7

                  If you want to try out this script, edit the above to this"
                  # Verify it's backup day;
                  if [ `date +%w` = 0 ]

                  Please Read Me

                  Comment


                    #10
                    I also realized that Sunday backup operation didn't delete the three-day old snapshot and would result in a back-log of snapshots. Also, the snapshot taken for backup purposes is read-only thus might not be the best for use if you needed to revert to it to boot.

                    I decided it made more sense to snapshot everyday for potential roll-back uses and take a separate read-only snapshot for backup use - deleting it after the backup operation.

                    The other thing that needs to be addressed is the fact that this really only works if your computer is on everyday, all day. I did write it for my server, but if there was a power outage, the result could potentially be extra older snapshots and backups not being properly cleaned.

                    Newest version:
                    Code:
                    **[I][COLOR=#000000]code removed**[/COLOR][/I]
                    EDITS: I had to change the device names from "/dev/sde2/" to "/dev/sde2" to make the lsblk command work. I also had an extra "else" in there.
                    Last edited by oshunluvr; Oct 30, 2017, 07:59 AM.

                    Please Read Me

                    Comment


                      #11
                      A revision to the previous autosnapshot script. I found a major flaw (a single space in the wrong place that resulted in the deletion of my boot subvolume rather than the backup) and revised the backup functionality a bit make it work as intended and be more logical:
                      Code:
                      [FONT=monospace]#!/bin/bash
                      
                      # Log the script activity
                      exec 1> >(logger -s -t $(basename $0)) 2>&1
                      
                      # Set the variables
                      # Distro subvolume name to snapshot and backup;
                      distro="@Ubuntu_Server_14_04"
                      
                      # Source and backup parent file system mount points and devices with their UUIDs;
                      source_mount="/mnt/sde2/"
                      backup1_mount="/mnt/sdb2/"
                      backup2_mount="/mnt/sdd2/"
                      source_device="/dev/sde2"
                      backup1_device="/dev/sdb2"
                      backup2_device="/dev/sdd2"
                      source_UUID=`lsblk -no UUID "$source_device"`
                      backup1_UUID=`lsblk -no UUID "$backup1_device"`
                      backup2_UUID=`lsblk -no UUID "$backup2_device"`
                      
                      # Dates to do things;
                      today=_`date +%y%m%d`
                      ## Age in days to remove snapshots.
                      snapremoveday=_`date +%y%m%d --date="-3 day"`
                      ## Age in days to keep backups.
                      backupkeepday=_`date +%y%m%d --date="-7 day"`
                      ## Age in days to remove backups.
                      backupremoveday=_`date +%y%m%d --date="-14 day"`
                      
                      # Assemble the above variables into single variables;
                      ## The source is the full path and distro subvolume name combined;
                      source="$source_mount""$distro"
                      ## Snapshot name is the source with today's date appended;
                      newsnap="$source""$today"
                      ## Snapshot to be deleted is the source with the snapshot removal date appended;
                      oldsnap="$source""$snapremoveday"
                      ## Name for read-only snapshot used for backups is the source with _ro and today's date appended;
                      newbackup="$distro"_ro"$today"
                      ## Name for the oldest backup is the subvolume name with the backup removal date appended;
                      oldbackup="$distro""$backupremoveday"
                      
                      # Verify the parent file system is mounted and mount it if not;
                      if ! mountpoint -q "$source_mount"
                      then  
                          mount "$source_device" "$source_mount"
                      fi
                      
                      # Delete the oldest snapshot;
                      if [ -d "$oldsnap" ]
                          then  
                             btrfs su de -c "$oldsnap"
                          fi
                      
                      # Take a new read-write snapshot;
                          btrfs su sn "$source" "$newsnap"
                      
                      # For backups; Take a read-only snapshot and send it to the backup
                      # locations, make backup snapshots bootable, and delete oldest backups.
                      
                      # Verify it's backup day;
                      if [ `date +%w` = 0 ]
                      then
                      
                      # Take a read-only snapshot
                        btrfs su sn -r "$source" "$source_mount""$newbackup"
                      
                      # Verify the backup file systems are mounted and mount them it if not;
                          if ! mountpoint -q "$backup1_mount"
                          then
                              mount "$backup1_device" "$backup1_mount"
                          fi
                          if ! mountpoint -q "$backup2_mount"
                          then
                              mount "$backup2_device" "$backup2_mount"
                          fi
                      
                      # Delete oldest backups;
                          if [ -d "$backup1_mount""$oldbackup" ]
                          then
                              btrfs su de -c "$backup1_mount""$oldbackup"
                          fi
                          if [ -d "$backup2_mount""$oldbackup" ]
                          then
                              btrfs su de -c "$backup2_mount""$oldbackup"
                          fi
                      
                      # Rename previous week's backups by adding backupkeepday;
                          mv "$backup1_mount""$distro" "$backup1_mount""$distro""$backupkeepday"
                          mv "$backup2_mount""$distro" "$backup2_mount""$distro""$backupkeepday"
                      ## The renaming prepares the oldest backup to be deleted next week and renames  
                      ## the current backup so we're ready to receive the newest backup.
                      
                      # Send the read-only snapshot to the backup locations;
                          btrfs send "$source_mount""$newbackup" | btrfs receive "$backup1_mount"
                          btrfs send "$source_mount""$newbackup" | btrfs receive "$backup2_mount"
                      
                      # Take a read-write snapshot of the received backups and using the distro name;
                          btrfs su sn "$backup1_mount""$newbackup" "$backup1_mount""$distro"
                          btrfs su sn "$backup2_mount""$newbackup" "$backup2_mount""$distro"
                      
                      # Delete read-only snapshots because they are no longer needed;
                          btrfs su de -c "$backup1_mount""$newbackup"
                          btrfs su de -c "$backup2_mount""$newbackup"
                          btrfs su de -c "$source_mount""$newbackup"
                      
                      # Edit the UUIDs in grub and fstab so the backups are bootable;
                      ### NOT YET VERIFIED AS WORKING###
                          sed -i "s@$source_UUID@$backup1_UUID@" "$backup1_mount""$distro"/etc/fstab
                          sed -i "s@$source_UUID@$backup2_UUID@" "$backup2_mount""$distro"/etc/fstab
                          sed -i "s@$source_UUID@$backup1_UUID@" "$backup1_mount""$distro"/boot/grub/grub.cfg
                          sed -i "s@$source_UUID@$backup2_UUID@" "$backup2_mount""$distro"/boot/grub/grub.cfg
                      ## This sed command is a replace-in-place action to change the UUIDs from the source to  
                      ## the backup file systems. The goal here is to make the backups bootable.
                      fi
                      
                      # Notify of completion of tasks;
                      # Verify it's backup day;
                      if [ `date +%w` = 0 ]
                      # Notify that backup tasks are complete and the old backups and snapshot are deleted;
                      then
                          ssh office "DISPLAY=:0 notify-send -i package-install -t 0 'Server backup:' 'Backup made and sent. Old backups and snapshot removed.'"
                      else
                      # Notify that a snapshot has been taken and the old one deleted;
                          ssh office "DISPLAY=:0 notify-send -i package-install -t 0 'Server snapshot:' 'Daily snapshot made. Old snapshot removed.'"
                      fi
                      
                      # Script complete
                      exit 0
                      
                      [/FONT]

                      Please Read Me

                      Comment


                        #12
                        Log file from my server showing my script activity:
                        Oct 29 04:30:01 server CRON[6061]: (root) CMD (/root/autosnapshot)
                        Oct 29 04:30:02 server autosnapshot: Create a snapshot of '/mnt/sde2/@Ubuntu_Server_14_04' in '/mnt/sde2/@Ubuntu_Server_14_04_171029'
                        Oct 29 04:30:02 server autosnapshot: Create a readonly snapshot of '/mnt/sde2/@Ubuntu_Server_14_04' in '/mnt/sde2/@Ubuntu_Server_14_04_ro_171029'
                        Oct 29 04:30:02 server autosnapshot: mv: cannot stat ‘/mnt/sdb2/@Ubuntu_Server_14_04’: No such file or directory
                        Oct 29 04:30:02 server autosnapshot: mv: cannot stat ‘/mnt/sdd2/@Ubuntu_Server_14_04’: No such file or directory
                        Oct 29 04:30:02 server autosnapshot: At subvol /mnt/sde2/@Ubuntu_Server_14_04_ro_171029
                        Oct 29 04:30:02 server autosnapshot: At subvol @Ubuntu_Server_14_04_ro_171029
                        Oct 29 04:39:02 server autosnapshot: At subvol /mnt/sde2/@Ubuntu_Server_14_04_ro_171029
                        Oct 29 04:39:02 server autosnapshot: At subvol @Ubuntu_Server_14_04_ro_171029
                        Oct 29 04:48:52 server autosnapshot: Create a snapshot of '/mnt/sdb2/@Ubuntu_Server_14_04_ro_171029' in '/mnt/sdb2/@Ubuntu_Server_14_04'
                        Oct 29 04:48:59 server autosnapshot: Create a snapshot of '/mnt/sdd2/@Ubuntu_Server_14_04_ro_171029' in '/mnt/sdd2/@Ubuntu_Server_14_04'
                        Oct 29 04:48:59 server autosnapshot: Delete subvolume (commit): '/mnt/sdb2/@Ubuntu_Server_14_04_ro_171029'
                        Oct 29 04:49:00 server autosnapshot: Delete subvolume (commit): '/mnt/sdd2/@Ubuntu_Server_14_04_ro_171029'
                        Oct 29 04:49:00 server autosnapshot: Delete subvolume (commit): '/mnt/sde2/@Ubuntu_Server_14_04_ro_171029'
                        and the current state of subvolumes (sde2 being the boot folder and sdb2 and sdd2 backup locations):
                        smith@server:~$ ll /mnt/sde2
                        total 16
                        drwxr-xr-x 1 root root 268 Oct 30 04:30 ./
                        drwxr-xr-x 1 root root 124 Sep 11 09:54 ../
                        drwxr-xr-x 1 root root 292 Oct 11 06:27 @Ubuntu_Server_14_04/
                        drwxr-xr-x 1 root root 292 Oct 11 06:27 @Ubuntu_Server_14_04_171028/
                        drwxr-xr-x 1 root root 292 Oct 11 06:27 @Ubuntu_Server_14_04_171029/
                        drwxr-xr-x 1 root root 292 Oct 11 06:27 @Ubuntu_Server_14_04_171030/
                        smith@server:~$ ll /mnt/sdb2
                        total 16
                        drwxr-xr-x 1 root root 118 Oct 29 04:48 ./
                        drwxr-xr-x 1 root root 124 Sep 11 09:54 ../
                        drwxr-xr-x 1 root root 292 Oct 29 04:39 @Ubuntu_Server_14_04/
                        smith@server:~$ ll /mnt/sdd2
                        total 16
                        drwxr-xr-x 1 root root 118 Oct 29 04:48 ./
                        drwxr-xr-x 1 root root 124 Sep 11 09:54 ../
                        drwxr-xr-x 1 root root 292 Oct 29 04:48 @Ubuntu_Server_14_04/
                        smith@server:~$
                        A success if I say so myself!
                        Last edited by oshunluvr; Oct 30, 2017, 08:01 AM.

                        Please Read Me

                        Comment


                          #13
                          New addition to script: I noticed after posting above that the snapshots did not have a new date/time stamp to reflect when they were created. I suppose this is because I mount using the "noatime" option to reduce unnecessary disk i/o. To fix this I added a touch command below the snapshot command:

                          Code:
                          # Take a new read-write snapshot;
                              btrfs su sn "$source" "$newsnap"
                              touch "$newsnap"
                          This effectively stamps the snapshot when it was made. This makes it more clear to me that all is well. I'll post back the results again next week and the week after to verify the backups are working properly (keeping one old one) and the time stamps are good.

                          Please Read Me

                          Comment


                            #14
                            Clever!
                            "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                            – John F. Kennedy, February 26, 1962.

                            Comment


                              #15
                              Originally posted by GreyGeek View Post
                              Clever!
                              The simplest solution is often the best!

                              Please Read Me

                              Comment

                              Working...
                              X