Announcement

Collapse
No announcement yet.

My newest BTRFS backup script - I found a new bash function - arrays

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    My newest BTRFS backup script - I found a new bash function - arrays

    I rearranged my server which means rearranging the backup automation scripts. I now have 2 backup drives that are 6TB and the media storage drive is 10TB, so I want to divide the 15 subvolumes across the 2 backup drives. Basically, I'm left with source, backup1, and backup2 and I want subvolumes 1-15 on different backups drives.

    My previous script didn't work this way so I began anew. The first riddle was "Can I have one script that successfully divides the subvolumes between the two backup devices?" I'm not so concerned with current sizes of the subvolume because they grow and shrink as needed but I wanted to use all the capacity I have available.

    Here's what I came up with. I removed everything but the parts that pertain to this post. I would like comments if there's a better way to do it:

    Code:
    declare -a set=(@Audio @Backups @Documents @Downloads @Home_Movies @Incoming @Movies @Music @Music_Videos @Pictures @Projects @TV_Shows @Videos @WWW)
    declare -a set1=(@Audio @Backups @Documents @Downloads @Home_Movies @Incoming @Music @Music_Videos @Pictures @Projects @Videos @WWW)
    
    for subvol in ${set[@]} ; do  
    if [[ ${set1[@]} = *"$subvol"* ]]; then backup_mount=backup1_mount ; else backup_mount=backup2_mount ; fi
    "set" = all the subvolume names, "set1" = those I want to go to backup1, any missing from set1 are sent to backup2.

    It's my first time using an array variable this way. Works like I want and amazingly short. I wanted to avoid repetitive code or dual scripts. If I decide later to re-arrange the backups or add more subvolumes I only have to add or subtract from the "set" and "set1" arrays.

    It occurred to me that I could have the "set" array built from a simple "ls" command. Then any newly appearing subvolumes would be automatically added to the process and sent to backup2.

    Or I could even adjust the target backup location by using a trailing number in the subvolume name - like a 1 or 2 at the end of the name to set the backup location or read the list from a text file instead of in the script itself, but I doubt I'll be changing this around much.

    Please Read Me

    #2
    Wow, you do good work, Stuart! Very impressive.

    Comment


      #3
      Indeed!
      I always used Python when I wanted to use arrays. I wasn't aware that bash offered them. So, I went searching for them:
      https://www.thegeekstuff.com/2010/06...The+Geek+Stuff
      "If the index number is @ or *, all members of an array are referenced. "
      "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
      – John F. Kennedy, February 26, 1962.

      Comment


        #4
        I can see a lot of functionality by using arrays. I have a couple other scripts that need re-writes too.

        Please Read Me

        Comment


          #5
          Your array use has a potential problem. It is not seen in your example, but if you removed @Music from set1, your test will still find it in set1 because it matches @Music_Videos.

          There isn't a simple way to check if something is in a bash array that I like, there's only what seems to me tricky, or involved:
          • mark the end of each item with something sure to be not found in the names, f.ex.
            ${set1[@]/%/ZZZ} = *${subvol}ZZZ*
          • use regex matching with spaces or something around the names:
            " ${set1[@]} " =~ " $subvol "
          • define a function, then use that in the test
            Code:
            is_in () {local x item=$1;shift;for x;do [[ "$x" = "$item" ]] && return 0;done; return 1; }
            if is_in $subvol "${set1[@]}"; then ...
          Regards, John Little

          Comment


            #6
            Originally posted by jlittle View Post
            Your array use has a potential problem. It is not seen in your example, but if you removed @Music from set1, your test will still find it in set1 because it matches @Music_Videos.

            There isn't a simple way to check if something is in a bash array that I like, there's only what seems to me tricky, or involved:
            • mark the end of each item with something sure to be not found in the names, f.ex.
              ${set1[@]/%/ZZZ} = *${subvol}ZZZ*
            • use regex matching with spaces or something around the names:
              " ${set1[@]} " =~ " $subvol "
            • define a function, then use that in the test
              Code:
              is_in () {local x item=$1;shift;for x;do [[ "$x" = "$item" ]] && return 0;done; return 1; }
              if is_in $subvol "${set1[@]}"; then ...
            I expected you'd have some sage advice for me

            Nice catch, I didn't think of that - "Music" exists in two names. Also "Videos" and "Movies" does too.

            So I ran a test. Before the test I had already decided to swap the "sets" because set2 (the group going to backup2) was much smaller - only @Movies and @TV_Shows. For the test , I added @Music to set2. Here's the test script:
            Code:
            #!/bin/bash
            
            declare -a set=(@Alt_Movies @Audio @Backups @Documents @Downloads @Home_Movies @Incoming @Movies @Music @Music_Videos @Pictures @Projects @TV_Shows @Videos @WWW)
            declare -a set2=(@Movies @TV_Shows @Music)
            
            for subvol in${set[@]};do
            if[[${set2[@]}= *"$subvol"* ]];thenecho"set2"$subvol;elseecho"set1"$subvol;fi
            
            done
            
            exit 0
            and the output:
            Code:
            [FONT=monospace][COLOR=#000000]root@server:~# ./autotest    [/COLOR]
            set1 @Alt_Movies
            set1 @Audio
            set1 @Backups
            set1 @Documents
            set1 @Downloads
            set1 @Home_Movies
            set1 @Incoming
            set2 @Movies
            set2 @Music
            set1 @Music_Videos
            set1 @Pictures
            set1 @Projects
            set2 @TV_Shows
            set1 @Videos
            set1 @WWW
            
            [/FONT]
            So clearly, "@Music" is in set 2 and "@Music_Videos" is in set 1.
            Last edited by oshunluvr; May 25, 2018, 06:32 AM.

            Please Read Me

            Comment


              #7
              Thinking maybe the order would matter, I added "@Music_test" near the top of the list and "@Music_test2" near the bottom, but the results look accurate:
              Code:
              set1 @Alt_Movies
              set1 @Audio
              set1 @Music_test
              set1 @Backups
              set1 @Documents
              set1 @Downloads
              set1 @Home_Movies
              set1 @Incoming
              set2 @Movies
              set2 @Music
              set1 @Music_Videos
              set1 @Pictures
              set1 @Projects
              set1 @Music_test2
              set2 @TV_Shows
              set1 @Videos
              set1 @WWW

              Please Read Me

              Comment


                #8
                Finally, I added "@Music_test2 to the front of "set2" list and:
                Code:
                root@server:~# ./autotest    
                set1 @Alt_Movies
                set1 @Audio
                set2 @Music_test
                set1 @Backups
                set1 @Documents
                set1 @Downloads
                set1 @Home_Movies
                set1 @Incoming
                set2 @Movies
                set2 @Music
                set1 @Music_Videos
                set1 @Pictures
                set1 @Projects
                set2 @Music_test2
                set2 @TV_Shows
                set1 @Videos
                set1 @WWW
                it FAILED. Both @Music_test2 and @Music_test1 appeared in the set2 list.

                Please Read Me

                Comment


                  #9
                  Clearly something is wacky. Strange that it seems to work fine until I get to the @Music_test vs @Music_test2. Also, if I change "@Music_test" to "@Music_test1" it doesn't fail.

                  Finally, I changed "@Music" in set 2 to "@Music_Videos" and they both listed as set 2. Seems the order of the sets controls the outcome.

                  Anyway: Adding the spaces as JLittle suggests, changing this:
                  Code:
                  if [[ ${set2[@]} = *"$subvol"* ]];
                  to this
                  Code:
                  if [[ " ${set2[@]} " = *" $subvol "* ]];
                  Seems to work fine.
                  Last edited by oshunluvr; May 25, 2018, 06:49 AM.

                  Please Read Me

                  Comment

                  Working...
                  X