rclone for cloud backups :]

[wizard10000]

'morning all

Spent the weekend working on a cloud backup solution and I'm fairly happy with the results. In addition to dumping /etc, /root and my home directory to an external drive I spent some time working with rclone and am now also syncing those directories with my Google Drive. Heck, they give you 15GB and my home directory is a little over ten, so no-brainer

Anyway, lookit here - http://rclone.org/

Rclone is a command line program to sync files and directories to and from

Google Drive
Amazon S3
Openstack Swift / Rackspace cloud files / Memset Memstore
Dropbox
Google Cloud Storage
Amazon Drive
Microsoft One Drive
Hubic
Backblaze B2
Yandex Disk
The local filesystem

Features

MD5/SHA1 hashes checked at all times for file integrity
Timestamps preserved on files
Partial syncs supported on a whole file basis
Copy mode to just copy new/changed files
Sync (one way) mode to make a directory identical
Check mode to check for file hash equality
Can sync to and from network, eg two different cloud accounts
Optional encryption (Crypt)
Optional FUSE mount (rclone mount)

In the archive is one binary that goes in /sbin, one man page and one text and one html readme. After much experimenting here what I ended up with -

#!/bin/bash
if [ -h /dev/disk/by-label/external ]
	then
		rsync -aEquX /root /mnt/external/archive
		rsync -aEquX /etc /mnt/external/archive
		rsync -aEquX --exclude-from=/home/wizard/.rsync/exclude /home/wizard /mnt/external/home
fi

rclone --delete-during --transfers=8 --exclude-from /root/.rclone-exclude --stats=5m --log-file=/var/log/rclone.log sync /root remote:private/linux/root

rclone --delete-during --transfers=8 --exclude-from /root/.rclone-exclude --stats=5m --log-file=/var/log/rclone.log sync /etc remote:private/linux/etc

rclone --delete-during --transfers=8 --exclude-from /root/.rclone-exclude --stats=5m --log-file=/var/log/rclone.log sync /home/wizard remote:private/linux/home

exit 0

The rsync job just dumps to an external drive (after insuring the external drive is mounted), the rclone tasks sync local and remote files - if I delete a file from the local machine it will also be deleted from gdrive the next time the job runs - but I still have a local copy on external hard drive until I'm sure I don't need the file any more

It took me a little more than eight hours to copy 11GB of data up to gdrive; subsequent runs take about five minutes.

I'm still working on directories to exclude and there is a difference in exclude format between rclone and rsync - rsync's exclude file looks like this

.cache/
Desktop/
Downloads/
temp/
Videos/
Music/

while rclone's is a little different -

/.cache/**
/Desktop/**
/Downloads/**
/temp/**
/Videos/**
/Music/**

I'm sure there's lots more stuff to exclude so the exclude file is a work in progress, but as you can see I don't back up ~/.cache in either instance.

Documentation was good and the man page has everything you need but the html readme is a nice touch. Although rclone should run just fine out of the box, I elected to create a developer account with Google and assign rclone to a fake project. The defaults will work, but google throttles bandwidth based on developer ID, so if you use the one that comes with rclone you're sharing bandwidth with everyone else that's using the defaults. With my own developer ID uploading averaged about 550KB/sec on a 5Mb outbound pipe, so as mentioned it took a bit more than eight hours for initial sync.

Although it appears you can use rclone to *mount* the remote store, I prefer to just run the thing as a backup job. One thing that made me a little sad, though - rclone does not follow links at all. Symnlinking everything I wanted to backup into a single directory would have been pretty cool, but is not happening - at least not with the current release.

Anyway, I think it's pretty cool. If you're allergic to Google you can use another service and if you're *really* concerned you could pipe rclone through whatever encryption method you cared to use.

cheers -