Announcement

Collapse
No announcement yet.

Short Essay - File System Terms, BTRFS on devices, and it's SUBVOLUMES and SNAPSHOTS

Collapse
This topic is closed.
X
This is a sticky topic.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Short Essay - File System Terms, BTRFS on devices, and it's SUBVOLUMES and SNAPSHOTS

    More and more users here are starting to use the BTRFS file system, many at my encouragement. I thought it was a good time to discuss some of the reasons why BTRFS is so great and some of the differences between BTRFS and the default EXT4 file systems. Today a primer on:

    File System Terms, How BTRFS Works with Devices, and SUBVOLUMES

    I brought these topics together because they are related to each other.

    First, a little bit on basic file system terminology:
    • DEVICE = A hard drive, USB drive, SD card, etc.
    • PARTITION = A divided section on a device.
    • PARTITION TABLE = The list and definitions (locations) of partitions on a device.
    • FORMAT verb = The action taken to create a file system on a partition or device.
    • FORMAT adjective = The type of file system on a partition or device.
    • FILE SYSTEM = The methods and data structures that an operating system uses to keep track of files on a device or partition; i.e. the way the files are stored on a device.


    This is important because using the correct terms helps tremendously with communication and understanding. Most of us incorrectly say we mount a partition. However, we never actually mount a partition, you can only mount a file system. Here's how it works: You want to divide a device into sections so you create partitions on it. Then you create a file system (format) on a partition. Then you mount the file system. Then you can begin using it.

    BTRFS handles partitions and devices slightly differently than most file systems. Most file systems require and wholly occupy a partition - even if you're using the entire device. For example, you buy a new hard drive (a DEVICE) and install it into your computer;
    • Using EXT4: You must first create a partition table and at least one partition - even if you have no intention of dividing the available space. Assuming this drive's device name is SDB (the second SATA or SCSI device), each partition would have it's own device name as SDB followed by the partition number: SDB1, SDB2, and so on. After partitioning, you would have to create a file system on each partition. Then you could mount and use the space on the drive for files.
    • Using BTRFS: You may first create a partition table and at least one partition if you wish, but you are not required to. BTRFS can occupy an entire device. You can skip the partition table and go directly to creating (formatting) your BTRFS file system on the entire SDB device.


    For example, here's the output of "lsblk" from my server:

    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    sda 8:0 0 5.5T 0 disk
    sdb 8:16 0 1.8T 0 disk
    ├─sdb1 8:17 0 1007K 0 part
    ├─sdb2 8:18 0 55.9G 0 part
    └─sdb3 8:19 0 1.8T 0 part
    sdc 8:32 0 5.5T 0 disk
    sdd 8:48 0 1.8T 0 disk
    ├─sdd1 8:49 0 1007K 0 part
    ├─sdd2 8:50 0 55.9G 0 part
    └─sdd3 8:51 0 1.8T 0 part
    sde 8:64 0 55.9G 0 disk
    ├─sde1 8:65 0 1007K 0 part
    └─sde2 8:66 0 55.9G 0 part
    Notice that sda and sdc are not partitioned and have no partition table. These two drives are a single BTRFS file system. The other three drives are partitioned.


    BTRFS SUBVOLUMES


    Partitions can be quite useful. With devices of 1 terabyte or more, using all the space in a single file system can be cumbersome and difficult to organize. Partitioning seems like a necessity. BTRFS can replace partitioning with SUBVOLUMES. Similar to partitions with their separate file systems, subvolumes can be individually mounted. One big advantage to this approach is while portions of data are segregated (separately existing) from other portions of data just like with partitioning, all the free space on the file system is available to ALL of the subvolumes that are on the same file system. The subvolumes share the file system space without sharing data.

    If you've ever had to resize a couple of partitions or had to wipe a hard drive clean and start over because your partitions were not the right sizes, you can see that subvolumes are a huge advantage over partitions in this regard. Subvolumes resize themselves as the amount of data they hold increases or decreases. The only limit is the size of the BTRFS file system the subvolumes reside on. Subvolume-to-subvolume data is kept separate, but free space is shared. Partitions can't do this. Subvolumes can be created or deleted instantly and at will without risking other data or lengthy and dangerous movement of data along with partition boundaries.

    Subvolumes can even be easily transferred from one BTRFS file system to another. To do this, BTRFS uses the send and receive commands. You can send a subvolume to a file or you can receive it on another BTRFS file system with a single command.


    BTRFS SNAPSHOTS

    Snapshots are a form of subvolume. The snapshot command makes a functioning copy of a source subvolume on the same BTRFS file system. You can mount, view and edit the files within it, delete or add files to within the snapshot just like you could with the source subvolume. When a snapshot is initially created, it takes no data space because at this point the data is not actually copied. Instead, a copy of the metadata (the location information of each file) is held in the snapshot. As changes are made to either the source subvolume or the snapshot, the changes are stored so the snapshot can begin to grow and start to use data space. One of the effects of this is if you delete a file or number of files from the source subvolume you would not see a corresponding increase in available space on the file system. This is because, although you've deleted a file from the source subvolume, the snapshot retains it's metadata and now the deleted file(s) are in the snapshot subvolume only. If you edit or delete files in the snapshot rather than in the source subvolume the same thing occurs - the deleted file remains in the source subvolume although it has been deleted from the snapshot.

    So what's the point or purpose of a snapshot?
    Snapshots can protect you from accidental deletions: You can retrieve a file deleted from the source subvolume from the previously made snapshot (or vice-versa).

    Snapshots can protect you from a failed upgrade or installation: Prior to upgrading your video driver or your install version, take a snapshot. If the upgrade doesn't go well - you can actually delete the source subvolume, replace it with the snapshot, and be right back where you were before the upgrade just moments before.

    Snapshots are also part of the backup process. To create a full copy of a subvolume and put it on different file system for backup purposes, you first take a read-only snapshot of a subvolume. The read-only status insures no data within the snapshot itself is manipulated during the send procedure which might cause data corruption. You can "send" the snapshot directly to another btrfs file system or if the backup device is not using the btrfs file system (or is only available over a network connection), you can send the snapshot to a file. The file then can be "received" to a btrfs file system later or on another computer.

    Conclusion
    With BTRFS and subvolumes and their snapshots, you can make easy and simple backups, prevent unintended consequences from upgrade or install operations, and protect yourself from your own mistakes. All of this and more right at your fingertips!

    A couple of final notes: If you use the whole disk for btrfs instead of a partition, you can't boot from that disk. btrfs will occupy the space normally reserved for grub. BTRFS can also span multiple devices for a single filesystem like LVM or RAID. This topic will be covered another time.
    Last edited by oshunluvr; Apr 14, 2017, 03:00 PM.
    Please Read Me
    Be not the first by whom the new are tried, Nor yet the last to lay the old aside. - Alexander Pope, An Essay on Criticism, 1711

    #2
    I did my first Incremental backup today.


    I used the @_bkup-2017-04-20 ro snapshot to create the @_backup-2017-04-21 snapshot, and the @home_bkup-2017-04-20 to make the @home_bkup-2017-04-21 snapshot.


    btrfs send -p /mnt/snapshots/@home_bkup-2017-04-20 /mnt/snapshots/@home_bkup-2017-04-21 | btrfs receive /backup/snapshots
    btrfs send -p /mnt/snapshots/@_bkup-2017-04-20 /mnt/snapshots/@_bkup-2017-04-21 | btrfs receive /backup/snapshots


    I plan to use the previous incremental backup to make the next one.

    Then, out of curiosity, I decided to mount the latest backup of my April 14th @home volume:
    :~$ sudo -i
    [sudo] password for jerry:


    root@jerry-Aspire-V3-771:~# mount /dev/sdb /backup

    root@jerry-Aspire-V3-771:~# btrfs su list /backup/snapshots

    ID 257 gen 91 top level 5 path snapshots
    ID 258 gen 79 top level 257 path @_bkup-2017-04-14
    ID 271 gen 83 top level 257 path @home_bkup-2017-04-14
    ID 346 gen 83 top level 257 path @_bkup-2017-04-20
    ID 347 gen 91 top level 257 path @home_bkup-2017-04-20


    root@jerry-Aspire-V3-771:~# umount /backup


    root@jerry-Aspire-V3-771:~# mount -o subvolid=271 /dev/sdb /backup


    root@jerry-Aspire-V3-771:~# vdir /backup
    total 0
    drwxr-xr-x 1 jerry jerry 21468 Apr 14 18:33 jerry
    root@jerry-Aspire-V3-771:~# vdir /backup/jerry/
    total 2431252
    -rw-r----- 1 jerry jerry 150528 Oct 10 2010 2009PlantList.xls
    -rw-r----- 1 jerry jerry 212611 Mar 3 15:47 2366WPocket_Bike_Brochure.pdf
    -rwxr-x--- 1 jerry jerry 49579 Oct 7 2009 3D_Least_Squares_land_leveling_solution.odt
    -rw-r----- 1 jerry jerry 31866 Aug 22 2015 486px-ECGafleidingen.jpg
    -rwxr-x--- 1 jerry jerry 514624 Jan 31 2004 4_wheel_tractor_patent.jpg
    -rw-r--r-- 1 jerry jerry 33450 Jul 28 2014 Bow_Down_smiley.gif
    -rw-r----- 1 jerry jerry 1584 Jul 20 2010 bp_video_script.txt
    -rw-r----- 1 jerry jerry 40062 Sep 21 2015 btrbk-0.20.0.tar.bz2
    ....
    .... LOTS and LOTS of files and directories ...
    ....
    -rw-r----- 1 jerry jerry 1123 Jul 4 2010 wrt54gl_hang_fix.txt
    -rw-r----- 1 jerry jerry 1597 Jul 4 2010 wrt54gl_hang_fix.txt~
    -rwxr-x--- 1 jerry jerry 3028316 Sep 7 2009 WRT54GL_V11_UG_C-Web,0.pdf
    -rwxr-x--- 1 jerry jerry 3028316 Apr 2 2009 WRT54GL_V11_UG_C-Web.pdf
    root@jerry-Aspire-V3-771:~# root@jerry-Aspire-V3-771:~# umount /backup


    root@jerry-Aspire-V3-771:~#
    [#]BTRFS[/#]
    Last edited by GreyGeek; Sep 22, 2017, 12:01 PM.
    "I would rather have questions that can't be answered, than answers that can't be questioned." ― Richard Feynman

    Comment

    Working...
    X