Announcement

Collapse
No announcement yet.

Would like some assistance

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    [SOLVED] Would like some assistance

    For me, understanding btrfs is voodoo magic. I just have a difficult time wrapping my head around it. That said, I have my laptop formatted with btrfs. I also have an external 500 GB HDD, also formatted with btrfs that has nothing on it.

    I'd like to play with btrfs full and incremental backups, and that's where I'd like some assistance. If one of you btrfs witch doctors would be so kind as to write two scripts (or two .bash_aliases file entries) that would let me manually do full backups and incremental backups, I'd be indebted. I don't want anything complicated. KISS. The laptop isn't a production unit, and it doesn't have anything important on it.

    If one of you is game, let me know what you need to know about my laptop and the external HDD.
    Windows no longer obstructs my view.
    Using Kubuntu Linux since March 23, 2007.
    "It is a capital mistake to theorize before one has data." - Sherlock Holmes

    #2
    Here is the script that I use when I don't do it manually.
    Code:
    #!/bin/bash
    #
    # script to create backup snapshots to /mnt/snapshots and a differential backup to /backup
    # To be run as root from /
    #
    echo "Mounting drives"
    eval "mount /dev/disk/by-uuid/038bd596-59b4-4bb0-98e5-3cbeec1dd339 /mnt"
    eval "mount /dev/disk/by-uuid/c16e0247-d866-47f3-8d5e-770e746cd51c /backup"
    
    NOW=$(date +%Y%m%d%H%M)
    echo "Making today's snapshot"
    MKSNAP='btrfs su snapshot -r /mnt/@ /mnt/snapshots/@'$NOW
    eval $MKSNAP
    eval 'sync'
    eval 'sync'
    
    echo "Finding previous snapshot as parent "
    PREVSNAP=""
    list=($(ls /mnt/snapshots/))
    PREVSNAP=${list[-2]}
    NOW=${list[-1]}
    echo "Attempting incremental backup"
    if [[ -s "/mnt/snapshots/"$PREVSNAP ]];
    then
      MKINC='btrfs send -p /mnt/snapshots/'$PREVSNAP
      MKINC=$MKINC' /mnt/snapshots/'$NOW
      MKINC=$MKINC' | btrfs receive /backup'
      echo $MKINC
      eval $MKINC
      eval 'sync'
      eval 'sync'
      eval "sync"
      DELSNAP='btrfs subvol delete -C /mnt/snapshots/'${list[0]}
      eval $DELSNAP
      eval 'sync'
      eval 'sync'
      eval 'sync'
      list=''
      list=($(ls /backup))
      DELSNAP='btrfs subvol delete -C /backup/'${list[0]}
      eval $DELSNAP
      eval 'sync'
      eval 'sync'
      eval 'sync'
      echo "Snapshots completed, oldest snapshots deleted"
      eval 'umount /backup'
      eval 'umount /mnt'
      eval 'sync'
      eval 'sleep 1'
      eval 'sync'
      eval 'sleep 1'
      eval 'sync'
      eval 'sleep 1'
      echo "Drives unmounted and 3 syncs done"
    else
      echo 'Incremental backup failed using '$PREVSNAP' and '$NOW
      echo 'Drives still mounted, ready to clean up'
    fi
    The "PREVSNAP" has to exist on the rootfs subvolume and on the backup subvolume.

    Here is what one of my incremental snapshots looked like:
    btrfs send -p /mnt/snapshots/@202202262040 /mnt/snapshots/@202202281609 | btrfs receive /backup

    The @202202262040, taken on Feb 26, 2022 at 8:40PM, resided on both /mnt/snapshots and /backup. It was sent to the /backup drive by my previous incremental send command. It is the parent snapshot. The snapshot taken on Feb 28, 2022 at 4:09PM is the child snapshot. The receive portion of the incremental command will see the name of the parent snapshot and look for it on the subvolume /backup. Using it as a template, the send portion of the command sends ONLY THE DIFFERENCE between the parent and child snapshots to the receive command. The receive command takes those differences and builds the child snapshot on /backup.

    My external 500Gb spinner is plugged in before I run the script. I do not mount it. The script does that. It mounts the external drive to /backup, creates the child snapshot (which becomes the latest snapshot) and then using the previous snapshot, as determined by the list command, the incremental snapshot is taken.

    I also have an internal NVMe 1Tb SSD which I mount manually on occasions to make incremental snapshots that may be 3 or more days between the parent and child. Even so, the incremental snapshot rarely takes more than 15 to 20 seconds to complete. Usually less than 10 secs.

    Note that in order to use the send command the parent and child snapshots must be "read only", that is, made using the -r switch. My /mnt/snapshot contains:
    Code:
    :~# !1996
    mount /dev/sda3 /mnt
    root@jerry-hp17cn1xxx:~# vdir /mnt/snapshots/
    total 0
    drwxr-xr-x 1 root root 334 Feb 10 16:28 @202202242057
    drwxr-xr-x 1 root root 334 Feb 10 16:28 @202202251913
    drwxr-xr-x 1 root root 334 Feb 10 16:28 @202202262040
    drwxr-xr-x 1 root root 334 Feb 10 16:28 @202202281609
    drwxr-xr-x 1 root root 334 Feb 10 16:28 @202203011815
    root@jerry-hp17cn1xxx:~#
    My usage shows:
    Code:
    :~# btrfs filesystem usage /mnt
    Overall:
       Device size:                 441.04GiB
       Device allocated:            160.05GiB
       Device unallocated:          280.99GiB
       Device missing:                  0.00B
       Used:                        133.96GiB
       Free (estimated):            305.43GiB      (min: 305.43GiB)
       Data ratio:                       1.00
       Metadata ratio:                   1.00
       Global reserve:              218.44MiB      (used: 0.00B)
    
    Data,single: Size:157.01GiB, Used:132.56GiB (84.43%)
      /dev/sda3     157.01GiB
    
    Metadata,single: Size:3.01GiB, Used:1.39GiB (46.30%)
      /dev/sda3       3.01GiB
    And it shows, with my entire system and 5 snapshots, 280.99Gib unallocated. Those same 5 snapshots would fill up my 500Gb spinner and the sixth one would fail to complete because it would run out of space. I learned that by experience.

    PS- The UUID's I got by using "blkid" with the external HDD plugged in.
    Last edited by GreyGeek; Mar 02, 2022, 07:38 PM.
    "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
    – John F. Kennedy, February 26, 1962.

    Comment


      #3
      Here is the result of my backup this evening.
      Code:
      $ sudo -i
      root@jerry-hp17cn1xxx:~# cd /
      root@jerry-hp17cn1xxx:/# /make_snapshot.sh
      Mounting drives
      mount: /mnt: /dev/sda3 already mounted on /.
      Making today's snapshot
      Create a readonly snapshot of '/mnt/@' in '/mnt/snapshots/@202203022213'
      Finding previous snapshot as parent
      Attempting incremental backup
      btrfs send -p /mnt/snapshots/@202203011815 /mnt/snapshots/@202203022213 | btrfs receive /backup
      At subvol /mnt/snapshots/@202203022213
      At snapshot @202203022213
      Delete subvolume (commit): '/mnt/snapshots/@202202242057'
      Delete subvolume (commit): '/backup/@202202171957'
      Snapshots completed, oldest snapshots deleted
      Drives unmounted and 3 syncs done
      root@jerry-hp17cn1xxx:/#
      "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
      – John F. Kennedy, February 26, 1962.

      Comment


        #4
        Thanks, but that's just way too much voodoo foo for me. Consider that your script is "at the marathon running stage", and I'm not even able to crawl yet.

        My internal HDD (/dev/sda) that Kubuntu 22.04 is installed on is formatted btrfs with one partition (/dev/sda1) (UUID=67fab1ac-b53c-4e10-9faf-f9a434156e) and two sub-volumes: subvol=@ and subvol=@home.

        My external USB HDD (/dev/sdb) is formatted btrfs with one partition (/dev/sdb1) (UUID=7baf740a-eab6-40ce-9225-a99a1d0ee969) and no sub-volumes.

        K(eep) I(t) S(imple) S(tupid) for me.

        All I really want at this point are extremely simple, step-by-step instructions that will create a snapshot of @ that includes @home, and sends it to the external USB.
        Windows no longer obstructs my view.
        Using Kubuntu Linux since March 23, 2007.
        "It is a capital mistake to theorize before one has data." - Sherlock Holmes

        Comment


          #5
          Sure enough ...
          Code:
          sudo -i
          cd /
          to get to root.

          Mount the btrfs root file system (<ROOTFS>) to /mnt
          Code:
          mount /dev/sda1 /mnt
          If you do
          vdir /mnt
          you will see

          /mnt/@
          /mnt/@home

          Here's mine:
          /mnt
          ├── @
          └── /mnt/snapshots/
          ├─── @202202251913
          ├─── @202202262040
          ├─── @202202281609
          ├─── @202203011815
          └─── @202203022213


          You don't see a @home subvolume because I moved @home into @ and commented out the stanza in /etc/fstab that mounts it.

          Those two subvolumes, @ and @home, are your live, running system, / and /home. You didn't have to stop the system to do this step. You can continue to use the system while /dev/sda1 is mounted to /mnt.


          Plug in and mount the External HDD drive
          Code:
          mount /dev/sdb1 /backup
          Create a place to store snapshots that won't be visible to users (you) when /mnt is umounted, and because you don't want to clutter up /mnt with snapshots.
          Code:
          mkdir /mnt/snapshots
          Create READ-ONLY snapshots for both @ and @home. Only read-only snapshots can be sent to external subvolumes using the btrfs send command. Using any other tool to create a copy on an external drive can corrupt the subvolume.
          Code:
          btrfs su snapshot -r /mnt/@           /mnt/snapshots/@yyyymmddhhmm
          btrfs su snapshot -r /mnt/@home  /mnt/snapshots/@homeyyyymmddhhmm
          Now, send the newly creates snapshots to the external USB drive.
          Code:
          btrfs send /mnt/snapshots/@yyyymmddhhmm  | btrfs receive /backup
          btrfs send /mnt/snapshots/@homeyyyymmddhhmm  | btrfs receive /backup
          Depending on the size of @ and @home, each send can take up to 20-30 minutes. After @yyyymmddhhmm and @homeyyyymmddhhmm are on /backup they will constitute the "parent" snapshot for a subsequent incremental backup, which will take only a few seconds for each one.

          Note: do not attempt to use mv to send a snapshot to /backup. It won't work. The mv command only works within a subvolume, NOT between subvolumes. If, in the future, you want to restore your system using a snapshot on HDD then reversing the source and destination is how it is done.

          I use the history command to recall previous mount commands and then issue !nnnn, where nnnn is the number of the specific history command I want to use. Saves lots of typing. If done manually, the btrfs send command is also among the history listings and can be copy+pasted and then edited for the new date before hitting the return key.

          Next lesson when you are ready.
          Last edited by GreyGeek; Mar 03, 2022, 10:40 PM.
          "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
          – John F. Kennedy, February 26, 1962.

          Comment


            #6
            That's much more helpful (to me). Thank you.

            First Issue:

            mount /dev/sdb /backup

            /backup didn't exist, so I created it on /dev/sda1

            Second Issue:

            When I plug in my external HDD, the Disk & Devices pop-up notifier appears. I just close the pop-up; I don't click on Mount and Open. In the konsole I type:

            mount /dev/sdb /backup

            and am told:

            mount: /backup: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error.

            If I then click on Mount and Open in Disk & Devices, the drive gets mounted and Dolphin opens showing the drive in a separate tab (465.8 GiB Removable Media). If I then repeat the mount command in konsole I get:

            mount: /backup: /dev/sdb already mounted or mount point busy.



            Update:

            After the external HDD was mounted via Disk & Devices, I opened KDE Partition Manager. I selected /dev/sdb and highlighted /dev/sdb1 and right-clicked and selected Unmount.

            Opened Konsole and typed:

            sudo -i
            mount /dev/sdb1 /backup

            That worked.

            So your instruction: mount /dev/sdb /backup is wrong? Needs to be /dev/sdb1?
            Last edited by Snowhog; Mar 03, 2022, 09:13 PM.
            Windows no longer obstructs my view.
            Using Kubuntu Linux since March 23, 2007.
            "It is a capital mistake to theorize before one has data." - Sherlock Holmes

            Comment


              #7
              Originally posted by Snowhog View Post
              That's much more helpful (to me). Thank you.

              First Issue:

              mount /dev/sdb /backup

              /backup didn't exist, so I created it on /dev/sda1

              Second Issue:

              When I plug in my external HDD, the Disk & Devices pop-up notifier appears. I just close the pop-up; I don't click on Mount and Open. In the konsole I type:

              mount /dev/sdb /backup

              and am told:

              mount: /backup: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error.

              If I then click on Mount and Open in Disk & Devices, the drive gets mounted and Dolphin opens showing the drive in a separate tab (465.8 GiB Removable Media). If I then repeat the mount command in konsole I get:

              mount: /backup: /dev/sdb already mounted or mount point busy.



              Update:

              After the external HDD was mounted via Disk & Devices, I opened KDE Partition Manager. I selected /dev/sdb and highlighted /dev/sdb1 and right-clicked and selected Unmount.

              Opened Konsole and typed:

              sudo -i
              mount /dev/sdb1 /backup

              That worked.

              So your instruction: mount /dev/sdb /backup is wrong? Needs to be /dev/sdb1?
              Correct. Typo on my part. Fixed it in the post so that others won't trip over it.

              How did your first snapshots go? How long did they take to send to the external HDD?

              "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
              – John F. Kennedy, February 26, 1962.

              Comment


                #8
                Originally posted by GreyGeek View Post
                Correct. Typo on my part. Fixed it in the post so that others won't trip over it.

                How did your first snapshots go? How long did they take to send to the external HDD?
                Thanks for confirming and correcting the typo.

                I haven't done a snapshot yet. I'd like to understand why my external HDD is being automounted. I get that that is likely the default configuration for this action in KDE Plasma. But it does make following your steps 'complicated'. When plugged in, it is being mounted to /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969. I guess I can just issue:

                umount /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969
                mount /dev/sdb1 /backup

                I just tried that, and that does work.

                I've got /dev/sda1 mounted to /mnt
                I've got /dev/sdb1 mounted to /backup
                I created /mnt/snapshots
                I created snapshots of @ and @home in /mnt/snapshots
                I'm in the process of sending the @ snapshot to the external drive. When it's finished, I'll send @home.
                Both sent/received without errors. Sweet.

                The process of sending @ didn't take very long; five+ minutes I think. @home took even less time. This isn't a production laptop, so there isn't much on it other than the Kubuntu 22.04 install.
                Last edited by Snowhog; Mar 03, 2022, 11:16 PM.
                Windows no longer obstructs my view.
                Using Kubuntu Linux since March 23, 2007.
                "It is a capital mistake to theorize before one has data." - Sherlock Holmes

                Comment


                  #9
                  Using the blkid requires the use of "/dev/disk/by-uuid/..." as in
                  mount /dev/disk/by-uuid/038bd596-59b4-4bb0-98e5-3cbeec1dd339 /mnt
                  The blkid command gives a list of UUID's for attached devices. There are several paths that can be used.
                  Code:
                  $ vdir /dev/disk
                  total 0
                  drwxr-xr-x 2 root root 320 Mar  4 07:26 by-id
                  drwxr-xr-x 2 root root 140 Mar  4 07:26 by-label
                  drwxr-xr-x 2 root root 140 Mar  4 07:26 by-partuuid
                  drwxr-xr-x 2 root root 180 Mar  4 07:26 by-path
                  drwxr-xr-x 2 root root 140 Mar  4 07:26 by-uuid
                  The /dev/sdXn path of a device can change randomly without warning. That's why UUID is used. The same UUID points to the /dev/sdXn path as it currently is following bootup. When mounting manually using /dev/sdXn is fine because it is determined at bootup or plugin. Inside a script using /dev/sdXn can fail because /dev/sdXn may not point to the same device. When first testing btrfs I used a raid1 using two devices, sda1 and sda2. When I plugged another SATA drive into my Acer and booted up I expected it to be sdc1. To my shock I discovered that my raid1 was sda1 and sdc1, even though they were the same two devices. The new SATA drive was sdb1.

                  Here are the paths given in MY /dev/disk/by-uuid directory
                  Code:
                  $ vdir /dev/disk/by-uuid
                  total 0
                  lrwxrwxrwx 1 root root 15 Mar  4 07:26 2b08ee29-bf29-4b5c-bb40-6aafaf66cec8 -> ../../nvme0n1p2
                  lrwxrwxrwx 1 root root 15 Mar  4 07:26 4ea5991d-7da6-4387-907d-acef20d047de -> ../../nvme0n1p1
                  lrwxrwxrwx 1 root root 10 Mar  4 07:26 83cb8dfd-aadc-4558-884d-eeafac2303bf -> ../../sda2
                  lrwxrwxrwx 1 root root 10 Mar  4 07:26 9bc19383-57db-4a32-938d-8466a133d94d -> ../../sda3
                  lrwxrwxrwx 1 root root 10 Mar  4 07:26 C853-95B5 -> ../../sda1
                  The paths following the "-->" are determined at boot, but the UUID is always the same for the same device. That's why I use the UUID in my scripts but I use the /dev/sdXn manually.
                  Last edited by Snowhog; Mar 05, 2022, 08:42 PM.
                  "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                  – John F. Kennedy, February 26, 1962.

                  Comment


                    #10
                    So I had a thought, and no, it didn't hurt much.

                    If I accept the Disk & Devices option to Mount and Open the external HDD, it will always be mounted at /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969. I can take advantage of that and use:
                    Code:
                    btrfs send /mnt/snapshots/@yyyymmddhhmm | btrfs receive /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969
                    btrfs send /mnt/snapshots/@homeyyyymmddhhmm | btrfs receive /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969
                    Windows no longer obstructs my view.
                    Using Kubuntu Linux since March 23, 2007.
                    "It is a capital mistake to theorize before one has data." - Sherlock Holmes

                    Comment


                      #11
                      Originally posted by Snowhog View Post
                      So I had a thought, and no, it didn't hurt much.

                      If I accept the Disk & Devices option to Mount and Open the external HDD, it will always be mounted at /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969. I can take advantage of that and use:
                      Code:
                      btrfs send /mnt/snapshots/@yyyymmddhhmm | btrfs receive /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969
                      btrfs send /mnt/snapshots/@homeyyyymmddhhmm | btrfs receive /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969
                      Mmm.. never used that approach. Interesting.

                      Those times aren't bad, but once you have a "parent" on your backup drive the incremental send is extremely fast. In your case I'd guess that it would complete in under 5 seconds.
                      "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                      – John F. Kennedy, February 26, 1962.

                      Comment


                        #12
                        Originally posted by GreyGeek View Post
                        Here is the script that I use when I don't do it manually.
                        You run this script from / as root, but where do you have the script saved at?

                        So, your script, it creates snapshots that are kept on your internal HDD (/mnt/snapshots) and incremental snapshots that are kept on the external HDD (/backup)?

                        If that is correct, how are you actually protected if your internal HDD fails?
                        Last edited by Snowhog; Mar 04, 2022, 01:21 PM.
                        Windows no longer obstructs my view.
                        Using Kubuntu Linux since March 23, 2007.
                        "It is a capital mistake to theorize before one has data." - Sherlock Holmes

                        Comment


                          #13
                          It is save at:
                          $ sudo vdir /
                          [sudo] password for jerry:
                          total 40
                          drwxr-xr-x 1 root root 0 Jan 3 16:23 backup
                          lrwxrwxrwx 1 root root 7 Oct 28 08:30 bin -> usr/bin
                          drwxr-xr-x 1 root root 1422 Feb 24 21:01 boot
                          drwxr-xr-x 22 root root 4580 Mar 4 07:26 dev
                          drwxr-xr-x 1 root root 4562 Mar 2 15:29 etc
                          -rw-r--r-- 1 root root 515 Jan 5 2020 he-ipv6.service
                          drwxr-xr-x 1 root root 10 Jan 3 16:28 home
                          lrwxrwxrwx 1 root root 7 Oct 28 08:30 lib -> usr/lib
                          lrwxrwxrwx 1 root root 9 Oct 28 08:30 lib32 -> usr/lib32
                          lrwxrwxrwx 1 root root 9 Oct 28 08:30 lib64 -> usr/lib64
                          lrwxrwxrwx 1 root root 10 Oct 28 08:30 libx32 -> usr/libx32
                          -rwxr-xr-x 1 root root 1600 Feb 8 15:58 make_snapshot.sh
                          drwxr-xr-x 1 root root 10 Jan 2 14:59 media
                          drwxr-xr-x 1 root root 0 Oct 28 08:30 mnt
                          -rwxr-xr-x 1 root root 343 Feb 8 15:59 mount_drives.sh
                          ...
                          I have two internal SSD's and a host of external HDDs & and an SSD.
                          I plug my external 1Tb spinner (which was the HDD that came with this new HP and I put it into a USB HD Caddy) and then run the script using:
                          Code:
                          sudo -i
                          cd /
                          /make_snapshots.sh
                          The script umounts my external USB HDD and /mnt when it is done.

                          Using the history command, I the use !nnnn to run the most recent command to mount my NVMe 1Tb SSD to /backup and the system to /mnt. Then I issue
                          Code:
                          vdir /mnt/snapshots
                          and
                          Code:
                          vdir /backup
                          to see what's on them. I pick the most recent snapshot that is the same on both to use as the parent, and then choose the newest snapshot on /mnt/snapshots as the child, the snapshot I want to send to /backup.
                          Code:
                          send -p /mnt/snapshots/@parentyyyymmddhhss /mnt/snapshots/@mostrecentsnapshotyyyymmddhhss | btrfs receive /backup
                          Then I use
                          Code:
                          btrfs filesystem sync /backup
                          two or three times, or until it returns immediately, indicating nothing was done. (You can also use 'sync')

                          Then I umount /mnt and /backup and then exit root and then exit the Konsole.
                          I use the NVMe SSD less frequently than I do the USB Caddy containing the 1Tb spinner, sometimes only once a week or so. I've used the send command sans -p a couple times and it takes about 25 minutes to send my snapshot to the NVMe SSD. I do that if I don't have a "parent" snapshot on the NVMe.

                          So, all in all, with the aid of the history command and the ability to execute previous history command using "!nnnn" I've pretty much been doing my snapshots manually. In all I have four USB drives that I send backup snapshots to, and some 128Gb and 256Gb USB sticks as well. For a long time I always kept a LiveUSB stick of Kubuntu in my pants watch pocket and a USB stick containing a recent snapshot of my system. Since I've combined @home into @, I only need to track and keep one snapshot and have no risk of mixing up the @ with the wrong @home.
                          Last edited by GreyGeek; Mar 04, 2022, 03:02 PM.
                          "A nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
                          – John F. Kennedy, February 26, 1962.

                          Comment


                            #14
                            Originally posted by Snowhog View Post
                            If I accept the Disk & Devices option to Mount and Open the external HDD, it will always be mounted at /media/paul/7baf740a-eab6-40ce-9225-a99a1d0ee969.
                            If the volume has a label, I would expect it to be mounted in /media/paul with that label, rather than the UUID.
                            Regards, John Little

                            Comment


                              #15
                              jlittle The drive doesn't have a label, so the UUID is identified.
                              Windows no longer obstructs my view.
                              Using Kubuntu Linux since March 23, 2007.
                              "It is a capital mistake to theorize before one has data." - Sherlock Holmes

                              Comment

                              Working...
                              X