Announcement

Collapse
No announcement yet.

Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Technique

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Technique

    Edited / partially rewritten 1 NOV 2010 to use Grub 2 and demonstrate alternative approaches.

    The 'net is full of techniques with which to make a bootable USB stick that will run your favorite Linux distro. Here on Kubuntu Forums, Qqmike has laid it out well and thoroughly:

    http://www.kubuntuforums.net/showthr...h-Drive-How-To

    Other sources:

    http://www.pendrivelinux.com/

    http://unetbootin.sourceforge.net/


    My issue has been, the installed Linux system needs only 200 - 800MB, yet the smallest USB stick at the store is 2GB, resulting in a big waste of storage space. And the further problem is, the Live USB stick installer packages insist on using the first partition on the stick (if not the entire stick), and Windows will not "see" a FAT data partition anywhere but in the first partition of the device. So, you pick up a 8GB SanDisk Cruzer for $20, and you'd like to have 7GB of FAT32 for general-purpose data storage, and a bootable Linux OS available if needed, and none of the installers support this setup. Fortunately, we're working in the open source community here -- we have choices. For ease of reading, I don't want to load this up with code boxes, so I'll bold your terminal commands.

    OK, as usual when working on a Linux system, there are a couple of approaches to get to the result that is wanted.

    Method #1. If your Linux Live CD has a "make a bootable USB stick" option that does NOT insist on reformatting and using the entire USB stick, then you can use this approach:

    - Use GParted to make a small first partition and a matching small second partition at the end of the USB stick
    - install your Linux in the first partition from the Live CD, using the "make a USB stick" utility
    - then use "dd" to copy it out of the first partition and into the second partition
    - then use GParted to enlarge and format the first partition to FAT32, for data storage
    - then install Grub to the MBR of the stick and configure it to boot the applicable Linux OS

    However, the *buntu "Startup Disk Creator" utility does indeed reformat and use the entire USB stick, so that's not desirable if you're looking for efficiency.


    Method #2. For *buntu and other Linux Live CD distributions:

    - Use GParted to set up the needed partitions on your USB stick
    - then mount your ISO image and the USB stick partition
    - copy all the files out of the ISO into the chosen partition on your USB stick
    - install grub to the MBR of the stick and configure it to boot the applicable Linux OS



    In this Guide, I will provide details on how to use both approaches . The initial preparation of the USB stick is the same for both methods, as follows:

    1. Insert your (configuration unknown/any) USB stick, and then make sure it is unmounted. In the console, run sudo fdisk -lu to verify the device number. Let's assume it is /dev/sdd for Step 2. KISS ANY AND ALL DATA ON THIS USB STICK "GOOD-BYE"!

    2. In root console (or prefixed with "sudo"), dd if=/dev/zero of=/dev/sdd bs=512 count=1

    You should see this:

    Code:
    dibl@meerkat:~/$ sudo dd if=/dev/zero of=/dev/sdd bs=512 count=1
    1+0 records in
    1+0 records out
    512 bytes (512 B) copied, 0.784765 s, 0.7 kB/s

    3. Next, Alt-F2 "kdesudo gparted" with no quote marks, to open Gparted in super-user mode. Use the drop-down graphical window in the upper right corner to browse to the device corresponding to your USB stick -- it should come up showing "unallocated" and a yellow alarm symbol, indicating that there is no partition table. From the menu choose "Device > Create Partition Table", accept the default "MS-DOS" table type, and then "Apply". The yellow alarm will disappear.

    At this point, the two approaches diverge, so jump down to Method #2 below to continue, if you need to do it that way ...


    Method #1

    Right-click on the unallocated space, and choose "New", and set the fileystem type to ext2, and pull the right edge of the graphical box to the left until the size of the first partition is the desired size, perhaps a little larger than the size of Live CD ISO file that you will use to install the OS. No label needed at this point. Right-click again on the remaining unallocated space, and this time pull the left edge of this second partition to the right, until you have exactly the same size in MB as the first partition, leaving an unallocated space between the two partitions. It is important that the size of these two partitions is an exact match.

    You should now see something that looks like this (4GB stick example):





    4. From your running Live CD or installed system, use the "Install To USB" utility, which lets you choose the first partition, without formatting.

    - a "persist" feature could be optional, but for this exercise I wanted the equivalent of the Live CD, so this guideline does not provide space, or create the loop device for storing added packages (follow Qqmike's guideline to add that feature, if needed)

    * NOTE: *buntu's "usb-creator" and many other "automatic" installers won't work for this purpose -- it insists on reformatting the entire USB stick as FAT32 before installing the ISO. If that's your situation, you need to be using Method #2.

    5. If grub was installed during the OS installation (step #4 above), test boot it on another computer or two, to confirm it works -- if it does, you're ready to proceed. If it doesn't boot, something is already wrong and you need to troubleshoot the problem before going any further.

    6. Insert the stick in your working computer and again use sudo fdisk -lu to ensure you know the partition device names for your two partitions.

    7. As root, edit the on-device file /boot/grub/grub.cfg and change (hd0,1) to (hd0,2), or for the current ver. 1.98 of grub-pc, you will find "set root='hd0,msdos1'" which needs to be changed to "set root='hd0,msdos2'" in the two or three places you might find it, and save the changes (nano, vi, and so forth are good for doing this from the CLI). Also, if the boot menu entries in /boot/grub/grub.cfg use UUID numbers, i.e. you see "search --no-floppy --fs-uuid --set ec17a5f9-d3b5-4332-97c5-af3074aa0904" and also (for non-*buntu kernels) "linux /boot/vmlinuz-2.6.36-blahblahblahkernelname-amd64 root=UUID=dfd5d21c-b8d7-4dd7-8d44-ac3146008354" then (with the USB stick connected) open a terminal and run sudo blkid to obtain the UUID number for the second partition of the USB stick, and use your editor to paste that number into all the appropriate locations in the grub.cfg file.

    8. Copy the installed (first) partition to the empty (second) partition, using (as root, or with "sudo" prefix) dd:

    dd if=/dev/sdx1 of=/dev/sdx2 bs=4096 conv=notrunc,noerror (where "x" is of course your device number)

    9. Open GParted. At this point, your view should look like this (example is 8GB stick):



    Now use GParted to expand (or delete and replace) the first partition to use all the free space available (don't touch the second partition!), reformat the first partition to FAT32, and change the "boot" flag to the second partition.

    Afterwards, your GParted view should look like this (4GB example):



    If Grub was not installed automatically (or correctly) by the installer, follow the "Installing Grub" step at the end of Method #2 below.


    Method #2: Right-click on the unallocated space, and choose "New", set the filesystem type to FAT32, and pull the right end of the graphical "box" toward the left, freeing up enough space for the second partition. You can see the "Free space following" number change as you move the right end of the first partition, and you want that number to be a little larger than you need for the files on the ISO image. Give this first partition a nice "MYDATA" or similar label. The partition size required for a Linux Live CD varies -- Kubuntu and other "full" distributions will need close to 800MB, while smaller ISOs like Parted Magic can get by with as little as 175MB. Once your first partition settings are correct, click "Add". Then right-click in the remaining unallocated space, set the filesystem type to ext2, give it a label if you wish, click "Add", and now you're ready to click the big green "Apply" checkmark on the Gparted menu. When it finishes, the graphic should look like the last image above, except the partition will still be empty.

    In the terminal, use
    Code:
    sudo fdisk -lu
    to confirm that you have a device with 2 partitions, the first being MS-DOS and the second a Linux filesystem.

    Now, with the terminal still open, follow this generic instruction to copy the contents of your Linux Live CD ISO file to the second (ext2) partition of your USB stick:


    How to Mount Your ISO image and Copy All the Files out of it into your USB memory stick partition

    Assumptions:

    1. You have downloaded xyz.ISO, and verified the md5sum

    2. You want all the files from the ISO in a new empty partition /dev/sdd2, which you just formatted to ext2


    Steps:

    1. Open a console window. As root (or with "sudo" prefix) make a suitable mount point:

    Code:
    mkdir -p /mnt/myiso
    {all the following commands are "as root", either at the "#" prompt, or with a "sudo" prefix, except where indicated otherwise}

    2. Change to the directory where the downloaded ISO is located, and using the Linux mount command:

    Code:
    mount -o loop xyz.iso /mnt/myiso
    3. Change to the /mnt/myiso directory. Verify that the ISO files are there:

    Code:
    ls -l
    ("sudo" not required for this command)

    4. Make a mount point for the USB device and partition

    Code:
    mkdir -p /mnt/SDD
    5. Mount the USB partition

    Code:
    mount -t ext2 /dev/sdd2 /mnt/SDD
    6. Change to the /mnt/SDD directory -- confirm you are there with

    Code:
    pwd
    ("sudo" not required for this command)

    7.a. Make sure your bash shell has the dotglob option enabled

    Code:
    shopt -s dotglob
    7.b. Copy all the files and directories, including any hidden files, from the ISO to the current directory:
    Code:
    cp -r /mnt/myiso/* .

    This completes the task of copying the ISO contents to the target partition. Verify that the number of files and directories in the target partition is identical to the contents of the mounted ISO; for example view them in Dolphin's split window, right-click on the applicable top folders and check "Properties". (There might be one more folder -- a "Lost&Found" folder, due to the ext2 filesystem -- that is OK.)


    Install Grub 2:

    With the /dev/sdd2 partition mounted on /mnt/SDD as described above, on a running Linux system, like Kubuntu ver. 9.10 or later that uses Grub 2 (aka grub-pc) for booting, you can proceed to install Grub 2 to your USB stick with this command:

    Code:
    sudo grub-install --root-directory=/mnt/SDD /dev/sdd
    and then you will need to manually edit the on-device file /boot/grub/grub.cfg to fix the partition ID references and UUID numbers, same as Step 7 for Method #1 above.

    See Qqmike's Grub 2 Guide if you want to know more about installing and configuring Grub 2.

    NOTE HOWEVER: For a "Live CD" USB stick Linux, that is not intended to get new packages or updates, you DO NOT EVER want to run "update-grub" or "grub-mkconfig"! Instead, for this limited case application, you can edit the on-device file /boot/grub/grub.cfg to make any needed adjustments to the device ID or the UUID numbers that are used for booting. When booting the USB stick, the partition that you are now seeing as /dev/sdd2 will always be (hd0,2) to Grub, and the UUID number needs to be correct, on the lines in /boot/grub/grub.cfg that begin with "search ..." and also where you find the phrase "root=UUID=dfd5d21c-b8d7-4dd7-8d44-ac3146008354". You can always find the correct UUID for each storage partition on your system, including the partitions on your USB stick, with
    Code:
    sudo blkid
    Test the stick -- when inserted in a running Windows or Linux box, it should offer a large first partition available for data storage. When booted on a PC set to boot from USB device (and capable of doing so), it should boot your installed Linux Live CD version.
    Last edited by dibl; Jan 28, 2012, 03:23 PM.

    #2
    Re: Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Techn

    Nice job (again! ).

    Smart strategy of making both partitions at the start the same size (965 MB in this case). Then , using dd you are going from a partition to another same-size partition, thus avoiding issues that occur when going from small to big. The dd will also copy over the filesystem (ext2); thus clever that you started with ext2 in the source partition and then reformatted it to FAT32 after doing the dd copy.

    I was surprised about Windows only seeing the first partition, and not many folks know that--probably because most Windows users only use a 1-partition flash drive, until last year when I searched this and found it to be an "issue" among heavy Windows users.

    Nice how it all fits together, the use of all the tools GParted, dd, and the GRUB install.

    Nice and clean, good job, dibl.
    An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

    Comment


      #3
      Re: Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Technique

      I am a new user to Kubuntu 9. I have it installed on a USB Flash drive and it all went well. However, none of the changes I make to the system such as graphics driver, sound etc are there after a shut down.

      At the risk of upsetting people, the details in this thread are good but to me it was very intimidating and the difficulty in my opinion is what is holding Linus back. As much as I dislike Windows, it is so much easier to change things

      Surely if the community can write such a fantastic op system why can they not write a utility to make the changes that thousands of people post on the internet

      Comment


        #4
        Re: Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Techn

        dibl's how-to doesn't apply to your question.

        I have not yet had time to try bootable flash drive with 9.10, but you should try usb-creator to do it and check the box for persistence when you run it. Here is some explanation for recent past versions of Kubuntu, and I'm pretty sure it will be the same for 9.10:
        Build a LIVE Kubuntu Flash Drive
        http://kubuntuforums.net/forums/inde...089474.new#new
        with persistence
        See this:
        Reply #1: usb-creator program (8.04 & 8.10)
        An intellectual says a simple thing in a hard way. An artist says a hard thing in a simple way. Charles Bukowski

        Comment


          #5
          Re: Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Technique

          Originally posted by KesGuy09

          the details in this thread are good but to me it was very intimidating
          I don't disagree -- that's why the title says "Advanced" and it isn't posted in the "Help the New Guy" forum.

          Comment


            #6
            Re: Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Technique

            I pretty much re-wrote this guide today, to address the two major ways to approach the problem, to put the ISO file-copying process in sequence, and to address the use of Grub 2 for booting the USB stick. Corrections and additions are welcomed.

            Comment


              #7
              Re: Bootable USB stick -- Lots of data storage plus your Linux -- Advanced Technique

              Nice piece of work. Thank you for updating.
              Using Kubuntu Linux since March 23, 2007
              "It is a capital mistake to theorize before one has data." - Sherlock Holmes

              Comment

              Working...
              X