Problem creating bootable usb thumb drives

[bobbicat]

The expert on Gparted-live has replied on the Gparted Forum in a negative fashion and has closed the bug as needing no action.
He denies that gparted offer a dd method, though at this time it is plainly visible on the website
I posted a reply - this is no way to encourage bug reporting.


Incidentally, on the Gparted live website, the claim is that their iso's are, isohybrid, made from Debian live and so should copy to and boot from a thumb drive with no problems.
My research above is therefore only background and not directly connected with the issue.

[bobbicat]

I have always started with a standard fat partitioned dos thumb drive when creating a bootable from an iso. It has always worked.
My Kubuntu thumb drives boot happily as UEFI and are produced in this way. Of course my computer drives, ssd and sdX, are GPT partition tabled and all have Linux partitions, apart from the required efi vfat boot partition.


I had toyed with the idea of GPT partition tabled thumb drives, but came to the conclusion that it produced no advantage. A standard usb drive comes with a dos partition table and a fat partition, so unless absolutely necessary why change the default - apart from the need to experiment and broaden knowledge there is no reason I can see to change.
I haven't used any OS apart from Linux for many, many years so my system doesn't have the 'adjustments' some have to make to allow for those 'other ways' of doing things.
Ever since one of those alternate commercial operation systems offered to correct 'errors' then proceeded to chew up all the data it contained I have given 'other methods' a wide berth.
I am happy with the reliability I have found in Kubuntu and away from all that.

So you're getting UEFI boot mode on MBR (instead of GPT)?

I cannot give a simple explanation. I am no expert. Google is to blame. It led me to this point.


from wikipedia:

An ISO image is an archive file of an optical disc.
Hybrid disc formats include the ability to be read by different devices, operating systems, or hardware.

There is a procedure I am not exoert enough to explain fully, but I will try to convey what I understand.
An iso of itself will not boot from a thumb drive. To make it do that it needs a structure added that the computer can work with.
An iso and a hybridised iso, look the same but the hybridized iso has this function.
When the hybridized iso is copied to the thumb drive a number of background processes run. I don't think the initial partition table and partitioning are important, as the hybrid iso overwrites what is there, creating non-standard structures, which then permit a boot option for the PC. You see it isn't simply a file that is copied, some sort of partition adjustment also takes place, these changes are probably not read correctly by ordinary disk utilities.

There is quite a lot of technical info out there for anyone willing to Google. Searches for syslinux-utils, isohybrid and UDF are productive.


All of this comes with the caveat that I am an amateur poking around in stuff I probably shouldn't. The outcome is not guaranteed. My expertise is perhaps only in Googling, not in the subject I am researching.

...AND at this point I must quote your quote, Mike:

You must act as if you know what you are doing; when, in effect, you know nothing.

THEN there is also my fav quote from Picasso:

Computers are useless - they only give answers

*addition*
it occurs to me that if you have, like me, been messing with this stuff, then when you are done it might be an idea to dd zero the thumb drive, to clear out those 'odd bits' that have been inserted here and there.

Warning: as always with dd, "be careful with that axe, Eugene!!!"

(I only put the last bit in cos I like the Warning wrap tag )

ALSO
NEVER use three words when you can fit in 33

(I was twenty minutes late to work this morning - I had got so caught up in writing this)

[Qqmike]

I'll try to look at this again sometime soon. (I'm not crazy about the isohybrid stuff ...) I'm tied up doing some unrelated experiments with UEFI. I'm noticing your use of fdisk instead of gdisk. So you're getting UEFI boot mode on MBR (instead of GPT)?

[bobbicat]

I've had a reply on the Gparted Forums acknowledging the problem and promising to look into it.
In the mean time, I suggest the safest way to address this problem would be to use Method B as outlined at the Gparted web pages.
My experimental results, above, are just shots in the dark, there are no guarantees of outcome or reliability. Don't take chances with your valuable data.

[bobbicat]

After more Googling and experimenting I came up with a procedure that appears to solve the problems.
I'm not sure that it is the right answer, just hacking really, but appears to work.

This is how:

download gparted-live-0.22.0-1-amd64.iso

install:

sudo apt-get install syslinux-utils

then run:

sudo isohybrid --uefi /path/to/gparted-live-0.22.0-1-amd64.iso

plug in your thumb drive at /dev/sdX
and run:

sudo dd if=/path/to/gparted-live-0.22.0-1-amd64.iso of=/dev/sdX bs=4M; sync

try booting up from it
It worked for me

sudo parted /dev/sdd print
 
Model: Ut163 USB2FlashStorage (scsi)
Disk /dev/sdd: 4127MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 


Number  Start  End    Size    Type     File system  Flags
 2      232MB  236MB  4280kB  primary               esp

maybe it will for you?

sudo fdisk -lu /dev/sdd


Disk /dev/sdd: 3.9 GiB, 4127194624 bytes, 8060927 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x656f8811


Device     Boot  Start    End Sectors  Size Id Type
/dev/sdd1  *         0 464895  464896  227M  0 Empty
/dev/sdd2       453500 461859    8360  4.1M ef EFI (FAT-12/16/32)

It probably needs tweaking up somewhat, its a strange looking thumb drive.
I'm no coder, just a cat with a mouse.

I'd appreciate feedback, if anyone reading this gets results or no...

[bobbicat]

Like a terrier with a rat, I just can't let go...

Here is some discussion on the Gparted Forums, it is very relevant. I'm beginning to wonder whether the recent iso versions of Gparted are isohybrid. I might try to convert them.

Heres a link to 2009 and the pertinent discussion: http://gparted-forum.surf4.info/viewtopic.php?id=13581 also in 2013 http://gparted-forum.surf4.info/viewtopic.php?id=16838

[bobbicat]

yup - saw your post - thanks for the support
I went through the same maze and found the answer after Googling for it.
I was unable to find the answer on the Gparted web-site
Perhaps I'll mention that on the Gparted Forum - though I'm beginning to think there's no activity there ( apart from ourselves that is)

I'm not a coder so I wouldn't have a clue how to write a patch for this bug.
I might be barking up the wrong tree but I think I've pointed out the anomaly - its up to someone with the appropriate abilities to tackle it now.

It has also occurred to me that as it is a gnome based utility this isn't really the place to find the answers I'm looking for.
If anything positive occurs I'll post here again, but for now I think its time to let the matter rest.

[Qqmike]

I got it. VERY frustrating! Bart Hakvoort
but I only saw this quite by accident when clicking on some Help link TWICE.

So I posted right after yours.

[Qqmike]

I tried to post at GParted forum, couldn't do it even though I have a valid account, very frustrating: Question: Who was the creator of GParted? Pun stops bots, answer on GParted site, etc. I clicked everything and couldn't find what they expected as an answer. How the h*ll do you post there? (something I've done many times in the past)

[bobbicat]

I'll try the skip and seek routine you mentioned above in connection with dd and see if I can create a start at sector 63. If I am successful the result might be enlightening.


not able to get a result, tried without success


I'm watching gparted forum and bugzilla but no interest is being shown there as yet

[Qqmike]

Looks like you are correct -- there seems to be the problem you've identified with the iso and dd-ing it to the flash drive. I tried two versions of GParted amd64: the current version gparted-live-0.22.0-1-amd64.iso, and the previous version: gparted-live-0.21.0-1-amd64.iso.
Same result: My PC did not recognize it as a UEFI boot choice; it DID see it as a BIOS boot choice, though. GParted's main menu indicated it was booted in BIOS mode (it was a * footnote at bottom of a main menu screen).

There seems to be a problem with that iso not setting up a proper UEFI installation and bootloader.

[bobbicat]

This is the result for my 1504 vervet installation thumb drive, it was created by using dd with the 64bit iso on an msdos partition tabled thumb drive.
It works fine, throwing up no errors and functioning correctly in UEFI.

sudo fdisk -lu /dev/sdd

Disk /dev/sdd: 7.5 GiB, 8053063680 bytes, 15728640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000ea6ea                                                                            
                                                                                                       
Device     Boot Start      End  Sectors  Size Id Type                                                  
/dev/sdd1  *       63 15727634 15727572  7.5G  c W95 FAT32 (LBA)

If you look you will see that start in this case is not at zero, it is at 63, so the partition table is not overwritten.
I'm now almost certain this is where the problem lies.

[bobbicat]

I have the correct iso - gparted-live-0.22.0-1-amd64.iso - it md5sums OK

GParted do not specify the type of partition table for the usb drive, so I assumed a standard thumb drive would be right. I have only ever used standard (msdos partition tabled) thumb drives for my iso's in the past. They have always functioned correctly with the UEFI system.

I created a standard clean thumb drive, using fdisk, for the results shown above.
The drive had an msdos partition table and one full sized fat32 partition before the iso was copied across.

I'll scrub the drive down to zero, create a GPT with a fat32 partition using gdisk and dd the iso across as before.
...and will post the outcome in a few minutes.

I'll also be looking at other bootable thumb drives I have that do not have this problem. Maybe something will show up there.


Oh and thanks for sparing your time Qqmike.


*results*

I'm getting the same result for both msdos and GPT partition tables. The utility boots up, declares a recursive partition on usb drive and is not recognized as UEFI by the PC.
The same utility installed from the appropriate zip file sets up correctly, is recognized as UEFI by the PC and throws up no error messages.

This leads me to suspect that the iso is not setting up correctly.

I have never had this problem dding other iso's. (for example - the iso for 64bit 1504 Kubuntu vervet and also all its predecessors)

[Qqmike]

Nor am I the expert I'll throw some stuff out for thought.

dd:
Yes you can control what is written and where it is written. Use skip and seek. man dd. This time, you'd use seek, skip past 512 bytes, seek=512 bs=1; or seek=1 bs=512, etc. That would skip 512 bytes on the target flash drive.

But, for standard stuff such as this--building GParted on a flash drive--I can't imagine any seek should be necessary, or they would tell you so at GParted.

I think what could be going on is some confusion among the utilities/PC/whatever as to what MODE the work is being done in: is it older BIOS+MBR mode or newer UEFI+GPT mode? You want UEFI, right? And you want straight UEFI, not some hybrid like UEFI with any old MBR in use (no CSM stuff).

So, making sure everything is always done in UEFI mode ...

GParted live iso file: is it the amd64 version (for UEFI)?

amd64

64-bit version runs on x86-64 based computers, with ability to access more than 4 gigabytes of memory, and supports use of multiple processor cores. For newer (>2010) computers with UEFI instead of legacy BIOS try this one.

http://gparted.org/download.php

Booting your PC: are you booting it in UEFI mode? That would be determined by your settings in the PC's UEFI setup menus.

Booting the flash drive: in UEFI mode?
That's determined when you put the flash drive in the PC, re-boot, see the POST, hit the magic key on your PC to enter its UEFI-"BIOS" setup (bad terminology), and you find a list of booting options, perhaps called a Boot Override menu. You'll see the flash drive if it is bootable. In fact, you may see it listed two or more times. You want to select the flash drive instance that has the descriptor UEFI or EFI on it.

That's all I can think of at the moment. If I can find time today, I'll try to build the GParted flash drive in UEFI and see what happens. I have done other UEFI bootable live flash drives with dd, straight iso-copying--with no problems.

Edit

Also, I would make sure the flash drive is clean before using it.
Best: dd it to zeros, dd if=/dev/zero of=/dev/sdX bs=4096 (or whatever bs you like, maybe bs=1M).
Also OK: Use GParted or a partitioning tool to delete all filesystems and partitioning and start from scratch with a GPT partition table and the rest of the drive one FAT32 partition. Of course, dd-ing the GParted iso will take over and wipe this off anyway, it will overwrite anything and everything on the flash drive to set up its own files for booting.

gdisk
And for any partition work with GPT, we are using gdisk, not fdisk, right?

[bobbicat]

No results as yet from Gparted Forum or Bugzilla.

BUT
after more extensive Googling I seem to be part way to an answer.

With:

sudo parted /dev/sdd print

I get:

libparted bug found
Invalid partition table - recursive partition on /dev/sdd.

Performing:

sudo fdisk -lu /dev/sdd

gives:

Disk /dev/sdd: 3.9 GiB, 4127194624 bytes, 8060927 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x656f8811

Device     Boot Start    End Sectors  Size Id Type
/dev/sdd1  *        0 464895  464896  227M 17 Hidden HPFS/NTFS

I found this comment which, perhaps, explains the above:

The first partition starts at sector 0, so it includes the MBR and the partition table that contains itself. This makes it look like an extended partition recursively containing itself.

-----------------------------------------------
Should something be done to make the dd copy skip a few sectors to allow space for the msdos partition table? Is that even possible?

This still doesn't address the fact that my computer cannot see that the file on the usb drive has UEFI capabilities.

Also,is the file type reported by fdisk relevant?

I am very much out of my depth here but I will continue tread water in the hope that a rescue is possible.