starting sector number, 4532393984 exceeds the msdos-partition-table-imposed maximum

[oshunluvr]

I've rarely seen a successful recovery of data by restoring starting and ending sectors of a partition, even if one had the foresight to write them down before changing them.

I had a disk several years ago - like 12-14 - that I re-partitioned in error. I was able to use testdisk to restore the entire thing without losing anything on it. After that, I kept off-line backups of all my partition tables just in case. I did actually run some testing using dd to backup and restore partition tables with total success.

Now, with the relative low cost of storage media and the ease of backing up with btrfs, I just make full backups more often. Also, GPT tables have a built in backup that can be loaded and relocated if a sector goes bad. Drives are much more robust these days.

The primary thing when something goes wrong with accidental deletes or partition table troubles is to stop using the disk immediately. IF you do that, your chances of at least a partial recovery are very good.

[oshunluvr]

Assuming you didn't write anything to the old /home partition, the calibre folder and it's contents are likely still intact there. extundelete should be able to restore the original folder.

As far as the new sda4, try running sudo parted -l and see what it lists as the file system. To attempt mounting it as ext4, try sudo mount /dev/sda4 -t ext4 /mnt/somewhere

[GreyGeek]

Well, if you moved partitions about in the middle of this you may have trashed it. As I said in the beginning, mucking about with partitions can be dangerous business, especially without a backup.

One is on dangerous ground when disagreeing with the lover of oceans, but I've rarely seen a successful recovery of data by restoring starting and ending sectors of a partition, even if one had the foresight to write them down before changing them. This situation was looking to me like a slow motion train wreck and I just wanted to cut to the chase. If nothing else it is a lesson on the necessity of having off line backups, and nothing is better and easier at that in the Linux world than Btrfs.

.... On a positive note, there is still a last resort - a tool called "photorec" which has an excellent track record of finding files on a drive,....

Photorec is excellent indeed. (It is part of testdisk, which is in the repository) I have used it in two cases with outstanding success to recover data from HD's that failed to read because of sector errors. I used dd to create an image.dd file of the drives and then used photorec to extract from that image what it could. In one case photorec recovered over 95% of 50,000 jpg images of official documents and indexes linking them to the paper documents. In another case it recovered 1,490 files out of 1,510 files, including the thesis, that were on a grad student's corrupted laptop HD. For the Dept of Revenue I had photorec also save corrupted files and on those I used a binary editor to extract readable portions of documents and indexes.

[rwbehne1]

Where did you move the Calibre database from?

1) I moved the partition all the way to the end of the drive.
2) I moved the Calibre directory from /home (sda3) to sda4.
Then I did other things elsewhere, and didn't touch those partitions again until now.

[rwbehne1]

Re-doing the partition table for sda4 won't do anything. The partition table doesn't contain file system format. If the file system is in fact ext4 you might be able to manually force mount it as ext4. There is a conversion tool that rolls btrfs back to ext4 IF it was converted from ext4 to btrfs. Unless you ran a conversion on the ext4 filesystem, it's likely to do more harm than good.

At this point, I'd run testdisk and analyse the disk for trouble.

Ok, while testdisk is running, how do I manually force mount it as ext4?

I don't think it was converted, just relabeled.

[oshunluvr]

Where did you move the Calibre database from?

[oshunluvr]

Re-doing the partition table for sda4 won't do anything. The partition table doesn't contain file system format. If the file system is in fact ext4 you might be able to manually force mount it as ext4. There is a conversion tool that rolls btrfs back to ext4 IF it was converted from ext4 to btrfs. Unless you ran a conversion on the ext4 filesystem, it's likely to do more harm than good.

At this point, I'd run testdisk and analyse the disk for trouble.

[rwbehne1]

I mounted it and...

root@kubuntu:/media/root/_usr_local# dir
BACKUPS  Calibre-customization  games   i2p      lib64       man        sbin   systemd-custom-scripts  webmin
bases    dosutils               gopher  include  libexec     nohup.out  share  TARBALLS
bin      etc                    HOLD    lib      lost+found  opt        src    tNOS
root@kubuntu:/media/root/_usr_local#

Well, that's PART of it, but somethings missing! This is what it was before the partition move. This is what I would expect. However, after I moved the partition I moved a Calibre database directory into the partition, and I see it is not here. I expect that the ext4 really is the actual /usr/local, but after the partition move. I think we should test my theory by fixing sda4 the same way - it couldn't hurt, eh?. If we find the missing Calibre directory then that means I'm right. So shall we try it? Should I use exactly the same method, but with 5403539456 to 5860532223?
Note again that the partition is the correct size as I recall:

4 5403539456 5860532223 217.9 GiB 8300 Linux filesystem

[oshunluvr]

Ok, try and mount /dev/sda5 somewhere.

[rwbehne1]

I would first attempt to rebuild the missing partition as I suggested using gdisk. Launch it, select "n" for new partition, allow it to be the default partition number "5", then in first sector put 4075401216, in last sector put 4532393983, leave the sector type the default "8300", then write it to disk. Then run "sudo partprobe" (if it errors again, reboot to the liveUSB), then run the partition manager and see where we are.

root@kubuntu:~# gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
 MBR: protective
 BSD: not present
 APM: not present
 GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/sda: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): F9C78116-9D2E-4304-A329-E3A7704EF117
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 8-sector boundaries
Total free space is 1328139151 sectors (633.3 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
  1              34            2047   1007.0 KiB  EF02  BIOS boot partition
  2            2048       209922047   100.1 GiB   8300  
  3       209922048      4075401215   1.8 TiB     8300  Linux filesystem
  4      5403539456      5860532223   217.9 GiB   8300  Linux filesystem

Command (? for help): n
Partition number (5-128, default 5): 
First sector (4075401216-5860533134, default = 4075401216) or {+-}size{KMGTP}: 4075401216
Last sector (4075401216-5403539455, default = 5403539455) or {+-}size{KMGTP}: 4532393983
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 
Changed type of partition to 'Linux filesystem'

Command (? for help): p
Disk /dev/sda: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): F9C78116-9D2E-4304-A329-E3A7704EF117
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 8-sector boundaries
Total free space is 871146383 sectors (415.4 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
  1              34            2047   1007.0 KiB  EF02  BIOS boot partition
  2            2048       209922047   100.1 GiB   8300  
  3       209922048      4075401215   1.8 TiB     8300  Linux filesystem
  4      5403539456      5860532223   217.9 GiB   8300  Linux filesystem
  5      4075401216      4532393983   217.9 GiB   8300  Linux filesystem

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
root@kubuntu:~# partprobe
Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.
root@kubuntu:~#

[oshunluvr]

Well, if you moved partitions about in the middle of this you may have trashed it. As I said in the beginning, mucking about with partitions can be dangerous business, especially without a backup. If it we're me, I'd stop focusing on the future plans for a minute and work on recovering your data. Likely you attempted the move before the new partition table went into effect so there's no telling what actually happened. You used several tools with overlapping functions. On a positive note, there is still a last resort - a tool called "photorec" which has an excellent track record of finding files on a drive, however since you have no where else to put data I have no idea who it would work for you, but...

...first things first: Make a partition where your old one was using gdisk like I said in the last post. Then attempt to mount it and see what happens. Under no circumstances should you be writing anything to anywhere on this disk except the partition tables - there's no actual data in the partition table space.

I can't really "be graphic" in the sense of screen shots or whatever, since GUI tools won't be of use.

Post back the results.

[rwbehne1]

GREY GEEK: If he deletes all his data how is he to recover it?

rwbehne1: DON'T WIPE THE DISK OR WRITE ANYTHING TO IT RIGHT NOW!

Possibly you have not lost the data, but you may need to do some recovery. I see the partition tables match up now. See the unallocated section in between sda3 and 4? It looks to me like the old sda4 is not in the partition table but it's still there. The old sector boundaries of sda4 were 4075401216 to 4532393983 - those sectors are in the unallocated portion of your disk, so it's very likely the data is all still there. You can also see the current sda3 partition ends at 4075401215 - exactly one sector before the old sda4 began.

testdisk or parted should be able to recreate the missing partition.

Ok, YOU just got my full attention!

You're saying along the lines of what I've been thinking. But this is unfamiliar territory for me. Could you be a little more graphic about how to repair it, since I'm just learning this?

Here's what gdisk says right now:

Number  Start (sector)    End (sector)  Size       Code  Name
1              34            2047   1007.0 KiB  EF02  BIOS boot partition
2            2048       209922047   100.1 GiB   8300  
3       209922048      4075401215   1.8 TiB     8300  Linux filesystem
4      [B]5403539456      5860532223[/B]   [B]217.9 GiB[/B]   8300  Linux filesystem

...and fdisk -l shows:

Device          Start        End    Sectors   Size Type
/dev/sda1          34       2047       2014  1007K BIOS boot
/dev/sda2        2048  209922047  209920000 100.1G Linux filesystem
/dev/sda3   209922048 4075401215 3865479168   1.8T Linux filesystem
/dev/sda4  [B]5403539456 5860532223[/B]  456992768 [B]217.9G[/B] Linux filesystem

I have verified that sda3 is in fact my old 1.8T /home partition, the size is right and it is still intact. I can mount it and browse around in it.

I know that 4075401216 to 5403539455 actually is unused space, not formatted. The 217.9G /usr/local partition used to be in there, following immediately behind the /home partition, but yesterday I moved it with KDE part tool to the end of the drive where it is now. Therefore 5403539456 to 5860532223 ought to still be the old /usr/local (the size is right), but KDE part tool shows it as having btrfs when it should be ext4, and it's labeled as / instead of /usr/local. I can mount it, but it looks blank because of the wrong label for the filesystem. (No, I refuse to write anything to it because I think it can be recovered.)

That's what changed - the labels for that partition.

So I think what has to be done is to change the btrfs back to say ext4 again, and change the / back to say /usr/local and that should recover that partition.

Let's try that first. I have installed testdisk, and parted is already installed. I have never used either before, and I don't want to guess, so I'm going to defer to your judgment on how to proceed oshunluvr.

My plans after recovering the partition:
1) Once that partition is recovered I'll want to find out what caused this problem in the first place and fix that too.
2) Then I can safely install a small system on the sda2 and boot into that, so I can stop using this USB drive for the rest of the work. (I think using the external drive may work faster when using a regular install instead of a USB stick.)
3) Next, I have to find a way to quickly move as much data as possible from sda3 to the external drive. That way I can make a new temporary partition in the unused space only big enough to move the remaining files from sda3 to the new partition, and then delete the empty sda3. That will free up space so I can grow sda2 to fill the available space, and move files from the old ext4 partitions into their new home in sda2.
4) After all files are moved from the ext4 partitions they'll be empty, so I can delete them and then grow the sda2 to fill the entire drive.

This is what I love about Linux: There's always more than one way to accomplish any task!

[oshunluvr]

I would first attempt to rebuild the missing partition as I suggested using gdisk. Launch it, select "n" for new partition, allow it to be the default partition number "5", then in first sector put 4075401216, in last sector put 4532393983, leave the sector type the default "8300", then write it to disk. Then run "sudo partprobe" (if it errors again, reboot to the liveUSB), then run the partition manager and see where we are.

[rwbehne1]

Your existing /home and /usr/local were on an EXT4 partition. You gave / to Btrfs and it also includes /home and /user/local, so you were putzing the system. You don't put chevy parts on a ford engine, do you?

Why would it "putz" the system? sda3 simply mounts as /media/root/_home/ in the live system, and in the installed system I can mount it any way I like, then copy files from it to @home.

I haven't played with 17.04 but every other version of Kubuntu & Neon I've used shows a progress bar. And, I doubt that 17.04 touched partitions that you didn't select during the install.

BTW, the size of the disk is NO indication as to how much DATA is on the disk. What I was asking was how many GB of DATA were in the partitions you wanted to preserve.

Data size:
/dev/sda3 1.8T 1.4T 404G 78% /media/root/_home

IF you haven't backed up that data you were concerned with then it has joined its brethren in the infinite bit bucket in the sky. IF you have then all is not lost.

Either way, what I'd do is reboot the installation USB and this time delete ALL the partitions on sda and create only one, sda1. That's all you really need. Select Btrfs as the file system and assign it to "/". Don't get fancy with small partitions and names like "target". /home will be created automatically. You don't need to create special partitions for subdirectories. Select the option to install multimedia apps but do not select the option to do updates during the install. You can apply updates later. Accept its default grub installation location. After the install is down don't reboot, POWER DOWN.

Remove the USB stick and power back up.

I can't attempt another install until I have a means of backing up everything, AND fixing whatever gdisk royally screwed up first. Deleting all my partitions, along with the data, is an unkind suggestion. I have apparently already lost everything that WAS in my former /usr/local due to whatever gdisk did to mess everything up the way it did. I don't want to loose another d... thing!

I've been having problems with the external drive, trying to copy anything to it is pathetically slow, and it keeps stalling for 20 minutes at a time. WHY? I don't know. With things not working the way they aren't, I'm now thinking of just going back to ext4, but the problem then is that no one has mentioned how to go about fixing the problem with the partition id being screwed up. Why did the installer think that sda4 needed a btrtfs installed when I explicitly set it to NOT touch that space? Why can't I set aside a portion yo just get a small system in and use usable WITHOUT touching the rest of the drive? Why is that too much to ask? If btrfs cannot live without hogging the entire drive and deleting everything else on it, then it can't be that wonderful.

What exactly happened to my sda4? Did something change the label somewhere, or forget to do it?
Can the data on it be recovered somehow? There must be SOME WAY to recover from this.
How can I tell it to change the label BACK to ext4?
How did gdisk so badly mis-configure the tables so as to confuse the installer?
How can it be corrected?
How can I effectively use this external drive to temporarily back up sda3?

Answering those questions will be much more helpful than telling me to throw away years of data.

[oshunluvr]

Actually, since you have the exact sector numbers, even gdisk should be able to re-create the missing partition. Simply doing a sector add and entering the specific beginning and ending sector numbers should remake the partition right were it was before.