Help me understand partitioning and disk space

[Skal1]

I repartitioned and reinstalled this weekend and everything worked out fine. I freed some more diskspace by reducing it to (a still pretty conservative) 1 respective .5%.

That doesn't look right. If it was GPT formatted it would say "gpt", like your 1 TB drive. Is the computer booting in a "compatibility" or "legacy" mode? I thought UEFI needed GPT.

IMO to list stuff "sudo parted -l" is better, but maybe I just remember when fdisk choked on GPT drives.

I checked just now and both fdisk and parted tell me both of my drives are GPT. I don't know what the problem was before. I partitioned before installing this time which worked out way better. Maybe the installer messed up?
I'm booting in UEFI now and I booted in UEFI before the repartitioning.

[jlittle]

... Also I wanted to use the UEFI my device offers.

                                                                                                 
dani@dani-ThinkPad-E570:~$ sudo fdisk -l                                                                                             
...
Disklabel type: [B][COLOR="#FF0000"]dos[/COLOR][/B]

That doesn't look right. If it was GPT formatted it would say "gpt", like your 1 TB drive. Is the computer booting in a "compatibility" or "legacy" mode? I thought UEFI needed GPT.

IMO to list stuff "sudo parted -l" is better, but maybe I just remember when fdisk choked on GPT drives.

[jglen490]

@jglen490: Its mostly for convenience I would say. This laptop is going to be my everyday-carry and I tend to amass data. I just chucked in the additional drive so I wouldn't have to worry. I'm fairly certain I wont use ALL of this space but its nice-to-have.

@oshunluvr: Thank you a lot for this in-depth explanation. I will definitely change the 5%.

Which tools do you personally use? Just a quick df?

Regarding the GRUB confusion: I read this thread https://www.kubuntuforums.net/showth...lp-the-new-guy here and it says to use 'apt-cache show grub | grep ersion' to find out which version you're using. This produces the strange output with Legacy etc that got me confused. I tried 'grub-install --version' which gives the correct version. And I'm indeed running 2.02 like you all said.

I think I will stick to EXT4 for now. Theres plenty of other things to learn first. Although the snapshot feature I've read about in the other thread sounds really nice so I might come back to it.

I will reinstall later today or tomorrow and report back. Thank you all for your help, it is greatly appreciated!

The big thing is welcome to Linux, and especially Kubuntu world. It is not the most popular distro, by some measures, but for me it just works and almost never gives me a hard time. Enjoy your laptop and acres of room!

Come here anytime you want to, have questions or have a problem to solve. And if you ever get into the "crack" that is BTRFS, just enjoy it, too, and pay no never mind to the bantering about BTRFS and ext4 - it is literally ALL good.

[DoYouKubuntu]

I don't disagree that ext4 is ancient - like me! But, I also don't doubt that it is mature and useful - like me, well sort of ...

I don't doubt that BTRFS is a useful approach to hosting a computing platform. I don't hate it, but at the same time I don't use it - I just ignore it and will continue to until Linux no longer supports ext4. ext4 is sufficient for my needs. If the OP were to choose going the BTRFS route, I wouldn't feel bad at all!

Yep, what you said!

[vinnywright]

BTW, for you EXT4 users, you can free those reserved blocks anytime your file system is unmounted. The tunefs command above is non-destructive. If you're working on a system (boot) drive, leave a small percentage reserved. If it's a pure data drive, free them all.

nice ,,,you get a good bit of info from a tune2fs -l /dev/sdxx looks like my neon-/dev/stable (on now) has a Reserved block count: of 695360 that seems to calculate out to 339.53125 MB on a 53GB partition .

VINNY

[oshunluvr]

Which tools do you personally use? Just a quick df?

I rarely check drive capacity, but when I do it's usually in the terminal and df is quick and easy. I did just recently add a drive percentage indicator to my conky script and I can monitor my install and backup partitions. I have a ton of available space so it is not much of a concern. My boot file system is approx. 460GB (2x250 SSD in RAID0) and two aging hard drives that were pulled from my server over the years. I use the HDs for backups and "play ground" - like wiping and creating partitions for the test I did above. most of my partition/file systems are of fixed use - meaning I'm not adding anything to them. With my backups and regular snapshots, my install file system sits at about 60% full. I check my server more often, but it's even less full at this point.

As far as EXT4 vs. BTRFS, I've written a ton of threads about it elsewhere on this forum so there's no need to go into it here. No one I know of has switched back to EXT4 once they experience all the benefits of BTRFS. It's just too easy.

[oshunluvr]

BTW, for you EXT4 users, you can free those reserved blocks anytime your file system is unmounted. The tunefs command above is non-destructive. If you're working on a system (boot) drive, leave a small percentage reserved. If it's a pure data drive, free them all.

[Skal1]

@jglen490: Its mostly for convenience I would say. This laptop is going to be my everyday-carry and I tend to amass data. I just chucked in the additional drive so I wouldn't have to worry. I'm fairly certain I wont use ALL of this space but its nice-to-have.

@oshunluvr: Thank you a lot for this in-depth explanation. I will definitely change the 5%.

Which tools do you personally use? Just a quick df?

Regarding the GRUB confusion: I read this thread https://www.kubuntuforums.net/showth...lp-the-new-guy here and it says to use 'apt-cache show grub | grep ersion' to find out which version you're using. This produces the strange output with Legacy etc that got me confused. I tried 'grub-install --version' which gives the correct version. And I'm indeed running 2.02 like you all said.

I don't disagree that ext4 is ancient - like me! But, I also don't doubt that it is mature and useful - like me, well sort of ...

I don't doubt that BTRFS is a useful approach to hosting a computing platform. I don't hate it, but at the same time I don't use it - I just ignore it and will continue to until Linux no longer supports ext4. ext4 is sufficient for my needs. If the OP were to choose going the BTRFS route, I wouldn't feel bad at all!

I think I will stick to EXT4 for now. Theres plenty of other things to learn first. Although the snapshot feature I've read about in the other thread sounds really nice so I might come back to it.

I will reinstall later today or tomorrow and report back. Thank you all for your help, it is greatly appreciated!

[vinnywright]

I don't disagree that ext4 is ancient - like me! But, I also don't doubt that it is mature and useful - like me, well sort of ...

I don't doubt that BTRFS is a useful approach to hosting a computing platform. I don't hate it, but at the same time I don't use it - I just ignore it and will continue to until Linux no longer supports ext4. ext4 is sufficient for my needs. If the OP were to choose going the BTRFS route, I wouldn't feel bad at all!

I use both ,,,but am slowly ,,,very slowly ,,,converting to btrfs ,,,I just love the ease of recovering from a update/install catastrophe

vinny@vinny-Bonobo-Extreme:~$ sudo parted -l
[sudo] password for vinny: 
Model: ATA HGST HTS725050A7 (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start   End    Size    Type      File system     Flags
1      8225kB  323GB  323GB   primary   btrfs           boot
3      323GB   379GB  56.3GB  primary   ext4
4      379GB   496GB  117GB   extended
6      379GB   436GB  57.0GB  logical   ext4
5      436GB   496GB  59.8GB  logical   ext4
2      496GB   500GB  4295MB  primary   linux-swap(v1)

Model: ATA HGST HTS721010A9 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
1      1049kB  1000GB  1000GB  btrfs        primary

VINNY

[jglen490]

I don't disagree that ext4 is ancient - like me! But, I also don't doubt that it is mature and useful - like me, well sort of ...

I don't doubt that BTRFS is a useful approach to hosting a computing platform. I don't hate it, but at the same time I don't use it - I just ignore it and will continue to until Linux no longer supports ext4. ext4 is sufficient for my needs. If the OP were to choose going the BTRFS route, I wouldn't feel bad at all!

[oshunluvr]

EXT4 is an ancient file system. By default, it reserves 5% of disk space for root and system services - whatever that means. This was fine and needed back when drives were 10-20GB. Now it's a ridiculous default. Try this on a terminal:

sudo tune2fs -l /dev/nvme0n1p3 | grep "Reserved block count"

and you should get a large number. You can reduce the percentage with:

sudo tune2fs -m .5 /dev/sdb1

which is half-a-percent. This will recover most of that space. I use zero percent for data storage drives when using EXT4 (which is hardly ever).

Also though, you need to accept that file systems have overhead (meta data, bad blocks, lost and found, etc.) and drive manufacturers play fast and loose with measurements of capacity and not all Linux tools are perfect. You just aren't ever going to get to a zero-diff result.

For your re-install, boot to the LiveUSB and use the Partition Manager to reformat the disk. 4096.00 MB on the swap partition will work there (partition manager is "smarter" than the installer program). After creating your EXT4 install partition, partition manager will show you the "used" space in the Properties window. I tested creating a new 16GB partition and it showed 430MB as "used" even though it was empty and even after I freed the reserved blocks. Here's what df reports before and after changing the reserved blocks to zero percent.

/dev/sda7           16G   44M   15G   1% /mnt/test
/dev/sda7           16G   44M   16G   1% /mnt/test

So so can see df shows more available space but the same amount of used space. These tools just aren't that accurate. I'm not saying disregard them totally, but it's really a comparison tool. Format your partition, free most of the reserved blocks, and do df. Then install and afterward do df again. The difference is a reasonably accurate accounting of how much space you used. That's what they're for.

Just for fun, I reformatted the same 16GB partition with BTRFS and NTFS;

NTFS:   16G   65M   16G   1% /mnt/test
BTRFS:  16G   17M   14G   1% /mnt/test

To GRUB: GRUB 1 aka Grub Legacy was shelved in 2010 and is no longer supported. It's last version number was actually 0.97. Ubuntu variants have been using GRUB 2 since then. They stopped calling it "Grub 2" or "Grub PC" around 2012 and just call it GRUB now. I think we're on GRUB 2.02 or something like that version-wise. Grub 2 supports EFI and Legacy boot modes.

[jglen490]

So this whole thing of disk size and numbers of this or that is more magic than forensic accounting. The disk sizes of 256GB and 1TB are "marketing specs" and are (fortunately) conservative based on the computations based on 1,000 or 1,024. Then you throw in the overhead of the actual file system once the drives are formatted - journaling and partition structure take away otherwise "usable" space. Then there's what the OS needs beyond the physical size of the files, such as all those temp spaces that show up when querying the size or the remaining space. Then there is the nature of the portrayal of sizes when using things like the df command. So the bottom line on that is, unless you are charging rent on your hard drive space, the specific numbers down to the nearest byte, or word boundary even, is pretty much immaterial.

What I use on my personal machine is a "sufficiency" judgement.
a) Do I want swap space and is what I've allocated sufficient? With today's RAM availability, in most cases whatever swap space you allocate will never be used. So that's the last thing I worry about.
b) What size do I need for the OS and application programs? This deal with what specifically falls under the / directory, other than /home. 20GB, 30GB, even 50GB is probably moe than most people will need, so dedicating your 256GB drive to / (including your UEFI needs) is probably way beyond sufficient. But that's O.K. it's a decision.
c) I require that /home be in its own partition, so how much space do I need? In my math, it's whatever is left after reserving OS, application, and swap space. So again you have a ton of space for configuration files identified specifically to your user name/space, plus data files (multimedia, docs, personal business).

So what you have is generous, at the very least.

I run 16.04.3 LTS on my desktop box. It came out of the box, or out of the download actually, with GRUB2. So unless you made the effort to bring in GRUB1, you're using GRUB2 - and that's terrific.

So you have a machine that works and that has generous space with flexibility to move around in, which equates to the best of all worlds and plenty of "sufficiency".

Use it, enjoy it! Don't worry about the decimal points!!

[Skal1]

First of all, thank you for your reply.

Yeah, your sizing was a little off. You need 8388608 sectors to get to 4GB. Remember it's 4*1024*1024. 4096 is 4*1024 so 4194 would have been closer. This is why partitioning before using the installer is the best way to go. fidisk or gdisk will allow you to enter a partition size in GB or MB then this doesn't happen. Having said that, you're likely going to be fine unless you're using suspend. Resizing and moving the other partition is very time consuming and risky. If you really feel the need to fix this, I'd delete the third partition, re-create the swap partition the correct size, then re-install. It will be faster than resize operations.

Yes thats what I figured.

I think I will just reinstall and partition before. Its a fresh install and I want to set it up properly. If I use the Partitionmanager it will automatically align the partitions right? I don't want to worry about alignment right now.

Checking disk usage is always a non-specific thing. Sector sizes, file system overhead, and other things all play a part. The point is you'll never really know exactly how much space is used or available, but you'll know enough to use it correctly. Does it really matter if it's 214GB available or 213.7GB? Not really. EXT4, like all other file systems, reserves some space for it's own use. Files take more space on the drive than their actual size because of sector sizes. You really needn't worry about drive space until you reach the 90% used area. Then start taking a look at what you've got on there.

Of course it doesn't really matter if it is 214G or 213,4G and I understand that the filesystem needs some space aswell. But the differences I seem to be getting are quite extreme.

Of a 931GB partition I can use 917GB of which 870GB are only available. The decrease from 931 to 917 I can accept as reserved space for the file system - no biggie. But where does the rest come from? Even df -h says only ~400MB are used so where are the other 50GB? I just lack the experience to judge if this is within the norm are some kind of issue.
Or are you saying I should just disregard those disk-usage tools alltogether?

Regarding GRUB, not sure what your question is exactly. "GRUB Legacy" usually refers to "GRUB 1" or the old version of grub we left behind years ago. We now use "GRUB PC" or "GRUB 2". Now "Legacy booting" - a BIOS setting - means "Non-UEFI booting" and doesn't actually refer to GRUB. GRUB 2 can boot in legacy mode or EFI mode or you can use some other boot manager. It sounds like you've set up GRUB/EFI booting so leave it that way. If it ain't broke...

Okay, that clears it up. Does 16.04 ship with GRUB 1? I guess GRUB 1 supports UEFI mode aswell since it seems to be working for me. So no need to update to GRUB 2 right?

Really appreciate the help, I'm loving this forum so far!

Regards
Skal

[oshunluvr]

Yeah, your sizing was a little off. You need 8388608 sectors to get to 4GB. Remember it's 4*1024*1024. 4096 is 4*1024 so 4194 would have been closer. This is why partitioning before using the installer is the best way to go. fidisk or gdisk will allow you to enter a partition size in GB or MB then this doesn't happen. Having said that, you're likely going to be fine unless you're using suspend. Resizing and moving the other partition is very time consuming and risky. If you really feel the need to fix this, I'd delete the third partition, re-create the swap partition the correct size, then re-install. It will be faster than resize operations.

Checking disk usage is always a non-specific thing. Sector sizes, file system overhead, and other things all play a part. The point is you'll never really know exactly how much space is used or available, but you'll know enough to use it correctly. Does it really matter if it's 214GB available or 213.7GB? Not really. EXT4, like all other file systems, reserves some space for it's own use. Files take more space on the drive than their actual size because of sector sizes. You really needn't worry about drive space until you reach the 90% used area. Then start taking a look at what you've got on there.

Regarding GRUB, not sure what your question is exactly. "GRUB Legacy" usually refers to "GRUB 1" or the old version of grub we left behind years ago. We now use "GRUB PC" or "GRUB 2". Now "Legacy booting" - a BIOS setting - means "Non-UEFI booting" and doesn't actually refer to GRUB. GRUB 2 can boot in legacy mode or EFI mode or you can use some other boot manager. It sounds like you've set up GRUB/EFI booting so leave it that way. If it ain't broke...

[Skal1]

Hello people,
I have some questions regarding the installation/partitioning process. But let me first say that my installation is up and running right now and I've not ran into any problems (yet). However I'm a bit confused and suspect I did something wrong.

What I wanted to do: My new laptop (no OS) has a 256GB SSD and a 1TB HDD. My plan was to have my /home on the HDD and root on the SSD. Also I wanted to use the UEFI my device offers.
What I did: I used the partitioning tool inside the installer (manual option). As for my HDD I just made one large ext4 partition with mounting point '/home'. On the SSD the first partition I created was the UEFI one. I choose the EFI option and entered the size '256'. As the second partition I choose the swap option and entered the size '4096'. Then I created a third partition the size of the remaining space with ext4 and mounting point '/'.

I ran the installation which seemed to work fine.

1. Did I choose the partition sizes correctly? IIRC the installer said that the sizes are in MiB. Thats why I choose values like 256 and 4096. However after partitioning the sizes are all different (smaller). Is this because the file system needs some space aswell? Or did I make a conversion error? E.g. my swap should be 4GiB as I set it to 4096MiB but the partition manager only sees 3,81GiB. You can see the different sizes on the screenshots below.

2. How do I check disk usage correctly? This came to my attention when I was looking at the KDE Partition Manager after the successful installation. Some disk usage values were downright impossible. E.g. the KDE partition manager tells me that my 3,81GiB swap has a usage of 47,05GiB. Funnily thats the same usage my /home HDD is supposed to have. Keep in mind that this is on a fresh install. I hadn't moved anything into my home folders yet and they're supposed to be almost 50GiB?! This made me suspicious so I installed Gparted. And voila, gparted offered yet another set of different usage values.

Although they were more believable I still wasn't convinced so I looked up how to check disk space in the terminal. Running 'fdisk -l', 'df -a' and 'df -h' left me even more confused.

Let me give you an example: My HDD is 1TB so the physical size should be 931,3GiB. The partition should be the same and in fact the KDE Partition Manager, Gparted and fdisk display a size of 931,5GiB. Running df -h shows a size of 917G. Now the strange part: KDE Partition Manager shows a disk usage of 47,05GiB, Gparted says 15,2GiB, df -a shows 476332B, df -h shows 473M. This leaves me with different available diskspace: Gparted says 916,3GiB are unused, KDE partman would leave me with 931,5-47,05=884,45GiB, Dolphin and df -h says 870G. I have linked some screens and output below.

Maybe you can see where my confusion comes from. Its hard for me to believe that a fresh install would consume so much space (eg. 47G on a fresh /home). And why I didn't have the full partition size available in the first place (eg. only 917G of 931G). Maybe I made a mistake during the installation?

Info
Screenshots: https://imgur.com/a/iiwa3

                                                                                                 
dani@dani-ThinkPad-E570:~$ sudo fdisk -l                                                                                             
[sudo] password for dani:                                                                                                            
Disk /dev/nvme0n1: 238,5 GiB, 256060514304 bytes, 500118192 sectors                                                                  
Units: sectors of 1 * 512 = 512 bytes                                                                                                
Sector size (logical/physical): 512 bytes / 512 bytes                                                                                
I/O size (minimum/optimal): 512 bytes / 512 bytes                                                                                    
Disklabel type: dos                                                                                                                  
Disk identifier: 0x810f1709                                                                                                          
                                                                                                                                  
Device         Boot   Start       End   Sectors   Size Id Type                                                                       
/dev/nvme0n1p1 *       2048    499711    497664   243M ef EFI (FAT-12/16/32)                                                         
/dev/nvme0n1p2       499712   8499199   7999488   3,8G 82 Linux swap / Solaris                                                       
/dev/nvme0n1p3      8499200 500115455 491616256 234,4G 83 Linux                                                                      
                                                                                                                                  
                                                                                                                                  
Disk /dev/sda: 931,5 GiB, 1000204886016 bytes, 1953525168 sectors                                                                    
Units: sectors of 1 * 512 = 512 bytes                                                                                                
Sector size (logical/physical): 512 bytes / 4096 bytes                                                                               
I/O size (minimum/optimal): 4096 bytes / 4096 bytes                                                                                  
Disklabel type: gpt

dani@dani-ThinkPad-E570:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3,9G     0  3,9G   0% /dev
tmpfs           789M  9,3M  779M   2% /run
/dev/nvme0n1p3  231G  5,7G  214G   3% /                                                                                              
tmpfs           3,9G   51M  3,8G   2% /dev/shm                                                                                       
tmpfs           5,0M  4,0K  5,0M   1% /run/lock                                                                                      
tmpfs           3,9G     0  3,9G   0% /sys/fs/cgroup                                                                                 
/dev/nvme0n1p1  240M  3,4M  236M   2% /boot/efi                                                                                      
/dev/sda1       917G  473M  870G   1% /home                                                                                          
tmpfs           789M     0  789M   0% /run/user/118                                                                                  
tmpfs           789M   12K  789M   1% /run/user/1000

• What release of Kubuntu you are using: 16.04 LTS
• If Kubuntu is installed, is it installed 'inside' of Windows (Wubi installation): No
• What version of KDE you are using: Plasma 5.5.5
• What version of Grub you are using: Version: 0.97-29ubuntu68
  Description-en: GRand Unified Bootloader (Legacy version)
• What other Operating Systems are installed: None

• Type: Laptop (Lenovo ThinkPad E570 20H6)
• CPU: Intel i5 7200U (64bit)
• GPU: Intel integrated graphics + Nvidia 950M (no driver yet)
• RAM: 8GB
• HDs: 256GB internal nvme SSD, 1TB internal Sata HDD
• Optical Drives: Internal DVD

Unrelated: GRUB Legacy Version. Is that 'Legacy' as in 'not-UEFI'? I've set my bootoptions to 'UEFI only' and Kubuntu boots without problem so I guess UEFI boot is working correctly?

Thanks in advance for your answers

Regards,
Skal