The dd Command

[Qqmike]

dd: Showing the progress as dd executes


As dd runs, it does not show you its progress; it only reports at the end when it is done--and that can take quite awhile for some jobs. There are a couple easy (non-expert) ways to see the progress of dd as it copies.

Do you have GNU Coreutils 8.24+ (Ubuntu 16.04 and newer)?

(1) If so, you can use status=progress in your dd statement. See below.

(2) If not, then you can use the utility pv in your dd statement. See below.


What version of coreutils do you have?
Here are four ways to check:
-- Muon, type coreutils, left-click on coreutils entry to highlight it, click the version tab.
-- At Konsole, run apt-cache policy coreutils
-- At Konsole, run sudo dpkg -s coreutils
-- Check the Ubuntu packages:
http://packages.ubuntu.com/search?keywords=coreutils
It shows the version according to each K(U)buntu version; for example:
Package coreutils
precise (12.04LTS) (utils): GNU core utilities, 8.13-3ubuntu3.3
trusty (14.04LTS) (utils): GNU core utilities 8.21-1ubuntu5.1
xenial (16.04LTS) (utils): GNU core utilities 8.25-2ubuntu2
yakkety (16.10) (utils): GNU core utilities 8.25-2ubuntu2
zesty (utils): GNU core utilities 8.25-2ubuntu2


Btw, what is coreutils?
https://en.wikipedia.org/wiki/GNU_Core_Utilities
http://www.gnu.org/software/coreutils/coreutils.html



(1) dd progress for GNU Coreutils 8.24+ (K(U)buntu 16.04 and newer)


--> Use the status=progress option in your dd statement.
See man dd (at Konsole in your K(U)buntu 16.04 or newer version OS).

Example To zero-out a 16 GB flashdrive and see the progress as dd executes:

 [FONT=Liberation Serif][COLOR=#000000]mike@mike-All-Series:~$ [/COLOR][SIZE=3][COLOR=#000000][B]sudo dd if=/dev/zero of=/dev/sdb bs=16M status=progress[/B][/COLOR][/SIZE][/FONT]
 [COLOR=#000000][FONT=Liberation Serif]...[/FONT][/COLOR]
 [COLOR=#000000][FONT=Liberation Serif]it will show you the #bytes completed and the rate (MB/s) as it goes along; and when done, it shows you the result:[/FONT][/COLOR]
 [FONT=monospace]
[FONT=Liberation Serif]1593835520 bytes (1.6 GB, 1.5 GiB) copied, 46.8188 s, 34.0 MB/s
...[/FONT][/FONT]
 [FONT=Liberation Serif][COLOR=#000000]1610612736 bytes (1.6 GB, 1.5 GiB) copied, 71.1188 s, 22.6 MB/s[/COLOR]
...[/FONT]
 [FONT=Liberation Serif][COLOR=#000000]2164260864 bytes (2.2 GB, 2.0 GiB) copied, 248.283 s, 8.7 MB/s [/COLOR]
...
[COLOR=#000000]6325010432 bytes (6.3 GB, 5.9 GiB) copied, 1425.35 s, 4.4 MB/s [/COLOR]
...[/FONT]
 [FONT=Liberation Serif][COLOR=#000000]15602810880 bytes (16 GB, 15 GiB) copied, 4003.56 s, 3.9 MB/s   [/COLOR]
dd: error writing '/dev/sdb': No space left on device[/FONT]
[FONT=Liberation Serif][COLOR=#000000]931+0 records in [/COLOR]
930+0 records out 
15610576896 bytes (16 GB, 15 GiB) copied, 4508.64 s, 3.5 MB/s[/FONT]

Note: We used a block size of 16 MiB, bs=16M, which is kind of arbitrary, and up to the user. You can omit bs= and it will use the default bs=512. Or try bs=4096. See man dd.


(2) dd progress for GNU Coreutils older than GNU Coreutils 8.24

--> Use the pv utility in your dd statement.


First, get the pv package.
For example, run at Konsole: sudo apt-get install pv. Or, open Muon, type pv in the search window, mark it for Installation, and click Apply.
Then see man pv.

From the package description:

pv - Pipe Viewer - is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

ref.: https://www.ivarch.com/programs/pv.shtml


Example Zero-out a 16 GB flashdrive and see the progress as dd executes

dd occurs twice in the statement below, and each occurrence requires sudo (sudo dd). Instead of doing it that way (which would also work), we will first get a root terminal using sudo su; then proceed (without sudo's); when done, issue an exit so we return to the normal user (non-root) prompt. The | is a pipe symbol. Like this:

mike@mike-desktop:~$ sudo su
root@mike-All-Series:/home/mike# [B][SIZE=3]dd if=/dev/zero | pv -s 16G  -pterb |  dd  of=/dev/sdb bs=16M[/SIZE][/B]

 ...

as it goes along, it will show you its progress from time to time
 ...
and then it will finish:
0+188740 records in 
0+188739 records out 
14.5GiB 1:11:08 [3.49MiB/s] [====================================>     ] 90%             

 15610576896 bytes (16 GB, 15 GiB) copied, 4268.09 s, 3.7 MB/s 
root@mike-All-Series:/home/mike# exit 
exit 
mike@mike-All-Series:~$

Note about pv

See man pv. Thus, we used the five default options -pterb, along with our estimate of the size of the copy, which we set at 16 GiB: -s 16G . If you want progress reports on percent complete, you must supply an estimate of the size; i.e., you must use the -s option.

[Qqmike]

jlittle, and so, with tapes, and with dd, that's where they used the more technical (and, to me, confusing) ibs and obs (input, output block sizes) rather than the single bs (block size).

[jlittle]

<nostalgia mode="old fart">
I can remember solving problems using dd in the middle 1980's. With half inch tape, I think. Block sizes really were block sizes.
</nostalgia>

[Qqmike]

Edits to:

The dd Command Parts 1,2, and 3:
https://www.kubuntuforums.net/showth...The-dd-Command

https://www.kubuntuforums.net/showth...l=1#post110611

https://www.kubuntuforums.net/showth...l=1#post117130

I went through all three parts and caught some typos, nothing really significant (except one statement, where b2=4096 should be bs=4096): Edits include Parts 1, 2, 3 in Posts #1, #2, #3.

Boy, if you are content with how your Kubuntu OS is running, bored, looking for something to do, don't know dd or are rusty, try this stuff ;-) (if you're nuts enough to do so).

At one time, when I wrote this, and did all the experiments/examples, I was hot on it, suitably obsessed. But then didn't look it, except to do occasional tasks with dd. Now I see there's some neat things here I had forgotten and that one could play with some more:

-- Cloning a flash drive to an identical one (which I do find myself doing on occasion).
-- Using sync after dd (I notice S-R doing this on occasion).
-->> Backup your Kubuntu OS root partition to your separate /home partition (as a file). Which, at one time, I used to do, and it worked well--do see the discussion in Part 1 on this, though.
-- A reminder about using Qalculate.
-- Using dcfldd (it's listed in Muon Package Manager)--pretty neat.
-- And using dd to clean stuff for privacy, before reusing it, like unused space on your home or root partition; a file; a partition; a flash drive or HDD; a MBR/partition table, or the first 63 sectors of a MBR scheme. For example, after running Bleachbit, how to clean the empty space leftover.


The examples involving BIOS are based on the traditional MBR scheme, its 512 bytes, the first 63 sectors, and so on. Thought about extending the examples to the GPT scheme, and I may do that sometime. But I realize that the reader can easily do it simply by applying the exact principles in the how-to. Easy enough: just get your skips, seeks, and counts right ... and your bs (that would be, of course, block size). It would involve exploring (uhmmm, playing with) the many features of a GPT setup:

The Protective Master Boot Record
1 MiB gap and alignment
Partition Table Header
Partition entry array
The last sector of the disk (LBA -1) contains the secondary or backup Partition Table Header
and the sector preceding the last sector (LBA -2) contains the secondary or backup partition table.
And possibly, in some setups, the Bios Boot Partition.

For this stuff, see the study guide and references therein:
GPT, UEFI -- Study Guide
https://www.kubuntuforums.net/showth...l=1#post346604

Added:
I just noticed that TestDisk has been extended for EFI GPT, very nice. Thought of this with the dd experiments, messing up a partition (intentionally, or as a test) and recovering it using TestDisk.

[#!/bin/bash]

Re: The dd Command

Nice example.

Let's just check one thing:
$ sudo dd if=/dev/zero of=/boot/swap.img bs=1024 count=500000

Should that be count=1000000? (To get 1 GB) (Or, use bs=2048?)

Thanks.

You are right. I ran the command twice and created a 500Mb file the first time then created the 1Gb file. I guess I copied the wrong command. I think the rest of it is correct though.

[Qqmike]

Re: The dd Command

Nice example.

Let's just check one thing:
$ sudo dd if=/dev/zero of=/boot/swap.img bs=1024 count=500000

Should that be count=1000000? (To get 1 GB) (Or, use bs=2048?)

[#!/bin/bash]

Use dd to create a swap file.

Here is one for your examples section:
Create a 1GB swap file.

This command creates the empty file
$ sudo dd if=/dev/zero of=/boot/swap.img bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 3.41333 s, 300 MB/s

This command turns the empty file into a swap file
$ sudo mkswap /boot/swap.img
Setting up swapspace version 1, size = 999996 KiB
no label, UUID=795987b3-d0d5-4c5b-8737-0a29199716b7

This command tells the system to start using the swap file
$ sudo swapon /boot/swap.img

This command verifies that the swap file is being used
$ swapon -s
Filename Type Size Used Priority
/dev/sdb5 partition 975868 16 -1
/boot/swap.img file 999996 0 -2

Lastly you need to edit /etc/fstab and add the swapfile entry
$ sudo nano /etc/fstab

/boot/swap.img none swap sw 0 0

Now I can reclaim my /dev/sdb5 partition by removing the entry from fstab. Or I can leave it as is. If you increase your memory you don't have to fdisk your drive and add more swap space, just create a swap file.

<edit> Corrected count in dd command.

[Qqmike]

NEW-&gt; The dd Command

The dd Command

Brand new, more tutorial, more complete, tons of examples, much improved, completely rewritten.

How to use the dd command

In three parts:
Part 1: Contains the basics, plenty of discussion and guidance.
Part 2 is in Reply #1 and contains lots of applications/examples.
Part 3 is in Reply #2 and repeats 6 methods from Part 2, methods useful for
my how-to Privacy Cleanup 101, and is expanded and even more tutorial for beginners.

You can scroll up to the first post and read the Table of Contents for all three parts.

This one is loaded for the beginning-to-intermediate level.
16,500 words.

Not to say that I won't improve it and add a bit more here and there, I will. But it may be the most complete guide (I know of) on the Internet, mainly because there isn't much out there. Just a fact. I know, because I searched for stuff to read, and much of it is either incorrect or impossible to read or both. It also clarifies issues that are not correctly reported elsewhere, issues people seem to be confused about. (It no doubt needs some proofing and copyediting for some typos, but all the examples are fully tested.)

Could be fun stuff for folks wanting to learn some more Linux methods.

You don't have to use dd for everything, but almost everyone could find an application or two that would make their work easier.

[Qqmike]

Re: dd Command

Update: dcfldd on Helix CD

dcfldd no longer available on free Helix CD:
The Helix reference for free CD with dcfldd and how-to's:
Helix: http://www.efense.com/helix
No longer available free; by membership only.
Helix3 vs Helix3 Pro -- changes in the program and products
http://www.youtube.com/watch?v=soBNib2ydt0

To get dcfldd:
sudo apt-get install dcfldd
man dcfldd


dcfldd
> dcfldd(1) - Linux man page
http://linux.die.net/man/1/dcfldd
> See also, dev notes re verify capability:
http://www.networksecurityarchive.or.../msg00004.html
> dcfldd - Latest version 1.3.4-1
http://dcfldd.sourceforge.net/


I have modified the reference section of the OP to reflect this change in Helix products.

[txHarleyMan]

Re: dd Command

Anyone using these methods is smart enough to take precautions, just as they do when editing system files like fstab, menu.lst, X.org files, sources.list, ...
Or, smart enough to fix things/recover when they don't make those backups, which I'm sure is more often the case.

LOL... Well, I am just not so quick to make assumptions about people.

[Qqmike]

Re: dd Command

I think by now, that goes without saying, in reference to your anachronism for dd.
We did read the first post, right?
Anyone using these methods is smart enough to take precautions, just as they do when editing system files like fstab, menu.lst, X.org files, sources.list, ...
Or, smart enough to fix things/recover when they don't make those backups, which I'm sure is more often the case.
And, there will always be people who will never get if= and of= straight.

[txHarleyMan]

Re: dd Command

dd ( data destroyer ) is great as long as you are very careful. Have a backup, right?

[Qqmike]

Re: dd Command

S-H: "... but I wonder about eod (End of Disc) varieties??"

Yes, and spyware? trojans? various informational plants? who knows!

It's a good reason, after infection/suspicion, before reformatting and re-installing, to wipe the drive by a zero-fill (or zero-out the drive). If the drive is sdb, that would be, from a live CD:
dd if=/dev/zero of=/dev/sdb
or using the DoD version of dd from Helix Live CD:
dcfldd if=/dev/zero of=/dev/sdb
or, use a zero-fill program like the SeaTools utility.

[Snowhog]

Re: dd Command

Your post is informative and a good example for others to see and try.

Thank you. Not one to simply 'leap before looking', I'll leave the data in these last bytes alone (for now). As you stated, "Probably no harm is keeping it,"

The usefulness of this comparison - that my 'unaddressable sectors' do contain 'data' and yours does not - is that stuff can be written there! While not a Windoze user anymore, outside of keeping Bill trapped within a VM, this might explain why a Windoze user hit with a particularly nasty virus, can't seem to get rid of it, even after reformatting the HD and reinstalling the OS. mbr (Master Boot Record) viri are common enough, but I wonder about eod (End of Disc) varieties??

[Qqmike]

Re: dd Command


“First question: Does formatting a drive only affect the addressable sectors? Based on my output above, this would seem to be the case.”

I think so. There's a matter of terminology here, which is always confusing unless you program this stuff: “addressable” by the filesystem? But, yes, it certainly seems so.


“Second question: As I have only Linux on my HD now (Windoze runs in a VM), is it safe to zero out these bytes in the unaddressable sectors?”

Yes, you can zero it out. Probably no harm is keeping it, though, in your case – it looks like Vista put a phantom MBR there (note the 55 aa end-of-file signature at the end). Very strange message it put there, too.

I really don't like giving dd recipes, especially after a day of making typos as I have!
But it seems it would be this (>>>> please check the seek= bytes):
dd if=/dev/zero of=/dev/sda bs=512 seek=234436545

(and as you know from the dd stuff, you use seek rather than skip on the output file (“of”) end of the equation; so before writing zeros to sda, the dd command will skip 234436545 bytes on sda; and, as you know, if anything slips here, you will write zeros to that drive ...)

I heard somewhere that Vista plants such a thing at the end of the drive ...

Your post is informative and a good example for others to see and try.