mount: / is busy and system hangs without shutting down

[oshunluvr]

Can't hurt. Likely cant redirect to a file this late in the shutdown.

You could remove the redirect, record the scrolling text with your phone, play it back in slo-mo

[Frank616]

Could still use some help with this, thanks.

Did the pipe fail because the machine is mostly shut down? It worked on my own machine when I tried it from a console. Should I try

lsof | more

?

Frank.

[Frank616]

OK, just got back to sleuthing this issue.

I edited /etc/init.d/umountroot to look as follows:

MOUNT_FORCE_OPT=
    [ "$(uname -s)" = "GNU/kFreeBSD" ] && MOUNT_FORCE_OPT=-f
    # This:
    #     mount -n -o remount,ro /
    # will act on a bind mount of / if there is one.
    # See #339023 and the comment in checkroot.sh
    mount    $MOUNT_FORCE_OPT -n -o remount,ro -t dummytype dummydev / 2>/dev/null \
    || mount $MOUNT_FORCE_OPT -n -o remount,ro              dummydev / 2>/dev/null \
    || mount $MOUNT_FORCE_OPT -n -o remount,ro                       /
    lsof       >  /home/irma/lsof.txt
    ps ajx    >  /home/irma/ps.txt
    ES=$?
    [ "$VERBOSE" = no ] || log_action_end_msg $ES
}

As you can see, I played with the commands in an effort to pipe the output to a text file that I could read later, as otherwise the output goes by on the screen too quickly to read. Anyway, the system did not like my little effort, calling my command a 'garbage option'! (snif). The output I got is on the attached screenshot.

Suggestions? Corrections?

Thanks.

Frank.

[oshunluvr]

Its two commands, so try

lsof
ps ajx

[Frank616]

oshunluvr:

OK, I finally stopped and READ you example of your SEARCH for umount. I have opened /etc/init.d/umountroot

The script is as follows:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          umountroot
# Required-Start:
# Required-Stop:
# Should-Stop:       halt reboot kexec
# Default-Start:
# Default-Stop:      0 6
# Short-Description: Mount the root filesystem read-only.
### END INIT INFO

PATH=/sbin:/bin
. /lib/init/vars.sh

. /lib/lsb/init-functions

do_stop () {
    [ "$VERBOSE" = no ] || log_action_begin_msg "Mounting root filesystem read-only"
    # Ask init to re-exec itself before we go down if it has been
    # upgraded this cycle.  It'll lose all its state, but at least
    # it won't hold open files on the root filesystem (lp:#672177).
    if [ -f /var/run/init.upgraded ]
    then
        old_map=$( cat /proc/1/maps )
        map=$old_map
        telinit u || :
        i=0
        timeout=5
        while [ "$map" = "$old_map" ]
        do
            sleep 1
            map=$( cat /proc/1/maps )
            /usr/bin/logger "waiting for init to respawn"
            i=$((i+1))
            if [ $i -eq $timeout ] ; then
                break
            fi
        done

        if [ "$map" = "$old_map" ] ; then
            /usr/bin/logger "FAIL: init failed to respawn in $timeout seconds - unmounting anyway"
        else
            /usr/bin/logger "SUCCESS: init respawned after $i seconds (within $timeout seconds timeout)"
        fi
    fi

    MOUNT_FORCE_OPT=
    [ "$(uname -s)" = "GNU/kFreeBSD" ] && MOUNT_FORCE_OPT=-f
    # This:
    #     mount -n -o remount,ro /
    # will act on a bind mount of / if there is one.
    # See #339023 and the comment in checkroot.sh
    mount    $MOUNT_FORCE_OPT -n -o remount,ro -t dummytype dummydev / 2>/dev/null \
    || mount $MOUNT_FORCE_OPT -n -o remount,ro              dummydev / 2>/dev/null \
    || mount $MOUNT_FORCE_OPT -n -o remount,ro                       /
    ES=$?
    [ "$VERBOSE" = no ] || log_action_end_msg $ES
}

case "$1" in
  start)
    # No-op
    ;;
  restart|reload|force-reload)
    echo "Error: argument '$1' not supported" >&2
    exit 3
    ;;
  stop)
    do_stop
    ;;
  *)
    echo "Usage: $0 start|stop" >&2
    exit 3
    ;;
esac

:

So, If I have finally understood correctly, I should end up with the following snippet, right?

MOUNT_FORCE_OPT=
    [ "$(uname -s)" = "GNU/kFreeBSD" ] && MOUNT_FORCE_OPT=-f
    # This:
    #     mount -n -o remount,ro /
    # will act on a bind mount of / if there is one.
    # See #339023 and the comment in checkroot.sh
    mount    $MOUNT_FORCE_OPT -n -o remount,ro -t dummytype dummydev / 2>/dev/null \
    || mount $MOUNT_FORCE_OPT -n -o remount,ro              dummydev / 2>/dev/null \
    || mount $MOUNT_FORCE_OPT -n -o remount,ro                       /
    lsof ps ajx
    ES=$?
    [ "$VERBOSE" = no ] || log_action_end_msg $ES
}

Sorry, its been a long day already....

Thanks for your patience.

Frank.

[Frank616]

oshunluvr and vinny:

next time it hangs ,,,,,,try ,,,,,press and hold "ctrl+alt+SysRq" wile holding them down press one at a time S then U then B ,,,,,,,,,,the system should reboot.

Did nothing. System is shut down, right? Should the keyboard do anything at this point?

Just for fun, boot up that system and kill network manager completely then try shutting down

As soon as I kill it, it restarts.


I will try yet another update today and see if that changes anything. If not, then I'll try as you suggest. I just looked up those commands and now understand that they should give more information so we know where to go from here. However, I cannot find an 'ajx' command Is there a spelling error there?

Correction: I see that ajx are flags. sorry.

try adding the lsof and/or ps ajx commands below the three lines that contain the remount command.

Finally, do I understand correctly that the commands should go after the last UNMOUNT command? I see no 'remount' commands in there. The file should look like:

:~$ locate umountroot
/etc/init.d/umountroot
/etc/rc0.d/K11umountroot
/etc/rc6.d/K11umountroot
lsof ps ajx
/lib/systemd/system/umountroot.service

I thank you both for taking the time to help out.

Correction: I totally misread the above LISTING. sorry. Another post to follow this one....

Frank.

[oshunluvr]

You should be able to kill networkmanager from ksysguard.

My system shows umountroot at:

:~$ locate umountroot
/etc/init.d/umountroot
/etc/rc0.d/K11umountroot
/etc/rc6.d/K11umountroot
/lib/systemd/system/umountroot.service

The last one is a link to null, so must be a place holder for future use. The 2nd and 3rd both link to the first, so edit /etc/init.d/umountroot. try adding the lsof and/or ps ajx commands below the three lines that contain the remount command.

[vinnywright]

Vinny:

I'll try that.

I booted from a Kubuntu USB stick, and ran fsck on the offending sdb1 partition. It found a block number error. I asked it to fix it, and now it comes back 'clean', but I still get the kernel panic. I'll try gparted tomorrow or Thursday when I have some time off again.

Frank.

if this is the same one you did this to ,,,,,,,

1. I changed this line in file /etc/default/grub to look like this: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash init=/lib/systemd/systemd"
2. After that, I ran sudo update-grub in a Terminal

,,,,,,,then undo it ,,,,their is no systemd in 14.04

OOPS wrong about that ,,,,,,,,,,but I would still try undoing it.

VINNY

[Frank616]

Vinny:

I'll try that.

I booted from a Kubuntu USB stick, and ran fsck on the offending sdb1 partition. It found a block number error. I asked it to fix it, and now it comes back 'clean', but I still get the kernel panic. I'll try gparted tomorrow or Thursday when I have some time off again.

Frank.

[vinnywright]

next time it hangs ,,,,,,try ,,,,,press and hold "ctrl+alt+SysRq" wile holding them down press one at a time S then U then B ,,,,,,,,,,the system should reboot.


VINNY

[Frank616]

oshunluvr,

I thought I had edited that KDM comment. Sorry. It is indeed a kernel panic, and I knew that when I posted. Just missed editing it out, I guess.

OK, how can I kill network manager before trying a shutdown?

Where will I find the unmountroot script?

Thanks for the suggestions.

Frank.

[oshunluvr]

Could be a process still has an open file and is preventing umountroot from remounting the root filesystem as read-only (this is part of the shutdown process). I see a couple Network Manager messages there on your screen shot. It could be the culprit. Just for fun, boot up that system and kill network manager completely then try shutting down. If that doesn't work, maybe adding this to the umountroot script will give some useful output:

ps ajx or maybe lsof

As always, have a liveUSB bootable handy before doing any system edits.

The other screenie shows a kernel panic (not KDM panic?). Is that what you meant? We haven't used KDM by default in quite awhile.

[Frank616]

It seems this problem is back again:

https://www.kubuntuforums.net/showth...ght=mount+busy


I'm about ready to pull my hair....

The more I try, it seems the worse it gets. Please see screenshot.

I tried a solution suggested here:

http://askubuntu.com/questions/55897...hutdown-reboot

The last message in that thread suggests:

I had the same problem with Kubuntu 14.10. My solution was:

1. I changed this line in file /etc/default/grub to look like this: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash init=/lib/systemd/systemd"
2. After that, I ran sudo update-grub in a Terminal

Now the system shuts down again...:-)

This did not appear to do anything on the first machine, the screenshot of which I have attached.

It worked once on the second machine (more about that later), but now that second one won't boot at all. It stops midway with a kdm panic of some description.

I can only get these machines to shut off if I press and hold the power button, and that messes up the system drive. Restarting normally then gives a kernel panic. Restarting again in safe mode, and running fsck, then continuing the boot brings it back up again, but the next shutdown is the same thing all over again.

The first machine has a brand-new fresh install of 14.04 with the latest updates all installed. Boot disk is a brand new Samsung 850 SSD. It has a second mechanical hdd, but I have not been able to get to the point of using it yet, as I can't resolve the shutdown issue.

I would chalk this up to a specific issue with this machine if my second desktop machine were not displaying similar problems. It too now hangs on shutdown, with similar error messages. It is a mature install from two years ago, and it too was recently updated to the latest software available. Trying to boot into safe mode on that second machine and running fsck never seems to terminate, and rebooting again just gives the kernel panic. I can't get it going again at all. Please see second screenshot.

I tried booting a previous kernel on both machines, but no change on exit.

Both machines are running Kubuntu 14:04 with 3.13.0-83-generic, and are both 64 bit machines with 64 bit kernels.

I am down to my laptop now, and, for the time being, I will NOT update it.

At this point, at least the one machine works if I run fsck in recovery mode. As noted, the second one seems to be hung in fsck at the moment. I could probably drop to a command shell and try running fsck that way, but I will need some help with that.

Any help or suggestions appreciated.

Thanks.

Frank.