No dns lookup after upgrade

[doctordruidphd]

After upgrading from Cosmic, which works as expected, and rebooting, I am unable to do any network name resolution. Pinging and connecting with firefox via ip address works fine, but name lookups do not. nslookup just sits there and eventually times out. I have tried stopping and restarting systemd-resolved.service and resolvconf-pull-resolved.service, with no effect. Now here is the really strange part: if I wait 5-10 minutes, then the miserable thing starts working!! I have no idea why, or what is kicking in.

The only thing I have been able to pin down is this: if I do a
systemctl status systemd-resolved.service
both before and after it works, I get the following:

Not-working:

greenman@Crynfyd19.04 ~$ cat not-working 
● systemd-resolved.service - Network Name Resolution
Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2019-04-28 14:44:21 PDT; 3min 24s ago
  Docs: man:systemd-resolved.service(8)
        https://www.freedesktop.org/wiki/Software/systemd/resolved
        https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
        https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
Main PID: 1010 (systemd-resolve)
Status: "Processing requests..."
 Tasks: 1 (limit: 4915)
Memory: 4.0M
CGroup: /system.slice/systemd-resolved.service
        └─1010 /lib/systemd/systemd-resolved

Apr 28 14:44:10 Crynfyd systemd[1]: Starting Network Name Resolution...
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: Positive Trust Anchors:
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: Using system hostname 'Crynfyd'.
Apr 28 14:44:21 Crynfyd systemd[1]: Started Network Name Resolution.

And working:

greenman@Crynfyd19.04 ~$ cat working
● systemd-resolved.service - Network Name Resolution
Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2019-04-28 14:44:21 PDT; 9min ago
  Docs: man:systemd-resolved.service(8)
        https://www.freedesktop.org/wiki/Software/systemd/resolved
        https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
        https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
Main PID: 1010 (systemd-resolve)
Status: "Processing requests..."
 Tasks: 1 (limit: 4915)
Memory: 4.3M
CGroup: /system.slice/systemd-resolved.service
        └─1010 /lib/systemd/systemd-resolved

Apr 28 14:44:10 Crynfyd systemd[1]: Starting Network Name Resolution...
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: Positive Trust Anchors:
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Apr 28 14:44:14 Crynfyd systemd-resolved[1010]: Using system hostname 'Crynfyd'.
Apr 28 14:44:21 Crynfyd systemd[1]: Started Network Name Resolution.
Apr 28 14:50:13 Crynfyd systemd-resolved[1010]: Using degraded feature set (UDP) for DNS server 192.168.1.1.

diff says the difference between the two, other than times and memory locations, is the last line, present in the "working" status:

Apr 28 14:50:13 Crynfyd systemd-resolved[1010]: Using degraded feature set (UDP) for DNS server 192.168.1.1.

Well this is bizarre. I don't know why it should start working several minutes after bootup.

PS I should mention two things: During the upgrade, somehow most of kde got whacked. I reinstalled kubuntu-desktop, and all SEEMS well, but maybe not. Also, the network problem exists both from the console and from inside kde.

[Snowhog]

If you create a new user and log in using it, does this problem present? If it does not, then something within your other login is creating the problem, and maybe removing the appropriate hidden directories (.local .config ....) and then logging out, rebooting, and logging in with your regular user may resolve the issue.

[doctordruidphd]

I tried that -- same behavior. Also, I tried booting into the recovery console, starting networking, dropping to root shell prompt. Same behavior -- pinging to ip address works, nslookup hangs, as does pinging to a name.

Thing is, this obviously worked during the upgrade from 18.10, so it's something that happened during reboot into 19.04. I boot without graphics, so I see all the messages that flash by, and I don't see any error messages.

[guest007]

Is your router providing DNS on 192.168.1.1 ?
Please cat /etc/resolv.conf and check your nameserver.
Are you using /etc/systemd/network/*.network or /etc/network/interfaces to set your nameserver, or using DHCP?

[doctordruidphd]

Is your router providing DNS on 192.168.1.1 ?
Please cat /etc/resolv.conf and check your nameserver.
Are you using /etc/systemd/network/*.network or /etc/network/interfaces to set your nameserver, or using DHCP?

OK, some progress in finding the problem.

1. Yes, nameserver is through the router at 192.168.1.1.
2. Nothing in /etc/systemd/network/*.network or /etc/network/interfaces, using DHCP.
3. Here is the interesting part, regarding resolv.conf:

When disco boots, and nslookup does not work, here is what /etc/resolv.conf looks like:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

When it starts working, here is what it looks like:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
search myhome.westell.com

Further:

When it is NOT working, here is the result from systemd-resolve --status:

Global
    LLMNR setting: no
MulticastDNS setting: no
DNSOverTLS setting: no
   DNSSEC setting: no
 DNSSEC supported: no
       DNSSEC NTA: 10.in-addr.arpa
                   16.172.in-addr.arpa
                   168.192.in-addr.arpa
                   17.172.in-addr.arpa
                   18.172.in-addr.arpa
                   19.172.in-addr.arpa
                   20.172.in-addr.arpa
                   21.172.in-addr.arpa
                   22.172.in-addr.arpa
                   23.172.in-addr.arpa
                   24.172.in-addr.arpa
                   25.172.in-addr.arpa
                   26.172.in-addr.arpa
                   27.172.in-addr.arpa
                   28.172.in-addr.arpa
                   29.172.in-addr.arpa
                   30.172.in-addr.arpa
                   31.172.in-addr.arpa
                   corp
                   d.f.ip6.arpa
                   home
                   internal
                   intranet
                   lan
                   local
                   private
                   test

Link 2 (eth0)
   Current Scopes: DNS
DefaultRoute setting: yes
    LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
   DNSSEC setting: no
 DNSSEC supported: no
      DNS Servers: 192.168.1.1
       DNS Domain: ~.
                   myhome.westell.com

And when it IS working:

Global
    LLMNR setting: no
MulticastDNS setting: no
DNSOverTLS setting: no
   DNSSEC setting: no
 DNSSEC supported: no
       DNSSEC NTA: 10.in-addr.arpa
                   16.172.in-addr.arpa
                   168.192.in-addr.arpa
                   17.172.in-addr.arpa
                   18.172.in-addr.arpa
                   19.172.in-addr.arpa
                   20.172.in-addr.arpa
                   21.172.in-addr.arpa
                   22.172.in-addr.arpa
                   23.172.in-addr.arpa
                   24.172.in-addr.arpa
                   25.172.in-addr.arpa
                   26.172.in-addr.arpa
                   27.172.in-addr.arpa
                   28.172.in-addr.arpa
                   29.172.in-addr.arpa
                   30.172.in-addr.arpa
                   31.172.in-addr.arpa
                   corp
                   d.f.ip6.arpa
                   home
                   internal
                   intranet
                   lan
                   local
                   private
                   test

Link 2 (eth0)
   Current Scopes: DNS
DefaultRoute setting: yes
    LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
   DNSSEC setting: no
 DNSSEC supported: no
Current DNS Server: 192.168.1.1
      DNS Servers: 192.168.1.1
       DNS Domain: ~.
                   myhome.westell.com

The only difference between the two being the line "Current DNS Server: 192.168.1.1".

So something is going on with how resolv.conf is being generated.

[doctordruidphd]

I found a temporary workaround, by no means a "fix".
I deleted the symbolic link from /etc/resolv.conf that points to /run/resolvconf/resolv.conf and copied /run/resolvconf/resolv.conf from Cosmic to /etc/resolv.conf in Disco. That got the system to work, but when I checked, I found that my hard copied /etc/resolv.conf had been replaced by the following:

greenman@Crynfyd19.04 ~$ cat /etc/resolv.conf
# Generated by NetworkManager
search myhome.westell.com
nameserver 127.0.0.53

As I said this is by no means a fix, something is still wrong. In fact there are quite a few things wrong with this "upgrade" -- programs hogging way more memory and executing much more slowly than on Cosmic, and there is no way it should have removed plasma during the upgrade if it went properly. It didn't go properly -- do-release-upgrade quit after having upgraded only a small fraction of the system, and the rest had to be done with apt-get dist-upgrade, apt-get install -f, dpkg --reconfigure -a, and so on to finish it. The whole thing is a mess and I may have to do it over some time in the future.

But I still would like to figure out what it is that actually went wrong.

[doctordruidphd]

I have tried upgrading yet again, and this time the upgrade itself went through smoothly. However, still no network connectivity (more correctly, no dns lookup -- ping by ip number does work). Same problem -- resolv.conf is not being set. It's desperation time as I can't come up with any ideas as to what is going wrong. Thanks for any ideas.

[doctordruidphd]

Solved. Evidently the problem was caused by the presence of the resolvconf package, which should have been removed during an earlier upgrade. It wasn't, and removing it after the upgrade didn't work either. It needed to be removed before the upgrade.