Mail problems

[SteveRiley]

That seems to be a common complaint (certain pieces of software being not-quite-standards-compliant, other software changes to work with it until none of it is actually following the standard). One of the comments in that article was about Google doing that with Jingle. Why do you think that happens so often? If you're the big player and you were planning on doing things differently, why not influence the standards committee to write your methods into the standard? And if it's your own standard, then there really is no excuse for not complying with it!

Standards create level playing fields and give consumers choice. If you're a profit seeking company, you don't like that -- you want to monopolize as much of your market as you can. If Google used only standard XMPP, then people could connect with whatever client they wish. The result is overall loss of control for Google and certainly an inability to maximize ad placement. So instead they bastardize the standards just enough to make using a non-Google client very painful. So painful for most people that they'll instead capitulate and use Google's client. Now Google is in control, they can track your activities better, and shove more ads at you.

So... the consensus seems to be that everything is a complete mess. What do you think is the way forward? Stick with XMPP or something new? Setting up an XMPP server for real is on my to-do list, but if it's on the way out it's a bit of a waste of time. I can't help but think XMPP has enough inertia to live on, though... there are so many nice clients out there on every platform.

The tale of standards-based presence and instant messaging is a rather sad one. In 1999 an IETF working group was chartered to begin defining the standards. The results were two informational RFCs: 2778, "A model for presence and instant messaging" and 2779, "Instant messaging / presence protocol requirements". Incidentally, both were created in part by my former manager at Riverbed. IMPP concerned itself with defining the architecture and gateway semantics for connecting messaging and presence systems together. Some individuals in the WG wanted to go further and define transport protocols, too; various factions couldn't come to an agreement and IMPP kind of imploded. Acting outside the IMPP WG, two other standards-based transport protocols emerged: SIMPLE and XMPP are implementations of IMPP concepts.

Alas, SIMPLE (given its reliance on SIP) and XMPP suffer from two problems: (1) they're late to the game, and (2) they're bloated.

The only reason we have standards-based email (SMTP) today is because it emerged long before the commercialization of the Internet. Interoperability was paramount -- not walled gardens. Instant messaging's roots are in consumer grade online networks -- think AOL Messenger, MSN Messenger; there's also ICQ, which had its own protocol. The incentive here is not to interoperate but to capture as many users as possible. Enterprise IT was late in adding presence and messaging to its stable of services; if presence and messaging had emerged as a business function in the late 1990s, I suspect we'd see a very different landscape today. Enterprises, after all, need interoperability over anything else.

SIP/SIMPLE and XMPP are monstrous and unwieldy. As you've seen, there's just a lot of stuff in SIP and its hangers on. Sure, real time audio and video are harder than text. But not that much. There's been a lot of bickering in the standards groups; protocol-by-committee isn't pretty. XMPP's albatross is XML. Anything XML touches turns to dust. And dust, unfortunately, is probably where XMPP is headed. Google is nudging people away; Facebook will drop it later this year.

Good reads:
http://jeff.ecchi.ca/blog/2013/12/20...aging-in-2013/
https://news.ycombinator.com/item?id=7151857
https://bloggeek.me/death-signaling/

Oh, and jabber.org's X.509 certificate expired today: https://xmpp.net/result.php?domain=j...rg&type=server

[DoYouKubuntu]

@DYK, I'm sorry for threadjacking...

No, no, no! I LOVE thread drift! It's what makes talking online more like real life, you know?

it looks like you have your answer now

Yep, that's why it dawned on me that I should post an update. Since changing a few of the outgoing server's settings, that 'sending failure issue' hasn't happened again...at all! Coincidence? I don't know. At any rate, here are the outgoing server's settings now:

Server name: smtpauth.earthlink.net
Connection security: none (as above)
Authentication method: Encrypted password
Username: my_earthlink_address@earthlink.net
Port: 25 Default: 587

Thanks again, all around, for the great input and insight.

[Feathers McGraw]

I'm not aware of any SIP server that uses DTLS, so I'm not surprised you didn't see it as an option.

But you do realize that SIP is only used for call setup and signaling, right? The actual audio and video is carried in RTP, which almost always uses UDP. RTP is the bulk of the call; the SIP signaling is minimal. I can't think of any efficiency gains one might realize by putting SIP in UDP.

That makes sense, I don't think I had put two and two together.

Very minimal. My experience is limited to the continual frustration I encounter trying to get $RANDOM_SOFT_PHONE working with my employer's Avaya phone switch. I'm convinced its SIP implementation is from the devil; various clients exhibit weird, unexplained behavior. The only one that worked consistently well was that ancient Twinkle -- probably because it cared little for "proper" protocol behavior and brute-forced every call setup. Alas, it no longer runs on modern Qt and is unmaintained.

That seems to be a common complaint (certain pieces of software being not-quite-standards-compliant, other software changes to work with it until none of it is actually following the standard). One of the comments in that article was about Google doing that with Jingle. Why do you think that happens so often? If you're the big player and you were planning on doing things differently, why not influence the standards committee to write your methods into the standard? And if it's your own standard, then there really is no excuse for not complying with it!

LWN's recent writeup about Matrix is illuminating. Particularly the comments.

That's a great article. Pretty sharp comments too like you said, they certainly aren't pulling punches.

So... the consensus seems to be that everything is a complete mess. What do you think is the way forward? Stick with XMPP or something new? Setting up an XMPP server for real is on my to-do list, but if it's on the way out it's a bit of a waste of time. I can't help but think XMPP has enough inertia to live on, though... there are so many nice clients out there on every platform.

[SteveRiley]

Neat! I'll have to remember that, I don't remember seeing it as an option when I was looking into SIP. There's so much to learn, I was a bit overwhelmed by the plethora of options available for SIP at the time to be honest - I got as far as "you have to choose which method to use: TCP or UDP" and looking up the differences between the two. I didn't want to send the PW in plaintext without TLS, so that left me with TCP (or so I thought).

I'm not aware of any SIP server that uses DTLS, so I'm not surprised you didn't see it as an option.

But you do realize that SIP is only used for call setup and signaling, right? The actual audio and video is carried in RTP, which almost always uses UDP. RTP is the bulk of the call; the SIP signaling is minimal. I can't think of any efficiency gains one might realize by putting SIP in UDP.

Do you have any experience with videocalling with SIP? If so, I'm curious to see which client and server programs you used, and why.

Very minimal. My experience is limited to the continual frustration I encounter trying to get $RANDOM_SOFT_PHONE working with my employer's Avaya phone switch. I'm convinced its SIP implementation is from the devil; various clients exhibit weird, unexplained behavior. The only one that worked consistently well was that ancient Twinkle -- probably because it cared little for "proper" protocol behavior and brute-forced every call setup. Alas, it no longer runs on modern Qt and is unmaintained.

It's a real shame that Google abandoned XMPP/Google Talk and switched to a proprietary protocol with Hangouts. Federated XMPP is something I think is pretty cool for the same reasons that federated SMTP is great.

LWN's recent writeup about Matrix is illuminating. Particularly the comments.

[Feathers McGraw]

@DYK, I'm sorry for threadjacking... it looks like you have your answer now but if not I'm sure we can move this discussion to another thread

@Steve...
Neat! I'll have to remember that, I don't remember seeing it as an option when I was looking into SIP. There's so much to learn, I was a bit overwhelmed by the plethora of options available for SIP at the time to be honest - I got as far as "you have to choose which method to use: TCP or UDP" and looking up the differences between the two. I didn't want to send the PW in plaintext without TLS, so that left me with TCP (or so I thought). Maybe I should have another go - I got quite far last time...

What I found with SIP is that it's easy to get diverted by guides that are setting up systems with some kind of video transcoding, whereas what I really wanted was a server-side program that could just route everything for me and let the clients decide on an audio/video format they could both use. GNU SIP Witch seems to be the most suitable for this, and I managed to set up a one-way video two-way audio session between two Android devices running CSipSimple... I couldn't get two-way video, which was a shame.

I see that routing everything through a central server isn't ideal for big installations because of the server load, but for a small home server that won't have more than a couple of calls at any one time I don't see the problem and it cuts out a whole world of pain with STUN. I actually looked into how much it would cost me to have 2 IP addresses so that I could run a STUN server - I think it would have taught me lots about NAT. Turns out you can only do that if you're a business, and it would be more than double what I pay now, so no dice.

Do you have any experience with videocalling with SIP? If so, I'm curious to see which client and server programs you used, and why.

XMPP was quite simple by comparison. Ejabberd seems to be the only free option that is actively developed and supports a good number of extensions, even if it does feel a bit odd due to erlang. One thing that bothered me was not being able to use the current stable version of fail2ban (0.8) to check failed password attempts (entries in the log file are split over multiple lines!). The new version of fail2ban (0.9) can do multiline regex, so I'm hoping it will make its way into the ubuntu repos soon.

It's a real shame that Google abandoned XMPP/Google Talk and switched to a proprietary protocol with Hangouts. Federated XMPP is something I think is pretty cool for the same reasons that federated SMTP is great.

[SteveRiley]

over UDP but of course you can't use TLS which is a bummer.

Sure you can: Datagram Transport Layer Security, RFC 6347.

[Feathers McGraw]

BTW, proper port number specifications follow the format number/transport. transport is either tcp or udp depending on the protocol. That's why I almost always use this format, even though in configuration dialogs you typically see only the numbers in isolation.

I've noticed you always do that. I didn't appreciate the reason why until I started experimenting with SIP and XMPP (basically trying to replace Hangouts with something free and federated) - some of those things can be configured to use TCP or UDP on the same port, which is something I hadn't come across before. SIP in particular seems to be more workable over UDP but of course you can't use TLS which is a bummer.

[SteveRiley]

You'll definitely want to use smtpauth.earthlink.net for outgoing mail:

steve@t520:~$ [B]telnet smtpauth.earthlink.net 587[/B]
Trying 207.69.189.208...
Connected to smtpauth.earthlink.net.
Escape character is '^]'.
220-elasmtp-kukur.atl.sa.earthlink.net ESMTP Exim 4.67 #1 Thu, 19 Feb 2015 17:02:19 -0500
220-NO UCE.  EarthLink does not authorize the use of its computers or network
220 equipment to accept, transmit, or distribute unsolicited e-mail.
[B]ehlo example.com[/B]
250-elasmtp-kukur.atl.sa.earthlink.net Hello example.com [63.224.42.106]
250-SIZE 29360128
250-PIPELINING
250-AUTH PLAIN LOGIN CRAM-MD5
250-STARTTLS
250 HELP
[B]^][/B]
telnet> [B]quit[/B]
Connection closed.

On port 587/tcp, using STARTTLS, with PLAIN authentication. This will protect your password.

BTW, proper port number specifications follow the format number/transport. transport is either tcp or udp depending on the protocol. That's why I almost always use this format, even though in configuration dialogs you typically see only the numbers in isolation.

[DoYouKubuntu]

Let me just address this for right now:

For the sake of correctness and security, change your outgoing server's port to 25/tcp, using STARTTLS connection security. I can't seem to force the server to tell me which authentication method it wants, so you'll need to experiment here.

I can, and did just now, change the outgoing server's port to 25, but I have no TCP choice; I set connection security to STARTTLS and then attempted to send mail. Nope:

Sending of message failed.
An error occurred sending mail: Unable to establish a secure link with SMTP server smtpauth.earthlink.net using STARTTLS since it doesn't advertise that feature. Switch off STARTTLS for that server or contact your service provider.

Leaving its port set at 25 but changing its connection security back to none, sending mail succeeds.

I changed "Authentication method" to "Encrypted password," left everything else as above, and sending mail succeeded. PLEASE NOTE that years ago I set SM's settings per Earthlink's instructions at that time, and using "Encrypted password" did not work, hence the reason I'd chosen "Password, transmitted insecurely." Really.

[Feathers McGraw]

Ah dude you've told me that before! Stupid sleepy post! Apologies.

So i guess it is possible they changed something before you got your turbo internet connection. I'm probably just misremembering though!

[DoYouKubuntu]

Here's an interesting update.

I changed the [outgoing server] port from 1 to 587--and could no longer send mail! It didn't do the failure thing, it just sat FOREVER trying to connect to Earthlink. So, fine, I said to myself, I'll just change it back to 1. And I did. AND IT HUNG, too! WTF?! I could no longer send mail from any account, not just the problem child Earthlink address. I logged in to Earthlink's webmail and sent a message, one copy to the problem Earthlink address and one to one of my domains, then attempted to retrieve them via SM as usual--they arrived, no problem, it was just OUTGOING mail that wouldn't work.

I knew I had KMail installed but had never used it [on this computer], so I went to it and set up its settings; it gave me the option to let it find the correct settings online, and I let it. As soon as it was done I attempted to send mail--and it worked. Then I looked at its settings and they're different from SM's:

Outgoing mail server: smtpauth.earthlink.net
[x] Server requires authentication
login: my_earthlink_address@earthlink.net
password: ************ (my Earthlink password)
[x] Store SMTP password

Encryption: [ ] None [ ] SSL [x] TLS
Port: 587
Authentication: CRAM-MD5

So back to SM I go, and I change its outgoing server from mail.earthlink.net to smtpauth.earthlink.net -- which, by the way, I remember from MANY years ago--that's what Earthlink used to instruct its users to enter as the outgoing server; mail.earthlink.net was a newer instruction

I changed its port to 587, didn't change anything else, and attempted to send/receive mail, both of which worked. I've since sent numerous test messages via SM and haven't seen the sending failure issue yet.

What does it all mean?!

ETA: I just noticed Steve's newest reply... I'll be back.

[SteveRiley]

Wow, Earthlink forces you to send passwords in clear text! mail.earthlink.net doesn't take connections on the POP3 secure port (995/tcp):

steve@t520:~$ [B]telnet mail.earthlink.net 995[/B]
Trying 209.86.93.206...
[B]^C[/B]

And TLS (via STARTTLS) is not shown as a capability on the regular port (110/tcp):

steve@t520:~$ [b]telnet mail.earthlink.net 110[/B]
Trying 209.86.93.208...
Connected to mail.earthlink.net.
Escape character is '^]'.
+OK NGPopper vEL_0_1_39_P at earthlink.net ready <13711.1424378387@mp-venomous.atl.sa.earthlink.net>
[B]CAPABILITY[/B]
+OK
apop
pass
top
uidl
user
.
[B]^][/B]
telnet> [B]quit[/B]
Connection closed.

Trying their IMAP server -- imap.earthlink.net -- shows that it isn't listening on the IMAP secure port (993/tcp), either:

steve@t520:~$ [B]telnet imap.earthlink.net 993[/B]
Trying 209.86.93.191...
[B]^C[/B]

But it does allow switching to TLS on the regular port (143/tcp):

steve@t520:~$ [B]telnet imap.earthlink.net 143[/B]
Trying 209.86.93.191...
Connected to imap.earthlink.net.
Escape character is '^]'.
* OK earthlink.net IMAP Service 8684 imapd EL_0_1_41_P at oim-genesis.atl.sa.earthlink.net ready
[B]. CAPABILITY[/B]
* CAPABILITY IMAP4 IMAP4rev1 QUOTA LITERAL+ UIDPLUS NO_ATOMIC_RENAME UNSELECT SORT X-Move X-Count THREAD=ORDEREDSUBJECT THREAD=REFERENCES AUTH=CRAM-MD5 AUTH=EL-TAC AUTH=EL-VOICE STARTTLS ID CHILDREN
. OK Completed
[B]^][/B]
telnet> [B]quit[/B]
Connection closed.

So, to stop leaking your password in clear text, you should change your incoming server to imap.earthlink.net, the port to 143, using STARTTLS connection security, with CRAM-MD5 authentication.

---

Now for sending email, mail.earthlink.net is not listening on the submission port (587/tcp), commonly used by clients for relaying messages to ISPs:

steve@t520:~$ [B]telnet mail.earthlink.net 587[/B]
Trying 209.86.93.202...
[B]^C[/B]

Nor is it listening on the (now deprecated) port for SMTP/SSL (465/tcp):

steve@t520:~$ [B]telnet mail.earthlink.net 465[/B]
Trying 209.86.93.211...
[B]^C[/B]

However, this server does respond to STARTTLS commands on the regular SMTP port (25/tcp):

steve@t520:~$ [B]telnet mail.earthlink.net 25[/B]
Trying 209.86.93.203...
Connected to mail.earthlink.net.
Escape character is '^]'.
220-pop-satin.atl.sa.earthlink.net ESMTP Exim 3.36 #1 Thu, 19 Feb 2015 15:46:30 -0500
220-NO UCE.  EarthLink does not authorize the use of its computers or network
220 equipment to deliver, accept, transmit, or distribute unsolicited e-mail.
[B]STARTTLS[/B]
220 OpenSSL/0.9.7beta go ahead
[B]^][/B]
telnet> [B]quit[/B]
Connection closed.

Oh, and I suspect your mail client is reverting to the standard port (and still sending your password in clear text) becuase port 1/tcp is definitely not being used:

steve@t520:~$ [B]telnet mail.earthlink.net 1[/B]
Trying 209.86.93.211...
[B]^C[/B]

For the sake of correctness and security, change your outgoing server's port to 25/tcp, using STARTTLS connection security. I can't seem to force the server to tell me which authentication method it wants, so you'll need to experiment here.

[DoYouKubuntu]

Thanks, guys, for the efforts and insight. Special thanks to Steve.

Just for the record, the e-mail I sent Steve DIDN'T do the failure thing, i.e., it successfully sent on its first try. There is no rhyme or reason to the failing, as far as I can tell. It's so random...one fails, then three don't, then two do, and ten don't... And nothing changes as far as settings or anything. So I don't know.

Here's some additional info about my server settings (within SeaMonkey's mail client); starting with the settings for the Earthlink address I have problems with:

Server type: POP Mail Server
Server name: mail.earthlink.net
Port: 110 Default: 110
Username: my_earthlink_address@earthlink.net
Connection security: None (other choices are STARTTLS and SSL/TLS -- if I choose either of those, it won't work)
Authentication method: Password, transmitted insecurely (other choices: Encrypted password, Kerberos / GSSAPI, NTLM, TLS certificate -- I've only tried encrypted password, and it failed)

Now for the "Outgoing server (SMTP) settings":

Server name: mail.earthlink.net
Connection security: none (as above)
Authentication method: Password, transmitted insecurely (as above)
Username: my_earthlink_address@earthlink.net
Port: 1 Default: 587

I don't think I'd ever noticed, or registered, the port number for the outgoing server. Offhand I don't know why I have it set to 1, but have to assume I set it that way per Earthlink's instructions at some time way in the past. Whenever I get another computer I copy an existing SeaMonkey over to its hard drive. So chances are pretty good that it's been set to 1 for a very long time! Yes, I'm going to change it to 587 and see what happens!

[SteveRiley]

This testing includes years ago I take it?

Nope, was recent for me.

When I was on Comcast, I had to use Dyn for mail transport. I relayed all outbound mail through them because Comcast blocks direct connections to any-ip:25/tcp from their network. Dyn's PTR stuff is (of course) set up correctly, and Gmail would accept inbound mail from me. When I switched to CenturyLink and got a static IP, that allows me to make direct connections to SMTP servers. I noticed then that GMail started to block inbound mail from me. My first attempt to solve this problem was to set up SPF. And that worked.

I probably could have instead just got the PTR set up correctly, and thus mirrored what Dyn was doing -- that is, correct PTR. But I wanted to tackle SPF. I also set up correct PTR, but that required a call to CenturyLink's technical support since they own the IP address range.

[Feathers McGraw]

This testing includes years ago I take it?

I'm sure PTR and no SPF is fine because that was my setup until a few months ago and I haven't had any problems.

Must be a coincidence then... or maybe I get fewer questions about other things now because I've clarified most of the bits where people used to go wrong!