Note about OAuth2

This page is obsolete after the "upgrade" of Office365 to OAuth2 authentication: see the current version instead.

This page retained for history only, about the initial transition to Office365.
See also the even older version about the transition to the Uni Exchange server.

Re-Introducing davmail

for Uni outsourced Office365

In June2018 our emails were migrated from the USyd internal Exchange server, to the cloud-based, outsourced Office365 service. Though Office365 supports IMAP/SMTP "natively", those are woefully slow and "clunky". Using davmail to convert IMAP/SMTP to EWS (Exchange Web Services) to that Office365 server, allows emails to be processed much faster, despite the intervening conversions and extra network traffic.

You want davmail if you use IMAP (e.g. Thunderbird) to access your email. You do not need to know about it if you use Outlook or AppleMail, or the web interface, or most mobile phone apps.

Native Office365 settings
Our davmail server
Setup instructions for Thunderbird, Apple Mail, Gmail web interface (and oddities), Mutt, Alpine, other mail clients
Check your setup
Notes, blurb

Native Office365 settings

Emails on Office365 can be accessed via its web interface, at either:
sydney.edu.au/office365
sydney.edu.au/email
http://outlook.com/owa/unisyd.onmicrosoft.com or even
outlook.office365.com
or by using Outlook or MacMail or many mobile phone apps (as Exchange mail).

For other email clients, Office365 supports IMAP/POP and SMTP, as per
POP and IMAP settings for Office 365 or
Uni sharepoint O365 FAQ
so using settings:

proto host port security

IMAP outlook.office365.com 993 SSL/TLS

POP outlook.office365.com 995 SSL/TLS

SMTP smtp.office365.com 587 STARTTLS

Username your @sydney email address

Password your unikey password

(or you could set forwarding).

Access to the Online Archive is possible with the web interface or Outlook, but not most other clients and not via IMAP, as per Microsoft documentation.

The IMAP/SMTP response of Office365 is woefully slow. Maybe this is as mentioned in Microsoft documentation:

  Note
  Each time a person accesses a POP-based or IMAP-based email program
  to open his or her Microsoft 365 or Office 365 email, that user will
  experience a delay of several seconds. The delay results from using a
  proxy server ...

... or may be just a sneaky way for Microsoft to promote Outlook and discourage other email clients.

Our davmail server

Our davmail server has host name

davmail.maths.usyd.edu.au

and it supports/accepts:

POP (pop3s) on port 995

IMAP (imaps) on port 993

SMTP (smtps) on either port 465 or port 587

with SSL/TLS encryption and "normal password" authentication, with the @sydney email address as username (not unikey), and unikey password. Davmail can be used for any "IMAP" services e.g. Thunderbird or Apple Mail, or for the gmail web interface, from anywhere.

Our davmail server could also support/accept:

LDAP (ldaps) on port 636

CalDAV on port 1080

both with SSL/TLS encryption, but it does not: LDAP not because it would be blocked by the ICT border router on some bogus security grounds, and CalDAV not because it does not seem needed or wanted.

Our davmail server uses http://davmail.sourceforge.net/ software. The server accepts POP/IMAP/SMTP connections, and "translates" the requests into EWS (Exchange Web Services) access: provides standard interfaces, using only supported EWS access to Office365 mail. Our server talks to the Office365 Exchange server at outlook.office365.com/EWS/Exchange.asmx, as set within its configuration; that choice is not part of the "conversation" with the client; it cannot be used to access any other Exchange servers: to access another, a different davmail service would need to be set up. It could be used "as is" for any other Office365 clients or login schemes, e.g. it should work for "student" email @uni.sydney.edu.au accounts.

Our davmail server runs on a "virtual machine" using just some idle CPU cycles, for zero cost. This service might be used by the whole Uni community (or even worldwide?), not just Maths. It would not be able to handle the network bandwidth if it became popular. Laptop users might instead run davmail themselves, locally.

Davmail SMTP effectively sends via EWS, and that does not keep an original "Date:" header, but replaces it with UTC timezone and at the time the message is handled by Office365. Some other SMTP headers are also added or deleted. Send yourself a message, then look at the headers in the Office365 Sent and Inbox folders, and weep.

The Office365 Online Archive can be accessed via davmail, at least with Thunderbird.

Curiously and amazingly, davmail is faster than Office365 IMAP or SMTP, e.g. "send" is in the blink of an eye, no 10-second wait.

Setup instructions

Thunderbird

In your Thunderbird go to

Edit or Settings/Preferences

Account Settings

Account Actions

Add Mail account

and there set:

Name: your name

Email address: your @sydney email address

Password: (none), un-check do not "remember password"

Incoming: IMAP, davmail.maths.usyd.edu.au, 993, SSL/TLS, normal password

Outgoing: SMTP, davmail.maths.usyd.edu.au, 465, SSL/TLS, normal password

Username: (both Incoming and Outgoing): your @sydney email address

then Re-Test, Done. (When asked for a password, use the matching unikey password.)

To avoid duplicates in "Sent", still in

Edit or Settings/Preferences

Account Settings

your new @sydney account

Copies & Folders

Un-check (not select) the setting:

When sending messages, automatically:

[ ] Place a copy in ... "Sent" Folder on ...

(since Office365 or davmail does pretty much the same anyway).

Click OK.

To access the Online Archive follow the instructions (or my rip-off):

Edit or Settings/Preferences

Account Settings

your new @sydney account

Server Settings

Advanced

Set

Personal namespace: (blank)

Public (shared): /archive

and maybe un-check (not select) the setting:

[ ] Show only subscribed folders

Click OK, then re-start Thunderbird.

Go to Check your setup.

Apple Mail

You do not need (cannot use?) IMAP or davmail... so just for the record.
Set things up as an Exchange account:

In your Apple Mail go to

Preferences

Accounts

Add (the "+" sign under the list)

and there set:

Full Name: your name

Email address: your @sydney email address

Password: your matching unikey password

Continue, let it check, then Create. Go to Check your setup.

Gmail web interface

You may (instead?) set redirect forwarding from Office365 to gmail, see set forwarding as mentioned above.

On the gmail web interface, go to

Settings

Accounts and Import

Check mail from other accounts / Add a mail account

and add your "central" mail account via our davmail server:

your @sydney.edu.au email address

(choose Import emails ... POP3)

Username: your @sydney email address, use matching unikey password

change POP server to davmail.maths.usyd.edu.au on port 995

select "leave copy on server" (so the Uni keeps backing up your mail)

select "always use SSL" (leave selected).

Say "yes" to send mail as this new account, or in

Settings

Accounts and Import

Send mail as / Add another email address

un-select "treat as alias", then set:

SMTP server davmail.maths.usyd.edu.au on port 587

Username: your @sydney email address, use matching unikey password

select to use SSL (not TLS, not sure why TLS does not work)

then wait for the verification code to arrive in your email, add it.
Maybe also choose "Reply from the same address the message was sent to".

This setting "gives away" your unikey password to your email service. Not an issue if you trust them. (Probably your laptop and phone also "remember" this password, anyway.)

Two or three Gmail oddities to BEWARE of.

Gmail does not like duplicates, and is "too smart" for its own good. When it sees a message with the same Message-ID it drops/ignores the second-arrived one as a duplicate: it will not show messages it has already seen (sent or forwarded). Suppose you send a message from gmail to some mailing list to which you are subscribed, or try to check the forwarding e.g. from @sydney to your gmail address, then your own (or test) message never "arrives": because a message with that Message-ID is already in the Sent folder. (When sending from gmail to yourself, the copy in Sent and Inbox is in fact the one and same, deleting one deletes the other also.)
To Check your setup below, on Office365 send a message to some new or non-existent user, copy that Sent or Draft into your Inbox.

The above allows you to set gmail to "send as" your @sydney address. You cannot set gmail to "send as" any other @sydney or @maths address, because gmail refuses to "forge" such emails: it wants to send those in a "legit" way. Gmail wants to verify that you "own" the sender address you are about to set, wants connection info (SMTP server, maybe username/password) that will allow sending such emails; it also will send a verification email. Just as you cannot (normally, from within USyd) send emails with "random" @sydney or @maths addresses, gmail refuses to "do the forgery" for you.
This makes it hard to set gmail to "send as" for most users, even when using their "true" addresses. In the past, gmail would just send a confirmation email, and if you replied then set that as "sender" address. Seems gmail noticed this was unsafe, e.g. as it allowed anyone to send as a mailing list they were subscribed to.

Gmail (correctly!) uses DMARC, and drops messages with broken DKIM signatures. Because the Mimecast spam filter wrecks DKIM signatures on @maths and @sydney messages, gmail will not show (most) messages e.g. from PayPal or eBay, sent via @maths or @sydney.

Go to Check your setup.

Mutt

Seems that in your ~/.mutt/muttrc file, you need to add lines like
(example for Paul Szabo, address paul.szabo@sydney.edu.au):

# IMAP settings

set imap_user = "paul.szabo@sydney.edu.au"

set spoolfile = imaps://davmail.maths.usyd.edu.au:993/INBOX

set folder = "imaps://davmail.maths.usyd.edu.au:993/"

set imap_keepalive = 30

# SMTP settings

set smtp_url = "smtps://$imap_user@davmail.maths.usyd.edu.au:465"

set ssl_force_tls = yes

# davmail only accepts from the "right" sender

set realname = "Paul Szabo"

set from = "paul.szabo@sydney.edu.au"

set use_from = yes

set use_envelope_from = yes

# other settings

set header_cache = ~/.mutt/cache/headers

set message_cachedir = ~/.mutt/cache/bodies

set certificate_file = ~/.mutt/certificates

# keep cache clean

set message_cache_clean = yes

Go to Check your setup.

Alpine

Seems that in your ~/.pinerc file, you need to add lines like
(example for Paul Szabo, address paul.szabo@sydney.edu.au):

inbox-path={davmail.maths.usyd.edu.au:993/ssl/novalidate-cert/user=paul.szabo@sydney.edu.au}inbox

smtp-server=davmail.maths.usyd.edu.au:587/ssl/novalidate-cert/user=paul.szabo@sydney.edu.au

customized-hdrs=From: Paul Szabo <paul.szabo@sydney.edu.au>

disable-these-authenticators=PLAIN

In the alpine SETUP Config menu, you need to enable Expose Hidden Config (then exit and re-enter config) to set Disable These Authenticators.

One problem may(?) remain: a message sent by alpine then shown by it, may say:

[ The following text contains the unknown encoding type ]

[ "X-UNKNOWN". ]

[ Some or all of the text may be displayed incorrectly. ]

I do not know what causes this.

Go to Check your setup.

Other mail clients

Seems the gmail app on phones can use your Exchange account to be added, more directly (or it could use IMAP). That alone would be enough if you only ever used that gmail app; not sure whether necessary (or would cause duplicates) once you have set gmail via the web interface; I did not yet test the phone app.

Other mail services may have "add account" features (similar to gmail). Succeeded on mail.com (its "mail collector" using IMAP to webmail.sydney on port 993, it could also send email as if it was from @sydney, no davmail at all).

Any other clients or any problems, please ask Paul.

Check your setup

After setting up your email client, check that email reception works: log in to Office365, copy some message into your Inbox, see it appear in your mail client.

Notes, blurb

See also older version about the transition to the Uni Exchange server.

Some of the blurb below is non-original, been (wrongly!) updated since the change to OAuth2.

Do not store old, long-term, or important messages on Office365, but keep in "local" folders.
BEWARE that when you leave the Uni, ICT will disable your unikey and you will lose access to Uni email.
BEWARE of Office365 Online Archive settings: they move messages older than some time into some "Online Archive". You can access this with Outlook or web interface, or via davmail and with Thunderbird (but maybe not other clients?), and not with IMAP as per Microsoft documentation, and not with Apple Mail. Maybe, change the archiving policy using the web interface: right-click on (each) email folder and choose Assign Policy > Achive Policy : Personal never move to archive (Never).
BEWARE of the Outlook recall function: messages recalled and still in your Inbox (or other Office365 folders?), will disappear.
BEWARE that ICT will sometimes delete some (bad? virus?) messages from your mail folders.

BEWARE of unikey password changes. Currently there is an enforced yearly change, and if you change then you may need to re-do the settings in your mail client (gmail or thunderbird or phone etc). (Or if you forget, then you may end up with your account locked after too many bad tries.) Best to leave your unikey password as it was: go through 5 or 10 changes, then back.

Note how "student" email on @uni.sydney.edu.au is outsourced to the same Office365 cloud, though with a different login scheme.

Note that with IMAP you can copy messages (in either direction) between Office365 and other folders: try to take advantage of the unlimited storage offered by Office365.

The Uni wants to store data only on servers under trusted jurisdictions, and gmail/google has servers in some Asian countries. The Uni trusts Microsoft (both @sydney and @uni.sydney are really Office365), Mimecast (our spam filter), trusted Symantec (previous spam filter), and say Cloudstor and Dropbox; so far the Uni does not seem to worry about Google Drive. There is a push to have mobile devices (their data, and the passwords they remember) encrypted but that does not seem monitored or enforced.

BEWARE of the Uni Mimecast spam filter, noting that all @maths and @sydney messages received, and any sent by Office365, go through it.

The Mimecast filter sometimes alters the encoding of messages: dislikes "Content-Transfer-Encoding: 7bit" or 8bit, prefers quoted-printable, may leave base64; specifically for "Content-Type: multipart/xxx" messages, does not put in any (useless?) Content-Transfer-Encoding lines into the email header; and corrects the capitalization of those headers. Worse: with the "URL rewrite" feature it changes URL links on purpose, rewriting the content. These actions wreck any DKIM signatures, and may result in messages being rejected later on (by clients that use DMARC correctly).
The Mimecast filter "URL rewrite" feature checks all URL links when you click on them, in effect clicking just before you do. This wrecks single-use links, as often used e.g. for "password reset" functions: you receive "link expired" by the time you click.

Apologies for the verbiage.

Paul Szabo psz@maths.usyd.edu.au 17 Nov 23

proto	host	port	security
IMAP	outlook.office365.com	993	SSL/TLS
POP	outlook.office365.com	995	SSL/TLS
SMTP	smtp.office365.com	587	STARTTLS

	Username		your @sydney email address
	Password		your unikey password

Note about OAuth2

Re-Introducing davmail

for Uni outsourced Office365

Contents

Native Office365 settings

Our davmail server

Setup instructions

Thunderbird

Apple Mail

Gmail web interface

Mutt

Alpine

Other mail clients

Check your setup

Notes, blurb