What's the best / most popular open-source IMAP server these days?

On Thu, 3 Jul 2008, Mark Crispin wrote:

I forked, and created Panda imapd (imap-2008). There is limited
ongoing work for my own purposes. For example, I discovered that UW
imapd is not fully compliant with RFC 5051, and that is fixed in Panda
imapd.

Bugs in UW-IMAP should be reported to alpine-contact@washington.edu, so
that they can be fixed in the next release of UW-IMAP and Alpine, which
will help a lot of users out there.

--
Eduardo
http://staff.washington.edu/chappa/alpine/

P.S. it's nice to see you back, and regrettably I think you are right about
everything you said in this thread.

On Thu, 03 Jul 2008 07:22:49 -0700, Mark Crispin wrote:

> Another open-source IMAP server that supports mbox format is Dovecot.
> The developer of Dovecot and I have been engaged in a friendly
> competition for a while now, in which he develops some new torture test
> that breaks all IMAP servers, and then he and I scramble to see if it
> gets fixed in Dovecot or UW/Panda first. This has had the effect of
> making Dovecot and UW/Panda be quite a bit more compliant and robust,
> particularly with high-intensity multiple access, than other servers.
>
> I see that he currently has an one-up on me, so it's my turn to play
> catch-up. I'll have to ask him what he found.

First was about the i;unicode-casemap. I don't remember if I've mentioned
the second though. It seems to be about SORTing addresses with groups.
The test is:

Cc: user2@domain2.org (foo bar)
Cc: user2: blah@domain1.org;
Cc: user1@domain1.org

CC
[IMAP] addr-mailbox of the first "cc" address.

In the second Cc line's first addr-mailbox is "user2", but UW-IMAP thinks
it's something else. Unless the group name isn't actually thought of as
the first address?

> However, note that this is not with mbox format; this is with the
> preferred mail stores (extended maildir on Dovecot, mix in UW/Panda).
> mbox format has legacy issues that keep it from working optimally for
> IMAP, no matter which server you use.

Dovecot with index files passes all imaptest tests with both mbox and
maildir formats. Without index files both fail some tests because changes
aren't noticed as easily since mtimes don't change. I think the tests
would pass if used on a filesystem supporting nanosecond (or probably
even microsecond) resolution. I guess I should try some day.

On Fri, 11 Jul 2008, Timo Sirainen posted:
> First was about the i;unicode-casemap.

Thank you. That is fixed in Panda IMAP.

> I don't remember if I've mentioned
> the second though. It seems to be about SORTing addresses with groups.

Thank you. I found the problem; it only shows up if you haven't first
done an RFC 822 parse of the message. Equivalent issues exist with To
and From (although you aren't supposed to have groups with From).

The problem is fixed in Panda IMAP as of now.

Please update your list to add "Panda IMAP 2008, mix format", and show
both issues as fixed there. Thanks again!!!!!!

> Dovecot with index files passes all imaptest tests with both mbox and
> maildir formats.

What about when you have legacy software mucking with the mbox files and
possibly deleting/corrupting the index files? Do you store metadata in
the mbox file, and if so how do you deal with the legacy software issues?
If you store metadata in the index file, how do you prevent problems with
loss of metadata due to legacy software (or users deleting what they think
to be a "junk file")?

These were the issues that prevented index files in mbox format with UW
imapd. It was easy enough to do this for mbox format if we could define
away legacy software and users mucking with "junk files"; but if that was
the case there were better choices than mbox format.

Thus, I designed mbox support to work in the worst possible environment
(which included NFS-mounted directories!), and that if things were any
better the site should use a better format.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Fri, 11 Jul 2008 09:44:42 -0700, Mark Crispin wrote:

> Please update your list to add "Panda IMAP 2008, mix format", and show
> both issues as fixed there. Thanks again!!!!!!

Added. Although I think it's a bit pointless if it's not possible to
download it..

>> Dovecot with index files passes all imaptest tests with both mbox and
>> maildir formats.
>
> What about when you have legacy software mucking with the mbox files and
> possibly deleting/corrupting the index files? Do you store metadata in
> the mbox file, and if so how do you deal with the legacy software
> issues? If you store metadata in the index file, how do you prevent
> problems with loss of metadata due to legacy software (or users deleting
> what they think to be a "junk file")?

The index files could be thought more of as a "shared and permanent in-
memory state of the mailbox". Dovecot's mbox handling is very close to
how UW-IMAP handles mbox files. The exact same headers are used in the
exact same way. The headers are written at same times (CHECK, EXPUNGE,
mailbox close). I guess the main differences are:

- If mbox file's size increases, Dovecot first checks if there are new
mails at the end of the file. If so, those are read. This is still just
like in UW-IMAP, but:
- If mbox file's mtime or size changes and the above check fails for
some reason, the mbox file is synchronized (read from the beginning) to
see if messages were expunged or flags were changed. Those changes are
applied to index.
- If some flags had been changed in the index but not yet written to
mbox file, those messages' flags aren't updated when synchronizing, even
if the flags had changed.

If index file gets deleted or corrupted it's created again from the
metadata in the mbox file. If the index doesn't have all updates, mbox is
synchronized to update it.

I think that's a simplified version of it. Dovecot also has this kind of
a "dirty state" where it's not entirely sure that the mbox has what
Dovecot thinks it should have. In dirty state when trying to access
messages (read, change flags, etc) Dovecot first reads the message's
headers to verify that the expected message really is there (based on X-
UID: header or MD5 sum of some specific headers if X-UID: wasn't written
yet). If the expected message isn't there, the entire mbox is again
synchronized to find out what had changed.

Anyway, I'm not too concerned about legacy programs modifying the
mailbox. Some people want/need to keep using mbox format (for all kinds
of different reasons) and I don't think they should be denied the full
IMAP functionality just because there exist some legacy tools (that they
don't use anyway).

The code is quite complex though, but there are other reasons for that as
well. Probably the main one is that Dovecot rewrites growing mbox files
backwards, so that if it crashes in the middle less data gets lost.

> Thus, I designed mbox support to work in the worst possible environment
> (which included NFS-mounted directories!), and that if things were any
> better the site should use a better format.

A lot of people want to use NFS, so recently I spent a lot of time
figuring out how to make Dovecot work reliably over the NFS. With Linux
(and probably Solaris) everything should work now reliably, but FreeBSD's
NFS caching needs some fixing..

On Fri, 11 Jul 2008, Timo Sirainen posted:
>> Please update your list to add "Panda IMAP 2008, mix format", and show
>> both issues as fixed there. Thanks again!!!!!!
> Added. Although I think it's a bit pointless if it's not possible to
> download it..

Thanks.

Your page lists several other closed-source products, so I think that it's
only fair to list Panda too...

> - If mbox file's mtime or size changes and the above check fails for
> some reason, the mbox file is synchronized (read from the beginning) to
> see if messages were expunged or flags were changed. Those changes are
> applied to index.

Beware; you can't always count on the mtime or even size being updated
correctly over NFS. From what you said later, you may have already
encountered some of this.

> - If some flags had been changed in the index but not yet written to
> mbox file, those messages' flags aren't updated when synchronizing, even
> if the flags had changed.

So, a legacy program that doesn't know about the index may see an outdated
state of the flags? That's making me remember bug reports from 15 or so
years ago...*shudder* :-(

> If index file gets deleted or corrupted it's created again from the
> metadata in the mbox file. If the index doesn't have all updates, mbox is
> synchronized to update it.

I'm not sure what you mean by the second sentence. You have some sort of
synchronization step in which you attempt to decide which is more
authoritative, the mbox or the index, and you update the one to reflect
the other, correct? If the index is recreated, then the mbox is the
authoritative one? Do you have a case where both may have updates that
need to be synchronized with the other, and if so, how do you resolve it?

> Anyway, I'm not too concerned about legacy programs modifying the
> mailbox. Some people want/need to keep using mbox format (for all kinds
> of different reasons) and I don't think they should be denied the full
> IMAP functionality just because there exist some legacy tools (that they
> don't use anyway).

That makes sense given your design of not being excessively concerned over
legacy. For me, legacy compatibility was the overriding concern.
Sheesh, I even had to make it work from a PC using Samba/NFS (oh, the
horror, the horror...).

I agree that there are people who want to stay with mbox even though they
don't have legacy compatibility needs. Fortunately, that class is fading
away simply because mailbox sizes are growing too fast for flat files to
scale any longer.

> Dovecot rewrites growing mbox files
> backwards, so that if it crashes in the middle less data gets lost.

That's a very good idea. I should have done that.

> A lot of people want to use NFS, so recently I spent a lot of time
> figuring out how to make Dovecot work reliably over the NFS. With Linux
> (and probably Solaris) everything should work now reliably, but FreeBSD's
> NFS caching needs some fixing..

Take my advice and flee, just as fast as you can! You won't believe how
many alligators are in that swamp. ;-)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Fri, 11 Jul 2008 12:09:31 -0700, Mark Crispin wrote:

> On Fri, 11 Jul 2008, Timo Sirainen posted:
>>> Please update your list to add "Panda IMAP 2008, mix format", and show
>>> both issues as fixed there. Thanks again!!!!!!
>> Added. Although I think it's a bit pointless if it's not possible to
>> download it..
>
> Thanks.
>
> Your page lists several other closed-source products, so I think that
> it's only fair to list Panda too...

Yes, but I've still been able to download and test myself all those
products (well, except for one of them I was only given remote access).

>> - If mbox file's mtime or size changes and the above check fails for
>> some reason, the mbox file is synchronized (read from the beginning) to
>> see if messages were expunged or flags were changed. Those changes are
>> applied to index.
>
> Beware; you can't always count on the mtime or even size being updated
> correctly over NFS. From what you said later, you may have already
> encountered some of this.

Right. I'm doing some OS-specific NFS cache flushing calls to force
getting the latest info from NFS server.

>> - If some flags had been changed in the index but not yet written to
>> mbox file, those messages' flags aren't updated when synchronizing,
>> even if the flags had changed.
>
> So, a legacy program that doesn't know about the index may see an
> outdated state of the flags? That's making me remember bug reports from
> 15 or so years ago...*shudder* :-(

Yes, but only while the mailbox is opened.

>> If index file gets deleted or corrupted it's created again from the
>> metadata in the mbox file. If the index doesn't have all updates, mbox
>> is synchronized to update it.
>
> I'm not sure what you mean by the second sentence. You have some sort
> of synchronization step in which you attempt to decide which is more
> authoritative, the mbox or the index, and you update the one to reflect
> the other, correct?

Pretty much, yes. Dovecot internally does this "synchronization" step
after every command and its purpose is to get index and backend mailbox
states equivalent, as well as do any pending changes to the backend
mailbox (e.g. flag changes/expunges are first written to a transaction
(index) log and later the synchronization step notices them and actually
does those changes to the mailbox).

> If the index is recreated, then the mbox is the
> authoritative one? Do you have a case where both may have updates that
> need to be synchronized with the other, and if so, how do you resolve
> it?

For each changed flag I initially mark it "dirty" in indexes. When the
flag is written to mbox file it's marked undirty in indexes. So when
reading the mbox file I always update the flag changes to undirty
messages, but ignore all flag changes to dirty messages. When the flags
are being written back to mbox they overwrite any changes there might
have been.

>> A lot of people want to use NFS, so recently I spent a lot of time
>> figuring out how to make Dovecot work reliably over the NFS. With Linux
>> (and probably Solaris) everything should work now reliably, but
>> FreeBSD's NFS caching needs some fixing..
>
> Take my advice and flee, just as fast as you can! You won't believe how
> many alligators are in that swamp. ;-)

I'm already pretty familiar with Linux kernel's NFS client code now.
Once I have some more time I'll probably spend a bit more time studying
how file opening works in Linux NFS and then figure out how to get
FreeBSD NFS work in a similar way to fix the problems I see with it.

I don't necessarily need to fully support NFS with all OSes, just be able
to say that some specific kernel versions work, and use the rest at your
own risk.

On Fri, 11 Jul 2008, Timo Sirainen posted:
> Yes, but I've still been able to download and test myself all those
> products (well, except for one of them I was only given remote access).

I'm willing to set up remote access for you and/or a private download.

> Right. I'm doing some OS-specific NFS cache flushing calls to force
> getting the latest info from NFS server.

Which calls, and when were they introduced? AFACT, the only way to do it
on some platforms is to make sure that all processes close the file.

> I don't necessarily need to fully support NFS with all OSes, just be able
> to say that some specific kernel versions work, and use the rest at your
> own risk.

That's probably a viable strategy to follow today. It wasn't 15 years
ago, when you couldn't even count upon having an ANSI compiler or POSIX.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Fri, 11 Jul 2008 15:41:04 -0700, Mark Crispin wrote:

> On Fri, 11 Jul 2008, Timo Sirainen posted:
>> Yes, but I've still been able to download and test myself all those
>> products (well, except for one of them I was only given remote access).
>
> I'm willing to set up remote access for you and/or a private download.

I'll ask about it later when I've done some new changes.

>> Right. I'm doing some OS-specific NFS cache flushing calls to force
>> getting the latest info from NFS server.
>
> Which calls, and when were they introduced?

I've written about them in http://unixcoding.org/NFSCoding (attribute
cache flushing).

> AFACT, the only way to do
> it on some platforms is to make sure that all processes close the file.

I remember seeing this somewhere, maybe on Linux. But it got fixed in a
newer version.

Mark Crispin wrote:
> I'm sorry. The silver lining in this cloud is that UW IMAP with mix
> format is in pretty good state, and ought to provide good service for
> the next several years before it will need to be retired. This will
> give you time to deploy Exchange and/or choose a cloud provider (which
> is the direction that most enterprises are taking).

Hello. I'm running UW-Imap with Debian (in Etch, I know it's an old
version of UW-imap, but it works fine and it is just for a home server).
I'm using the "MBX" mailbox format.

What is this "MIX" format? Not the same as the "MBX" format? Is "MBX"
deprecated?

Yes, I've been out of the loop for a long time...

Thanks for the nice UW-imap server, by the way!

---

Also, I'm shocked by you endorsing Microsoft Exchange... Do you think
Exchange is a good option? (Exchange means Active Directory, which means
surrendering the kingdom's keys to Microsoft...)

Pepe <pepe@naleco.com> writes:

> Hello. I'm running UW-Imap with Debian (in Etch, I know it's an old
> version of UW-imap, but it works fine and it is just for a home
> server). I'm using the "MBX" mailbox format.
>
> What is this "MIX" format? Not the same as the "MBX" format? Is "MBX"
> deprecated?

mbx is a single file format which means, amongst other things, that
performance suffers when a folder gets very large.

mix has other advantages; the 2006 release announcement mentioned some:

http://mailman2.u.washington.edu/pip...er/000610.html

In addition to improved performance the most noticable differences are
that mix is "dual-use", i.e. a mix folder is also a directory that can
contain other folders, and error recovery is better (I think because
data and metadata are in separate files).

Cheers,

- Joel