Otdfbt

I'm looking for an imap server that is fast with larger folders. Say 20'000-100'000 emails per folder.

Currently I'm using dovecot, and opening a folder can take 10 seconds, and the HD light on the imap server is brinking like crazy.

I'm using alpine as a client, and it only lists the newest mails by default, so it's not that my client is trying to transfer everything when opening the mailbox. This can be seen in that when I scroll it has to load the subject lines for the next page (the first time I scroll there).

I'm using maildir on XFS.

Edit: I ask since it's not much data, in the grand scheme of things. If this was in a SQL database then getting the subject lines of the newest 40 messages would not take 10 seconds for a folder of 40'000 emails. The only data needed is:

SELECT date, from, subject FROM emails ORDER BY date DESC LIMIT 40;

Any ideas?

Dovecot is actually pretty good, performance-wise. Dovecot's Performance Tuning wiki page has a few tips and tricks to further improve performance. Keeping indices and maildirs on separate disks is a good thing to start with, if at all possible for you. You could also evaluate switching to Dovecot's dbox storage format.

Maybe you could try using a database engine for message storage, instead of using Maildir/Maildir++ mailboxes. This can be done with dbmail.

I don't know how reliable dbmail is for a production environment, but since you already have virtualization working, you could set it up on another VM for testing purposes and see how it performs on your environment.

Here's an overview of dbmail's architecture:

_{(source: dbmail.org)}

You don't mention the server specs...how much memory are you using, processor, network card/switches are gigabit? And if you look at the server, can you tell what's being maxed out? If it's drive throughput you aren't going to get very far changing the server software

I've been cloning systems over a network and was puzzled to have two systems on a gigabit switch pulling only about 15 MB/sec when I knew that my system was capable of bursts in the 50 MB/s range. Turned out it was the disks bottlenecking on the end-systems (I put a drive into a second IDE channel and did a direct DD, got the same transfer speeds).

You might want to check out the processor/disk/network usage as well as the switch and see if any of those are causing issues. If not those, you could look for ways to boost throughput using separate disks, separating mailboxes to different spindles, check and see if you can get any better throughput using hardware RAID mirroring (I'm not sure how much of a boost in read times off the disks you can get), or possibly moving to higher-performing hard disks with lower latency and bigger caches.

Since you're using dovecot I presume you're already using it's indexing features? I'm not aware of anything (at least, not anything free), that's faster than dovecot.