SOGo and Cyrus IMAP: 2.4 works well.

| | TrackBacks (0)
The open source group-ware suite SOGo can be married happily together with Cyrus IMAP, using the Version 2.4.x of the powerful IMAP server.

The reason lies in the handling of the very important \Seen-Flag.

In Cyrus 2.3.x, you had to type this in your imapd.conf file:

flushseenstate: 1
And still you would have inconsistencies when using the same mailbox in parallel on multiple clients/interfaces.

We did an upgrade from 2.3.18 to 2.4.13 some days ago and all these problems vanished.

Why is the \Seen-Flag so important for SOGo and not for e.g. Thunderbird?

Imagine two connections to the INBOX. Number one is in IDLE state and a new messages comes in:

+ idling
* 24459 EXISTS

On the second connection to the INBOX just FETCH this message:

. FETCH 24459 RFC822.TEXT

The server will reply:

* 24459 FETCH (FLAGS (\Seen) RFC822.TEXT {263}

The "FLAGS (\Seen)" is important here. The server tells itself, that it has set the \Seen-Flag on this message. SOGo relies on this. Thunderbird STOREs explicitly the \Seen-Flag after having retrieved the message information.

"flushseenstate: 1" was an Option in Cyrus 2.3.x to really set this flag immediately if it is triggered by a FETCH, otherwise it would only have been set in reality after re-SELECT-ing the mailbox or closing the connection - resulting in a crazy behavior of read/unread-marks in SOGo.

The IDLEing connection just tells correctly then:

* 24459 FETCH (FLAGS (\Recent \Seen))

For Cyrus 2.4, these problems do not exist any more, as the core mailbox routines handling ACLs and Flags were rewritten for consistency.

Additionally you get QRESYNC support, ESEARCH, compressed replication streams and other extensions.

What's different in Cyrus 2.4 from 2.3?

  • All databases are recommended to be in skiplist. Building Cyrus 2.4 without Berkeley DB is something I felt very comfortable (no more DBERROR).
  • The \Seen-Flag of the user's own mailboxes/folders are now part of cyrus.index. The Seen-Files only store \Seen-Flags of shared mailboxes which are shared to the user.
  • The cyrus.expunge merged into cyrus.index.
  • IMAP2, IMAP3 and IMAP4 were removed, resulting in smaller code. Only IMAP4rev1 is supported. But - be honest - do you know any kind of IMAP client not compatible with 4rev1?
  • The POP3 server has a quick check whether the mailbox is empty - so POP3 checks on "no mail" are extremely rapid and less resource consuming. Statuscache has to be enabled for this.
  • Charset-Handling is now based an Unicode 5.2, and UTF-8 mailbox names can be used within sieve.
  • Deleted folders can also be included in delayed expunges, the new option is "delete_mode".

Re-indexing mailboxes

These performance and stability bumps do come however with a certain pain: The cyrus.index-files of each mailbox/folder have to be recreated.

In fact it is a reconstruct which merges Seen- and Expunge-Files:


This rebuild is triggered as soon as the mailbox gets accessed by a pop3d, imapd or lmtpd process. cyr_expunge and squatter also trigger this.

And now, imagine 8,000 imapd processes triggering this reconstruct. You won't be able to say "IOPS" slower than your system will be. And this is exactly the scenario when you do the update and mail comes in through your SMTP gateway.

One solution (if really "on-line" re-indexing is wanted) is to shorten retransmit times of your SMTP server which feeds your Cyrus system and to set the timeout for the first delivery attempt very low (some seconds). So index triggering gets started, and the delivery process is free for other jobs. Next try the index rebuild will hopefully be finished.

The better solution is to announce downtime, just do a guess based on your experience how many time it will take to reconstruct all your mailboxes/folders.

The best solution - sure - would be a storage system where you can migrate all your mailbox partitions/volumes transparently to the fastest device (regarding random IOPS) you have (e.g. SSD) and to make the on-line re-indexing there - once 70% or 80% of the INBOXes are done (due to LMTP delivery) the system will get slowly back to normal. The remainder of the mailboxes gets updated due user access or a running squatter job.

0 TrackBacks

Listed below are links to blogs that reference this entry: SOGo and Cyrus IMAP: 2.4 works well..

TrackBack URL for this entry:

December 2015

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    


This blog is owned by:

Pascal Gienger
J├Ągerstrasse 77
8406 Winterthur

Google+: Profile
YouTube Channel: pascalgienger