My South

SOGo and Cyrus IMAP: 2.4 works well.

2012-01-15T10:00:26Z

The open source group-ware suite SOGo can be married happily together with Cyrus IMAP, using the Version 2.4.x of the powerful IMAP server.

The reason lies in the handling of the very important \Seen-Flag.

In Cyrus 2.3.x, you had to type this in your imapd.conf file:

flushseenstate: 1

]]>
We did an upgrade from 2.3.18 to 2.4.13 some days ago and all these problems vanished.

Why is the \Seen-Flag so important for SOGo and not for e.g. Thunderbird?

Imagine two connections to the INBOX. Number one is in IDLE state and a new messages comes in:

+ idling
* 24459 EXISTS
* 1 RECENT

On the second connection to the INBOX just FETCH this message:

. FETCH 24459 RFC822.TEXT

The server will reply:

* 24459 FETCH (FLAGS (\Seen) RFC822.TEXT {263}

The "FLAGS (\Seen)" is important here. The server tells itself, that it has set the \Seen-Flag on this message. SOGo relies on this. Thunderbird STOREs explicitly the \Seen-Flag after having retrieved the message information.

"flushseenstate: 1" was an Option in Cyrus 2.3.x to really set this flag immediately if it is triggered by a FETCH, otherwise it would only have been set in reality after re-SELECT-ing the mailbox or closing the connection - resulting in a crazy behavior of read/unread-marks in SOGo.

The IDLEing connection just tells correctly then:

* 24459 FETCH (FLAGS (\Recent \Seen))

For Cyrus 2.4, these problems do not exist any more, as the core mailbox routines handling ACLs and Flags were rewritten for consistency.

Additionally you get QRESYNC support, ESEARCH, compressed replication streams and other extensions.

What's different in Cyrus 2.4 from 2.3?

All databases are recommended to be in skiplist. Building Cyrus 2.4 without Berkeley DB is something I felt very comfortable (no more DBERROR).
The \Seen-Flag of the user's own mailboxes/folders are now part of cyrus.index. The Seen-Files only store \Seen-Flags of shared mailboxes which are shared to the user.
The cyrus.expunge merged into cyrus.index.
IMAP2, IMAP3 and IMAP4 were removed, resulting in smaller code. Only IMAP4rev1 is supported. But - be honest - do you know any kind of IMAP client not compatible with 4rev1?
The POP3 server has a quick check whether the mailbox is empty - so POP3 checks on "no mail" are extremely rapid and less resource consuming. Statuscache has to be enabled for this.
Charset-Handling is now based an Unicode 5.2, and UTF-8 mailbox names can be used within sieve.
Deleted folders can also be included in delayed expunges, the new option is "delete_mode".

Re-indexing mailboxes

These performance and stability bumps do come however with a certain pain: The cyrus.index-files of each mailbox/folder have to be recreated.

In fact it is a reconstruct which merges Seen- and Expunge-Files:

This rebuild is triggered as soon as the mailbox gets accessed by a pop3d, imapd or lmtpd process. cyr_expunge and squatter also trigger this.

And now, imagine 8,000 imapd processes triggering this reconstruct. You won't be able to say "IOPS" slower than your system will be. And this is exactly the scenario when you do the update and mail comes in through your SMTP gateway.

One solution (if really "on-line" re-indexing is wanted) is to shorten retransmit times of your SMTP server which feeds your Cyrus system and to set the timeout for the first delivery attempt very low (some seconds). So index triggering gets started, and the delivery process is free for other jobs. Next try the index rebuild will hopefully be finished.

The better solution is to announce downtime, just do a guess based on your experience how many time it will take to reconstruct all your mailboxes/folders.

The best solution - sure - would be a storage system where you can migrate all your mailbox partitions/volumes transparently to the fastest device (regarding random IOPS) you have (e.g. SSD) and to make the on-line re-indexing there - once 70% or 80% of the INBOXes are done (due to LMTP delivery) the system will get slowly back to normal. The remainder of the mailboxes gets updated due user access or a running squatter job.

login account failure: No account present for user

2012-01-14T10:20:00Z

doukoula console login: pascal
Password:            login account failure: No account present for user
   instance svc:/system/console-login:default exited with status 1

What's the problem?
It is the nscd, the Name Service Cache Daemon.

What did I wrong?

]]> set up SSH, and ntp at the end. I did some patch updates and after a reboot I could not log in.

I could not log in any more as "pascal". The error message was:

login account failure: No account present for user

And - additionally - the SMF instance "system/console-login" just quit (and was restarted).

Bad.

I disabled the name service cache to be able to log in again:

svcadm disable name-service-cache

I think it has s.th. to do with ntp, as the user was added when ntp was not active so passwd/shadow and group were "wrong" seeing the time stamp (it was 13 minutes in the future), and nscd did not take the new information for his cache.

In fact, I am not using NIS, no NIS+ and no heavy DNS requests. So I won't need nscd anyway. And the speed "bump" to cache passwd/shadow/group information is really not noticeable for a dozen system users...

The future of IT: CLOSED.

2012-01-13T19:27:24Z

I just read this (in German):

http://heise.de/-1413109
(in short: For ARM devices, Microsoft insists on having UEFI secure boot for their Windows 8 system, and metro apps will only be available on the Microsoft app store with signed code).

My comment:
The game is already at its end, my opinion.

Everybody wants closed platforms, "ecosystems". Nobody wants an open platform, besides us "freaks".

Developers DO want a closed platform, as a radical DRM/Licensing will generate guaranteed revenue and they can bully the user like they want to motivate them to pay more.

]]>
The big manufacturers want closed platforms because they create a Lock-In: If someone bought Apps for some $100 (which they would have for free on Ubuntu and others) they won't change the platform afterwards. They will be happy to pay $10 "upgrade fees" to register their new serial numbers of their new closed device.

Guys and girls who just program for fun are out of the game. We're not interesting anymore.

IT as a simple commodity like electric current coming out of the wall socket. And time will come when a "programmer" ("software engineer") will have to have a master degree and a damage insurance covering $10 million and other "nice things" to be able to be certified for the "App stores". You'll see it!

That's the game you started with your iOS devices and now you won't be able to put back the ghost in the bottle.

So begin to count your money, every little step will cost money - just as every little minute of turning on the water boiling kettle will show up on your electrical bill. And forget the idea that "you just program some lines of code to help you with a job"...

There are however some things which could break up things:

More and more very cheap computer boards are coming out, there is this one for $15:

http://www.freesoftwaremagazine.com/articles/allwinner_a10_gplcompliant_computer_15

And this one - when available - with HDMI, USB and (for $10 additionally) Ethernet for a price of $25:

http://mashable.com/2011/12/28/raspberry-pi-coming-january/

Imagine if people in all countries would use this thingies to create something completely new - unconctrollable by big corporations and governments....

And there is Google - they offer an App Store but it is NOT mandatory. And HTC has began to open their bootloader of Android devices.... Can they withstand the massive media press which tells people that "Openness" is "insecure"?

known_hosts in hash format - OpenSSH

2012-01-13T18:09:03Z

When you wonder why your known_hosts file has changed or you don't find your hosts:

|1|AEf8B/QCP+1wRKA761Cq/woad4s=|HVEWXKDdRpZmO3GeQ5T37xCFwgE= ssh-rsa AAAAB3NzaC1

yc2EAAAABIwAAAQEAt20HN/iasjy9KEW/BGjtxlsy8oebyEEnZvKAnjMck0EEVbA5rtE4dzisnbrYwZ4

67JP+p9UGZDQa4jbVo2ZHmz28nQapmw1WpLBD2wSN66PsMk5QCxICxBC6PDCOlwakQvLNES2B9R2cuev

G9Ag2ni+2Qdb17gEkkh2MZ91INylAzM7QWW7soGoSf1TsshSHexfMt12zb6kWxRuRCeT4fOzlJNOmPgr

uE3wTt/kfEbvPBZwDyUbEKApfYIxO8ic+FyO7qFEjHkhqT7px/oJMLS279uUHlhG+KWtPxYPhWaYulPZ

hhdn4D6pR+6shjPsH4VTyAUGwf7usKUvJZP59Ww==

Then your ssh server software is a new version which hashes the known hosts file entries. To remove entries from there because the host key differs (you installed the host completely from scratch), just type:

ssh-keygen -R hostname

or

ssh-keygen -R ipadress

That's it.

If you have a new ssh package, you may hash yourself your known_hosts file by typing:

ssh-keygen -H

Solaris 10 Minimal, getting started with SSH server

2012-01-13T15:24:49Z

Just because it happens again today:

After doing a very minimal Solaris 10 installation without remote services enabled, it is always the same thing to get the SSH server running (Solaris 10 DVD in drive):

# mount -F hsfs /dev/dsk/c2t0d0s0 /mnt

# cd /mnt/Solaris_10/Product

# pkgadd -d . SUNWsshdr SUNWgssc SUNWgss SUNWsshcu SUNWuiu8 SUNWsshdu SUNWsshr SUNWsshu

# cd /etc/ssh

# ssh-keygen -t rsa -b 2048 -f ssh_host_rsa_key

# ssh-keygen -t dsa -b 2048 -f ssh_host_dsa_key

# svcadm enable ssh

Replace device of the DVD drive with your actual configuration.

]]>

# cd /mnt/Solaris_10/Product

# pkgadd -d . SUNWntpr SUNWntpu SUNWbash

# vi /etc/inet/ntp.conf

# ntpdate ntp1.ptb.de (replace with your time server)

# svcadm enable ntp

To get the Patch Utility "pca" running:

# pkgadd -d . SUNWgcmn SUNWwgetr SUNWwgetu

# mkdir -p /usr/local/bin

# cd /usr/local/bin

# wget http://www.par.univie.ac.at/solaris/pca/stable/pca

PCA:
http://www.par.univie.ac.at/solaris/pca/installation.html

Mount local patch repository on this machine, configure pca, cd to the repository and run "pca -i". Reboot. Ready.

OpenSSL 1.0.0: New CApath hashes!

2012-01-09T12:04:50Z

A real case of RTFM of OpenSSL ...

After un upgrade of OpenSSL to Version 1.0.0 (from 0.9.8) the certificate authority chain of my certificate did not show up any more (was not given by the TLS server). A look in the OpenSSL manual could have helped to save 20 minutes of error searching :)

A

for i in *.pem; do ln -s $i `openssl x509 -noout -subject_hash -in $i`.0; done

was enough to restore the hash index links for my CA certs (the files did not have any whitespace or punctuation marks in their filename so $i was enough).

The manual states:

-subject_hash
outputs the ``hash'' of the certificate subject name. This is used in OpenSSL to form an index to allow certificates in a directory to be lookedup by subject name.

-subject_hash_old
outputs the ``hash'' of the certificate subject name using the olde ralgorithm as used by OpenSSL versions before 1.0.0.

Google Earth clone from Nokia using WebGL

2012-01-07T09:21:07Z

Fascinating.

Because it uses only WebGL a decent browser dows not need any plugin to render it.

It is faster than Google Earth on my machine and much faster than Google Maps.

But - it does not add additional layers with information, so speed competition my a little bit unfair.

If you want to try:

http://maps3d.svc.nokia.com/webgl/

Merry Christmas and a happy new year!

2011-12-24T10:27:21Z

May many of your wishes may become true in 2012.

Ubuntu 11.10 Oneiric Ocelot available

2011-10-13T17:31:36Z

Just a note: Ubuntu updater offers 11.10 final:

I'll try it to see whether it works.

New Solaris 10 Kernel patch 147441-03 (x86), 147440-03 (SPARC)

2011-10-02T08:36:25Z

Just a short note:

It contains important fixes for PKCS11 handling of KMS (Crypto Key Management).

Android 2.2: wrong arabic characters

2011-09-13T14:14:32Z

Apparently Android 2.2 is not capable of displaying arabic characters possible.

Has this changed in Android 2.3, 3.0 and 3.1?

]]> and K-9 Mail on Android 2.2 vs Thunderbird on Ubuntu 11.04. Click on them to enlarge.

Solaris 10 / 11 and Oracle VM Virtualbox

2011-08-08T18:34:13Z

As I want to test an IP multipathed iSCSI storage configuration under Solaris 10 and 11 I wanted to set them up in Oracle VirtualBox. It was more or less easy.

Please do not forget to install the VirtualBox guest tools, otherwise the guest will crash as soon as the host system "steals" CPU cycles.

On text-only installs, mount the CD image like this (Solaris 10, CD is IDE master):

# mount -F hsfs /dev/dsk/c0t0d0p0 /mnt

On Solaris 11, the CD was attached to controller "7" target "1" (SATA, Port 1)

# mount -F hsfs /dev/dsk/c7t1d0p0 /mnt

Then:

# pkgadd -d /mnt/VBoxSolarisAdditions.pkg

]]>

Solaris 10: upgrading root zfs pool: rewrite GRUB!

2011-08-08T11:45:18Z

If you accidentally upgraded your root zfs pool (which is not recommended until it is supposed to be done from Oracle) do not forget to update your boot signature, your boot archive and your grub installation -

but do not reboot before having done this, otherwise the system won't boot any more.

Don't upgrade the root pool! You won't be able to repair your system when booting from the actual Solaris 10 boot CD (09/10) as the root pool cannot be mounted then. The following steps are to make sure that your system will at least boot when you did the upgrade accidentally.

Example:

]]> # zpool status
pool: rpool
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
        pool will no longer be accessible on older software versions.
scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t0d0s0 ONLINE       0     0     0

errors: No known data errors
Now you're a little bit too fast and you upgrade the wrong pool (not the big storage pool you wanted to upgrade but the zfs root pool:

# zpool upgrade rpool
This system is currently running ZFS pool version 29.

Successfully upgraded 'rpool' from version 22 to version 29

# zfs upgrade -r rpool
6 filesystems upgraded

To clear the situation just do this:

# bootadm update-archive
# touch /rpool/boot/grub/bootsign/pool_rpool
# /boot/solaris/bin/update_grub
Creating GRUB menu in /
Installing grub on /dev/rdsk/c1t0d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 273 sectors starting at 50 (abs 16115)
# reboot

This will work, after reboot zpool status tells:

# zpool status
  pool: rpool
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t0d0s0  ONLINE       0     0     0

errors: No known data errors

Solaris Kernel 144501-19: zpool version 29, zfs version 5

2011-08-07T17:11:52Z

Since Kernel patch/update 144501-19, Oracle now puts zpool version 29 and zfs version 5 into production.

One visible change is the more detailed status when doing a scrub or a resilver operation:

# zpool status
  pool: rpool
 state: ONLINE
 scan: scrub in progress since Mon Aug  8 16:18:21 2011
    1.98G scanned out of 3.66G at 50.7M/s, 0h0m to go
    1.98G scanned out of 3.66G at 50.7M/s, 0h0m to go
    0 repaired, 54.13% done
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t0d0s0  ONLINE       0     0     0

errors: No known data errors

]]>

VER  DESCRIPTION
---  --------------------------------------------------------
 [...]
 23  Slim ZIL
 24  System attributes
 25  Improved scrub stats
 26  Improved snapshot deletion performance
 27  Improved snapshot creation performance
 28  Multiple vdev replacements
 29  RAID-Z/mirror hybrid allocator

History of zfs versions:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS filesystem version
 2   Enhanced directory entries
 3   Case insensitive and File system unique identifier (FUID)
 4   userquota, groupquota properties
 5   System attributes

My linux month: Start!

2011-08-06T13:38:51Z

My "day-book" of my Ubuntu experiences as a day-by-day system can be found here:

http://southbrain.com/south/articles/me-and-my-desktop-ubuntu-1104.html

The article will grow and be split in multiple ones.