Cross-folder fulltext search with Dovecot deprecated
The mail search in OX App Suite can utilize fulltext and cross-folder search capabilities, as they are provided by Dovecot. This article aims to be a short walkthrough for setting up Dovecot and the OX App Suite middleware accordingly. I assume that you already have a working Dovecot installation that is used as the primary mail backend for your (again already existing and basically configured) OX App Suite installation. I further assume that you are running Dovecot in version 2.2.9 and OX App Suite 7.6.0 on Debian Wheezy. We will use Solr as mail index.
Dovecot
Both search features are realized via Dovecot plugins. Fulltext search relies on FTS and FTS Solr. Cross-folder search is realized via a virtual folder, that claims to contain all mails from all other folders. Debian Wheezy comes with Dovecot 2.1.7. As this version is quite old, we take the packages from the backports repository, which contains version 2.2.9. Nevertheless this should all work with 2.1.7 also. After adding the backports entries my /etc/apt/sources.list looks like this:
deb http://ftp2.de.debian.org/debian/ wheezy main contrib non-free
deb-src http://ftp2.de.debian.org/debian/ wheezy main contrib non-free
deb http://ftp2.de.debian.org/debian/ wheezy-updates main contrib non-free
deb-src http://ftp2.de.debian.org/debian/ wheezy-updates main contrib non-free
deb http://ftp2.de.debian.org/debian/ wheezy-backports main contrib non-free
deb-src http://ftp2.de.debian.org/debian/ wheezy-backports main contrib non-free
Installing and configuring the fulltext index
We start with installing Solr. I use solr-jetty here because it's easy to configure and more leightweight than a full tomcat installation. Later we will use the Solr admin panel to see if everything works. The admin panel uses Java Server Pages (JSP), therefore we also need a JDK. We install both via
$ aptitude install solr-jetty openjdk-6-jdk
We also need Dovecots FTS Solr plugin. When installing Dovecot packages, we have to explicitly choose the backports repository or we get the original 2.1.7 packages. Install the plugin via
$ aptitude -t wheezy-backports install dovecot-solr
Before we can start Jetty, which in turn starts Solr as a web app, we have to configure it. The Solr admin panel refers to a shared jQuery library, that is linked into the web app. Therefore we have to allow the delivery of symlinks by Jetty. Open /etc/jetty/webdefault.xml and add the following to the <servlet> section:
<init-param>
<param-name>aliases</param-name>
<param-value>true</param-value>
</init-param>
Then open /etc/default/jetty and change NO_START to 0. To access Solrs admin panel Jetty must accept connections from the outside. So you have to adjust JETTY_HOST accordingly. Further there is a problem with the softlink that links the solr webapp into Jetty. /var/lib/jetty/webapps/solr points to /usr/share/solr/webapp, what is wrong. The correct path is /usr/share/solr/web. We fix this with
$ rm /var/lib/jetty/webapps/solr
$ ln -s /usr/share/solr/web /var/lib/jetty/webapps/solr
You may also check the permissions for the jquery library. By default, it's a symlink at /usr/share/solr/web/admin/jquery-1.4.3.min.js which throws a 404 when visiting the solr admin panel. Examinate /var/log/jetty/*stdout.log for any Jetty related issues.
The productive use has shown to change these values in the /etc/jetty/jetty.xml to get jetty work with big mailboxes:
<Set name="maxIdleTime">100000</Set>
<Set name="headerBufferSize">65536</Set>
Now you can start Jetty with
$ service jetty start
If everything was done correctly, you should reach Solrs admin panel at http://<ip or host>:8080/solr/admin.
Now its time to activate the FTS Solr plugin in Dovecot and to make use of our freshly set up indexing server. Open /etc/dovecot/conf.d/10-mail.conf and change
#mail_plugins
to
mail_plugins = fts fts_solr
To configure the plugins open /etc/dovecot/conf.d/90-plugin.conf and change the plugin-section to
plugin {
fts = solr
fts_autoindex = yes
fts_solr = url=http://localhost:8080/solr/
}
fts_autoindex = yes will enable automatic indexing of new mails. Dovecot 2.2.9 or newer is required by this feature.
Solr uses a so called schema that defines the structure of the index. A schema is defined in an XML file. The schema for Dovecot is provided by the dovecot-solr package and can be found at /usr/share/doc/dovecot-core/dovecot/solr-schema.xml. We have to replace Solrs default schema with this one:
$ cp /usr/share/doc/dovecot-core/dovecot/solr-schema.xml /etc/solr/conf/schema.xml
Afterwards we restart Jetty and Dovecot to apply the changes.
$ service jetty restart
$ service dovecot restart
Now we can start indexing some mails, to see if everything works. According to the FTS-Solr manual we can index a mailbox with
$ doveadm fts rescan -u <user>
$ doveadm index -u <user> '*'
The first call instructs Dovecot to remember that a rebuild of all indexes for the given user is necessary. The second one then forces the rebuild-operation and blocks until all mails are indexed. We use * as a wildcard for all mailboxes of the given user here. http://<ip or host>:8080/solr/admin/schema.jsp should now show the numDocs value as equal to the number of mails contained in the mailbox.
Congratulations, searching within mail bodies now utilizes Solr and is blazing fast!
Configuring the all-messages folder
To enable cross-folder search we configure Dovecot to add a special folder to every mailbox. From the outside the folder looks like it contains all mails from all other folders. Some more information can be found here. Add the following to your Dovecot configuration:
mail_plugins = $mail_plugins virtual
namespace Virtual {
prefix = Virtual/
separator = /
hidden = yes
list = no
subscriptions = no
location = virtual:/etc/dovecot/virtual:INDEX=/var/vmail/%u/virtual
}
This makes use of the dovecot virtual folders plugin. A new hidden namespace Virtual is created, which will not be contained in IMAP LIST responses and not accept subscriptions. However, folders below that namespace can be selected and examined. In our case we define a global configuration for virtual folders below /etc/dovecot/virtual, which makes configured folders appear in every users account. However, indexes for such folders need to be created per-mailbox of course, which we expect to be located under /var/vmail/.
To create a virtual folder, a file system folder carrying the target name needs to be created below the denoted path. In our case we create a directory /etc/dovecot/virtual/All. Folder owner of the virtualand virtual/All folders needs to be the system user running the dovecot process. In our case it's vmail.
mkdir -p /etc/dovecot/virtual/All
chown -R vmail:vmail /etc/dovecot/virtual
Now we need to create the virtual folders configuration. Create a new file /etc/dovecot/virtual/All/dovecot-virtual and open it in your favorite editor. E.g. you might decide to include all mails from all folders, but Trash and Spam.
*
-INBOX/Trash
-INBOX/Trash/*
-INBOX/Spam
-INBOX/Spam/*
all
The file can be owned by root but must be readable by the user running the dovecot process.
As a result every mail account will contain a selectable mailbox Virtual/All which pretends to contain all messages from all other mailboxes but Trash and Spam (given that these are named like this and located below the INBOX anmespace with / as separator).
For App Suite to be able to display the original folders within the search results Dovecot needs to announce an additional capability which needs to be added via
imap_capability = +XDOVECOT
Excursion: Make the all-messages folder visible
If you want to make the All-folder available within the folder tree, don't hide the Virtual namespace and enable listing. You should also think about more user-friendly names for the namespace and folder then. Furthermore you would probably want to mark the folder with the special-use flag \All. This can be done via adding the following to /etc/dovecot/conf.d/15-mailboxes.conf:
namespace Virtual {
mailbox All {
special_use = \All
}
}
OX App Suite
The last step is to configure OX App Suite. At first open /opt/open-xchange/etc/imap.properties and change the value of com.openexchange.imap.imapSearch to force-imap. Then open /opt/open-xchange/etc/findbasic.properties and configure
com.openexchange.find.basic.mail.allMessagesFolder = Virtual/All
Restart the server with
$ /etc/init.d/open-xchange restart
OX App Suite uses that folder now when starting a search from within the inbox:
