Document Store

By default, when making API requests, EmailEngine will still use IMAP as the backend, even if the Document Store has been enabled. To force EmailEngine to use the Document Store, add a GET query argument documentStore=true to all API requests.

Document Store is only available for regular read requests, such as listing or searching messages. If you want to fetch attachments or update flags for messages, these requests will always go against the IMAP server.

Requests against the Document Store do not require setting the path option. Unlike with IMAP, where you can only search for emails from a specific folder, Document Store allows you to search all messages from the entire account.

The following example searches for unseen emails from the entire account (path is not set for the request):

curl -XPOST -s "http://example.com/v1/account/example/search?documentStore=true" \
    -H "Authorization: Bearer test123" \
    -H "Content-type: application/json" \
    -d '{
  "search": {
    "unseen": true
  }
}'

With searches against the Document Store, you can use an additional filtering option called documentQuery. This is a search object structure that EmailEngine would pass directly to ElasticSearch when running the query. The custom query would be merged with regular search options, so you'd still have to provide the search query object, but you can leave it empty.

The following example searches for unseen emails with attachments. IMAP searches do not allow searching for attachments, but in ElasticSearch, it is possible. Note, though, that the attachment field is stored as a nested document, so the query must be built as a nested query as well. This is why the resulting query looks quite complicated.

All attachments have an "id" property, so the query matches any document with attachment.id property set.

curl -XPOST -s "http://example.com/v1/account/example/search?documentStore=true" \
    -H "Authorization: Bearer test123" \
    -H "Content-type: application/json" \
    -d '{
  "search": {
    "unseen": true
  },
  "documentQuery": {
    "nested": {
      "path": "attachments",
      "query": {
        "bool": {
          "must": [
            {
              "exists": {
                "field": "attachments.id"
              }
            }
          ]
        }
      }
    }
  }
}'

Note that the query is basically a filter. This means you can only use filtering terms but not anything more advanced like aggregations or grouping. You'd have to run your queries against the ElasticsEarch server directly for these types of queries.

In the Document Store configuration page, there is a tab called "Pre-processing". This allows you to filter and modify email contents before these are synced to the Document Store.

Message pre-processing consists of two components.

Filtering function

First is the filtering function. This is a tiny program written in Javascript. The function takes the email payload as an argument and returns if the email should be synced or not based on that input.

The following example filter skips all emails for the account "example". These emails are not synced to the Document Store.

if(payload.account === "example"){
    return false;
}
return true;

Mapping function

Second is the mapping function. It takes the email payload and morphs it into the required output structure for syncing. Mapping is mostly useful for adding new fields, or removing PII data from existing fields. This function is also a JavaScript code, just like the filtering function. Whatever the function returns will be synced to the Document Store. Protected fields like id, account and path can not be modified and will be always included as part of the output.

The following example adds a new field called type to the synced email.

payload.type = 'email';
return payload;

NB! If you add additional fields to synced emails, you probably should create a field type mappings for these fields first. Otherwise, ElasticSearch generates the type mapping dynamically itself, and it might not be what you expect.

NB! Beware, once a field mapping has been set, you can't modify it unless you migrate the entire index. The same applies to dynamically mapped fields – if ElasticSearch finds a field value without a type mapping, it will set it dynamically, and you can't modify that type later. So define any mappings you want to use before storing data in ElasticSearch.

EmailEngine allows you to create basic types only. For example, keywords, numbers, etc. For more complex object types, you would have to manage these manually.

You can generate new field mappings using curl like this:

curl -XPUT -k "https://localhost:9200/emailengine/_mapping?pretty=true" \
  -u "elastic:PASSWORD" \
  -H "Content-type: application/json" \
  -d '{
    "properties": {
      "type": {
        "type": "keyword",
        "ignore_above" : 128
      }
    }
  }'

Read more about it here.