Features

This page describes what the tmail library does: how it talks to servers, reads and writes mail, uses GPG, and runs in the background. The terminal interface uses it to read, compose and act on mail; see Current state for what the shipped tmail command offers. For the settings, see Configuration.

Accounts and identities

tmail handles several accounts. Each account has its own IMAP server (or a local Maildir) and its own SMTP server. security is tls (implicit TLS), start_tls or none; the defaults are IMAP on 993 with tls and SMTP on 587 with start_tls.

Connections:

One long-lived IMAP connection per account serves the commands one at a time. After a drop it is reopened on demand, and a command that failed because the connection was lost is retried once when that is safe.
Watching for new mail uses a connection of its own, because IDLE blocks the connection it runs on.
Each send opens its own SMTP connection.

A sender identity is an extra From: address of an account, for example an alias that forwards to the mailbox. An identity has no server of its own: its mail goes out through the account’s SMTP server with the account’s login. Each identity can override the display name, the GPG settings, the signature file and the SMTP envelope sender (see Sender identities).

Maildir accounts

An account can read its mail from a local Maildir instead of an IMAP server. Another program keeps the Maildir filled: mbsync, offlineimap, fetchmail or a local mail delivery agent. tmail does not download mail itself.

[account.maildir]                  # instead of [account.imap]
path = "~/Mail/home"

An account has either [account.imap] or [account.maildir], never both. Sending still goes through SMTP, so [account.smtp] stays required. The folder layouts and a matching mbsync setup are in Maildir layout.

On a Maildir, tmail lists the folders, pages and reads messages, searches, sets flags, moves messages between folders, watches for new mail and saves the sent copy in the files. It never expunges: messages flagged deleted stay until your sync tool removes them.

UIDs. A Maildir has no UIDs, so tmail keeps a small index per account in ~/.config/tildemail/maildir/, never inside the Maildir, which may be synced or read-only. The index gives each message a UID and each folder a UIDVALIDITY, keyed by the unique part of the file name, so a flag change (a rename) or the move from new to cur never changes a UID. A UID is never reused. A message that a listing misses, because another program is renaming it at that moment, is looked for again before its UID is dropped. If you delete the index, every folder gets a new UIDVALIDITY and is numbered again.
Reading. A message that nothing has opened yet is moved from new to cur when you read it; it is not marked seen. A file that cannot be parsed shows as (unreadable message) in a listing and reports an error when opened. An empty file is never opened.
Links. Symbolic links, to a message or to a folder, are ignored, and tmp must not be a link: the programs that fill a Maildir never make one, and a link could point outside it.
Search is a scan. There is no server-side search, so tmail reads every message of the folder (every folder when you ask for all) and matches each word, ignoring case, anywhere in the sender, the subject or the text, as described in Search. It is fine for thousands of messages and slows down with the size of the folder. The local search index of the IMAP cache does not cover a Maildir.
New mail is found by looking at the folder every 5 seconds, not with inotify, so a Maildir on a network file system works too.
No cache. The files are already local, so the SQLite cache is not used for these accounts, and tmail cache status and tmail cache clear skip them.
Concurrent writers. mbsync can rename a file while tmail acts on it. When a rename finds the file gone, tmail lists the folder again and tries once more. If the file is still not there, it reports that another program changed the message.
Moves across filesystems. A move between folders on different filesystems is a copy and a delete, not one atomic rename.

Credentials

username and password are never stored in clear in the file. They take two placeholders: ${NAME} reads an environment variable and #{command} runs a command. Both are resolved only when connecting, and the value is kept in memory only. The rules and the known limitations are in Credentials.

GPG

Every cryptographic operation runs the gpg binary found on PATH, with your keyring and gpg-agent, so pinentry asks for passphrases as usual.

How gpg runs. tmail runs gpg with --batch --status-fd 2 and a private, temporary --log-file, so text gpg prints about a key or a file can never be read as a status line. It sets GPG_TTY from the terminal on standard input when the variable is not set. Only exit codes are logged, never content.
Format. Messages are PGP/MIME (RFC 3156): multipart/signed with a detached application/pgp-signature (SHA-256, micalg=pgp-sha256), and multipart/encrypted with application/pgp-encrypted. Inline PGP is not produced.
Decrypting runs gpg --no-use-embedded-filename --no-auto-key-retrieve --decrypt. The plaintext is read from standard output, never written to a file name the sender chose, and no keyserver is asked per incoming message. Verifying also runs with --no-auto-key-retrieve.
Encrypting. Recipients are matched by exact address (<bob@example.org>, never a substring search) or by hex key id or fingerprint. The gpg.key of the sender identity is always added as a recipient, so the copy in Sent stays readable by you. Before the editor’s text is confirmed, tmail checks that a usable public key exists for every recipient and stops with the missing ones listed. An expired or revoked key, a sign-only key, a key whose encryption subkeys are all expired or revoked, and a key whose user ids for the address are all revoked count as no key, because encrypting to them fails. So does a disabled key matched by address; a disabled key named by key id or fingerprint is still found, and gpg still encrypts to it. Recipient trust is left to gpg’s trust model: an untrusted recipient key counts as present, and makes encryption fail.
Finding and importing public keys. IGpgService.SearchKeysAsync looks a key up for an e-mail address, first in the web key directory of the address’s domain and then on the key server named in dirmngr.conf (or dirmngr’s default, hkps://keys.openpgp.org; a key server in gpg.conf is not used). It downloads into a temporary keyring and never writes yours, keeps only keys with a user id for exactly that address, and returns them flagged when expired or revoked. The lookup tells the web key directory host and the key server which address was asked for. IGpgService.ImportKeyAsync imports one key by its full fingerprint (40 or 64 hex digits): the key is downloaded first, its fingerprint must be exactly the requested one, data that holds another key besides it is refused, a key over 1 MiB is refused, and it is imported without other people’s signatures. The imported key is not trusted or certified: check its fingerprint with its owner and certify it with gpg before encrypting to it. Each import writes one log line with the fingerprint. Nothing runs these automatically. tmail gpg fetch runs them from the command line and imports only the key you choose (see Commands); Lua scripts can call them (gpg.search_key, gpg.import_key, see Scripting with Lua), including from an event handler, without asking you.

Signature status

A signature is reported as None, Good, Bad, UnknownKey or Untrusted.

A good signature only proves which key signed, and the key holder can add any user id to a key. So SignatureAssessor keeps Good only when both hold:

the From header holds exactly one address;
that address belongs to a user id of the signing key that is valid in your trust model (marginal, full or ultimate validity: a key you certified, or your own).

Every other good signature is Untrusted. If your gpg.conf sets trust-model always, gpg prints no trust line, so every signature is shown as untrusted.

For a Good or Untrusted signature made by a key named by its full fingerprint, OpenedMessage.SignerTrust holds what the keyring says about the key: its validity (Unknown, Invalid, Disabled, Revoked, Expired, Undefined, Never, Marginal, Full or Ultimate) and the owner trust you gave to its owner. The reader prints both at the end of the Signature: line. It is null when the key is not in the keyring or the keyring cannot be read: the status is then all there is to show.

Looking up the key of a signature

When the signature says Unknown key, K in the reader (signature.fetch_key, also the command :signature.fetch_key) looks the signing key up. Nothing is asked of any server before you press it, and nothing is imported before you answer y.

tmail looks for the key published for the From address (the web key directory of its domain and the key server) and keeps the one whose fingerprint, or the fingerprint of one of its subkeys, is the signing key (a signature made by a signing subkey names the subkey, not the primary key). When the From header holds more than one address, or that finds nothing, and the signature gave a full fingerprint, it asks the key server for that fingerprint. A signature that gave only a long key id is matched against the keys of the address and never sent to a key server by itself: a long key id is short enough to be forged.
A window shows the fingerprint in groups of four, the user ids the key claims (they are claims of the key holder, not certified identities; at most five, each cut to a line), the dates, whether it is expired or revoked, where it was found, and that the key is not in your keyring yet, so its validity and owner trust are unknown. It says “the key this signature names”, because the sender chooses the key a signature names. A key found only by its fingerprint was not published for the From address, and the window says so: its user ids may be anyone’s. One lookup runs at a time, and if you leave the message before it ends, nothing is asked.
On y the key is imported as tmail gpg fetch imports it (checked against its fingerprint, without other people’s signatures), and the message is opened again, so the Signature: and Key: lines show the new verdict.

An imported key is not trusted. The signature turns Untrusted, with the validity and owner trust of the key on the Key: line, until you check the fingerprint with its owner and certify the key with gpg (gpg --quick-sign-key), which makes it Good when the key has a valid user id with the From address. The lookup tells the hosts which address and fingerprint you are interested in, which is why it runs only when you ask. A key that matches the signature is still only a candidate: the signature is checked against it after the import, and it stays Unknown key if the key does not verify it.

Reading a message

IMessageReader.OpenAsync turns a loaded message into what a reader shows: the text and the attachments of the readable content, and what is known about the signature and the encryption. It parses the raw source and follows up to four PGP/MIME layers: an encrypted message is decrypted, a signed one is verified and its signed part becomes the content, in either order (a signature inside the encryption, or one that gpg reports while decrypting). S/MIME and inline PGP are read as ordinary text. One more shape is read as PGP/MIME: the encrypted mail that Proton Mail sends as a multipart/mixed, with the control part (application/pgp-encrypted, attachment.gpg) and the armored message (encrypted.asc) as attachments and often the public key of the sender (application/pgp-keys) as a third. The outer message is not authenticated, so the rule is strict: the message is put back in the shape of RFC 3156 and opened like any other only when every part of it is one of those (a text part that says null or nothing, one control part that says Version: 1, one armored message as application/octet-stream, and public keys), and only that text is left out. Any other part, a longer text, a second armored part or a key part that is not a key block leaves the message as it is, so that nothing a person could read is hidden. When the message is opened, the public key is listed as an attachment after the attachments of the message itself; it comes from outside the encryption, so nothing vouches for it: check its fingerprint before you trust it. The list marks such a message as encrypted from its structure alone (a multipart/mixed with one small application/pgp-encrypted part and an application/octet-stream part), like a real multipart/encrypted.

Signature. Good only under the rule in Signature status. A message with several signatures gets the worst status. The signer’s user id is shown as gpg printed it: it says who the key claims to be, not who sent the message.
Failures do not throw. A message gpg cannot decrypt is reported as encrypted but not decrypted, with an empty body and no attachments, and its ciphertext is never shown as text. A signature that cannot be checked (malformed, gpg missing) is Bad, and the body is still shown. Any other failure of a gpg step (an I/O error, gpg missing) is reported the same way. All of these carry a Problem text, and so does an encrypted or signed part the reader does not open (nested more than four layers deep, or inside another part such as a mailing-list wrapper). Cancelling kills gpg.
HTML. OpenedMessage.BodyText is always plain text; an HTML body is converted by stripping tags, dropping scripts and styles, decoding entities and starting a new line at block tags. A message with an HTML part and no text/plain part also gets OpenedMessage.BodyMarkdown: the HTML converted to Markdown (headings, lists, links, emphasis, quotes, code and tables) for a reader that can draw it. Scripts, styles, comments and hidden elements are dropped, only http, https and mailto links are kept, embedded images are skipped, and a table used to lay out a newsletter becomes plain blocks (only a table with header cells stays a table). An element the converter does not know keeps its text (font, center, u); the text of head, title, noscript, svg, select, textarea, iframe and similar is dropped. A body over 2,000,000 characters, nested more than 100 levels deep, or one that cannot be converted, has no Markdown and is read as plain text. A message that has a plain part is read as that part. Hidden elements are dropped from BodyText too, so a reply never quotes text the reader did not show.
Conversations. A message summary carries the Message-ID, In-Reply-To and References headers (the IMAP backend fetches References with the summary, the Maildir backend reads them from the file, and the cache keeps them). ConversationBuilder.Group links messages through those ids, also through ids of messages that are not loaded, and then joins a reply that has no thread header to the message with the same subject (without Re:, Fwd:, AW:, WG:, R:, I: and similar prefixes) that came just before it, if that was in the last 60 days. Only the last 50 ids of a References header are followed, and a text that is not a message id is ignored. Conversations come newest first, and so do the messages inside. The interface draws them.
Attachments. SaveAttachmentAsync writes from the content that was already opened, so an encrypted message is not decrypted twice. Attachment names from the sender are reduced to a safe file name before use. Saving never overwrites an existing file and saves one attachment at a time per opened message. Decrypting can ask for a passphrase through pinentry, so open an encrypted message where the terminal can be handed over.

Safe to draw

The body, the signer and the problem text are cleaned before they reach the screen:

control characters (escape sequences, the C1 range) become spaces; in a body, line breaks and tabs stay;
bidi controls and other invisible format characters are removed; zero-width joiners stay, for emoji and scripts that need them;
combining marks are limited to three per letter;
look-alikes of a blank (Hangul fillers, the braille blank) become spaces.

The same cleaning is available as ToDisplayLine() and ToDisplayBlock() for any other text taken from a message, such as a subject or a sender. The fields of the loaded message itself (subject, sender, recipients, headers, attachment names) are not cleaned. Draw them through those two methods, draw the attachment names through SafeFileName, and next to a signature result draw the From address that was checked, not only its display name.

Composing and replying

To send a message without the editor, from a script, see tmail send.

Writing a message opens $VISUAL or $EDITOR on a file like this:

To: bob@example.org
Cc:
Subject: Lunch
Sign: yes
Encrypt: no

Body text.

Only those five headers can be edited. Sign takes yes or no, and Encrypt takes yes, no or auto. Both start from the settings of the sender identity; changing them in the editor changes how that message is sent.
The message is refused when it has no recipient, when a header is malformed, or when Sign or Encrypt: yes is set without a gpg.key.
Encrypt: auto is decided when the editor closes. The message is encrypted when the sender has a gpg.key and every address in To and Cc has a usable public key, and sent in clear otherwise. Unlike Encrypt: yes, a recipient without a key never stops the send, but an unusable key of your own (gpg.key) still does, as with yes. The confirmation shows the message as not encrypted and says why, for example not encrypted: no public key for bob@example.org, or not encrypted: no gpg.key for this sender.
If gpg itself cannot be run or reports an error while the keys are checked, the message is not sent and the text you wrote is kept, exactly as with yes.
Looking up a missing key. With fetch_keys = true in [account.gpg] (or in an identity’s [account.identity.gpg]), a recipient without a public key is looked up when the editor closes, in the web key directory of the address’s domain and on the key server (see GPG); both are asked every time, and a key found by the first and a different one found by the second count as several. The lookup runs only for messages you are asked to confirm, never for tmail send. A recipient with exactly one key that is neither expired nor revoked gets an offer; with no key, several keys, only an expired or revoked key, or a failed lookup, the recipient stays without a key and the message behaves as without the setting: with Encrypt: auto it goes out in clear and the confirmation says why (not encrypted: no public key found for bob@example.org), with Encrypt: yes it is stopped. When every missing key has an offer, the message is encrypted and the confirmation lists each key in a window with the address, the fingerprint in groups of four and where it was found (web key directory or key server). Nothing is imported before you answer: on y, each key is imported (as tmail gpg fetch does, without other people’s signatures), and the message is encrypted with those keys accepted without a trust check for that one message only: the other recipients are checked first with your own trust model, and your trust database is neither read nor changed. In your keyring the new keys stay untrusted, so a later message to them (or a second try after a failure that came after the import) needs the key certified with gpg first, as for any imported key. If an import fails, nothing is sent and the text you wrote is kept.
If auto chose to encrypt and gpg then fails to encrypt, typically because it refuses an untrusted recipient key, the message is not sent and the text you wrote is kept. The error ends with a hint: Encrypt was chosen by "auto"; set "Encrypt: no" in the editor to send in clear. An explicit Encrypt: yes gets no hint.
Closing the editor without changes, or with no recipient and an empty body, cancels the message.
If sending fails, the text written is kept so it can be opened again.
After a successful send, a copy is stored in the account’s sent_folder with the Seen flag. A copy that could not be stored is reported; the message has left anyway.

The file is created with mode 0600 in $XDG_RUNTIME_DIR, or in ~/.config/tildemail/cache/drafts (mode 0700) when that variable is not set, and deleted afterwards.

Sender

A draft is sent as the identity whose address is in its From:. A From: that is neither the account’s address nor one of its identities is refused, and nothing is sent. The SMTP envelope sender (MAIL FROM) is the account address, or the identity address when the identity sets envelope_from = "identity".

Replies and forwards

A reply goes to Reply-To when the message has one, else to the sender.
Reply-all adds every other recipient, never lists your own addresses (the account’s and its identities’) and lists each address once.
A reply to a message you wrote yourself keeps the address you wrote it from. Any other reply is sent from the identity the original message was addressed to (found in To or Cc, the account’s own address first, then the identities in file order), or from the account when none matches. Addresses match exactly, case aside: squid+news@linux.it does not match squid@linux.it.
A forward is written from the account, quotes the original inline under a header and has no recipients.
Re: and Fwd: are not added twice.
Answering or forwarding an encrypted message always starts with Encrypt: yes, whatever the account default. With an identity set to encrypt = "auto", any other reply or forward starts with Encrypt: auto.

Text signatures

Each account and identity can name a file in ~/.config/tildemail/signatures/ with signature_file. The signature of the address you send as is added at the bottom of the compose file, after the standard -- separator, so you can still edit or delete it before sending.

The file is read when you start writing, so a change applies to the next message.
A missing file, or one over 16 KiB, stops the message with an error before the editor opens. An empty file adds nothing.
A message with no recipient and nothing but the signature in its body is cancelled, not sent.
A symbolic link to a signature file is followed; a device or a pipe is ignored.

Search

A search can cover one folder or every folder of the account:

Free words must all match, each one in the sender, the subject or the body.
"a phrase" stays together, also after a prefix.
from: and subject: filter the sender and the subject. Prefixes are case-insensitive, and repeated prefixes join their values with a space.
Parsing never fails: an unterminated quote runs to the end, and a prefix without a value is ignored.

The search goes to the server, which holds every message. When the server cannot be reached, tmail answers from the local cache instead, and the result may be incomplete (see Searching the cache).

Local cache

tmail keeps a local copy of each account’s mail in a SQLite database, so that listing a folder is fast and mail already read can be read without a network. It stores the folder list, the headers of the messages, and the whole text of every message tmail reads. Writes are never served from it, and a search is served from it only when the server cannot be reached. Whenever the cache fails or is in doubt, the server answers. The details, including what is on disk, are in Mail cache.

Address book

tmail keeps a small address book in ~/.config/tildemail/state/address-book.db (readable by you only) to complete the To: of a new message. An entry is an address with a name, where it came from (Mail or Chat), how many times it was seen and when it was last seen. It is filled from two places:

The mail cache. Whenever messages are stored in the cache (a folder is listed, a new message arrives, cache.download), the senders now in the cache replace what the book knew from that account, so rebuilding the cache never counts anyone twice. The sent folder, the trash and the folders called Junk or Spam are left out, and so are the addresses of the account and its identities.
The messages you send. Once the server accepted a message, its To and Cc recipients are added.

A failing address book never fails a listing or a send. Two limits: the cache keeps the sender only, not To and Cc, and a Maildir account has no sender in its index, so a Maildir account teaches the book only through the messages you send. cache clear does not empty the book, and neither does removing an account: its entries stay.

Lua scripting

Lua scripts in ~/.config/tildemail/scripts/, starting from init.lua, run in a sandbox that only sees what tmail exposes. They can:

react to new mail and to other events, such as a sent message or a lost connection (events.on);
move and flag messages (mail);
show desktop notifications (sys.notify) and write to the log (log);
check that the keyring holds a usable key (gpg.has_key), look public keys up for an address (gpg.search_key) and import one by its fingerprint (gpg.import_key, which a script can call on its own and which leaves the key untrusted);
bind keys (keys.bind);
run work in the background through the job scheduler (jobs);
declare rules that tag, flag, mark read or move messages (rules, see Rules).

The engine writes definitions.lua and .luarc.json into the scripts directory, so an editor with the Lua language server completes the API, and it can reload the scripts without restarting. See Scripting with Lua.

Background jobs

IJobSchedulerService runs and lists background work of three kinds:

Queued jobs run once, in parallel, on a fixed number of workers ([jobs] workers). A queued job can report how far it is (IProgress<string>).
Recurring jobs run every N minutes (up to a little under 50 days) and never overlap with themselves: a tick that finds the previous run still pending or running is skipped.
Tracked jobs are run by the caller on its own thread; the scheduler only lists them, from Track(name) until the returned handle is disposed. They take no worker and wait for none. The terminal interface tracks a search this way.

GetJobs() lists the active jobs with their kind, state, interval, last and next run, run count, last error and progress. The Changed event fires whenever that list or a progress changes; the terminal interface listens to it to keep its status line and its :jobs window right. A job that fails is logged and nothing is retried; a recurring job simply runs again at its next tick.

Stopping the scheduler cancels the queued jobs and the token of the running ones, and waits stop_timeout_seconds for them. tmail watch and the terminal interface start the scheduler and stop it on exit; the other commands do not start it. Lua jobs are described in Jobs from Lua.

New-mail watcher

NewMailWatcher runs one watch per account and watched folder (watch, default ["INBOX"]). tmail watch starts it for every account that lists a folder; an account with watch = [] is not watched (see tmail watch).

Baseline. The watch takes its baseline from the server (the folder’s next UID), so mail that arrived while tmail was closed is not reported as new.
Waiting. It waits with IMAP IDLE, renewed every 9 minutes, or polls every 60 seconds on a server without IDLE.
Failures. After a failure it reconnects after 2 seconds, doubling the wait up to 5 minutes.
Reporting. Each new message is reported once with its headers, which puts it in the cache. Every handler (notifications, scripts) runs even when another one failed.
UIDVALIDITY changes. When a folder’s UIDVALIDITY changes during a watch, the messages the server received after the last known point are reported, at most 20 of them: notifying a whole rebuilt mailbox is worse than missing a few.