An ag(e)ing hacker, Luca Saiu's blog
2023-08-07 03:05 (last update: 2024-01-11 12:15) Luca Saiu

p≡p-mail-tool: easy privacy for email with existing Mail User Agents (p≡p is working with Gnus!)

During the last month and a half I have unfortunately mostly disappeared from GNU, having been busy and focused writing p≡p-mail-tool, a new work project I have let overflow into my personal time as a beautiful little hack in which I believe. p≡p-mail-tool is of course free software.


Freedom of speech and privacy are more and more threatened by governments and hostile corporations working against the public interest. In this season of death of liberty the minimum we can do to respond is making surveillance more difficult, by providing the general public with easy tools to use for private communication.

That is the p≡p Project’s mission. Current p≡p software is based on email, which the younger generation no longer use as its primary communication medium — to the chagrin of greybeards like me who still like to take the time to compose a well-crafted message.

The p≡p project is also working on something new and more chat-like which will make make casual users feel at home; the new software (also, of course, free software) will reuse most of the current existing p≡p code base; email was always intended to be just a “transport” among others1, to carry our messages with automatic encryption and verification.
However, the new application will be some other day’s topic for me or for somebody else.

Today I am writing about email, embracing my grey-bearded grumpy self with its futile taste for writing with care and cultivating an impractically sentimental attachment to language. Email is my favourite communication mean, maybe after Usenet. As a hacker I must not be the only one.

p≡p for email before p≡p-mail-tool: p≡p for Thunderbird

Before p≡p-mail-tool the p≡p software for email working on GNU/Linux systems consisted in a Thunderbird addon. While the software is very easy to use (the usual Thunderbird, plus essentially only the coloured “privacy bar” showing the privacy status of each incoming or outgoing message), the addon hides some complexity: inserting encryption and verification everywhere interferes with the application logic in a deep way, to a degree that was not anticipated in the Thunderbird extension API. The addon itself is a set of JavaScript files communicating over HTTP with a local server holding the local persistent state and performing calling the cryptographic primitives.

Even with a few minor warts one cannot really complain of p≡p for Thunderbird: it works and is in fact easy to use.

One uses Thunderbird with the p≡p addon exactly like an ordinary Thunderbird, without losing compatibility with ordinary unencrypted email. If we receive a message from a new communication partner using p≡p, the system automatically imports the partner’s attached public key (TOFU: Trust On First Use). A communication partner not using p≡p and receiving a message from us will get an unencrypted email, with a sender-key attachment which is perfectly safe to ignore.

When a public key for a communication partner is known, it is automatically used2 to encrypt outgoing messages: this is everything a user needs in normal circumstances.

p≡p for Thunderbird’s UI composing an outgoing message with yellow privacy bar
p≡p for Thunderbird, composing a message to a communication partner to whom we can send a protected (encrypted and authenticated) message
p≡p for Thunderbird displaying an incoming message with yellow privacy bar
p≡p for Thunderbird, displaying an incoming message from the same communication partner: again the message is yellow: encrypted and authenticated.

The coloured “privacy bar” shows the privacy status of an incoming or outgoing message: red for “under attack”, yellow for “protected”, green for “trusted”; and without any colour if the message is unprotected.

p≡p for Thunderbird displaying an incoming message with no colour
An unprotected incoming message with no colour, from a non-p≡p communication partner or the first exchange with a p≡p communication partner.

TOFU is vulnerable to a Man-in-the-Middle attack only inside a very narrow window: as long as the very first message between them is not compromised two communication partners can rely on protected (encrypted and authenticated) exchanges. This level of protection is already reasonable for most circumstances, but it is possible to do even better and close the initial attack window as well, as an optional measure:

p≡p for Thunderbird UI displaying trustwords to check
Optionally one can contact the communication partner out-of-band, and check that her trustwords (a function of the fingerprints of both communication partners) match: if the users accept that the trustwords match, messages between them will be shown with a green bar.

For the unusual cases in which one needs to replace a lost or compromised key (typically after a disk crash or device destruction, loss or theft) there are mechanisms to manually reset keys; these are trivial to use, and key management is otherwise completely automatic.

Now, by adding that p≡p is also compatible with standard OpenPGP I should have mentioned most of its features.


Jörg Knobloch, who wrote p≡p for Thunderbird and large parts of Thunderbird itself, is an expert whose work should not be overlooked; p≡p for Thunderbird is an achievement in itself.

I do not dislike Thunderbird: in fact I used it myself (actually first the old “Mozilla suite”, then Thunderbird) for about a decade, between 2002 or 2003 and early 2012. Its message search capabilities are excellent, and its “message filters” my second favourite client automatic classification system after Gnus’s “fancy mail splitting”.

For me the real problem with Thunderbird is that it is not a piece of software I love. It is not a set of functionalities running on top of my favourite editor, keyboard-controllable, extensible to no end, written in Lisp which is itself a thing of beauty. Since I love Gnus and to me email and editing are very important I will not accept a compromise.
Other hackers have preferences different from mine, but usually just as strong. We are demanding, particular: we want power and control.

Even with p≡p for Thunderbird being a sensible choice for the general public many hackers will not want to adopt it. I use p≡p for Thunderbird almost every day for testing the p≡p Engine, but Thunderbird it is not my production MUA.

My proposed solution: p≡p-mail-tool

I want users to be able to switch to p≡p while keeping their MUA, adapted as little as possible; ideally with no changes at all.

A good solution to this problem is to provide a standard interface to an MUA: the new software must be able to use existing standard protocols or formats, for example IMAP and SMTP: the user’s MUA will connect to the local host, possibly on non-standard ports, to send and receive ordinary unencrypted messages. All of the encryption, decryption and verification will happen automatically, out of the MUA’s view.

This turns out to be almost entirely feasible; for the single exception see Displaying the outgoing message rating below.

p≡p-mail-tool downloads incoming messages from the user’s “incoming nodes”: usually remote IMAP or POP servers, but possibly also mbox files, Maildir spools or something similar, decrypting messages as necessary and storing the decrypted versions into a local store. A good candidate for a local store is an IMAP server but a Maildir directory or an mbox file will also do, as long as the user’s MUA can handle the formats3.

When the MUA submits a message it will connect to the submitter side of p≡p-mail-tool over SMTP (or over a sendmail-compatible command-line interface); p≡p-mail-tool will automatically encrypt the message and then deliver it to the appropriate outgoing node: usually a remote SMTP server, but possibly another sendmail, or some spool file.

diagram showing p≡p-mail-tool the user MUA Dovecot as a local store incoming nodes and outgoing nodes
An example using a POP or IMAP incoming nodes, a Dovecot IMAP server as a local source, and SMTP servers as outgoing nodes. The connections inside the black box happen within the local host.

It is quite easy to have p≡p-mail-tool supporting multiple mail accounts, by receiving from multiple incoming nodes and submitting to multiple outgoing nodes: the specific outgoing node for a message can be chosen from the From address, via a regular expression. This account configuration is stored in p≡p-mail-tool’s configuration file; the actual MUA is simpler to handle, since it only needs to know about stores and submitters.

The following excerpt comes from p≡p-mail-tool’s sample configuration file. This imaginary user has three accounts, and

incoming_rules = [
    #  (own-username, own-email-address),
    #  LOCAL-STORE),
    (('pop',   '',    None, 'quirky',  'PASSWORD'),
     ('Mr Quirky', ''),
     ('dovecot-subprocess', '~/.pEp-dovecot', 1143, 'john',    'PASSWORD')),
    (('imaps', '',   None, 'johndoe', 'PASSWORD'),
     ('John Doe',  ''),
     ('dovecot-subprocess', '~/.pEp-dovecot', 1143, 'john',    'PASSWORD')),
    (('imaps', '',            None, 'john',    'PASSWORD'),
     ('John Doe',  ''),
     ('dovecot-subprocess', '~/.pEp-dovecot', 1143, 'john',    'PASSWORD')),

outgoing_rules = [
    # (regexp,
     ('smtps', '',    465,  'johndoe', 'PASSWORD')),
     ('smtp', '',    587,  'johndoe', 'PASSWORD')),
    # The last server always matches, so the regular expression is
    # irrelevant and in fact ignored.
     ('smtp', '', 465, 'johndoe', 'PASSWORD')),

Early on I noticed the need for a fast and reliable local IMAP server, able to hold a high number of messages. Being a happy long-time user I chose Dovecot for my own tests; but soon I had the interesting realisation that I could have p≡p-mail-tool machine-generate the entire Dovecot configuration from p≡p-mail-tool’s configuration and entirely handle Dovecot as a subprocess, its complexity hidden from the user. This feature is used in the example above and I expect it to be frequently used in practice.

The configuration file has more details than shown here (see but p≡p-mail-tool ends up being easy to configure for its power. For example it is trivially possible to start multiple Dovecot instances, all started up automatically by p≡p-mail-tool, as separate sources, presenting as separate mailboxes to the MUA.

Privacy bar and colours with p≡p-mail-tool

If the MUA is unmodified, how can it show the privacy bar for an incoming message?

This turns out to be easy as long as the MUA can display arbitrary headers, and the kind of flexible MUAs we hackers like of course will let us do that. The solution, therefore, is for p≡p-mail-tool to add a non-standard header after decryption, displaying the message rating. The header looks like:

X-pEp-Rating: yellow (reliable 6)

Seen from Gnus this would be:

Gnus screenshot displaying headers

While this privacy display does not look as friendly as the one in Thunderbird above, it does contain the relevant information.

There remains another major unsolved problem: how can we display the colour of an outgoing message while it is being composed, since the message has not passed through p≡p-mail-tool yet?

Displaying the outgoing message rating

First of all, it is necessary to modify the MUA for this. In the case of Gnus we can add some message-mode hook to be called at the right time every time the message is edited, or periodically with a timer4.

The hook to call is an Emacs Lisp function, computing the rating. But where does the function get the information from?

The solution is having a command-line interface in p≡p-mail-tool.

p≡p-mail-tool can be invoked in the command line to compute, among the rest, the rating a message would have if sent; the message is passed through p≡p-mail-tool’s standard input; from Emacs’s point of view the message-mode hook invokes p≡p-mail-tool as a subprocess using the Lisp equivalent of the system libc function.

This help message documents the current command-line interface of p≡p-mail-tool, including the option --outgoing-rating-stdin I am discussing now.

[luca@moore ~/pep-src/pEp-mail-tool]$ ./pEp-mail-tool --help
usage: pEp-mail-tool [-h] [-v] [-s] [--conf FILE] [--no-op] [--wake-up]
                     [--exit] [--ping EMAIL EMAIL]
                     [--incoming-stdin EMAIL] [--outgoing-stdin]
                     [--outgoing-rating-stdin] [--trustwords-language LL]
                     [--trustwords-full EMAIL EMAIL]
                     [--trustwords-partial EMAIL EMAIL] [--trust EMAIL]
                     [--mistrust EMAIL] [--trust-reset EMAIL]
                     [--key-reset EMAIL] [--key-reset-all-own]

A command-line tool for using p≡p for mail without a special client.

  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -s, --server          run as (foreground) server [default: run as
  --conf FILE           use FILE as a configuration file (only useful
                        when running as server); this can be given
                        multiple times. If never given use the hardwired
                        path ~/.pEp-mail-tool
  --no-op               do nothing, exiting with sucess as long as the
                        server is responding. This is intended to check
                        whether the server is up
  --wake-up             wake up every server thread that is currenly
                        waiting. This is useful to force the server to
                        fetch mail immediately, skipping the random wait
                        time for one iteration of the infinite loop
  --exit                kill the server
  --ping EMAIL EMAIL    send a Distribution.Ping message from the own
                        identity having the given first email address to
                        another identity having the second email address
  --incoming-stdin EMAIL
                        handle an incoming message from stdin, to be
                        received by an own identity with the given
  --outgoing-stdin      handle an outgoing message from stdin
                        print the rating of the outgoing message from
                        stdin (without encrypting or sending the message)
  --trustwords-language LL
                        use the given two-letter language code for
                        trustwords, instead of the server default. This
                        only makes sense combined with --trustwords-full
                        or --trustwords-partial
  --trustwords-full EMAIL EMAIL
                        print full trustwords (10 words or 160 bits) for
                        the identities having the two given email
  --trustwords-partial EMAIL EMAIL
                        exactly like --trustwords-full , except that this
                        only prints partial trustwords (5 words or 80
  --trust EMAIL         make the default key of the given identity
  --mistrust EMAIL      make the default key of the given identity
                        mistrusted, and no longer use it as a default key
                        for any identity
  --trust-reset EMAIL   clear the trust bits for the default key of the
                        given identity
  --key-reset EMAIL     reset the default key for the identity with the
                        given address, be it own or non-own
  --key-reset-all-own   reset all own keys

Invoke with a basename containing the substring "sendmail" for an
alternative command-line syntax compatible with (a subset of) sendmail's.

Please report bugs by opening a ticket on
or by contacting the author Luca Saiu <>.

The rapid frequency of p≡p-mail-tool invocations, potentially once per key press, makes this use case performance-critical. In order to make this important feature efficient p≡p-mail-tool is itself divided into a client part and a server part, communicating with each other over a Unix-domain socket.

p≡p-mail-tool --outgoing-rating-stdin will just start up as a client performing a request to the server: the server, already initialised with p≡p-mail-tool’s configuration and with a thread pool ready, will reply quickly, on my machine within 40~50 ms.

As the output of p≡p-mail-tool --help above shows, one an use this command-line interface for other p≡p operations such as checking trustwords, or resetting compromised keys.

Thanks to p≡p-mail-tool’s command-line interface and p≡p-mail-tool’s Emacs Lisp hooks for Gnus we can now have a rating header automatically updated as the user composes a message:

Gnus screenshot composing a message no pmt mode
The X-pEp-Rating header appears automatically and is updated as the user edits the message sender and recipients.

This would already suffice to cover the basic functionality but it took me relatively little effort to do better and use an Emacs overlay to display the header value in its intended colour:

Gnus screenshot composing a message with pmt mode
Now the X-pEp-Rating header value is covered by an overlay making it look nicer — and in actual colour.

Gnus uses message-mode for messages being composed, and article-mode for messages (incoming or outgoing) being read. I reused the overlay code to decorate article-mode buffers as well:

Gnus screenshot displaying headers with pmt mode

This makes Gnus quite pleasant to use with p≡p. It is also possible to customize (in the GNU Emacs sense) some parameters in pEp-mail-tool.el.

Other MUAs

It should not be difficult to adapt other MUAs to p≡p-mail-tool.

I plan to work on Alpine at some point, and should also look into Wanderlust and the other Emacs MUAs, for which I should be able to reuse much of the current code.


When I talked to Volker Birk maybe six months ago about my still very rough design he convinced me to write my tool in Python, because of the availability of Python wrappers for the p≡p primitives: we call that component the p≡p Python Adapter ( Later when I actually started the plan turned out to be correct and using the p≡p Python Adapter, Python’s library and aiosmtpd it turned out to be indeed relatively easy to put together a first working version of p≡p-mail-tool. aiosmtpd remains the one component I do not master completely.

I have never hidden my personal dislike of Python. I used the language mostly guided by my preferences, by avoiding or wrapping the language features I wanted hidden. For example I have always loathed the threading.Thread class, obviously influenced by the terrible Java API without even the excuse of language constraints; and so my threads use a thin function wrapper on top of that class. I do not use inheritance at all, anywhere in the project. The code remains quite simple and understandable. p≡p-mail-tool is after all a very high-level application, quite different from my usual p≡p Engine work, and switching to it for a while has been a refreshing diversion.

After seeing the code already in an advanced state Volker remarked, correctly, that my way of using Python displayed my Lisper’s aesthetics.

More about p≡p-mail-tool

I gave a presentation about p≡p-mail-tool to my p≡p Foundation colleagues in late June, “p≡p-mail-tool: Using p≡p for email with ordinary mail clients — A status report with some personal opinions”. I am publishing a recording with permission.

Notice that this presentation has not been updated to cover the recent p≡p-mail-tool developments, and the software has already become considerably more general and powerful since the time of the presentation. The video should still be suitable as an introduction.

Please also notice that some of the naming conventions in p≡p-mail-tool changed since the presentation.

  • Slides (PDF, 1.4MiB)
  • Video (WebM, 322MiB, 1h18m)
  • Abstract: I have written a new tool for using pEp for email, particularly with traditional clients different from Thunderbird, on Unix systems; it is not complete but already at a point where a random reasonably technical person could use it production. I normally use Gnus as my mail client – Gnus being one of the mail applications running on top of GNU Emacs.
    I wrote a small extension to Gnus for taking advantage of pEp-mail-tool, making the interface quite easy and friendly.
    Other mail programs can be adapted; but even without any adaptation the tool is useful.

I am willing to give an updated presentation over Jitsi, possibly in the evening, if sufficient interest arises.

About the software

p≡p-mail-tool is free software, released under the GNU Affero General Public Licence; written by me, Luca Saiu, for the p≡p Project.

The official p≡p-mail-tool repository is

Trying p≡p-mail-tool

Building the p≡p stack from its source, while not difficult, is a long process because of the multiple repositories involved.

The scripts in make it easier to build the entire pEp stack up to p≡p-mail-tool from its sources, as an ordinary non-root user; the script has been tested on several GNU/Linux distributions. At the time of writing:

  • Debian 11;
  • Fedora 38, “basic functionality”;
  • Ubuntu 22.04.02-deskop, “minimal installation”;
  • Mint 21.2 XFCE.

We are working to also have p≡p-mail-tool officially packaged by distributions such as Debian.


Volker Birk, the founder of the p≡p Project, is a Mutt user who like me had always delayed a personal switch to p≡p in order to keep his favourite MUA. Volker’s feedback has always been particularly encouraging.
Volker provided me with a lot of feedback about p≡p-mail-tool; in order to support his own use cases I understood ways of generalising the utility and making it remarkably more powerful. Volker spent quite some time teaching me and giving me suggestion about notation and syntax, many of which I had to begrudgingly concede to be pragmatically correct, no matter how much I disliked them: p≡p-mail-tool’s configuration file syntax will at some point allow some more idiomatically Pythonic structures as well, as alternatives to my positional nested-sequence Lisp-style notation.
Most of all, I am grateful to Volker for the freedom of design he ultimately left me.

Thanks to Nkls, now working on Mixnet and Onion-Routing support for p≡p, who was the first p≡p-mail-tool user and who helped me to discover and solve many problems.

Thanks to my other p≡p Foundation colleagues: Ariodante, Heck and Sva for their interest and feedback listening to my rehersal presentation; and all the others who gave me positive and encouraging feedback.

[2023-10-19 update: the public repositories and issue trackers have been moved to Codeberg. I updated every affected link in this article without marking each change.]

— Luca Saiu, 2023-08-07 03:05 (last update: 2024-01-11 12:15)

aiosmtpd, bash, command-line, dovecot, emacs, english, free-software, gnu, gnu-linux, gnus, hacking, imap, lisp, minimalism, myself, parentheses, pop, p≡p, privacy, python, script, smtp, software, surveillance, thunderbird, unix

Next post Previous post

You might want to go to the main blog index (Atom feedfeeds for every post: Atom 1.0, RSS 2.0) or to my web site

[my photo]
Luca Saiu

The opinions I express here are my own and do not necessarily reflect the beliefs or policies of my employer or for that matter of anyone else. In case you felt that the public statement of my thoughts threatened your warm sense of security and your emotional stability feel free to leave at any time.
The system does not support user comments and probably never will. Anyway you can contact me if you want to discuss some topic with me. I might update my posts if you provide interesting insights.

You might be interested in my web site

Copyright © 2009, 2011-2014, 2017, 2018, 2021-2024 Luca Saiu
Verbatim copying and redistribution of this entire page are permitted in any medium without royalties, provided this notice is preserved.
This page was generated by
trivialblog. trivialblog is free software, available under the GNU GPL.
Tag icon copyright information is available in this file.



Email will remain supported even in the new application, as one of the transports.


There is no need to even show a key fingerprint to the user: the interface is intentionally designed to be unobtrusive.


Independently from the format and the protocol any store containing unencrypted messages should reside in a crypto container: an encrypted home directory, an encrypted filesystem.


Both alternatives are in fact implemented, and the user can choose between better latency or lower CPU usage.