feature request: fetch missing keys in the background

[Antoine Beaupré]

Hi!

Because I have a rather large keyring, fetching new keys or refreshing
old ones takes a long time. When I click on the red red button saying
"Unknown key ID [...] or unsupported algorithm", Emacs freezes for a
good 90 seconds. Considering I use that editor^Wvirtual machine for
nearly everything, it's quite annoying and sends me off browsing the web
in Firefox forever. I end up having a horde of shaven yaks and then
finally remember that I had that key update, by which time I had
forgotten what I was doing reading email in the first place, let alone
what I was doing *before* opening my inbox...

So. It would be great if Notmuch would run those key updates
asynchronously. I am not sure how that would work: as Bremner said on
IRC, it might make it difficult to update the button automatically. But
I don't mind that: I can refresh the page myself. He suggested running
things in the background when clicking with a prefix (C-u?) but I would
argue that freezing Emacs is just a no-no in general.

I couldn't find directly what function was called behind that button:
"C-h k" just says it's, obviously, `push-button'. Bremner says it might
be `notmuch-crypto-sigstatus-good-callback' which looks reasonable. I'm
only a junior elisp programmer, but it seems to me the
`notmuch-show-refresh-view' call there could be an asynchronous callback
to an async `make-process' call, as opposed to `call-process', which is
synchronous.

But I'd like others to chime in here: is this something that would be
accepted? Would the above work?

A.

[David Edmondson]

Post by Antoine Beaupré
Hi!
Because I have a rather large keyring, fetching new keys or refreshing
old ones takes a long time. When I click on the red red button saying
"Unknown key ID [...] or unsupported algorithm", Emacs freezes for a
good 90 seconds. Considering I use that editor^Wvirtual machine for
nearly everything, it's quite annoying and sends me off browsing the web
in Firefox forever. I end up having a horde of shaven yaks and then
finally remember that I had that key update, by which time I had
forgotten what I was doing reading email in the first place, let alone
what I was doing *before* opening my inbox...
So. It would be great if Notmuch would run those key updates
asynchronously. I am not sure how that would work: as Bremner said on
IRC, it might make it difficult to update the button automatically. But
I don't mind that: I can refresh the page myself. He suggested running
things in the background when clicking with a prefix (C-u?) but I would
argue that freezing Emacs is just a no-no in general.
"C-h k" just says it's, obviously, `push-button'. Bremner says it might
be `notmuch-crypto-sigstatus-good-callback' which looks reasonable. I'm
only a junior elisp programmer, but it seems to me the
`notmuch-show-refresh-view' call there could be an asynchronous callback
to an async `make-process' call, as opposed to `call-process', which is
synchronous.
But I'd like others to chime in here: is this something that would be
accepted? Would the above work?

I'm puzzled about the currently expected behaviour based on your
question.

I have “notmuch-crypto-process-mime” set to t.

I *don't* have “keyserver-options auto-key-retrieve” set in my gpg.conf,
so I don't expect to get any automatic retrieval of keys (it was
annoyingly slow sometimes).

When I read a message with a signature using a key that I don't have, I
get the expected “Unknown key ID ...” button. If I click on that button
a new window appears showing me the details of that key (with a slight
delay).

The referenced key is *not* imported into my keyring, so updating the
view of the message does nothing to change the display of the button -
it still says “Unknown key ID ...”.

Enabling “auto-key-retrieve” changes this behaviour, as you would expect
(I see a delay opening the message while the key is retrieved, and then
the button shows it as good).

(Caveat: I'm using “remote-notmuch”, so emacs and notmuch/gpg are on
different machines, but I don't know that this changes any of the
above.)

dme.

[David Edmondson]

Post by David Edmondson
(Caveat: I'm using “remote-notmuch”, so emacs and notmuch/gpg are on
different machines, but I don't know that this changes any of the
above.)

Ah, of course it does - “gpg --recv-keys ...” runs on the same machine
as emacs, not on the server in this case. Ick.

dme.

[Antoine Beaupré]

On 2018-09-03 17:46:00, David Edmondson wrote:

[...]

Post by David Edmondson
I'm puzzled about the currently expected behaviour based on your
question.

[...]

Post by David Edmondson
Enabling “auto-key-retrieve” changes this behaviour, as you would expect
(I see a delay opening the message while the key is retrieved, and then
the button shows it as good).

The current behavior is somewhat expected: I'm happy notmuch fetches the
keys on click. The problem is it takes forever (that's GPG's fault) and
that Emacs freezes while it happens (that's Notmuch's fault).

Post by David Edmondson
(Caveat: I'm using “remote-notmuch”, so emacs and notmuch/gpg are on
different machines, but I don't know that this changes any of the
above.)

I'm not sure either.

A.

[David Edmondson]

How about this patch?

You'll need to set “notmuch-crypto-get-keys-asynchronously” to “t” to
see any benefit.

[David Edmondson]

Post by David Edmondson
How about this patch?
You'll need to set “notmuch-crypto-get-keys-asynchronously” to “t” to
see any benefit.

Oops, and delete the old version of
“notmuch-crypto-sigstatus-error-callback” from just below the new one.

Post by David Edmondson
diff --git a/emacs/notmuch-crypto.el b/emacs/notmuch-crypto.el
index fc2b5301..97396ba0 100644
--- a/emacs/notmuch-crypto.el
+++ b/emacs/notmuch-crypto.el
@@ -43,6 +43,11 @@ mode."
:package-version '(notmuch . "0.25")
:group 'notmuch-crypto)
+(defcustom notmuch-crypto-get-keys-asynchronously nil
+ "Retrieve gpg keys asynchronously."
+ :type 'boolean
+ :group 'notmuch-crypto)
+
(defface notmuch-crypto-part-header
'((((class color)
(background dark))
@@ -145,6 +150,36 @@ mode."
(call-process epg-gpg-program nil t t "--list-keys" fingerprint))
(recenter -1))))
+(defun notmuch-crypto--async-key-sentinel (process event)
+ (let ((status (process-status process))
+ (exit-status (process-exit-status process)))
+ (when (memq status '(exit signal))
+ (message "Asynchronous GPG key retrieval %s."
+ (if (= exit-status 0)
+ "completed"
+ "failed")))))
+
+(defun notmuch-crypto-sigstatus-error-callback (button)
+ (let* ((sigstatus (button-get button :notmuch-sigstatus))
+ (keyid (concat "0x" (plist-get sigstatus :keyid)))
+ (buffer (get-buffer-create "*notmuch-crypto-gpg-out*")))
+ (if notmuch-crypto-get-keys-asynchronously
+ (progn
+ (message "Getting the GPG key %s asynchronously..." keyid)
+ (make-process :name "notmuch GPG key retrieval"
+ :buffer buffer
+ :command (list epg-gpg-program "--recv-keys" keyid)
+ :sentinel #'notmuch-crypto--async-key-sentinel))
+ (let ((window (display-buffer buffer t nil)))
+ (with-selected-window window
+ (with-current-buffer buffer
+ (goto-char (point-max))
+ (call-process epg-gpg-program nil t t "--recv-keys" keyid)
+ (insert "\n")
+ (call-process epg-gpg-program nil t t "--list-keys" keyid))
+ (recenter -1))
+ (notmuch-show-refresh-view)))))
+
(defun notmuch-crypto-sigstatus-error-callback (button)
(let* ((sigstatus (button-get button :notmuch-sigstatus))
(keyid (concat "0x" (plist-get sigstatus :keyid)))
dme.
--
For a long time I felt, without style and grace.

dme.

[Antoine Beaupré]

Post by David Edmondson
How about this patch?
You'll need to set “notmuch-crypto-get-keys-asynchronously” to “t” to
see any benefit.

This is great!

As you said on IRC, the patch is more likely:

[David Edmondson]

Post by Antoine Beaupré

Post by David Edmondson
How about this patch?
You'll need to set “notmuch-crypto-get-keys-asynchronously” to “t” to
see any benefit.

This is great!

Another version, which attempts to redisplay the buffer if it thinks
that it's safe and sensible to do so.

This version *defaults* to async behaviour, as I think that it's safe to
do so.

[Antoine Beaupré]

That's great! It's basically what I was looking for, but unfortunately
it does not work here. I am not sure why, but it never actually updates
the widget, even if I do not move the point.

A.

[David Edmondson]

Post by Antoine Beaupré
That's great! It's basically what I was looking for, but unfortunately
it does not work here. I am not sure why, but it never actually updates
the widget, even if I do not move the point.

That's odd. Forcing the connection type to 'pipe improves things on
macOS, but I figured that you were using Linux.

Could you test the patch that I posted and see if it works better?

dme.

[Antoine Beaupré]

Post by David Edmondson

Post by Antoine Beaupré
That's great! It's basically what I was looking for, but unfortunately
it does not work here. I am not sure why, but it never actually updates
the widget, even if I do not move the point.

That's odd. Forcing the connection type to 'pipe improves things on
macOS, but I figured that you were using Linux.
Could you test the patch that I posted and see if it works better?

Err... That *was* with the new patch in ***@disaster-area.hh.sledj.net

Did i miss something?

[David Edmondson]

Post by Antoine Beaupré

Post by David Edmondson

Post by Antoine Beaupré
That's great! It's basically what I was looking for, but unfortunately
it does not work here. I am not sure why, but it never actually updates
the widget, even if I do not move the point.

That's odd. Forcing the connection type to 'pipe improves things on
macOS, but I figured that you were using Linux.
Could you test the patch that I posted and see if it works better?

Did i miss something?

I just sent another set.

dme.

[Antoine Beaupré]

Post by David Edmondson

Post by Antoine Beaupré

Post by David Edmondson

Post by Antoine Beaupré
That's great! It's basically what I was looking for, but unfortunately
it does not work here. I am not sure why, but it never actually updates
the widget, even if I do not move the point.

That's odd. Forcing the connection type to 'pipe improves things on
macOS, but I figured that you were using Linux.
Could you test the patch that I posted and see if it works better?

Did i miss something?

I just sent another set.

Ah yes, in another thread.. I'm not on the list so I missed that,
sorry. :)

So I tried that and it works, if I really don't touch anything. It's
certainly an improvement over the previous behavior. Once the "message"
is gone (ie. as soon as anything else is done in Emacs), it's a little
hard to figure out what is happening without going through the status
buffer.

Maybe that's fine, but it would be nice if the button would immediately
update with that status. For example, the text could change to
"... updating" or something. Of course, we're back to the same place
that it can't be updated without shuffling the buffer around if the
point moved, but at least the UI would "remember" there was some change
in the status (it's not just "Failure" but "We're working on it").

I also quite like the changes to the *..-gpg-out* buffer: it's a little
more readable than it was before.

Anyways, as is, I think it's a great improvement already and would
recommend this for merging.

A.

PS: As it turns out, I had a lot of trouble importing those patches. I
tried first to download the 2018 archive from here:

https://notmuchmail.org/pipermail/notmuch/2018.txt.gz

I tried converting this to a maildir and move it to ~/Maildir/.notmuch/
with spwhitton's mailscripts:

mbox2maildir notmuch-2018.txt notmuch-2018
mdmv notmuch-2018/*/* ~/Maildir/.notmuch/
notmuch new

But the last command did not find any new messages, which left me quite
puzzled. The same thing occured with the full archive at:

https://notmuchmail.org/archives/notmuch.mbox

So I am not sure what I was doing wrong there... I ended up using mutt
(bleeergh! ;)) to extract the patchset...

[David Bremner]

Post by Antoine Beaupré
https://notmuchmail.org/pipermail/notmuch/2018.txt.gz

The short answer is that pipermail does not produce correct mboxes. I'm
not sure all that is wrong with them, but I noticed it mangles the
'From ' and the 'From: ' lines. It is probably some "anti-spam" measure
that was effective in the early 1980s.

The other mbox (full archive) should be correct.

[David Edmondson]

Post by Antoine Beaupré
So I tried that and it works, if I really don't touch anything. It's
certainly an improvement over the previous behavior. Once the "message"
is gone (ie. as soon as anything else is done in Emacs), it's a little
hard to figure out what is happening without going through the status
buffer.
Maybe that's fine, but it would be nice if the button would immediately
update with that status. For example, the text could change to
"... updating" or something. Of course, we're back to the same place
that it can't be updated without shuffling the buffer around if the
point moved, but at least the UI would "remember" there was some change
in the status (it's not just "Failure" but "We're working on it").

v2 of the patch does this.

dme.