[PATCH v2 0/4] Retrieve GPG keys asynchronously.

[David Edmondson]

Retrieve GPG keys asynchronously.

v2:
- Update the label on buttons when they are used to request
asyncronous key retrieval.
- Always update the buffer when key retrieval completes, even if the
retrieval failed, so that the label is properly re-generated.


David Edmondson (4):
emacs: Asynchronous retrieval of GPG keys
emacs: Minor refactoring of crypto code
emacs: Add notmuch-crypto-gpg-program and use it
emacs: Improve the reporting of key activity

emacs/notmuch-crypto.el | 191 +++++++++++++++++++++++++++++++++---------------
1 file changed, 134 insertions(+), 57 deletions(-)

[David Edmondson]

Allow the user to specify the gpg program to use when retriving keys,
etc., defaulting to the value of `epg-gpg-program'.
---
emacs/notmuch-crypto.el | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/emacs/notmuch-crypto.el b/emacs/notmuch-crypto.el
index 81c0eca0..760461e8 100644
--- a/emacs/notmuch-crypto.el
+++ b/emacs/notmuch-crypto.el
@@ -48,6 +48,11 @@ mode."
:type 'boolean
:group 'notmuch-crypto)

+(defcustom notmuch-crypto-gpg-program epg-gpg-program
+ "The gpg executable."
+ :type 'string
+ :group 'notmuch-crypto)
+
(defface notmuch-crypto-part-header
'((((class color)
(background dark))
@@ -149,7 +154,7 @@ by user FROM."
(with-selected-window window
(with-current-buffer buffer
(goto-char (point-max))
- (call-process epg-gpg-program nil t t "--list-keys" fingerprint))
+ (call-process notmuch-crypto-gpg-program nil t t "--list-keys" fingerprint))
(recenter -1))))

(declare-function notmuch-show-refresh-view "notmuch-show" (&optional reset-state))
@@ -202,7 +207,7 @@ corresponding key when the status button is pressed."
button (format "Retrieving key %s asynchronously..." keyid))
(let ((p (make-process :name "notmuch GPG key retrieval"
:buffer buffer
- :command (list epg-gpg-program "--recv-keys" keyid)
+ :command (list notmuch-crypto-gpg-program "--recv-keys" keyid)
:connection-type 'pipe
:sentinel #'notmuch-crypto--async-key-sentinel
;; Create the process stopped so that
@@ -214,13 +219,14 @@ corresponding key when the status button is pressed."
(process-put p :notmuch-show-point (point))
(message "Getting the GPG key %s asynchronously..." keyid)
(continue-process p)))
+
(let ((window (display-buffer buffer)))
(with-selected-window window
(with-current-buffer buffer
(goto-char (point-max))
- (call-process epg-gpg-program nil t t "--recv-keys" keyid)
+ (call-process notmuch-crypto-gpg-program nil t t "--recv-keys" keyid)
(insert "\n")
- (call-process epg-gpg-program nil t t "--list-keys" keyid))
+ (call-process notmuch-crypto-gpg-program nil t t "--list-keys" keyid))
(recenter -1))
(notmuch-show-refresh-view)))))

[David Bremner]

Post by David Edmondson
Allow the user to specify the gpg program to use when retriving keys,
etc., defaulting to the value of `epg-gpg-program'.

typo in retriving.

More interestingly, would you mind documenting the use case in the
commit message? I'm unclear on a couple points

- why one needs a different gpg for key retrieval
- whether the new variable is just for key retrieval, or as the name
notmuch-crypto-gpg-program suggests, all crypto operations from notmuch.

[David Edmondson]

Post by David Bremner

Post by David Edmondson
Allow the user to specify the gpg program to use when retriving keys,
etc., defaulting to the value of `epg-gpg-program'.

typo in retriving.

Will fix.

Post by David Bremner
More interestingly, would you mind documenting the use case in the
commit message? I'm unclear on a couple points
- why one needs a different gpg for key retrieval

I want to use “gpg” for most operations that deal with encryption within
emacs, but when using notmuch remotely over an ssh connection (that is
the UI and database on different machines) I need to update the keyring
on the database machine so that notmuch itself sees the key, not the
keyring on my local machine (arguably I need to update both, but hey),
so use “ssh <server> gpg” instead.

Post by David Bremner
- whether the new variable is just for key retrieval, or as the name
notmuch-crypto-gpg-program suggests, all crypto operations from notmuch.

Just for key retrieval.

dme.

[David Bremner]

Post by David Edmondson

Post by David Bremner

Post by David Edmondson
Allow the user to specify the gpg program to use when retriving keys,
etc., defaulting to the value of `epg-gpg-program'.

typo in retriving.

Will fix.

Post by David Bremner
More interestingly, would you mind documenting the use case in the
commit message? I'm unclear on a couple points
- why one needs a different gpg for key retrieval

I want to use “gpg” for most operations that deal with encryption within
emacs, but when using notmuch remotely over an ssh connection (that is
the UI and database on different machines) I need to update the keyring
on the database machine so that notmuch itself sees the key, not the
keyring on my local machine (arguably I need to update both, but hey),
so use “ssh <server> gpg” instead.

Post by David Bremner
- whether the new variable is just for key retrieval, or as the name
notmuch-crypto-gpg-program suggests, all crypto operations from notmuch.

Just for key retrieval.

I guess renaming the variable and documenting it as used only for key
retrieval would help both questions here.

[David Edmondson]

Rather than blocking emacs while gpg does its' thing, by default run
key retrieval asynchronously, possibly updating the display of the
message on successful completion.
---
emacs/notmuch-crypto.el | 79 ++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 69 insertions(+), 10 deletions(-)

diff --git a/emacs/notmuch-crypto.el b/emacs/notmuch-crypto.el
index fc2b5301..ddb447a6 100644
--- a/emacs/notmuch-crypto.el
+++ b/emacs/notmuch-crypto.el
@@ -43,6 +43,11 @@ mode."
:package-version '(notmuch . "0.25")
:group 'notmuch-crypto)

+(defcustom notmuch-crypto-get-keys-asynchronously t
+ "Retrieve gpg keys asynchronously."
+ :type 'boolean
+ :group 'notmuch-crypto)
+
(defface notmuch-crypto-part-header
'((((class color)
(background dark))
@@ -145,19 +150,73 @@ mode."
(call-process epg-gpg-program nil t t "--list-keys" fingerprint))
(recenter -1))))

+(defun notmuch-crypto--async-key-sentinel (process event)
+ "When the user asks for a GPG key to be retrieved
+asynchronously, handle completion of that task."
+ (let ((status (process-status process))
+ (exit-status (process-exit-status process))
+ (keyid (process-get process :gpg-key-id)))
+ (when (memq status '(exit signal))
+ (message "Getting the GPG key %s asynchronously...%s."
+ keyid
+ (if (= exit-status 0)
+ "completed"
+ "failed"))
+ ;; If the original buffer is still alive and point didn't move
+ ;; (i.e. the user didn't move on or away), refresh the buffer to
+ ;; show the updated signature status.
+ (let ((show-buffer (process-get process :notmuch-show-buffer))
+ (show-point (process-get process :notmuch-show-point)))
+ (when (and (bufferp show-buffer)
+ (buffer-live-p show-buffer)
+ (= show-point
+ (with-current-buffer show-buffer
+ (point))))
+ (with-current-buffer show-buffer
+ (notmuch-show-refresh-view)))))))
+
+(defun notmuch-crypto--set-button-label (button label)
+ "Set the text displayed in BUTTON to LABEL."
+ (save-excursion
+ (let ((inhibit-read-only t))
+ ;; This knows rather too much about how we typically format
+ ;; buttons.
+ (goto-char (button-start button))
+ (forward-char 2)
+ (delete-region (point) (- (button-end button) 2))
+ (insert label))))
+
(defun notmuch-crypto-sigstatus-error-callback (button)
(let* ((sigstatus (button-get button :notmuch-sigstatus))
(keyid (concat "0x" (plist-get sigstatus :keyid)))
- (buffer (get-buffer-create "*notmuch-crypto-gpg-out*"))
- (window (display-buffer buffer t nil)))
- (with-selected-window window
- (with-current-buffer buffer
- (goto-char (point-max))
- (call-process epg-gpg-program nil t t "--recv-keys" keyid)
- (insert "\n")
- (call-process epg-gpg-program nil t t "--list-keys" keyid))
- (recenter -1))
- (notmuch-show-refresh-view)))
+ (buffer (get-buffer-create "*notmuch-crypto-gpg-out*")))
+ (if notmuch-crypto-get-keys-asynchronously
+ (progn
+ (notmuch-crypto--set-button-label
+ button (format "Retrieving key %s asynchronously..." keyid))
+ (let ((p (make-process :name "notmuch GPG key retrieval"
+ :buffer buffer
+ :command (list epg-gpg-program "--recv-keys" keyid)
+ :connection-type 'pipe
+ :sentinel #'notmuch-crypto--async-key-sentinel
+ ;; Create the process stopped so that
+ ;; we have time to store the key id,
+ ;; etc. on it.
+ :stop t)))
+ (process-put p :gpg-key-id keyid)
+ (process-put p :notmuch-show-buffer (current-buffer))
+ (process-put p :notmuch-show-point (point))
+ (message "Getting the GPG key %s asynchronously..." keyid)
+ (continue-process p)))
+ (let ((window (display-buffer buffer t nil)))
+ (with-selected-window window
+ (with-current-buffer buffer
+ (goto-char (point-max))
+ (call-process epg-gpg-program nil t t "--recv-keys" keyid)
+ (insert "\n")
+ (call-process epg-gpg-program nil t t "--list-keys" keyid))
+ (recenter -1))
+ (notmuch-show-refresh-view)))))

(defun notmuch-crypto-insert-encstatus-button (encstatus)
(let* ((status (plist-get encstatus :status))

[David Bremner]

Post by David Edmondson
+(defun notmuch-crypto--async-key-sentinel (process event)
+ "When the user asks for a GPG key to be retrieved
+asynchronously, handle completion of that task."
+ (let ((status (process-status process))
+ (exit-status (process-exit-status process))
+ (keyid (process-get process :gpg-key-id)))
+ (when (memq status '(exit signal))
+ (message "Getting the GPG key %s asynchronously...%s."
+ keyid
+ (if (= exit-status 0)
+ "completed"
+ "failed"))
+ ;; If the original buffer is still alive and point didn't move
+ ;; (i.e. the user didn't move on or away), refresh the buffer to
+ ;; show the updated signature status.

This is a pretty conservative condition for the update. I can live with
the tradeoff to get non-blocking updates, but I think it will surprise
people used to the button updating to have it not do so when they move
the point away. So I guess this behavior should be documented in a more
user visible way?

[David Edmondson]

Post by David Bremner

Post by David Edmondson
+(defun notmuch-crypto--async-key-sentinel (process event)
+ "When the user asks for a GPG key to be retrieved
+asynchronously, handle completion of that task."
+ (let ((status (process-status process))
+ (exit-status (process-exit-status process))
+ (keyid (process-get process :gpg-key-id)))
+ (when (memq status '(exit signal))
+ (message "Getting the GPG key %s asynchronously...%s."
+ keyid
+ (if (= exit-status 0)
+ "completed"
+ "failed"))
+ ;; If the original buffer is still alive and point didn't move
+ ;; (i.e. the user didn't move on or away), refresh the buffer to
+ ;; show the updated signature status.

This is a pretty conservative condition for the update. I can live with
the tradeoff to get non-blocking updates, but I think it will surprise
people used to the button updating to have it not do so when they move
the point away. So I guess this behavior should be documented in a more
user visible way?

The condition is conservative because the refresh redraws the whole
buffer. If it just updated the button directly (which I've been fiddling
with, but don't have working well yet) then we might relax the
condition.

Where would we document the behaviour?

dme.

[David Bremner]

Post by David Edmondson
The condition is conservative because the refresh redraws the whole
buffer. If it just updated the button directly (which I've been fiddling
with, but don't have working well yet) then we might relax the
condition.
Where would we document the behaviour?

In the appropriate docstring?

[David Edmondson]

---
emacs/notmuch-crypto.el | 96 +++++++++++++++++++++++++------------------------
1 file changed, 50 insertions(+), 46 deletions(-)

diff --git a/emacs/notmuch-crypto.el b/emacs/notmuch-crypto.el
index ddb447a6..81c0eca0 100644
--- a/emacs/notmuch-crypto.el
+++ b/emacs/notmuch-crypto.el
@@ -96,36 +96,40 @@ mode."
:supertype 'notmuch-button-type)

(defun notmuch-crypto-insert-sigstatus-button (sigstatus from)
+ "Insert a button describing the signature status SIGSTATUS sent
+by user FROM."
(let* ((status (plist-get sigstatus :status))
- (help-msg nil)
- (label "Signature not processed")
(face 'notmuch-crypto-signature-unknown)
- (button-action (lambda (button) (message (button-get button 'help-echo)))))
+ (button-action (lambda (button) (message (button-get button 'help-echo))))
+ (keyid (concat "0x" (plist-get sigstatus :keyid)))
+ label help-msg)
(cond
((string= status "good")
- (let ((fingerprint (concat "0x" (plist-get sigstatus :fingerprint))))
- ;; if userid present, userid has full or greater validity
- (if (plist-member sigstatus :userid)
- (let ((userid (plist-get sigstatus :userid)))
- (setq label (concat "Good signature by: " userid))
- (setq face 'notmuch-crypto-signature-good))
- (progn
- (setq label (concat "Good signature by key: " fingerprint))
- (setq face 'notmuch-crypto-signature-good-key)))
- (setq button-action 'notmuch-crypto-sigstatus-good-callback)
- (setq help-msg (concat "Click to list key ID 0x" fingerprint "."))))
+ (let ((fingerprint (concat "0x" (plist-get sigstatus :fingerprint)))
+ (userid (plist-get sigstatus :userid)))
+ ;; If userid is present it has full or greater validity.
+ (if userid
+ (setq label (concat "Good signature by: " userid)
+ face 'notmuch-crypto-signature-good)
+ (setq label (concat "Good signature by key: " fingerprint)
+ face 'notmuch-crypto-signature-good-key))
+ (setq button-action 'notmuch-crypto-sigstatus-good-callback
+ help-msg (concat "Click to list key ID 0x" fingerprint "."))))
+
((string= status "error")
- (let ((keyid (concat "0x" (plist-get sigstatus :keyid))))
- (setq label (concat "Unknown key ID " keyid " or unsupported algorithm"))
- (setq button-action 'notmuch-crypto-sigstatus-error-callback)
- (setq help-msg (concat "Click to retrieve key ID " keyid " from keyserver and redisplay."))))
+ (setq label (concat "Unknown key ID " keyid " or unsupported algorithm")
+ button-action 'notmuch-crypto-sigstatus-error-callback
+ help-msg (concat "Click to retrieve key ID " keyid
+ " from keyserver and redisplay.")))
+
((string= status "bad")
- (let ((keyid (concat "0x" (plist-get sigstatus :keyid))))
- (setq label (concat "Bad signature (claimed key ID " keyid ")"))
- (setq face 'notmuch-crypto-signature-bad)))
+ (setq label (concat "Bad signature (claimed key ID " keyid ")")
+ face 'notmuch-crypto-signature-bad))
+
(t
(setq label (concat "Unknown signature status"
(if status (concat ": " status))))))
+
(insert-button
(concat "[ " label " ]")
:type 'notmuch-crypto-status-button-type
@@ -134,22 +138,22 @@ mode."
'mouse-face face
'action button-action
:notmuch-sigstatus sigstatus
- :notmuch-from from)
- (insert "\n")))
-
-(declare-function notmuch-show-refresh-view "notmuch-show" (&optional reset-state))
+ :notmuch-from from))
+ (insert "\n"))

(defun notmuch-crypto-sigstatus-good-callback (button)
(let* ((sigstatus (button-get button :notmuch-sigstatus))
(fingerprint (concat "0x" (plist-get sigstatus :fingerprint)))
(buffer (get-buffer-create "*notmuch-crypto-gpg-out*"))
- (window (display-buffer buffer t nil)))
+ (window (display-buffer buffer)))
(with-selected-window window
(with-current-buffer buffer
(goto-char (point-max))
(call-process epg-gpg-program nil t t "--list-keys" fingerprint))
(recenter -1))))

+(declare-function notmuch-show-refresh-view "notmuch-show" (&optional reset-state))
+
(defun notmuch-crypto--async-key-sentinel (process event)
"When the user asks for a GPG key to be retrieved
asynchronously, handle completion of that task."
@@ -187,6 +191,8 @@ asynchronously, handle completion of that task."
(insert label))))

(defun notmuch-crypto-sigstatus-error-callback (button)
+ "When signature validation has failed, try to retrieve the
+corresponding key when the status button is pressed."
(let* ((sigstatus (button-get button :notmuch-sigstatus))
(keyid (concat "0x" (plist-get sigstatus :keyid)))
(buffer (get-buffer-create "*notmuch-crypto-gpg-out*")))
@@ -208,7 +214,7 @@ asynchronously, handle completion of that task."
(process-put p :notmuch-show-point (point))
(message "Getting the GPG key %s asynchronously..." keyid)
(continue-process p)))
- (let ((window (display-buffer buffer t nil)))
+ (let ((window (display-buffer buffer)))
(with-selected-window window
(with-current-buffer buffer
(goto-char (point-max))
@@ -219,25 +225,23 @@ asynchronously, handle completion of that task."
(notmuch-show-refresh-view)))))

(defun notmuch-crypto-insert-encstatus-button (encstatus)
- (let* ((status (plist-get encstatus :status))
- (help-msg nil)
- (label "Decryption not attempted")
- (face 'notmuch-crypto-decryption))
- (cond
- ((string= status "good")
- (setq label "Decryption successful"))
- ((string= status "bad")
- (setq label "Decryption error"))
- (t
- (setq label (concat "Unknown encryption status"
- (if status (concat ": " status))))))
- (insert-button
- (concat "[ " label " ]")
- :type 'notmuch-crypto-status-button-type
- 'help-echo help-msg
- 'face face
- 'mouse-face face)
- (insert "\n")))
+ "Insert a button describing the encryption status ENCSTATUS."
+ (insert-button
+ (concat "[ "
+ (let ((status (plist-get encstatus :status)))
+ (cond
+ ((string= status "good")
+ "Decryption successful")
+ ((string= status "bad")
+ "Decryption error")
+ (t
+ (concat "Unknown encryption status"
+ (if status (concat ": " status))))))
+ " ]")
+ :type 'notmuch-crypto-status-button-type
+ 'face 'notmuch-crypto-decryption
+ 'mouse-face 'notmuch-crypto-decryption)
+ (insert "\n"))

;;

[David Bremner]

Post by David Edmondson
- (setq help-msg (concat "Click to retrieve key ID " keyid " from keyserver and redisplay."))))
+ (setq label (concat "Unknown key ID " keyid " or unsupported algorithm")
+ button-action 'notmuch-crypto-sigstatus-error-callback
+ help-msg (concat "Click to retrieve key ID " keyid
+ " from keyserver and redisplay.")))
+

I wonder if we should still claim we're going to redisplay here, since I
have the feeling most of the time that won't happen?

[David Edmondson]

Post by David Bremner

Post by David Edmondson
- (setq help-msg (concat "Click to retrieve key ID " keyid " from keyserver and redisplay."))))
+ (setq label (concat "Unknown key ID " keyid " or unsupported algorithm")
+ button-action 'notmuch-crypto-sigstatus-error-callback
+ help-msg (concat "Click to retrieve key ID " keyid
+ " from keyserver and redisplay.")))
+

I wonder if we should still claim we're going to redisplay here, since I
have the feeling most of the time that won't happen?

Will update.

dme.

[David Edmondson]

Improve the information provided about key retrieval and key validity.
---
emacs/notmuch-crypto.el | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-crypto.el b/emacs/notmuch-crypto.el
index 760461e8..4629c233 100644
--- a/emacs/notmuch-crypto.el
+++ b/emacs/notmuch-crypto.el
@@ -147,13 +147,16 @@ by user FROM."
(insert "\n"))

(defun notmuch-crypto-sigstatus-good-callback (button)
- (let* ((sigstatus (button-get button :notmuch-sigstatus))
+ (let* ((id (notmuch-show-get-message-id))
+ (sigstatus (button-get button :notmuch-sigstatus))
(fingerprint (concat "0x" (plist-get sigstatus :fingerprint)))
(buffer (get-buffer-create "*notmuch-crypto-gpg-out*"))
(window (display-buffer buffer)))
(with-selected-window window
(with-current-buffer buffer
(goto-char (point-max))
+ (insert (format "-- Key %s in message %s:\n"
+ fingerprint id))
(call-process notmuch-crypto-gpg-program nil t t "--list-keys" fingerprint))
(recenter -1))))

@@ -205,10 +208,14 @@ corresponding key when the status button is pressed."
(progn
(notmuch-crypto--set-button-label
button (format "Retrieving key %s asynchronously..." keyid))
+ (with-current-buffer buffer
+ (goto-char (point-max))
+ (insert (format "--- Retrieving key %s:\n" keyid)))
(let ((p (make-process :name "notmuch GPG key retrieval"
+ :connection-type 'pipe
:buffer buffer
+ :stderr buffer
:command (list notmuch-crypto-gpg-program "--recv-keys" keyid)
- :connection-type 'pipe
:sentinel #'notmuch-crypto--async-key-sentinel
;; Create the process stopped so that
;; we have time to store the key id,
@@ -224,6 +231,7 @@ corresponding key when the status button is pressed."
(with-selected-window window
(with-current-buffer buffer
(goto-char (point-max))
+ (insert (format "--- Retrieving key %s:\n" keyid))
(call-process notmuch-crypto-gpg-program nil t t "--recv-keys" keyid)
(insert "\n")
(call-process notmuch-crypto-gpg-program nil t t "--list-keys" keyid))