[PATCH] introduce new.rename_tags for renamed (moved) messages

[Michael J Gruber]

IMAP clients (such as webmail) use folders to mark messages as junk
etc., some even to mark messages as trash ("move to trash"). Such a
change is reported by notmuch as a rename; the message is not tagged
with new.tags since it is not new, so that there is no way to act upon a
rename.

Introduce new.rename_tags (default: not set) which are added by `notmuch
new` to renamed messages. This allows to act upon renames, e.g. to keep
the IMAP folder structure in sync with tags with a tool like `afew` or
homecooked scripts simply by filtering for this tag in the same ways as
one would filter for new messages using new.tags.

Signed-off-by: Michael J Gruber <***@grubix.eu>
---
NEWS | 11 +++++++++++
doc/man1/notmuch-config.rst | 6 ++++++
notmuch-client.h | 8 ++++++++
notmuch-config.c | 26 ++++++++++++++++++++++++++
notmuch-new.c | 23 +++++++++++++++++++++++
5 files changed, 74 insertions(+)

diff --git a/NEWS b/NEWS
index 240d594b..e3b75e74 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,14 @@
+Notmuch 0.28 (UNRELEASED)
+=========================
+
+New command-line features
+-------------------------
+
+User-configurable tags for renamed messages
+
+ A new "new.rename_tags" option is available in the configuration file to
+ determine which tags are applied to renamed (moved) messages.
+
Notmuch 0.27 (2018-06-13)
=========================

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 89909808..9e4198a1 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -77,6 +77,12 @@ The available configuration items are described below.

Default: ``unread;inbox``.

+**new.rename_tags**
+ A list of tags that will be added to all messages which
+ **notmuch new** identifies as renamed (moved).
+
+ Default: not set.
+
**new.ignore**
A list to specify files and directories that will not be searched
for messages by **notmuch new**. Each entry in the list is either:
diff --git a/notmuch-client.h b/notmuch-client.h
index 6c84ecc0..5e1e6b66 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -316,6 +316,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
const char *new_tags[],
size_t length);

+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config,
+ size_t *length);
+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+ const char *rename_tags[],
+ size_t length);
+
const char **
notmuch_config_get_new_ignore (notmuch_config_t *config,
size_t *length);
diff --git a/notmuch-config.c b/notmuch-config.c
index e1b16609..02f7d247 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -132,6 +132,8 @@ struct _notmuch_config {
size_t user_other_email_length;
const char **new_tags;
size_t new_tags_length;
+ const char **rename_tags;
+ size_t rename_tags_length;
const char **new_ignore;
size_t new_ignore_length;
bool maildir_synchronize_flags;
@@ -712,6 +714,14 @@ notmuch_config_get_new_tags (notmuch_config_t *config, size_t *length)
&(config->new_tags_length), length);
}

+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config, size_t *length)
+{
+ return _config_get_list (config, "new", "rename_tags",
+ &(config->rename_tags),
+ &(config->rename_tags_length), length);
+}
+
const char **
notmuch_config_get_new_ignore (notmuch_config_t *config, size_t *length)
{
@@ -738,6 +748,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
&(config->new_tags));
}

+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+ const char *list[],
+ size_t length)
+{
+ _config_set_list (config, "new", "rename_tags", list, length,
+ &(config->rename_tags));
+}
+
void
notmuch_config_set_new_ignore (notmuch_config_t *config,
const char *list[],
@@ -867,6 +886,13 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
tags = notmuch_config_get_new_tags (config, &length);
for (i = 0; i < length; i++)
printf ("%s\n", tags[i]);
+ } else if (strcmp(item, "new.rename_tags") == 0) {
+ const char **tags;
+ size_t i, length;
+
+ tags = notmuch_config_get_rename_tags (config, &length);
+ for (i = 0; i < length; i++)
+ printf ("%s\n", tags[i]);
} else if (STRNCMP_LITERAL (item, BUILT_WITH_PREFIX) == 0) {
printf ("%s\n",
notmuch_built_with (item + strlen (BUILT_WITH_PREFIX)) ? "true" : "false");
diff --git a/notmuch-new.c b/notmuch-new.c
index 6a54a1a1..e6d3dc82 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -50,6 +50,8 @@ typedef struct {
bool full_scan;
const char **new_tags;
size_t new_tags_length;
+ const char **rename_tags;
+ size_t rename_tags_length;
const char **ignore_verbatim;
size_t ignore_verbatim_length;
regex_t *ignore_regex;
@@ -948,9 +950,18 @@ remove_filename (notmuch_database_t *notmuch,

status = notmuch_database_remove_message (notmuch, path);
if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+ const char **tag;
add_files_state->renamed_messages++;
+ notmuch_message_freeze (message);
+
+ for (tag = add_files_state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+ notmuch_message_add_tag (message, *tag);
+ }
+
+
if (add_files_state->synchronize_flags == true)
notmuch_message_maildir_flags_to_tags (message);
+ notmuch_message_thaw (message);
status = NOTMUCH_STATUS_SUCCESS;
} else if (status == NOTMUCH_STATUS_SUCCESS) {
add_files_state->removed_messages++;
@@ -1095,6 +1106,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
add_files_state.verbosity = VERBOSITY_VERBOSE;

add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+ add_files_state.rename_tags = notmuch_config_get_rename_tags (config, &add_files_state.rename_tags_length);
add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
db_path = notmuch_config_get_database_path (config);
add_files_state.db_path = db_path;
@@ -1113,6 +1125,17 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
}
}

+ for (i = 0; i < add_files_state.rename_tags_length; i++) {
+ const char *error_msg;
+
+ error_msg = illegal_tag (add_files_state.rename_tags[i], false);
+ if (error_msg) {
+ fprintf (stderr, "Error: tag '%s' in rename.tags: %s\n",
+ add_files_state.rename_tags[i], error_msg);
+ return EXIT_FAILURE;
+ }
+ }
+
if (hooks) {
ret = notmuch_run_hook (db_path, "pre-new");
if (ret)

[David Bremner]

Post by Michael J Gruber
Introduce new.rename_tags (default: not set) which are added by `notmuch
new` to renamed messages. This allows to act upon renames, e.g. to keep
the IMAP folder structure in sync with tags with a tool like `afew` or
homecooked scripts simply by filtering for this tag in the same ways as
one would filter for new messages using new.tags.

The idea seems OK to me. I was hoping for some more feedback from
others, but here we are.

Post by Michael J Gruber
+ const char **tag;
add_files_state->renamed_messages++;
+ notmuch_message_freeze (message);
+
+ for (tag = add_files_state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+ notmuch_message_add_tag (message, *tag);
+ }
+
+

extra blank line

Post by Michael J Gruber
if (add_files_state->synchronize_flags == true)
notmuch_message_maildir_flags_to_tags (message);
+ notmuch_message_thaw (message);

Did you have a specific reason for putting the _thaw after the existing
maildir_flags_to_tags? It's probably not important, but if there's no
good reason it seems more natural before.

As a new feature this needs some tests before it can be merged. You can
start by looking at the tests for new.tags for inspiration I guess
(T050-new.sh and T340-maildir-sync.sh). One case that occured to me is
that when duplicate files (with the same message-id) exist, deleting one
of them is detected as a rename.

[Michael J Gruber]

IMAP clients (such as webmail) use folders to mark messages as junk
etc., some even to mark messages as trash ("move to trash"). Such a
change is reported by notmuch as a rename; the message is not tagged
with new.tags since it is not new, so that there is no way to act upon a
rename.

Introduce new.rename_tags (default: not set) which are added by `notmuch
new` to renamed messages. This allows to act upon renames, e.g. to keep
the IMAP folder structure in sync with tags with a tool like `afew` or
homecooked scripts simply by filtering for this tag in the same ways as
one would filter for new messages using new.tags.

Signed-off-by: Michael J Gruber <***@grubix.eu>
---
Changed since v1:
- acted upon review comments (blank line, _thaw position)
- added 3 tests (mv, cp, cp-rm)
- treat copies as renames, too

The reasoning behind the latter is: If you use a mapping between folders
and tags, then a copy to an additional location should alert the
"mapper" to update that mapping; that's what the rename tag is for.
Maybe it should be named "renew" after all? But it's just the
folder/label name that is/needs to be renewed, nothing else about the
message.

Interdiff against v1:
diff --git a/notmuch-new.c b/notmuch-new.c
index e6d3dc82..e893fa21 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -401,6 +401,13 @@ add_file (notmuch_database_t *notmuch, const char *filename,
break;
/* Non-fatal issues (go on to next file). */
case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ notmuch_message_freeze (message);
+
+ for (tag = state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+ notmuch_message_add_tag (message, *tag);
+ }
+
+ notmuch_message_thaw (message);
if (state->synchronize_flags)
notmuch_message_maildir_flags_to_tags (message);
break;
@@ -958,10 +965,9 @@ remove_filename (notmuch_database_t *notmuch,
notmuch_message_add_tag (message, *tag);
}

-
+ notmuch_message_thaw (message);
if (add_files_state->synchronize_flags == true)
notmuch_message_maildir_flags_to_tags (message);
- notmuch_message_thaw (message);
status = NOTMUCH_STATUS_SUCCESS;
} else if (status == NOTMUCH_STATUS_SUCCESS) {
add_files_state->removed_messages++;
diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh
index 7fece5f2..44f32ad2 100755
--- a/test/T340-maildir-sync.sh
+++ b/test/T340-maildir-sync.sh
@@ -196,6 +196,36 @@ notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
test_expect_equal "$(< output)" \
"thread:XXX 2001-01-05 [1/1] Notmuch Test Suite; File in new/ (test unread)"

+test_begin_subtest "Renamed files get default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "renamed"
+mv $MAIL_DIR/new/file-in-new $MAIL_DIR/new/file-in-new-renamed
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX 2001-01-05 [1/1] Notmuch Test Suite; File in new/ (renamed test unread)"
+
+test_begin_subtest "Copied files do get new default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "copied"
+cp $MAIL_DIR/new/file-in-new-renamed $MAIL_DIR/new/file-in-new-copied
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX 2001-01-05 [1/1(2)] Notmuch Test Suite; File in new/ (copied renamed test unread)"
+
+test_begin_subtest "Renamed files (cp+rm) get default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "cprm"
+rm $MAIL_DIR/new/file-in-new-renamed
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX 2001-01-05 [1/1] Notmuch Test Suite; File in new/ (copied cprm renamed test unread)"
+
for tag in draft flagged passed replied; do
test_begin_subtest "$tag is valid in new.tags"
OLDCONFIG=$(notmuch config get new.tags)

NEWS | 11 +++++++++++
doc/man1/notmuch-config.rst | 6 ++++++
notmuch-client.h | 8 ++++++++
notmuch-config.c | 26 ++++++++++++++++++++++++++
notmuch-new.c | 29 +++++++++++++++++++++++++++++
test/T340-maildir-sync.sh | 30 ++++++++++++++++++++++++++++++
6 files changed, 110 insertions(+)

diff --git a/NEWS b/NEWS
index 240d594b..e3b75e74 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,14 @@
+Notmuch 0.28 (UNRELEASED)
+=========================
+
+New command-line features
+-------------------------
+
+User-configurable tags for renamed messages
+
+ A new "new.rename_tags" option is available in the configuration file to
+ determine which tags are applied to renamed (moved) messages.
+
Notmuch 0.27 (2018-06-13)
=========================

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 89909808..9e4198a1 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -77,6 +77,12 @@ The available configuration items are described below.

Default: ``unread;inbox``.

+**new.rename_tags**
+ A list of tags that will be added to all messages which
+ **notmuch new** identifies as renamed (moved).
+
+ Default: not set.
+
**new.ignore**
A list to specify files and directories that will not be searched
for messages by **notmuch new**. Each entry in the list is either:
diff --git a/notmuch-client.h b/notmuch-client.h
index 6c84ecc0..5e1e6b66 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -316,6 +316,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
const char *new_tags[],
size_t length);

+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config,
+ size_t *length);
+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+ const char *rename_tags[],
+ size_t length);
+
const char **
notmuch_config_get_new_ignore (notmuch_config_t *config,
size_t *length);
diff --git a/notmuch-config.c b/notmuch-config.c
index e1b16609..02f7d247 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -132,6 +132,8 @@ struct _notmuch_config {
size_t user_other_email_length;
const char **new_tags;
size_t new_tags_length;
+ const char **rename_tags;
+ size_t rename_tags_length;
const char **new_ignore;
size_t new_ignore_length;
bool maildir_synchronize_flags;
@@ -712,6 +714,14 @@ notmuch_config_get_new_tags (notmuch_config_t *config, size_t *length)
&(config->new_tags_length), length);
}

+const char **
+notmuch_config_get_rename_tags (notmuch_config_t *config, size_t *length)
+{
+ return _config_get_list (config, "new", "rename_tags",
+ &(config->rename_tags),
+ &(config->rename_tags_length), length);
+}
+
const char **
notmuch_config_get_new_ignore (notmuch_config_t *config, size_t *length)
{
@@ -738,6 +748,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
&(config->new_tags));
}

+void
+notmuch_config_set_rename_tags (notmuch_config_t *config,
+ const char *list[],
+ size_t length)
+{
+ _config_set_list (config, "new", "rename_tags", list, length,
+ &(config->rename_tags));
+}
+
void
notmuch_config_set_new_ignore (notmuch_config_t *config,
const char *list[],
@@ -867,6 +886,13 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
tags = notmuch_config_get_new_tags (config, &length);
for (i = 0; i < length; i++)
printf ("%s\n", tags[i]);
+ } else if (strcmp(item, "new.rename_tags") == 0) {
+ const char **tags;
+ size_t i, length;
+
+ tags = notmuch_config_get_rename_tags (config, &length);
+ for (i = 0; i < length; i++)
+ printf ("%s\n", tags[i]);
} else if (STRNCMP_LITERAL (item, BUILT_WITH_PREFIX) == 0) {
printf ("%s\n",
notmuch_built_with (item + strlen (BUILT_WITH_PREFIX)) ? "true" : "false");
diff --git a/notmuch-new.c b/notmuch-new.c
index 6a54a1a1..e893fa21 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -50,6 +50,8 @@ typedef struct {
bool full_scan;
const char **new_tags;
size_t new_tags_length;
+ const char **rename_tags;
+ size_t rename_tags_length;
const char **ignore_verbatim;
size_t ignore_verbatim_length;
regex_t *ignore_regex;
@@ -399,6 +401,13 @@ add_file (notmuch_database_t *notmuch, const char *filename,
break;
/* Non-fatal issues (go on to next file). */
case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ notmuch_message_freeze (message);
+
+ for (tag = state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+ notmuch_message_add_tag (message, *tag);
+ }
+
+ notmuch_message_thaw (message);
if (state->synchronize_flags)
notmuch_message_maildir_flags_to_tags (message);
break;
@@ -948,7 +957,15 @@ remove_filename (notmuch_database_t *notmuch,

status = notmuch_database_remove_message (notmuch, path);
if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) {
+ const char **tag;
add_files_state->renamed_messages++;
+ notmuch_message_freeze (message);
+
+ for (tag = add_files_state->rename_tags; tag != NULL && *tag != NULL; tag++) {
+ notmuch_message_add_tag (message, *tag);
+ }
+
+ notmuch_message_thaw (message);
if (add_files_state->synchronize_flags == true)
notmuch_message_maildir_flags_to_tags (message);
status = NOTMUCH_STATUS_SUCCESS;
@@ -1095,6 +1112,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
add_files_state.verbosity = VERBOSITY_VERBOSE;

add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+ add_files_state.rename_tags = notmuch_config_get_rename_tags (config, &add_files_state.rename_tags_length);
add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
db_path = notmuch_config_get_database_path (config);
add_files_state.db_path = db_path;
@@ -1113,6 +1131,17 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
}
}

+ for (i = 0; i < add_files_state.rename_tags_length; i++) {
+ const char *error_msg;
+
+ error_msg = illegal_tag (add_files_state.rename_tags[i], false);
+ if (error_msg) {
+ fprintf (stderr, "Error: tag '%s' in rename.tags: %s\n",
+ add_files_state.rename_tags[i], error_msg);
+ return EXIT_FAILURE;
+ }
+ }
+
if (hooks) {
ret = notmuch_run_hook (db_path, "pre-new");
if (ret)
diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh
index 7fece5f2..44f32ad2 100755
--- a/test/T340-maildir-sync.sh
+++ b/test/T340-maildir-sync.sh
@@ -196,6 +196,36 @@ notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
test_expect_equal "$(< output)" \
"thread:XXX 2001-01-05 [1/1] Notmuch Test Suite; File in new/ (test unread)"

+test_begin_subtest "Renamed files get default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "renamed"
+mv $MAIL_DIR/new/file-in-new $MAIL_DIR/new/file-in-new-renamed
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX 2001-01-05 [1/1] Notmuch Test Suite; File in new/ (renamed test unread)"
+
+test_begin_subtest "Copied files do get new default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "copied"
+cp $MAIL_DIR/new/file-in-new-renamed $MAIL_DIR/new/file-in-new-copied
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX 2001-01-05 [1/1(2)] Notmuch Test Suite; File in new/ (copied renamed test unread)"
+
+test_begin_subtest "Renamed files (cp+rm) get default renamed tags"
+OLDCONFIG=$(notmuch config get new.rename_tags)
+notmuch config set new.rename_tags "cprm"
+rm $MAIL_DIR/new/file-in-new-renamed
+notmuch new
+notmuch config set new.rename_tags $OLDCONFIG
+notmuch search 'subject:"File in new"' | notmuch_search_sanitize > output
+test_expect_equal "$(< output)" \
+"thread:XXX 2001-01-05 [1/1] Notmuch Test Suite; File in new/ (copied cprm renamed test unread)"
+
for tag in draft flagged passed replied; do
test_begin_subtest "$tag is valid in new.tags"
OLDCONFIG=$(notmuch config get new.tags)

[Gaute Hope]

Post by Michael J Gruber
IMAP clients (such as webmail) use folders to mark messages as junk
etc., some even to mark messages as trash ("move to trash"). Such a
change is reported by notmuch as a rename; the message is not tagged
with new.tags since it is not new, so that there is no way to act upon a
rename.
Introduce new.rename_tags (default: not set) which are added by `notmuch
new` to renamed messages. This allows to act upon renames, e.g. to keep
the IMAP folder structure in sync with tags with a tool like `afew` or
homecooked scripts simply by filtering for this tag in the same ways as
one would filter for new messages using new.tags.

Hi,

think this would be very useful. I suggested something similar way back[0]
(no doubt bit-rotted by now), and seem to remember there were some
issues with cases where a rename would not be detected. Might be worth
checking out !

[0] id:1396800683-9164-1-git-send-email-***@gaute.vetsj.com

BTW: My use-case was solved by using `lastmod:` queries in keywsync [1]
(later obsoleted by gmailieer[2]).

[1] https://github.com/gauteh/abunchoftags
[2] https://github.com/gauteh/gmailieer

Regards, Gaute

[Gaute Hope]

Lastmod and (re)new tags serve different purposes, I would thank. The
latter is a flag that can be cleared to signal that a certain job has
been done (by whomever). Whereas, if one actor acts upon a lastmod
change by modifying the db that will change the lastmod and (possibly)
trigger another actor to act upon that. Different scenarios which look
the same if you have one "actor" only.

Agreed, very useful if all scenarios can be reliably caught! Maybe it
was the "message changed without rename" that was problematic, since
offlineimap would change the X-Keywords header without renaming the
file.


- gaute