tell me how to do this right (mail sent to lists)

[Jeff Templon]

Hi,

The use case: I send a message to a mailing list of which I am a member.

I can see via another imap client (in this case MailMate / OS X) that
there are two messages in imap, both with the same messageid. One
example right now (printing the Subject header below):

in my sent mail folder:

[***@Veldhoven] Inschrijving/Registration ***@Veldhoven 2019 deadline 9 November 2019

and in my inbox:

[pdp] [***@Veldhoven] Inschrijving/Registration ***@Veldhoven 2019: deadline 9 November 2019


However, in notmuch-emacs, I only see the first one. My guess is that
notmuch has first seen the one in "sent", indexed that, and then seeing
the second one with the same messageId, skips it. I'd prefer to have
both, but if I have to pick one or the other, I'd actually prefer to
have the second one, as I'd then have

[ pdp ]

as the start of the subject header for ALL mails to that list, and as an
added bonus, afew (should?) then tag them with the list name making them
easily searchable.

So how does one do this "right"?

Thanks,

JT

[Daniel Kahn Gillmor]

Hi Jeff--

Post by Jeff Templon
I can see via another imap client (in this case MailMate / OS X) that
there are two messages in imap, both with the same messageid. One
However, in notmuch-emacs, I only see the first one. My guess is that
notmuch has first seen the one in "sent", indexed that, and then seeing
the second one with the same messageId, skips it.

notmuch doesn't "skip" anything -- it just knows that they are the same
e-mail due to message-id matching. it actually indexes both texts. you
can see this with:

notmuch search --output=files id:$MESSAGEID

Post by Jeff Templon
I'd prefer to have both, but if I have to pick one or the other, I'd
actually prefer to have the second one, as I'd then have
[ pdp ]
as the start of the subject header for ALL mails to that list, and as an
added bonus, afew (should?) then tag them with the list name making them
easily searchable.

I'd recommend doing your tagging based on something other than a
substring of the subject line -- wouldn't tagging based on the recipient
of the message (To: or Cc:) be sufficient to group your mailing list
messages under one heading, regardless of which variant notmuch chooses
to prefer?

i recognize this doesn't answer your proximal question -- and i'd love
to hear if someone more knowledgable has a good answer for it. i'm just
offering a workaround for what i think is your ultimate goal.

all the best,

--dkg

[Jeff Templon]

Hi Daniel,

Thanks for answering!

Post by Daniel Kahn Gillmor
notmuch doesn't "skip" anything -- it just knows that they are the same
e-mail due to message-id matching. it actually indexes both texts.
you
notmuch search --output=files id:$MESSAGEID

Indeed - with the command line tool in this form, I do see both files.
Read
on :-)

Post by Daniel Kahn Gillmor

Post by Jeff Templon
I'd prefer to have both, but if I have to pick one or the other, I'd
actually prefer to have the second one, as I'd then have
[ pdp ]
as the start of the subject header for ALL mails to that list, and as an
added bonus, afew (should?) then tag them with the list name making them
easily searchable.

I'd recommend doing your tagging based on something other than a
substring of the subject line -- wouldn't tagging based on the
recipient
of the message (To: or Cc:) be sufficient to group your mailing list
messages under one heading, regardless of which variant notmuch chooses
to prefer?

So here’s the deal : I am tagging with afew, it has a special mailing
list filter,
which does the right thing - it tags the copy that was delivered from
the mailing
list with the list name. However (and maybe this is just the Emacs
interface?) when
I view the message in notmuch-emacs, it only shows one of them, and this
seems to be
(always? at least usually) the copy that is Fcc’d to my sent folder
and tagged only
with “sent” because it doesn’t have a list-id yet.

So I’m still looking for how to do this “right”. I must not be
the only one that sends
mails to mailing lists?

JT

[Carl Worth]

So here’s the deal : I am tagging with afew, it has a special mailing
list filter, which does the right thing - it tags the copy that was
delivered from the mailing list with the list name.

I think you may be mistaken about the above details.

If the two messages have the same message ID, then as far as notmuch is
concerned that is just a single message, (that happens to be backed by
multiple files).

However (and maybe this is just the Emacs interface?) when I view the
message in notmuch-emacs, it only shows one of them, and this seems to
be (always? at least usually) the copy that is Fcc’d to my sent
folder and tagged only with “sent” because it doesn’t have a list-id
yet.

It is true that if a single message is backed by multiple files, the
notmuch-emacs interface will only display one of them. But even if you
changed which one it chooses (which would require some new code as far
as I know) shouldn't change what tags you are seeing.

So I’m still looking for how to do this “right”. I must not be the
only one that sends mails to mailing lists?

You're certainly not the only one. Hopefully, what you really want
doesn't require the ability to maintain separate tags for these two
very-similar messages, (since notmuch cannot do that currently).

I guess the question I have is: What has given you the idea that afew is
tagging one message but not the other?

-Carl

[Jeff Templon]

Hi,

Thanks again for the answers.

Post by Carl Worth
If the two messages have the same message ID, then as far as notmuch is
concerned that is just a single message, (that happens to be backed by
multiple files).

OK.

Post by Carl Worth
You're certainly not the only one. Hopefully, what you really want
doesn't require the ability to maintain separate tags for these two
very-similar messages, (since notmuch cannot do that currently).

I wasn't clear - I don't need different tags. I do want the
list-associated tags, even if it only shows me the "sent copy".

Post by Carl Worth
I guess the question I have is: What has given you the idea that afew is
tagging one message but not the other?

Well, this:

medina:~> notmuch search --output=files id:***@simeto.nikhef.nl
/Users/templon/Maildir/Nikhef/Sent/cur/1539098270.M625625000P91131Q71R9d5148db.dhcp-134-239.nikhef.nl:2,S
/Users/templon/Maildir/Nikhef/Unclassified Bulk/cur/1539098271.M415561000P91131Q113R5b534386.dhcp-134-239.nikhef.nl:2,

medina:~> notmuch search --output=tags id:***@simeto.nikhef.nl
sent

So the one in "Sent" is, well, sent :-) The other one (Unclassified
Bulk) is the one from the list. There are no list-associated tags.
Hence my conclusion is that notmuch and/or afew encounters first the
sent copy (maybe because S comes before U?) and tags it, then, when
encountering the second file while processing, says in essence "oh i
already have this message ID - skip".

JT

[David Bremner]

Post by Jeff Templon

Post by Carl Worth
I guess the question I have is: What has given you the idea that afew is
tagging one message but not the other?

/Users/templon/Maildir/Nikhef/Sent/cur/1539098270.M625625000P91131Q71R9d5148db.dhcp-134-239.nikhef.nl:2,S
/Users/templon/Maildir/Nikhef/Unclassified Bulk/cur/1539098271.M415561000P91131Q113R5b534386.dhcp-134-239.nikhef.nl:2,
sent
So the one in "Sent" is, well, sent :-) The other one (Unclassified
Bulk) is the one from the list. There are no list-associated tags.
Hence my conclusion is that notmuch and/or afew encounters first the
sent copy (maybe because S comes before U?) and tags it, then, when
encountering the second file while processing, says in essence "oh i
already have this message ID - skip".

The tag is not associated with the file in Sent, it is associated
with the message-id.

Searching currently for words in the subject of either copy of the
message should work, so you should be able to tag the message(-id) as
you like. I can't help with afew, but if you have an example that
doesn't work on the command line, I'm interested. Display of such
messages (and threads) is still less than ideal as it still essentially
choose an arbitrary file with that message-id.

d

[Jeff Templon]

Post by David Bremner
The tag is not associated with the file in Sent, it is associated
with the message-id.

I guess I didn't make myself clear enough, again. I didn't mean that
the tag is associated with the file. What I am guessing is something
like this:

for message in new_messages:
if message.id not in database:
process message and determine list of tags
appy those tags to the messageID

new_messages is the list of files that notmuch had not yet seen.

notice the if statement. I am guessing, based on the behavior, that
this is how it works. So the implication would be:

if the first message file encountered for that messageID is the one in
Sent, it does not have a ListID field and hence does not get the list
tags applied. The second message file (the one sent via the mailing
list) DOES have a ListID field, but notmuch "already knows this
messageID" so it does not look at that message and the list tags never
get applied.

I'm not sure whether it's an afew problem or a notmuch problem ...

JT

[David Bremner]

Post by Jeff Templon
if the first message file encountered for that messageID is the one in
Sent, it does not have a ListID field and hence does not get the list
tags applied. The second message file (the one sent via the mailing
list) DOES have a ListID field, but notmuch "already knows this
messageID" so it does not look at that message and the list tags never
get applied.
I'm not sure whether it's an afew problem or a notmuch problem ...

It sounds like an afew problem to me. notmuch proper doesn't know about
the List-ID field.

d

[Martin Jambor]

Hi,

I am a notmuch newbie and do not use afew (but have my own post-new hook
script), but...

Post by Jeff Templon

Post by David Bremner
The tag is not associated with the file in Sent, it is associated
with the message-id.

I guess I didn't make myself clear enough, again. I didn't mean that
the tag is associated with the file. What I am guessing is something
process message and determine list of tags
appy those tags to the messageID
new_messages is the list of files that notmuch had not yet seen.

...I don't think the above is entirely accurate. IIUC, what happens is
that when notmuch sees a message for the first time during "notmuch
new", it simply and only applies the tags specified in the tags entry of
[new] section in .notmuch-config file to them - the default I believe is
"unread;inbox;new".

Then the post-hook is executed, which in your case I presume runs afew
somehow, and typically that hook goes over all messages with the tag
new, process them in an arbitrary way, sometimes adding further tags
according to the contents, and eventually drops the new tag.

I guess a potential problem is that notmuch sees the message the first
time when you send it and it adds the new tag and post-hook processes
the outgoing version. When it arrives back, I believe notmuch does not
apply the new tag to it anymore. This means that post-hook does not get
to process the incoming version, which might be a problem if you want to
see stuff added by a remote party, for example when you want to grep the
file and look for Sender header in order to mark it as coming from a
particular mailing list.

That is the reason I do not attempt this in my post-hook and rely on
notmuch "tag:new AND to:***@address" search which is not really
precise but at least it should be probably faster.

Perhaps it would be a good idea to split the tags entry in [new] section
into two, one for tags added to each previously unknown message-id
(default would be unread and inbox) and a new one for each new file
(default new)?

Let me stress again that I am a newbie and can be completely wrong about
anything above, so take it with a grain of salt.

Thanks,

Martin

[Jeff Templon]

Hi Martin,

Post by Martin Jambor
Then the post-hook is executed, which in your case I presume runs afew
somehow, and typically that hook goes over all messages with the tag
new, process them in an arbitrary way, sometimes adding further tags
according to the contents, and eventually drops the new tag.
I guess a potential problem is that notmuch sees the message the first
time when you send it and it adds the new tag and post-hook processes
the outgoing version. When it arrives back, I believe notmuch does not
apply the new tag to it anymore. This means that post-hook does not get
to process the incoming version, which might be a problem if you want to
see stuff added by a remote party, for example when you want to grep the
file and look for Sender header in order to mark it as coming from a
particular mailing list.

Thanks for your clarification, this is indeed what I meant when I said
that what happens is dependent on the order in which notmuch encounters
the message files / copies possessing the same messageID.

Rereading the afew documentation (not very clear) it hints at this
point:

"SentMailFilter"

It can be used for example to easily tag posts sent to mailing lists
which at this stage don’t have List-Id field.

without making it explicit, unfortunately, that if you choose not to use
this, messages you send to the mailing list won't be tagged in the same
way as those received from others through the mailing list.

Post by Martin Jambor
Perhaps it would be a good idea to split the tags entry in [new] section
into two, one for tags added to each previously unknown message-id
(default would be unread and inbox) and a new one for each new file
(default new)?

If I understand you correctly, you're suggesting that newly-detected
files always have their messageIDs tagged as "new" even if that
messageID has already been "seen"? Maybe that solves my case here, I
guess it would depend on how the message files are fed from notmuch to
afew.

JT

[Carl Worth]

Post by Jeff Templon

Post by David Bremner
The tag is not associated with the file in Sent, it is associated
with the message-id.

I guess I didn't make myself clear enough, again. I didn't mean that
the tag is associated with the file. What I am guessing is something

Hi Jeff,

Thanks for persevering so that we can all try to understand what's
happening. I appreciate the patience on all sides. :-)

Post by Jeff Templon
process message and determine list of tags
appy those tags to the messageID

Well, there are actually a couple of different processing loops that you
might be describing with the above. Let me try to walk through things:

First, there's a loop where "notmuch new" finds previously-unseen files
and indexes the content, adding it to the database:

for message_file in new_files:
message_id = get_headers_message_id (message_file)
(message,is_new) = database_lookup (message_id)
index_file (message, message_file)
if is_new:
add_new_message_tags (message)

The above pseudo-code is based on the loop in notmuch-new.c:add_files(),
add_file() as well as lib/add-message.cc:notmuch_database_index_file()
and more or less trying to use naming consistent with the code.

Something to not in the above loop: The database_lookup above, (which is
actually _notmuch_message_create_for_message_id), can either create a
new message object in the database or return an existing object. But,
either way, the content of the message will be indexed. So, the
significant feature is that notmuch will always be able to search the
content it indexes for any message file, (regardless of the order it was
encountered given any duplication).

However, as can also be seen in the above loop, tags that are added
to new messages, (these are as configured in the "new.tags" entry in
~/.notmuch-config) will only be added if the message is new in this pass
of notmuch-new.

I'm not looking into the code for afew right now, but I can guess a
couple of places the undesired bug could be happening:

1. It could be looping over all messages with the "new" tag. And if your
sent message gets tagged "new" in a pass before the mailing-list
duplicate is present, then afew will not have access to the
mailing-list version when it does its processing. Then, later, when a
pass does have the mailing-list duplicate present, it won't be
considered a "new" message so would not get picked processed in a
loop considering messages tagged "new".

2. It's possible that both message files are present at the time that
afew does its processing, but that it only opens one of the files to
go looking for the List-Id header, (which it must be doing
somewhere---as David mentioned, the List-Id header is not ever
indexed by notmuch itself).

And in the above discussion, I'm assuming that it's even notmuch-new
that's doing the detection of new files. Some people use mail flows that
have some external mechanism for processing new incoming mail and then
calling "notmuch insert" for each one.

In conclusion, you have a few different options to get reliable
behavior:

One option is to use notmuch-based searches to find the mailing-list
mail that is of interest for you. To do this you would want to key off
of a header that is indexed by notmuch. For example, you could do
something like:

notmuch tag +my-list-tag to:my-list-recipient-address

Another option is to continue to tag messages by inspecting the file
(outside of notmuch) to look for a header like List-Id (like you are
apparently doing now). To make this reliably, you would simply want to
ensure that that processing happens on every new file that is added. And
note that the "new" tag as added by "notmuch new" is not reliable for
that. That tag _is_ reliable for learning that a new message ID has
become available in the database, but is not reliably for know that a
new message file has appeared, (for a message ID that was present
previously).

Does that help explain things?

-Carl

[Jeff Templon]

Hi All,

Coming back to this, I think (I need to watch it for a week or so) that
I have "solved" the mystery.

When working from mail, I noticed that sometimes, a mail that I had sent
could not be found as "sent". I had borrowed an elisp function from
somewhere that marked "interesting messages as sent", and I think what I
inadvertently did here was to bypass all the afew tagging for such
messages.

When I stopped using this elisp function, the messages seem to be tagged
correctly again. It's only been a few days (you know the "let's see if
it does it again joke, right?") so like I said, I'll keep an eye on it
to see whether it's really "fixed" now.

Thanks for all your help and patience in this matter!

JT