Notmuch DB Problems

[m***@nawaz.org]

Hi,

A few days ago I noticed notmuch new was no longer working (I have it
as a cron job so it took a while to figure it out).

It just freezes. I do have a Python hook, and it was freezing on the
line that opens the database.

I tried a notmuch dump. Same problem - freezes

Based on some earlier threads, I tried a notmuch compact. Same problem
- freezes.

All these freezes seem to use no memory/CPU.

Interestingly, queries work fine - from both the command line and the
Emacs interface. So I can read old stuff just fine. But all the
commands above cause a freeze.

Currently using notmuch-0.24.2. I tried notmuch-0.27 - same problem.

Results of a xapian check:

docdata:
blocksize=8K items=6 firstunused=3 revision=6442 levels=0 root=0
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=178562 firstunused=53441 revision=6442 levels=2
root=46086
/usr/bin/xapian-check: DatabaseError: 1 unused block(s) missing from
the free list, first is 0

What are my options? Unfortunately the last dump I have is many months
old, so I'm a bit wary of deleting the database and rebuilding. Given
that the show and search commands work, I was wondering if I can write
a script to get all the message/thread ID's for all the tags and store
them, and then rebuild the database and use that stored information to
retag all my messages (all without using the dump command)?

Thanks!

Mueen

[Jani Nikula]

Post by m***@nawaz.org
Hi,
A few days ago I noticed notmuch new was no longer working (I have it
as a cron job so it took a while to figure it out).
It just freezes. I do have a Python hook, and it was freezing on the
line that opens the database.
I tried a notmuch dump. Same problem - freezes
Based on some earlier threads, I tried a notmuch compact. Same problem
- freezes.
All these freezes seem to use no memory/CPU.
Interestingly, queries work fine - from both the command line and the
Emacs interface. So I can read old stuff just fine. But all the
commands above cause a freeze.
Currently using notmuch-0.24.2. I tried notmuch-0.27 - same problem.
blocksize=8K items=6 firstunused=3 revision=6442 levels=0 root=0
B-tree checked okay
docdata table structure checked OK
blocksize=8K items=178562 firstunused=53441 revision=6442 levels=2
root=46086
/usr/bin/xapian-check: DatabaseError: 1 unused block(s) missing from
the free list, first is 0
What are my options? Unfortunately the last dump I have is many months
old, so I'm a bit wary of deleting the database and rebuilding. Given
that the show and search commands work, I was wondering if I can write
a script to get all the message/thread ID's for all the tags and store
them, and then rebuild the database and use that stored information to
retag all my messages (all without using the dump command)?

It might be interesting to see an strace log to possibly get an idea
where it gets stuck.

Is the filesystem writable and working okay?

If search and show work, I'm guessing it gets stuck in trying to open
the database writable. One hackish idea is to patch notmuch dump to open
the database in read-only mode, and dump the tags. See below. The dump
command opens the database writable to prevent changes while
dumping. (Arguably this could be a command line option for cases like
yours.)

BR,
Jani.

diff --git a/notmuch-dump.c b/notmuch-dump.c
index ef2f02dfeb5c..d06dbcf50224 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -364,7 +364,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
int ret;

if (notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_WRITE, &notmuch))
+ NOTMUCH_DATABASE_MODE_READ_ONLY, &notmuch))
return EXIT_FAILURE;

notmuch_exit_if_unmatched_db_uuid (notmuch);

[Mueen Nawaz]

Post by Jani Nikula
It might be interesting to see an strace log to possibly get an idea
where it gets stuck.
Is the filesystem writable and working okay?
If search and show work, I'm guessing it gets stuck in trying to open
the database writable. One hackish idea is to patch notmuch dump to open
the database in read-only mode, and dump the tags. See below. The dump
command opens the database writable to prevent changes while
dumping. (Arguably this could be a command line option for cases like
yours.)

Thanks - your patch worked. I dumped all the tags, deleted the database,
rebuilt it and restored the tags. All was well.

Until the following day at noon I noticed the problem was back. By
evening, I could not even do queries - it wouldn't open even in read
only mode. The database was dead.

After a lot of poking around, I figured out the problem, and this may be
of interest to the developers (although not sure if it is a xapian issue
or a notmuch issue).

Here's why it would freeze:

I have a post-new hook that runs a Python script. Depending on whether
the new email it is processing matches a rule I have, it will fire off
an email to the sender using the SMTP library in Python.

I had recently upgraded my MTA (PostFix), and it had a backward
incompatible change that broke my config. I don't know why, but I could
still send emails via Emacs, but when I tried to send them via Python,
Postfix would log an error and it would not send. The Python statement
would freeze (I guess Postfix doesn't return an appropriate response?
Not sure why).


I have a cron job to run "notmuch new" 3 times an hour. Since the hook
was frozen, so was the notmuch new command. I had quite a lot of
"notmuch new" processes. I assume this meant the DB was locked all this
time for writing.

Now killing all those jobs did not fix the database. It was still
broken. And as we saw the second time round, it was /really/ broken - it
would not even open in read-only mode.

It is scary that if a post-new hook freezes while the database is
locked, it could (eventually) clobber the database. I don't know if
notmuch can do anything to prevent this outcome?

BTW, I think the DB would die only after a while. In my experiments, if
I killed the hook soon (e.g. under 1 minute), the database seemed fine.

[David Bremner]

Post by Mueen Nawaz
After a lot of poking around, I figured out the problem, and this may be
of interest to the developers (although not sure if it is a xapian issue
or a notmuch issue).
I have a post-new hook that runs a Python script. Depending on whether
the new email it is processing matches a rule I have, it will fire off
an email to the sender using the SMTP library in Python.
I had recently upgraded my MTA (PostFix), and it had a backward
incompatible change that broke my config. I don't know why, but I could
still send emails via Emacs, but when I tried to send them via Python,
Postfix would log an error and it would not send. The Python statement
would freeze (I guess Postfix doesn't return an appropriate response?
Not sure why).
I have a cron job to run "notmuch new" 3 times an hour. Since the hook
was frozen, so was the notmuch new command. I had quite a lot of
"notmuch new" processes. I assume this meant the DB was locked all this
time for writing.

notmuch unlocks the database before running the hook, so I don't
understand how a hung hook results in a locked database. If it happens
again (or you're motivated to set up a testbed) I'd be interested in the
output of

lsof ~/Maildir/.notmuch/xapian/flintlock

Also, is this by chance a network file system? Because those often
break locking.

Post by Mueen Nawaz
Now killing all those jobs did not fix the database. It was still
broken. And as we saw the second time round, it was /really/ broken - it
would not even open in read-only mode.

That seems like something the Xapian devs (in copy) might be interested
in fixing, if you could come up with a simple reproducer.

Post by Mueen Nawaz
It is scary that if a post-new hook freezes while the database is
locked, it could (eventually) clobber the database. I don't know if
notmuch can do anything to prevent this outcome?

notmuch could be cleverer about timing out on trying to acquire a
lock. I suspect it's a bit delicate to get that right, and I've been
hoping the underlying primitives would get a bit more flexible
w.r.t. locking.

We could also potentially run hooks in the equivalent of "timeout", but
I don't know how much code that would be. A simpler option (once we
understand what the real problem is) would be to suggest that users use
timeout themselves in hooks to be run unattended.

[Mueen Nawaz]

Post by David Bremner

Post by Mueen Nawaz
I have a post-new hook that runs a Python script. Depending on
whether the new email it is processing matches a rule I have,
it will fire off an email to the sender using the SMTP library
in Python.
I had recently upgraded my MTA (PostFix), and it had a backward
incompatible change that broke my config. I don't know why, but
I could still send emails via Emacs, but when I tried to send
them via Python, Postfix would log an error and it would not
send. The Python statement would freeze (I guess Postfix
doesn't return an appropriate response? Not sure why).
I have a cron job to run "notmuch new" 3 times an hour. Since
the hook was frozen, so was the notmuch new command. I had
quite a lot of "notmuch new" processes. I assume this meant the
DB was locked all this time for writing.

notmuch unlocks the database before running the hook, so I don't
understand how a hung hook results in a locked database. If it
happens again (or you're motivated to set up a testbed) I'd be
interested in the output of

Well, it results in a locked database because I have this in the
(Python) hook:

DATABASE = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)

Soon after that I freeze the new messages. And at the end I thaw
them out. The hang occurs in between the two, I think.

Post by David Bremner
Also, is this by chance a network file system? Because those
often break locking.

No - regular hard drive.

Post by David Bremner

Post by Mueen Nawaz
Now killing all those jobs did not fix the database. It was
still broken. And as we saw the second time round, it was
/really/ broken - it would not even open in read-only mode.

That seems like something the Xapian devs (in copy) might be
interested in fixing, if you could come up with a simple
reproducer.

I can think of two experiments:

1. Write a hook that opens the database as above, and then just
does nothing (e.g. while True). Let it run, say, for 24
hours. (Not sure if the "freeze" part is relevant.

2. Same as the above, but have a cron job that fires "notmuch new"
every 20 minutes. This will freeze on the database line above (all
except the first invocation which will be stuck at while True).

After a day of this, check if you can open the database in
READ_WRITE mode.

Post by David Bremner
notmuch could be cleverer about timing out on trying to acquire
a lock. I suspect it's a bit delicate to get that right, and
I've been hoping the underlying primitives would get a bit more
flexible w.r.t. locking.

I agree having notmuch handle it is not ideal. I was originally
thinking there should be a default timeout that one can adjust as
needed. However, when someone does "notmuch new" to build a new
database, that can take several minutes. And others may have flows
very different from mine.

At the very least, we probably should know why the DB be clobbered
at all.

[David Bremner]

Post by Mueen Nawaz
DATABASE = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)

OK. So your code is locking the database, and never unlocking it
(because of the hang). So that part is at least not mysterious.
I was thinking more along the lines of something that could be part of
the notmuch test suite, i.e. run in a few seconds. Or at worst in 10
minutes or so to be usable to debug.

d

[Mueen Nawaz]

Post by David Bremner
I was thinking more along the lines of something that could be
part of the notmuch test suite, i.e. run in a few seconds. Or at
worst in 10 minutes or so to be usable to debug.

I don't know if this can reliably be done. As I pointed out in my
earlier post, hanging for a short time (1-2 minutes) did not seem
to clobber the database. I don't know what the threshold is, and
whether the threshold is constant.

[Olly Betts]

Post by David Bremner

Post by Mueen Nawaz
Now killing all those jobs did not fix the database. It was still
broken. And as we saw the second time round, it was /really/ broken - it
would not even open in read-only mode.

That seems like something the Xapian devs (in copy) might be interested
in fixing, if you could come up with a simple reproducer.

I'm certainly happy to investigate if someone can provide a way for
me to make it happen on demand.

It doesn't make much sense to me that holding the lock alone could be
causing any sort of corruption - that's just an fcntl() lock.

I would suggest to make sure you're running Xapian 1.4.7 as that fixed a
cursor handling bug which affected notmuch. I didn't find a way to make
it corrupt on-disk data, but it's hard to be completely certain that it
couldn't ever do that, so ruling out that as a cause would be good.

Post by David Bremner
notmuch could be cleverer about timing out on trying to acquire a
lock. I suspect it's a bit delicate to get that right, and I've been
hoping the underlying primitives would get a bit more flexible
w.r.t. locking.

You mean in Xapian? If so, a wishlist bug saying what you're hoping
for might help it happen.

Cheers,
Olly

[David Bremner]

Post by Olly Betts
You mean in Xapian? If so, a wishlist bug saying what you're hoping
for might help it happen.
Cheers,
Olly

I filed

https://trac.xapian.org/ticket/769#ticket

Maybe I'm overthinking it and I should just impliment some kind of loop
around a try catch block to do the open. For some reason I had the
impression that was a bit tricky to get right.

[Mueen Nawaz]

Post by Olly Betts
It doesn't make much sense to me that holding the lock alone
could be causing any sort of corruption - that's just an fcntl()
lock.
I would suggest to make sure you're running Xapian 1.4.7 as that
fixed a cursor handling bug which affected notmuch. I didn't
find a way to make it corrupt on-disk data, but it's hard to be
completely certain that it couldn't ever do that, so ruling out
that as a cause would be good.

I was running 1.4.5 - maybe that's the cause?