[issue7759] mhlib fails on Btrfs filesystem (test_mhlib failure)

[Neil Schemenauer]

New submission from Neil Schemenauer <nas-python at arctrix.com>:

Btrfs does not maintain a link count for directories (MacOS does the same I think). That confuses mhlib.py because it uses os.stat().st_nlinks as an optimization.

The attached patch removes the optimization and make test_mhlib pass on Btrfs (and probably HFS+) filesystems.

----------
files: mhlib_nlinks.txt
messages: 98169
nosy: nascheme
priority: normal
severity: normal
status: open
title: mhlib fails on Btrfs filesystem (test_mhlib failure)
type: behavior
versions: Python 2.7, Python 3.2
Added file: http://bugs.python.org/file15975/mhlib_nlinks.txt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Chris Withers]

Chris Withers <chris at simplistix.co.uk> added the comment:

Please can you write a test for your patch?

----------
nosy: +cjw296
stage: -> test needed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Antoine Pitrou]

Antoine Pitrou <pitrou at free.fr> added the comment:

The documentation mentions that mhlib is deprecated and mailbox should be used instead. Is there any point in trying to fix it?

----------
nosy: +pitrou

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Antoine Pitrou]

Changes by Antoine Pitrou <pitrou at free.fr>:


----------
stage: test needed -> patch review
versions: +Python 2.6 -Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Antoine Pitrou]

Changes by Antoine Pitrou <pitrou at free.fr>:


----------
nosy: +akuchling

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Neil Schemenauer]

Post by Antoine Pitrou
The documentation mentions that mhlib is deprecated and mailbox
should be used instead. Is there any point in trying to fix it?

It looks like Btrfs will eventually conform to traditional st_nlink
behavior. However, that still leaves HFS+. Perhaps the easiest fix
would be to have the unit test check for weird st_nlink behavior by
creating a directory with a subdirectory. If something is weird,
skip testing mhlib. The downside to that solution is that someone
might use mhlib on a HFS+ filesystem and encounter buggy behavior.

I can imagine that removing the optimization can make mhlib much
slower for large mail boxes. Maybe that would be better than
risking lost mail though. On modern machines maybe it doesn't
matter much.

Neil

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Antoine Pitrou]

Post by Neil Schemenauer

Post by Antoine Pitrou
The documentation mentions that mhlib is deprecated and mailbox
should be used instead. Is there any point in trying to fix it?

It looks like Btrfs will eventually conform to traditional st_nlink
behavior. However, that still leaves HFS+.

That wasn't really my question. What I ask is: since mhlib is
deprecated, why do we need to fix it while people are encouraged to use
mailbox instead?
And, besides, does mailbox show the same problem?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Neil Schemenauer]

Post by Antoine Pitrou
That wasn't really my question. What I ask is: since mhlib is
deprecated, why do we need to fix it while people are encouraged to use
mailbox instead?

Sorry, I don't understand what you are proposing. Do you mean we
should just let the test fail for people who develop on HFS+ and
Btrfs filesystems? That seems not so good.

Post by Antoine Pitrou
And, besides, does mailbox show the same problem?

No, it doesn't have that optimization.

Neil

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Antoine Pitrou]

Post by Neil Schemenauer
Sorry, I don't understand what you are proposing. Do you mean we
should just let the test fail for people who develop on HFS+ and
Btrfs filesystems? That seems not so good.

Hmm, you are right. From a quick glance, the patch looks ok. I assume you've checked it doesn't break anything else :)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Chris Withers]

Changes by Chris Withers <chris at simplistix.co.uk>:


----------
nosy: -cjw296

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Mark Lawrence]

Mark Lawrence <breamoreboy at yahoo.co.uk> added the comment:

Since mhlib has gone from py3k is there any interest in applying this to 2.6 or 2.7, given that there's been no response to msg98232?

----------
components: +Library (Lib)
nosy: +BreamoreBoy

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Neil Schemenauer]

Neil Schemenauer <nas-python at arctrix.com> added the comment:

Closing this bug. I don't think it makes sense to change the mhlib module in bugfix release. My patch is fairly simple but not simple enough to make me feel comfortable.

----------
resolution: -> wont fix
status: open -> closed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The new buildbot edelsohn-sles-z is red from its setting at 19 Aug 2015. test_mhlib is the only failed test.

http://buildbot.python.org/all/builders/s390x%20SLES%202.7/builds/114/steps/test/logs/stdio
======================================================================
FAIL: test_listfolders (test.test_mhlib.MhlibTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/dje/cpython-buildarea/2.7.edelsohn-sles-z/build/Lib/test/test_mhlib.py", line 185, in test_listfolders
eq(folders, tfolders)
AssertionError: Lists differ: [] != ['deep', 'deep/f1', 'deep/f2',...

Second list contains 6 additional elements.
First extra element 0:
deep

- []
+ ['deep', 'deep/f1', 'deep/f2', 'deep/f2/f3', 'inbox', 'wide']

----------------------------------------------------------------------

I think we should fix this issue. Proposed patch adds a test that we can use nlinks for count a number of subdirectories.

----------
keywords: +patch
nosy: +David.Edelsohn, serhiy.storchaka
status: closed -> open
versions: -Python 2.6
Added file: http://bugs.python.org/file41010/mhlib_nlinks_2.patch

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is even simpler and more reliable patch. It works even if the subfolder is a symlink to the directory on the filesystem that doesn't support links counting for directories.

----------
Added file: http://bugs.python.org/file41011/mhlib_nlinks_3.patch

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue7759>
_______________________________________