[issue25498] Python 3.4.3 core dump with simple sample code

[Jake Montgomery]

New submission from Jake Montgomery:

We are seeing a consistent "invalid memory access" crash in Python3.4.3.

*Test setup*
This is occurring on Ubuntu 14.04.1 LTS. It is occurring on multiple unrelated installs, but has not been tested with any other operating system. It occurs using the "Python 3.4.3 (default, Oct 14 2015, 20:28:29)" that is installed using apt-get. It does *not* appear to occur using Python 3.4.0. Other versions of python were not tested.

*Reproducing*
I was able to reduce the code needed to a pretty minimal python program, which is attached. Running this as "pyton3 ./pythoncrash.py" will trigger the crash, and a core dump if enabled.

If run under python3.4.3-dbg The error is:

python3.4-dbg: ../Objects/memoryobject.c:115: mbuf_dealloc: Assertion `self->exports == 0' failed.

In some versions of this code, it had to be run twice, since it would only occur when the pyc files were already created. The attached version does not appear to have this behavior, but run it twice, at least, if you do not see the crash.

The code supplied is "minimal", in that removing any line will cause the crash not to occur.

*More clues*
I suspect that the key is the multiprocessing.Value object, and its destruction. The other "import" statements may just be happenstance needed to create the correct memory conditions for the crash. I was able to start "stubbing out" the actual code in those imported packages, and the crash continued as long as the number of functions, classes, and class dependencies were maintained. But that became too tedious. In earlier versions of the test code, adding or removing significant numbers of function or classes could make the crash not manifest.

*Stack trace*
Running under python3.4.3-dbg, gdb gives the following stack trace:

#0 0x00007fc0425b4cc9 in __GI_raise (sig=***@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fc0425b80d8 in __GI_abort () at abort.c:89
#2 0x00007fc0425adb86 in __assert_fail_base (
fmt=0x7fc0426fe830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=***@entry=0x707199 "self->exports == 0",
file=***@entry=0x70715a "../Objects/memoryobject.c", line=***@entry=115,
function=***@entry=0x708923 <__PRETTY_FUNCTION__.10355> "mbuf_dealloc") at assert.c:92
#3 0x00007fc0425adc32 in __GI___assert_fail (assertion=0x707199 "self->exports == 0",
file=0x70715a "../Objects/memoryobject.c", line=115,
function=0x708923 <__PRETTY_FUNCTION__.10355> "mbuf_dealloc") at assert.c:101
#4 0x00000000004beb92 in mbuf_dealloc (self=0x7fc042068058) at ../Objects/memoryobject.c:115
#5 0x00000000004cd67b in _Py_Dealloc (op=<managedbuffer at remote 0x7fc042068058>)
at ../Objects/object.c:1749
#6 0x00000000004c0ec6 in memory_clear (self=0x7fc042026bf8) at ../Objects/memoryobject.c:1079
#7 0x00000000004409c3 in delete_garbage (collectable=0x7fff21065180,
old=0x9a4f40 <generations+64>) at ../Modules/gcmodule.c:866
#8 0x0000000000440f56 in collect (generation=2, n_collected=0x7fff210651e8,
n_uncollectable=0x7fff210651f0, nofail=0) at ../Modules/gcmodule.c:1032
#9 0x000000000044144e in collect_with_callback (generation=2) at ../Modules/gcmodule.c:1140
#10 0x00000000004421f2 in PyGC_Collect () at ../Modules/gcmodule.c:1616
#11 0x00000000004226a0 in Py_Finalize () at ../Python/pythonrun.c:607
#12 0x0000000000428f38 in Py_Exit (sts=0) at ../Python/pythonrun.c:2713
#13 0x0000000000426077 in handle_system_exit () at ../Python/pythonrun.c:1812
#14 0x00000000004260a2 in PyErr_PrintEx (set_sys_last_vars=1) at ../Python/pythonrun.c:1822
#15 0x0000000000425cef in PyErr_Print () at ../Python/pythonrun.c:1718
#16 0x00000000004255c4 in PyRun_SimpleFileExFlags (fp=0x29824c0,
filename=0x7fc042442b68 "./pythoncrash.py", closeit=1, flags=0x7fff21065520)
at ../Python/pythonrun.c:1611
#17 0x0000000000424418 in PyRun_AnyFileExFlags (fp=0x29824c0,
filename=0x7fc042442b68 "./pythoncrash.py", closeit=1, flags=0x7fff21065520)
at ../Python/pythonrun.c:1292
#18 0x000000000043e9fb in run_file (fp=0x29824c0, filename=0x28b62f0 L"./pythoncrash.py",
p_cf=0x7fff21065520) at ../Modules/main.c:319
#19 0x000000000043f781 in Py_Main (argc=2, argv=0x28b5020) at ../Modules/main.c:751
#20 0x000000000041e6d6 in main (argc=2, argv=0x7fff21065798) at ../Modules/python.c:69

The full core dump is available, if it would help.

----------
components: Interpreter Core
files: pythoncrash.py
messages: 253613
nosy: JakeMont
priority: normal
severity: normal
status: open
title: Python 3.4.3 core dump with simple sample code
versions: Python 3.4
Added file: http://bugs.python.org/file40876/pythoncrash.py

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[R. David Murray]

R. David Murray added the comment:

The program does not crash for me on gentoo running python 3.4 tip (--with-pydebug). It does crash running gentoo's stock 3.4.3. So, perhaps this bug has already been fixed.

----------
nosy: +r.david.murray

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Zachary Ware]

Zachary Ware added the comment:

I get the crash on OSX with recent builds of 3.4 and 3.6:

Assertion failed: (self->exports == 0), function mbuf_dealloc, file Objects/memoryobject.c, line 115.
Abort trap: 6

----------
nosy: +jnoller, sbt, zach.ware
stage: -> needs patch
type: -> crash
versions: +Python 3.5, Python 3.6

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Martin Panter]

Martin Panter added the comment:

Crashes on Linux x86-64 for me with and without --with-pydebug, with the default 3.6 branch, 3.5.0, and 3.4 tip. No extra debugging output though in the --with-pydebug cases though.

----------
nosy: +martin.panter

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Martin Panter]

Martin Panter added the comment:

I traced this down to part of revision 1da9630e9b7f (Issue #22896). If I revert the changes to CDataType_from_buffer() at <https://hg.python.org/cpython/rev/1da9630e9b7f/#l4.3>, the crash no longer happens. I suspect the offending code is this, that is trying to stash the original buffer protocol supporting object into a memory view:

mv = PyMemoryView_FromBuffer(&buffer);
if (mv == NULL) {
PyBuffer_Release(&buffer);
return NULL;
}
/* Hack the memoryview so that it will release the buffer. */
((PyMemoryViewObject *)mv)->mbuf->master.obj = buffer.obj;
((PyMemoryViewObject *)mv)->view.obj = buffer.obj;
//~ Py_INCREF(buffer.obj);
if (-1 == KeepRef((CDataObject *)result, -1, mv))
result = NULL;

If I enable my INCREF() line it also stops the crash, but I guess at the expense of a memory leak.

----------
components: +ctypes -Interpreter Core
keywords: +3.4regression
nosy: +serhiy.storchaka

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[eryksun]

Changes by eryksun <***@gmail.com>:


Added file: http://bugs.python.org/file40883/ctypes_crash.py

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[eryksun]

Changes by eryksun <***@gmail.com>:


----------
keywords: +patch
Added file: http://bugs.python.org/file40884/ctypes_from_buffer_1.patch

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[eryksun]

Changes by eryksun <***@gmail.com>:


Removed file: http://bugs.python.org/file40884/ctypes_from_buffer_1.patch

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[eryksun]

Changes by eryksun <***@gmail.com>:


Added file: http://bugs.python.org/file40890/ctypes_from_buffer_2.patch

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Serhiy Storchaka]

Changes by Serhiy Storchaka <***@gmail.com>:


----------
assignee: -> serhiy.storchaka
priority: normal -> high
stage: needs patch -> patch review

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Stefan Krah]

Stefan Krah added the comment:

We should ultimately do something like I suggested in msg235256.

Everything else is a hack (N.B.: the issues that 1da9630e9b7f
tried to fix were also hacks, so it isn't really the fault of
that commit).

----------
nosy: +skrah

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Martin Panter]

Martin Panter added the comment:

Before the Issue 22896 changes, PyObject_AsWriteBuffer() was used, which requests a buffer with the PyBUF_WRITABLE flag. Currently we use “w*” argument parsing format, which also uses PyBUF_WRITABLE. (Incidentally, I suspect the C-contiguity check is redundant for “w*”; non-contiguous buffers trigger the “read-write” error instead.)

Now Eryksun’s patch changes to PyMemoryView_FromObject(), which requests a buffer with the PyBUF_FULL_RO flag (the most liberal), and only then checks for writability and contiguity. Could this be a problem for some kind of object that returns different buffers depending on the request flags?

I agree that the existing buffer and memory view APIs don’t seem to be very practical and well understood. PyMemoryView_FromBuffer() is also used in the “io” module. It doesn’t even try to save a reference to the underlying buffers, meaning it is possible for Python code to write into freed memory by saving the memory view object passed to the readinto() method.

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Before committing any solution we first should have understood the cause of the crash. The peculiarity of example ctypes_crash.py is that the argument of ctypes.c_char.from_buffer is a memoryview. With current code we have following chain (the list creates a reference loop with itself):

list->memoryview->ManagedBuffer->memoryview->ManagedBuffer->bytearray

Using PyMemoryView_FromObject or hypothetical PyMemoryView_FromObjectEx we could get shorter chain:

list->memoryview->ManagedBuffer->bytearray

May be the cause not in current hack, but in memoryview that can't survive with breaking long chain? Or may be we just are lucky in latter case?

I share Martin's doubts about writability in Eryksun’s patch.

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Stefan Krah]

Stefan Krah added the comment:

If memoryview B is created from memoryview A, then B must be
registered with the same ManagedBuffer as A, otherwise the
whole scheme breaks down (PyMemoryView_FromObject() performs
this check).


PyMemoryView_FromBuffer() is really a legacy function that
should be only used for creating temporary views within a
function. As far as I know, it has usually been used that
way from the start (since 3.0).

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Stefan Krah]

Stefan Krah added the comment:

Per the docs the readonly flag must be consistent for all consumers,
so checking for writability after getting the view should be okay in
principle.

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Martin Panter]

Martin Panter added the comment:

Okay, so <https://docs.python.org/dev/c-api/buffer.html#c.PyBUF_WRITABLE> says writability must be consistent. As far as I can see, there is no similar requirement for contiguity. So in theory PyBUF_FULL_RO could produce a discontiguous buffer when PyBUF_WRITABLE would have produced a contiguous one. But I guess this is rather unlikely, so Eryksun’s approach might be good enough.

I am starting to wonder why the current memory view hack should even be necessary. It seems like a memory view created by PyMemoryView_FromBuffer() will eventually do the equivalent of PyBuffer_Release(), except that it leaks a reference to the underlying Python object. This was also brought up at <https://bugs.python.org/issue15903#msg170658>, but the question is not resolved that I can see.

Possibly also related is Issue 15821, but I don’t know enough details or history to be sure.

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Stefan Krah]

Stefan Krah added the comment:

The current hack isn't necessary. I didn't review eryksun's patch
in depth, but on the surface it looks good.

I don't think the other issues you mentioned are closely related:
The cause here is that the obj fields of both the second memoryview
and the second mbuf are manually set to the address of the first
memoryview, which plays havoc with the rather complex deallocation
scheme.

I've opened #25524 in order to make it easier to create views
with detailed request flags.



As a practical matter, I think we should just revert the ctypes
change for 3.4 and 3.5 (after verifying a second time that
PyObject_AsWriteBuffer() is exactly the same as in 3.4.0).


Then we use the new function from #25524 for 3.6, which should
alleviate concerns about writablility (not all extension writers
read the docs:) etc.

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I rather inclined to commit eryksun's patch.

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Martin Panter]

b = bytearray(b"a")
c = ctypes.c_char.from_buffer(b)
b *= 1000

BufferError: Existing exports of data: object cannot be re-sized

PyObject_AsWriteBuffer() and PyBuffer_Release() did get changed in 3.4, also by Issue 22896. Reviewing the history (mainly <https://hg.python.org/cpython/diff/1da9630e9b7f/Objects/abstract.c>), the only behaviour change I see is that it now clears view->obj to NULL, via the new PyBuffer_Release() call, before calling DECREF on the object.

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Serhiy Storchaka]

Changes by Serhiy Storchaka <***@gmail.com>:


----------
assignee: serhiy.storchaka ->

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Martin Panter]

Martin Panter added the comment:

I propose this patch, based on Eryksun’s patch (thanks Eryksun). New changes:

* Added test case
* Applied Serhiy’s review suggestion
* Changed the new BufferError exceptions back to TypeError to match previous behaviour and test cases

----------
nosy: +eryksun
Added file: http://bugs.python.org/file41009/ctypes_from_buffer_3.patch

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Added new comments on Rietveld. Would be nice to add tests for read-only and non-contiguous buffers.

eryksun, what is your real name? What we have to add in the Misc/ACKS file?

----------

_______________________________________
Python tracker <***@bugs.python.org>
<http://bugs.python.org/issue25498>
_______________________________________