[issue20074] open() of read-write non-seekable streams broken

[Dolda2000]

New submission from Dolda2000:

It seems open() is slightly broken in Python 3, in that one cannot open non-seekable files in read-write mode. One such common use is open("/dev/tty", "r+") for interacting directly with the controlling TTY regardless of standard stream redirections. Note that this is a regression for Python 2, where this worked as expected.

open("/dev/tty", "r+")

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
io.UnsupportedOperation: File or stream is not seekable.

Just for the record, the same thing happens with "w+" and "rb+".

This also means that the getpass module is slightly broken, since it will always fail whenever stdin is redirected.

----------
components: IO
messages: 206957
nosy: Dolda2000
priority: normal
severity: normal
status: open
title: open() of read-write non-seekable streams broken
type: behavior
versions: Python 3.1, Python 3.3

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[R. David Murray]

R. David Murray added the comment:

I don't think getpass will fail, since it uses os.open, not open.

Presumably you can work around the problem by opening the stream twice: once for reading and once for writing. I'll leave it to Antoine to say whether or not that is in fact the expected solution, or if this is a design bug.

----------
nosy: +pitrou, r.david.murray
versions: +Python 3.4 -Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Use unbuffered binary file.

open("/dev/tty", "r+b", buffering=0)

<_io.FileIO name='/dev/tty' mode='rb+'>

----------
nosy: +serhiy.storchaka

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Dolda2000]

Post by R. David Murray
I don't think getpass will fail, since it uses os.open, not open.

It also uses fdopen() on the resulting file descriptor, however, which has the same problem.

Post by R. David Murray
Use unbuffered binary file.

It's nice that that's at least possible, but I, for one, would still consider it a bug that it isn't possible to open it in text mode.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Dolda2000]

Dolda2000 added the comment:

Just to demonstrate failure of getpass, by the way:

$ cat >/tmp/pwtest.py
import getpass
print(getpass.getpass())
$ python3 /tmp/pwtest.py </dev/null
/usr/lib/python3.3/getpass.py:83: GetPassWarning: Can not control echo on the terminal.
passwd = fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
Password: Traceback (most recent call last):
File "/usr/lib/python3.3/getpass.py", line 63, in unix_getpass
old = termios.tcgetattr(fd) # a copy to save
termios.error: (25, 'Inappropriate ioctl for device')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/pwtest.py", line 2, in <module>
print(getpass.getpass())
File "/usr/lib/python3.3/getpass.py", line 83, in unix_getpass
passwd = fallback_getpass(prompt, stream)
File "/usr/lib/python3.3/getpass.py", line 118, in fallback_getpass
return _raw_input(prompt, stream)
File "/usr/lib/python3.3/getpass.py", line 134, in _raw_input
raise EOFError
EOFError

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Serhiy Storchaka]

Post by Dolda2000
It's nice that that's at least possible, but I, for one, would still consider it a bug that it isn't possible to open it in text mode.

io.TextIOWrapper(open("/dev/tty", "r+b", buffering=0))

<_io.TextIOWrapper name='/dev/tty' encoding='UTF-8'>
Looks as this was fixed in issue18116 for 3.4. David, perhaps issue18116 should be backported to older Python versions.

----------
nosy: +benjamin.peterson, hynek, stutzbach

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Dolda2000]

Dolda2000 added the comment:

Python 3.3.3 (default, Dec 8 2013, 14:51:59)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

import io
io.TextIOWrapper(open("/dev/tty", "rb+"))

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
io.UnsupportedOperation: File or stream is not seekable.

Was this also fixed in 3.4?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Serhiy Storchaka]

Post by Dolda2000

import io
io.TextIOWrapper(open("/dev/tty", "rb+"))

File "<stdin>", line 1, in <module>
io.UnsupportedOperation: File or stream is not seekable.

buffering=0

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Dolda2000]

Dolda2000 added the comment:

Oh sorry, my bad. I messed up. :)

Given that that works, though, why can't open() handle opening /dev/tty directly in text mode? Clearly, TextIOWrapper can handle the necessary buffering without the stream having to be seekable.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Roundup Robot]

Roundup Robot added the comment:

New changeset 100f632d4306 by R David Murray in branch '3.3':
#18116: backport fix to 3.3 since real-world failure mode demonstrated.
http://hg.python.org/cpython/rev/100f632d4306

----------
nosy: +python-dev

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[R. David Murray]

R. David Murray added the comment:

Having buffering doesn't make the stream seekable. So the question is, is the *design* of the IO module that '+' requires a seekable stream the best behavior, or can that constraint be relaxed? You have to keep in mind that the IO module is a bunch of building blocks, which are plugged together automatically for the most common scenarios. The goal is a portable, consistent IO system, not one that completely mimics unix/C IO primitives.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Antoine Pitrou]

Post by R. David Murray
Having buffering doesn't make the stream seekable. So the question
is, is the *design* of the IO module that '+' requires a seekable
stream the best behavior, or can that constraint be relaxed?

A non-seekable read/write stream doesn't really make sense (think about
it).
What you may be thinking about, instead, is a pair of non-seekable
streams, one readable and one writable. There is BufferedRWPair for
that:
http://docs.python.org/dev/library/io.html#io.BufferedRWPair

(granted, BufferedRWPair isn't wired in open(), so you have to do all
the wrapping yourself)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Dolda2000]

Post by R. David Murray
So the question is, is the *design* of the IO module that '+' requires a seekable stream the best behavior, or can that constraint be relaxed?

What purpose does that constraint serve? Is there any reason it shouldn't be relaxed?

It seems to work quite well without it in Python 2.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[R. David Murray]

R. David Murray added the comment:

Antoine already answered that question: it does not make sense to have a single stream that is open for *update* if it is not seekable. The fact C conflates "update" with "both read and write" can be seen as a design bug in C :)

The remaining question might be: is there a sensible way (that fits with the design of the IO system) to hook BufferedRWPair up to open?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________

[Martin Panter]

Changes by Martin Panter <vadmium+py at gmail.com>:


----------
nosy: +vadmium

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20074>
_______________________________________