forked from python/peps
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathpep-0461.txt
256 lines (180 loc) · 8.27 KB
/
pep-0461.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
PEP: 461
Title: Adding % formatting to bytes and bytearray
Version: $Revision$
Last-Modified: $Date$
Author: Ethan Furman <[email protected]>
Status: Final
Type: Standards Track
Content-Type: text/x-rst
Created: 2014-01-13
Python-Version: 3.5
Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25,
2014-03-27
Resolution: https://mail.python.org/pipermail/python-dev/2014-March/133621.html
Abstract
========
This PEP proposes adding % formatting operations similar to Python 2's ``str``
type to ``bytes`` and ``bytearray`` [1]_ [2]_.
Rationale
=========
While interpolation is usually thought of as a string operation, there are
cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the
work needed to make up for this missing functionality detracts from the overall
readability of the code.
Motivation
==========
With Python 3 and the split between ``str`` and ``bytes``, one small but
important area of programming became slightly more difficult, and much more
painful -- wire format protocols [3]_.
This area of programming is characterized by a mixture of binary data and
ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a
restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in
writing new wire format code, and in porting Python 2 wire format code.
Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
formats, and ``FTP`` and ``HTTP`` communications, among many others.
Proposed semantics for ``bytes`` and ``bytearray`` formatting
=============================================================
%-interpolation
---------------
All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``,
``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently added
to Python 3) will be supported, and will work as they do for str, including
the padding, justification and other related modifiers (currently ``#``, ``0``,
``-``, `` `` (space), and ``+`` (plus any added to Python 3)). The only
non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a
synonym for b).
For the numeric codes, the only difference between ``str`` and ``bytes`` (or
``bytearray``) interpolation is that the results from these codes will be
ASCII-encoded text, not unicode. In other words, for any numeric formatting
code `%x`::
b"%x" % val
is equivalent to::
("%x" % val).encode("ascii")
Examples::
>>> b'%4x' % 10
b' a'
>>> b'%#4x' % 10
' 0xa'
>>> b'%04X' % 10
'000A'
``%c`` will insert a single byte, either from an ``int`` in range(256), or from
a ``bytes`` argument of length 1, not from a ``str``.
Examples::
>>> b'%c' % 48
b'0'
>>> b'%c' % b'a'
b'a'
``%b`` will insert a series of bytes. These bytes are collected in one of two
ways:
- input type supports ``Py_buffer`` [4]_?
use it to collect the necessary bytes
- input type is something else?
use its ``__bytes__`` method [5]_ ; if there isn't one, raise a ``TypeError``
In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is rejected
as the string to bytes conversion requires an encoding, and we are refusing to
guess; numbers are rejected because:
- what makes a number is fuzzy (float? Decimal? Fraction? some user type?)
- allowing numbers would lead to ambiguity between numbers and textual
representations of numbers (3.14 vs '3.14')
- given the nature of wire formats, explicit is definitely better than implicit
``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 code
bases easier to maintain. Python 3 only code should use ``%b``.
Examples::
>>> b'%b' % b'abc'
b'abc'
>>> b'%b' % 'some string'.encode('utf8')
b'some string'
>>> b'%b' % 3.14
Traceback (most recent call last):
...
TypeError: b'%b' does not accept 'float'
>>> b'%b' % 'hello world!'
Traceback (most recent call last):
...
TypeError: b'%b' does not accept 'str'
``%a`` will give the equivalent of
``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated
value. Use cases include developing a new protocol and writing landmarks
into the stream; debugging data going into an existing protocol to see if
the problem is the protocol itself or bad data; a fall-back for a serialization
format; or any situation where defining ``__bytes__`` would not be appropriate
but a readable/informative representation is needed [6]_.
``%r`` is included as a synonym for ``%a`` for the sole purpose of making 2/3
code bases easier to maintain. Python 3 only code use ``%a`` [7]_.
Examples::
>>> b'%a' % 3.14
b'3.14'
>>> b'%a' % b'abc'
b"b'abc'"
>>> b'%a' % 'def'
b"'def'"
Compatibility with Python 2
===========================
As noted above, ``%s`` and ``%r`` are being included solely to help ease
migration from, and/or have a single code base with, Python 2. This is
important as there are modules both in the wild and behind closed doors that
currently use the Python 2 ``str`` type as a ``bytes`` container, and hence
are using ``%s`` as a bytes interpolator.
However, ``%b`` and ``%a`` should be used in new, Python 3 only code, so ``%s``
and ``%r`` will immediately be deprecated, but not removed from the 3.x series
[7]_.
Proposed variations
===================
It has been proposed to automatically use ``.encode('ascii','strict')`` for
``str`` arguments to ``%b``.
- Rejected as this would lead to intermittent failures. Better to have the
operation always fail so the trouble-spot can be correctly fixed.
It has been proposed to have ``%b`` return the ascii-encoded repr when the
value is a ``str`` (b'%b' % 'abc' --> b"'abc'").
- Rejected as this would lead to hard to debug failures far from the problem
site. Better to have the operation always fail so the trouble-spot can be
easily fixed.
Originally this PEP also proposed adding format-style formatting, but it was
decided that format and its related machinery were all strictly text (aka
``str``) based, and it was dropped.
Various new special methods were proposed, such as ``__ascii__``,
``__format_bytes__``, etc.; such methods are not needed at this time, but can
be visited again later if real-world use shows deficiencies with this solution.
A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [8]_,
also exists.
Objections
==========
The objections raised against this PEP were mainly variations on two themes:
- the ``bytes`` and ``bytearray`` types are for pure binary data, with no
assumptions about encodings
- offering %-interpolation that assumes an ASCII encoding will be an
attractive nuisance and lead us back to the problems of the Python 2
``str``/``unicode`` text model
As was seen during the discussion, ``bytes`` and ``bytearray`` are also used
for mixed binary data and ASCII-compatible segments: file formats such as
``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc.
``bytes`` and ``bytearray`` already have several methods which assume an ASCII
compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name
just a few. %-interpolation, with its very restricted mini-language, will not
be any more of a nuisance than the already existing methods.
Some have objected to allowing the full range of numeric formatting codes with
the claim that decimal alone would be sufficient. However, at least two
formats (dbf and pdf) make use of non-decimal numbers.
Footnotes
=========
.. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting
.. [2] neither string.Template, format, nor str.format are under consideration
.. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html
.. [4] http://docs.python.org/3/c-api/buffer.html
examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes``
.. [5] http://docs.python.org/3/reference/datamodel.html#object.__bytes__
.. [6] https://mail.python.org/pipermail/python-dev/2014-February/132750.html
.. [7] http://bugs.python.org/issue23467 -- originally ``%r`` was not allowed,
but was added for consistency during the 3.5 alpha stage.
.. [8] http://python.org/dev/peps/pep-0460/
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: