forked from python/peps
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathpep-0406.txt
275 lines (199 loc) · 10.2 KB
/
pep-0406.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
PEP: 406
Title: Improved Encapsulation of Import State
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <[email protected]>, Greg Slodkowicz <[email protected]>
Status: Withdrawn
Type: Standards Track
Content-Type: text/x-rst
Created: 4-Jul-2011
Python-Version: 3.4
Post-History: 31-Jul-2011, 13-Nov-2011, 4-Dec-2011
Abstract
========
This PEP proposes the introduction of a new 'ImportEngine' class as part of
``importlib`` which would encapsulate all state related to importing modules
into a single object. Creating new instances of this object would then provide
an alternative to completely replacing the built-in implementation of the
import statement, by overriding the ``__import__()`` function. To work with
the builtin import functionality and importing via import engine objects,
this PEP proposes a context management based approach to temporarily replacing
the global import state.
The PEP also proposes inclusion of a ``GlobalImportEngine`` subclass and a
globally accessible instance of that class, which "writes through" to the
process global state. This provides a backwards compatible bridge between the
proposed encapsulated API and the legacy process global state, and allows
straightforward support for related state updates (e.g. selectively
invalidating path cache entries when ``sys.path`` is modified).
PEP Withdrawal
==============
The import system has seen substantial changes since this PEP was originally
written, as part of :pep:`420` in Python 3.3 and :pep:`451` in Python 3.4.
While providing an encapsulation of the import state is still highly
desirable, it is better tackled in a new PEP using PEP 451 as a foundation,
and permitting only the use of PEP 451 compatible finders and loaders (as
those avoid many of the issues of direct manipulation of global state
associated with the previous loader API).
Rationale
=========
Currently, most state related to the import system is stored as module level
attributes in the ``sys`` module. The one exception is the import lock, which
is not accessible directly, but only via the related functions in the ``imp``
module. The current process global import state comprises:
* sys.modules
* sys.path
* sys.path_hooks
* sys.meta_path
* sys.path_importer_cache
* the import lock (imp.lock_held()/acquire_lock()/release_lock())
Isolating this state would allow multiple import states to be
conveniently stored within a process. Placing the import functionality
in a self-contained object would also allow subclassing to add additional
features (e.g. module import notifications or fine-grained control
over which modules can be imported). The engine would also be
subclassed to make it possible to use the import engine API to
interact with the existing process-global state.
The namespace PEPs (especially PEP 402) raise a potential need for
*additional* process global state, in order to correctly update package paths
as ``sys.path`` is modified.
Finally, providing a coherent object for all this state makes it feasible to
also provide context management features that allow the import state to be
temporarily substituted.
Proposal
========
We propose introducing an ImportEngine class to encapsulate import
functionality. This includes an ``__import__()`` method which can
be used as an alternative to the built-in ``__import__()`` when
desired and also an ``import_module()`` method, equivalent to
``importlib.import_module()`` [3]_.
Since there are global import state invariants that are assumed and should be
maintained, we introduce a ``GlobalImportState`` class with an interface
identical to ``ImportEngine`` but directly accessing the current global import
state. This can be easily implemented using class properties.
Specification
=============
ImportEngine API
~~~~~~~~~~~~~~~~
The proposed extension consists of the following objects:
``importlib.engine.ImportEngine``
``from_engine(self, other)``
Create a new import object from another ImportEngine instance. The
new object is initialised with a copy of the state in ``other``. When
called on ``importlib engine.sysengine``, ``from_engine()`` can be
used to create an ``ImportEngine`` object with a **copy** of the
global import state.
``__import__(self, name, globals={}, locals={}, fromlist=[], level=0)``
Reimplementation of the builtin ``__import__()`` function. The
import of a module will proceed using the state stored in the
ImportEngine instance rather than the global import state. For full
documentation of ``__import__`` functionality, see [2]_ .
``__import__()`` from ``ImportEngine`` and its subclasses can be used
to customise the behaviour of the ``import`` statement by replacing
``__builtin__.__import__`` with ``ImportEngine().__import__``.
``import_module(name, package=None)``
A reimplementation of ``importlib.import_module()`` which uses the
import state stored in the ImportEngine instance. See [3]_ for a full
reference.
``modules, path, path_hooks, meta_path, path_importer_cache``
Instance-specific versions of their process global ``sys`` equivalents
``importlib.engine.GlobalImportEngine(ImportEngine)``
Convenience class to provide engine-like access to the global state.
Provides ``__import__()``, ``import_module()`` and ``from_engine()``
methods like ``ImportEngine`` but writes through to the global state
in ``sys``.
To support various namespace package mechanisms, when ``sys.path`` is altered,
tools like ``pkgutil.extend_path`` should be used to also modify other parts
of the import state (in this case, package ``__path__`` attributes). The path
importer cache should also be invalidated when a variety of changes are made.
The ``ImportEngine`` API will provide convenience methods that automatically
make related import state updates as part of a single operation.
Global variables
~~~~~~~~~~~~~~~~
``importlib.engine.sysengine``
A precreated instance of ``GlobalImportEngine``. Intended for use by
importers and loaders that have been updated to accept optional ``engine``
parameters and with ``ImportEngine.from_engine(sysengine)`` to start with
a copy of the process global import state.
No changes to finder/loader interfaces
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Rather than attempting to update the PEP 302 APIs to accept additional state,
this PEP proposes that ``ImportEngine`` support the content management
protocol (similar to the context substitution mechanisms in the ``decimal``
module).
The context management mechanism for ``ImportEngine`` would:
* On entry:
* Acquire the import lock
* Substitute the global import state with the import engine's own state
* On exit:
* Restore the previous global import state
* Release the import lock
The precise API for this is TBD (but will probably use a distinct context
management object, along the lines of that created by
``decimal.localcontext``).
Open Issues
===========
API design for falling back to global import state
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The current proposal relies on the ``from_engine()`` API to fall back to the
global import state. It may be desirable to offer a variant that instead falls
back to the global import state dynamically.
However, one big advantage of starting with an "as isolated as possible"
design is that it becomes possible to experiment with subclasses that blur
the boundaries between the engine instance state and the process global state
in various ways.
Builtin and extension modules must be process global
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Due to platform limitations, only one copy of each builtin and extension
module can readily exist in each process. Accordingly, it is impossible for
each ``ImportEngine`` instance to load such modules independently.
The simplest solution is for ``ImportEngine`` to refuse to load such modules,
raising ``ImportError``. ``GlobalImportEngine`` would be able to load them
normally.
``ImportEngine`` will still return such modules from a prepopulated module
cache - it's only loading them directly which causes problems.
Scope of substitution
~~~~~~~~~~~~~~~~~~~~~
Related to the previous open issue is the question of what state to substitute
when using the context management API. It is currently the case that replacing
``sys.modules`` can be unreliable due to cached references and there's the
underlying fact that having independent copies of some modules is simply
impossible due to platform limitations.
As part of this PEP, it will be necessary to document explicitly:
* Which parts of the global import state can be substituted (and declare code
which caches references to that state without dealing with the substitution
case buggy)
* Which parts must be modified in-place (and hence are not substituted by the
``ImportEngine`` context management API, or otherwise scoped to
``ImportEngine`` instances)
Reference Implementation
========================
A reference implementation [4]_ for an earlier draft of this PEP, based on
Brett Cannon's importlib has been developed by Greg Slodkowicz as part of the
2011 Google Summer of Code. Note that the current implementation avoids
modifying existing code, and hence duplicates a lot of things unnecessarily.
An actual implementation would just modify any such affected code in place.
That earlier draft of the PEP proposed change the PEP 302 APIs to support passing
in an optional engine instance. This had the (serious) downside of not correctly
affecting further imports from the imported module, hence the change to the
context management based proposal for substituting the global state.
References
==========
.. [1] PEP 302, New Import Hooks, J van Rossum, Moore
(http://www.python.org/dev/peps/pep-0302)
.. [2] __import__() builtin function, The Python Standard Library documentation
(http://docs.python.org/library/functions.html#__import__)
.. [3] Importlib documentation, Cannon
(http://docs.python.org/dev/library/importlib)
.. [4] Reference implementation
(https://bitbucket.org/jergosh/gsoc_import_engine/src/default/Lib/importlib/engine.py)
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: