Unable to install wheel containing .data/headers using from non-writable pip installation into writable --prefix

[pelson]

With the following scenario:

• pip installed in a virtual environment, not owned by the executing user
• pip install --prefix to a location that is owned by the user

pip is able to install simple wheels, but fails to install wheels with header .data (didn't check for other types of data)

I have confirmed that this is still the case on master with the following Dockerfile (placed in the pip repository root in order to copy the source and install pip):

FROM python

RUN python -m venv /opt/venv

# Install the latest pip (from source) into both the base Python, and the venv we just created.
COPY . /tmp/pip-repo
RUN python -m pip install /tmp/pip-repo
RUN /opt/venv/bin/python -m pip install /tmp/pip-repo

RUN useradd -ms /bin/bash not_root
USER not_root

# Fine
RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable wheel

# Fine 
RUN python -m pip install --prefix $HOME/writable pybind11

# Not fine as it attempts to write to {venv}/include/site
RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable pybind11

The output of building this Dockerfile with docker build --network=host . is:

$ git rev-parse HEAD
09f9b0030d0a3fe98cfe14e6fd9511100c3155e3

$ docker build --network=host .
Sending build context to Docker daemon  517.3MB
Step 1/10 : FROM python
 ---> b55669b4130e
Step 2/10 : RUN python -m venv /opt/venv
 ---> Using cache
 ---> 12c12f10ec01
Step 3/10 : COPY . /tmp/pip-repo
 ---> 6a2a8b641e26
Step 4/10 : RUN python -m pip install /tmp/pip-repo
 ---> Running in 50f5d4a53d52
Processing /tmp/pip-repo
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517): started
  Building wheel for pip (PEP 517): finished with status 'done'
  Created wheel for pip: filename=pip-20.2.dev1-py2.py3-none-any.whl size=1504387 sha256=c3f4a919e6797f6f477ab912ddc6567f8f7f845b10eb84a09e57b1f6ca612204
  Stored in directory: /root/.cache/pip/wheels/77/78/ae/d6e97e1b65b04ed7f7f18819f65a629c406fdfab1661304f08
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-20.2.dev1
Removing intermediate container 50f5d4a53d52
 ---> 62acdf090a08
Step 5/10 : RUN /opt/venv/bin/python -m pip install /tmp/pip-repo
 ---> Running in 269e4a89a909
Processing /tmp/pip-repo
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517): started
  Building wheel for pip (PEP 517): finished with status 'done'
  Created wheel for pip: filename=pip-20.2.dev1-cp38-none-any.whl size=1504387 sha256=6646a166c762c1b13bf407b81a5c428d1df949fd36b83a5c4f9b02ef43d9616d
  Stored in directory: /root/.cache/pip/wheels/3a/92/63/e66cd23efd91ab6b7552fc37293c6076634bd7c6e5925abf50
Successfully built pip
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.2.dev1
Removing intermediate container 269e4a89a909
 ---> d2b0273bd03d
Step 6/10 : RUN useradd -ms /bin/bash not_root
 ---> Running in dd9e454eed3d
Removing intermediate container dd9e454eed3d
 ---> e6ea8e7d10c6
Step 7/10 : USER not_root
 ---> Running in 17075cdc597b
Removing intermediate container 17075cdc597b
 ---> 09dae5d7fe5b
Step 8/10 : RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable wheel
 ---> Running in 252992c43507
Collecting wheel
  Downloading wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Installing collected packages: wheel
  WARNING: The script wheel is installed in '/home/not_root/writable/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed wheel-0.34.2
Removing intermediate container 252992c43507
 ---> 34705392a77b
Step 9/10 : RUN python -m pip install --prefix $HOME/writable pybind11
 ---> Running in 3d7d6e35d52a
Collecting pybind11
  Downloading pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
Installing collected packages: pybind11
Successfully installed pybind11-2.5.0
Removing intermediate container 3d7d6e35d52a
 ---> 495db85a9fdb
Step 10/10 : RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable pybind11
 ---> Running in f0287949ea8f
Collecting pybind11
  Using cached pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
Installing collected packages: pybind11
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/opt/venv/include/site'
Consider using the `--user` option or check the permissions.

The command '/bin/sh -c /opt/venv/bin/python -m pip install --prefix $HOME/writable pybind11' returned a non-zero code: 1

The full traceback from this command is:

Step 10/10 : RUN /opt/venv/bin/python -m pip -vvv install --prefix $HOME/writable pybind11
 ---> Running in fcd054a27042
Non-user install due to --prefix or --target option
Created temporary directory: /tmp/pip-ephem-wheel-cache-4jufpvad
Created temporary directory: /tmp/pip-req-tracker-4g9cr9c4
Initialized build tracking at /tmp/pip-req-tracker-4g9cr9c4
Created build tracker: /tmp/pip-req-tracker-4g9cr9c4
Entered build tracker: /tmp/pip-req-tracker-4g9cr9c4
Created temporary directory: /tmp/pip-install-c8x570pg
1 location(s) to search for versions of pybind11:
* https://pypi.org/simple/pybind11/
Fetching project page and analyzing links: https://pypi.org/simple/pybind11/
Getting page https://pypi.org/simple/pybind11/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pybind11/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pybind11/ HTTP/1.1" 304 0
  Found link 
... snip ...
Given no hashes to check 46 links for project 'pybind11': discarding no candidates
Using version 2.5.0 (newest of versions: 1.0, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8, 1.8.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.3.0, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0)
Collecting pybind11
  Created temporary directory: /tmp/pip-unpack-w_3s0md1
  Looking up "https://files.pythonhosted.org/packages/89/e3/d576f6f02bc75bacbc3d42494e8f1d063c95617d86648dba243c2cb3963e/pybind11-2.5.0-py2.py3-none-any.whl" in the cache
  Current age based on date: 2
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 2
  Using cached pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
  Added pybind11 from https://files.pythonhosted.org/packages/89/e3/d576f6f02bc75bacbc3d42494e8f1d063c95617d86648dba243c2cb3963e/pybind11-2.5.0-py2.py3-none-any.whl#sha256=205d9f9615eac190811cb8c3c2b2f95f6844ddba0fa0a1d45d00793338741601 to build tracker '/tmp/pip-req-tracker-4g9cr9c4'
  Removed pybind11 from https://files.pythonhosted.org/packages/89/e3/d576f6f02bc75bacbc3d42494e8f1d063c95617d86648dba243c2cb3963e/pybind11-2.5.0-py2.py3-none-any.whl#sha256=205d9f9615eac190811cb8c3c2b2f95f6844ddba0fa0a1d45d00793338741601 from build tracker '/tmp/pip-req-tracker-4g9cr9c4'
Installing collected packages: pybind11
  Created temporary directory: /tmp/pip-unpacked-wheel-m2gac4cg

ERROR: Could not install packages due to an EnvironmentError.
Consider using the `--user` option or check the permissions.
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 393, in run
    installed = install_given_reqs(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
    requirement.install(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 819, in install
    install_wheel(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 683, in install_wheel
    install_unpacked_wheel(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 497, in install_unpacked_wheel
    clobber(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 404, in clobber
    ensure_dir(dest)  # common for the 'include' path
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 111, in ensure_dir
    os.makedirs(path)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/venv/include/site'
Removed build tracker: '/tmp/pip-req-tracker-4g9cr9c4'
The command '/bin/sh -c /opt/venv/bin/python -m pip -vvv install --prefix $HOME/writable pybind11' returned a non-zero code: 1

[uranusjr]

From a quick glance, it seems like something in the wheel installation logic does not correctly respect --prefix, and wants to install into the “real” scheme of the host Python. I’m guessing similar flags like --target may also have the same issue.

[chrahunt]

Here we unconditionally use sys.prefix when calculating the header path when running under virtualenv (or venv). That's definitely not the right thing to do when someone provides --prefix explicitly. I don't have a good answer for what we should do, though.

[uranusjr]

The prefix argument looks suspecious…