Why you should set redirect_urls = True

pypicloud using S3/Cloudfront will generate signed urls for your clients to download. When you run pip install <package> it will hit the /simple/<package> endpoint and attempt to render urls for all versions of that package. That will look like this:

<a href="https://bucket.s3.amazonaws.com/package-0.1.tar.gz?Signature=SIGNATURE">package-0.1.tar.gz</a><br>
<a href="https://bucket.s3.amazonaws.com/package-0.2.tar.gz?Signature=SIGNATURE">package-0.2.tar.gz</a><br>
<a href="https://bucket.s3.amazonaws.com/package-0.3.tar.gz?Signature=SIGNATURE">package-0.3.tar.gz</a><br>

If you have a lot of versions of that package, that’s a lot of cryptographic signatures that have to be run just for one pip install. It used to be that boto used M2Crypto for these signatures, but then this pull request landed which changed it to use rsa, a pure-python library that’s easier to install.

It has some advantages, but speed is not one of them. Signing all of these urls can now take an obscenely long time.

Solution: Why don’t we just render dummy urls in the /simple/<package> endpoint that will then return a HTTP redirect to the signed S3 url? Then we only have to sign one url per pip install.

Problem: Because legacy code is the worst thing in the world. For reasons that I am unable/unwilling to fully debug, easy_install cannot handle that. It just can’t.

So to compromise I added the storage.redirect_urls option. When set to true, it will generate redirect urls instead of signed S3 urls at the /simple endpoint. This is much much faster, but breaks for easy_install.

Please, please stop using easy_install. Just stop.