Installing Scrapy 1.1 On Ubuntu 15.10

I don’t do a huge amount of Python, but when I do, it usually involves Scrapy.

As a result of not doing a lot of Python, it usually means I don’t have all the relevant gubbins (yes, that is the correct technical description) installed to make things ‘just work ™’.

On the contrary, things usually go quite wrong indeed:

Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-I6du0W/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-hwGRh6-record/install-record.txt --single-version-externally-managed --compile --user failed with error code 1 in /tmp/pip-build-I6du0W/lxml

or

Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-dLB7U_/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-KNAq8K-record/install-record.txt --single-version-externally-managed --compile --user failed with error code 1 in /tmp/pip-build-dLB7U_/lxml
Storing debug log for failure in /home/chris/.pip/pip.log

and so on.

Now, also, I am not the world’s foremost expert on Linux, and / or the package dependencies required to get Python to play nicely.

This is a polite way of saying I blindly follow the top answers on StackOverflow until whatever it is I am trying to install (e.g. Scrapy) installs without some horrendous error message. Hey, a little brutal honesty never hurt anybody.

So, to save my future sanity, here are all the things I ran today to get Scrapy 1.1 installed on Ubuntu 15.10:

➜  ~ python --version
Python 2.7.10

➜  ~ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 15.10
Release:	15.10
Codename:	wily

You will need pip installed. (sudo apt-get install python-pip worked for me.)

Then this – I am aware there are duplicates, this is a concat of about 6 different attempts. Feel free to de-dupe if you want, it makes no difference (afaik):

sudo apt-get install build-essential autoconf libtool pkg-config python-opengl python-imaging python-pyrex python-pyside.qtopengl idle-python2.7 qt4-dev-tools qt4-designer libqtgui4 libqtcore4 libqt4-xml libqt4-test libqt4-script libqt4-network libqt4-dbus python-qt4 python-qt4-gl libgle3 python-dev libxml2-dev libxslt1-dev python-dev libxml2-dev libxslt1-dev python-dev libxml2-dev libxslt-dev python-dev lib32z1-dev build-essential libssl-dev libffi-dev python-dev

And then Scrapy installed ‘first time’ using:

pip install Scrapy

Marvellous.

If you’d like to learn more Python, I’m not the best person to speak to. I would highly recommend you check out one or more of the courses listed at CourseDuck.