Installing Scrapy 1.1 On Ubuntu 15.10

I don’t do a huge amount of Python, but when I do, it usually involves Scrapy.

As a result of not doing a lot of Python, it usually means I don’t have all the relevant gubbins (yes, that is the correct technical description) installed to make things ‘just work ™’.

On the contrary, things usually go quite wrong indeed:

or

and so on.

Now, also, I am not the world’s foremost expert on Linux, and / or the package dependencies required to get Python to play nicely.

This is a polite way of saying I blindly follow the top answers on StackOverflow until whatever it is I am trying to install (e.g. Scrapy) installs without some horrendous error message. Hey, a little brutal honesty never hurt anybody.

So, to save my future sanity, here are all the things I ran today to get Scrapy 1.1 installed on Ubuntu 15.10:

You will need pip installed. ( sudo apt-get install python-pip worked for me.)

Then this – I am aware there are duplicates, this is a concat of about 6 different attempts. Feel free to de-dupe if you want, it makes no difference (afaik):

And then Scrapy installed ‘first time’ using:

Marvellous.

Shares

Published by

Code Review

Code Review

CodeReviewVideos is a video training site helping software developers learn Symfony faster and easier.

2 thoughts on “Installing Scrapy 1.1 On Ubuntu 15.10”

  1. Still not good 🙁

    apt-get problems:
    Note, selecting ‘libxslt1-dev’ instead of ‘libxslt-dev’
    E: Unable to locate package lib32z1-dev

    pip install scrapy problems:
    collect2: error: ld returned 1 exit status
    error: command ‘i686-linux-gnu-gcc’ failed with exit status 1

    —————————————-
    Command “/usr/bin/python -u -c “import setuptools, tokenize;__file__=’/tmp/pip-build-TY9l_h/lxml/setup.py’;f=getattr(tokenize, ‘open’, open)(__file__);code=f.read().replace(‘\r\n’, ‘\n’);f.close();exec(compile(code, __file__, ‘exec’))” install –record /tmp/pip-P8mryP-record/install-record.txt –single-version-externally-managed –compile” failed with error code 1 in /tmp/pip-build-TY9l_h/lxml/

    1. I’m unsure how best to advise on the specific issue you are facing.

      What I would do now – if needing to use Scrapy – would be to either use an Ansible role (google for “ansible scrapy” to find plenty), or maybe use a Docker container.

      One tip would be to google for “ansible scrapy” and then even if not using Ansible, you could still follow the playbook(s) manually – e.g. https://github.com/arbabnazar/ansible-roles/blob/master/scrapy/tasks/main.yml – and see if they have any dependencies or steps you haven’t tried.

      Sorry I can’t help further, but best of luck getting it working.

Leave a Reply

Your email address will not be published. Required fields are marked *