EuroPython 2016

Ingesting 35 million hotel images with python in the cloud.

Speaker(s) Alex Vinyals

This talk covers the distributed architecture that Skyscanner built to solve the data challenges involved in the generation of images of all hotels in the world. Putting together a distributed system in Python, based on queues, surfing on the AWS Cloud.

Our goal? To build an incremental image processing pipeline that discards poor quality and duplicated images, scaling the final images to several sizes to optimise for mobile devices.

Among the challenges:

  1. Ingest all the input images that partners provide us.
  2. Detect and remove bad quality + duplicated images from reaching production.
  3. Resize all the generated images to optimise for mobile devices.
  4. Ensure the process scales and behaves in an incremental way.
  5. Ensure the whole process fits in a time constrained window.

Among the tools we used? Pillow, ImageHash, Kombu and Boto.

in on Tuesday 19 July at 16:15 See schedule

Do you have some questions on this talk?

New comment