{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Split Up! Fighting the Monolith!\n",
"
\n",
"## Patrick Muehlbauer, Europython 2016\n",
"
\n",
"### [@tmuxbee](https://twitter.com/tmuxbee)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Sorry, no Microservices :)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Python Package Structure\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mysuperlibrary/\r\n",
"├── .git\r\n",
"├── MANIFEST.in\r\n",
"├── mysuperlibrary\r\n",
"│ └── __init__.py\r\n",
"├── README\r\n",
"├── requirements.txt\r\n",
"├── setup.cfg\r\n",
"├── setup.py\r\n",
"└── tests\r\n",
"\r\n",
"3 directories, 6 files\r\n"
]
}
],
"source": [
"!tree -a mysuperlibrary/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Monorepo containing multiple packages\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"monorepo/\r\n",
"├── .git\r\n",
"├── library_1\r\n",
"│ ├── library_1\r\n",
"│ └── tests\r\n",
"├── library_2\r\n",
"│ ├── library_2\r\n",
"│ └── tests\r\n",
"├── library_3\r\n",
"│ ├── library_3\r\n",
"│ └── tests\r\n",
"├── library_n\r\n",
"│ ├── library_n\r\n",
"│ └── tests\r\n",
"├── myapplication\r\n",
"│ ├── myapplication\r\n",
"│ └── tests\r\n",
"└── requirements.txt\r\n",
"\r\n",
"16 directories, 1 file\r\n"
]
}
],
"source": [
"!tree -a monorepo/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"source": [
"![](images/monorepo_selection.jpg)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Why moving subpackages to their own repos?\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* nicer experience for contributers"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* avoiding 'spaghetti code'"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* easier to do releases of the libraries"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* easier to integrate something like `setuptools_scm`"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## setuptools_scm\n",
"### manage your versions by scm tags\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# setup.py\n",
"from setuptools import setup, find_packages\n",
"\n",
"# defining the version manually\n",
"setup(\n",
" name='mypackage',\n",
" version='0.0.1'\n",
" packages=find_packages(exclude=['tests']),\n",
" install_requires=['requests']\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# version from setuptools_scm\n",
"setup(\n",
" name='mypackage',\n",
" packages=find_packages(exclude=['tests']),\n",
" setup_requires=['setuptools_scm'],\n",
" use_scm_version=True,\n",
" install_requires=['requests']\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.0.2.dev1+ngdcd0b11\r\n"
]
}
],
"source": [
"!cd mypackage && python setup.py --version # one commit (dcd0b11...) after 0.0.1 "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## What about my commit history?\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### git subtree"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"`git subtree split -P library_3 -b lib_3-branch`"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"```mkdir ../library_3\n",
"cd ../library_3```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"`git init`"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"`git pull ../monorepo lib_3-branch`"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Continuous Integration"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Before\n",
"Application is build with latest commits of all libraries, since there is only one repository."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"### After (1)\n",
"\n",
"Try mimicing old behaviour by checking out latest commit of every single repository."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Ended up in a really messy Jenkins job for creating the application build."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* bugfix releases... :|"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Why not just putting our internal libraries to the dependencies in `requirements.txt`?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### After (2)\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* run libraries unit tests"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* if successful, upload wheel to internal devpi server"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* use `pip install --pre` for internal dependencies when creating the build artefact"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* extra job for release builds where `--pre` is not used"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Devpi-Server\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* PyPI server: can be used as local self-updating caching mirror"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* whitelisting/blacklisting certain packages"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* ..."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Requirements Pinning Strategy\n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* library packages: don't pin dependencies in `setup.py`'s `install_requires`"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* application is responsible to use correct requirements"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Will avoid lots of `VersionConflict` errors."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Let's see what we got from splitting up\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* `(+)` automatic versioning for our libraries powered by `setuptools_scm`"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"\n",
"* `(+)` happy library contributers"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* `(-)` a much more complex development workflow for core commiters"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* (+) improved code quality and structure\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## What's Next?\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Microservices? ;)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Thanks!\n",
"\n",
"Slides: https://github.com/blue-yonder/documents"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 0
}