This post explains how Kedro can guide an analytics team to follow best practices and avoid technical debt.
In a recent article, I explained that following software principles can help you create a well-ordered analytics project to share, extend and reuse in the future. In this post we'll review how you can benefit from using Kedro as a toolbox to apply best practices to data science code.
How data science projects fail
As data scientists, we aspire to unlock valuable insights by building
well-engineered prototypes that we can take forward into production.
Instead, there is a tendency for us to make poor engineering decisions
in the face of tight deadlines or write code of dubious quality through
a lack of expertise.
The result is technical debt and prototype code that is difficult to understand,
maintain, extend, and fix. Projects that once looked promising fail to transition past the experimental stage into production.
"A cycle of quick and exciting research leads to high expectations of
great improvement, followed by a long series of delays and
disappointments where frustrating integration work fails to recreate
those elusive improvements, made all the worse by the feeling of sunk
costs and a need to justify the time spent."Joe Plattenburg, Data Scientist at Root Insurance
How to write well-engineered data science code
When you start to cut code on a prototype, you may not prioritize
maintainability and consistency. Adopting a team culture and way of
working to minimize technical debt can make the difference between
success and failure.
Some of the most valuable techniques a data scientist can pick up are
those that generations of software engineers already use, such as the
following guidelines:
Use a standard and logical project structure: It is easier to
understand a project, and share it with others, if you follow a standard
structure.
Don't use hardcoded values: instead, use precisely named constants
and put them all into a single configuration file so you can find and
update them easily.
Refactor your code: In data science terms, it often makes sense to
use a Jupyter notebook for experimentation. But once your experiment is
done, it's time to clean up the code to remove elements that make it
unmaintainable, and to remove accidental complexity. Refactor the code
into Python functions and packages to form a pipeline that can be
routinely tested to ensure repeatable behaviour.
"Testing after each change means that when I make a mistake, I only
have a small change to consider in order to spot the error, which
makes it far easier to find and fix."Martin Fowler, Author of Refactoring: Improving the Design of Existing
Code
Make code reusable by making it readable: Write your pipelines as a
series of small functions that do just one task, with single return
paths and a limited number of arguments.
Many data scientists say they've learned from their colleagues through
pair programming, code reviews and in-house mentoring that enables them
to build expertise suitable to their roles and requirements.
We see Kedro as the always-available team lead that steers the direction
of the analytics project from the outset and encourages use of a
well-organized folder structure, software design that supports regular
testing, and a culture of writing readable, clean code.
What is Kedro?
Kedro is an open-source toolbox for production-ready data science. The
framework was born at QuantumBlack to solve the challenges faced
regularly in data science projects and promote teamwork through
standardised team workflows. It is now hosted by the LF AI & Data
Foundation as an incubating project.
Kedro = Consistent project structure
Kedro is built on the learnings of Cookie Cutter Data Science. It helps you to standardise how configuration, source
code, tests, documentation, and notebooks are organised with an
adaptable project template. If your team needs to build with multiple
projects that have similar structure, you can also create your own
Cookie Cutter project templates with Kedro starters.
Kedro = Maintainable code
Kedro helps you refactor your business logic and data processing into
Python modules and packages to form pipelines, so you can keep your
notebooks clean and tidy.
Kedro-Viz then visualises the pipelines to help you navigate .
"People started from scratch each time, the same pitfalls were
experienced independently, reproducibility was time consuming and only
members of the original project team really understood each
codebase...We needed to enforce consistency and software engineering best
practices across our own work. Kedro gave us the super-power to move
people from project to project and it was game-changing. After working
with Kedro once, you can land in another project and know how the
codebase is structured, where everything is and most importantly how
you can help".Joel Schwarzmann, Principal Product Manager, QuantumBlack Labs, blog
post
Kedro = Code quality
Kedro makes it easy to avoid common code smells such as hard-coded
constants and magic numbers. The configuration library enables your code
to be reusable through data, model, and logging configuration. An
ever-expanding data catalog supports multiple formats of data access.
Kedro also makes it keep your code quality up to standard, through
support for black, isort, and flake8 for code linting and formatting,
pytest for testing, and Sphinx for documentation.
Kedro = Standardisation
Kedro integrates with standard data science tools, such as TensorFlow,
scikit-learn, or Jupyter notebooks for experimentation, and commonly
used routes to deployment such as Databricks.
Summary
Kedro is an open-source Python toolbox that applies software engineering
principles to data science code. It makes it easier for a team to apply
software engineering principles to data science code, which reduces the
time spent rewriting data science experiments so that they are fit for
production.
When you follow established best practice, you have a better chance of
success.
Software engineering principles only work if the entire team follows
them. A tool like Kedro can guide you just like an experienced technical
lead, making it second nature to use established best practices, and
supporting a culture and set of processes based upon software
engineering.
Look forward to greater collaboration and productivity with Kedro in
your team!
Find out more about Kedro
There are many ways to learn more about Kedro:
Join our Slack organisation to reach out to us directly if you've a question or want to stay up to date with our news. There's an archive of past past conversations on Slack too.
Read our docs or look at the Kedro source code on GitHub.
Check out our "Crash course in Kedro video on YouTube.
Look out for an upcoming training session tailored to help your team get
on-board with Kedro.
Top comments (0)