Enfold the odyssey to Python brilliance, a journey illuminated with the allure of elegance and efficiency. This article, inspired by Saa, my Python package that translates time into human-friendly spoken expressions, takes on a voyage to explore four advanced Python concepts, unearthing the true prowess of Python language. With every stride, witness your code transform, mirroring the elegance of poetry and the finesse of an art masterpiece.
β οΈ code heavy: set out in not-too-distant galax, this article explores advance concepts
1. Single Dispatcher: The Chameleon Functions
Observe the allure of singledispatch from functools, a decorator that bestows your functions the gift to morph based on the type of the first argument, a beacon for writing seamless generic code.
from __future__ import annotations
from functools import singledispatch
from datetime import time, datetime
TimeType = str | time | datetime
@singledispatch
def clock(_: TimeType) -> time:
"""Clock Parser
Accepts string, time or datetime and return time object
Args:
_ (TimeType): string, time or datetime object
Raises:
NotImplementedError: shell for dispatching
Returns:
time: python time object
"""
raise NotImplementedError
@clock.register(str)
def _(t: str) -> time:
return datetime.strptime(t, "%H:%M").time()
@clock.register(datetime)
def _(t: datetime) -> time:
return t.time()
@clock.register(time)
def _(t: time) -> time:
return t
Test we must, young Padawan. Reflect upon the use cases, we shall. In the vast galaxy where stars twinkle, mirror the universeβs myriad possibilities, our trials will.
from datetime import datetime, time
import pytest
import clock
@pytest.fixture(params=["12:34", datetime.now().replace(hour=12, minute=34), time(12, 34)])
def valid_time_input(request):
yield request.param
@pytest.fixture(params=[1234, 12.34, [12, 34], {"hour": 12, "minute": 34}])
def invalid_time_input(request):
yield request.param
def test_clock_with_valid_input(valid_time_input):
"Test with valid inputs"
result = clock(valid_time_input)
assert isinstance(result, time)
assert result.hour == 12
assert result.minute == 34
def test_clock_with_invalid_input(invalid_time_input):
"Test with invalid input, expecting NotImplementedError"
with pytest.raises(NotImplementedError):
clock(invalid_time_input)
Reflect and Contemplate:
Pave this path cautiously. Ensure the base function is etched to raise a NotImplementedError and adorn each additional implementation with @function_name.register(type).
2. Setters and Getters: The Shield and Sword
Conjure the @property decorator, your shield and sword, ensuring your objects stand resilient, cloaked in encapsulation.
Beginning with elemental steps, let's unveil the potential to not only encapsulate but also validate inputs at the setting.
from __future__ import annotations
Ο = 22/7 #3.14...
def radius_validator(value: int|float) -> int | float:
if not isinstance(value, (int, float)):
raise TypeError(f"Radius has to be a positive int or float π")
elif value < 0:
raise ValueError(f"Radius cannot be negative π")
return value
class Circle:
def __init__(self, radius: int|float):
self.radius = radius
@property
def radius(self) -> int|float:
return self._radius
@radius.setter
def radius(self, value: int|float):
self._radius = radius_validator(value)
def area(self) -> float:
return Ο * self._radius**2
def circumference(self) -> float:
return 2 * Ο * self._radius
def __repr__(self):
return f"area={self.area(): .2f};circumference={self.circumference():.2f}"
# c = Circle(radius=42)
# c.radius = -1 # => throws ValueError(f"Radius cannot be negative π")
# c.radius = "42" # => throws TypeError(f"Radius has to be a positive int or float π")
Up to this point, things are looking promising. The power of setters and getters extends far beyond mere value validation at assignments; they unlock Jedi-like capabilities. Consider this: in the realm of classical Machine Learning, what if we employed a setter to save a fitted transformer and later, during prediction, utilized a getter to retrieve it? Intriguing, isn't it?
import pickle
from pathlib import Path
from typing import Literal
import pandas as pd
from sklearn.compose import ColumnTransformer
class TransformerTask:
def __init__(self, transformer:ColumnTransformer,
stage: Literal["train", "predict"] = "train",
file_path:str="models/transformer.pkl",):
self.stage = stage
self.file_path = Path(file_path)
self._transformer = transformer
@property
def transformer(self) -> ColumnTransformer:
if self.stage == "predict" and not self.file_path.exists():
raise FileNotFoundError(f"{self.file_path} was not found.")
if self.stage == "predict" and self.file_path.exists():
self._transformer = pickle.loads(self.file_path.read_bytes())
return self._transformer
@transformer.setter
def transformer(self, transformer:ColumnTransformer) -> None:
self.file_path.write_bytes(pickle.dumps(transformer))
def run(self, data: pd.DataFrame) -> pd.DataFrame:
operations = {
"train": self._train,
"predict": self._predict
}
return operations.get(self.stage, lambda d: d)(data)
def _train(self, data: pd.DataFrame):
cleaned_data = self.transformer.fit_transform(data)
self.transformer = self._transformer
return cleaned_data
def _predict(self, data: pd.DataFrame):
return self.transformer.transform(data)
Test we must, again young Padawan ...
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OneHotEncoder
from sklearn import set_config
from srp import config
set_config(transform_output="pandas")
transformer = make_column_transformer(
(
OneHotEncoder(sparse_output=False),[
config.COLUMNS_TO_ONEHOTENCODE,
],
),
verbose_feature_names_out=False,
remainder="passthrough",
)
Oops, now test, we can π₯Ή
from pathlib import Path
import pytest
import pandas as pd
from tasks import TransformerTask
from preprocessors import transformer
# URI for the dataset
URI = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"
# Model File path
MODEL_FILE = Path("models/transformer.pkl")
def cleanup(file=MODEL_FILE):
if MODEL_FILE.exists():
MODEL_FILE.unlink()
@pytest.fixture(autouse=True)
def startup_and_teardown():
"Execute before and after a test run"
# Startup: remove model file
cleanup()
yield
# Teardown : remove model file
cleanup()
@pytest.fixture
def load_data():
yield pd.read_csv(URI)
@pytest.fixture
def transformer_task():
yield TransformerTask(transformer, stage="train")
def test_transformer_train(load_data, transformer_task):
"Test the training stage"
result = transformer_task.run(load_data)
assert not result.empty
assert MODEL_FILE.exists()
def test_transformer_predict(transformer_task, load_data):
"Test the prediction stage"
# First, run the training stage to ensure the transformer is fitted and saved
transformer_task.run(load_data)
transformer_task.stage = "predict"
# Testing prediction stage
assert MODEL_FILE.exists()
assert transformer_task.stage == "predict"
result = transformer_task.run(load_data)
assert not result.empty
Reflect and Contemplate:
Wield this tool with a balance to avert the shadows of accidental recursion or the echoes of unwanted side effects.
3. Decorators: The Enchanters
Summon the might of Decorators, enchanters that weave their spells to transform the behaviour of your functions or methods, bestowing upon them new realms of possibilities.
Create a decorator, we must. Allow it will, to profile bottlenecks within our functions/methods, hmmm.
Start simple, we shall. Expand later, we will, young Padawan.
from typing import Callable
import numpy as np
def regression(func: Callable) -> Callable:
func.kind = "regression"
return func
@regression
def mse(y_true, y_pred):
return ((y_true - y_pred)**2).mean()
print((mse.__name__, mse.kind))
# ('mse', 'regression')
Let's level up, doing a hyperspace jump
from cProfile import Profile
from functools import wraps
import pstats
def profiler(func):
@wraps(func)
def wrapper(*args, **kwargs):
with Profile() as p:
# run the function and collect profile data
results = p.runcall(func, *args, **kwargs)
# sort => name, time, file https://docs.python.org/3/library/profile.html
ps = pstats.Stats(p, stream=None).sort_stats('cumulative')
ps.print_stats(5) # restriction of print
return results
return wrapper
@profiler
@regression
def mse(y_true, y_pred):
return ((y_true - y_pred)**2).mean()
if __name__ == "__main__":
y_true = np.array([1, 2, 3, 4, 5])
y_pred = np.array([1, 3, 2, 3, 5])
print((mse.__name__, mse.kind, results))
# some profile logs
# ('mse', 'regression', 0.6)
Another way of writing the same decorator, using class, would look like this:
class profiler:
def __init__(self, func):
self.func = func
self.__name__ = func.__name__
def __call__(self, *args, **kwargs):
with Profile() as p:
results = p.runcall(self.func, *args, **kwargs)
ps = pstats.Stats(p, stream=None).sort_stats('cumulative')
ps.print_stats(5)
return results
Hark! A disturbance in the Force. Hardcoded sorting and stats printing lines. A desire to inject them, I sense. Embark on this quest, we must. Bring balance to the code, we shall. Doing that, let us.
# functional way
def profiler(sort_by='cumulative', restriction=5, streams=None):
def inner_function(func):
@wraps(func)
def wrapper(*args, **kwargs):
with Profile() as p:
result = p.runcall(func, *args, **kwargs)
ps = pstats.Stats(p, stream=streams).sort_stats(sort_by)
ps.print_stats(restriction)
return result
return wrapper
return inner_function
# class way
class profiler:
def __init__(self, sort_by='cumulative', restriction=5, streams=None):
self.sort_by = sort_by
self.restriction = restriction
self.streams = streams
def __call__(self, func):
def wrapper(*args, **kwargs):
with Profile() as p:
result = p.runcall(func, *args, **kwargs)
ps = pstats.Stats(p, stream=self.streams).sort_stats(self.sort_by)
ps.print_stats(self.restriction)
return result
wrapper.__name__ = func.__name__
return wrapper
@regression
@profiler(sort_by="time", restriction=3)
def mse(y_true, y_pred):
return ((y_true - y_pred)**2).mean()
if __name__ == "__main__":
y_true = np.array([1, 2, 3, 4, 5])
y_pred = np.array([1, 3, 2, 3, 5])
results = mse(y_true, y_pred)
print((mse.__name__, mse.kind, results))
# some profile logs are sorted by time and show 3 lines
# ('mse', 'regression', 0.6)
Ah, a wise query emerges amidst the stars! Speak of adorning methods within a cosmic class, do you? Fear not, for a class decorator we shall craft. Bestow our decorator upon the class methods, it shall. In unity, traverse the celestial pathways of Python, we will!
def profilerx(decorate=None):
if decorate is None:
decorate = lambda d: d
def wrapper(cls):
name_functions = {name:func for name, func in vars(cls).items() if not name.startswith("__")}
for name, func in vars(cls).items():
if callable(func):
setattr(cls, name, decorate(func))
return cls
wrapper.__name__ = cls.__name__
return wrapper
# Decorate our class, we can βΊοΈ
@profilerx(decorate=profiler(sort_by="cumulative", restriction=5))
class Circle:
def __init__(self, radius: int|float):
self._radius = radius_validator(radius)
...
Reflect and Contemplate:
Embrace this magic with mindfulness, for within its allure, lies the labyrinth of complexity, whispering tales of code entangled in its own enchantment.
4. Metaprogramming: The Arcane Arts
As we traverse the realms of decorators, we find ourselves at the gateway to Metaprogramming, a universe where code breathes life into more code. Imagine a scenario where the reins of our crafted code slip out of our grasp, handed over to realms and teams beyond our dominion. In this abyss, where control eludes our touch, how do we ensure our shields are not tampered with? Behold the luminescence of metaprogramming, a beacon in the shadows of constraint and order.
Envision a creation of our own, a code entity named RejectPrint, destined to embark upon a voyage to distant teams and uses. As it journeys beyond our realm, we yearn to engrave a solemn vow upon its essence β the exile of the print command, guarding the sanctity of silence amidst its ventures in the unknown.
# our modulex.py
import logging
import inspect
from typing import Callable
def check_bad_usage(name: str, obj: Callable)-> bool:
"""
checks if an object has a function with certain name
"""
func_names = obj.__code__.co_names
is_bad_usage = False
if name not in func_names:
return is_bad_usage
is_bad_usage = True
for func_name in func_names:
if name == func_name:
# echo the code with bad usage
logging.error(inspect.getsource(obj.__code__))
return is_bad_usage
class RejectPrintMeta(type):
def __new__(cls, name, bases, body):
callable_obj = (v for v in body.values() if callable(v))
for obj in callable_obj:
is_bad_usage = check_bad_usage("print", obj)
if is_bad_usage:
raise UserWarning(f"No way, `{obj.__name__}` contains `print` function!")
return super().__new__(cls, name, bases, body)
class RejectPrint(metaclass=RejectPrintMeta):
pass
As users employ our package, a silent guardian emerges at the birth of their classes. The use of print is gently yet firmly barred, ensuring unbroken harmony in the world of our toolβs operation.
# example.py
# from modulex import RejectPrint
class UserPrint(RejectPrint):
def no_print_used(self):
pass
def print_used(self):
print('using print :-o')
Reflect and Contemplate:
Tread with care in the realm of metaprogramming, for the powers it unleashes, while potent, whisper the tales of complexity and enigma.
Conclusion: Beginning of the Start
I incorporated these techniques in my repertoire, as a young Padawan. I watched the doors to Python mastery unfold before me. It is my hope that you continue to explore, learn, and adapt, and let the Force guide your own enlightening journey.
And so, the odyssey through the realms of advanced Python concludes yet begins. May your code flourish in elegance and efficiency, nurtured by the seeds of wisdom sown today.
Until then, may the force keep you coding.
Note: Generators, context managers, and plugins are additional Jedi tools that I've had to set aside for this post to avoid making it overly lengthy. If your curiosity is piqued and you desire to explore these realms further, do let me know. A sequel awaits the beckoning of your interest, ready to guide you further on this path to Python mastery.
Top comments (1)
Great article, it is rare to see explanations of Python advanced concepts in dev to!