DEV Community

Cover image for Odyssey to Python Mastery: 4 Jedi Techniques
Prayson Wilfred Daniel
Prayson Wilfred Daniel

Posted on • Updated on

Odyssey to Python Mastery: 4 Jedi Techniques

Enfold the odyssey to Python brilliance, a journey illuminated with the allure of elegance and efficiency. This article, inspired by Saa, my Python package that translates time into human-friendly spoken expressions, takes on a voyage to explore four advanced Python concepts, unearthing the true prowess of Python language. With every stride, witness your code transform, mirroring the elegance of poetry and the finesse of an art masterpiece.

โš ๏ธ code heavy: set out in not-too-distant galax, this article explores advance concepts


1. Single Dispatcher: The Chameleon Functions

chameleon

Observe the allure of singledispatch from functools, a decorator that bestows your functions the gift to morph based on the type of the first argument, a beacon for writing seamless generic code.

from __future__ import annotations
from functools import singledispatch
from datetime import time, datetime


TimeType = str | time | datetime


@singledispatch
def clock(_: TimeType) -> time:
    """Clock Parser

    Accepts string, time or datetime and return time object

    Args:
        _ (TimeType): string, time or datetime object

    Raises:
        NotImplementedError: shell for dispatching

    Returns:
        time: python time object
    """
    raise NotImplementedError


@clock.register(str)
def _(t: str) -> time:
    return datetime.strptime(t, "%H:%M").time()


@clock.register(datetime)
def _(t: datetime) -> time:
    return t.time()


@clock.register(time)
def _(t: time) -> time:
    return t
Enter fullscreen mode Exit fullscreen mode

Test we must, young Padawan. Reflect upon the use cases, we shall. In the vast galaxy where stars twinkle, mirror the universeโ€™s myriad possibilities, our trials will.

from datetime import datetime, time
import pytest
import clock  


@pytest.fixture(params=["12:34", datetime.now().replace(hour=12, minute=34), time(12, 34)])
def valid_time_input(request):
    yield request.param


@pytest.fixture(params=[1234, 12.34, [12, 34], {"hour": 12, "minute": 34}])
def invalid_time_input(request):
    yield request.param

def test_clock_with_valid_input(valid_time_input):
    "Test with valid inputs"

    result = clock(valid_time_input)
    assert isinstance(result, time)
    assert result.hour == 12
    assert result.minute == 34


def test_clock_with_invalid_input(invalid_time_input):
    "Test with invalid input, expecting NotImplementedError"
    with pytest.raises(NotImplementedError):
        clock(invalid_time_input)

Enter fullscreen mode Exit fullscreen mode

Reflect and Contemplate:

Pave this path cautiously. Ensure the base function is etched to raise a NotImplementedError and adorn each additional implementation with @function_name.register(type).


2. Setters and Getters: The Shield and Sword

setter

Conjure the @property decorator, your shield and sword, ensuring your objects stand resilient, cloaked in encapsulation.

Beginning with elemental steps, let's unveil the potential to not only encapsulate but also validate inputs at the setting.

from __future__ import annotations

ฯ€ = 22/7 #3.14...

def radius_validator(value: int|float) -> int | float:

    if not isinstance(value, (int, float)):
        raise TypeError(f"Radius has to be a positive int or float ๐Ÿ˜ž")
    elif value < 0:
        raise ValueError(f"Radius cannot be negative ๐Ÿ˜ž")       
    return value


class Circle:
    def __init__(self, radius: int|float):
        self._radius = radius_validator(radius)

    @property
    def radius(self) -> int|float:
        return self._radius

    @radius.setter
    def radius(self, value: int|float):

        self._radius = radius_validator(value)

    def area(self) -> float:
        return ฯ€ * self._radius**2 

    def circumference(self) -> float:
        return 2 * ฯ€ * self._radius

    def __repr__(self):

        return f"area={self.area(): .2f};circumference={self.circumference():.2f}"

# c = Circle(radius=42) 
# c.radius = -1   # => throws ValueError(f"Radius cannot be negative ๐Ÿ˜ž")
# c.radius = "42" # => throws TypeError(f"Radius has to be a positive int or float ๐Ÿ˜ž") 

Enter fullscreen mode Exit fullscreen mode

Up to this point, things are looking promising. The power of setters and getters extends far beyond mere value validation at assignments; they unlock Jedi-like capabilities. Consider this: in the realm of classical Machine Learning, what if we employed a setter to save a fitted transformer and later, during prediction, utilized a getter to retrieve it? Intriguing, isn't it?

import pickle
from pathlib import Path
from typing import Literal
import pandas as pd
from sklearn.compose import ColumnTransformer


class TransformerTask:
    def __init__(self, transformer:ColumnTransformer, 
                 stage: Literal["train", "predict"] = "train", 
                 file_path:str="models/transformer.pkl",):
        self.stage = stage
        self.file_path = Path(file_path)
        self._transformer = transformer

    @property
    def transformer(self) -> ColumnTransformer:
        if self.stage == "predict" and not self.file_path.exists():
            raise FileNotFoundError(f"{self.file_path} was not found.")

        if self.stage == "predict" and self.file_path.exists():
            self._transformer = pickle.loads(self.file_path.read_bytes())

        return self._transformer

    @transformer.setter
    def transformer(self, transformer:ColumnTransformer) -> None: 
        self.file_path.write_bytes(pickle.dumps(transformer))


    def run(self, data: pd.DataFrame) -> pd.DataFrame:
        operations = {
            "train": self._train,
            "predict": self._predict
        }
        return operations.get(self.stage, lambda d: d)(data)

    def _train(self, data: pd.DataFrame):
        cleaned_data = self.transformer.fit_transform(data)
        self.transformer = self._transformer
        return cleaned_data

    def _predict(self, data: pd.DataFrame):
        return self.transformer.transform(data)
Enter fullscreen mode Exit fullscreen mode

Test we must, again young Padawan ...

from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OneHotEncoder
from sklearn import set_config

from srp import config

set_config(transform_output="pandas")


transformer = make_column_transformer(
    (
        OneHotEncoder(sparse_output=False),[
            config.COLUMNS_TO_ONEHOTENCODE,
        ],
    ),
    verbose_feature_names_out=False,
    remainder="passthrough",
)
Enter fullscreen mode Exit fullscreen mode

Oops, now test, we can ๐Ÿฅน

from pathlib import Path
import pytest
import pandas as pd
from tasks import TransformerTask 
from preprocessors import transformer 



# URI for the dataset
URI = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"

# Model File path
MODEL_FILE = Path("models/transformer.pkl")

def cleanup(file=MODEL_FILE):
    if MODEL_FILE.exists():
        MODEL_FILE.unlink() 

@pytest.fixture(autouse=True)
def startup_and_teardown():
    "Execute before and after a test run"
    # Startup: remove model file
    cleanup()
    yield
    # Teardown : remove model file
    cleanup()

@pytest.fixture
def load_data():
    yield pd.read_csv(URI)


@pytest.fixture
def transformer_task():
    yield TransformerTask(transformer, stage="train")


def test_transformer_train(load_data, transformer_task):
    "Test the training stage"

    result = transformer_task.run(load_data) 
    assert not result.empty
    assert MODEL_FILE.exists()


def test_transformer_predict(transformer_task, load_data):
    "Test the prediction stage"
    # First, run the training stage to ensure the transformer is fitted and saved
    transformer_task.run(load_data)
    transformer_task.stage = "predict"

    # Testing prediction stage
    assert MODEL_FILE.exists()
    assert transformer_task.stage == "predict" 

    result = transformer_task.run(load_data)
    assert not result.empty

Enter fullscreen mode Exit fullscreen mode

Reflect and Contemplate:

Wield this tool with a balance to avert the shadows of accidental recursion or the echoes of unwanted side effects.


3. Decorators: The Enchanters

russian doll

Summon the might of Decorators, enchanters that weave their spells to transform the behaviour of your functions or methods, bestowing upon them new realms of possibilities.

Create a decorator, we must. Allow it will, to profile bottlenecks within our functions/methods, hmmm.

Start simple, we shall. Expand later, we will, young Padawan.

from typing import Callable
import numpy as np

def regression(func: Callable) -> Callable:
    func.kind = "regression"
    return func

@regression
def mse(y_true, y_pred):
    return ((y_true - y_pred)**2).mean()

print((mse.__name__, mse.kind))
# ('mse', 'regression')
Enter fullscreen mode Exit fullscreen mode

Let's level up, doing a hyperspace jump

from cProfile import Profile
from functools import wraps
import pstats

def profiler(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        with Profile() as p:
            # run the function and collect profile data
            results = p.runcall(func, *args, **kwargs)

            # sort => name, time, file https://docs.python.org/3/library/profile.html
            ps = pstats.Stats(p, stream=None).sort_stats('cumulative') 
            ps.print_stats(5) # restriction of print
        return results
    return wrapper


@profiler
@regression
def mse(y_true, y_pred):
    return ((y_true - y_pred)**2).mean()

if __name__ == "__main__":
    y_true = np.array([1, 2, 3, 4, 5])
    y_pred = np.array([1, 3, 2, 3, 5])
    print((mse.__name__, mse.kind, results))
    # some profile logs
    # ('mse', 'regression', 0.6)


Enter fullscreen mode Exit fullscreen mode

Another way of writing the same decorator, using class, would look like this:

class profiler:
    def __init__(self, func):
        self.func = func
        self.__name__ = func.__name__

    def __call__(self, *args, **kwargs):
        with Profile() as p:
            results = p.runcall(self.func, *args, **kwargs)

            ps = pstats.Stats(p, stream=None).sort_stats('cumulative')
            ps.print_stats(5) 

        return results
Enter fullscreen mode Exit fullscreen mode

Hark! A disturbance in the Force. Hardcoded sorting and stats printing lines. A desire to inject them, I sense. Embark on this quest, we must. Bring balance to the code, we shall. Doing that, let us.

# functional way
def profiler(sort_by='cumulative', restriction=5, streams=None):
    def inner_function(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            with Profile() as p:
                result = p.runcall(func, *args, **kwargs)  
                ps = pstats.Stats(p, stream=streams).sort_stats(sort_by)  
                ps.print_stats(restriction)  
            return result
        return wrapper
    return inner_function

# class way
class profiler:
    def __init__(self, sort_by='cumulative', restriction=5, streams=None):
        self.sort_by = sort_by
        self.restriction = restriction
        self.streams = streams

    def __call__(self, func):
        def wrapper(*args, **kwargs):
            with Profile() as p:
                result = p.runcall(func, *args, **kwargs)  
                ps = pstats.Stats(p, stream=self.streams).sort_stats(self.sort_by)  
                ps.print_stats(self.restriction)  
            return result
        wrapper.__name__ = func.__name__
        return wrapper

@regression
@profiler(sort_by="time", restriction=3)
def mse(y_true, y_pred):
    return ((y_true - y_pred)**2).mean()

if __name__ == "__main__":
    y_true = np.array([1, 2, 3, 4, 5])
    y_pred = np.array([1, 3, 2, 3, 5])
    results = mse(y_true, y_pred)

    print((mse.__name__, mse.kind, results))
    # some profile logs are sorted by time and show 3 lines
    # ('mse', 'regression', 0.6)
Enter fullscreen mode Exit fullscreen mode

Ah, a wise query emerges amidst the stars! Speak of adorning methods within a cosmic class, do you? Fear not, for a class decorator we shall craft. Bestow our decorator upon the class methods, it shall. In unity, traverse the celestial pathways of Python, we will!

def profilerx(decorate=None):
    if decorate is None:
        decorate = lambda d: d

    def wrapper(cls):
        name_functions = {name:func for name, func in vars(cls).items() if not name.startswith("__")}

        for name, func in vars(cls).items():
            if callable(func):
                setattr(cls, name, decorate(func))

        return cls
        wrapper.__name__ = cls.__name__

    return wrapper

# Decorate our class, we can โ˜บ๏ธ

@profilerx(decorate=profiler(sort_by="cumulative", restriction=5))
class Circle:
    def __init__(self, radius: int|float):
        self._radius = radius_validator(radius)
   ...
Enter fullscreen mode Exit fullscreen mode

Reflect and Contemplate:

Embrace this magic with mindfulness, for within its allure, lies the labyrinth of complexity, whispering tales of code entangled in its own enchantment.


4. Metaprogramming: The Arcane Arts

meta

As we traverse the realms of decorators, we find ourselves at the gateway to Metaprogramming, a universe where code breathes life into more code. Imagine a scenario where the reins of our crafted code slip out of our grasp, handed over to realms and teams beyond our dominion. In this abyss, where control eludes our touch, how do we ensure our shields are not tampered with? Behold the luminescence of metaprogramming, a beacon in the shadows of constraint and order.

Envision a creation of our own, a code entity named RejectPrint, destined to embark upon a voyage to distant teams and uses. As it journeys beyond our realm, we yearn to engrave a solemn vow upon its essence โ€“ the exile of the print command, guarding the sanctity of silence amidst its ventures in the unknown.

# our modulex.py
import logging
import inspect
from typing import Callable


def check_bad_usage(name: str, obj: Callable)-> bool:
    """
    checks if an object has a function with certain name
    """

    func_names = obj.__code__.co_names
    is_bad_usage = False

    if name not in func_names:
        return is_bad_usage

    is_bad_usage = True
    for func_name in func_names:


        if name == func_name:
            # echo the code with bad usage
            logging.error(inspect.getsource(obj.__code__))
    return is_bad_usage



class RejectPrintMeta(type):

    def __new__(cls, name, bases, body):

        callable_obj = (v for v in body.values() if callable(v))

        for obj in callable_obj:
            is_bad_usage = check_bad_usage("print", obj)
            if is_bad_usage:
                raise UserWarning(f"No way, `{obj.__name__}` contains `print` function!")

        return super().__new__(cls, name, bases, body)



class RejectPrint(metaclass=RejectPrintMeta):
    pass

Enter fullscreen mode Exit fullscreen mode

As users employ our package, a silent guardian emerges at the birth of their classes. The use of print is gently yet firmly barred, ensuring unbroken harmony in the world of our toolโ€™s operation.

# example.py
# from modulex import RejectPrint

class UserPrint(RejectPrint):

    def no_print_used(self):
        pass

    def print_used(self):
        print('using print :-o')
Enter fullscreen mode Exit fullscreen mode

beautiful error message

Reflect and Contemplate:

Tread with care in the realm of metaprogramming, for the powers it unleashes, while potent, whisper the tales of complexity and enigma.


Conclusion: Beginning of the Start

I incorporated these techniques in my repertoire, as a young Padawan. I watched the doors to Python mastery unfold before me. It is my hope that you continue to explore, learn, and adapt, and let the Force guide your own enlightening journey.

And so, the odyssey through the realms of advanced Python concludes yet begins. May your code flourish in elegance and efficiency, nurtured by the seeds of wisdom sown today.

Until then, may the force keep you coding.

Note: Generators, context managers, and plugins are additional Jedi tools that I've had to set aside for this post to avoid making it overly lengthy. If your curiosity is piqued and you desire to explore these realms further, do let me know. A sequel awaits the beckoning of your interest, ready to guide you further on this path to Python mastery.

Top comments (1)

Collapse
 
shahnoza profile image
Shahnoza Bekbulaeva

Great article, it is rare to see explanations of Python advanced concepts in dev to!