How circuits actually works

circuits is 2 circuit breaker classes, that help you manage failures in your code.

Both classes have:

Circuit

Circuit is an auto-resetting circuit that sleeps, and then automatically resets to continue.

When the short counter reaches the num_shorts_to_trip threshold, the circuit trips.

┌─────────┐    ┌───────────────────┐    ┌──────────────────────────────────┐
│ short() │───▶│ increment counter │───▶│ shorts == num_shorts_to_trip ? │
└─────────┘    └───────────────────┘    └───────────────┬──────────────────┘
                                                         │
                                   ┌─────────────────────┴────────────────────┐
                                   │                                          │
                                yes│                                          │no
                                   ▼                                          ▼
                            ┌──────────┐                              ┌──────────┐
                            │   trip   │                              │ continue │
                            └────┬─────┘                              └──────────┘
                                 │
                                 ▼
                         ┌────────────────┐
                         │ sleep + reset  │
                         └────────────────┘

Circuit uses a threading.RLock to ensure that internal state is thread-safe.

Tracking state

_num_shorts_to_trip: int Number of shorts required before the circuit trips. Set at initialization.

_times_shorted: int Counter tracking shorts since the last trip. Resets to 0 after each trip.

_total_trips: int Lifetime count of all trips. Never resets.

_current_sleep_time: float Current sleep duration after backoff is applied. Starts at sleep_time_after_trip and grows with each trip.

_lock: threading.RLock Reentrant lock for thread-safe state access.

Properties

num_shorts_to_trip: Number of shorts required before the circuit trips.

times_shorted: Counter tracking shorts since the last trip. Resets to 0 after each trip.

total_trips: Lifetime count of all trips. Never resets.

current_sleep_time: Current sleep duration after backoff is applied. Starts at sleep_time_after_trip and grows with each trip.

Methods

short(custom_sleep: float | None = None) -> bool

Increments _times_shorted by 1, calling _trip_circuit() if the num_shorts_to_trip threshold is reached.

If custom_sleep is provided, it will be used instead of _current_sleep_time.

  1. Acquires self._lock
  2. Increments _times_shorted by 1
  3. Checks if _times_shorted >= num_shorts_to_trip
  4. Releases self._lock
  5. If _times_shorted >= num_shorts_to_trip, calls _trip_circuit():
  1. Returns True if slept, False otherwise

trip(custom_sleep: float | None = None) -> bool

trip() immediately triggers the circuit, bypassing the short counter.

If custom_sleep is provided, it will be used instead of _current_sleep_time.

  1. Calls _trip_circuit():
  1. Returns True (always sleeps)

reset_backoff() -> None

Restores the original sleep time to sleep_time_after_trip.

  1. Acquires self._lock
  2. Sets _current_sleep_time to sleep_time_after_trip
  3. Releases self._lock
  4. Returns None

Exponential Backoff

Circuit supports exponential backoff to progressively increase sleep time after repeated trips.

circ = Circuit(
    num_shorts_to_trip=5,
    sleep_time_after_trip=1.0,  # initial sleep time == 1.0s
    backoff_factor=2.0,         # double it each time it is tripped
    max_sleep_time=30.0         # sleep time is capped at 30s
)

With the above parameters:

Formula for calculating the next sleep time:

_current_sleep_time = min(
    _current_sleep_time * backoff_factor,
    max_sleep_time
)

Jitter and max sleep time

Jitter adds randomness to sleep durations.

When multiple processes trip their circuits at the same time, they all wake up at the same time, which puts pressure on the system. Jitter spreads out the wake-up times to prevent this.

The jitter parameter is a decimal (0.2), not a percentage (20).

Values are clamped to [0.0, 1.0].

For a jitter of 0.2 and a sleep_duration of 1.0, the range is [0.8, 1.2].

Thread Safety

All state access is protected by a reentrant lock (threading.RLock).

A reentrant lock is needed because _trip_circuit() may be called from short() and trip() and both need lock access.

The sleep operation itself happens outside the lock to avoid blocking other threads during the sleep.

All public properties acquire the lock for reads.

@property
def times_shorted(self) -> int:
    with self._lock:
        return self._times_shorted

This ensures that all reads are consistent and thread-safe.

Async Support

Circuit supports async usage via the _AsyncableMethod pattern from suitkaise.sk.

_AsyncableMethod wraps a sync method and an async method into a single attribute that can be called either way.

short = _AsyncableMethod(_sync_short, _async_short)

Usage:

# sync usage
circ.short()

# async usage
await circ.short.asynced()()

The async versions use asyncio.sleep() instead of blocking time.sleep():

async def _async_trip_circuit(self, custom_sleep: float | None = None) -> bool:
    with self._lock:
        sleep_duration = custom_sleep if custom_sleep is not None else self._current_sleep_time
        self._total_trips += 1
        self._times_shorted = 0
        
        if self.backoff_factor != 1.0:
            self._current_sleep_time = min(
                self._current_sleep_time * self.backoff_factor,
                self.max_sleep_time
            )
    
    sleep_duration = self._apply_jitter(sleep_duration)
    if sleep_duration > 0:
        await asyncio.sleep(sleep_duration)
    
    return True

The lock usage is the same - only the sleep call differs.

Methods with async support:

Methods like reset_backoff() and properties do not need async versions because they don't sleep.

Share Integration

Circuit includes _shared_meta for integration with suitkaise.processing.Share.

_shared_meta is a dictionary that declares which attributes each method/property reads from or writes to. The Share class uses this metadata to synchronize state across processes.

_shared_meta = {
    'methods': {
        'short': {'writes': ['_times_shorted', '_total_trips', '_current_sleep_time']},
        'trip': {'writes': ['_times_shorted', '_total_trips', '_current_sleep_time']},
        'reset_backoff': {'writes': ['_current_sleep_time']},
    },
    'properties': {
        'times_shorted': {'reads': ['_times_shorted']},
        'total_trips': {'reads': ['_total_trips']},
        'current_sleep_time': {'reads': ['_current_sleep_time']},
    }
}

This allows a Share instance to wrap a circuit and automatically synchronize state across multiple processes.

Sleep Implementation

Circuit uses suitkaise.timing.sleep() for blocking sleeps:

from suitkaise.timing import api as timing

# in _trip_circuit():
timing.sleep(sleep_duration)

This uses the timing module's sleep implementation, which provides consistent behavior across different environments.

BreakingCircuit

Breaking circuit that stops when the failure threshold is reached.

Unlike Circuit, it stays broken until you manually reset it. Use this for stopping after a threshold is reached and deciding what to do next.

When the short counter reaches the num_shorts_to_trip threshold, the circuit breaks.

┌─────────┐    ┌───────────────────┐    ┌──────────────────────────────────┐
│ short() │───▶│ increment counter │───▶│ shorts >= num_shorts_to_trip ? │
└─────────┘    └───────────────────┘    └───────────────┬──────────────────┘
                                                         │
                                   ┌─────────────────────┴────────────────────┐
                                   │                                          │
                                yes│                                          │no
                                   ▼                                          ▼
                            ┌──────────┐                              ┌──────────┐
                            │  break   │                              │ continue │
                            └────┬─────┘                              └──────────┘
                                 │
                                 ▼
                      ┌───────────────────────┐
                      │ sleep (stay broken)   │
                      └───────────────────────┘

BreakingCircuit uses a threading.RLock to ensure that internal state is thread-safe.

Tracking state

_num_shorts_to_trip: int Number of shorts required before the circuit breaks. Set at initialization.

_times_shorted: int Counter tracking shorts since the last trip/reset. Resets to 0 after each trip or reset.

_total_trips: int Lifetime count of all trips. Incremented on every short() call, never resets.

_current_sleep_time: float Current sleep duration after backoff is applied. Starts at sleep_time_after_trip and grows with each reset().

_broken: bool Whether the circuit is currently broken. Set to True on trip, cleared by reset().

_lock: threading.RLock Reentrant lock for thread-safe state access.

Properties

num_shorts_to_trip: Number of shorts required before the circuit breaks.

broken: Whether the circuit is currently broken.

times_shorted: Counter tracking shorts since the last trip/reset. Resets to 0 after each trip or reset.

total_trips: Lifetime count of all trips. Never resets.

current_sleep_time: Current sleep duration after backoff is applied. Starts at sleep_time_after_trip and grows with each reset().

Methods

short(custom_sleep: float | None = None) -> None

Increments _times_shorted and _total_trips by 1, calling _break_circuit() if the num_shorts_to_trip threshold is reached.

If custom_sleep is provided, it will be used instead of _current_sleep_time.

  1. Captures sleep_duration (custom_sleep or _current_sleep_time)
  2. Acquires self._lock
  3. Increments _times_shorted by 1
  4. Increments _total_trips by 1
  5. Checks if _times_shorted >= num_shorts_to_trip
  6. Releases self._lock
  7. If _times_shorted >= num_shorts_to_trip, calls _break_circuit():
  1. Returns None

Note: Unlike Circuit, BreakingCircuit increments _total_trips on every short() call, not just when the circuit trips.

trip(custom_sleep: float | None = None) -> None

trip() immediately breaks the circuit, bypassing the short counter.

If custom_sleep is provided, it will be used instead of _current_sleep_time.

  1. Acquires self._lock
  2. Increments _total_trips by 1
  3. Releases self._lock
  4. Captures sleep_duration (custom_sleep or _current_sleep_time)
  5. Calls _break_circuit(sleep_duration):
  1. Returns None

reset() -> None

Resets the circuit to operational state and applies exponential backoff.

  1. Acquires self._lock
  2. Sets _broken to False
  3. Resets _times_shorted to 0
  4. If backoff_factor != 1.0:
  1. Releases self._lock
  2. Returns None

Note: Unlike Circuit which applies backoff on trip, BreakingCircuit applies backoff on reset(). This means the next trip will use the increased sleep time.

reset_backoff() -> None

Restores the original sleep time to sleep_time_after_trip.

  1. Acquires self._lock
  2. Sets _current_sleep_time to sleep_time_after_trip
  3. Releases self._lock
  4. Returns None

Note: Does NOT reset the broken state - use reset() for that.

Exponential Backoff

BreakingCircuit supports exponential backoff to progressively increase sleep time after repeated resets.

circ = BreakingCircuit(
    num_shorts_to_trip=5,
    sleep_time_after_trip=1.0,  # initial sleep time == 1.0s
    backoff_factor=2.0,         # double it each time reset() is called
    max_sleep_time=30.0         # sleep time is capped at 30s
)

With the above parameters:

Formula for calculating the next sleep time (applied on reset()):

_current_sleep_time = min(
    _current_sleep_time * backoff_factor,
    max_sleep_time
)

Jitter and max sleep time

Jitter adds randomness to sleep durations.

When multiple processes trip their circuits at the same time, they all wake up at the same time, which puts pressure on the system. Jitter spreads out the wake-up times to prevent this.

The jitter parameter is a decimal (0.2), not a percentage (20).

Values are clamped to [0.0, 1.0].

For a jitter of 0.2 and a sleep_duration of 1.0, the range is [0.8, 1.2].

Thread Safety

All state access is protected by a reentrant lock (threading.RLock).

A reentrant lock is needed because _break_circuit() may be called from short() and trip() and both need lock access.

The sleep operation itself happens outside the lock to avoid blocking other threads during the sleep.

All public properties acquire the lock for reads.

@property
def broken(self) -> bool:
    with self._lock:
        return self._broken

This ensures that all reads are consistent and thread-safe.

Async Support

BreakingCircuit supports async usage via the _AsyncableMethod pattern from suitkaise.sk.

_AsyncableMethod wraps a sync method and an async method into a single attribute that can be called either way.

short = _AsyncableMethod(_sync_short, _async_short)

Usage:

# sync usage
circ.short()

# async usage
await circ.short.asynced()()

The async versions use asyncio.sleep() instead of blocking time.sleep():

async def _async_break_circuit(self, sleep_duration: float) -> None:
    with self._lock:
        self._broken = True
        self._times_shorted = 0

    sleep_duration = self._apply_jitter(sleep_duration)
    if sleep_duration > 0:
        await asyncio.sleep(sleep_duration)

The lock usage is the same - only the sleep call differs.

Methods with async support:

Methods like reset(), reset_backoff(), and properties do not need async versions because they don't sleep.

Share Integration

BreakingCircuit includes _shared_meta for integration with suitkaise.processing.Share.

_shared_meta is a dictionary that declares which attributes each method/property reads from or writes to. The Share class uses this metadata to synchronize state across processes.

_shared_meta = {
    'methods': {
        'short': {'writes': ['_times_shorted', '_total_trips', '_broken']},
        'trip': {'writes': ['_total_trips', '_broken', '_times_shorted']},
        'reset': {'writes': ['_broken', '_times_shorted', '_current_sleep_time']},
        'reset_backoff': {'writes': ['_current_sleep_time']},
    },
    'properties': {
        'broken': {'reads': ['_broken']},
        'times_shorted': {'reads': ['_times_shorted']},
        'total_trips': {'reads': ['_total_trips']},
        'current_sleep_time': {'reads': ['_current_sleep_time']},
    }
}

This allows a Share instance to wrap a circuit and automatically synchronize state across multiple processes.

Sleep Implementation

BreakingCircuit uses suitkaise.timing.sleep() for blocking sleeps:

from suitkaise.timing import api as timing

# in _break_circuit():
timing.sleep(sleep_duration)

This uses the timing module's sleep implementation, which provides consistent behavior across different environments.