CODEBASE SYSTEM MAP

Huey Task Queue

Huey is a lightweight task queue that allows developers to declare tasks via decorators, enqueue them for asynchronous execution, and manage results, retries, and failures through a coordinated system of workers and a scheduler.

System overview5 Modules
Technical view · Deepcoleifer/huey6e8acd022346eeb1b09f46c7fa2612f760ee194ben
System overview

Product runtime architecture

Solid lines show the primary path. Dashed lines show reads, writes, async work, or feedback. Select an element to drill down.

Primary pathSupporting relation
Drawing…
01API LayerProvides the public interface for declaring tasks, enqueuing them, and managing results and schedules.02Task RegistryMaintains a mapping of task names to task classes and handles serialization/deserialization of task messages.03Storage BackendAbstracts the underlying queue and result store, providing operations for enqueue, dequeue, schedule, and result retrieval.04Consumer & WorkersRuns the worker processes and scheduler that dequeue tasks, execute them, and manage retries and failures.05SerializationHandles serialization and deserialization of task messages, including optional compression and signing.
01
Modules

API Layer

Provides the public interface for declaring tasks, enqueuing them, and managing results and schedules.

Module evidencehuey/api.py:1-1400

Dashed nodes are interfaces outside this module.

Drawing…
entryTask DeclarationAllows developers to define tasks using the @huey.task() or @huey.periodic_task() decorators, capturing retry and scheduling parameters.
Input
Function to be decoratedRetry parametersSchedule validation function
Output
TaskWrapper instance
Source evidence
huey/api.py:192-215

No prompt evidence is directly associated with this node.

stageEnqueue TaskSerializes a task into a message and pushes it onto the queue via the storage backend.
Input
TaskWrapper instanceArguments and keyword arguments
Output
Serialized message in queue
Source evidence
huey/api.py:307-335

No prompt evidence is directly associated with this node.

stageDequeue TaskRetrieves a serialized message from the queue and deserializes it back into a task object.
Input
Queued message
Output
Task object
Source evidence
huey/api.py:383-386

No prompt evidence is directly associated with this node.

stageExecute TaskRuns the task's function, handling pre/post execute hooks, retries, and result storage.
Input
Task object
Output
Result or exception
Source evidence
huey/api.py:424-571

No prompt evidence is directly associated with this node.

decisionRetry DecisionDetermines whether a failed task should be retried based on retry count and delay parameters.
Input
ExceptionTask retry metadata
Output
Retry or failure result
Source evidence
huey/api.py:602-617

No prompt evidence is directly associated with this node.

stageGet ResultRetrieves a task's result from the result store, optionally blocking until it is available.
Input
Task ID
Output
Result value
Source evidence
huey/api.py:1315-1321

No prompt evidence is directly associated with this node.

outputResult OutputRepresents the final result of a task execution that is returned to the caller.
Input
Result value
Output
Final result
Source evidence
huey/api.py:1315-1321

No prompt evidence is directly associated with this node.

02
Modules

Task Registry

Maintains a mapping of task names to task classes and handles serialization/deserialization of task messages.

Module evidencehuey/registry.py:1-133

Dashed nodes are interfaces outside this module.

Drawing…
stageSerialize Task MessageConverts a task object into a Message structure, handling nested callbacks and chord configuration.
Input
Task object
Output
Message object
Source evidence
huey/registry.py:54-95

No prompt evidence is directly associated with this node.

stageDeserialize Task MessageReconstructs a task object from a serialized message, resolving the task class from the registry.
Input
Serialized message
Output
Task object
Source evidence
huey/registry.py:97-128

No prompt evidence is directly associated with this node.

03
Modules

Storage Backend

Abstracts the underlying queue and result store, providing operations for enqueue, dequeue, schedule, and result retrieval.

Module evidencehuey/storage.py:1-1700

Dashed nodes are interfaces outside this module.

Drawing…
storeQueue StoragePersists the serialized task message in the underlying queue (e.g., Redis, SQLite) for later retrieval by workers.
Input
Serialized message
Output
Queued message
Source evidence
huey/storage.py:58-70

No prompt evidence is directly associated with this node.

storeResult StoreStores the result of a task execution, keyed by task ID, for later retrieval by callers.
Input
Task IDResult value
Output
Stored result
Source evidence
huey/storage.py:198-216

No prompt evidence is directly associated with this node.

storeSchedule StorePersists scheduled tasks with their ETA, allowing the scheduler to enqueue them when due.
Input
Task messageETA timestamp
Output
Scheduled task entry
Source evidence
huey/storage.py:107-128

No prompt evidence is directly associated with this node.

04
Modules

Consumer & Workers

Runs the worker processes and scheduler that dequeue tasks, execute them, and manage retries and failures.

Module evidencehuey/consumer.py:1-664

Dashed nodes are interfaces outside this module.

Drawing…
workerScheduler LoopPeriodically checks for due periodic tasks and enqueues them, also managing scheduled tasks with ETA.
Input
Current timestampPeriodic task registry
Output
Enqueued periodic tasks
Source evidence
huey/consumer.py:168-199

No prompt evidence is directly associated with this node.

workerWorker LoopContinuously dequeues tasks from the queue and executes them, handling retries and failures.
Input
Queued messages
Output
Executed tasksResults or errors
Source evidence
huey/consumer.py:116-145

No prompt evidence is directly associated with this node.

stageConsumer Health CheckMonitors worker and scheduler processes, restarting any that have died to maintain system availability.
Input
Worker process statusScheduler process status
Output
Restarted processes
Source evidence
huey/consumer.py:580-610

No prompt evidence is directly associated with this node.

05
Modules

Serialization

Handles serialization and deserialization of task messages, including optional compression and signing.

Module evidencehuey/serializer.py:1-92

Dashed nodes are interfaces outside this module.

Drawing…
toolSerializer InterfaceProvides serialize and deserialize methods used by the API to convert messages to and from bytes.
Input
Message objectBytes
Output
BytesMessage object
Source evidence
huey/serializer.py:1-92

No prompt evidence is directly associated with this node.

Generated from traceable source evidence · IR 2.0