audit_trail.maintenance¶
Partition management, retention and health checks.
ensure_partitions creates the partitions writes will need: every
severity partition of the activity table, and the monthly partitions of the
transaction table and of each severity partition, from the current UTC month
months_ahead months forward. There is no DEFAULT partition, so a row
without a partition fails with SQLSTATE 23514; run it on a schedule
(cron, worker, application start) with enough months ahead to cover a missed
run.
Existing partitions are found by their bounds in the catalog, not by name: a
partition created by hand or by another tool under a different name, with the
same (or wider) bounds, counts as present. Names and bounds are described in
audit_trail.migrations.
Locking: CREATE TABLE ... PARTITION OF takes an ACCESS EXCLUSIVE lock
on the parent, so it waits for every open transaction that has touched the
parent, including business transactions with uncommitted audit rows. Under
PostgreSQL's documented lock queueing, later lock requests that conflict with
the waiting one, such as new audit inserts, queue behind it. lock_timeout
bounds that wait: past it the whole call is rolled back and
PartitionLockTimeoutError is raised; retrying later is safe.
drop_expired removes monthly partitions older than a per-severity
retention, with ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY and then
DROP TABLE. health reports how many months ahead are covered, detaches
left pending and leftover tables that look like detached partitions.
Locks taken by drop_expired, as observed in pg_locks on PostgreSQL 14
and 18:
DETACH ... CONCURRENTLYfirst takesSHARE UPDATE EXCLUSIVEon the parent, which does not conflict with inserts. It then waits for every transaction that has used the parent, holding no lock on any table while it waits, so new audit inserts go ahead. Last, it takesSHARE UPDATE EXCLUSIVEon the parent andACCESS EXCLUSIVEon the partition, which waits for readers of that partition.DETACH ... FINALIZEtakes the same locks as that last step, and waits for the same transactions.DROP TABLEof the detached table takesACCESS EXCLUSIVEon that table only.
lock_timeout bounds each of these waits, including the wait for other
transactions. An application transaction left open on an audit table for
longer than lock_timeout therefore makes drop_expired fail with
PartitionLockTimeoutError; this is expected. If that happens after a
detach has started, the partition stays "pending detach" and the next call
finalizes and drops it.
LOCK_NOT_AVAILABLE
module-attribute
¶
SQLSTATE raised when lock_timeout expires.
PartitionError ¶
Bases: Exception
Partition maintenance could not be done.
PartitionLockTimeoutError ¶
Bases: PartitionError
A lock needed for partition maintenance was not granted within lock_timeout.
lock_timeout applies to each lock wait, not to the whole call, so
ensure_partitions blocked behind drop_expired and then behind an
open transaction can take about twice lock_timeout before this is
raised. From ensure_partitions, nothing was created; months_ahead
leaves room for a later retry. From drop_expired, the partitions dropped
before the timeout stay dropped (each one is logged), and a detach left
pending is finalized by the next call.
PartitionHealth
dataclass
¶
How far ahead one partitioned parent is covered by monthly partitions.
Attributes:
| Name | Type | Description |
|---|---|---|
table |
str | None
|
Schema-qualified name of the parent (the transaction table or a
severity partition), or |
covers_now |
bool
|
Whether the current UTC month has a partition. |
months_ahead |
int | None
|
Whole months after the current one covered without a
gap; |
below |
bool
|
Whether the current month is not covered or |
HealthReport
dataclass
¶
HealthReport(
min_months_ahead: int,
transaction: PartitionHealth,
activity: dict[int, PartitionHealth],
pending_detach: list[str],
orphaned: list[str],
)
Result of health.
Attributes:
| Name | Type | Description |
|---|---|---|
min_months_ahead |
int
|
The threshold the report was made with. |
transaction |
PartitionHealth
|
Coverage of the transaction table. |
activity |
dict[int, PartitionHealth]
|
Coverage of each requested severity, by severity value. |
pending_detach |
list[str]
|
Schema-qualified names of partitions left "pending
detach" by an interrupted |
orphaned |
list[str]
|
Schema-qualified names of tables in the audit schemas that
are named like monthly partitions but are no partition. A crash
between the detach and the drop in |
PartitionManager ¶
Runs partition maintenance on the library's own connections.
ensure_partitions runs in one transaction; drop_expired switches
its connection to AUTOCOMMIT itself.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
engine
|
Engine | AsyncEngine
|
Sync or async engine with DDL privileges. Must not be
configured for |
required |
tables
|
AuditTables
|
The audit tables, from |
required |
severities
|
Iterable[int]
|
A severity |
required |
Source code in audit_trail/maintenance.py
ensure_partitions ¶
Create missing partitions in one transaction; see ensure_partitions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
months_ahead
|
int
|
Months to create after the current UTC month. |
3
|
lock_timeout
|
str | None
|
PostgreSQL |
'5s'
|
Returns:
| Type | Description |
|---|---|
list[str]
|
Schema-qualified names of the partitions created. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the manager was built with an |
PartitionLockTimeoutError
|
If a lock was not granted in time. |
Source code in audit_trail/maintenance.py
aensure_partitions
async
¶
Async ensure_partitions, for a manager built with an AsyncEngine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
months_ahead
|
int
|
Months to create after the current UTC month. |
3
|
lock_timeout
|
str | None
|
PostgreSQL |
'5s'
|
Returns:
| Type | Description |
|---|---|
list[str]
|
Schema-qualified names of the partitions created. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the manager was built with a sync |
PartitionLockTimeoutError
|
If a lock was not granted in time. |
Source code in audit_trail/maintenance.py
drop_expired ¶
drop_expired(
retention: Mapping[_Severity, timedelta | None],
*,
transaction_retention: timedelta | None = None,
lock_timeout: str | None = "5s",
) -> list[str]
Drop expired partitions on an AUTOCOMMIT connection; see drop_expired.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
retention
|
Mapping[_Severity, timedelta | None]
|
Retention by severity value; |
required |
transaction_retention
|
timedelta | None
|
Retention of the transaction table. |
None
|
lock_timeout
|
str | None
|
PostgreSQL |
'5s'
|
Returns:
| Type | Description |
|---|---|
list[str]
|
Schema-qualified names of the partitions dropped. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the manager was built with an |
PartitionLockTimeoutError
|
If a lock was not granted in time. |
Source code in audit_trail/maintenance.py
adrop_expired
async
¶
adrop_expired(
retention: Mapping[_Severity, timedelta | None],
*,
transaction_retention: timedelta | None = None,
lock_timeout: str | None = "5s",
) -> list[str]
Async drop_expired, for a manager built with an AsyncEngine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
retention
|
Mapping[_Severity, timedelta | None]
|
Retention by severity value; |
required |
transaction_retention
|
timedelta | None
|
Retention of the transaction table. |
None
|
lock_timeout
|
str | None
|
PostgreSQL |
'5s'
|
Returns:
| Type | Description |
|---|---|
list[str]
|
Schema-qualified names of the partitions dropped. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the manager was built with a sync |
PartitionLockTimeoutError
|
If a lock was not granted in time. |
Source code in audit_trail/maintenance.py
health ¶
Report partition health; see health.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_months_ahead
|
int
|
Coverage threshold in months after the current one. |
2
|
Returns:
| Type | Description |
|---|---|
HealthReport
|
The report. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the manager was built with an |
Source code in audit_trail/maintenance.py
ahealth
async
¶
Async health, for a manager built with an AsyncEngine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_months_ahead
|
int
|
Coverage threshold in months after the current one. |
2
|
Returns:
| Type | Description |
|---|---|
HealthReport
|
The report. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the manager was built with a sync |
Source code in audit_trail/maintenance.py
ensure_partitions ¶
ensure_partitions(
connection: Connection,
tables: AuditTables,
severities: Iterable[int],
*,
months_ahead: int = 3,
now: datetime | None = None,
lock_timeout: str | None = "5s",
) -> list[str]
Create the missing severity and monthly partitions.
Covers the UTC month of now and the months_ahead months after it,
for the transaction table and for every severity. Idempotent: a partition
that already exists with the same or wider bounds, under any name, is
left alone.
Runs in the connection's current transaction, which must not be
AUTOCOMMIT; the caller commits, or rolls back on error. Concurrent
calls are serialized with a transaction-level advisory lock.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection
|
Connection
|
Connection with DDL privileges. |
required |
tables
|
AuditTables
|
The audit tables, from |
required |
severities
|
Iterable[int]
|
A severity |
required |
months_ahead
|
int
|
Months to create after the current one. |
3
|
now
|
datetime | None
|
Reference time. |
None
|
lock_timeout
|
str | None
|
PostgreSQL |
'5s'
|
Returns:
| Type | Description |
|---|---|
list[str]
|
Schema-qualified names of the partitions created, parents first. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
PartitionError
|
If a parent table does not exist, or a monthly partition name derived from an existing severity partition would be longer than PostgreSQL allows. |
PartitionLockTimeoutError
|
If a lock was not granted within
|
Source code in audit_trail/maintenance.py
drop_expired ¶
drop_expired(
connection: Connection,
tables: AuditTables,
retention: Mapping[_Severity, timedelta | None],
*,
transaction_retention: timedelta | None = None,
now: datetime | None = None,
lock_timeout: str | None = "5s",
) -> list[str]
Detach and drop the monthly partitions past their retention.
A partition is dropped when its upper bound is earlier than
now - retention; one whose upper bound is exactly that instant is
kept. Each partition is detached with DETACH PARTITION ...
CONCURRENTLY from its parent (the transaction table or a severity
partition) and then dropped, one at a time. Detaches left pending by an
interrupted call are finalized first. Concurrent calls, and
ensure_partitions, are serialized with an advisory lock.
DETACH ... CONCURRENTLY cannot run inside a transaction block, so
connection must be in AUTOCOMMIT mode. lock_timeout is set for
the session during the call and restored afterwards. The locks taken are
listed in the module documentation. Waiting for application transactions
that have used an audit table counts against lock_timeout, so a long
open transaction makes this call fail; that is expected, and retrying
later is safe.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection
|
Connection
|
|
required |
tables
|
AuditTables
|
The audit tables, from |
required |
retention
|
Mapping[_Severity, timedelta | None]
|
Retention by severity value. |
required |
transaction_retention
|
timedelta | None
|
Retention of the transaction table. It is
capped at the shortest finite severity retention, with a warning
if it is longer; |
None
|
now
|
datetime | None
|
Reference time, timezone-aware. |
None
|
lock_timeout
|
str | None
|
PostgreSQL |
'5s'
|
Returns:
| Type | Description |
|---|---|
list[str]
|
Schema-qualified names of the partitions dropped, in order. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the connection is not in |
PartitionError
|
If a parent table does not exist. |
PartitionLockTimeoutError
|
If a lock was not granted within
|
Source code in audit_trail/maintenance.py
314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 | |
health ¶
health(
connection: Connection,
tables: AuditTables,
severities: Iterable[int],
*,
min_months_ahead: int = 2,
now: datetime | None = None,
) -> HealthReport
Report partition coverage, pending detaches and orphaned partitions.
Read-only; runs on any connection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection
|
Connection
|
Connection that can read the catalog. |
required |
tables
|
AuditTables
|
The audit tables, from |
required |
severities
|
Iterable[int]
|
A severity |
required |
min_months_ahead
|
int
|
Coverage below this many months after the current
one is flagged. The default of |
2
|
now
|
datetime | None
|
Reference time, timezone-aware. |
None
|
Returns:
| Type | Description |
|---|---|
HealthReport
|
The report. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
PartitionError
|
If a parent table does not exist. |
Source code in audit_trail/maintenance.py
396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 | |