Planning Your Enterprise AI Investments? Discover How to Choose the Right AI Use Case →

Solution Accelerator

Data Governance

· Data Retention · Databricks

Data Retention & Archival Framework

Reference workflows, reusable notebooks, and implementation guidance to apply configurable retention rules in Databricks, protect records on hold, and mark eligible records as archived with decision logs and run details.

Language

Python / PySpark

Deployment

Databricks Notebooks

Type

Framework + Reference Code

Trusted By :

How it works

Apply retention rules while protecting records on hold

Retention rules need to account for both record age and conditions that require records to remain available. Maintaining these rules in separate scripts can make them harder to update, apply consistently, and review.

DataTheta’s Data Retention & Archival Framework reads settings from one JSON configuration file and applies KEEP, HOLD, or ARCHIVE decisions. It updates record statuses, captures archival details, and exposes active and archived records through separate views, with held records included in the active view.

  • Configure and validate: Define the date column for record age, archive threshold, hold rules, and evaluation date in config/retention_config.json. Validate the configuration before applying retention rules.
  • Evaluate each record: Calculate record age using the configured date column and a fixed as_of_date or today’s date. Apply KEEP, HOLD, or ARCHIVE decisions according to the configured rules.
  • Apply archival and hold decisions: Mark eligible records ARCHIVED and capture the archive date, run ID, and applied rule. Protect records matching hold rules from archival.
  • Review the results: Inspect active and held records in the active view, archived records in the archive view, and record counts, mode, and status in the pipeline run log.
AI systems in production
0 +
Avg. time to first outcome
0 Weeks
Forecast accuracy
0 %
Faster decision cycles
0 X
Revenue influenced by AI
$ 0 M+
Manual processing eliminated
0 %

40+

AI systems in production

8 Weeks

Avg. time to first outcome

34%

Forecast accuracy improvement

3×

Faster decision cycles

$180M+

Revenue influenced by AI

68%

Manual processing eliminated

Key Capabilities

What this accelerator helps you do

Configurable Retention Rules

Define retention settings, the date column for record age, and hold conditions in one JSON file. Update rules without changing the main pipeline code.

Record Age & Decision Logic

Evaluate record age using a fixed as_of_date or today’s date. Apply KEEP, HOLD, or ARCHIVE decisions according to the configured rules.

Hold Rule Protection

Protect records matching configured hold conditions from archival. Keep held records available in the active view alongside other retained records.

Archival Status & Traceability

Mark eligible records ARCHIVED and capture the archive date, run ID, and applied rule. Expose archived records through a dedicated archive view.

Results Views & Run Logging

Review active and archived datasets alongside pipeline record counts, processing mode, and execution status. Use the results notebook to inspect each run’s outputs.

Solution Accelerators

Apply reusable data lifecycle patterns with DataTheta Solution Accelerators.

Explore reusable accelerators for data retention, governance, data quality, reconciliation, synchronization, and Databricks data engineering.

Ready to simplify data retention and archival?

Use DataTheta’s Data Retention & Archival Framework to apply retention rules, protect held records, and review archival decisions and pipeline results in Databricks.

©2026 Copyright DataTheta – Lance Labs Inc.

Let's Build Your Next Data & AI Solution

Tell us about your business goals, data challenges, or technology requirements. Our team will review your enquiry and connect with you to discuss the right solution.

No spam. Your information stays private.