Engine

Code. Compile. Check.

The SDF engine provides a holistic view of your SQL ecosystem by considering all queries in a warehouse simultaneously. Uncover problems proactively and optimize your data infrastructure with compute, not more.

Book Demo

SDF build, SDF lineage, and SDF deploy output wrapper

Introducing Code Checks for SQL

Code Checks are rules for your SQL. Code checks are powerful and super easy to use on your laptop or in CI/CD. Check out our first blog post for Code Check examples.

Enhanced Security
Write custom compile-time code checks to safeguard sensitive data from unauthorized access
Debug Your Data Like Code
Code checks bring data engineering closer to software engineering with visibility rules, type correctness, naming conventions, and more.
Proactive Error Prevention
Leverage compile time analysis and CI/CD integration to catch mistakes before they impact your data integrity
Lower Cost
Data tests and backfills are expensive. SDF allows you to provide strong guarantees across your warehouse, org-wide, all from the comfort of your laptop

Code Check Use Cases

Protect Sensitive Data with SDF Code Checks

Ensure All PII is Appropriately Anonymized

Prevent Mixing Currencies with SDF Code Checks

Prevent Different Currencies from Combining

Prevent time grain disparities with SDF Code Checks

Ensure Tables Respect their Visibility Specification

Workspace Blocks

SDF Workspaces organize and manage SQL project dependencies. A workspace is a directory defined by a workspace.sdf.yml. The file must start with a Workspace block, and may contain YML blocks for tables, functions, classifiers and more.
‍
Use workspace blocks to define:

The location of SQL source files and YML configuration files
The default SQL dialect, catalog and schema, globally or per location
Code-checks and their locations
References to other workspaces

Table Blocks

Tables can be defined in either SQL or in YML. But only SDF Table blocks specify metadata for SQL tables.

Use Table blocks to:

Define a table (if it not given in SQL), including its columns
Add descriptions to tables and columns to grow your catalog
Add classifiers to table and/or column.

Classifier Blocks

Classifiers are a powerful extended typing system for data. In SDF they are used to provide more meaning to data, i.e. they can provide units of measurement (e.g. USD, meter), represent the state of data representation (e.g. PII is in clear text or anonymized), or provide the purpose of a table (e.g. this table is public).

Attach a classifier to a table or column; SDF then tracks the flow of a classifier within the whole workspace.

Use Classifier blocks to:

Define classifiers & labels including
Add descriptions to classifiers and labels to grow your catalog
Configure classifier behavior: their scope, propagation & cardinality

Function Blocks

Functions are either builtin or user defined. They are dialect specific. SDF Function blocks specify re-classifications for builtin- and user defined functions.

Use Function blocks to:

Define the signature of dialect specific function, including its parameters and return types
Add descriptions to function names and parameters, a section tag and executable examples of proper usage to generate an indexed function catalog
Add reclassify lists, to define how a function reclassifies the label(s) of a classifier

Profile Blocks

SDF profiles are used to specify different configurations for your project. Technically a profile blocks simply overrides anything that appears in a workspace, sources, classifiers, checks. Think of it like a built-in A/B.
‍
Use Profile blocks to:

Segment your warehouse
Customize the workspace for the intended use, such as development, test, release.
Test different classification or propagation rules in real time
Deploy multiple profiles to track effects and changes

Extensive SQL DialectCompatibility

SDF is a true multi-dialect compiler, supporting: Redshift, Presto, Trino, Big Query, and ANSI SQL.

SDF Supports Amazon Redshift SQL Dialect

Coming Soon...

An Example of SDF Propagating Classifiers through a data warehouse

Label the Sources. Automate the Rest.

Label root tables and watch as your classifiers flow through your entire warehouse automatically. Configure rules to tell SDF where to stop propagation or reclassify based on logic you define in YML

<1% of Tables Need Manual Classifiers

Classifiers Propagate to the Other 99%

Flexible reclassification

Step into Modern SQL

Run the Engine Locally with Ease

Frictionless onboarding with no migration required

Experience the speed of a compiler written in Rust with built-in caching and datafusion under the hood

Learn More