Databricks Advanced Security: Implementing Row Filters and Column Masks

While Unity Catalog provides a powerful foundation for centralised data governance, standard table level GRANT statements are often insufficient for modern enterprise security. Organisations frequently encounter scenarios where different user groups must query the same table, but access to specific rows or sensitive columns must be restricted based on their identity.

Historically, this required engineering complex dynamic views or maintaining duplicate datasets. Databricks resolves this natively within Unity Catalog through Row Filters and Column Masks. This eliminates architectural redundancy while ensuring strict, programmatic compliance.

Transitioning from Dynamic Views to Native Functions

Before the introduction of native row filters and column masks in Unity Catalog, data engineers relied on dynamic views utilising functions like is_account_group_member(). While effective, views decoupled the security logic from the underlying table, complicating lineage tracing and increasing administrative overheads.

Native row filters and column masks are standard SQL User-Defined Functions (UDFs) that are bound directly to the table itself. Whenever the table is queried, regardless of the compute context (Databricks SQL, a Spark DataFrame, or a third party BI tool), the security logic is automatically evaluated and enforced entirely at the engine level.

Implementing Row Level Security

Row filters evaluate a boolean expression for every row. If the function returns TRUE, the row is visible to the executing user; if FALSE, the row is silently filtered out.

Consider a sales table containing global data. We want European analysts to only see European rows, while global administrators can see everything.

First, we define the filter function within our catalog:

CREATE FUNCTION catalog.security.eu_region_filter(region STRING)
RETURN IF(
  is_account_group_member('global_admins'),
  TRUE,
  region = 'EU'
);

Next, we attach this function as a row filter to our target table during creation (or via an ALTER TABLE statement):

CREATE TABLE catalog.sales.transactions (
  transaction_id INT,
  region STRING,
  amount DECIMAL(10,2)
)
WITH ROW FILTER catalog.security.eu_region_filter ON (region);

Because the function leverages is_account_group_member(), the filter dynamically adapts to the user executing the query.

Implementing Column Level Masking

Column masks operate similarly but manipulate the contents of a specific column rather than filtering the entire row. This is particularly critical for PII (Personally Identifiable Information) such as email addresses or national insurance numbers.

Let us create a masking function that exposes only the domain of an email address unless the user belongs to the HR department.

CREATE FUNCTION catalog.security.email_domain_mask(email STRING)
RETURN IF(
  is_account_group_member('hr_department'),
  email,
  regexp_replace(email, '^.*(@.*)$', '****$1')
);

We then apply this mask directly to the column definition:

CREATE TABLE catalog.hr.employees (
  employee_id INT,
  name STRING,
  email STRING MASK catalog.security.email_domain_mask
);

When an unauthorised user queries the employees table, they will see rows formatted as ****@company.com, preserving analytical capabilities (such as grouping by domain) while protecting the sensitive identifier.

Architectural Advantages and Challenges

The primary benefit of this approach is centralised, foolproof governance. Because the security policies are bound to the table structure within Unity Catalog, there is zero risk of an analyst circumventing the rules by accessing the Parquet files directly or querying via an unexpected endpoint.

However, engineers must be cautious of performance implications. Because these UDFs are evaluated dynamically at query time, highly complex filtering logic containing nested subqueries or massive conditional trees can degrade query performance. It is best practice to keep the logic within row filters and column masks as streamlined as possible.

Conclusion

Mastering native row filters and column masks in Unity Catalog is essential for designing resilient, secure data architectures. By moving away from fragile dynamic views to declarative, table bound policies, organisations can dramatically enhance their security posture whilst simplifying administrative workflows.

Ready to implement advanced data security? A secure architecture protects your most valuable assets. Contact us to discuss how we can safeguard your data platform.