onprema

Airflow Secrets Backend

Sensitive information can be stored in the Airflow metastore (usually as Variables or Connections). By default, the metastore can encrypt sensitive values using a Fernet key.

But if someone gets ahold on the Fernet key and has access to the database, they can read those sensitive values.

This is a huge no-no for customers of Astro because they own the database and could therefore view any sensitive information stored in it (because they also know the fernet key)

Another strategy is to use a third-party Secrets Backend. Supported ones include: Vault, AWS Secrets Manager and Parameter Store, Azure Key Vault and Google Cloud Secrets Manager.

You can configure a secrets backend using known AIRFLOW_ variables. For example:

AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>"}
AWS_DEFAULT_REGION=<region>

This way, you aren't storing any secrets in the metastore!

#airflow #security