Adding attributes to assets in a subdirectory
dg and Dagster Components are under active development. You may encounter feature gaps, and the APIs may change. To report issues or give feedback, please join the #dg-components channel in the Dagster Community Slack.
Attaching and modifying meta information on definitions such as assets is important in Dagster projects. You want to have groups, teams, owners, kinds, tags, and metadata set correctly to organize definitions and ensure that other tools and processes that rely on them correctly function.
Within a dg-driven defs project layout, you can apply attribute transformations at any point in the directory structure. This supports uses cases ranging from ensuring that all definitions in a particular folder have an owner set to a particular team to more complex workflows involving applying a group conditionally based on other properties of the definition.
Example
First, we'll look at a project with assets defined in a subdirectory. You can quickly replicate this example by running the following commands:
dg scaffold project my-project \
&& cd my-project/src \
&& dg scaffold dagster.asset team_a/subproject/a.py \
&& dg scaffold dagster.asset team_a/b.py \
&& dg scaffold dagster.asset team_b/c.py
There are three assets defined in various subdirectories:
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━┳━━━━━━━━━ ┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│ │ │ a │ default │ │ │ │ │
│ │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│ │ │ b │ default │ │ │ │ │
│ │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│ │ │ c │ default │ │ │ │ │
│ │ └─────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴────────────────────────────────────────────────┘
tree my_project/defs
my_project/defs
├── __init__.py
├── team_a
│ ├── b.py
│ └── subproject
│ └── a.py
└── team_b
└── c.py
4 directories, 4 files
Let's say we want to add a team_a asset group to all assets in the team_a folder. We can add a component.yaml file to the my_project/defs/team_a directory with the following contents:
type: dagster.components.DefsFolderComponent
attributes:
asset_post_processors:
- target: "*"
attributes:
group_name: "team_a"
tree my_project/defs
my_project/defs
├── __init__.py
├── team_a
│ ├── b.py
│ ├── component.yaml
│ └── subproject
│ └── a.py
└── team_b
└── c.py
4 directories, 5 files
Finally, we can run dg list defs again to see the new group applied to the assets in the team_a subdirectory:
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│ │ │ a │ team_a │ │ │ │ │
│ │ ├─────┼─────────┼──────┼───────┼─────────────┤ │
│ │ │ b │ team_a │ │ │ │ │
│ │ ├─────┼─────────┼──────┼─────── ┼─────────────┤ │
│ │ │ c │ default │ │ │ │ │
│ │ └─────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴────────────────────────────────────────────────┘
Advanced usage
The configuration for the dagster.components.DefsFolderComponent can be more complex than the example above. You can apply multiple transforms to the assets in the subdirectory, each of which will be processed in order.
You can also target the configuration to a specific selection of assets using the target field. This field uses Dagster's Selection Syntax. All selections are evaluated against the assets defined within the subdirectory.