Looker (dagster-looker)
Dagster allows you to represent your Looker project as assets, alongside other your other technologies like dbt and Sling. This allows you to see how your Looker assets are connected to your other data assets, and how changes to other data assets might impact your Looker project.
Looker API
Here, we provide interfaces to manage Looker projects using the Looker API.
Assets (Looker API)
- class dagster_looker.LookerResource
- experimental
This API may break in future versions, even between dot releases.
Represents a connection to a Looker instance and provides methods to interact with the Looker API.
- build_defs
- deprecated
This API will be removed in version 1.9.0. Use dagster_looker.load_looker_asset_specs instead.
Returns a Definitions object which will load structures from the Looker instance and translate it into assets, using the provided translator.
Parameters:
- request_start_pdt_builds (Optional[Sequence[RequestStartPdtBuildRequestStartPdtBuild]]) – A list of
- dagster_looker_translator (Optional[DagsterLookerApiTranslatorDagsterLookerApiTranslator]) – The translator to
Returns: A Definitions object which will contain return the Looker structures as assets.Return type: Definitions
- class dagster_looker.RequestStartPdtBuild
A request to start a PDT build. See https://developers.looker.com/api/explorer/4.0/types/DerivedTable/RequestStartPdtBuild?sdk=py for documentation on all available fields.
Parameters:
- model_name – The model of the PDT to start building.
- view_name – The view name of the PDT to start building.
- force_rebuild – Force rebuild of required dependent PDTs, even if they are already materialized.
- force_full_incremental – Force involved incremental PDTs to fully re-materialize.
- workspace – Workspace in which to materialize selected PDT (‘dev’ or default ‘production’).
- source – The source of this request.
- class dagster_looker.LookerFilter
Filters the set of Looker objects to fetch.
Parameters:
- dashboard_folders (Optional[List[List[str]]]) – A list of folder paths to fetch dashboards from.
- only_fetch_explores_used_in_dashboards (bool) – If True, only explores used in the fetched dashboards
- dagster_looker.load_looker_asset_specs
- experimental
This API may break in future versions, even between dot releases.
Returns a list of AssetSpecs representing the Looker structures.
Parameters:
- looker_resource (LookerResourceLookerResource) – The Looker resource to fetch assets from.
- dagster_looker_translator (Type[DagsterLookerApiTranslatorDagsterLookerApiTranslator]) – The translator to use
Returns: The set of AssetSpecs representing the Looker structures.Return type: List[AssetSpec]
- dagster_looker.build_looker_pdt_assets_definitions
- experimental
This API may break in future versions, even between dot releases.
Returns the AssetsDefinitions of the executable assets for the given the list of refreshable PDTs.
Parameters:
- resource_key (str) – The resource key to use for the Looker resource.
- request_start_pdt_builds (Optional[Sequence[RequestStartPdtBuildRequestStartPdtBuild]]) – A list of requests to start PDT builds.
- dagster_looker_translator (Optional[DagsterLookerApiTranslatorDagsterLookerApiTranslator]) – The translator to
Returns: The AssetsDefinitions of the executable assets for the given the list of refreshable PDTs.Return type: AssetsDefinition
lkml (LookML)
Here, we provide interfaces to manage Looker projects defined a set of locally accessible LookML files.
Assets (lkml)
- dagster_looker.build_looker_asset_specs
- experimental
This API may break in future versions, even between dot releases.
Build a list of asset specs from a set of Looker structures defined in a Looker project.
Parameters:
- project_dir (Path) – The path to the Looker project directory.
- dagster_looker_translator (Optional[DagsterLookerTranslator]) – Allows customizing how to
Examples:
from pathlib import Path
from dagster import external_assets_from_specs
from dagster_looker import build_looker_asset_specs
looker_specs = build_looker_asset_specs(project_dir=Path("my_looker_project"))
looker_assets = external_assets_from_specs(looker_specs)
- class dagster_looker.DagsterLookerLkmlTranslator
- experimental
This API may break in future versions, even between dot releases.
Holds a set of methods that derive Dagster asset definition metadata given a representation of a LookML structure (dashboards, explores, views).
This class is exposed so that methods can be overriden to customize how Dagster asset metadata is derived.
- get_asset_key
A method that takes in a LookML structure (dashboards, explores, views) and returns the Dagster asset key that represents the structure.
The LookML structure is parsed using
lkml. You can learn more about this here: https://lkml.readthedocs.io/en/latest/simple.html.You can learn more about LookML dashboards and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/param-lookml-dashboard.
You can learn more about LookML explores and views and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/lookml-quick-reference.
This method can be overriden to provide a custom asset key for a LookML structure.
Parameters: lookml_structure (Tuple[Path, str, Mapping[str, Any]]) – A tuple with the path to file defining a LookML structure, the LookML structure type, and a dictionary representing a LookML structure.Returns: The Dagster asset key that represents the LookML structure.Return type: AssetKey
- get_deps
A method that takes in a LookML structure (dashboards, explores, views) and returns the Dagster dependencies of that the structure.
The LookML structure is parsed using
lkml. You can learn more about this here: https://lkml.readthedocs.io/en/latest/simple.html.You can learn more about LookML dashboards and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/param-lookml-dashboard.
You can learn more about LookML explores and views and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/lookml-quick-reference.
This method can be overriden to provide custom dependencies for a LookML structure.
Parameters: lookml_structure (Tuple[Path, str, Mapping[str, Any]]) – A tuple with the path to file defining a LookML structure, the LookML structure type, and a dictionary representing a LookML structure.Returns: The Dagster dependencies for the LookML structure.Return type: AssetKey
- get_description
A method that takes in a LookML structure (dashboards, explores, views) and returns the Dagster description of the structure.
The LookML structure is parsed using
lkml. You can learn more about this here: https://lkml.readthedocs.io/en/latest/simple.html.You can learn more about LookML dashboards and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/param-lookml-dashboard.
You can learn more about LookML explores and views and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/lookml-quick-reference.
This method can be overriden to provide a custom description for a LookML structure.
Parameters: lookml_structure (Tuple[Path, str, Mapping[str, Any]]) – A tuple with the path to file defining a LookML structure, the LookML structure type, and a dictionary representing a LookML structure.Returns: The Dagster description for the LookML structure.Return type: Optional[str]
- get_group_name
A method that takes in a LookML structure (dashboards, explores, views) and returns the Dagster group name of the structure.
The LookML structure is parsed using
lkml. You can learn more about this here: https://lkml.readthedocs.io/en/latest/simple.html.You can learn more about LookML dashboards and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/param-lookml-dashboard.
You can learn more about LookML explores and views and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/lookml-quick-reference.
This method can be overriden to provide a custom group name for a LookML structure.
Parameters: lookml_structure (Tuple[Path, str, Mapping[str, Any]]) – A tuple with the path to file defining a LookML structure, the LookML structure type, and a dictionary representing a LookML structure.Returns: A Dagster group name for the LookML structure.Return type: Optional[str]
- get_metadata
A method that takes in a LookML structure (dashboards, explores, views) and returns the Dagster metadata of the structure.
The LookML structure is parsed using
lkml. You can learn more about this here: https://lkml.readthedocs.io/en/latest/simple.html.You can learn more about LookML dashboards and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/param-lookml-dashboard.
You can learn more about LookML explores and views and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/lookml-quick-reference.
This method can be overriden to provide custom metadata for a LookML structure.
Parameters: lookml_structure (Tuple[Path, str, Mapping[str, Any]]) – A tuple with the path to file defining a LookML structure, the LookML structure type, and a dictionary representing a LookML structure.Returns: A dictionary representing the Dagster metadata for the LookML structure.
Return type: Optional[Mapping[str, Any]]
- get_owners
A method that takes in a LookML structure (dashboards, explores, views) and returns the Dagster owners of the structure.
The LookML structure is parsed using
lkml. You can learn more about this here: https://lkml.readthedocs.io/en/latest/simple.html.You can learn more about LookML dashboards and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/param-lookml-dashboard.
You can learn more about LookML explores and views and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/lookml-quick-reference.
This method can be overriden to provide custom owners for a LookML structure.
Parameters: lookml_structure (Tuple[Path, str, Mapping[str, Any]]) – A tuple with the path to file defining a LookML structure, the LookML structure type, and a dictionary representing a LookML structure.Returns: A sequence of Dagster owners for the LookML structure.Return type: Optional[Sequence[str]]
- get_tags
A method that takes in a LookML structure (dashboards, explores, views) and returns the Dagster tags of the structure.
The LookML structure is parsed using
lkml. You can learn more about this here: https://lkml.readthedocs.io/en/latest/simple.html.You can learn more about LookML dashboards and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/param-lookml-dashboard.
You can learn more about LookML explores and views and the properties available in this dictionary here: https://cloud.google.com/looker/docs/reference/lookml-quick-reference.
This method can be overriden to provide custom tags for a LookML structure.
Parameters: lookml_structure (Tuple[Path, str, Mapping[str, Any]]) – A tuple with the path to file defining a LookML structure, the LookML structure type, and a dictionary representing a LookML structure.Returns: A dictionary representing the Dagster tags for the LookML structure.
Return type: Optional[Mapping[str, str]]