Skip to main content

Great Expectations (dagster-ge)

dagster_ge.ge_validation_op_factory

Generates ops for interacting with Great Expectations.

Parameters:

    • name (str) – the name of the op

    • datasource_name (str) – the name of your DataSource, see your great_expectations.yml

    • data_connector_name (str) – the name of the data connector for this datasource. This should

    • data_asset_name (str) – the name of the data asset that this op will be validating.

    • suite_name (str) – the name of your expectation suite, see your great_expectations.yml

    • batch_identifier_fn (dict) – A dicitonary of batch identifiers to uniquely identify this

    • input_dagster_type (DagsterTypeDagsterType) – the Dagster type used to type check the input to the op.

    • runtime_method_type (str) – how GE should interperet the op input. One of (“batch_data”, extra_kwargs (Optional[dict]) –

      adds extra kwargs to the invocation of ge_data_context’s get_validator method. If not set, input will be:

      \{
          "datasource_name": datasource_name,
          "data_connector_name": data_connector_name,
          "data_asset_name": data_asset_name,
          "runtime_parameters": \{
              "\<runtime_method_type>": \<op input>
          },
          "batch_identifiers": batch_identifiers,
          "expectation_suite_name": suite_name,
      }

Returns: An op that takes in a set of data and yields both an expectation with relevant metadata and an output with all the metadata (for user processing)