adc_toolkit.data.validators.gx

 20@dataclass
 21class BatchManager:
 22    """
 23    Coordinate batch metadata and operations for Great Expectations validation.
 24
 25    The BatchManager dataclass serves as a central coordination point in the
 26    Great Expectations (GX) validation workflow. It encapsulates all essential
 27    metadata about a validation batch, including the dataset name, the data to
 28    be validated, and the GX data context. Upon initialization, it automatically
 29    creates a BatchRequest that can be used by downstream validation components
 30    (expectation strategies, checkpoint managers) to execute validations.
 31
 32    This class acts as a bridge between raw data and the Great Expectations
 33    validation engine. It delegates datasource management to DatasourceManager,
 34    which handles the details of registering pandas or PySpark datasources with
 35    the GX context, and constructs the BatchRequest that defines how GX should
 36    access and validate the data.
 37
 38    The BatchManager is typically instantiated by the validate_dataset function
 39    and passed to ExpectationAdditionStrategy implementations (which add
 40    validation rules) and CheckpointManager (which executes validations and
 41    evaluates results).
 42
 43    Parameters
 44    ----------
 45    name : str
 46        The logical name identifying this dataset or validation batch. This
 47        name is used to identify the data asset within the GX datasource and
 48        is typically used as the basis for naming expectation suites (e.g.,
 49        "{name}_suite"). Must be a valid identifier string.
 50    data : Data
 51        The dataset to be validated. This can be a pandas DataFrame, PySpark
 52        DataFrame, or any other data structure conforming to the Data protocol.
 53        The data will be registered with GX as a dataframe asset for validation.
 54    data_context : AbstractDataContext
 55        The Great Expectations data context that manages datasources, expectation
 56        suites, checkpoints, and validation results. This can be an
 57        EphemeralDataContext (in-memory, for testing or transient workflows),
 58        FileDataContext (persistent, file-based), or cloud-backed contexts
 59        (AWS, GCP, Azure). The context provides the validation infrastructure
 60        and configuration.
 61
 62    Attributes
 63    ----------
 64    name : str
 65        The logical name of the dataset being validated.
 66    data : Data
 67        The dataset to be validated.
 68    data_context : AbstractDataContext
 69        The Great Expectations data context managing validation infrastructure.
 70    batch_request : BatchRequest
 71        The Great Expectations BatchRequest object created during initialization.
 72        This request encapsulates the datasource name, data asset name, and
 73        dataframe reference needed by GX to access and validate the data.
 74        Automatically created by __post_init__ via create_batch_request().
 75
 76    See Also
 77    --------
 78    DatasourceManager : Manages GX datasource registration for pandas and PySpark data.
 79    CheckpointManager : Executes validation checkpoints using BatchManager metadata.
 80    validate_dataset : Main validation function that instantiates BatchManager.
 81    ExpectationAdditionStrategy : Adds validation expectations using BatchManager.
 82
 83    Notes
 84    -----
 85    The BatchManager uses a dataclass design with automatic initialization via
 86    __post_init__. The batch_request field is not part of the constructor
 87    signature (field(init=False)) but is automatically created after the main
 88    fields are initialized.
 89
 90    The workflow is as follows:
 91
 92    1. User calls validate_dataset() with data, name, and strategies
 93    2. validate_dataset() creates a BatchManager instance
 94    3. BatchManager.__post_init__() calls create_batch_request()
 95    4. create_batch_request() delegates to DatasourceManager to register datasource
 96    5. A data asset is added to the datasource with the specified name
 97    6. A BatchRequest is built referencing the dataframe
 98    7. The BatchManager (with batch_request populated) is passed to strategies
 99    8. ExpectationAdditionStrategy adds expectations to the suite
100    9. CheckpointManager runs validation and evaluates results
101
102    The BatchManager supports both pandas and PySpark DataFrames through the
103    DatasourceManager abstraction, which automatically detects the dataframe
104    type and registers the appropriate GX datasource (PandasDatasource or
105    SparkDFDatasource).
106
107    Examples
108    --------
109    Create a BatchManager for a pandas DataFrame with an ephemeral context:
110
111    >>> import pandas as pd
112    >>> from great_expectations.data_context import EphemeralDataContext
113    >>> from adc_toolkit.data.validators.gx.batch_managers.batch_manager import BatchManager
114    >>> data = pd.DataFrame(
115    ...     {
116    ...         "col1": [1, 2, 3],
117    ...         "col2": [4.0, 5.0, 6.0],
118    ...         "col3": ["a", "b", "c"],
119    ...     }
120    ... )
121    >>> context = EphemeralDataContext()
122    >>> batch_manager = BatchManager(
123    ...     name="my_dataset",
124    ...     data=data,
125    ...     data_context=context,
126    ... )
127    >>> print(batch_manager.batch_request.data_asset_name)
128    my_dataset
129    >>> print(batch_manager.batch_request.datasource_name)
130    pandas_datasource
131
132    Use BatchManager within the full validation workflow:
133
134    >>> from adc_toolkit.data.validators.gx.batch_managers.batch_validation import validate_dataset
135    >>> from adc_toolkit.data.validators.gx.batch_managers.expectation_suite_lookup_strategy import (
136    ...     AutoExpectationSuiteCreation,
137    ... )
138    >>> from adc_toolkit.data.validators.gx.batch_managers.expectation_addition_strategy import (
139    ...     SchemaExpectationAddition,
140    ... )
141    >>> validated_data = validate_dataset(
142    ...     name="my_dataset",
143    ...     data=data,
144    ...     data_context=context,
145    ...     expectation_suite_lookup_strategy=AutoExpectationSuiteCreation(),
146    ...     expectation_addition_strategy=SchemaExpectationAddition(),
147    ... )
148
149    Access the batch request for custom validation logic:
150
151    >>> batch_request = batch_manager.batch_request
152    >>> validator = context.get_validator(batch_request=batch_request, expectation_suite_name="my_dataset_suite")
153    >>> validation_result = validator.validate()
154    """
155
156    name: str
157    data: Data
158    data_context: AbstractDataContext
159    batch_request: BatchRequest = field(init=False)
160
161    def __post_init__(self) -> None:
162        """
163        Initialize computed fields after dataclass initialization.
164
165        This method is automatically called by the dataclass machinery after
166        __init__ completes. It creates and populates the batch_request field
167        by calling create_batch_request(). This two-stage initialization allows
168        the batch_request to be automatically computed from the name, data, and
169        data_context fields without requiring explicit initialization by the
170        caller.
171
172        The method ensures that every BatchManager instance has a valid
173        batch_request immediately after construction, ready to be used by
174        downstream validation components.
175
176        Notes
177        -----
178        This method is called automatically and should not be invoked manually.
179        It is part of the dataclass lifecycle and implements the deferred
180        initialization pattern for computed fields.
181
182        See Also
183        --------
184        create_batch_request : Creates the BatchRequest for this batch.
185        """
186        self.batch_request = self.create_batch_request()
187
188    def create_batch_request(self) -> BatchRequest:
189        """
190        Create a Great Expectations BatchRequest for this validation batch.
191
192        This method orchestrates the creation of a BatchRequest by delegating
193        datasource management to DatasourceManager, adding a dataframe asset to
194        the datasource, and building a batch request that references the data.
195
196        The process involves:
197
198        1. Instantiate DatasourceManager with the data and context
199        2. Add or update the appropriate datasource (pandas or PySpark) in the
200           data context based on the detected dataframe type
201        3. Add a dataframe asset to the datasource with the specified name
202        4. Build and return a BatchRequest linking the data asset to the
203           in-memory dataframe
204
205        The resulting BatchRequest can be used by Great Expectations validators,
206        checkpoints, and other components to access and validate the data.
207
208        Returns
209        -------
210        BatchRequest
211            A Great Expectations BatchRequest object containing the datasource
212            name (e.g., "pandas_datasource" or "pyspark_datasource"), data
213            asset name (same as self.name), and a reference to the dataframe.
214            This request can be passed to GX validators and checkpoints to
215            execute validations.
216
217        Notes
218        -----
219        This method is called automatically during __post_init__ and typically
220        should not be called manually. It delegates the complexity of datasource
221        type detection and registration to DatasourceManager, keeping the
222        BatchManager logic clean and focused on coordination.
223
224        The datasource is added or updated (not just retrieved) to ensure it
225        exists in the data context. If a datasource with the same name already
226        exists, GX updates it; otherwise, it creates a new one.
227
228        The dataframe asset is ephemeral and exists only for the duration of
229        this validation batch. Each call to create_batch_request() creates a
230        new asset with the specified name, which may overwrite previous assets
231        with the same name.
232
233        See Also
234        --------
235        DatasourceManager.add_or_update_datasource : Registers the datasource with GX.
236
237        Examples
238        --------
239        The create_batch_request method is called automatically, but its behavior
240        can be understood through manual usage:
241
242        >>> import pandas as pd
243        >>> from great_expectations.data_context import EphemeralDataContext
244        >>> from adc_toolkit.data.validators.gx.batch_managers.batch_manager import BatchManager
245        >>> data = pd.DataFrame({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})
246        >>> context = EphemeralDataContext()
247        >>> batch_manager = BatchManager(name="test_data", data=data, data_context=context)
248        >>> # batch_request is automatically created via __post_init__
249        >>> batch_request = batch_manager.batch_request
250        >>> print(f"Datasource: {batch_request.datasource_name}")
251        Datasource: pandas_datasource
252        >>> print(f"Asset: {batch_request.data_asset_name}")
253        Asset: test_data
254
255        The batch request can be used to get a validator:
256
257        >>> validator = context.get_validator(batch_request=batch_request, expectation_suite_name="test_suite")
258        """
259        datasource = DatasourceManager(self.data, self.data_context).add_or_update_datasource()
260        data_asset = datasource.add_dataframe_asset(name=self.name)
261        return data_asset.build_batch_request(dataframe=self.data)

 12class ConfigurationBasedExpectationAddition:
 13    """
 14    Add Great Expectations to a suite from dictionary-based configuration.
 15
 16    This class implements the ExpectationAddition protocol to add expectations
 17    to Great Expectations suites based on structured dictionary configurations.
 18    It provides a declarative approach to defining data validation rules without
 19    writing code, making it suitable for configuration-driven validation workflows.
 20
 21    The class processes expectation dictionaries where each dictionary contains
 22    a single expectation type as the key and its parameters as the value. It
 23    leverages the `parse_expectations_dict` function to extract and validate
 24    the expectation structure before creating ExpectationConfiguration objects
 25    and adding them to the target suite.
 26
 27    This implementation is particularly useful for:
 28
 29    - Loading expectations from YAML or JSON configuration files
 30    - Dynamically building validation suites from user-defined configurations
 31    - Separating validation logic from code for better maintainability
 32    - Enabling non-technical users to define validation rules
 33
 34    Attributes
 35    ----------
 36    None
 37        This class is stateless and requires no instance attributes.
 38
 39    See Also
 40    --------
 41    parse_expectations_dict : Parses and validates expectation dictionary structure.
 42    ExpectationAddition : Protocol defining the expectation addition interface.
 43    BatchManager : Manages Great Expectations batches and data contexts.
 44
 45    Notes
 46    -----
 47    The class expects expectation dictionaries to follow a specific format:
 48
 49    - Each dictionary must contain exactly one key-value pair
 50    - The key is the expectation type (e.g., "expect_column_values_to_be_in_set")
 51    - The value is a dictionary of expectation parameters (kwargs)
 52
 53    The expectation suite is retrieved using the naming convention
 54    "{batch_manager.name}_suite" and is automatically updated in the data
 55    context after all expectations are added.
 56
 57    This implementation does not validate whether the expectation types exist
 58    in Great Expectations or whether the provided kwargs are valid for the
 59    expectation type. Such validation is delegated to Great Expectations itself
 60    when the expectations are added to the suite.
 61
 62    Examples
 63    --------
 64    Basic usage with column value expectations:
 65
 66    >>> from adc_toolkit.data.validators.gx.batch_managers.batch_manager import BatchManager
 67    >>> adder = ConfigurationBasedExpectationAddition()
 68    >>> expectations = [
 69    ...     {
 70    ...         "expect_column_values_to_be_in_set": {
 71    ...             "column": "status",
 72    ...             "value_set": ["active", "inactive", "pending"],
 73    ...         }
 74    ...     },
 75    ...     {
 76    ...         "expect_column_values_to_not_be_null": {
 77    ...             "column": "user_id",
 78    ...         }
 79    ...     },
 80    ... ]
 81    >>> adder.add_expectations(batch_manager, expectations)
 82
 83    Configuration with multiple expectation types:
 84
 85    >>> expectations = [
 86    ...     {
 87    ...         "expect_table_row_count_to_be_between": {
 88    ...             "min_value": 100,
 89    ...             "max_value": 10000,
 90    ...         }
 91    ...     },
 92    ...     {
 93    ...         "expect_column_mean_to_be_between": {
 94    ...             "column": "price",
 95    ...             "min_value": 0.0,
 96    ...             "max_value": 1000.0,
 97    ...         }
 98    ...     },
 99    ...     {
100    ...         "expect_column_unique_value_count_to_be_between": {
101    ...             "column": "product_id",
102    ...             "min_value": 50,
103    ...             "max_value": 500,
104    ...         }
105    ...     },
106    ... ]
107    >>> adder.add_expectations(batch_manager, expectations)
108
109    Loading expectations from a configuration file:
110
111    >>> import yaml
112    >>> with open("expectations.yaml") as f:
113    ...     config = yaml.safe_load(f)
114    >>> expectations = config["expectations"]
115    >>> adder = ConfigurationBasedExpectationAddition()
116    >>> adder.add_expectations(batch_manager, expectations)
117    """
118
119    def add_expectations(self, batch_manager: BatchManager, expectations: list[dict]) -> None:
120        """
121        Add expectations to the expectation suite from configuration dictionaries.
122
123        Parse each expectation dictionary to extract the expectation type and
124        parameters, create ExpectationConfiguration objects, and add them to
125        the expectation suite associated with the provided batch manager. The
126        updated suite is persisted to the data context.
127
128        This method processes expectations sequentially, ensuring each is properly
129        validated and added before moving to the next. If any expectation dictionary
130        is malformed, the method will raise an exception before adding subsequent
131        expectations.
132
133        Parameters
134        ----------
135        batch_manager : BatchManager
136            The batch manager containing the data context and batch information.
137            The expectation suite is retrieved using the pattern
138            "{batch_manager.name}_suite" from the batch manager's data context.
139        expectations : list of dict
140            List of expectation dictionaries to add to the suite. Each dictionary
141            must contain exactly one key-value pair where the key is the expectation
142            type (string) and the value is a dictionary of expectation parameters.
143
144            Expected format for each dictionary:
145            {
146                "expectation_type_name": {
147                    "param1": value1,
148                    "param2": value2,
149                    ...
150                }
151            }
152
153        Returns
154        -------
155        None
156            This method modifies the expectation suite in place and persists
157            changes to the data context but does not return a value.
158
159        Raises
160        ------
161        InvalidExpectationDictionaryError
162            If any expectation dictionary does not contain exactly one key-value pair.
163        InvalidExpectationNameTypeError
164            If the expectation type (dictionary key) is not a string.
165        InvalidExpectationKwargsTypeError
166            If the expectation parameters (dictionary value) are not a dictionary.
167
168        See Also
169        --------
170        parse_expectations_dict : Function that validates and extracts expectation
171            components from dictionary format.
172        ExpectationConfiguration : Great Expectations class representing a single
173            expectation with its type and parameters.
174
175        Notes
176        -----
177        The method performs the following steps for each expectation:
178
179        1. Parse the expectation dictionary using `parse_expectations_dict`
180        2. Create an `ExpectationConfiguration` object with the extracted type and kwargs
181        3. Add the configuration to the expectation suite
182        4. Update the suite in the data context after all expectations are added
183
184        The expectation suite must exist in the data context before calling this
185        method. The suite is identified by the naming convention "{batch_manager.name}_suite".
186
187        All expectations are added to the same suite in a single batch. If you need
188        to add expectations to multiple suites, call this method separately for each
189        batch manager.
190
191        This method does not validate the semantic correctness of expectations
192        (e.g., whether column names exist or parameter values are appropriate).
193        Such validation occurs when Great Expectations evaluates the expectations
194        against actual data.
195
196        Examples
197        --------
198        Add column validation expectations:
199
200        >>> adder = ConfigurationBasedExpectationAddition()
201        >>> expectations = [
202        ...     {
203        ...         "expect_column_values_to_be_in_set": {
204        ...             "column": "status",
205        ...             "value_set": ["active", "inactive"],
206        ...         }
207        ...     },
208        ...     {
209        ...         "expect_column_values_to_not_be_null": {
210        ...             "column": "user_id",
211        ...         }
212        ...     },
213        ... ]
214        >>> adder.add_expectations(batch_manager, expectations)
215
216        Add table-level and column-level expectations:
217
218        >>> expectations = [
219        ...     {
220        ...         "expect_table_row_count_to_be_between": {
221        ...             "min_value": 1000,
222        ...             "max_value": 100000,
223        ...         }
224        ...     },
225        ...     {
226        ...         "expect_column_mean_to_be_between": {
227        ...             "column": "temperature",
228        ...             "min_value": -50.0,
229        ...             "max_value": 150.0,
230        ...         }
231        ...     },
232        ... ]
233        >>> adder.add_expectations(batch_manager, expectations)
234
235        Add expectations with complex parameter types:
236
237        >>> expectations = [
238        ...     {
239        ...         "expect_column_values_to_match_regex": {
240        ...             "column": "email",
241        ...             "regex": r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$",
242        ...         }
243        ...     },
244        ...     {
245        ...         "expect_column_pair_values_to_be_equal": {
246        ...             "column_A": "expected_total",
247        ...             "column_B": "actual_total",
248        ...         }
249        ...     },
250        ... ]
251        >>> adder.add_expectations(batch_manager, expectations)
252        """
253        suite = batch_manager.data_context.get_expectation_suite(expectation_suite_name=f"{batch_manager.name}_suite")
254        for expectation in expectations:
255            expectation_type, expectation_kwargs = parse_expectations_dict(expectation_dictionary=expectation)
256            expectation_configuration = ExpectationConfiguration(
257                expectation_type=expectation_type,
258                kwargs=expectation_kwargs,
259            )
260            suite.add_expectation(expectation_configuration)
261        batch_manager.data_context.update_expectation_suite(suite)

 84class GXValidator:
 85    """
 86    Great Expectations validator implementing the DataValidator protocol.
 87
 88    This validator provides comprehensive data quality validation using Great
 89    Expectations (GX). It orchestrates the complete validation workflow including
 90    data context management, expectation suite lookup, expectation creation,
 91    batch management, checkpoint execution, and validation result evaluation.
 92
 93    The validator operates in two modes depending on configuration:
 94
 95    1. **Auto-creation mode** (default): Automatically creates expectation suites
 96       and freezes schemas on first validation. Subsequent validations enforce
 97       the captured schema.
 98
 99    2. **Custom mode**: Uses pre-defined expectation suites and custom strategies
100       for suite lookup and expectation addition, enabling manual control over
101       validation rules.
102
103    Schema freezing captures the structure of a DataFrame (column names and types)
104    and creates expectations that enforce this schema in future validations. This
105    provides automatic protection against schema drift while allowing manual
106    expectation customization when needed.
107
108    Parameters
109    ----------
110    data_context : great_expectations.data_context.AbstractDataContext
111        Great Expectations data context managing expectation suites, checkpoints,
112        validation results, and data source configurations. This can be a
113        filesystem-based context, cloud-based context (S3, GCS, Azure), or
114        ephemeral in-memory context for testing.
115    expectation_suite_lookup_strategy : ExpectationSuiteLookupStrategy or None, optional
116        Strategy for handling expectation suite lookup when validating datasets.
117        Controls behavior when a suite does not exist for a dataset:
118
119        - AutoExpectationSuiteCreation (default): Automatically creates missing
120          suites, enabling zero-configuration validation.
121        - CustomExpectationSuiteStrategy: Raises ExpectationSuiteNotFoundError
122          for missing suites, enforcing explicit suite definitions.
123
124        Default is None, which uses AutoExpectationSuiteCreation.
125    expectation_addition_strategy : ExpectationAdditionStrategy or None, optional
126        Strategy for adding expectations to expectation suites during validation.
127        Controls how expectations are populated when a suite is created or updated:
128
129        - SchemaExpectationAddition (default): Automatically adds schema
130          expectations by inspecting DataFrame structure, freezing the schema.
131        - SkipExpectationAddition: Skips automatic expectation addition,
132          requiring manual expectation definition.
133
134        Default is None, which uses SchemaExpectationAddition.
135
136    Attributes
137    ----------
138    data_context : great_expectations.data_context.AbstractDataContext
139        The Great Expectations data context instance used for all validation
140        operations.
141    expectation_suite_lookup_strategy : ExpectationSuiteLookupStrategy
142        The strategy used for expectation suite lookup.
143    expectation_addition_strategy : ExpectationAdditionStrategy
144        The strategy used for adding expectations to suites.
145
146    See Also
147    --------
148    adc_toolkit.data.abs.DataValidator : Protocol defining validator interface.
149    adc_toolkit.data.validators.gx.data_context.RepoDataContext : Filesystem-based data context.
150    adc_toolkit.data.validators.gx.data_context.S3DataContext : AWS S3-based data context.
151    adc_toolkit.data.validators.gx.data_context.GCPDataContext : Google Cloud Storage context.
152    adc_toolkit.data.validators.gx.data_context.AzureDataContext : Azure Blob Storage context.
153    adc_toolkit.data.validators.gx.batch_managers.AutoExpectationSuiteCreation : Auto-create suites.
154    adc_toolkit.data.validators.gx.batch_managers.CustomExpectationSuiteStrategy : Require suites.
155    adc_toolkit.data.validators.gx.batch_managers.SchemaExpectationAddition : Auto-add schema expectations.
156    adc_toolkit.data.validators.gx.batch_managers.SkipExpectationAddition : Skip expectation addition.
157
158    Notes
159    -----
160    **Design Patterns:**
161
162    The GXValidator implements several design patterns:
163
164    - **Strategy Pattern**: Pluggable strategies for suite lookup and expectation
165      addition enable flexible validation workflows without modifying core logic.
166    - **Facade Pattern**: Simplifies Great Expectations' complex API by providing
167      a clean, high-level interface for validation.
168    - **Dependency Injection**: Data context and strategies are injected,
169      enabling testability and configuration flexibility.
170
171    **Validation Workflow:**
172
173    When validate() is called, the following sequence occurs:
174
175    1. Look up or create expectation suite for the dataset
176    2. Create a batch from the data using BatchManager
177    3. Add expectations to the suite using the configured strategy
178    4. Create or update a checkpoint for the dataset
179    5. Execute the checkpoint to validate data against expectations
180    6. Evaluate validation results and raise ValidationError on failure
181    7. Return the original data if validation succeeds
182
183    **Performance Considerations:**
184
185    - First validation of a dataset (suite creation) is slower than subsequent
186      validations due to suite initialization overhead.
187    - Schema freezing requires full DataFrame inspection, which scales with
188      the number of columns (not rows).
189    - Validation performance depends on the number and complexity of expectations.
190    - Consider using sampling for large datasets with expensive expectations.
191
192    **Thread Safety:**
193
194    GXValidator instances are not thread-safe. The underlying Great Expectations
195    data context may perform file I/O and maintain internal state. For concurrent
196    validation, create separate validator instances per thread.
197
198    **Cloud Storage:**
199
200    When using cloud-based data contexts (S3, GCS, Azure), ensure appropriate
201    credentials and permissions are configured. The data context stores expectation
202    suites, checkpoints, and validation results in cloud storage.
203
204    Examples
205    --------
206    Create a validator with default auto-creation behavior:
207
208    >>> from adc_toolkit.data.validators.gx import GXValidator
209    >>> from great_expectations.data_context import EphemeralDataContext
210    >>> import pandas as pd
211    >>> context = EphemeralDataContext()
212    >>> validator = GXValidator(data_context=context)
213    >>> df = pd.DataFrame({"id": [1, 2, 3], "value": [10, 20, 30]})
214    >>> validated = validator.validate("dataset_name", df)
215
216    Create a validator requiring pre-defined expectation suites:
217
218    >>> from adc_toolkit.data.validators.gx.batch_managers import (
219    ...     CustomExpectationSuiteStrategy,
220    ...     SkipExpectationAddition,
221    ... )
222    >>> validator = GXValidator(
223    ...     data_context=context,
224    ...     expectation_suite_lookup_strategy=CustomExpectationSuiteStrategy(),
225    ...     expectation_addition_strategy=SkipExpectationAddition(),
226    ... )
227    >>> # This will raise ExpectationSuiteNotFoundError if suite doesn't exist
228    >>> validated = validator.validate("strict_dataset", df)
229
230    Use with a filesystem-based data context:
231
232    >>> validator = GXValidator.in_directory("/path/to/gx/config")
233    >>> df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]})
234    >>> validated = validator.validate("my_data", df)
235
236    Validate with automatic schema freezing:
237
238    >>> df_first = pd.DataFrame({"id": [1, 2], "name": ["Alice", "Bob"]})
239    >>> validator.validate("users", df_first)  # Creates suite, freezes schema
240    >>> df_second = pd.DataFrame({"id": [3, 4], "name": ["Charlie", "Dave"]})
241    >>> validator.validate("users", df_second)  # Validates against frozen schema
242    >>> df_invalid = pd.DataFrame({"id": [5], "age": [30]})  # Different schema
243    >>> validator.validate("users", df_invalid)  # Raises ValidationError
244
245    Integration with ValidatedDataCatalog:
246
247    >>> from adc_toolkit.data.catalog import ValidatedDataCatalog
248    >>> catalog = ValidatedDataCatalog.in_directory(
249    ...     catalog_dir="config/catalog", validator=GXValidator.in_directory("config/gx")
250    ... )
251    >>> # Load with automatic validation
252    >>> df = catalog.load("customer_data")
253    >>> # Process data
254    >>> processed = transform(df)
255    >>> # Save with automatic validation
256    >>> catalog.save("processed_customer_data", processed)
257    """
258
259    __slots__ = [
260        "data_context",
261        "expectation_addition_strategy",
262        "expectation_suite_lookup_strategy",
263    ]
264
265    def __init__(
266        self,
267        data_context: AbstractDataContext,
268        expectation_suite_lookup_strategy: ExpectationSuiteLookupStrategy | None = None,
269        expectation_addition_strategy: ExpectationAdditionStrategy | None = None,
270    ) -> None:
271        """
272        Initialize a Great Expectations validator with specified configuration.
273
274        Creates a new GXValidator instance with the provided data context and
275        validation strategies. The data context manages expectation suites,
276        checkpoints, and validation results. The strategies control how the
277        validator handles missing expectation suites and how it populates
278        suites with expectations.
279
280        Parameters
281        ----------
282        data_context : great_expectations.data_context.AbstractDataContext
283            Great Expectations data context to use for all validation operations.
284            This context manages the storage and retrieval of expectation suites,
285            checkpoints, and validation results. Can be:
286
287            - RepoDataContext: Filesystem-based context stored in a directory
288            - S3DataContext: AWS S3-backed context for cloud deployments
289            - GCPDataContext: Google Cloud Storage-backed context
290            - AzureDataContext: Azure Blob Storage-backed context
291            - EphemeralDataContext: In-memory context for testing
292
293        expectation_suite_lookup_strategy : ExpectationSuiteLookupStrategy or None, optional
294            Strategy for handling expectation suite lookup operations. Controls
295            the behavior when an expectation suite is not found for a dataset:
296
297            - None (default): Uses AutoExpectationSuiteCreation, which
298              automatically creates missing suites with zero configuration.
299            - AutoExpectationSuiteCreation(): Explicitly auto-creates suites.
300            - CustomExpectationSuiteStrategy(): Raises an error for missing
301              suites, enforcing that all suites must be pre-defined.
302
303            Default is None, which is equivalent to AutoExpectationSuiteCreation().
304
305        expectation_addition_strategy : ExpectationAdditionStrategy or None, optional
306            Strategy for adding expectations to expectation suites. Controls
307            how expectations are populated when validating data:
308
309            - None (default): Uses SchemaExpectationAddition, which inspects
310              DataFrame structure and adds schema validation expectations.
311            - SchemaExpectationAddition(): Explicitly adds schema expectations
312              by freezing the DataFrame's column names and types.
313            - SkipExpectationAddition(): Skips automatic expectation addition,
314              requiring all expectations to be manually defined.
315
316            Default is None, which is equivalent to SchemaExpectationAddition().
317
318        Returns
319        -------
320        None
321
322        See Also
323        --------
324        in_directory : Factory method to create validator from configuration directory.
325        validate : Validate data using this validator.
326        adc_toolkit.data.validators.gx.data_context.RepoDataContext : Create filesystem context.
327        adc_toolkit.data.validators.gx.batch_managers.AutoExpectationSuiteCreation : Auto-create strategy.
328        adc_toolkit.data.validators.gx.batch_managers.SchemaExpectationAddition : Schema freeze strategy.
329
330        Notes
331        -----
332        The constructor performs minimal initialization, only storing the provided
333        parameters. No I/O operations, file system access, or data context
334        initialization occurs during construction. This enables fast instantiation
335        and lazy initialization patterns.
336
337        **Default Strategies:**
338
339        When strategy parameters are None, the validator uses sensible defaults:
340
341        - AutoExpectationSuiteCreation: Enables zero-configuration validation
342          by automatically creating expectation suites on first use.
343        - SchemaExpectationAddition: Provides automatic schema drift protection
344          by freezing the DataFrame structure on first validation.
345
346        These defaults are ideal for development, exploration, and rapid
347        prototyping. For production deployments with explicit validation rules,
348        consider using CustomExpectationSuiteStrategy and pre-defined suites.
349
350        **Strategy Immutability:**
351
352        Once a validator is instantiated, its strategies cannot be changed.
353        To use different strategies, create a new validator instance. This
354        design ensures consistent validation behavior throughout a validator's
355        lifetime.
356
357        **Data Context Lifecycle:**
358
359        The validator does not own the data context lifecycle. The caller is
360        responsible for creating and properly disposing of the data context.
361        For ephemeral contexts used in testing, ensure proper cleanup:
362
363        >>> context = EphemeralDataContext()
364        >>> try:
365        ...     validator = GXValidator(data_context=context)
366        ...     # Use validator
367        ... finally:
368        ...     # Clean up context if needed
369        ...     pass
370
371        Examples
372        --------
373        Create a validator with default auto-creation strategies:
374
375        >>> from great_expectations.data_context import EphemeralDataContext
376        >>> context = EphemeralDataContext()
377        >>> validator = GXValidator(data_context=context)
378        >>> # Automatically creates suites and freezes schemas
379
380        Create a validator with strict, manual suite management:
381
382        >>> from adc_toolkit.data.validators.gx.batch_managers import (
383        ...     CustomExpectationSuiteStrategy,
384        ...     SkipExpectationAddition,
385        ... )
386        >>> validator = GXValidator(
387        ...     data_context=context,
388        ...     expectation_suite_lookup_strategy=CustomExpectationSuiteStrategy(),
389        ...     expectation_addition_strategy=SkipExpectationAddition(),
390        ... )
391        >>> # Requires pre-defined suites, no automatic expectations
392
393        Create a validator with auto-creation but manual expectations:
394
395        >>> validator = GXValidator(
396        ...     data_context=context,
397        ...     expectation_suite_lookup_strategy=AutoExpectationSuiteCreation(),
398        ...     expectation_addition_strategy=SkipExpectationAddition(),
399        ... )
400        >>> # Creates suites automatically but expects manual expectation definition
401
402        Use with a filesystem-based data context:
403
404        >>> from adc_toolkit.data.validators.gx.data_context import RepoDataContext
405        >>> context = RepoDataContext("/path/to/gx").create()
406        >>> validator = GXValidator(data_context=context)
407
408        Use with a cloud-based data context:
409
410        >>> from adc_toolkit.data.validators.gx.data_context import S3DataContext
411        >>> context = S3DataContext("s3://my-bucket/gx-config").create()
412        >>> validator = GXValidator(data_context=context)
413        """
414        self.data_context = data_context
415        self.expectation_suite_lookup_strategy = expectation_suite_lookup_strategy or AutoExpectationSuiteCreation()
416        self.expectation_addition_strategy = expectation_addition_strategy or SchemaExpectationAddition()
417
418    @classmethod
419    def in_directory(cls, path: str | Path) -> "GXValidator":
420        """
421        Create a GXValidator with a filesystem-based Great Expectations data context.
422
423        This factory method provides a convenient way to create a validator using
424        a repository-based (filesystem) data context. It initializes a RepoDataContext
425        from the specified directory and creates a validator with default strategies
426        for auto-creation and schema freezing.
427
428        The specified directory should contain a Great Expectations project structure
429        with configuration files, expectation suites, checkpoints, and validation
430        results. If the directory does not contain a valid GX project, the
431        RepoDataContext will initialize a new project structure.
432
433        Parameters
434        ----------
435        path : str or pathlib.Path
436            Path to the directory containing Great Expectations configuration.
437            This directory should have (or will be initialized with) the
438            following structure:
439
440            - great_expectations.yml : Main configuration file
441            - expectations/ : Directory containing expectation suite JSON files
442            - checkpoints/ : Directory containing checkpoint YAML files
443            - uncommitted/ : Directory for validation results and data docs
444            - plugins/ : Optional directory for custom expectations
445
446            If the directory does not exist or is empty, a new GX project
447            structure will be created. Both absolute and relative paths are
448            supported.
449
450        Returns
451        -------
452        GXValidator
453            A new GXValidator instance configured with:
454
455            - RepoDataContext based on the specified directory
456            - AutoExpectationSuiteCreation strategy (creates suites automatically)
457            - SchemaExpectationAddition strategy (freezes schemas automatically)
458
459        Raises
460        ------
461        FileNotFoundError
462            If the parent directory of the specified path does not exist
463            and cannot be created.
464        PermissionError
465            If the process lacks permissions to read from or write to the
466            specified directory.
467        ValueError
468            If the directory contains invalid Great Expectations configuration
469            files that cannot be parsed.
470
471        See Also
472        --------
473        __init__ : Constructor for custom data context and strategy configuration.
474        validate : Validate data using this validator.
475        adc_toolkit.data.validators.gx.data_context.RepoDataContext : Filesystem context implementation.
476        adc_toolkit.data.validators.gx.data_context.S3DataContext : AWS S3-based context.
477        adc_toolkit.data.validators.gx.data_context.GCPDataContext : Google Cloud context.
478
479        Notes
480        -----
481        **Repository Structure:**
482
483        Great Expectations uses a specific directory structure to organize
484        validation artifacts:
485
486        - Expectation suites are stored as JSON in expectations/
487        - Checkpoints are stored as YAML in checkpoints/
488        - Validation results go in uncommitted/validations/
489        - Data docs are generated in uncommitted/data_docs/
490
491        This structure enables version control of validation rules while keeping
492        validation results and documentation out of version control.
493
494        **Version Control:**
495
496        When using filesystem-based contexts, consider the following for version
497        control (Git):
498
499        - Commit: expectations/, checkpoints/, great_expectations.yml, plugins/
500        - Ignore: uncommitted/ (contains validation results and generated docs)
501
502        This approach version controls validation rules while excluding
503        environment-specific results.
504
505        **Performance:**
506
507        The in_directory method performs I/O operations to read configuration
508        and initialize the data context. For applications creating many validator
509        instances, consider caching the data context and passing it to __init__
510        instead of using in_directory repeatedly.
511
512        **Automatic Initialization:**
513
514        If the specified directory does not contain a great_expectations.yml file,
515        RepoDataContext will initialize a new GX project. This is useful for
516        quickly starting validation without manual GX project setup, but may not
517        be suitable for production deployments where explicit configuration is
518        preferred.
519
520        **Default Strategies:**
521
522        This factory method always uses default strategies (AutoExpectationSuiteCreation
523        and SchemaExpectationAddition). For custom strategies, use the __init__
524        constructor directly:
525
526        >>> from adc_toolkit.data.validators.gx.data_context import RepoDataContext
527        >>> context = RepoDataContext(path).create()
528        >>> validator = GXValidator(data_context=context, expectation_suite_lookup_strategy=CustomStrategy())
529
530        Examples
531        --------
532        Create a validator from a GX project directory:
533
534        >>> validator = GXValidator.in_directory("/path/to/gx_project")
535        >>> import pandas as pd
536        >>> df = pd.DataFrame({"id": [1, 2, 3], "value": [10, 20, 30]})
537        >>> validated = validator.validate("my_dataset", df)
538
539        Use with a relative path:
540
541        >>> validator = GXValidator.in_directory("config/validations")
542        >>> validated = validator.validate("dataset", data)
543
544        Use with pathlib.Path:
545
546        >>> from pathlib import Path
547        >>> config_path = Path("config") / "gx"
548        >>> validator = GXValidator.in_directory(config_path)
549
550        Initialize a new GX project and validator:
551
552        >>> # Directory doesn't exist yet
553        >>> validator = GXValidator.in_directory("./new_gx_project")
554        >>> # Now directory contains initialized GX project structure
555
556        Validate multiple datasets with one validator:
557
558        >>> validator = GXValidator.in_directory("config/gx")
559        >>> df1 = pd.DataFrame({"a": [1, 2]})
560        >>> df2 = pd.DataFrame({"b": ["x", "y"]})
561        >>> validated1 = validator.validate("dataset1", df1)
562        >>> validated2 = validator.validate("dataset2", df2)
563
564        Integration in a data pipeline:
565
566        >>> def validate_pipeline_data(data_path: str, gx_path: str) -> None:
567        ...     validator = GXValidator.in_directory(gx_path)
568        ...     for dataset_name in ["raw", "cleaned", "features"]:
569        ...         df = pd.read_csv(f"{data_path}/{dataset_name}.csv")
570        ...         validated = validator.validate(dataset_name, df)
571        ...         print(f"Validated {dataset_name}: {len(validated)} rows")
572        """
573        return cls(data_context=RepoDataContext(path).create())
574
575    def validate(self, name: str, data: Data) -> Data:
576        """
577        Validate data against Great Expectations rules for the named dataset.
578
579        Executes the complete Great Expectations validation workflow for the
580        specified dataset. This includes expectation suite lookup or creation,
581        batch request generation, expectation addition, checkpoint creation and
582        execution, and validation result evaluation.
583
584        The validation process ensures data quality by verifying that the data
585        meets all expectations defined in the associated expectation suite. If
586        validation fails, detailed error information identifies which expectations
587        failed and why.
588
589        On successful validation, the original data is returned unchanged. The
590        validation is side-effect free from the data perspective, but may create
591        or update expectation suites, checkpoints, and validation results in the
592        data context storage.
593
594        Parameters
595        ----------
596        name : str
597            Identifier for the dataset being validated. This name is used to:
598
599            - Look up the corresponding expectation suite (named "{name}_suite")
600            - Create or update the checkpoint for this dataset
601            - Store validation results associated with this dataset
602
603            The name should be consistent across validation calls for the same
604            logical dataset to ensure proper suite reuse and result tracking.
605            Use descriptive, stable names like "customer_data", "sales_features",
606            or "model_predictions".
607
608        data : Data
609            The dataset to validate. Must be a Data protocol-compatible object,
610            typically a pandas DataFrame or Spark DataFrame. The data should
611            have `columns` and `dtypes` properties for schema inspection.
612
613            The data is not modified by validation. If validation succeeds,
614            the same object (or an equivalent copy) is returned.
615
616        Returns
617        -------
618        Data
619            The validated data. This is the same object as the input `data`
620            parameter if validation succeeds. The return type matches the input
621            type (e.g., pandas.DataFrame returns pandas.DataFrame).
622
623            Returning the data enables method chaining and integration with
624            pipelines:
625
626            >>> validated = validator.validate("data", raw_data)
627            >>> processed = transform(validated)
628
629        Raises
630        ------
631        ValidationError
632            If the data fails validation against the expectation suite. The
633            exception contains detailed information about:
634
635            - Which expectations failed
636            - Observed values that violated expectations
637            - Expected values or constraints
638            - Summary statistics for failed validations
639
640            This exception indicates data quality issues that must be addressed
641            before proceeding with downstream processing.
642
643        ExpectationSuiteNotFoundError
644            If the expectation suite for the dataset does not exist and the
645            validator is configured with CustomExpectationSuiteStrategy. This
646            indicates that validation rules must be explicitly defined before
647            validation can proceed.
648
649            To resolve, either:
650            - Create the expectation suite manually in the data context
651            - Switch to AutoExpectationSuiteCreation strategy
652            - Ensure the correct data context is being used
653
654        TypeError
655            If the data type is incompatible with Great Expectations batch
656            creation. For example, if the data does not have the required
657            `columns` and `dtypes` attributes.
658
659        KeyError
660            If the batch manager cannot create a batch from the data due to
661            missing required attributes or metadata.
662
663        See Also
664        --------
665        __init__ : Constructor for configuring validation strategies.
666        in_directory : Factory method for filesystem-based validators.
667        adc_toolkit.data.validators.gx.batch_managers.validate_dataset : Underlying validation function.
668        adc_toolkit.data.abs.DataValidator.validate : Protocol method specification.
669
670        Notes
671        -----
672        **Validation Workflow:**
673
674        The validate method orchestrates these steps:
675
676        1. **Suite Lookup**: Check if an expectation suite exists for the dataset.
677           If not, behavior depends on the lookup strategy:
678
679           - AutoExpectationSuiteCreation: Create a new suite
680           - CustomExpectationSuiteStrategy: Raise ExpectationSuiteNotFoundError
681
682        2. **Batch Creation**: Convert the data into a GX Batch object using
683           BatchManager, making it compatible with GX validation operations.
684
685        3. **Expectation Addition**: Add expectations to the suite based on the
686           addition strategy:
687
688           - SchemaExpectationAddition: Inspect data schema and add schema expectations
689           - SkipExpectationAddition: Skip, expecting manual expectation definition
690
691        4. **Checkpoint Execution**: Create or update a checkpoint for the dataset
692           and execute it to validate the batch against the expectation suite.
693
694        5. **Result Evaluation**: Analyze validation results. If all expectations
695           pass, return the data. If any fail, raise ValidationError with details.
696
697        **First Validation vs. Subsequent Validations:**
698
699        The first time a dataset is validated (with AutoExpectationSuiteCreation
700        and SchemaExpectationAddition), the validator:
701
702        - Creates an expectation suite named "{name}_suite"
703        - Inspects the DataFrame schema (column names and types)
704        - Adds schema expectations that "freeze" this structure
705        - Creates a checkpoint for the dataset
706        - Validates the data (which should pass since expectations match the data)
707
708        Subsequent validations of the same dataset:
709
710        - Reuse the existing expectation suite and checkpoint
711        - Validate data against the frozen schema and any other expectations
712        - Detect schema drift or data quality issues
713
714        **Performance:**
715
716        Validation performance depends on several factors:
717
718        - Number of expectations in the suite
719        - Complexity of expectations (simple schema checks vs. statistical tests)
720        - Size of the dataset (some expectations scan all data)
721        - Data context backend (filesystem vs. cloud storage)
722
723        First validation is slower due to suite and checkpoint creation overhead.
724        Subsequent validations are faster, typically scaling with the number of
725        expectations rather than data size.
726
727        For large datasets with expensive expectations, consider:
728        - Sampling strategies to validate subsets
729        - Caching validation results
730        - Running validations asynchronously
731        - Using incremental validation for streaming data
732
733        **Idempotency:**
734
735        Validation is idempotent: validating the same data multiple times with
736        the same name produces the same result (pass or fail). However, validation
737        results are stored with timestamps, so each validation creates new result
738        artifacts in the data context.
739
740        **Thread Safety:**
741
742        The validate method is not thread-safe. Multiple threads validating
743        different datasets concurrently may encounter race conditions when
744        accessing the data context. For concurrent validation, create separate
745        validator instances (with separate data contexts) per thread.
746
747        **Side Effects:**
748
749        While validation does not modify the data, it may have side effects:
750
751        - Create or update expectation suites in the data context
752        - Create or update checkpoints in the data context
753        - Write validation results to storage (filesystem or cloud)
754        - Generate data documentation if configured
755
756        These artifacts are stored according to the data context configuration.
757
758        Examples
759        --------
760        Basic validation with automatic suite creation:
761
762        >>> import pandas as pd
763        >>> from adc_toolkit.data.validators.gx import GXValidator
764        >>> validator = GXValidator.in_directory("config/gx")
765        >>> df = pd.DataFrame({"id": [1, 2, 3], "value": [10, 20, 30]})
766        >>> validated = validator.validate("sales_data", df)
767        >>> # First validation creates suite and freezes schema
768        >>> validated.shape
769        (3, 2)
770
771        Subsequent validation detects schema drift:
772
773        >>> df_valid = pd.DataFrame({"id": [4, 5], "value": [40, 50]})
774        >>> validator.validate("sales_data", df_valid)  # Passes, schema matches
775        >>> df_invalid = pd.DataFrame({"id": [6], "price": [100]})
776        >>> validator.validate("sales_data", df_invalid)  # Raises ValidationError
777
778        Handle validation failures gracefully:
779
780        >>> try:
781        ...     validated = validator.validate("strict_data", df)
782        ... except ValidationError as e:
783        ...     print(f"Validation failed: {e}")
784        ...     # Log error, send alert, reject data, etc.
785        ...     raise
786
787        Validate multiple datasets in a pipeline:
788
789        >>> def etl_pipeline(validator: GXValidator) -> None:
790        ...     raw = load_raw_data()
791        ...     validated_raw = validator.validate("raw_data", raw)
792        ...     cleaned = clean(validated_raw)
793        ...     validated_clean = validator.validate("cleaned_data", cleaned)
794        ...     features = engineer_features(validated_clean)
795        ...     validated_features = validator.validate("features", features)
796        ...     save(validated_features)
797
798        Use validation in data loading:
799
800        >>> class ValidatedDataLoader:
801        ...     def __init__(self, validator: GXValidator):
802        ...         self.validator = validator
803        ...
804        ...     def load(self, name: str, path: str) -> pd.DataFrame:
805        ...         df = pd.read_csv(path)
806        ...         return self.validator.validate(name, df)
807
808        Integration with ValidatedDataCatalog:
809
810        >>> from adc_toolkit.data.catalog import ValidatedDataCatalog
811        >>> catalog = ValidatedDataCatalog.in_directory(
812        ...     "config/catalog", validator=GXValidator.in_directory("config/gx")
813        ... )
814        >>> # Validation happens automatically on load
815        >>> df = catalog.load("customer_data")  # Validates after loading
816        >>> processed = transform(df)
817        >>> catalog.save("processed_data", processed)  # Validates before saving
818
819        Validate with custom expectation suite:
820
821        >>> # Pre-create suite with custom expectations
822        >>> suite = context.create_expectation_suite("custom_data_suite")
823        >>> suite.add_expectation(
824        ...     ExpectationConfiguration(
825        ...         expectation_type="expect_column_values_to_be_in_range",
826        ...         kwargs={"column": "age", "min_value": 0, "max_value": 120},
827        ...     )
828        ... )
829        >>> # Now validate using the custom suite
830        >>> df = pd.DataFrame({"age": [25, 30, 35]})
831        >>> validator.validate("custom_data", df)  # Uses custom_data_suite
832        """
833        return validate_dataset(
834            name,
835            data,
836            self.data_context,
837            self.expectation_suite_lookup_strategy,
838            self.expectation_addition_strategy,
839        )