vcnet.data

VCNet data module. This module provides the classes to manage the data for VCNet.

class vcnet.data.PostHocRounder(precisions)[source]

Bases: object

A class dedicated to rounding values to a given number of decimals. This is a post-hoc rounder, as it rounds the numerical values generated by VCNet to make them more realistic.

The class implements only an inverse_transform, as it applies the transformation to the generated counterfactuals.

Parameters:

precisions (Dict(str,int)) – map giving the precision (number of decimals) to apply to each attribute

inverse_transform(df)[source]

Applies the inverse transformation to the dataframe df.

vcnet.data.fit_rounder(df)[source]

Builds a rounder from a dataframe.

Parameters:

df (pd.DataFrame) – DataFrame containing only continuous data.

Return type:

PostHocRounder

vcnet.data.fit_scaler(scaling_method, df)[source]

Fits a scaler on a dataframe.

Parameters:

  • scaling_method ({"MinMax", "Standard", "Identity"}) – String indicating which scaling method to use, or an sklearn.preprocessing scaler.

  • df (pd.DataFrame) – DataFrame containing only continuous data.

Return type:

sklearn.base.BaseEstimator

vcnet.data.fit_imputer(imputation_method, df)[source]

Fits an imputer of missing values on a dataframe.

Parameters:

  • imputation_method ({"SimpleImputer", "Identity"}) – String indicating which imputation method to use, or an sklearn.impute imputer.

  • df (pd.DataFrame) – DataFrame containing only continuous data.

Return type:

sklearn.base.BaseEstimator

vcnet.data.fit_encoder(encoding_method, df)[source]

Fits an encoder of categorical attributes on a dataframe.

Parameters:

  • encoding_method ({"OneHot", "OneHot_drop_binary", "Identity"}) – String indicating which encoding method to use, or an sklearn.preprocessing encoder.

  • df (pd.DataFrame) – DataFrame containing only categorical data.

Return type:

sklearn.base.BaseEstimator

vcnet.data.scale(fitted_scaler: sklearn.base.BaseEstimator, features: List[str], df: pandas.DataFrame) → pandas.DataFrame[source]

Pipeline function to normalize data with a fitted sklearn scaler.

Parameters:

  • fitted_scaler (sklearn Scaler) – Scaler used to normalize the input data

  • features (list) – List of continuous features

  • df (pd.DataFrame) – Data to normalize

Returns:

output – Whole DataFrame with normalized values

Return type:

pd.DataFrame

vcnet.data.descale(fitted_scaler: sklearn.base.BaseEstimator, features: List[str], df: pandas.DataFrame) → pandas.DataFrame[source]

Pipeline function to denormalize data with a fitted sklearn scaler.

Parameters:

  • fitted_scaler (sklearn Scaler) – Scaler used to normalize the input data

  • features (list) – List of continuous features

  • df (pd.DataFrame) – Data to denormalize

Returns:

output – Whole DataFrame with denormalized values

Return type:

pd.DataFrame

vcnet.data.encode(fitted_encoder: sklearn.base.BaseEstimator, features: List[str], df: pandas.DataFrame) → pandas.DataFrame[source]

Pipeline function to encode data with a fitted sklearn OneHotEncoder.

Parameters:

  • fitted_encoder (sklearn OneHotEncoder) – Encoder of the input data.

  • features (list) – List of categorical features.

  • df (pd.DataFrame) – Data to encode

Returns:

output – Whole DataFrame with encoded values

Return type:

pd.DataFrame

vcnet.data.decode(fitted_encoder: sklearn.base.BaseEstimator, features: List[str], df: pandas.DataFrame) → pandas.DataFrame[source]

Pipeline function to decode data with a fitted sklearn OneHotEncoder.

Parameters:

  • fitted_encoder (sklearn OneHotEncoder) – Encoder of the input data.

  • features (list) – List of categorical features.

  • df (pd.DataFrame) – Data to decode

Returns:

output – Whole DataFrame with decoded values

Return type:

pd.DataFrame

vcnet.data.impute(fitted_imputer: sklearn.base.BaseEstimator, features: List[str], df: pandas.DataFrame) → pandas.DataFrame[source]

Pipeline function to impute missing values in the dataset with a fitted sklearn imputer. In the preparation pipeline of DataCatalog, it is applied after the scaling.

Parameters:

  • fitted_imputer (sklearn Imputer) – Imputer of missing values.

  • features (list) – List of numerical features.

  • df (pd.DataFrame) – Data to modify

Returns:

output – Whole DataFrame without missing values (in the selected features)

Return type:

pd.DataFrame

vcnet.data.attribute_round(fitted_rounder: PostHocRounder, features: List[str], df: pandas.DataFrame) → pandas.DataFrame[source]

Pipeline function to round the numerical attributes of the data.

Parameters:

  • fitted_rounder (PostHocRounder) – Rounds the attributes of the data to a fitted level of precision

  • features (list) – List of continuous features

  • df (pd.DataFrame) – Data to round

Returns:

output – Whole DataFrame with rounded values

Return type:

pd.DataFrame

vcnet.data.order_data(feature_order: List[str], df: pandas.DataFrame) → pandas.DataFrame[source]

Restores the correct order of the input features for the ML model.

Only works for encoded data.

Parameters:

  • feature_order (list) – List of input features in the correct order

  • df (pd.DataFrame) – Data to reorder

Returns:

output – Whole DataFrame with ordered features

Return type:

pd.DataFrame

class vcnet.data.NumpyDataset(*args: Any, **kwargs: Any)[source]

Bases: TensorDataset

Dataset with only numerical attributes.

A numpy dataset is represented by a tensor with attributes in columns. When it is a training dataset, the last column is the numerical target feature.

data_loader(batch_size=128, shuffle=True, num_workers=4)[source]

Builds a torch data loader to be used for training VCNet.

Parameters:

  • batch_size (int, optional) – Size of the batches. Defaults to 128.

  • shuffle (bool, optional) –

    Whether to shuffle the examples before creating batches.

    Defaults to True.

  • num_workers (int, optional) – Number of workers. Defaults to 4.

Returns:

representation of a dataset for mini-batch optimization

Return type:

torch.DataLoader

features(test=False)[source]

Returns the feature part of the tensors.

Parameters:

test (bool, optional) – indicates whether the dataset contains labels (False) or not (True). Defaults to False.

target(test=False)[source]

Returns the labels of the dataset (if they exist, otherwise it returns None).

Parameters:

test (bool, optional) – indicates whether the dataset contains labels (False) or not (True). Defaults to False.

class vcnet.data.DataCatalog(*args: Any, **kwargs: Any)[source]

Bases: LightningDataModule

Generic framework for tabular datasets, using sklearn preprocessing.

The preprocessing pipeline is made of the following steps: scaling of numerical attributes, imputation of missing values and encoding of categorical attributes. The reverse pipeline decodes the categorical attributes, reverses the scaling and can also round the numerical attributes.

Parameters:

config –

Dict Configuration dictionary containing the required and optional settings to prepare the dataset for counterfactual generation. The settings are used to set up an internal pipeline (and its reverse pipeline).

The settings must at least define the following attributes:

  • target (str): name of the target attribute

  • continuous (List[str]): list of continuous attributes of the dataset

  • categorical (List[str]): list of categorical attributes of the dataset

  • immutables (List[str]): list of immutable attributes (among the continuous or categorical attributes)

If the dataset is stored in a file, it can be loaded in the pipeline by setting the filename attribute.

The following optional settings define the train/test sets:

  • test_size/val_size (float): proportions of the dataset dedicated to test and validation

  • stratify (bool): use a stratification strategy to sample the test/train sets

The following optional settings define the preprocessing pipeline:

  • scaling_method (str, default: “MinMax”): type of sklearn scaler. It can be set with the property setter to any sklearn scaler. Set to “Identity” for no scaling.

  • encoding_method (str, default: “OneHot_drop_binary”): type of one-hot encoding (“OneHot” or “OneHot_drop_binary”). “OneHot_drop_binary” drops one of the two columns of binary features. It can be set with the property setter to any sklearn encoder. Set to “Identity” for no encoding.

  • imputation_method (str, default: “Identity”): type of sklearn imputer (“SimpleImputer” or “Identity”). Set to “Identity” for no imputation.

  • activate_rounding (bool, default: False): if True, the values of the continuous attributes of generated counterfactuals are rounded to be more realistic.

Finally, other optional parameters:

  • batch_size (int): default value is 64

Variables:

  • df – pandas.DataFrame The complete DataFrame (restricted to the attributes listed in the settings). This is the combination of df_train, df_val and df_test before preprocessing, although not shuffled.

  • df_train – pandas.DataFrame Training portion of the complete DataFrame.

  • df_test – pandas.DataFrame Test portion of the complete DataFrame.

  • df_val – pandas.DataFrame Validation portion of the complete DataFrame.

Warning

Imputation works only for continuous variables.

Warning

The names of the attributes and of the target are not checked at this stage, but when the dataset is prepared.

Warning

Rounding is applied to all numerical attributes or to none of them. You cannot choose the attributes to which the rounding is applied. Nonetheless, the rounding precision (number of decimals) is automatically inferred from the training data for each attribute (two attributes can have different precisions).

prepare_data(raw_pd: pandas.DataFrame | None = None)[source]

Prepares the data: splits the dataset and fits the preprocessing pipeline.

Parameters:

raw_pd (pd.DataFrame, optional) – a pandas dataframe containing the data to prepare. If None, the data are loaded from the filename setting. Defaults to None.

Returns:

updated settings, ready to be used in a VCNet model. None means that the data preparation failed.

Return type:

Dict or None

train_dataloader() → torch.utils.data.DataLoader[source]
val_dataloader() → torch.utils.data.DataLoader[source]
test_dataloader() → torch.utils.data.DataLoader[source]
data_unloader(X: torch.tensor, y: torch.tensor | numpy.array) → pandas.DataFrame[source]

Recreates a dataframe from the numerical tensors of data and labels used internally by VCNet models.

It applies the inverse transformation to the data structures required internally by VCNet. Depending on the dataset parameters, it reverses the one-hot encoding to recreate categorical features readable by the user; it reverses the scaling of numerical attributes and can also round them; finally, it decodes the class labels from the probabilistic predictions.

Parameters:

  • X (torch.tensor) – examples, as represented internally by VCNet

  • y (torch.tensor or np.array) – class probabilities of the examples

Returns:

dataframe with the same columns as the input dataframe

Return type:

DataFrame

Warning

If the preprocessing included the imputation of missing values, this step is not reversed and the output dataset contains the imputed values.

class_encoding(y)[source]

Computes the internal encoding of the given class labels.

Parameters:

y – vector of class labels

Returns:

The probabilistic representation of the classes (one-hot encoding)

property settings

Settings of the dataset

property df_train: pandas.DataFrame

Dataframe containing prepared train data

property df_test: pandas.DataFrame

Dataframe containing prepared test data

property df_val: pandas.DataFrame

Dataframe containing prepared validation data

property raw_df_train: pandas.DataFrame

Dataframe containing raw train data

property raw_df_test: pandas.DataFrame

Dataframe containing raw test data

property raw_df_val: pandas.DataFrame

Dataframe containing raw validation data

property categorical: List[str]

List of categorical attributes

property continuous: List[str]

List of numerical attributes

property immutables: List[str]

List of immutable attributes

property target: str

Name of the target attribute

property scaler: sklearn.base.BaseEstimator

Contains a fitted sklearn scaler.

Return type:

sklearn.preprocessing.BaseEstimator

property encoder: sklearn.base.BaseEstimator

Contains a fitted sklearn encoder.

Return type:

sklearn.preprocessing.BaseEstimator

property imputer: sklearn.base.BaseEstimator

Contains a fitted sklearn imputer.

Return type:

sklearn.preprocessing.BaseEstimator

transform(df: pandas.DataFrame) → pandas.DataFrame[source]

Transforms raw data into the form expected by the models. Only applicable to DataFrames that have not been preprocessed yet.

It is recommended to use it to keep consistent encodings and normalization.

Parameters:

df (pd.DataFrame) – Contains raw (not normalized and not encoded) data.

Returns:

output – Prediction input normalized and encoded

Return type:

pd.DataFrame

inverse_transform(df: pandas.DataFrame) → pandas.DataFrame[source]

Transforms the output of the models back into the original form of the data. Only applicable to preprocessed DataFrames.

Parameters:

df (pd.DataFrame) – Contains normalized and encoded data.

Returns:

output – Prediction output denormalized and decoded

Return type:

pd.DataFrame

get_pipeline_element(key: str) → Callable[source]

Returns a specific element of the transformation pipeline.

Parameters:

key (str) – Name of the pipeline element to return

Return type:

Pipeline element