Data Producer, User, Owner in mloda¶

Roles in mloda¶

In this notebook, we will describe three roles which exists in the mloda framework.

• Data Producers: Provide access to raw data, business-layer data, or aggregated data.

  • A data scientist or analyst might create simpler datasets or analytical outputs
  • A data engineer might design access to complex data infrastructures, such as data lakes or warehouses
  • Shares plug-ins and with it access to data

• Data User: Interacts with mloda by applying plug-ins while making requests via the mlodaAPI.

  • A data scientist or analyst who needs data and data transformations (features)
  • Consumes features from other parts of the organizations

• Data Owner: Ensures lifetime value, availability and governance of data and features

Data Producer¶

Let us look closer into the role of a data producer.

What could a data producer create?

FeatureGroup API (plug-in) in short¶

In mloda, a data producer defines access by creating feature groups. Here's an example implementation:

class FeatureGroupClass(AbstractFeatureGroup):

    # Root feature definition
    @classmethod
    def input_data(...)
        ...

    # Features derived from other features
    def input_features(...)
        ...

    @classmethod
    def calculate_feature(...)
        ...

Simple example implementation of the FeatureGroup API¶

In the background, mloda loads the plug-ins, which were created before, like this one.

class ReadFileFeature(AbstractFeatureGroup):
    @classmethod
    def input_data(cls) -> Optional[BaseInputData]:
        return ReadFile()

    @classmethod
    def calculate_feature(cls, data: Any, features: FeatureSet) -> Any:
        reader = cls.input_data()
        if reader is not None:
            data = reader.load(features)
            return data
        raise ValueError(f"Reading file failed for feature {features.get_name_of_one_feature()}.")

We use composition to read different data sources. A ReadFile object looks like this:

class CsvReader(ReadFile):
    @classmethod
    def suffix(cls) -> Tuple[str, ...]:
        return (
            ".csv",
            ".CSV",
        )

    @classmethod
    def load_data(cls, data_access: Any, features: FeatureSet) -> Any:
        result = pyarrow_csv.read_csv(data_access)
        return result.select(list(features.get_all_names()))

    @classmethod
    def get_column_names(cls, file_name: str) -> Any:
        read_options = pyarrow_csv.ReadOptions(skip_rows_after_names=1)
        table = pyarrow_csv.read_csv(file_name, read_options=read_options)
        return table.schema.names

As you can see, the implementation is flexible in the sense that if you need something, you can adjust it quite easily. The other files like .json, .parquet and the sqlite access are implemented in a similar fashion.

Complex plug-ins¶

These can be quite varied:

• aggregate features
• entity features
• historical features

Additionally, one can also write feature groups for:

• using feature stores
• using orchestrator steps
• lazy evaluated functions

Quality¶

The producer must optimize quality, which includes:

• defining input and output validators
• manage the storage and retrieval of artifacts
• implementing software testing

An integration test could be done by using mlodaAPI.run_all and custom data.

An example of a unit test could look like:

def test_csv_reader_2(self) -> None: 
   def test_parse_options_are_customized(self, mock_read_csv):
        # Ensure the parse options are as expected
        expected_parse_options = pyarrow_csv.ParseOptions(
            delimiter=",",
            quote_char='"',
            ignore_empty_lines=True
        )

        # Call the method to trigger parse options usage
        CsvReader2.load_data(Mock(), Mock(spec=FeatureSet))

        # Verify that the _parse_options in CsvReader2 are customized
        self.assertEqual(CsvReader2._parse_options, expected_parse_options)

This allows us to apply software engineering practices consistently throughout the entire data workflow.

Consequences¶

Within mloda, the data producer is empowered as the primary driver, owing to the extensive and customizable range of available plugins.

Unlike traditional data toolchains, mloda provides data producers with the flexibility to define their specific start and end points. This enables the versatile application of mloda across different parts of machine learning lifecycle, such as prototyping, training data preparation, or real-time result monitoring.

This includes the autonomy to define the boundaries of the data producer's domain and to govern the outflow of data. Both aspects are managed through feature groups, which remain under the direct control of the data producer.

Data User Role¶

The Data User plays a pivotal role in configuring and utilizing the mloda API for machine learning and data workflows. The mloda API offers flexible configurations to cater to diverse use cases across the ML lifecycle. Below is an outline of the configurations and features that define the Data User's role:

class mlodaAPI:
    def __init__(
        self,
        requested_features: Union[Features, list[Union[Feature, str]]],
        compute_frameworks: Union[Set[Type[ComputeFrameWork]], Optional[list[str]]] = None,
        links: Optional[Set[Link]] = None,
        data_access_collection: Optional[DataAccessCollection] = None,
        global_filter: Optional[GlobalFilter] = None,
        api_input_data_collection: Optional[ApiInputDataCollection] = None,
        plugin_collector: Optional[PlugInCollector] = None,
    ) -> None:

data = mlodaAPI.run_all(requested_feature,...)

Let’s use the API to further explain the Data User role. As shown above, there are several configurations to consider. The key ones are:

• Which features to request and if the compute_frameworks should be limited?
• How data is linked?
• What specific access rights and permissions does the user have?
• How data is refined to meet the requirements of the use case?

With all the given configurations, the mloda core is designed, whenever feasible, to follow the process:

• First, formulate an optimized execution plan
• Second, to execute the plan accordingly

What the user mostly gains is that the process is repeatable and can be run in most environments, as long as the plug-ins are available and the accesses exist (firewalls, credentials).

The data user could run mloda API in following scenarios:

• POC notebooks
• Production code scenarios (model training or realtime prediction)
• Micro service endpont
• KPI or QA test data ingestion

With this, the whole ml lifecycle is represented and plug-ins can be reused in a testable and repeatable way along this cycle.

Data Owner Role¶

Data owners typically operate at various levels within an organization.

It can be the one who produces the data, the business stakeholder responsible for the service, or, in some cases, may not be explicitly defined.

In mloda, the data owner is the one in control of the governance. However, as to date, this system is not included in this open-source offering, as this platform is reserved for development until the plugin ecosystem has a higher degree of maturity.

We have the plug-in functionalities to integrate governance and operations systems in place. Two simple examples can be:

Using organization wide logging¶

class OtelExtender(WrapperFunctionExtender):
    def __init__(self) -> None:
        if trace is None:
            return

        # Function to be wrapped by the Extender
        self.wrapped = {WrapperFunctionEnum.FEATURE_GROUP_CALCULATE_FEATURE}

    def wraps(self) -> Set[WrapperFunctionEnum]:
        return self.wrapped

    def __call__(self, func: Any, *args: Any, **kwargs: Any) -> Any:
        logger.warning("OtelExtender")
        result = func(*args, **kwargs)
        return result

Logging data size¶

class LogSizeOfData(WrapperFunctionExtender):

    def wraps(self) -> Set[WrapperFunctionEnum]:
        # Function to be wrapped by the Extender
        return {WrapperFunctionEnum.VALIDATE_INPUT_FEATURE}

    def __call__(self, func: Any, *args: Any, **kwargs: Any) -> Any:
        result = func(*args, **kwargs)
        size = sys.getsizeof(result)
        print(f"Size: {size}")
        return result

Conclusion¶

In this notebook, we explored the roles of Data Producers, Data Users, and Data Owners within mloda. We delved into the responsibilities and functionalities associated with each role, highlighting how they contribute to the overall data lifecycle.

• Data Producers are responsible for implementing the plugins, ensuring the accuracy of data access processes, and defining relevant configuration options

• Data Users create usage configuration and apply the mlodaAPI to receive data

• Data Owners, while not fully covered in this open-source offering, they are critical for governance and ensuring the ongoing availability and maintenance of essential plugins

Understanding these roles and their interactions, shows how mloda's modular and extensible design is vital in bringing the change to efficient data management practices and processing throughout the machine learning lifecycle

Next steps¶

While mloda holds a great potential to become a unified portal, we face a key challenge: its plugin coverage is not comprehensive enough to integrate seamlessly across all available tools and technologies. Therefore, active community contributions are absolutely essential to accelerate both its adoption and its ability to transform data management practices