Python SDK - Virtual Data Tables

Modified on Thu, 25 Nov 2021 at 09:34 AM

Provided Methods

Create a new virtual data table out of an underlying data table
- createVirtual
Update the configuration of an existing virtual data table
- updateVirtual
Create a new virtual data table with a specified configuration out of an underlying data table
- createVirtualWithConfiguration

Method documentation

All the examples expect that there is already a OneDataApi object initialized and assigned to a variable called "onedata_api".

### Creation of the ONE DATA Api ###
onedata_api = OneDataApi(base_url=base_url, username=user,
                         password=pw, verify=False, timeout=999)

createVirtual

Description

Creates a virtual data table with the given name from a data table with the given ID.

Optional request options (request_transformers, response_transformers, timeout, verify, sleep_after_response_millis, deserializer) can be specified.

Parameters

Property	Type	Required	Default	Description
vdt_source_id	str	true		ID of the source data table
vdt_name	str	true		Name of the newly created virtual data table
target_project	str	true		The target project for the newly created virtual data table
description	str	false	None	Description for the new virtual data table
notes	str	false	None	Notes for the new virtual data table
tags	[str]	false	None	Tags for the new virtual data table

Usage

### Create new virtual data table ###
vdt: DataTable = onedata_api.datatables.create_virtual(vdt_source_id="87ee58aa-1d56-4ba9-b940-0a87762b4a27",
                                                       vdt_name="New Data Table",
                                                       target_project="94cc58bb-1d78-4ba9-s244-0a76562b4b78")

updateVirtual

Description

Updates the virtual data table with the given id XOR name and project id. The vdt can either be updated by specifying the data source configuration or the sql and the id of the underlying data table.

Optional request options (request_transformers, response_transformers, timeout, verify, sleep_after_response_millis, deserializer) can be specified.

Parameters

Property	Type	Required	Default	Description
vdt_source_id	str	true		ID of the source data table
vdt_name	str	true		Name of the newly created virtual data table
target_project	str	true		The target project for the newly created virtual data table
description	str	false	None	Description for the new virtual data table
notes	str	false	None	Notes for the new virtual data table
tags	[str]	false	None	Tags for the new virtual data table

Usage

### Update virtual data table ###

update_config: dict = {
    "origin": "datatable",
    "config": {
        "schema": "table",
        "dataId": "b06349ee-3169-4eee-901f-fcdaa12b9b35",
        "dataOptions": {
            'sql': 'Select * FROM inputTable'
        }
    }
}

vdt: DataTable = onedata_api.datatables.update_virtual(vdt_id="cd21e90f-d4bd-4462-9197-67e7526f6bdf",
                                                       datasource_configuration=update_config,
                                                       execution_target_project_id="ee548bc5-5853-4b32-90ba-5bf4f34e149f")

createVirtualWithConfiguration

Description

Creates a virtual data table with the specified name from the specified source id. The vdt can either be created by specifying the data source configuration or the sql and the id of the underlying data table.

Optional request options (request_transformers, response_transformers, timeout, verify, sleep_after_response_millis, deserializer) can be specified.

Returns: DataTable

Parameters

Property	Type	Required	Default	Description
vdt_source_id	Union[str, uuid.UUID]	true		ID of the source data table
vdt_name	str	true		Name of the newly created virtual data table
target_project	Union[str, uuid.UUID]	true		The ID of the project the virtual data table should be created in
datasource_configuration	dict	false	None	The new datasource configuration of the virtual data table
sql	str	false	None	The sql that should be applied to the source table
description	str	false	None	Description for the new virtual data table
notes	str	false	None	Notes for the new virtual data table
tags	[str]	false	None	Tags for the new virtual data table
execution_target_project_id	Union[str, uuid.UUID]	false	None	The ID of the target project which affects the selection of the Spark Execution Context.

Usage

### Create virtual data table with configuration ###

update_config: dict = {
    "origin": "datatable",
    "config": {
        "schema": "table",
        "dataId": "b06349ee-3169-4eee-901f-fcdaa12b9b35",
        "dataOptions": {
            'sql': 'Select * FROM inputTable'
        }
    }
}

vdt: DataTable = onedata_api.datatables.create_virtual_with_configuration(vdt_source_id="b06349ee-3169-4eee-901f-fcdaa12b9b35",
                                                                          vdt_name=f"Test-{uuid4()}",
                                                                          description="VDT by SDK",
                                                                          tags=["Guten", "Tag"], notes="Gute Noten,
                                                                          target_project_id=project_id,
                                                                          datasource_configuration=update_config)