Release 3.43.0 (77) - Partitioning in ONE DATA (48.3.0 // 1.203.0)

Modified on Thu, 02 Sep 2021 at 04:32 PM

Server Version: 48.3.0 Client Version: 1.203.0

Release Date: 26.08.2021

Compatible ONE DATA components versions are listed at the bottom of this article.

Summary

The ONE DATA release 3.43.0 contains the functionality of partitioning, prevention of CSV injection in CSV download, and data product transfer updates of the Scheduler and Scheduler rules.

Features

Finished

Scheduler & Scheduler rules Update

Scheduler & Scheduler rules Update
Delivered customer value:	The Data Product Update functionality includes Scheduler and the usability of the JSON config is created.
Parts in the release:	Implement endpoint for retrieving/creating schedulers by originId Implement endpoint for updating schedulers Production Line version exists endpoint Rest Assured tests for schedule getorcreate and update endpoints on onedata-backend

Improvement Bundles

Finished

Improvement Bundle: Prevent CSV injection in CSV download in ONE DATA

IB: Prevent CSV injection in CSV download in ONE DATA
Delivered customer value:	Extends the existing functionality to make the filter / escaping mechanism optional but enabled by default - so that the CSV downloads are not vulnerable to CSV injection
Parts in the release:	CSV download: filter for formula injection Add an option to Data Table Save Processor for writing CSVs to file system connections safely

Ongoing

Improvement Bundle: Partitioning in ONE DATA (parquet)

IB: Partitioning in ONE DATA (parquet)
Delivered customer value:	Partitioning as an expert mode feature which offers a possibility to save partitions of data based on a specified column in the data in order to reduce execution time of queries in case of partition with low volume of data a possibility to load a partitioned data table (via Spark) see, in DT save processor, that a partitioned data table is currently used
Parts in the release:	Partitioning - expert mode Client changes for partitioning

IB: Partitioning in ONE DATA (parquet)

Delivered customer value:

Partitioning as an expert mode feature which offers

a possibility to save partitions of data based on a specified column in the data in order to reduce execution time of queries in case of partition with low volume of data
a possibility to load a partitioned data table (via Spark)
see, in DT save processor, that a partitioned data table is currently used

Parts in the release:

Partitioning - expert mode
Client changes for partitioning

Compatible Versions

The following versions are recommended to be used for this release.

ONE DATA Components

ONE DATA versions:

OD Classic
- Server: 48.3.0
- Client: 1.203.0
Apps
- Server: 1.67.4
- Client: 1.97.0
- Docu: 2.19.3
- Styleguide: 0.63.3
Custom Components
- Simple Microservice Form: 0.2.1