Release 3.7.0 (41) - Migration of workflows (46.25.0 // 1.67.0)

Modified on Thu, 30 Jul 2020 at 09:58 AM

Features

In the following all features are listed which are (partly) included in the release. If no other open parts (stories) are left, the feature is finished and otherwise it is clarified as ongoing.

Finished Features

Migration of workflows and upload to parquet
Goal	Dataset can be uploaded and stored directly as parquet. Reports based on migrated Workflows or Data tables work as before. Workflows will be migrated, when the user triggers the migration manually in an old workflow. Datasets used in the old workflow will be migrated, after the user confirms the migration of the workflow being correct, or corrected by the user. Button in workflow detail view, which will be only displayed, if migration is necessary, to validate if migration can be achieved without problems. Warning and errors can be reused as they are currently used via the Job/Config Sidenav and small icons next to the processors. New endpoint to calculate converted workflow, without persisting it. Along with the updated workflow a list with warnings and errors, resulting from the conversion will be send. Errors indicate changes that mus be made to the workflow by the user, example: A FRTP is un-named and converted to a Data Table Save, where it must be named. This name has to be set by the user post conversion. Warnings will show up to indicate non existing options. We also want show the user which nodes have been updated/changed/converted/migrated. The converted workflow is persisted after the user clicks save. On execution the data sets will be migrated to data tables.
Finished parts in the release	Auto migrate on upload to parquet	Dataset can be uploaded and stored as Parquet. The user can choose if he wants to have the uploaded data as CSV or Parquet. If parquet is selected, types are computed automatically. Malformed lines in the uploaded .csv will be written in a second Data Table.
Finished parts in the release	Manual migration of workflows	Migration of a workflow will only be displayed to the user, if it is possible. The conversion has to be triggered manually. Workflows that contain old processors (Dataset Load, FRT-Processor, Dataset Save) can be migrated to Data Table-Load and/or -Save. The newly converted workflow is not persisted directly, as the user might want to do some modification beforehand. It is save when the user clicks actively on "save" or "run workflow". Along with the updated workflow, that is not persisted yet, warnings and errors resulting from the conversion should be communicated to the user. Example: A FRT is unnamed and converted to a Data Table Save, where it must be named. This name has to be set by the user post conversion. On execution of a newly created workflow, datasets are migrated to Data Tables. Migrated datasets → Data Table now, can be used in there respective processors and reports. Reports based on migrated Datasets to Data Table can be used as before (No manual effort needed).

Usability and Technical improvements for Data Tables
Goal	Improve the Data Tables feature with given feedback from data scientists.
Finished parts in the release	Usability and Technical improvements for Data Tables	Result of Data Table Save Processor can be used in Reports. Only one processor for saving and one processor for loading Data. A lot of processors with similar functionalities have to be maintained.