Release 3.6.0 (40) - File Connection Handling (46.24.0 // 1.166.0)

• Unit and integration coverage
• Issues with ORC column schema limitations addressed
• Possible issues with encoding of string cells analyzed and addressed (if necessary)

Optional: enable ORC support in ONE DATA

ORC is far more limited than Parquet is, allowing only Alphanumeric, dot and underscore

It needs to evaluated whether a substitution similar to Parquet is feasible and then implemented. Alternatively, only a forward substition is implemented

Connection can read .csv-files from a local filesystem.

It is possible to create a dataset from a connection. The dataset can be used properly in a Dataset Load Processor.

Filesystem Whitelist per Domain

Allow the superadmin to configure which filesystem directories are allowed to be used via filesystem connections per domain.
The current whitelisting implementation (configured via server's config.properties file) only allows whitelisting directories globally for all domains.
The new per-domain whitelists should work alongside the global whitelist.
A user is allowed to use a specific filesystem directory if at least one of the following two statements is true:

1. The directory is whitelisted via the global config.properties setting
2. The directory is whitelisted by the superadmin for a domain x and the user is a member of x.

Connection can handle multiple files using the same schema.
It is possible to create a dataset from the merged file and use it in the dataset load processor.

Merge rules for file system connections

MVP with ability to merge different filesystem files into a single files
and load it as a dataset.

• Show a sample of the dataset in the dataset details page with at least 5 rows

• Merge rules will be created in the connection settings that define which files will be merged and what the name of the resulting single file will be
• In the connection's file list page, the merged files and the original files are shown
• the connection's file list can be filtered for either only actual files or only merged files or both
• It should be possible to merge 2 - n files into one file
  • no warning if 0 or 1 file is met by merge rule
• It should be possible to create a Data Table from the merged file
• The merged file can be used with the Data Table Load Processor; the usage should be the same as with a Data Table from a single unmerged file
• The split files must have the same schema to merge them to one single Data Table
• Merging should work with file sizes of at least 50 GB (for example: 2 files with 25 GB each and 1000 files with 50 MB each)
• Error message should be shown, if files with different schemas are be merged
• for all files an extra column containing the relative filepath will be added.
  • The Name of the column was not defined yet
  • in case of name collision a prefix/suffix has to be added
• The regex/merge rule is applied on the relative file path
• Search for a regex string is available.

Deal with technical debts and workarounds in Debug-Mode

Bugs and Defects related to Debug Mode

Provide an alternative representation of the graph using node and edge lists, instead of the existing representation (node contains the list of successors), since this makes the substitution algorithm unnecessarily complex.

Instead of showing "ONE DATA" in the title of a tab, the name of the workflow, report, dataset, model, production line is shown.

Add a section in the Integrated Workflow Execution processor to show the name of the different result tables processors as a list.

The branding (= logo in the header) is shown on the list of project page. Inside a project the logo shall not be shown.

In the Data Table detailview, when changing the the data types of a Data Table under the section DATA TABLE SAMPLES the dataset statistics are updated automatically.