TABLE OF CONTENTS

• Overview
• Definition of a Data Product
• Necessary Rights for Export, Import and Update
  • Users Who Can Export Data Products
  • Users Who Can Import Data Products
  • Users Who Can Update Data Products
• Where to Find The Functionality
• Step-by-step
  • How to Export a Data Product
  • How to Import a Data Product
  • How to Update a Data Product
• Example Scenario of the Export/Import Feature
• Exported ZIP File Format
• Boundaries and Current Constraints

Overview

The Data Product Transfer feature offers the possibility to export Data Products and transfer them between instances by importing or updating them on the target instance. While a Data Product Import creates a new project, a Data Product Update creates new versions of resources in existing projects. Currently, Data Product Update can update Keys, Connections, Data Tables, Workflows, Production Lines, Functions, Schedules, and Apps. Data Product Transfer can also be used for creating a copy of a Data Product on a single instance. 

Data Product Transfer is useful for transferring Data Products from a development ONE DATA instance to production instances. The biggest advantage of the feature is that the user can smoothly export, import, and update Data Products without support from DevOps.

This is an advanced functionality. To use it efficiently, the user should be familiar with sharing resources and should at least be able to differentiate between modules and instances.

Definition of a Data Product

A Data Product consists of one ONE DATA Project. The manifestation of a Data Product in Software is a ONE DATA Project, containing the ONE DATA resources (Workflow, Production Lines, Data Tables...) and Apps.

Necessary Rights for Export, Import and Update

Users Who Can Export Data Products:

• Super Admins, as they have all rights.
• Domain Admins have authority on Projects belonging to the Domains that they manage. As a result, they can export Projects contained in these Domains.
• Other users can only export Projects if they are the Project Owner.

Users Who Can Import Data Products:

Any user who can create a Project inside the target Domain can import a Data Product. In addition, the same rights management applies as for Apps: App Builder Module is required and the user must have access to the Apps that are to be exported.

Users Who Can Update Data Products:

Any user who has access to the Project and has READ and WRITE rights for the resources can update the Project. 

Where to Find The Functionality

If the user has one of the roles mentioned in the previous section, the export and import functionality can be found at the following places.

The export, import and update functionalities are found within the action bar in the Module overview page of the respective instance or within the action bar of the Project of interest.

In both cases, the user will be redirected to a new page with the following link: https://{{od-instance}}/transfer/  (Example: https://my-instance.teams.onedata.de/transfer/)

Step-by-step

How to Export a Data Product

Go to the transfer page of your current instance either by button or URL (https://{{od-instance}}/transfer/). In the navigation bar on the left, select the export feature.

Then, check if you are in the correct Domain.

The Domain from which a Project will be exported can be modified by navigating to the top-right button of the page.

Select the Project to export. Within a Domain, to easily find the correct Project, the "Filter by modules" functionality can be used to narrow down the list of Projects.

There are some configuration fields under "Export Parameters" which need to be completed.

• Force lock of the Project (recommended): During the export procedure, resources within the Project are 'locked', so they can not be modified.
• Include all versions referenced by other resources: Whether to include only the latest version of the resources (e.g. Workflows), or all of the versions.
• Export Project without data: Whether to export all Data Tables or none of them.

When everything has been configured according to your needs, click the "Export" button. The Project will then be automatically exported in ZIP format, with all the used resources. 

The ZIP archive will include all Apps that are associated to the Project and visible to the current user. The following versions of each App are included in the export:

• latest version
• latest valid version
• published version

Note that the version numbers will be different after the import. To re-identify the versions, custom names can be given to them. 

The association of an App to a Project can be configured in the App's settings.

How to Import a Data Product

Go to the transfer page of your current instance either by button or URL (https://{{od-instance}}/transfer/). In the navigation bar on the left, select the import feature.

Then, check if you are in the correct Domain.

The Domain to which a Project will be imported can be modified by navigating to the top-right button of the page.

Select the target module from the drop-down.

Note that, depending on the selected module, some modifications will be introduced so that the imported Project fits in the module. An example would be that some resources get ignored.

Afterwards, choose a ZIP file using one of the two options and hit the "Import Project" button.

The user will then be informed by a small pop-up when the import procedure takes place and when it is done. The imported Project will be added to the selected Domain.

In some cases, importing a Project with specific resource dependencies can result in warnings. The user will then be informed about these warnings right after the import is done. The warnings can be downloaded in text format. These could be warnings about failed imports or resources that need adaptations, like File System Connections or Credentials.  

How to Update a Data Product

1. Open the transfer page of your current instance either by button or by URL (https://{od-instance}/transfer/), then select the update icon (navigation bar on the left).

2. Select Domain, Module, and the Project that will be updated, then click on NEXT button. 

3. Select (or drag-and-drop) a ZIP file that is to be uploaded and click on NEXT button.

4. Configure the Update Instructions and click on UPDATE PROJECT button.

5. The user will be informed about both successful or failed imports (and other logs).

Example Scenario of the Export/Import Feature

Here you can see a step-by-step scenario: A Project was first exported from the A-Team instance and then imported into the Internal instance.

Exported ZIP File Format

After the extraction procedure has successfully been executed, a ZIP file will be downloaded. This file has a specific format:

Every resource type is assigned a folder (data, models, productionLines, etc.).
There are two other JSON files:

• exportedVersions: Information about the versions of the exported resources.
• projectMeta: Some meta-information about the exported Project (id of original Project, name).

Within each resource folder we have a list of folders representing the resources that are included in the 'parent' folder. Each folder contains meta information of the selected resource and potentially other files. All files are in JSON format, which is very handy for the export procedure.

The structure of the Apps folder is as follows:

• Each App is represented by a folder with the App's ID as its name.
• The appMeta.json contains general metadata of the App.
• Each exported version of an App is contained in a nested folder that is named after the App version's UUID. It contains:
  • a file appConfigVersionMeta.json which contains metadata of the App version.
  • a file rawConfig.json which contains the configuration of the App version is it was written by the person who built it.
  • a file config.json which is a technical internal representation of rawConfig.json. It is needed by Apps.

Note that, per App, an arbitrary number of versions may be present in the ZIP. All of them will be imported in the order of their original version numbers. 

Boundaries and Current Constraints

• Project transfer can only be done between compatible instances. They need the same ONE DATA version and the same configuration (config.properties) for aspects like for example cold plug Processors.
• Before transferring Projects, FRTs within workflows have to be migrated to Data Tables.
• Resources that contain a password, like keys or credentials, need further adaptations after importing them. Their current password is replaced with "test".