Spark Execution Context

Modified on Fri, 09 Apr 2021 at 02:06 PM

TABLE OF CONTENTS

General Description

A functionality was added that enables the use of different Spark Execution Contexts (SEC) for different Domains and projects on an ONE DATA instance. Super and Domain Admins can manage these contexts.


This brings the following benefits:

  • You can allocate specific resources (hardware) to your context.
  • If one of the Spark contexts faces an issue, an instance restart is not required. Only that particular context would need to be restarted.

Necessary Rights

The SEC settings can only be accessed if the user is a Super Admin or a Domain Admin. There are differences in what the Super Admin and the Domain Admin can view and edit. These differences are explained in Spark Settings Views.


Where to Find the Functionality

The Spark Execution Context module can be accessed in two ways. In the application, Super and Domain Admins can access Spark settings through the "SPARK Settings" button in the general settings.

The other option is to access them via URL https://{{od-instance}}/spark-settings/ (e.g. https://team.onedata.de/spark-settings/). If a normal user tries to access the page this way, he will get the following error screen.


The SEC module contains 3 views: Context View, Domain View and Project View. More on these in the next section.



SPARK Settings Views

Context View

This view is available to both Super Admins and Domain Admins. It lists available Spark contexts, with the difference that Domain Admins only see contexts of Domains they are admin of. Super Admins can also register new contexts and delete existing ones. The buttons for these functions are not visible to Domain Admins.


Creating a Context

A new context can be created with the button "Register Context" (2), which opens the following dialog.



There are currently two types of contexts: Built-in and Remote Context. There is always only one Built-in Context, the default one. It's not possible to create more. Remote Contexts can be created as many times as the user wants. To create a Remote Context, a host needs to be available. These hosts are ONE DATA Computation Servers inside of Docker Containers, which have to be created by your DevOps. The name of the host needed to enter in the dialog is the name of the Docker Container.


Deleting a Context

A context can only be deleted, if it is not used in any Domains. The delete button is the trash can symbol (5) at the end of a row.


Setting the Instance Default Context

The default context of the ONE DATA instance is used as the context for Domains that a Super or Domain Admin has not manually set in the Domain View. It is marked with the "Instance Default" label in the list of contexts (3). The default context can either be set using the gear symbol (4) or by choosing a specific context with the "Set INSTANCE Default Context" button (1).

 

Domain View

In this view, the Super Admin can view all Domains on the instance and modify the Domain Default Spark Context for each Domain. Domain Admins can only view and update Domains they are admin of. By default, until changed manually, the Instance Default Spark Context is used as the Domain Default Spark Context.



By clicking on a Domain, the element opens and reveals a view similar to the Context View. 



The Default Context can be set using the same buttons, as in the Context View. After changing the default context, projects that are using the Domain Default Context will switch to the new context automatically. Furthermore, new projects in that Domain will use the new default context. 

The "Connect Context" button is only visible to the Super Admin and lets them add new contexts to the Domain.


Project View

In this view, a Super Admin can view all projects on the instance and modify their SEC. A Domain Admin can view and update projects of the Domain he is admin of.




Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons

Feedback sent

We appreciate your effort and will try to fix the article