Creating Data Transformers: Web Application

A data transformer is an object that transforms input data and (depending on the transformation type) loads transformed data into one or several consumer objects. Both data providers or other transformers can be used as transformer inputs. Similarly, both consumer objects or other transformers can receive data from transformer outputs. The length of transformation chain is not limited.

If the following transformers are used in a task: Delete Duplicates, Sort, Join, batch data processing will be ignored.

Creating a Data Transformer

To create a data transformer, open the context menu in the ETL task's working area and in the Transformer group select the required transformer type.

Editing Transformer Basic Properties

After executing one of the operations, the transformer's settings panel is displayed:

On the settings panel set basic properties that are common for all transformers:

To determine work parameters and to create links between other objects, click the Settings button.

Deleting Transformer

To delete a transformer, select the Delete item in the object's context menu. After executing the operation the operation confirmation is requested.

Editing Transformer Inputs

NOTE. The Input Editor tab is displayed in the example of the Union transformer. The panel appearance and the number of available inputs depends on the transformer type.

Inputs and Links

An input can be added in two ways:

Set up an input of already created transformer. To set up input of the created transformer, execute the operations:

Make sure that the working area contains at least one object, which data should be sent to the transformer input.
Click the Add New Input button on the Input Editor tab. The list of available sources is displayed on the tab:

NOTE. The list contains only the objects, for which a link with transformer has not been set up yet.

Click to select the required source from the list. The selected data source and its fields are displayed on the Input Editor tab. The default input identifier is IN, where N - number of the selected transformer input.

After executing the operations the input and its link with the object are created:

NOTE. The configured input is displayed in the example of the Union transformer.

If multiple inputs can be selected for a transformer, click the Add New Input button and select input source again.

Set up input simultaneously with adding a transformer to ETL task. To set up input on creating a transformer:

Make sure that the working area contains at least one object, which data should be sent to transformer input.
Hover the cursor to the selected object. The cursor is turned to the pointer, each side of the object displays connecting points:

Click to select one of the connecting points. In the Transformer group select the required transformer.

After executing the operations a transformer is added to the working area, a link between the object and the transformer will be set up automatically.

To edit input identifier, double-click the input identifier, and the identifier field is turned to the edit mode. Enter a new identifier. An input identifier should be unique and should start with a letter. Usage of Latin letters, numbers and the "_" character is allowed.

To edit input fields, go to the Editing Field section.

Duplication enables the user to create an input with fields of previously created input.

NOTE. Duplicating input is available for the transformers, which allow for adding multiple inputs.

To duplicate input:

Click the Actions button in the line with the identifier of the corresponding input on the Input Editor panel.
In the drop-down menu select the Duplicate item.

After executing the operations a new input is added with the fields of the previously created input.

To delete input:

Click the button in the line with the identifier of the corresponding input on the Input Editor panel.
In the drop-down menu select the Delete Input item.

After executing the operations the input is deleted without operation confirmation.

Editing Transformer Outputs

Outputs and Links

A transformer output can be set up in two ways:

Set up a link with already created data consumer. To set up a transformer output, execute the operations:

Make sure that the working area contains at least one object used as a data consumer.
Click the Create Link button next to the output identifier on the Output Editor tab. The default output identifier is ON, where N - number of transformer output.

The tab display a list of available data consumers:

NOTE. The list contains only the objects, for which a link with transformer has not been set up yet.

Select the required object input in the list. The link will be set up with the selected object.

After executing the operations the transformer output and its link with the object are created.

Set up a link simultaneously with adding a data consumer to ETL task. To set up a link between transformer output and object on creating an object:

Make sure that the working area contains the transformer, for which an output should be set up.
Hover the cursor on the transformer. The cursor is turned to the pointer, each side of the transformer displays connecting points.

Click to select one of the connecting points. In the drop-down list select the object type, which will be used as a data consumer.

NOTE. The command is available if a link between the transformer output and the data consumer has not been created yet, and the maximum number of transformer outputs is not exceeded.

After executing the operations the object is added to the working area, a link between the transformer output and the object is set up automatically.

To edit the output identifier, double-click the output identifier - the identifier field is turned to the edit mode. Enter a new identifier. An input identifier should be unique and should start with a letter. Usage of Latin letters, numbers and the "_" character is allowed.

To edit output fields, go to the Editing Field section.

To delete input:

Click the button in the line with the identifier of the corresponding output on the Output Editor panel.
In the drop-down menu select the Delete Output item.

After executing the operations the output is deleted without operation confirmation.

Editing Input and Output Fields

To set up input fields, go to the Input Editor tab. To set up output fields, go to the Output Editor tab. Available operations for input and output fields are similar:

To create a field:

Click the Add Field button next to the input or output identifier.

The dialog box opens:

Edit identifier and name. They are equal by default.
Select the required field type in the drop-down list, depending on the provider or consumer data type.

After executing the operations the field with the specified parameters is created.

To quickly edit field name, click the field name that should be changed. The name field will be turned to the edit mode. Enter a new name.

To quickly edit field type, click the field type to be edited. Select a new field type in the drop-down list.

To edit all field parameters, select the Edit item in the field's context menu. The dialog box for editing field parameters opens:

If required, change field identifier, name and type. To save the changes, click the Apply button.

To delete a field, select the Delete item in the field's context menu.

The field is deleted without confirmation of executing operation.

To automatically fill in input or output fields:

Click the button in the line with the identifier of the corresponding input on the Input Editor/Input Editor panel.
Select one of the drop-down menu items:

Fill from Provider. Fields from data provider are added to the list.
Fill from Consumer. Fields from data consumer are added to the list.

After executing the operations input or output fields are filled in automatically from data provider/data consumer.

Editing Outputs with Error Records

After executing the operations the Output for Records with Errors tab is displayed:

NOTE. The Output for Records with Errors tab is displayed in the example for the Union transformer. The panel appearance and the number of available outputs depends on the transformer type.

To set additional settings to be applied when there are error records, click the Settings button. The dialog box opens:

Specify the maximum number of error records, information about which is exported. The -1 value is set by default, in this case information about all error records is exported.

NOTE. If there is a number of error records, information export may slow down ETL task execution.

If the maximum number of output records is set, select the action executed if this number is exceeded. The following operations are available: