Data Validation with FME Server

From fmepedia

 (http://fmepedia.com/attachments/Data_Validation_with_FME_Server/images/ss_data_valid_invalid.png)

Scenario

This example demonstrates a Data Upload and Validation service hosted on an FME Server installation.

It shows how to create a workspace that allows upload of data to FME Server, that carries out validation tasks on that data, and that returns a HTML report of the validation process (shown right). In doing so it highlights the Data Streaming Service and the upload source data capabilities of the FME Server Web User Interface (WEB UI).

Table of contents

1 Scenario 2 Viewing the Example 3 How it Works 3.1 Workspace 4 Setting up the Example 4.1 Requirements 4.2 Steps 5 Running the Example 5.1 Uploading and Validating Data 5.2 When Data Passes Validation

---

Viewing the Example

This example is set up and running on Safe's FME Server demo site. You can try it out by clicking either of these links below.
There are no installation requirements other than a standard web browser. Data Upload is turned off on our example site; instead you are prompted to select a layer in a pre-loaded dataset.

---

How it Works

As with most FME Server scenarios, a Spatial ETL process (workspace) is designed using FME Desktop (Workbench) and uploaded to FME Server.

The workspace reads a set of pre-defined data (live it would allow selection of a user-defined dataset), validates the contents, and outputs a report of the validation results to a text file. When uploaded to the FME Server it is registered as an FME Data Streaming Service.

Now when the translation URL is called (as in the "Run it Live" link above) a request is made to FME Server. It runs the workspace and returns the text file report directly to a standard web browser.

If no errors are reported then a second workspace is executed. It uploads the validated data into a master spatial database.

Workspace

The workspace used to generate the output has this content:

1. To allow data upload, the source dataset path parameter is simply published in the workspace.
   • The Upload option allows you to pick a ZIP file from your system that contains the source dataset files.
   • Note: files uploaded to FME Server in this manner are temporary and will be deleted after a short period.
2. When the workspace is run, each of the features is run through a series of validation steps.
   • If a problem is found (ie the data is in some way invalid):
     • The error message is converted into a HTML string and is output using the Text File writer.
   • If no problems are found (ie the data is valid):
     • The repository and dataset name are converted into a HTML string which defines a form/button, and is output using the Text File writer.
     • When the Load Data button is pressed, the dataLoader.fmw workspace is run with the source dataset that was validated.
     • The dataLoader.fmw workspace loads the source data into the master database and the FME Server Web Interface displays a report of the translation.

---

Setting up the Example

To actually set up and run the example on your own system requires a certain set of software and setup.

Requirements

• FME Server
• FME Desktop
• Workspaces and Datasets

Steps

• Download the example Workspaces and Datasets zip file.
• Unzip the file.
• Open the 'dataLoader.fmw' workspace in FME Workbench 2008 or greater.
• Publish the workspace to FME Server into the 'Demos' repository and register it with the Job Submitter Service (you may need to create the Demos repository using the New button).
• Copy the 'dataValidatorReportPage.htm' and 'dataValidatorSuccessPage.htm' files into the 'c:\temp\' directory (these files are referenced in the 'dataValidator.fmw' workspace - so alternatively, you could modify the paths in the workspace).
• Open the 'dataValidator.fmw' workspace in FME Workbench 2008 or greater.
• Publish the workspace to FME Server into the 'Demos' repository and register it with the Data Streaming Service.

---

Running the Example

Uploading and Validating Data

• Open the FME Server web interface, expand the Data Streaming Service, expand the 'Demos' repository, click the Configure button next to the 'dataValidator.fmw' workspace.
• Click the Upload button.
  • In the popup, click the Browse button and browse to the 'hydro_poly_updates_bad.zip' file, click Open and then click Next twice.
  • Expand 'hydro_poly_updates_bad.zip', check on 'hydro_poly_updates_bad.gml' and click Finish to close the popup.
• Click the Run button.
• The 'dataValidator.fmw' workspace will validate the GML data and a web page, generated by the workspace, will open in your browser showing the results of the validation.
  • This GML data contains errors, so you will see an report detailing the errors. Try using the 'hydro_poly_updates_good.zip' file.

When Data Passes Validation

• Repeat the above process, but use the 'hydro_poly_updates_fixed.zip' ZIP file and make sure that only the 'hydro_poly_updates_good.gml' file is checked on.
  • This time the GML data will pass validation, so you will see a success message and a button will be displayed that will allow you to load the data into the master/production database.
• Click the Load Data button.
• The 'dataLoader.fmw' workspace will run and the FME Server Web Interface will display a report detailing the results of the translation to the master/production database.