Accuracy of Coordinate System Conversions

From Fmepedia

Created by: Dean Hintz

Q. How Accurate are Coordinate System Transformations in FME?

A. The following describes the general process that FME uses to do conversions and what the typical sources of error.

Distortion is limited more by the operating system / platform and format than FME itself. Typically we are accurate to something like 16 decimal places with the floating point math we do.

The format you are using is a big factor. Some formats such as mapinfo (http://www.fmepedia.com/index.php/MapInfo_Precision_Issues) are integer based and this can cause significant rounding errors depending on the spatial extent used. The same can occur with spatial databases. If you define the spatial extent for an SDE database (http://www.fmepedia.com/index.php/How_do_I_use_FME_to_create_a_new_GeoDatabase_Feature_Class) to be too broad, you lose spatial precision at any given point.

Our reprojection is done using the CSMAP libraries with algorithms that convert source coordinates to geocentric and then to the destination coordinate system. Another important factor with survey data is how exactly the datum conversion is done, such as in the case from Bessel - 1841 (AUT-BM-31) to WGS84. For AUT-BM31 -> LL-WGS84, the steps are:

1. Undo Transverse Mercator projection to get lat/long.
2. Convert to geocentric. (The Bessel ellipsoid definition comes into play here.)
3. Apply 7-parameter Bursa translation.
4. Convert to lat/long. (The WGS84 ellipsoid definition comes into play here.)

Many countries have implemented grid shift (http://fmepedia.com/index.php/Datum_Transformations_and_Grid_Shift_Files) systems, which involve using a grid of small corrections to adjust the x,y values for the datum shift based on location. In Europe, this has only been implemented thus far in Spain and Germany. Where available, you can configure the grid shift files used and the order or precedence under Tools-Options-Datum Grid Shifts in workbench.

If there is some doubt as to the accuracy of our reprojections, with ArcGIS installed, you can always try using the ESRI reprojection engine and compare the results. This can be set in the Workspace Advanced settings.

If you are using a floating point format such as shape, then besides the datum shift issues, your accuracy should be limited mostly by the accuracy of your source data. If your source data is accurate to say 1 thousandth of a millimetre, I would expect that FME should be accurate to at least a hundredth of a mm - you might expect to lose an order of magnitude of precision, but not much more. If you are seeing errors in excess of a mm, then please send us a test case that shows this. Note that the hard thing to do in reproducing reprojection problems is establishing a defensible base case. Often the best way to do this is to go to a Geodesy or Cadastral Authority and get a set of test points in say Bessel - 1841 and WGS84 and see if you can get FME to reproduce one based on the other. If there is a significant difference than please send us your complete test case as well.