Oliver's Python Corner
From fmepedia
Abstract
This tutorial will introduce you in working with FME's PythonCaller. Using PythonCaller and PythonCreator as well, allows you to extend FME by our own functions written in Python.
This means, to embed Python code in FME. The opposite, i.e. embedding FME in Python is not the main point of this tutorial.
Before you start, please have a look at Python with FME!
Installing pyfme
According to Python with FME the Python wrapper module for FME's FME Objects API is called pyfme.
Having Python installed before FME, there's probably nothing to do, because the FME installer up to FME release 2008 is looking for an existing Python installation and copies the necessary files to the appropriate directory.
To check your installation, just start a Python command line and type in:
>>> import pyfme >>> dir(pyfme)
If everything goes well, you'll see something like this:
['FMEBridge', 'FMEBridgePtr', 'FMECoordSysManager', 'FMECoordSysManagerPtr', ... ]
If you get error messages, you'll have to install pyfme by yourself. But don't worry, it's quite easy.
Go to your FME installation directory, for example C:\Program Files\FME2009 and then have a look at the \fmeobjects subdirectory. You'll find some other subdirectories there:
\python \python23 \python24 \python25
In \python23, \python24 there are two files each:
_pyfme.dll pyfme.py
In \python25 you'll find pyfme.py as well, but instead of _pyfme.dll there's a new style library called _pyfme.pyd.
_pyfme.pyd pyfme.py
These two files are everything you need for working with pyfme.
As you suppose, the names of these directories refer to a specific Python version:
\python23 --> Python 2.3.x (http://www.python.org/download/releases/2.3/) \python24 --> Python 2.4.x (http://www.python.org/download/releases/2.4/) \python25 --> Python 2.5.x (http://www.python.org/download/releases/2.5/)
This means that pyfme will definitely not run with other Python versions than 2.3, 2.4 and 2.5.
Just copy _pyfme.dll or _pyfme.pyd and pyfme.py, corresponding to your Python version, to Python's .\Lib\site-packages directory, e.g.
C:\Python24\Lib\site-packages\_pyfme.dll C:\Python24\Lib\site-packages\pyfme.py
respectively
C:\Python25\DLLs\_pyfme.pyd C:\Python25\Lib\site-packages\pyfme.py
Starting with FME 2009 (build 5573), FME comes with an embedded Python interpreter and a subset of the standard Python library (called "FME Python"). This means that users will no longer have to install Python separately in order to use the PythonCreator/PythonCaller transformers or Startup/Shutdown Python scripts.
According to Mark (The FME Evangelist), "subset" of Python means, that everything will be shipped except
- tkinter - the Python interface to the Tk GUI toolkit and - bsddb - the Python interface to the Berkeley DB library
But you won't miss them most likely.
The embedded Python interpreter lives in $FME_HOME/fmepython25. As you suppose, it is Python version 2.5. If you need another version like Python 2.3 or Python 2.4, you probably have to install it by yourself, see above.
The embedded interpreter will be used by default. If you want to use an allready installed Python 2.5, you'll have to copy pyfme.py and _pyfme.pyd by yourself. The reason for this new behaviour is to avoid conflicts with parallel installations of different FME releases.
Now check your installation again, as shown above.
For further installation details please have a look at PyFME Setup which you'll find in your FME installation, e.g.
C:\Program Files\FME2009\fmeobjects\python\apidoc\setup.html
or read PyFME - A Python wrapper for FME Objects (http://www.safe.com/support/fmeobjects/docs/pyfme/index.html) at http://www.safe.com.
Tutorial Data
At the bottom of this page, you'll find a ZIP (http://www.fmepedia.com/attachments//Oliver%27s_Python_Corner/sample_data.zip)-file containing some small sample datasets with only a few number of features. These data are artificial and are located somewhere in Germany.
You may use your own data, but note that the sample datasets are especially prepared for the following exercises, i.e. that some exercises may not run successfully when using your own data.
You'll also find a ZIP-file containing the Python scripts (http://www.fmepedia.com/attachments//Oliver%27s_Python_Corner/tutorial_scripts.zip) discussed in this tutorial.
Working with PythonCaller
Start the Workbench and load the Shapefile called points.shp as source dataset. In the transformers gallery search for Python and drag the PythonCaller into your main Workbench canvas.
There are two ways for embedding your own Python code in FME's PythonCaller:
1. writing your Python code in FME's integrated source code editor or 2. referencing a Python script, stored in an external file, e.g. myFactories.py
For further details please have a look at How Can I Modularize My Python Code (http://www.fmepedia.com/index.php/How_Can_I_Modularize_My_Python_Code).
Using FME's internal editor
Connect the PythonCaller at least with your source dataset and click the red highlighted Properties button for editing PythonCallers parameters. Now click the [...] button to launch FME's integrated source code editor.
Type in the following lines of code:
# script 1
class MyPythonFactory(object):
def __init__(self):
pass
def input(self,feature):
self.pyoutput(feature)
def close(self):
pass
Note: Please pay attention to the style of code block indentation! FME's source code editor uses real tab stops for indenting blocks, whereas other Python IDE's like IDLE substitute tab stops by four spaces (by default). Mixed styles in the same block are not allowed. You will probably have to replace tabs or spaces in code copied from another IDE to avoid errors.
Now submit the "Ok" button to close the editor and return to the parameters dialog.
In the "Edit PythonCaller Parameters" dialog you have to reference the Python class, you just defined in your code.
In Python symbol to use: type in the name of your class:
MyPythonFactory
Close the dialog by clicking "Ok" and connect the PythonCaller to a destination dataset and/or the visualizer.
Finally run your Workbench and have a look at the translation log.
If everything goes well, you'll find some lines similar to these:
(...) Module `PythonFactory' API version matches current core version (3.2 20070516) Loading fmepython module: fmepython24 ... Loading pyfme dll: C:\Programme\GIS\FME2007\fmeobjects\python24\_pyfme.dll ... _pyfme.dll version is 24 Initializing pyfme.dll... Loaded _pyfme library `C:\Programme\GIS\FME2007\fmeobjects\python24\_pyfme.dll' Importing pyfme module... Adding directory `C:\Programme\GIS\FME2007\fmeobjects\python24\' to the python path (...)
The resulting destination dataset seems to be unchanged, but every feature was processed by the PythonCaller.
You'll see later on what happend.
Using an external module
If you prefer working with your favorite Python IDE instead of FME's internal one, no problem. It works nearly the same way.
Write a script containing the same lines of code and save it to myFactories.py.
# script 1
class MyPythonFactory(object):
def __init__(self):
pass
def input(self,feature):
self.pyoutput(feature)
def close(self):
pass
Copy myFactories.py into a place where FME will search for it.
FME automatically adds the following directories to Python's search path (sys.path):
$FME_HOME/python all 'transformer' directories $FME_MF_DIR, i.e. where the mapping file/workspace is saved
Additionally, you can create a "Startup Python Script" (Workspace Settings - Advanced) that updates sys.path.
Return to FME, load your source dataset into the Workbench and connect it to the PythonCaller. In the "Edit PythonCaller Parameters" dialog you now have to reference your external script (=module) AND your MyPythonFactory class by specifying the following symbol to use:
myFactories.MyPythonFactory
Note:
In FME 2007 PythonCaller expects some input in it's own editor, although you want to use your external module.
Just open the editor and type in:
pass
This error will be fixed in FME 2008.
The rest is identical to the explanations above. Connect the PythonCaller to an output an run the workbench.
How does it work?
The Python class MyPythonFactory is composed of three methods:
1. __init__ 2. input 3. close
The __init__ method is the so called constructor. The constructor is called automatically when the class has been called (and an object has been created). As you recognize, only a pass statement is assigend just to do nothing (at the moment).
The input method takes two parameters: self and feature. feature is an instance of pyfme.FMEFeature.
The close method only has a self parameter.
In a nutshell:
The MyPythonFactory class receives a feature as input from FME. Now the __init___ method is called automatically. The input feature is handled by a corresponding input method. After feature processing has finished, the feature is passed back to FME using the pyoutput method. If a close method is present, it will be called finally.
Detour: FME feature
What is a FME feature?
According to Building Applications with FME Objects (http://downloads.safe.com/fme/fme_objects/BuildingAppsWithFMEObjects.pdf) a feature is the fundamental data unit of FME. "(...) a feature consists of a set of attributes and a geometry with an associated coordinate system."
Features with no geometry (attributes only) are also supported, but if a feature has a geometry, it is associated with only one coordinate system at a time.
"When an application reads from a dataset, it receives the data of one feature at a time. When an application writes to a dataset, it sends the data of one feature at a time."
What does this mean?
If FME processes a feature, it does not know anything about the feature it processed before and the feature it will process next. This means that you can only access one feature at a time, but you can't get any information about the whole dataset such as the datasets bounding box, the total number of features and so on.
Logging
At this point you may probably be disappointed that you wrote Python code and run your Workbenches, but apperently nothing happend. Now you'll get to know a really helpful component of pyfme: the FMELogfile class. Using FME's logging capabilities allows you to see what's going on while running a Workbench and to find errors by examining FME's logfile.
Return to your Python source code and modify it as follows:
# script 2
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
self.logger.log('Processing feature number: ' + str(self.count_features))
self.pyoutput(feature)
def close(self):
self.logger.log('Total number of features: ' + str(self.count_features))
Note: from now on we will use external Python scripts. Therefore it is necessary to import pyfme at the beginning of your script.
If you prefer working with FME's internal editor, you should import pyfme in a Startup Python Script. To do this, go to Workspace Settings and expand the Advanced option. Double click Startup Python Script and click [...] to open the Startup Python Script editor.
Type in:
import pyfme
and return to your workbench and run it. Have a look at the translation log and you'll find some new entries:
(...) Processing feature number: 1 FME Configuration: Destination coordinate system set to input coordinate system `DHDN.Gauss3d-3' Opened Shape File C:\xampplite\pyfme\out\points.shp for output Opened DBF file 'C:\xampplite\pyfme\out\points.dbf' for output Processing feature number: 2 Processing feature number: 3 Processing feature number: 4 Emptying factory pipeline Unexpected Input Remover(TestFactory): Tested 4 input features -- 4 features passed, 0 features failed. Unexpected Input Remover Nuker(TeeFactory): Cloned 0 input feature(s) into 0 output feature(s) Source -> Generic(TeeFactory): Cloned 4 input feature(s) into 4 output feature(s) points Feature Counter -1 7(TeeFactory): Cloned 4 input feature(s) into 4 output feature(s) Total number of features: 4 (...)
Additionally you should open the logfile (stored in the same directory and with the same prefix as your Workbench file) and search for lines similar to these:
(...) 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Processing feature number: 1 2007-11-01 16:38:26| 3.4| 0.0|INFORM|FME Configuration: Destination coordinate system set to input coordinate system `DHDN.Gauss3d-3' 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Opened Shape File C:\xampplite\pyfme\out\points.shp for output 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Opened DBF file 'C:\xampplite\pyfme\out\points.dbf' for output 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Processing feature number: 2 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Processing feature number: 3 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Processing feature number: 4 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Emptying factory pipeline 2007-11-01 16:38:26| 3.4| 0.0|STATS |Unexpected Input Remover(TestFactory): Tested 4 input features -- 4 features passed, 0 features failed. 2007-11-01 16:38:26| 3.4| 0.0|STATS |Unexpected Input Remover Nuker(TeeFactory): Cloned 0 input feature(s) into 0 output feature(s) 2007-11-01 16:38:26| 3.4| 0.0|STATS |Source -> Generic(TeeFactory): Cloned 4 input feature(s) into 4 output feature(s) 2007-11-01 16:38:26| 3.4| 0.0|STATS |points Feature Counter -1 7(TeeFactory): Cloned 4 input feature(s) into 4 output feature(s) 2007-11-01 16:38:26| 3.4| 0.0|INFORM|Total number of features: 4 (...)
As you recognize, there are the same entries in the logfile as in the translation log. But at the beginning of each line, there are some additional columns showing a time stamp and some statistics. The most interesting column is the fourth column where you can see the |INFORM| and |STATS | statements. These statements indicate if everything worked fine or errors occured. In case of errors, a severity code is typed out.
FMELogfile knows six different severities, which are printed out in different colors in the translation log.
Severity Color 0 INFORM FME_INFORM black 1 WARN FME_WARN blue 2 ERROR FME_ERROR red 3 FATAL FME_FATAL red 4 STATS FME_STATISTIC black 5 STATSREP FME_STATUSREPORT black
The default severity is INFORM (0), but you can assign a different severity if you want. Just add the code number to your self.logger.log statement:
self.logger.log('Processing feature number: ' + str(self.count_features),1) # switch severity to WARN (1) and print it out in blue
Check it out by changing the code number from 0 to 5, run your Workbench and examine the translation log and your logfile!
I you can't memorize the meaning of the numbers 0-5 you can use the corresponding text instead, e.g.:
self.logger.log('Processing feature number: ' + str(self.count_features),FME_WARN) # switch severity to WARN (1) and print it out in blue
Working with attributes
In this chapter you'll learn how to work with feature attributes. (Re-) Open the last workbench with points.shp as source dataset.
Accessing attributes
Edit your Python script the following way and run your Workbench:
# script 3
import pyfme
from string import upper
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
# Accessing attributes
self.surname = feature.getAttribute('SURNAME') # get attribute by it's name
self.lastname = feature.getAttribute('LASTNAME') # get attribute by it's name
# Process attributes, e.g. convert all characters to upper-case characters
self.up_surname = upper(self.surname)
self.up_lastname = upper(self.lastname)
# Write processed attributes to logfile
self.logger.log(str(self.up_surname) + ' ' + str(self.up_lastname),1)
self.pyoutput(feature)
def close(self):
self.logger.log('Total number of features: ' + str(self.count_features),1)
You see, that accessing attributes is quite easy. Just get them by calling their names. But note: Attribute names are case sensitive! 'LASTNAME' is not the same as 'lastname'.
Accessing all attributes of a feature
If you want to access all attributes of a feature, it's not necessary to call them all by their names one by one. Just get a list of all feature attributes and loop over them:
# script 4
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
# Accessing all attributes of a feature
self.all_attributes = feature.getAllAttributeNames() # return a list with all attribute names as result
self.all_attributes_count = len(self.all_attributes) # count attributes
self.logger.log('Feature: '+ str(self.count_features),1)
for i in range(self.all_attributes_count):
self.attribute_name = self.all_attributes[i] # get attribute name
self.attribute_value = feature.getAttribute(self.attribute_name) # get attribute value
self.logger.log('Attribute: ' + str(self.attribute_name) + '\tValue: ' + str(self.attribute_value), 1)
self.logger.log('\n',1)
self.pyoutput(feature)
def close(self):
self.logger.log('Total number of features: ' + str(self.count_features),1)
Run your Workbench and have a look at the translation log. You'll notice, that some more attributes are printed out, which do not exist in your source dataset:
Feature: 1 Attribute: X_INT Value: 3333317 Attribute: ID Value: 1 Attribute: fme_type Value: fme_point Attribute: LASTNAME Value: von Goethe Attribute: Y_INT Value: 5684892 Attribute: fme_feature_type Value: points Attribute: fme_geometry Value: fme_point Attribute: SHAPE_GEOMETRY Value: shape_point Attribute: X_FLOAT Value: 3333317 Attribute: SURNAME Value: Johann Wolfgang Attribute: Y_FLOAT Value: 5684892
The highlighted attributes are FME's internal, so called format attributes. We will discuss one of them, namely fme_geometry, later.
Setting attributes
You'll probably want to send your processed attributes back to FME. No problem using pyfme's setAttribute function.
Please return to the script before last and edit it as follows:
# script 5
import pyfme
from string import upper
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
# Accessing attributes
self.surname = feature.getAttribute('SURNAME')
self.lastname = feature.getAttribute('LASTNAME')
# Process attributes, e.g. convert all characters to upper-case characters
self.up_surname = upper(self.surname)
self.up_lastname = upper(self.lastname)
# Sending processed attributes back to FME
feature.setAttribute('SURNAME',self.up_surname) # return attribute back to FME
feature.setAttribute('LASTNAME',self.up_lastname) # return attribute back to FME
# Write processed attributes to logfile
self.logger.log(str(self.up_surname) + ' ' + str(self.up_lastname),1)
self.pyoutput(feature)
def close(self):
self.logger.log('Total number of features: ' + str(self.count_features),1)
And of course it's possible to create new attributes. Add a new line to the setAttribute section:
# Sending processed attributes back to FME
feature.setAttribute('SURNAME',self.up_surname)
feature.setAttribute('LASTNAME',self.up_lastname)
feature.setAttribute('FULLNAME',str(self.surname) + ' ' + str(self.lastname)) # create a new attribute
Note: Connect a Visualizer to PythonCaller's output port before you run the Workbench!
After running the Workbench examine feature attributes in the Vizualizer on the one hand and in the destination dataset on the other hand. You'll notice, that the new attribute called 'FULLNAME' is present in the Vizualizer, but it's missing in the destination dataset.
How to solve this problem?
You have to expose the new attribute! There are two ways for doing this:
1. exposing the new attribute on PythonCaller 2. using the AttributeExposer transformer
Exposing attributes on PythonCaller:
Exposing attributes using AttributeExposer
Alternatively search the Transformer Gallery for AttributeExposer and drag it into the Workspace Canvas. Then connect the AttributeExposer with PythonCaller's output port and open the AttributeExposer.
Typeless attribute handling
You may have realized, that we just used the getAttribute function for accessing an attribute and the setAttribute function for sending it back to FME. We did not specify which type of attribute we wanted to get or set. This means, that getAttribute and setAttribute are working typeless. If you get an attribute typeless the returned value will be automatically converted to a Python type that corresponds to the internal FME type and vice versa.
Of course you can get and set any type of attribute explicitly by using:
getStringAttribute setStringAttribute getIntAttribute setIntAttribute getRealAttribute setRealAttribute getBooleanAttribute setBooleanAttribute
If you're interesting in more details about attribute handling please read Toms Pyfme Attribute Type Handling.
Unicode strings
You should especially avoid to get/set UnicodeString objects by getStringAttribute/setStringAttribute, because this will propably cause an error,
compare pyfme.setStringAttribute error(?) (http://groups.google.de/group/fmetalk/browse_thread/thread/229d006a3fd493b8).
Just get/set them by explicitly using:
getUnicodeString setUnicodeString
Working with lists
It's also possible to work with Python respectively FME lists by using:
getListAttribute setListAttribute
The only difference to non-list attributes is, that you have to expose a list as a so-called unqualified list in Workbench, e.g.
foo{}
foo{}.color
Tom's insider tip:
"getListAttribute works very similarly to a pattern matcher.
I.e. foo{ } looks for all attributes with names foo{0}..foo{N}.
The match has to be complete, not partial. foo{ } will not match foo{0}.bar"
See PythonCaller + getListAttribute (http://groups.google.com/group/fmetalk/browse_thread/thread/a2c06dd976417e83/3861140fa239a842?lnk=gst&q=Python#3861140fa239a842) on FMETalk for an example.
Ok, let's look how it works.
Open the last Workbench and insert the following lines of code:
...
feature.setAttribute('FULLNAME',str(self.surname) + ' ' + str(self.lastname)) # create a new attribute
# script 5b
# Process Python/FME lists
# Due to a bug in pyfme for [get|set]ListAttribute this only works with FME 2008, build >=5150!
# Initialize some Python lists
list_goethe = ['Geistesgruss','Vor Gericht','An den Mond','Der Erlkoenig','Xenien','...']
list_schiller = ['Ode an die Freude','Resignation','Die Teilung der Erde','Der Handschuh','Der Taucher','...']
list_herder = ['Terpsichore','Kalligone','Die Sonne und der Wind','...']
list_wieland = ['Geschichte des Agathon','Musarion','Idris und Zenide','Nadine','...']
# Send lists back to FME
if self.lastname == 'von Goethe':
feature.setListAttribute('WORKS',list_goethe)
elif self.lastname == 'von Schiller':
feature.setListAttribute('WORKS',list_schiller)
elif self.lastname == 'von Herder':
feature.setListAttribute('WORKS',list_herder)
elif self.lastname == 'Wieland':
feature.setListAttribute('WORKS',list_wieland)
# Write processed attributes to logfile
...
Return to Workbench and add a new attribute, either by adding it to PythonCaller or to AttributeExposer:
WORKS{}
From now on, you can access your list attribute as a usual FME list.
Now you should know the basics for working with attributes, but there are many more functions for working with attributes in pyfme.
For example attributeExists, which returns true or false and can be used for checking, if an attribute is present or not:
# script 6
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.attribute_exists = feature.attributeExists('LASTNAME')
self.logger.log('Attribute "LASTNAME" exsists: ' + str(self.attribute_exists),1)
self.attribute_exists = feature.attributeExists('lastname')
self.logger.log('Attribute "lastname" exsists: ' + str(self.attribute_exists),1)
self.pyoutput(feature) # Note: the close method is optional and we don't need it at this point
Please explore the other functions by yourself by typing:
import pyfme dir(pyfme.FMEFeature)
on Python's command prompt.
Working with Geometries
Working with geometries is slightly more difficult than working with attributes, because geometries can't be handled typeless (unfortunatly). But don't be discouraged, you'll learn how to work with them!
Please open a blank new Workspace and load all sample data as source datasets. Additionally load points.dbf as dBASE III (dbf):
points.dbf # as dBASE III (dbf) points.shp # as Shapefile lines.shp area.shp multipoint.shp polygon_with_donut.shp multipolygon_with_donut.shp
Drag the PythonCaller into your Workbench and connect it with all source datasets incl. points.dbf.
Now write the following Python code in your preferred editor (PythonCaller or external one):
# script 7
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.feature_name = feature.getAttribute('fme_feature_type') # getting feature name
self.geometry_type = feature.getGeometryType() # getting Geometry Type
self.logger.log('Feature [' + str(self.feature_name) + '] has geometry type: ' + str(self.geometry_type), 1)
self.pyoutput(feature)
Run your Worbench an examine the translation log. You'll find the following entries typed in blue:
Feature [points] has geometry type: 0 # points.dbf Feature [points] has geometry type: 1 # points.shp Feature [lines] has geometry type: 2 Feature [area] has geometry type: 4 Feature [polygon_with_donut] has geometry type: 8 Feature [multipoint] has geometry type: 512 Feature [multipolygon_with_donut] has geometry type: 512
According to Building Applications with FME Objects (http://downloads.safe.com/fme/fme_objects/BuildingAppsWithFMEObjects.pdf) (see page 15), FME knows the following Geometry Types:
0 = fme_no_geom 1 = fme_point 2 = fme_line 4 = fme_polygon 8 = fme_donut 512 = fme_aggregate
They all have to be handled in a different manner. Therefore you allways have to find out which type of geometry you're currently working with.
Ok, let's start over.
Accessing points, lines and polygons
Open a blank Workbench and load points.shp as source dataset. Drag PythonCaller into the Workspace and write the following lines of Python code:
# script 8
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
self.my_coords = feature.getCoordinates() # get feature coordinates
self.my_coords_count = len(self.my_coords) # count feature coordinates
for i in range(self.my_coords_count): # loop over all coordinates
self.xy = self.my_coords[i]
self.x = float(self.xy[0])
self.y = float(self.xy[1])
self.logger.log('Feature ' + str(self.count_features) + ' : X=' + str(self.x) + ', Y=' + str(self.y), 1)
self.pyoutput(feature)
Connect PythonCaller with a Visualizer, run your Workbench and examine the translation log as usual. Switch to FME Viewer and select a feature. Now compare the coordinates in your translation log with the Coordinate Listing in FME Viewer. The coordinates in the log should be identical to those listed in FME Viewer.
Repeat these steps twice by replacing points.shp by lines.shp and area.shp as source dataset.
Exporting OGC WKT and WKB
getCoordinates returns plain coordinates as a result, but sometimes it's really useful to work with coordinates in a standardized notation. The most common interchange formats are defined by Open Geospatial Consortium, Inc.® (OGC (http://www.opengeospatial.org/)):
OGC WKT = Well Known Text OGC WKB = Well Known Binary
pyfme supports both formats, but we will discuss Well Known Text only, because Well Known Binary is a binary format as the name suggests and it can't be dumped as text in FME's translation log.
Exporting feature coordinates as OGC WKT is quite easy. Just modify your Python code as follows:
# script 9
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.ogc_wkt_coords = feature.exportGeomOGCWKT() # use feature.exportGeomOGCWKB() for WKB
self.logger.log(str(self.ogc_wkt_coords), 1)
self.pyoutput(feature)
If you execute your Workbench with points.shp, lines.shp and area.shp, you'll find the follwowing entries in your translation log:
POINT (3333317 5684892) LINESTRING (3333317 5684892,3333417 5684892) POLYGON ((3333317 5684892,3333317 5684992,3333417 5684992,3333417 5684892,3333317 5684892)) # first/last coordinate are identical to close polygon
Accessing donut features
Working with donut features is a little bit more complicated than working with simple geometries like points, lines and polygons, because donut features are compound features consisting of one outer shell and one or more inner shells. These parts have to be processed separately.
Open a new Workbench with polygon_with_donut.shp, PythonCaller and a Visualizer. Then edit the following Python code and run your Workbench:
# script 10
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
self.logger.log('Feature: ' + str(self.count_features),1)
self.my_geom_type = feature.getGeometryType()
if self.my_geom_type == 8: # 8 = fme_donut
self.logger.log("Processing donuts ...", 1)
self.my_donut_collection = pyfme.FMEFeatureVectorOnDisk() # create a container object, which holds the donut parts
self.my_donut_collection = feature.getDonutParts() # get donut parts, i.e. outer and inner shells
self.my_donut_collection_count = len(self.my_donut_collection) # count donut parts
for k in range(self.my_donut_collection_count): # loop over all parts (outer and inner shells)
self.logger.log('Feature part: ' + str(k), 1)
self.my_donut_part = self.my_donut_collection[k]
self.my_coords = self.my_donut_part.getCoordinates()
self.my_coords_count = len(self.my_coords)
for m in range(self.my_coords_count): # loop over part coordinates
self.xy = self.my_coords[m]
self.x = float(self.xy[0])
self.y = float(self.xy[1])
self.logger.log(str(m+1) + ': X=' + str(self.x) + ', Y=' + str(self.y), 1)
m += 1
k += 1
del self.my_donut_collection # destroy container object
else:
self.logger.log("Feature is not a donut!", 1)
self.pyoutput(feature)
With pyfme.FMEFeatureVectorOnDisk() we use an auxiliary function which is necessary for working with compound features like donut or aggregate features.
Accessing aggregate features
Processing aggregate features is roughly the same as processing donuts. Open a blank Workbench with multipoint.shp as source dataset, PythonCaller and Visualizer. Write your script containing the following lines of code:
#script 11
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
self.logger.log('Feature: ' + str(self.count_features),1)
self.my_geom_type = feature.getGeometryType()
if self.my_geom_type == 512: #512 = fme_aggregate
self.logger.log("Processing aggregates ...", 1)
self.my_feature_collection = pyfme.FMEFeatureVectorOnDisk() # initialize feature collection
self.my_feature_collection = feature.splitAggregate(True) # split aggregate into its constituent parts
self.my_collection_count = len(self.my_feature_collection)
for i in range(self.my_collection_count): # loop over all parts
self.logger.log('Feature part: ' + str(i), 1)
self.my_feature_part = self.my_feature_collection[i] # getting part
self.my_coords = self.my_feature_part.getCoordinates()
self.my_coords_count = len(self.my_coords)
for m in range(self.my_coords_count): # loop over part coordinates
self.xy = self.my_coords[m]
self.x = float(self.xy[0])
self.y = float(self.xy[1])
self.logger.log(str(m+1) + ': X=' + str(self.x) + ', Y=' + str(self.y), 1)
m += 1
i += 1
del self.my_feature_collection # destroy feature collection
else:
self.logger.log("Feature is not an aggregate!", 1)
self.pyoutput(feature)
Instead of getting donut parts you have to split the aggregate feature by feature.splitAggregate(True).
Putting it all together
Now we are ready for working with all geometry types known by FME. This script should also run with your own datasets (vectors only, of course).
# script 12
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.count_features = 0
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.count_features += 1
self.logger.log('Feature: ' + str(self.count_features),1)
self.my_geom_type = feature.getGeometryType()
if self.my_geom_type == 512:
self.aggregate_handler(feature)
elif self.my_geom_type == 8:
self.donut_handler(feature)
elif self.my_geom_type == 4:
self.simple_feature_handler(feature)
elif self.my_geom_type == 2:
self.simple_feature_handler(feature)
elif self.my_geom_type == 1:
self.simple_feature_handler(feature)
elif self.my_geom_type == 0:
self.logger.log('Feature has no geometry!',1)
else:
self.logger.log('Feature has an unknown geometry!',2)
self.pyoutput(feature)
# let's define some supporting functions
def simple_feature_handler(self,my_feature):
self.my_coords = my_feature.getCoordinates()
self.my_coords_count = len(self.my_coords)
self.logger.log("Number of Coords: " + str(self.my_coords_count),1)
for i in range(self.my_coords_count):
self.xy = self.my_coords[i]
self.x = float(self.xy[0])
self.y = float(self.xy[1])
self.logger.log(str(i+1) + ': X=' + str(self.x) + ', Y=' + str(self.y), 1)
i += 1
def donut_handler(self,my_feature):
self.logger.log("Processing donuts ...", 1)
self.my_donut_collection = pyfme.FMEFeatureVectorOnDisk()
self.my_donut_collection = my_feature.getDonutParts()
self.my_donut_collection_count = len(self.my_donut_collection)
for i in range(self.my_donut_collection_count):
self.logger.log('Donut part: ' + str(i), 1)
self.my_donut_part = self.my_donut_collection[i]
self.simple_feature_handler(self.my_donut_part)
i +=1
del self.my_donut_collection
def aggregate_handler(self,my_feature):
self.logger.log("Processing aggregates ...", 1)
self.my_aggregate_collection = pyfme.FMEFeatureVectorOnDisk()
self.my_aggregate_collection = my_feature.splitAggregate(True)
self.my_collection_count = len(self.my_aggregate_collection)
for i in range(self.my_collection_count):
self.logger.log('Aggregate part: ' + str(i), 1)
self.my_feature_part = self.my_aggregate_collection[i]
self.my_geom_type = self.my_feature_part.getGeometryType()
if self.my_geom_type == 8:
self.donut_handler(self.my_feature_part)
elif self.my_geom_type <= 8:
self.simple_feature_handler(self.my_feature_part)
else:
self.logger.log('Unknown geometry', 2)
i +=1
del self.my_aggregate_collection
Setting geometries
Settings geometries is as complex as getting them. But there's a shortcut luckily: just import geometries from OGC WKT.
Examine some datasets by yourself (see chapter 7.2) to find out which OGC WKT geometries exist or read the OGC (http://www.opengeospatial.org) specification: Simple Feature Access - Part 1: Common Architecture (http://www.opengeospatial.org/standards/sfa) abbr. "Simple Feature Access 1"!
In a nutshell, the following geometry types are known:
simple features: POINT, LINESTRING, POLYGON, aggregate features: MULTIPOINT, MULTILINESTRING, MULTIPOLYGON
And there are some additional types like POINTZ, POINTM, POINTZM and GEOMETRYCOLLECTION.
If you inspect the following OGC WKT geometry (normally it's one line without line breaks), you'll realize that it's exactly our sample dataset multipolygon_with_donut:
MULTIPOLYGON (((3333417.0 5684992.0,3333417.0 5684892.0,3333317.0 5684892.0,3333317.0 5684992.0,3333417.0 5684992.0), (3333337.0 5684972.0,3333337.0 5684912.0,3333397.0 5684912.0,3333397.0 5684972.0,3333337.0 5684972.0)), ((3333417.0 5685002.0,3333317.0 5685002.0,3333317.0 5685102.0,3333417.0 5685102.0,3333417.0 5685002.0)))
The keyword MULTIPOLYGON outside of the parenthesis indicates the geometry type. The outer parenthesis enclose one feature, in that case an aggregate feature:
MULTIPOLYGON (((aggregate part 1)),((aggregate part 2)))
The first aggregate part consists of a donut feature with one outer shell and one inner shell:
MULTIPOLYGON (((donut outer shell),(donut inner shell)),((aggregate part 2)))
X-value and Y-value of one coordinate are seperated by a space character and coordinate pairs are seperated by commas.
MULTIPOLYGON (((x1 y1,x2 y2,...,xn yn),(donut inner shell)),((aggregate part 2)))
Note: Polygon coordinates are ordered clockwise and donut outer shells too, but donut inner shells are directed counterclockwise!
Ok, let's start over with a new Workbench. Add the dBase file multipolygon_with_donut.dbf, which has currently no geometry, into the Workbench and connect a PythonCaller. Now write these few lines of code:
# script 13
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.logger.log('Importing feature geometry from OGC Well Known Text ...',1)
self.my_ogc_wkt = 'MULTIPOLYGON (((3333417.0 5684992.0,3333417.0 5684892.0,3333317.0 5684892.0,3333317.0 5684992.0,3333417.0 5684992.0),\
(3333337.0 5684972.0,3333337.0 5684912.0,3333397.0 5684912.0,3333397.0 5684972.0,3333337.0 5684972.0)),\
((3333417.0 5685002.0,3333317.0 5685002.0,3333317.0 5685102.0,3333417.0 5685102.0,3333417.0 5685002.0)))' # OGC WKT as one string
feature.setGeometryType(512) # set geometry type to fme_aggregate
feature.importGeomOGCWKT(self.my_ogc_wkt) # assign geometry to feature
self.pyoutput(feature)
You'll probably wonder at the obvious line breaks in the coordinate listing above, but don't worry these are no real line breaks, because the are masked by backslashes \.
Note: Please don't overwrite your source data! The destination dataset should get a different name than the input *.dbf and don't forget to change the Allowed Geometries settings to shape_polygon in destination datasets General Settings.
Add a visualizer or an output dataset to your Workbench and run it!
Now you should know the basics for working with geometries but there are many more functions for working with them in pyfme,
especially for writing geometries the non-OGC-way. Please explore the other functions by yourself or read Building Applications with FME Objects (http://downloads.safe.com/fme/fme_objects/BuildingAppsWithFMEObjects.pdf) to get an idea how it works.
Working with coordinate systems
If you remember the feature definition in chapter 4.4, you've probably kept in mind, that "if a feature has a geometry, its geometry is associated with only one coordinate system at a time." This means, that we have to deal with coordinate systems in this chapter.
Reading coordinate systems
Ok, same procedure as before: new Workbench + points.shp + PythonCaller + some lines of Python code:
# script 14
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def input(self, feature):
self.my_coordsys = feature.getCoordinateSystem() # get features coordinate system
self.pyoutput(feature)
def close(self):
self.fme_coordsys = pyfme.FMECoordSysManager() # initialize FMECoordSysManager
self.coordsys_params = self.fme_coordsys.getCoordSysParms(self.my_coordsys) # query coordinate system parameters; result is a dictionary!
self.logger.log("Coordinate system:\t\t\t" + str(self.coordsys_params),1)
self.cs_name = self.coordsys_params['CS_NAME']
self.cs_description = self.coordsys_params['DESC_NM']
self.cs_source = self.coordsys_params['SOURCE']
self.cs_epsg_code = self.coordsys_params['EPSG']
self.logger.log("Coordinate system name:\t\t" + str(self.cs_name),1)
self.logger.log("Coordinate system description:\t" + str(self.cs_description),1)
self.logger.log("Coordinate system source:\t\t" + str(self.cs_source),1)
self.logger.log("Coordinate system, EPSG code:\t" + str(self.cs_epsg_code),1)
While the input method only reads the coordinate system assigned to the current feature, the close method queries FME's Coordinate System Manager for it's parameters (assuming that all features of one dataset have the same coordinate system). If you explore FME's Coordinate System Manager by typing:
import pyfme dir(pyfme.FMECoordSysManager)
you'll notice some interesting get* functions like getDatum or getEllipsoid. Unfortunately they don't work in pyfme yet.
Assigning coordinate systems
Setting a coordinate system is quite easy. Make a copy of points.shp and delete it's projection file *.prj. Now load the Shapefile into a blank Workbench, connect a PythonCaller, an output dataset and write some Python code like this:
# script 15
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def input(self, feature):
feature.setCoordinateSystem('EPSG:31467') # assigning a coordinate system to a feature (dataset)
self.pyoutput(feature)
Run your workbench and have a look at the translation log, where you'll find this line (typed in black):
FME Configuration: Destination coordinate system set to input coordinate system `EPSG:31467'
This indicates, that the coordinate system was successfully assigned to the destination dataset without changing features Coordinate System in Workbench Navigator. There it's still set to <not set>.
A coordinate system has to be assigned by it's name like 'EPSG:31467', but which names are allowed?
The answer is simple: all names known by FME are allowed! Just open the Coordinate System Gallery (Tools - Browse Coordinate Systems ...) to get a list:
Note: The coordinate system names have to be spelled exactly as definied in the Coordinate System Gallery! Hence you should take to use EPSG (http://www.epsg.org/Geodetic.html) codes instead of verbal descriptions, because EPSG codes are short and the danger of misspelling them is low.
Modifying Features
In the last chapters you learned how to read and write features and how to work with coordinate systems. In this chapter you'll learn to manipulate features by performing FME functions and by working with Factory Pipelines.
Performing FME functions
First have a look at FME Functions and Factories (http://docs.safe.com/fme/html/FFT/fft.htm), chapter FME Functions. Explore the functions @Buffer and @Reproject.
@Buffer @Buffer(<bufferWidth> [,[<strokeAngle> [,<bufferStyle>]]) @Reproject @Reproject(<sourceCS>,<destCS>[,COORDINATES_ONLY][,<rasterInterpolationType>][,<rasterCellSize>])
Now return to FME and add points.dbf to a new Workbench. Connect a PythonCaller and a Visualizer and add this Python code:
# script 16
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def input(self,feature):
# read xy-attributes
self.x = feature.getAttribute('X_FLOAT')
self.y = feature.getAttribute('Y_FLOAT')
# create geometry with coordinate system
self.logger.log('Import feature geometry from OGC Well Known Text ...',1)
self.my_ogc_wkt = 'POINT (' + str(self.x) + ' ' + str(self.y) + ')'
feature.setGeometryType(1)
feature.importGeomOGCWKT(self.my_ogc_wkt)
self.logger.log('Assign coordinate system ...',1)
feature.setCoordinateSystem('EPSG:31467')
# perform some functions, see http://docs.safe.com/fme/html/FFT/fft.htm for detailed documentation
self.logger.log('Perform function @Buffer ...',1)
feature.performFunction('@Buffer(20.0)') # the complete function has to be passed as one(!) string
self.logger.log('Perform function @Reproject ...',1)
feature.performFunction('@Reproject(EPSG:31647,EPSG:4326)')
self.pyoutput(feature)
Run your Workbench and examine the coordinate system and coordinate listings in FME Viewer. You'll detect coordinates in decimal deegrees now.
Please keep in mind, that the complete function call must be passed as one string, i.e. have to be quoted!
Working with Factory Pipelines
Another way for manipulating features is to use so called Factory Pipelines. Factory Pipelines are chained feature and attribute value functions, which allow you to perform sophisticated processing tasks, see Building Applications with FME Objects (http://downloads.safe.com/fme/fme_objects/BuildingAppsWithFMEObjects.pdf) for further details.
First, you have to author a Factory Pipeline. Open a new Workbench and add points.dbf as source dataset. Then connect the 2DPointAdder and select X_FLOAT and Y_FLOAT attributes as X and Y values. Next add the Bufferer and set Buffer Amount to 25 and Interpolation Angle to 5. Finally link the Reprojector to the Bufferer and choose EPSG:31467 as Source Coordinate System and
EPSG:4326 as Destination Coordinate System.
Now select the three transformers (all three have to be marked) and copy (Ctrl+C) them into the clipboard.
Next open a text editor like Notepad and paste (Ctrl+V) the content of the clipboard into the text editor. The result should look like this:
DEFAULT_MACRO WB_CURRENT_CONTEXT
# -------------------------------------------------------------------------
FACTORY_DEF * TeeFactory \
FACTORY_NAME 2DPOINTADDER \
INPUT FEATURE_TYPE points \
OUTPUT FEATURE_TYPE 2DPOINTADDER_OUTPUT \
@Dimension(2) \
@Tcl2("FME_Coordinates addCoord {@Value(X_FLOAT)} {@Value(Y_FLOAT)}") \
@GeometryType(fme_polygon)
# -------------------------------------------------------------------------
FACTORY_DEF * TeeFactory \
FACTORY_NAME BUFFERER_passer_onner \
INPUT FEATURE_TYPE 2DPOINTADDER_OUTPUT \
OUTPUT FEATURE_TYPE __to_buffer__
FACTORY_DEF * TeeFactory \
FACTORY_NAME BUFFERER \
INPUT FEATURE_TYPE __to_buffer__ \
OUTPUT FEATURE_TYPE BUFFERER_BUFFERED \
@Buffer2(25,5,CAP_ROUND)
# -------------------------------------------------------------------------
FACTORY_DEF * TeeFactory \
FACTORY_NAME REPROJECTOR \
INPUT FEATURE_TYPE BUFFERER_BUFFERED \
OUTPUT FEATURE_TYPE REPROJECTOR_REPROJECTED \
@Reproject(EPSG:31467,EPSG:4326,NearestNeighbor,SquareCells)
Save it to MyFactoryPipeline.fmi.
After that open a blank Workbench and add points.dbf and a PythonCaller to it. Write this Python code:
# script 17
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def input(self,feature):
self.logger.log('Applying a factory pipeline ... ',1)
# 1) Create the pipeline
self.my_pipeline = pyfme.FMEFactoryPipeline('Test') # initialize factory pipeline named 'Test'
self.my_pipeline.addFactoriesFromFile('MyFactoryPipeline.fmi') # add pipeline from file; you have to define a path, if it's not in $FME_MF_DIR
# 2) Process pipeline; insert features into the pipeline
self.my_pipeline.process(feature)
# 3) Retreive processed feature from factory pipeline
new_feature = pyfme.FMEFeature() # initialize new feature object
self.my_pipeline.getOutputFeature(new_feature) # Retreive feature from factory pipeline
self.my_pipeline.allDone()
self.pyoutput(new_feature) # Return processed feature back to FME
Finally add a Visualizer and/or a polygon destination dataset, run the Workbench and inspect the results.
Using Factory Pipelines is a very powerful and flexible way for manipulating features. Imagine that you perform some tests on an input feature and, depending on the test results, you apply one or another Factory Pipeline.
Ok, these were the basics. Now you should be able to:
read/write attributes read/write geometries read/write coordinate systems perform functions apply factory pipelines
With this knowledge you are in a position to extend FME by our own functions written in Python. Of course you can combine FME with external Python modules like
Computational Geometry Algorithms Library (http://cgal-python.gforge.inria.fr/) or ReportLab (http://www.reportlab.org/) for writing PDF's (Don't do this! FME 2008 can write PDF's) many more ...
Accessing published parameters
In the previous chapters you learned, how to embed Python functions in FME, either as internal Python code or by referencing an external script. Even if everything works fine, there are still some options to make the interaction between Python and FME more convenient.
Imagine you wrote your own Python reader for an unsupported binary format or you use variables in your Python code. It will become really bothersome to edit your Python code permanently, only to reference another file or to change a variable.
The easiest way would be, to leave your Python code untouched and to make the necessary changes directly in Workbench instead.
No problem! Simply use FME's Published Parameters (Note: This only works with FME 2008 and above!)
The following example contains two Published Parameters:
1. a file descriptor 2. a variable
Open a new Workbench, add a PythonCreator and connect a Logger and/or a Visualizer. Next publish two parameters as shown as follows.
In PythonCreator reference this external script as script21.MyPythonFactory:
# script 21
# access published parameters from external script
import pyfme
import string
import __main__ # FME_MacroValues are part of the __main__ namespace
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def close(self):
# Read source data for creating features from file
try:
param_filename = __main__.FME_MacroValues['Filename'] # get published parameter by name
param_buffwidth = __main__.FME_MacroValues['BufferWidth'] # get published parameter by name
except:
self.logger.log('Required published parameters not set!',2)
my_file = open(param_filename,'r')
self.logger.log("Reading source data from file ...",1)
my_buffer = my_file.readlines()
my_file.close()
for line in my_buffer:
my_line = line.strip().split(';') # strip newline '\n'
ID = my_line[0]
SURNAME = my_line[1]
LASTNAME = my_line[2]
X = my_line[3]
Y = my_line[4]
# Create new feature
newFeature = pyfme.FMEFeature()
# Set attributes
newFeature.setAttribute("ID",int(ID))
newFeature.setAttribute("SURNAME",SURNAME)
newFeature.setAttribute("LASTNAME",LASTNAME)
newFeature.setAttribute("X",int(X))
newFeature.setAttribute("Y",int(Y))
# Create geometry with coordinate system
self.logger.log('Import feature geometry from OGC Well Known Text ...',1)
self.my_ogc_wkt = 'POINT (' + str(X) + ' ' + str(Y) + ')'
newFeature.setGeometryType(pyfme.FME_GEOM_POINT)
newFeature.importGeomOGCWKT(self.my_ogc_wkt)
newFeature.setCoordinateSystem("EPSG:31467")
# Buffer feature
self.logger.log('Perform function @Buffer ...',1)
self.logger.log('Buffer Amount: ' + str(param_buffwidth),1)
newFeature.performFunction('@Buffer(' + str(param_buffwidth) + ')')
self.logger.log("Output features to FME ...",1)
self.pyoutput(newFeature)
Unzip script21_22.zip (http://www.fmepedia.com/attachments//Oliver's_Python_Corner/script21_22.zip), go to Published Parameters and select points.txt as Input file for PythonCreator. You may also change the Buffer width parameter.
FME's Published Parameters can be accessed as a Python dictionary called FME_MacroValues. You can get the values by calling their key names:
FME_MacroValues['Filename'] FME_MacroValues['BufferWidth']
If you use external scripts, FME_MacroValues are published in __main__ namespace.
If you prefer to write the Python source code directly in PythonCreator or PythonCaller instead, FME_MacroValues are part of your modules namespace. For this reason, some changes are necessary:
# script 22
# access published parameters from embedded Python code
import pyfme
import string
global FME_MacroValues
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def close(self):
# Read source data for creating features from file
try:
param_filename = FME_MacroValues['Filename'] # get published parameter by name
param_buffwidth = FME_MacroValues['BufferWidth'] # get published parameter by name
except:
(... the rest is identical to script 21, see above ...)
Note: Due to a bug in FME prior to Build 5573 (20080711), FME_MacroValues are only populated if FME_BEGIN_PYTHON is executed.
Hence add a simple "pass" statement to Startup Python Script. This is sufficient to populate FME_MacroValues properly.
FME 2009 and subsequent builds
In addition to the FME_MacroValues object in the __main__ namespace, now there's also a reference in the pyfme namespace, so you can use it from within Python factory modules.
Now FME_MacroValues can be referenced directly as objects, e.g.:
param_filename = pyfme.macros.Filename param_buffwidth = pyfme.macros.BufferWidth
If you want to test the new features, just return to workbench script 21.
If published, go to Workspace Settings - Advanced - Startup Python Script and delete the pass statement. The script editor should be empty now.
After this, change your Python code as follows:
# script 23 (derived from script 21)
# access published parameters from external script
import pyfme
import string
#import __main__ # comment this out or delete line
#if you use internal code instead (= script 22) then delete the "global FME_MacroValues" line
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def close(self):
# Read source data for creating features from file
try:
param_filename = pyfme.macros.Filename # new style access to macro values
param_buffwidth = pyfme.macros.BufferWidth
except:
(... the rest is identical to script 21, see above ...)
Now run your Workbench. If everything goes well, four points will be created which are buffered by the specified buffer amount.
To get a general idea of all FME_MacroValues in your current workspace you can get a list using the following short (external) PythonCaller script:
# script 26
import pyfme
import __main__
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
myMacroValues = __main__.FME_MacroValues
self.logger.log('These FME_MacroValues are currently set:',1)
for key in myMacroValues.keys():
value = myMacroValues[key]
self.logger.log('Key: ' + str(key) + '\tValue: ' + str(value),1)
def input(self,feature):
self.pyoutput(feature)
Troubleshooting
Error handling using pyfme.FMEException.
To be done ...
Please have a look at pyfme.FMEException (http://groups.google.com/group/fmetalk/t/e224abac845b735d) in the meantime!
Excursus
Now let's have a look at some other points, where you can use Python in FME.
PythonCreator
PythonCreator is a transformer similar to PythonCaller. There's only one difference between PythonCreator and PythonCaller: PythonCreator receives no input from FME, i.e. features are created by PythonCreator itself.
Open a new Workbench and search Transformer Gallery for PythonCreator. Drag it into the Workspace Canvas and connect a Visualizer. Add the following Python script to PythonCreator:
# script 18
import pyfme
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
def close(self):
self.logger.log("Factory received the close signal, time to create features ...",1)
# Source data for creating features
self.myList = [(3342118.14,5680983.65,'Hans'),(3342218.14,5680883.65,'Fritz'),\
(3342318.14,5680783.65,'Willi'),(3342418.14,5680683.65,'Walter')]
feature_count = len(self.myList)
for i in range(feature_count):
# Accessing data from list self.myList
self.myData = self.myList[i]
self.myX = self.myData[0]
self.myY = self.myData[1]
self.myZ = 0.0
self.myName = self.myData[2]
# Creating a new feature
newFeature = pyfme.FMEFeature()
# Setting attributes
newFeature.setIntAttribute("Integer",(i+1))
newFeature.setRealAttribute("Float",(1.95583*(i+1)))
newFeature.setStringAttribute("Text",self.myName)
# Setting geometries the FME way
newFeature.resetCoords()
newFeature.setDimension(2)
newFeature.addCoordinates([self.myX],[self.myY],[self.myZ])
newFeature.setGeometryType(pyfme.FME_GEOM_POINT)
newFeature.setCoordinateSystem("EPSG:31467")
newFeature.performFunction('@Reproject(DHDN.Gauss3d-3,LL-WGS84)')
self.logger.log("Output features to FME ...",1)
self.pyoutput(newFeature)
Run your Workbench and inspect the results in FME Viewer. You may have noticed, that we created geometries the FME way and not the OGC way as described in chapter 7.6.
Startup Python Script
Startup Python Scripts can be used to initialize your Python environment by loading necessary modules (cp. chap. 5) or by adding directories to Python's search path. You can also do some checks, e.g. if all files and directories exists.
For additional information about Startup and Shutdown scripts please consult Startup and Shutdown Scripts (http://www.fmepedia.com/index.php/Startup_and_Shutdown_Scripts).
Shutdown Python Script
Writing Shutdown Python Scripts you can do the opposite, i.e. clean up your directories, copy some files or whatever you want.
For example imagine a Workbench which runs every night. You want to be notified when the Workbench has been finished and you additionally want to read the logfile. Just let FME send you an e-mail.
Add a PythonCaller AND a Shutdown Python Script to your existing Workbench.
# script 19
# PythonCaller script
# Purpose: saves the name of FME's logfile into a file
import pyfme
import os
class MyPythonFactory(object):
def __init__(self):
self.logger = pyfme.FMELogfile()
self.my_logfile = self.logger.getFileName() # get logfile name and path
self.my_logfile = os.path.normpath(self.my_logfile) # normalize pathname
self.my_tempfile = get_filename() # get a dynamic filename, see function below
self.my_file = open(self.my_tempfile,'w')
self.my_file.write(self.my_logfile) # write logfile name into a file
self.my_file.flush() # will be reread in Shutdown Python Script
def input(self,feature):
self.pyoutput(feature) # redirect untouched feature back to fme
def get_filename():
my_drive = os.environ.get("HOMEDRIVE")
my_path = os.environ.get("HOMEPATH")
my_filename = "pyfme_where_is_my_logfile.txt"
my_temp_file = os.path.join(my_drive, my_path, my_filename)
return my_temp_file
# script 20
# Shutdown Python Script
# Purpose: reads the name of FME's logfile from a file and attaches the logfile to an e-mail
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email.Utils import COMMASPACE, formatdate
from email import Encoders
import os
def mail():
# get name of file which stores the name of FME's logfile
# must be identical to PythonCaller!
my_drive = os.environ.get("HOMEDRIVE")
my_path = os.environ.get("HOMEPATH")
my_filename = "pyfme_where_is_my_logfile.txt"
my_temp_file = os.path.join(my_drive, my_path, my_filename)
logfile = my_temp_file
logfile = open(logfile, 'r').read()
# assembling the e-mail
sender = 'Your FME script <your.name@domain.com>'
to = 'your.name@domain.com'
subject = 'FME has finished ...'
mailtext = 'Workbench finished successfully, see attached logfile!' # you can also read the message from a file
AUTHREQUIRED = 0 # if you need to use SMTP Authentification set to 1
smtpuser = 'smtp.user@domain.com'
smtppass = 'smtppassword'
message = MIMEMultipart()
message["From"] = sender
message["To"] = to
message['Date'] = formatdate(localtime=True)
message["Subject"] = subject
message.attach( MIMEText(mailtext) )
# add FME's logfile as mail attachment
attachment = MIMEBase('application', "octet-stream")
attachment.set_payload( open(logfile,"rb").read() )
Encoders.encode_base64(attachment)
attachment.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(logfile))
message.attach(attachment)
# send e-mail
mailServer = smtplib.SMTP('smtp.server.com') # IP or name of your mailserver
if AUTHREQUIRED:
mailServer.login(smtpuser, smtppass)
mailServer.sendmail(sender, to, message.as_string())
mailServer.quit()
mail()
Run your Workbench and check your in-box.
Update: Since FME Build 5612, i.e. FME 2009 and above, there's a new variable called FME_LogFileName.
This simplifies logfile handling extremely. Regarding scripts 19+20, script 19 becomes needless, because you can now access the logfile name
directly from FME_LogFileName. For sending the logfile as e-mail attachment, modify the Shutdown Python Script (script 20) as follows:
# script 20 for FME 2009 and subsequent versions
# Shutdown Python Script
# Purpose: reads the name of FME's logfile from FME_LogFileName variable and attaches the logfile to an e-mail
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email.Utils import COMMASPACE, formatdate
from email import Encoders
import os
def mail():
# get name of FME's logfile from FME_LogFileName variable
logfile = FME_LogFileName
# assembling the e-mail
sender = 'Your FME script <your.name@domain.com>'
to = 'your.name@domain.com'
subject = 'FME has finished ...'
mailtext = 'Workbench finished successfully, see attached logfile!' # you can also read the message from a file
AUTHREQUIRED = 0 # if you need to use SMTP Authentification set to 1
smtpuser = 'smtp.user@domain.com'
smtppass = 'smtppassword'
message = MIMEMultipart()
message["From"] = sender
message["To"] = to
message['Date'] = formatdate(localtime=True)
message["Subject"] = subject
message.attach( MIMEText(mailtext) )
# add FME's logfile as mail attachment
attachment = MIMEBase('application', "octet-stream")
attachment.set_payload( open(logfile,"rb").read() )
Encoders.encode_base64(attachment)
attachment.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(logfile))
message.attach(attachment)
# send e-mail
mailServer = smtplib.SMTP('smtp.server.com') # IP or name of your mailserver
if AUTHREQUIRED:
mailServer.login(smtpuser, smtppass)
mailServer.sendmail(sender, to, message.as_string())
mailServer.quit()
mail()
FMEReader & FMEWriter
In the abstract I wrote, that embedding pyfme in stand-alone Python scripts is not in the main focus of this tutorial. That's right, but at this point you can close Workbench and can have a look at how pyfme works along with stand-alone Python scripts if you want.
Maybe you know, that there are some sample scripts coming along with FME in %FME_HOME%\fmeobjects\samples\Python.
The following script is derived from %FME_HOME%\fmeobjects\samples\Python\ShapeToTAB\shapeToTAB.py and writes an ESRI shape as output instead of a MapInfo TAB.
Please create a blank directory and save the following Python code to a file called script24.py. Additionally copy the points.* shape from the turorial dataset (http://www.fmepedia.com/attachments//Oliver%27s_Python_Corner/sample_data.zip) to your new directory.
# script 24
import sys
from pyfme import *
try:
logger=FMELogfile("script24.log")
logger.log("Opening reader",1)
reader=FMEReader("SHAPE")
reader.open(".") # Note: the SHAPE reader can read all shape files in a certain directory
logger.log("Opening writer")
writer=FMEWriter("SHAPE")
writer.open("out") # create new output directory
schemaFeature=FMEFeature() # ESRI shapes allways have schema features
logger.log("Copying schema features")
while reader.readSchema(schemaFeature):
logger.logFeature(schemaFeature) # introducing a "new" logger function .logFeature which writes features to the logfile
# overwrite schema definition
GeomList = []
GeomList.append('fme_polygon')
schemaFeature.setListAttribute('fme_geometry', GeomList)
writer.addSchema(schemaFeature)
feature=FMEFeature()
while reader.read(feature):
logger.log('Perform function @Buffer ...',1)
feature.performFunction('@Buffer(20.0)')
logger.log('Perform function @Reproject ...',1)
feature.performFunction('@Reproject(EPSG:31647,EPSG:4326)')
logger.logFeature(feature)
writer.write(feature)
reader.close()
writer.close()
print "Script 24 successfully finished."
except FMEException, err:
print "FMEException: %s" % err
sys.exit(1)
Now run "python.exe script24.py". If everything is going well, you should find a new out\ subdirectory containing a shape file with buffered points, i.e. circular polygons.
Tom's insider tip:
Run your script the following way: fme.exe script24.py
Q: What's the benefit of this procedure?
A: Running fme.exe as a Python interpreter option ensures that the FME environment is initialized properly.
FMEWriter: create features from scratch
Ok, the next step is to create features from scratch and to write them to an ESRI shape.
For a better understanding of what's going on, I would advise you to study chapter 5 "Writing Features to a Dataset - Writing Features" in Building Applications with FME Objects (http://downloads.safe.com/fme/fme_objects/BuildingAppsWithFMEObjects.pdf)!
Important note: The following explanations only pertain to ESRI shapes! Writing other formats may be different from writing shapes.
Please read the Quick Facts of the format you want to write.
You'll find the Quick Facts for example in Workbench "Help":
Help -> FME Readers and Writers Reference -> Quick Facts e.g. ESRI Shape Quick Facts
or at Supported formats (http://www.safe.com/products/desktop/formats/index.php) on www.safe.com.
The most important things you can learn from the Quick Facts are:
1. Format Type Identifier the exact name of the writer 2. Dataset Type whether a directory name or a filename has to be specified 3. Feature Type the dataset name 4. Schema Required does the format require a schema specification? Yes/No?
When you've checked these points a script for creating an ESRI shape could look like this:
# script 25
from pyfme import *
import time
# Define some functions to keep the code readable
# ESRI shapes do require a schema, so let's define a schema feature at first
def CreateSchemaFeature(myShapefileName):
schemaFeature = FMEFeature()
# "setFeatureType" sets the name fo the shapefile
schemaFeature.setFeatureType(myShapefileName)
# Define "User-Defined Attributes"
schemaFeature.setSeqAttribute('ID','fme_decimal(1,0)')
schemaFeature.setSeqAttribute('TEXT','fme_char(20)')
schemaFeature.setSeqAttribute('FLOAT','fme_decimal(10,2)')
schemaFeature.setSeqAttribute('DATE','fme_date(8,0)')
schemaFeature.setSeqAttribute('BOOLEAN','fme_logical(5)')
# Create a list of one or more fme_type values that are valid for data features
# In case of SHAPE, the list should contain one type only
GeomList = []
GeomList.append('fme_polygon')
schemaFeature.setListAttribute('fme_geometry', GeomList)
return schemaFeature
def CreateFeature(myShapefileName, myID, myX, myY, myText, myBoolean):
# Get schema feature
schemaFeature = CreateSchemaFeature(myShapefileName)
# Create feature as an instance of the schema feature
newFeature = schemaFeature
# Assign attributes to the new feature
newFeature.setAttribute("ID",myID)
newFeature.setAttribute("TEXT",myText)
newFeature.setAttribute("FLOAT",myX)
newFeature.setAttribute("DATE",str(time.strftime("%Y%m%d")))
newFeature.setAttribute("BOOLEAN",myBoolean)
# Assign a geometry to the new feature
my_ogc_wkt = 'POINT (' + str(myX) + ' ' + str(myY) + ')'
newFeature.setGeometryType(FME_GEOM_POINT)
newFeature.importGeomOGCWKT(my_ogc_wkt)
# Assign coordinate system and perform a function
newFeature.setCoordinateSystem("EPSG:31467")
newFeature.performFunction('@Buffer(10.0)')
# Write feature to logfile and dataset
logger.logFeature(newFeature,FME_INFORM)
writer.write(newFeature)
def OpenLogfile(myLogfileName):
global logger
# Write a logfile for debugging purposes
logger=FMELogfile(myLogfileName)
logger.log("Opening writer",FME_INFORM)
return logger
def OpenShapeWriter(myShapefileDir):
global writer
# "SHAPE" is the Format Type Identifier
writer=FMEWriter('SHAPE')
# In case of a shapefile just open a directory
writer.open(myShapefileDir, {'METAFILE':'SHAPE'})
# Now the SHAPE writer needs to know the schema
schemaFeature = CreateSchemaFeature(myShapefileName)
writer.addSchema(schemaFeature)
return writer
# ################################
# Ok, let's produce some features
# ################################
# Source data for creating features
myList = [(3342118.14,5680983.65,'Hans','true'),(3342218.14,5680883.65,'Fritz','true'),\
(3342318.14,5680783.65,'Willi','false'),(3342418.14,5680683.65,'Walter','false')]
# Define some names
myShapefileDir = 'out'
myShapefileName = 'script25Shape'
myLogfileName = 'script25.log'
# Open logfile and writer
OpenLogfile(myLogfileName)
OpenShapeWriter(myShapefileDir)
# Process data
feature_count = len(myList)
for i in range(feature_count):
# Get data from list myList
myID = i + 1
myData = myList[i]
myX = myData[0]
myY = myData[1]
myText = myData[2]
myBoolean = myData[3]
CreateFeature(myShapefileName, myID, myX, myY, myText, myBoolean)
# Close the writer
writer.close()
del writer
logger.log("script25.py successfully finished",FME_INFORM)
del logger
print "script25.py successfully finished"
Run "fme.exe script25.py".
I published another example (http://groups.google.com/group/fmetalk/web/check_database_and_run_FME_on_success.zip) on FME Talk.
Python on FME Server
If you want to use pyfme on FME Server, please read Michaels hints: Python on FME Server
Beta Changes
In this chapter you'll find some hints concerning the latest pyfme changes in FME betas. The things are changing really dynamically, so that I can't ensure that my hints are allways up-to-date.
If you're using a beta, you should regularly consult at least the following sources of information:
- The FME Evangelist (http://evangelism.safe.com/category/developer-tools/python-developer-tools/) - whatsnew.txt (ftp://ftp.safe.com/fme/beta/whatsnew.txt)
Please note, that I will not include the new stuff in the chapters above, until the next final build will be released.
Jotted
At this point you'll find some notes -without further comments- concerning the latest changes in pyfme.
These notes are extracts of whatsnew.txt (ftp://ftp.safe.com/fme/beta/whatsnew.txt).
BUILD 5614:
@Tcl2/FME_END_PYTHON|TCL/PythonFactory:
Added FME_NumFeaturesLogged variable to all of these places so that a translation can determine the number of features that have been logged. This can be helpful to detect features that a writer may have rejected.
BUILD 5613: FME_BEGIN/END_TCL/PYTHON:
Ensured that the WORKSPACE_NAME, FME_MF_NAME, and FME_MF_NAME_MASTER macros always hold the name the workspace was last saved as when running within workbench environment.
BUILD 5612: FME_BEGIN_TCL/PYTHON:
Added FME_LogFileName variable to the BEGIN Tcl/Python environments.
BUILD 6030 20090222: fmepython:
Cleanup framework issues that caused problems in FME Server (RT#125675 PR#19721)
BUILD 6074 20090514: Workbench Search:
Make workspace parameters searchable so that users can search python and tcl scripts.(PR#20827)
To the well-disposed reader
If you read this tutorial, you like it and it helps you to solve a special problem which could be interesting for others, please share your experiences by publishing a Python Sample (http://www.fmepedia.com/index.php/Category:Python_Sample) script or by completing this tutorial. For requesting editor status see Fmepedia:Feedback (http://www.fmepedia.com/index.php/Fmepedia:Feedback).
About the author
I studied geography in Münster (http://www.muenster.de/stadt/tourismus/en/) (Westphalia) and I'm working in the GIS department of a german mobile telecommunications company (the red one).
I'm neither a professional programmer, nor a FME guru, but I hope, my tutorial is somewhat useful for you!?
Thanks to an evaluation license, I was able to write this tutorial in my spare time.
Regarding the complexity of pyfme my tutorial is neither complete, nor error-free most likely. And you have certainly noticed, that English is not my first language, so please feel free to correct errors concerning the contents and linguistic/grammatical mistakes!
Please post your questions on FME Talk (http://groups.google.com/group/fmetalk) at Google Groups.
Oliver (mailto:pyfme@gmx.net)
My favorite Python ...
... IDE: EasyEclipse for Python (http://www.easyeclipse.org/site/distributions/python.html)
... Book: Beginning Python: From Novice to Professional (http://www.apress.com/book/view/159059519x) by Magnus Lie Hetland, ISBN13: 978-1-59059-519-0
... or from Mr. Python himself: An Introduction to Python - The Python Tutorial (http://www.network-theory.co.uk/python/intro/) by Guido van Rossum and Fred L. Drake, Jr., ISBN: 0-9541617-6-9
... Spatial ETL-Tool: FME : )
