The Definitive Guide to FME Server Log Files
From fmepedia
(http://fmepedia.com/index.php/Category:FME_Server)
| Table of contents |
|
|
Overview
This article investigates everything related to logging and FME Server. It describes the location of the log files and exactly what each log file contains. Configuring the log files is also studied in detail. Within this article we will assume Tomcat is the web servlet container. Since the structure of the web applications are the same no matter what servlet container you are using, the information is still be applicable if you are not using Tomcat. Below is an architecture diagram with an overview of all the critical log files. It helps give you an idea of which elements of the FME Server architecture produce which log files.
(http://www.fmepedia.com/attachments/The_Definitive_Guide_to_FME_Server_Log_Files/Images/Log_Files.png)
FME Server Log Files
Location of the Log Files
There is one main directory where FME Server outputs its log files:
<FMEServerDir>\Logs
Within here there are a series of log-files and a number of sub-folders also containing log files. We will now investigate what each log file is showing.
The location of this path can be modified. All FME Server Core log files use this path (i.e. process Monitor and FME Server log files, not engine), so if you change it they will write to this new location. The log files produced by the services do not use this value when building a path, the location can be configured on a service by service basis. See the discussion on the individual service for more details.
<FMEServerDir>\Server\resources\COM\safe\utility\messageLogger_en.properties
and change the property assignment near the top that reads:
LOG_FILE_PATH = C:/Program Files/FMEServer/Logs/
to set the log file root directory you wish to use. Note the required forward-slash character at the end of the path. The system combines the root directory path specified here with the filename/type specified in the processMonitorConfig/fmeServerConfig files to obtain the final log file path.
FME Engine Log
Structure: fmeEngineConfig_n.log. There is one FME_Engine log file for each Engine running on the server
Contents
This log details information about startup and translation processes of the engine. You can also view the parameters the engine is using. By default every time you restart the FME Server service this log file will be re-created. Each FME command (e.g. workspace job), is logged to the FME Engine log along with the details of any pre and post commands included in the sub-section. This log helps us see the actual FME command line run and the sub-section used when running this command.
When would I look in here?
- To see which jobs have been processed by a specific engine;
- View the actual FME command line run and the sub-section used to run the command. This is useful for debugging; if a translation has failed you can run it from the FME Server Console using this command. This just removes another layer of processing which is good practice when debugging;
- If you want to see the parameters an Engine is using to process workspaces.
Configuration
There is one log file for each running engine. Within here you can configure various aspects relating to the engine.
Configuration file: <FME Server Install DIR>\Server\fmeEngineConfig.txt.
Relevant Parameters
- FME_SERVER_LOG_FILE: Sets the name and full path of the FME Engine log files.
FME Server Core Log
Structure: fmeServer_1236300700696.log. There is one FME Server log being written to at any one time.
Contents
This log is also created each time the FME Server is started, this log is used to store messages created by the transformation manager and the repository manager. Parameters relating to the database FME Server is using are also visible in here. The full life-cycle of a job submission i.e. submitted to the queue, processed by an FME Engine and completed are also shown.
When would I look in here?
- If we need to know the job_id of a specific job in the queue we can find it in the fmeServer log.
- Identify if a job has been sent to an engine and a response returned.
- If you have moved FME Server to use a production database and are having problems you can check to see if a connection is being made within here.
Configuration
Configuration file: <FMEServerDir>\Server\fmeServerConfig.txt
Relevant Parameters
- LOG_FILENAME: Allows you to control the name of the log file.
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
Process Monitor Log
Structure: processMonitor_1236300700352.log.
Contents
A new copy of the processMonitor log is created in the FME Server logs directory each time the FME Server is started. This log file details the FME Server components which were started and if any components needed re-start at any time. If an FME Engine fails to start we would see this in the processMonitor log file along with an error message. A licensing issue with an FME Engine would show up in this log. Also, all FME Server translation jobs (i.e. workspace jobs) are shown in this log.
When would I look in here?
- If the FME Engines are not starting then you need to check in here to identify why. It is most likely a licensing issue.
- You can identify the build number of the Engines you are using on the FME Server.
- To view the life-cycle of the job: receiving the request, sending to the engine and a response on the state of the translation can be viewed in here.
Configuration
Configuration file: <FMEServerDir>\Server\processMonitorConfig.txt
Relevant Parameters
- LOG_FILENAME: Allows you to control the name of the log file
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
Service Logs
Each FME Server service creates its own log file within a sub-directory in the FME Server logs directory. We will now study each sub-directory in detail.
Important Note: The naming and structure of the folders in configurable. The conventions used below are the defaults.
Data Download
Location: <FMEServerDir>\Logs\datadownload
Description
Two different types of log file are created in here:
datadownload_<value>.log
- This lo file is created each time the data download service is restarted. It details every request received and processed using the data download service. Other useful information displayed is the job id, success of the job, and number of features returned by the translation. There is also a link to the corresponding log file described below
FME_<Server Name>_<Value>.log
- This log files is created for each translation processed by FME Server. The log file provides logging down to the workspace level. It is similar to the log produce when a workspace is run in FME Workbench. Crucially the number of features written out can be viewed in here. By default a new log file is produced for each job that is submitted using the console.
Configuration
There are two log files which are written to when a data download request is processed. The “datadownload_<value>.log” and the “FME_<Server Name>_<Value>.log”. Each of these are configured in a different file.
The "datadownload_<value>.log” is configured at the servlet level. On Apache the location of the configuration file is here:
<TomcatDIR>\webapps\fmedatadownload\WEB-INF\conf\propertiesFile.properties
Within here you can setup a large number of parameters relating to the log. Here are some of the key parameters you might want to think about changing:
- LOG_FILENAME: Allows you to control the name and path of the log file.
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
The "FME_<Server Name>_<Value>.log" is configured from here:
<FME Install DIR>\Logs\fmeEngineConfig_n.log.
Within this log file under the SUB_SECTION FILE_DOWNLOAD_SERVICE you can set the FME_TRANSFORMATION_LOG_DIR; this details where the log file is written out.
Managing the files
Since a new job is produced for every translation run, the volume of files could become an issue here. The size of individual files is likely to be small.
Data Streaming
Location: <FMEServerDir>\Logs\datastreaming
Description
Two different types of log file are created in here:
datastreaming_<value>.log
A new one of these is created each time the data streaming service is restarted. It details every request received and processed using the data streaming service. Other useful information displayed is the job id, success of the job, and number of features returned by the translation. There is also a link to the corresponding log file described below
FME_<Server Name>_<Value>.log
One of these log files is created for each translation processed by FME Server. The log file provides logging down to the workspace level. It is similar to the log produced when a workspace is run in FME Workbench. Crucially the number of features written out can be viewed in here. By default a new log file is produced for each job that is submitted using the console.
Configuration
There are two log files which are written to when a data streaming request is processed. The “datastreaming_<value>.log” and the “FME_<Server Name>_<Value>.log”. Each of these are configured in a different file.
The “datastreaming_<value>.log” is configured at the servlet level. On Apache the location of the configuration file is here:
<TomcatDIR>\webapps\fmedatastreaming\WEB-INF\conf\propertiesFile.properties
Within here you can setup a large number of parameters relating to the log. Here are some of the key parameters you might want to think about changing:
- LOG_FILENAME: Allows you to control the name and path of the log file.
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
The “FME_<Server Name>_<Value>.log” can be configured from here:
<FMEServerDir>\Logs\fmeEngineConfig_n.log.
Within this log file under the SUB_SECTION STREAM_DOWNLOAD_SERVICE you can set the FME_TRANSFORMATION_LOG_DIR; this details where the log file is written out.
Managing the files
Since a new job is produced for every translation run, the volume of files could become an issue here. The size of individual files is likely to be small.
Job Submitter Service
Location: <FMEServerDir>\Logs\jobsubmitterservice
Description
Two different types of log file are created in here:
jobsubmitterservice_<value>.log
A new one of these is created each time the job submitter service is restarted. It details every request received and processed using the job submitter service. Other useful information displayed is the job id, success of the job, and number of features returned by the translation. There is also a link to the corresponding log file described below.
FME_<Server Name>_<Value>.log
One of these log files is created for each translation processed by FME Server. The log file provides logging down to the workspace level. It is similar to the log produce when a workspace is run in FME Workbench. Crucially the number of features written out can be viewed in here. By default a new log file is produced for each job that is submitted using the console.
Configuration
There are two log files which are written to when a job submitter request is processed. The “jobsubmitterservice _<value>.log” and the “FME_<Server Name>_<Value>.log”. Each of these are configured in a different file.
The “jobsubmitterservice _<value>.log” is configured at the servlet level. On Apache the location of the configuration file is here:
<TomcatDIR>\webapps\fmejobsubmitter\WEB-INF\conf\propertiesFile.properties
Within here you can setup a large number of parameters relating to the log. Here are some of the key parameters you might want to think about changing:
- LOG_FILENAME: Allows you to control the name and path of the log file.
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
The “FME_<Server Name>_<Value>.log” can be configured from here:
<FMEServerDir>\Logs\fmeEngineConfig_n.log.
Within this log file under the SUB_SECTION JOB_SUBMITTER_SERVICE you can set the FME_TRANSFORMATION_LOG_DIR; this details where the log file is written out.
Managing the files
Since a new job is produced for every translation run, the volume of files could become an issue here. The size of individual files is likely to be small.
KML Network Link
Location: <FMEServerDir>\Logs\kmlnetworklink
Description
Only one log file is created in the kmlnetworklink folder: kml_networklink_<value>.log. Since the KML Network Link is an extension of the data streaming service the main logging is done in the data streaming folder (see section above). This file just logs the lifecycle of the request and whether or not it has been zipped to a KMZ file correctly.
Configuration
The kml log is configured at the servlet level. On Apache the location of the configuration file is here:
<TomcatDIR>\webapps\fmejobsubmitter\WEB-INF\conf\propertiesFile.properties
Within here you can setup a large number of parameters relating to the log. Here are some of the key parameters you might want to think about changing:
- LOG_FILENAME: Allows you to control the name and path of the log file.
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
Managing the files
Since a new job is not produced for every translation run the number of files produced is minimal. The size of individual files is also likely to be small as so little is logged for each translation.
OGC
Location: <FMEServerDir>\Logs\ogc
Description
Two different types of log file are created in here:
ogc_<value>.log
A new one of these is created each time the OGC service is restarted. It details every request received and processed using the OGC service. This includes both GetCapabilities and GET requests. If a user is having problems either retrieving the GetCapabilites file or submitting a request it is worth looking in here to view what happened to the request.
FME_<Server Name>_<Value>.log
One of these log files is created for each GET request processed by FME Server (not for the GetCapabilites request). The log file provides logging down to the workspace level. It is similar to the log produce when a workspace is run in FME Workbench. Crucially the number of features written out can be viewed in here. By default a new log file is produced for each job that is submitted using the console.
Configuration
The ogc_<value>.log is configured at the servlet level. On Apache the location of the configuration file is here:
<TomcatDIR>\webapps\fmeogc\WEB-INF\conf\propertiesFile.properties
Within here you can setup a large number of parameters relating to the log. Here are some of the key parameters you might want to think about changing:
- LOG_FILENAME: Allows you to control the name and path of the log file.
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
The “FME_<Server Name>_<Value>.log” can be configured from here:
<FMEServerDir>\Logs\fmeEngineConfig_n.log.
Within this log file under the SUB_SECTION WFS_SERVICE and SUB_SECTION WMS_SERVICE you can set the FME_TRANSFORMATION_LOG_DIR; this details where the log file is written out. By default the WFS and WMS services log to the same folder.
Managing the files
Since a new job is produced for every translation run, the volume of files could become an issue here. The size of individual files is likely to be small.
Server Job Submitter
Location: <FMEServerDir>\Logs\serverjobsubmitter
Description
Every time a workspace is run using the ServerJobSubmitter transformer in workbench a log file is created in this folder. The log file provides logging down to the workspace level. It is similar to the log produce when a workspace is run in FME Workbench. Crucially the number of features written out can be viewed in here.
SOAP
Location: <FMEServerDir>\Logs\soap
Description
soap_<Value>.log
This log file details every request that has come in using the SOAP servlet.
Configuration
The SOAP log is configured at the servlet level. On Apache the location of the configuration file is here:
<TomcatDIR>\webapps\fmesoap\WEB-INF\conf\propertiesFile.properties
Within here you can setup a large number of parameters relating to the log. Here are some of the key parameters you might want to think about changing:
- LOG_FILENAME: Allows you to control the name and path of the log file.
- MAX_LOGFILE_LINES: The maximum number of lines written to the current log file. Once the log file reaches that number of lines a new log is created.
- MAX_LOGFILE_AGE_SECONDS: The maximum allowable age in seconds of previous versions of log files. The event to search for older log files is triggered when a new log file is created.
Managing the files
A new log is produced when the SOAP servlet is restarted. If you are using the SOAP service a lot the log files may need managing as quite a large amount of information is output per request.
Spatial Direct
Location: <FMEServerDir>\Logs\spatialdirect
Description
If you have installed Spatial Direct with FME Server the log files relating to spatial Direct will be populated within this folder.
Web UI
Location: <FMEServerDir>\Logs\webui\webui.log
Description
This log file details output from the FME Server web user interface, both the standard and administrative are logged to the same file by default. It shows the registration of the services as a plug-in with the web application and any error or warning messages produced by the web application.
Configuration
By default the log files for both the FME Server Web UI and the FME Server Admin UI are logged to the same file. The FME Server Web UI log is configured at the servlet level. On Apache the location of the configuration file is here:
<TomcatDIR>\webapps\fmeserver\WEB-INF\conf\propertiesFile.properties
The FME Server Admin UI is configured from here:
<TomcatDIR>\webapps\fmeserver-admin\WEB-INF\conf\propertiesFile.properties
Both of these files contain the following parameters related to logging:
- LOG_FILE_PATH: Allows you to control the name and path of the log file.
- DEBUG_LEVEL: Allows you to control the level of logging.
Since you can control the LOG_FILE_PATH separately in each properties file, you therefore have the option to separate the admin logging from the general web UI logging.
Managing the files
If you are a heavy user then you may want to set some sort of scheduling up to manage this file as it can grow large. As stated above you can also control the level of logging using the DEBUG_LEVEL parameter, it accepts the following parameters:
- OFF (turn off logging)
- SEVERE (errors only)
- WARNING (warnings and errors only)
- INFO (no detailed debug output)
- FINE (basic debug output)
- FINER (somewhat detailed debug output)
- FINEST (highest level of detailed debug output)
Other Log Files
Console
Location: <FMEServerDir>\Logs\console
Description
This folder provides logging for any jobs run using the FME Server Console. The log file provides logging down to the workspace level. It is similar to the log produce when a workspace is run in FME Workbench. Crucially the number of features written out can be viewed in here. By default a new log file is produced for each job that is submitted using the console.
Managing the files
Since a new job is produced for every translation run, the volume of files could become an issue here. The size of individual files is likely to be small.
Installation
Location: <FMEServerDir>\Logs\installation
Description
The main log file in here is:
installation.log
It contains all the logging linked to the installation of FME Server. If you have had problems with the installation of FME Server you should look in here for clues. If FME Server installed correctly you do not need to worry about it.
Tomcat Logs
As well as the log files produced within the FME Sever directory log files are also produced by the web servlet container itself. I am not going to go into details about the specifics of the log files as it will vary depending upon the product you are using. It is just worth being aware of their presence so you can mange them. Tomcat’s log files are located here:
<Tomcat Install DIR>\Tomcat 6.0\logs
These log files are generic Tomcat log files applicable to any web application you have running on Tomcat. Files in here are logging Tomcat's internal workings. If FME Server was not installed these files would still be produced. For details on how to manage them visit Tomcat's web-page.
