Troubleshooting FME Server - Specific Issues
From fmepedia
(http://fmepedia.com/index.php/Category:FME_Server)
| Table of contents |
Why Are Jobs Queuing but Not Getting Done?
Symptom
In this situation you can submit jobs through the FME Server WebUI or any other client but no job ever gets done by an FME.
Resolving
Generally this occurs when the FME Server core is running but you have no FME Engines running. As described on the Trouble Shooting How To page #Is FME Server Running (http://fmepedia.com/index.php/Troubleshooting_FME_Server_-_How_To#Is_FME_Server_Running.3F,)] this most often occurs when you do not have a valid license for your FME Engines - the steps to install your license are also described in this article.
If however, you are sure your FME Engines are licensed there may be an issue with the start-up sequence of FME Server. This can be confirmed by looking in your most recent processMonitor log and fmeServer log here: <FMEServerDir>\Logs\. In these logs you may see that the FME Server took so long to start that the FME Engines gave up trying to connect to it when they were issued their start up commands. Often the root of this problem is that the default HSQL database used to run FME Server had become so large that it does not start up quickly enough. The best solution is to use a production level database (Oracle, SQLServer, MySQL are all supported) for your FME Server. The configuration steps for using a production database can be found in the FME Server Administrators guide in Appendix B. Be sure to back-up your repository database using the FME Server repository reader (Admin Guide Chapter 5 - Upgrading FME Server > Backing Up Your FME Server Repository Using FME
Desktop) before doing this configuration change.
Although we recommend using a production database, we have provided steps for cleaning up the HSQL database in this fmePedia article here:[Repository Database - Removing Records (http://fmepedia.com/index.php/Repository_Database_-_Removing_Records#Steps)]
My hard drive is full
Log files fill up my hard drive
Symptom
Hard drive is full and log files in <FMEServerDir>/Logs directory is to blame.
Resolving
You will need to create an automated task to delete all or some of these log files. In general a Windows Scheduled Task is the easiest way to setup a task that will automatically these delete log files.
Dataset Fanout doesn't work for Services
Symptom
Service fails indicating it can't write to the directory specified by the fanout prefix in the workspace.
Resolving
The FME Server needs to write the data to a specific file path to be accessible to the service but the fanout prefix over-rides this path. There are two steps to resolving this issue
1) In all of your FME Engine Config files (e.g. <FMEServerDir>\Server\fmeEngineConfig_1.txt) ensure that within each service sub-section you have the line: MACRO_DEF FME_SERVER_DEST_DIR !FME_AUTO_DIR_NAME! \
This line sets a published parameter whenever the workspace is run that includes the actual destination directory used by the service.
2) In the workspace publish the Fanout Directory parameter on the writer and give the parameter the name FME_SERVER_DEST_DIR.
When the workspace is run on FME Server this macro is replaced by the destination directory for the service and the data can be fanned out and still accessible to the service.
I can't publish my huge workspace
Symptom
When I try to publish the workspace through workbench it hangs and when I try to publish the workspace through the Admin Web Interface it reports the following error:
An Exception occurred COM.safe.fmeserver.api.FMEServerException QSMessageReadFailed
Or, when I try the add_workspace command from the FME Server Console to add my huge workspace to the repository I get the following error:
* Reading FME Server Response from channel failed with error number 392008
Resolving
This is caused by the FME Server java process running out of memory. To workaround this you need to:
- open <FMEServerDir>/Server/processMonitorConfig.txt in a text editor
- locate the line that starts with CMDStartFMEServer_1=
- immediately before the -Xrs, add the text -Xms32m -Xmx128m so that this line resembles:
CMDStartFMEServer_1="FMEServer\\Utilities\\jre\\bin\\java" -Xms32m -Xmx128m-Xrs COM.safe.fmeserver.FMEServer "FMEServer\\Server\\fmeServerConfig.txt"
My workspace fails with Write Permission Errors
Symptom
Workspace which completes successfully when run in FME Desktop, fails with permission issues on the Destination Datasets when run by FME Server.
Resolution
This happens because the FME Server process is running as a user which requires write access to the location of your destination datasets. Check the security on your destination datasets or folders and ensure that the FME Server logon user has write access.
Tomcat runs out of memory (due to user sessions)
Symptom
Some sort of memory or java error on the web page or in a log file after many user sessions are generated. Or possible "500 Internal" error on webpage.
Resolution
1. Lower session lifetime.
- Open the following file in a text editor:
- <TomcatDir>/conf/web.xml
- Locate the following line:
- <session-timeout>30</session-timeout>
- Decrease the timeout value
- Save and close
- Restart Tomcat.
2. Increase memory available to Tomcat.
To set the start and maximum heap size run the following command before starting your tomcat:
- export CATALINA_OPTS=”-Xms256m -Xmx512m” or
- export JAVA_OPTS=”-Xms256m -Xmx512m”
OR
- Start Menu > Programs > Apache Tomcat > Configure Tomcat
- Java tab > Set Initial memory pool AND Maximum memory pool, (64mb and 512mb seems to work)
FME Server won't start
There is an error about fmeutil_java.dll in the fmeServer_###.log
Symptom
The FME Server will attempt to start, but the FME Server core won't stay up. In the fmeServer_###.log there is the following java error:
java.lang.UnsatisfiedLinkError: C:\Program Files\FMEServer\Server\lib\fmeutil\fmeutil_java.dll: The specified procedure could not be found
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at COM.safe.fmeserver.RepositoryManager.init(RepositoryManager.java:102)
at COM.safe.fmeserver.FMEServer.init(FMEServer.java:376)
at COM.safe.fmeserver.FMEServer.main(FMEServer.java:1441)
Reason
This generally occurs when FME Desktop was installed before FME Server on the same machine. FME Server looks at the PATH environment variable, the Desktop path is first int he list, a fmeutil.dll is found in the FME Desktop installation and tries to use it, resulting in the error.
Resolution
There are a at least two options:
- Uninstall FME Desktop and/or reinstall it after FME Server is installed.
- Modify the PATH environmental variable so that any references to FME Desktop paths come after the FME Server paths.
After running running many jobs, some start failing for unknown reasons (Windows XP)
Symptom
You have an FME Server installed on a Windows XP machine and when you run a lot of jobs against it, they start to fail after a while.
Reason
In Windows XP (not Windows Server) there is a registry setting that limits the maximum number of TCP/IP connections that may be open at the same time. FME Server needs to open a TCP/IP connection for each job and eventually the maximum number is over come.
Resolution
The following shows some of the possible changes that can be made in the Windows Registry (under HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters):
- By increasing the number of available dynamic user ports from 5000 to a maximum of 65534, we have been able to avoid the connection errors previously experienced.
- Set MaxUserPort (a DWORD registry key) to the desired value (ensure you are inputting in decimal format).
- Further improvements can also be made by adjusting the maximum number of free TCP Control Blocks. By default, XP Pro has this set at 1000, while Server has it at 2000.
- Set MaxFreeTcbs (a DWORD registry key) to the desired value (ensure you are inputting in decimal format)
- Decreasing the TIME_WAIT delay interval is also possible (default is set to 240 seconds) to close sockets earlier.
- Set TcpTimedWaitDelay (a DWORD registry key) to the desired value (ensure you are inputting in decimal format)
- Speed up when to mark sockets as free by adjusting the TIME_WAIT connection limit before the freeing process occurs (default is 1000 connections).
- Set MaxFreeTWTcbs (a DWORD registry key) to the desired value (ensure you are inputting in decimal format)
If you find that you need to add/modify the majority of the settings above, it may be better just to migrate to a dedicated server OS.
