Title:
How to Resolve Fatal Error due to missing system mandatory properties for server startup
Fact:
Interstage BPM (Business Process Manager) V11.4.1 Build IF1141013
Fact:
Oracle 12c
Fact:
JBoss EAP (Enterprise Application Platform) V6.4.9 GA
Fact:
Windows Server 2012
Fact:
JDK 1.8.0_192
Symptom:
JBoss failed to start due to the following fatal errors found in server.log:
18:47:42,464 INFO [org.jboss.web] (ServerService Thread Pool -- 77) JBAS018210: Register web context: /fujitsu-ibpm-startup-webapp
18:47:43,964 INFO [stdout] (ServerService Thread Pool -- 77) [Dec 13 2018 - 18:47:43.961(001)] WARN : SYSTEM: TID=838425121: Some fatal error has occurred, server will be shutdown now. {Following system mandatory properties are missing for server startup ''ServerHost','ServerRoot','ServerSharedRoot','JMSNamingProviderURL''}
18:47:43,964 INFO [stdout] (ServerService Thread Pool -- 77) Some fatal error has occurred, server will be shutdown now. {Following system mandatory properties are missing for server startup ''ServerHost','ServerRoot','ServerSharedRoot','JMSNamingProviderURL''}
18:47:43,965 INFO [stdout] (ServerService Thread Pool -- 77) at com.fujitsu.iflow.commonee.SystemEnvironment.initialize(SystemEnvironment.java:144)
18:47:43,965 INFO [stdout] (ServerService Thread Pool -- 77) at com.fujitsu.iflow.serveree.impl.common.IflowStartup.startTimer(IflowStartup.java:107)
18:47:43,966 INFO [stdout] (ServerService Thread Pool -- 77) at com.fujitsu.iflow.startup.IflowStartupServlet.init(IflowStartupServlet.java:23)
18:47:43,966 INFO [stdout] (ServerService Thread Pool -- 77) at javax.servlet.GenericServlet.init(GenericServlet.java:242)
18:47:43,966 INFO [stdout] (ServerService Thread Pool -- 77) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1206)
18:47:43,967 INFO [stdout] (ServerService Thread Pool -- 77) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1112)
18:47:43,967 INFO [stdout] (ServerService Thread Pool -- 77) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3593)
18:47:43,967 INFO [stdout] (ServerService Thread Pool -- 77) at org.apache.catalina.core.StandardContext.start(StandardContext.java:3802)
18:47:43,968 INFO [stdout] (ServerService Thread Pool -- 77) at org.jboss.as.web.deployment.WebDeploymentService.doStart(WebDeploymentService.java:163)
18:47:43,968 INFO [stdout] (ServerService Thread Pool -- 77) at org.jboss.as.web.deployment.WebDeploymentService.access$000(WebDeploymentService.java:61)
18:47:43,968 INFO [stdout] (ServerService Thread Pool -- 77) at org.jboss.as.web.deployment.WebDeploymentService$1.run(WebDeploymentService.java:96)
18:47:43,969 INFO [stdout] (ServerService Thread Pool -- 77) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
18:47:43,969 INFO [stdout] (ServerService Thread Pool -- 77) at java.util.concurrent.FutureTask.run(FutureTask.java:266)
18:47:43,969 INFO [stdout] (ServerService Thread Pool -- 77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
18:47:43,970 INFO [stdout] (ServerService Thread Pool -- 77) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
18:47:43,970 INFO [stdout] (ServerService Thread Pool -- 77) at java.lang.Thread.run(Thread.java:748)
18:47:43,970 INFO [stdout] (ServerService Thread Pool -- 77) at org.jboss.threads.JBossThread.run(JBossThread.java:122)
18:47:43,971 INFO [stdout] (ServerService Thread Pool -- 77) Caused by: Following system mandatory properties are missing for server startup ''ServerHost','ServerRoot','ServerSharedRoot','JMSNamingProviderURL''
18:47:43,971 INFO [stdout] (ServerService Thread Pool -- 77) at com.fujitsu.iflow.common.BaseSystemEnvironment.checkIfMandatoryPropertyPresent(BaseSystemEnvironment.java:2009)
18:47:43,972 INFO [stdout] (ServerService Thread Pool -- 77) at com.fujitsu.iflow.common.BaseSystemEnvironment.validateMandatoryProperties(BaseSystemEnvironment.java:1954)
18:47:43,972 INFO [stdout] (ServerService Thread Pool -- 77) at com.fujitsu.iflow.common.BaseSystemEnvironment.checkMandatoryProperties(BaseSystemEnvironment.java:1937)
18:47:43,972 INFO [stdout] (ServerService Thread Pool -- 77) at com.fujitsu.iflow.commonee.SystemEnvironment.initialize(SystemEnvironment.java:87)
18:47:43,972 INFO [stdout] (ServerService Thread Pool -- 77) ... 16 more
18:47:43,973 INFO [stdout] (ServerService Thread Pool -- 77)
18:47:44,033 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.deployment.subunit."fujitsu-ibpm-engine.ear"."fujitsu-ibpm-engine-ejb.jar".INSTALL: org.jboss.msc.service.StartException in service jboss.deployment.subunit."fujitsu-ibpm-engine.ear"."fujitsu-ibpm-engine-ejb.jar".INSTALL: JBAS018733: Failed to process phase INSTALL of subdeployment "fujitsu-ibpm-engine-ejb.jar" of deployment "fujitsu-ibpm-engine.ear"
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:166) [jboss-as-server-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS011030: Could not configure component ProcessInstance
at org.jboss.as.ee.component.deployers.EEModuleConfigurationProcessor.deploy(EEModuleConfigurationProcessor.java:96)
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159) [jboss-as-server-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
... 5 more
Caused by: java.lang.IllegalStateException: Container is down
at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:691) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:243) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2433) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:243) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2433) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:345) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
at org.jboss.as.ejb3.security.EJBSecurityViewConfigurator.configure(EJBSecurityViewConfigurator.java:160)
at org.jboss.as.ee.component.DefaultComponentViewConfigurator.configure(DefaultComponentViewConfigurator.java:68)
at org.jboss.as.ee.component.deployers.EEModuleConfigurationProcessor.deploy(EEModuleConfigurationProcessor.java:82)
... 6 more
Fix:
The issue occurred during reverse lookup of IP to hostname at BPM startup.
Step 1:
Check IBPMPROPERTIES for the four missing system properties by executing sql query:
Execute sql query to see the key for the four system properties:
select * from IBPMPROPERTIES WHERE tenantid=-1;
'ServerHost','ServerRoot','ServerSharedRoot','JMSNamingProviderURL'
Refer to BPM Administration Guide of the four System parameters.
They must exist in IBPMPROPERTIES table in the PROPERTYKEY column in the following format for single server setup.
Cluster will be different. Refer to admin guide for cluster format:
ServerHost.{HOSTNAME}
ServerRoot.{HOSTNAME}
ServerSharedRoot.{HOSTNAME}
JMSNamingProviderURL.{HOSTNAME}
Step 2:
Check hosts file, typically found in C:\Windows\System32\drivers\etc\hosts for Windows or /etc/hosts Linux.
Make sure there is only one valid entry of IP to mapping.
This symptom can be reproduced if the following were found in hosts file:
192.168.217.132 interstagedemo2 #invalid alias which doesn't exist
192.168.217.132 interstagedemo #intended valid entry
ping -a 192.168.217.132 should return the intended hostname found in step 1.
To fix the issue, simply remove the invalid entry or correct the mapping in hosts file.