After a recent outage where our OMS host was bounced, we found that OMS wouldn’t start. The WebTier started without issue, but OMS didn’t.
:oracle:/u01/app/oracle/middleware/oms/bin >./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 220.127.116.11.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
WebTier Successfully Started
Starting Oracle Management Server…
Oracle Management Server is Down
Checking the OMS log: /u01/app/oracle/middlware/gc_inst/em/EMGC_OMS1.log only provided me what I already knew
2014-03-05 10:26:56,571 [main] DEBUG oms.StatusOMSCmd processStatusOMS.239 – console page status code is 404
I hen checked the EMGC_OMS1.out log: /u01/app/oracle/middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
Which provided me with the following notice. Apparently the sysman password was changed in the repository, but not in the credential store.
Repos details fetched from credstore
Fetched repository credentials from Credential Store
Invalid Connection Pool. ERROR = User credentials doesn’t match the existing ones
Failed to verify repository
We can change the sysman password in the repository by using emctl.
/u01/app/oracle/middlware/oms/bin/emctl config oms -store_repos_details -repos_host <host> -repos_port 1521 -repos_sid <SID> -repos_user SYSMAN -repos_pwd xxxxx
* Note replace the XXXXX after -repos_pwd with the actual password for your system, as well as the <host> with your host.
At this point I received an error message that I didn’t capture concerning the weblogic password being incorrect. The next step is to change the weblogic password.
. setDomainEnv.sh — this will set the environment variables
2. cd /u01/app/oracle/middleware/gc_inst/user_projects/domains/GCDomain/security
3. cp DefaultAuthenticatorInit.ldift DefaultAuthenticatorInit.ldift.20140305 — this file will be changed
4. java weblogic.security.utils.AdminAccount newAdmin newPassword .
* note there is a DOT at the end of the command line. Ensure that the JAVA_HOME and the CLASSPATH are set correctly
5. cd /u01/app/oracle/middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER
6. mv data to data_yyyymmdd
7. cd security
8. Modify the boot.properties file to have only these two lines:
Now I receive
<Server failed to bind to the configured Admin port. The port may already be used by another pr ocess.>
So there apparently are already processes running that our bound to the same port. A ps -ef|grep weblogic shows me the processes. I kill the processes and run the start again this time successful.
Once the server comes up return to the change password for the SYSMAN and the OMS and this time it should be
And lastly I start oms ./emctl start oms
It takes a while and I was bombarded by alerts since oms was down for a few days but it did in fact startup without issues.
Still need to investigate why the the passwords appeared to have changed since last startup.