EMCTL Script Empty

After several hours of trying to determine why when I ran ./emctl start oms absolute nothing happened.  I wasn’t even receiving an error message.  I set JAVA_HOME, OMS_HOME, ORACLE_HOME and yet nothing.  I checked ./emctl start agent and that worked without any issue, but not the emctl under the OMS home.  I was perplexed.  And then I saw metalink note How to Recreate the emctl Script for the Enterprise Manager OMS (Doc ID 1404623.1).  My file was indeed 0 bytes.  The note describes how to recreate the file.  I’ve since made a backup of the emctl for future reference.  Hopefully when I google “emctl start oms returns no information” this post will serve as a reminder.

Cluster Database and EM12c

Originally when we configured our cluster database on EM12c we used IP addresses simply because DNS wasn’t reliable in the sense setup.  Instead of fighting the battle we used IP addresses.  Which works provided you never change your IP addresses.  Three years later that’s exactly what we did.

During the configuration of the cluster database on EM12c, I realized I completely forgot how to make this happen.  So this is really a note to myself for the next time, (side note, DNS is internal and stable so we used the host names) I may have to re-configure the cluster database within EM12c.

The first step of course is to install the agent on all cluster host.  This is no different than any other host.

The second step adding the cluster database and high available is done through the manually add target.  Remember to select cluster database and not database instance.  The instances can’t be added until the cluster database.

After the configurations were complete, status remained pending for the cluster database and the targets all appeared down.  What I discovered within the configuration EM12c used the listener machine name as the host for connection string purposes, but that didn’t occur to me until I set specific connection string for both of the instances within the cluster database.  All that was required, change the listener machine name.

Why the listener machine name didn’t work?  It was the VIP fully qualified.  It should, but sadly the VIP fully qualified is not listed in the DNS nor the host file.  I don’t have access to either, and the length of time to have this resolved put me on a different path.  I changed the listener machine name to the host name.  Success.

I didn’t just leave it there, though I put in the proper work order to have the VIPs added to the DNS which is estimated at about 20 business days.  The work around provides us with EM working for the cluster database while we wait.

Standby Redo Log Issues After Create of Standby

We ended up recreating our standby’s at the DR site unrelated to any Oracle issue which is always good.  However, when the duplication finished the standby redo logs were not available.

RFS[12]: No standby redo logfiles available for thread 1

Our first step was checking to see if the standby logs existed and whether they were being used by checking the v$standby_log view.  The status showed unassigned and the no SCN was listed.

Then I found the following from John Hallas that explained different in sizes between the primary and secondary can be a cause:

http://jhdba.wordpress.com/2011/02/28/incorrectly-sized-standby-redo-logs/

Comparing the sizes they were different sizes 200mb secondary for 600mb on primary.

Time to recreate the the standby redo logs with the correct size matching the primary.  I pull the standby redo log scripts dynamically on the primary to drop and then recreate the log files.  Again John Hallas does a really good job providing this information.

http://jhdba.wordpress.com/2011/02/28/scripts-to-resize-standby-redolog-files/

 

Lack of Exception Handling Yet Again

What’s a good method of getting those deep layered exceptions to propagate for effective troubleshooting? My personal favorite is dbms_utility.format_error_backtrace . Even if the code doesn’t have exception handling I will at times add this to the code in order to effectively troubleshoot. Easier of course in non-production systems, but the same can be done in production. If someone of course, is monitoring recompiled code or creation of objects this might get flagged. I don’t recommend it for high executed code in production without of course going through your change control process. Could end up being nasty if you’re not careful.

Simply add dbms_output.put_line(dbms_utility.format_error_backtrace) to your exception handling:


CREATE OR REPLACE PROCEDURE third
IS
BEGIN
DBMS_OUTPUT.put_LINE('I'm the first');
first;
EXCEPTION
WHEN OTHERS
THEN
DBMS_OUTPUT.put_line(DBMS_UTILITY.FORMAT_ERROR_STACK);
DBMS_OUTPUT.put_line(dbms_utility.format_error_backtrace);
END;

I used both the FORMAT_ERROR_STACK and FORMAT_ERROR_BACKTRACE backtrace doesn’t include the error message. alternatively SQLERRM could be used as well. Depending on how deep into the call stack the error is being raised it may sever you will to substr the error message. Maybe you do only need the ORA error number and line number.

Remember if using multiple exceptions you will need to re-raise the exception to the outer most exception.

A developer can make friends with the DBA staff by including well-defined exception handling within their code. And if they don’t we can guide them to the light. When all else fails we can add in the needed lines for troubleshooting. I find this so much easier than trying to debug in production. Just because you can doesn’t mean that you should.

Linux OS Commands To Remember

Remove files older than a certain date:

find /path/* -mtime +180 -exec rm {} \;

include the whole path.  Substitue the +180 is the number of days.  There is a space after rm and after {}.

If you receive list argument too long error try the following instead:

cd /path to files/

find *.ext -mtime +180 -print0|xargs -0 rm -f