OEM12C Deployment and Mass Promotion

OEM12C Deployment and Mass Promotion

I was privileged to be a speaker at the Rocky Mountain training days #td17 on the topic of “Automagically” Deploying EM12c Agents and Mass promotion of targets once the hosts have been added.

The presentation is attached, but in a nut shell, using EM12C’s RPM generation and using Puppet we are able to push out agents, the moment the machines are built and validated.

The first step is to log into the OMS machine and generate the RPM using EMCLI (After logging into emcli )

> emcli get_supported_platforms  <== This will get the information for building the RPM’s


emcli get_agentimage_rpm -destination=/tmp -platform=”Linux x86-64″ -version=”″

This last command will generate a RPM in the /tmp directory called: “oracle-agt-”   << notice the version # and platform in the name

Now that the RPM is ready, we use Puppet to copy and to install the RPM on new machines.

A few notes about puppet installs with RPMS

  • We use a standard location / directory
  • Only run if the standard location is NOT in place
  • Only run if “emwd.pl” is not in place
  • Verifies the correct ownership of the directories
  • Uses oraInventory for location

There are 2 steps for the puppet process

  1. apply RPM – >> rpm -ivh oracle-agt-
  2. create agent.properties file – With host name, default password, etc.
  3. run – /etc/init.d/oracle-agt RESPONSE_FILE=/usr/lib/oracle/agent/agent.properties
  4. When Step 3 finishes, there should be a new OMS entry for that host and connected to the OMS.

Last step – Just added recently – (Not part of presentation) is to perform a mass promotion of non-host targets like databases, listeners, etc.


Mass Promote code is in GIT Hub – https://github.com/harry2040/OEM12cMassPromote

Thanks again to #td17 for a great conference and the opportunity.



Recently i was asked to generate a report that shows all the databases that were backed up via RMAN and they wanted a color code when a backup failed or wasn’t completed successfully.

That posed a challenge and took some research, but finally generated a report.   Below is a sample report and how to color code on afailed backup, but can be used on any sql report:

set pages 999
set feedback off
break on db_name page
alter session set nls_date_format='DD-MON-YY HH24:MI:SS';
spool backup_report
PROMPT <H2><center><b> RMAN Backup  details  </b></center></H2>
SET lines 132 pages 9999 feedback off
COLUMN start_time      format date              heading ‘Start Date’
COLUMN end_time      format date              heading ‘End Date’
COLUMN input_bytes format 999,999,999,999 heading ‘Input Bytes’
COLUMN output_bytes        format 999,999,999,999 heading ‘Output Bytes’
COLUMN cstatus format a40  heading ‘Status’
COLUMN object_type              format a10  heading ‘Backup Type’
COLUMN time_taken_display      format a40  heading ‘Time Taken’
select db_name, object_type,start_time,end_time,input_bytes, output_bytes,
when status like ‘%WARN%’ then ‘<font color=”green”>COMPLETED</font>’   <==== Note color default for non-failed jobs
when status like ‘COMPLETED’ then ‘<font color=”green”>COMPLETED</font>’
ELSE ‘ <font color=”red”>’||status||'</font>’                                                             <=== Note – falls through code and any noncompleted status uses this color
END) cstatus
from <RMAN CATALOG>.rc_rman_status
where object_type = ‘DB FULL’
and operation not in (‘RESTORE VALIDATE’)
order by db_name,start_time Asc;
spool off

Hope this helps with your reports and can be used for any type of report that requires color coding.


GLOC16 is here !!

This year there are more Oracle ACES presenting than ever before at the Great Lakes Oracle Users Conference. Check out the Schedule at a Glance and see which sessions you want to attend. Oracle Aces include Susan Behn, Karen Brownfield, Michael Gangler, Gustavo Gonzalez, Tim Gorman, Janis Griffin, Kyle Hailey, Francis Mignault, Daniel Morgan, Rich Niemiec, Kerry Osborne, Elke Phelps, Tanel Poder, Kellyn Pot’Vin-Gorman, Carlos Sierra, Scott Spendolini, Deanna Sunde, Mark Bobak, Ric Van Dyke, and Tim Vlamis.

I will be speaking on May 19th from 4 – 5:00pm on “Using RPM’s to distribute OEM Agents”

TO Register:



Granting Oracle Schema Permissions (Objects not created yet) – Part 2

After presenting at Collaborate 16 about granting schema permissions, a colleague – Frank Pound send me the following that can also be used to perform the trigger only on “Tables”:


create or replace trigger trigger_grant_dml

after create on schema
v_job number;
v_todo varchar2(200);
     if ora_sysevent = 'CREATE' and ora_dict_obj_type = 'TABLE' then
     v_todo:='execute immediate ''grant select, insert, update, delete on '||ora_dict_obj_name||' to my_new_role'';';
     dbms_job.submit(job=>v_job, what=>v_todo);
  end if;
  when others then


Please contact me with any questions - mjgangler@gmail.com

Collaborate 16 !

I will be presenting 3 presentations at this years Collaborate 16 in Las Vegas.

Monday – April 11 – 12 – 12:30 – Palm B – Using Puppet and RPM’s to deploy EM12C Agents

Wednesday – April 13 – – 12:00 – 12:30 – Jasmine B – Granting Oracle Schema Permissions When Objects not Created Yet !

Thursday – April 14 – 8:30 am – 9:30 am – Palm  D – Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP






MOUS15 is here !!!

If you are in the Indiana, Michigan, Illinois, Ohio or Canada region and want to see and experience great Oracle user and technical presentations, networking and hands-on workshops, please register to the Michigan Oracle User Summit (#MOUS15), on Wednesday, November 18th at Schoolcraft college in Livonia, Michigan (www.mous.us).    Its a full day of presentations (45) that include Big Data presentations, Hyperion presentations, JDEdwards, Oracle Applications, Middleware (SOA), Database (Oracle12c, RAC, others) and 2 hands-on workshops – #cloneattack and #racattack.   We also have 13 Oracle Ace and Ace directors, Rich Niemiec of Rolta will be our keynote speaker.   Please visit our website (www.mous.us) or guidebook for more details on the conference content.

So not only do you get some great user and technical presentations, but you get 2 great hands-on workshops – Clone Attack and Racattack (www.racattack.org).

Clone attack (www.delphix.com ) – did you ever want to create development databases from production with a small footprint and dynamic way ?   Well come to MOUS15 and create many development databases on your own PC.

RACATTACK – (www.racattack.org) – Racattack allows you to create a 12c RAC database on your own PC, yes, you too can have your own RAC database on your pc.

Wait there is more !! – WE ARE ON GUIDEBOOK – http://www.guidebook.com (Look for MOUS) its free !!

Lastly, this conference is a great way to network with other Oracle users, a full day of presentations, hands-on workshops, great food , education and its priced reasonably ($50 pre-register, $75 at door).

Hope to see you at the conference and please contact me with any questions,

--Mike Gangler

ORA-10997 and ORA-09968 on ASM startup (With NFS)

During a recent hardware upgrade, i received the following error on a ASM (NO-RAC) startup.   It was very confusing, because i was custom to seeing ASM on Storage LUNS and in this case it was using ASM (NO-RAC) and NFS storage.   Unfortunately, as always with late night changes, you don’t notice things until you get fresh eyes on the issue.   A simple “df -kh” would’ve told me that it was NFS, but made too many presumptions.

When i first told about the NFS settings, i thought i needed to restart nfs, but that didn’t work.   Than after getting another fresh pair of eyes from the Linux team, we realized that the directory “/var/lib/nfs” was set to 750???   After some investigation, we saw that puppet was changing it to 750 every 5 minutes, so even when we changed it to 755, it was 750.  so the next question, why ?

Well it appears that FTI requirements are the culprit.  Apparently the auditors don’t understand nfs and O/S’s, probably don’t care either.   Apparently, FTI companies are required to set this to 750…. Now i understand security, but this directory needs to be readable by Oracle to start up the nfs drives and ASM.

Until we get a permanent solution, the linux team offered the following solution:

the setting of 755 is only needed for startup, so the temporary solution is (as root):

root> chmod 755 /var/lib/nfs
root> chatter +i /var/lib/nfs <== protects
root> crsctl start has
== after successful startup ====
root> chatter -i /var/lib/nfs <== takes off the protection

The following is the error description:

grid> srvctl start asm

PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-10997: another startup/shutdown operation of this instance in progress
ORA-09968: unable to lock file
Linux-x86_64 Error: 37: No locks available
Additional information: 64
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/product/<machine_name>/agent/ohasd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.asm' on '<machine name>' failed