RMOUG 18 – Upgrading to 12.2 – Tales from the Battle Field

Here is my updated presentation for upgrading to 12.2.   Please contact me with any questions.




OEM12C Deployment and Mass Promotion

OEM12C Deployment and Mass Promotion

I was privileged to be a speaker at the Rocky Mountain training days #td17 on the topic of “Automagically” Deploying EM12c Agents and Mass promotion of targets once the hosts have been added.

The presentation is attached, but in a nut shell, using EM12C’s RPM generation and using Puppet we are able to push out agents, the moment the machines are built and validated.

The first step is to log into the OMS machine and generate the RPM using EMCLI (After logging into emcli )

> emcli get_supported_platforms  <== This will get the information for building the RPM’s


emcli get_agentimage_rpm -destination=/tmp -platform=”Linux x86-64″ -version=”″

This last command will generate a RPM in the /tmp directory called: “oracle-agt-”   << notice the version # and platform in the name

Now that the RPM is ready, we use Puppet to copy and to install the RPM on new machines.

A few notes about puppet installs with RPMS

  • We use a standard location / directory
  • Only run if the standard location is NOT in place
  • Only run if “emwd.pl” is not in place
  • Verifies the correct ownership of the directories
  • Uses oraInventory for location

There are 2 steps for the puppet process

  1. apply RPM – >> rpm -ivh oracle-agt-
  2. create agent.properties file – With host name, default password, etc.
  3. run – /etc/init.d/oracle-agt RESPONSE_FILE=/usr/lib/oracle/agent/agent.properties
  4. When Step 3 finishes, there should be a new OMS entry for that host and connected to the OMS.

Last step – Just added recently – (Not part of presentation) is to perform a mass promotion of non-host targets like databases, listeners, etc.


Mass Promote code is in GIT Hub – https://github.com/harry2040/OEM12cMassPromote

Thanks again to #td17 for a great conference and the opportunity.


RMAN Upgrade – ORA-00439: feature not enabled…

How many people have received letters from Oracle that have stated that partitioning and / or Fine-grained auditing were turned on for Oracle 12C and they had to remove it otherwise pay a higher license fee or penalties.   Well if you haven’t, your lucky as many customers have received this kind of letter, recently.

Well many have received the notice and then recompiled/de-installed the partitioning and/or fine grain auditing option on the databases that have contained a RMAN catalog user/schema.

When they removed the partition and/or the fine grain auditing and tried to upgrade the RMAN Catalog for the new Oracle 12c databases you will get this error:

Recovery Manager: Release - Production on Thu May 26 13:08:02 2016
Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.
RMAN> connect catalog catalog_user/PW@Database;
connected to recovery catalog database
PL/SQL package catalog_user.DBMS_BA version in RCVCAT database is too old

RMAN> upgrade catalog

recovery catalog owner is catalog_user
enter UPGRADE CATALOG command again to confirm catalog upgrade

RMAN> upgrade catalog

error creating create_deleted_object_seq
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-06004: ORACLE error from recovery catalog database: ORA-00439: feature not enabled: Partitioning
RMAN-06004: ORACLE error from recovery catalog database: ORA-00439: feature not enabled: Fine-grained access control

Well if you have received this error, I hope you have a backup of the database, as this will corrupt you RMAN catalog. Well at least it corrupted our RMAN catalog and many others.


New RMAN catalog – Oracle requires Partitioning, fine-grained auditing and other Enterprise edition options. Per Mike Dietrich’s BLOG, Going forward with Oracle 12c, you need to use Enterprise Edition for RMAN or Infrastructure databases:

“A separate single instance Oracle Database can be installed and used as an infrastructure repository for RMAN, Oracle Enterprise Manager Cloud Control, Automatic Workload Repository (AWR) Warehouse, Global Data Services Catalog, and Grid Infrastructure Management Repository without additional license requirements, provided that all the targets are correctly licensed. It may not be used or deployed for other uses.”



#1 – the RMAN catalog must be in its OWN Oracle EE database – no other applications – Partitions need to be included
#2 – DBMS_LOB and UTTL_HTTP Must be present
#3 – Execute this script before upgrading the catalog: SQL> @$ORACLE_HOME/rdbms/admin/dbmsrmansys.sql
#4 – no Partitioning license required
#5 – Make sure you have a backup
#6 – p20432873_121020_Generic.zip – is a patch to fix the corrupted RMAN catalog due to Oracle 12c requirement – basically its a different recover.bsq script to fix the corruption. – This is only if its currently corrupted.   One of the few bugs, that i was able to create an actually see a patch created for it – thank you Oracle.
#7 – See MOS Note:1927265.1 – this has more information

Please review Mike Dietrich’s Blog above regarding licenses and more information – https://blogs.oracle.com/UPGRADE/entry/rman_catalog_requires_ee_in

Hope this helps.   Again, please backup your rman catalog before upgrading .


On behalf of the Michigan Oracle User Summit (www.mous.us), I would like to personally thank all vendors, Speakers, attendee’s and volunteers for taking the time attend our annual technology conference. Your participation in the conference was the reason for the continued success of the conference and nearly 250 participants.

We have posted the presentations on our website (http://www.mous.us/presentations.html).  I would also like to extend an invitation to everyone for next year’s conference on Wednesday, November 8, 2017 – #MOUS17, (please visit our website for more information – www.mous.us or SEMOP) .   If you are interested please contact me mjgangler@gmail.com. We will be opening a call for papers in May.

Hope to see you at #MOUS17



Recently i was asked to generate a report that shows all the databases that were backed up via RMAN and they wanted a color code when a backup failed or wasn’t completed successfully.

That posed a challenge and took some research, but finally generated a report.   Below is a sample report and how to color code on afailed backup, but can be used on any sql report:

set pages 999
set feedback off
break on db_name page
alter session set nls_date_format='DD-MON-YY HH24:MI:SS';
spool backup_report
PROMPT <H2><center><b> RMAN Backup  details  </b></center></H2>
SET lines 132 pages 9999 feedback off
COLUMN start_time      format date              heading ‘Start Date’
COLUMN end_time      format date              heading ‘End Date’
COLUMN input_bytes format 999,999,999,999 heading ‘Input Bytes’
COLUMN output_bytes        format 999,999,999,999 heading ‘Output Bytes’
COLUMN cstatus format a40  heading ‘Status’
COLUMN object_type              format a10  heading ‘Backup Type’
COLUMN time_taken_display      format a40  heading ‘Time Taken’
select db_name, object_type,start_time,end_time,input_bytes, output_bytes,
when status like ‘%WARN%’ then ‘<font color=”green”>COMPLETED</font>’   <==== Note color default for non-failed jobs
when status like ‘COMPLETED’ then ‘<font color=”green”>COMPLETED</font>’
ELSE ‘ <font color=”red”>’||status||'</font>’                                                             <=== Note – falls through code and any noncompleted status uses this color
END) cstatus
from <RMAN CATALOG>.rc_rman_status
where object_type = ‘DB FULL’
and operation not in (‘RESTORE VALIDATE’)
order by db_name,start_time Asc;
spool off

Hope this helps with your reports and can be used for any type of report that requires color coding.


GLOC16 is here !!

This year there are more Oracle ACES presenting than ever before at the Great Lakes Oracle Users Conference. Check out the Schedule at a Glance and see which sessions you want to attend. Oracle Aces include Susan Behn, Karen Brownfield, Michael Gangler, Gustavo Gonzalez, Tim Gorman, Janis Griffin, Kyle Hailey, Francis Mignault, Daniel Morgan, Rich Niemiec, Kerry Osborne, Elke Phelps, Tanel Poder, Kellyn Pot’Vin-Gorman, Carlos Sierra, Scott Spendolini, Deanna Sunde, Mark Bobak, Ric Van Dyke, and Tim Vlamis.

I will be speaking on May 19th from 4 – 5:00pm on “Using RPM’s to distribute OEM Agents”

TO Register:



US Pro Rugby – premiers this weekend – AOL.com

Saturdays game was good to see, closest team for us in Midwest is Ohio…  

After an exciting opening week, PRO Rugby continues its debut season when Denver looks to win its second straight while taking on San Francisco. The much-anticipated showdown will be streamed live on AOL.com Sunday at 6 p.m. ET.

Fans can also enjoy a replay of Saturday’s game between San Diego and Sacramento.

PRO Rugby — the first team-sport professional league to launch in the United States since Major League Soccer in 1993 and Major League Lacrosse in 2001 — officially kicked off last Sunday, as Denver got by Ohio, 19-13, in overtime and Sacramento edged San Francisco 37-25 in a rivalry match.

For full rosters of each team, CLICK HERE.