Oracle EBS 12.2 + 19c RAC Clone Using RMAN Duplicate — Complete Runbook

1. Overview and Architecture

This runbook covers the end-to-end procedure to clone an Oracle E-Business Suite 12.2 environment backed by a 2-node Oracle 19c Real Application Clusters (RAC) database using RMAN Duplicate within the same datacenter. It also covers the process of adding a new RAC node to an existing cluster.

1.1 Scope

• Source: 2-node Oracle 19c RAC (CDB with EBS PDB), ASM storage, EBS 12.2 application tier (2 app nodes)
• Target: New 2-node RAC cluster + 2 new EBS app tier nodes — all within the same datacenter
• Database clone method: RMAN Duplicate (Active Database Duplicate — no backup required)
• EBS clone method: Rapid Clone (adcfgclone.pl) after DB duplicate
• New node addition: Oracle Clusterware + Grid Infrastructure extension to a third/fourth node

1.2 Environment Reference

2. Pre-Clone Checklist

2.1 Hardware and OS Requirements

• Identical CPU architecture (x86_64)
• Same OS version and patch level as source nodes
• Sufficient RAM: minimum 32GB per DB node (same as source)
• ASM disk groups provisioned and visible on target nodes
• Network: Public, Private (interconnect), and SCAN IPs configured
• SSH equivalence set up between all source and target nodes (oracle and grid users)
• NTP synchronized across all nodes

2.2 Oracle Software Prerequisites

• Oracle Grid Infrastructure 19c installed and configured on target cluster (clusterware up, SCAN listener running)
• Oracle Database 19c software installed on target nodes (ORACLE_HOME same path as source)
• ASM disk groups created on target: +DATA, +REDO, +FRA (same names as source, size ≥ source usage)
• Oracle Inventory consistent across all nodes

2.3 Network Prerequisites

2.4 Source Database Checks

Step	Task	Command / Action	Run As
1	Check DB open mode	SELECT name, open_mode, db_unique_name FROM v$database;	oracle/sqlplus
2	Check CDB/PDB status	SELECT name, open_mode FROM v$pdbs;	oracle/sqlplus
3	Check archive log mode	ARCHIVE LOG LIST;	oracle/sqlplus
4	Check FRA space	SELECT * FROM v$recovery_file_dest;	oracle/sqlplus
5	Check RMAN config	SHOW ALL;	rman target /
6	Verify ASM disk groups	SELECT name, state, total_mb, free_mb FROM v$asm_diskgroup;	grid/sqlplus
7	Check active sessions on source	SELECT count(*) FROM v$session WHERE status='ACTIVE';	oracle/sqlplus

3. Target Cluster Preparation

3.1 Grid Infrastructure Verification

Step	Task	Command / Action	Run As
1	Verify clusterware status	crsctl stat res -t	grid
2	Check SCAN listener	srvctl status scan_listener	grid
3	Verify ASM disk groups	asmcmd lsdg	grid
4	Check OCR and Voting Disk	ocrcheck crsctl query css votedisk	root
5	Verify interconnect	oifcfg getif	grid

3.2 ASM Disk Group Preparation

# On clone-rac1 as grid:
asmcmd
ASMCMD> lsdg
ASMCMD> mkdir +DATA/CLNECDB
ASMCMD> mkdir +REDO/CLNECDB
ASMCMD> mkdir +FRA/CLNECDB
ASMCMD> exit

3.3 Oracle Password File on Target

# On clone-rac1 as oracle:
orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB1 password=<sys_password> entries=10

# On clone-rac2 as oracle:
orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB2 password=<sys_password> entries=10

3.4 Static Listener Configuration on Target

Create/update $TNS_ADMIN/listener.ora on both target nodes:

LISTENER_CLNECDB =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = clone-rac1-vip)(PORT = 1521))))

SID_LIST_LISTENER_CLNECDB =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = CLNECDB)
      (ORACLE_HOME = /u01/app/oracle/product/19.0.0/dbhome_1)
      (SID_NAME = CLNECDB1)))

# Start the listener:
lsnrctl start LISTENER_CLNECDB

3.5 Auxiliary Init Parameter File

Create /tmp/init_aux.ora on clone-rac1:

db_name=CLNECDB
db_unique_name=CLNECDB
enable_pluggable_database=TRUE
sga_target=8G
pga_aggregate_target=2G
control_files='+DATA/CLNECDB/control01.ctl','+REDO/CLNECDB/control02.ctl'
db_file_name_convert='+DATA/PRODCDB','+DATA/CLNECDB','+REDO/PRODCDB','+REDO/CLNECDB'
log_file_name_convert='+REDO/PRODCDB','+REDO/CLNECDB'
db_recovery_file_dest='+FRA'
db_recovery_file_dest_size=500G
audit_file_dest='/u01/app/oracle/admin/CLNECDB/adump'
diagnostic_dest='/u01/app/oracle'
cluster_database=FALSE
instance_number=1
thread=1

4. RMAN Active Database Duplicate

4.1 Start Auxiliary Instance

# On clone-rac1 as oracle:
export ORACLE_SID=CLNECDB1
mkdir -p /u01/app/oracle/admin/CLNECDB/adump
sqlplus / as sysdba
SQL> STARTUP NOMOUNT PFILE='/tmp/init_aux.ora';
SQL> EXIT;

4.2 Configure RMAN Channels and Start Duplicate

Run the following RMAN script from clone-rac1 as oracle. This performs an Active Database Duplicate — no backup needed, data streams directly from source to target:

rman target sys/<sys_pwd>@PRODCDB auxiliary sys/<sys_pwd>@CLNECDB_AUX

Then at the RMAN prompt:

DUPLICATE TARGET DATABASE TO CLNECDB
  FROM ACTIVE DATABASE
  USING COMPRESSED BACKUPSET
  NOFILENAMECHECK
  SPFILE
    PARAMETER_VALUE_CONVERT 'PRODCDB','CLNECDB',
                            '+DATA/PRODCDB','+DATA/CLNECDB',
                            '+REDO/PRODCDB','+REDO/CLNECDB'
    SET db_unique_name='CLNECDB'
    SET db_recovery_file_dest='+FRA'
    SET db_recovery_file_dest_size='500G'
    SET cluster_database='FALSE'
    SET instance_number='1'
    SET thread='1'
  ALLOCATE AUXILIARY CHANNEL c1 DEVICE TYPE DISK
  ALLOCATE AUXILIARY CHANNEL c2 DEVICE TYPE DISK
  ALLOCATE AUXILIARY CHANNEL c3 DEVICE TYPE DISK
  ALLOCATE AUXILIARY CHANNEL c4 DEVICE TYPE DISK;

4.3 Post-Duplicate Verification

Step	Task	Command / Action	Run As
1	Verify DB is open	SELECT name, open_mode FROM v$database;	oracle/sqlplus
2	Verify PDB status	SELECT name, open_mode FROM v$pdbs;	oracle/sqlplus
3	Open EBS PDB	ALTER PLUGGABLE DATABASE CLNEPDB OPEN;	oracle/sqlplus
4	Check datafiles	SELECT name FROM v$datafile;	oracle/sqlplus
5	Verify redo logs	SELECT group#, status FROM v$log;	oracle/sqlplus
6	Check temp tablespace	SELECT name FROM v$tempfile;	oracle/sqlplus

5. Convert Single Instance to 2-Node RAC

5.1 Create SPFILE in ASM

# On clone-rac1 as oracle:
sqlplus / as sysdba
SQL> CREATE SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora' FROM MEMORY;
SQL> SHUTDOWN IMMEDIATE;
SQL> EXIT;

5.2 Create RAC-Enabled PFILE

Create /tmp/init_rac.ora on clone-rac1:

SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora'

5.3 Modify SPFILE for RAC

sqlplus / as sysdba
SQL> STARTUP PFILE='/tmp/init_rac.ora';
SQL> ALTER SYSTEM SET cluster_database=TRUE SCOPE=SPFILE;
SQL> ALTER SYSTEM SET instance_number=1 SID='CLNECDB1' SCOPE=SPFILE;
SQL> ALTER SYSTEM SET instance_number=2 SID='CLNECDB2' SCOPE=SPFILE;
SQL> ALTER SYSTEM SET thread=1 SID='CLNECDB1' SCOPE=SPFILE;
SQL> ALTER SYSTEM SET thread=2 SID='CLNECDB2' SCOPE=SPFILE;
SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS1' SID='CLNECDB1' SCOPE=SPFILE;
SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS2' SID='CLNECDB2' SCOPE=SPFILE;
SQL> SHUTDOWN IMMEDIATE;
SQL> EXIT;

5.4 Create Undo Tablespace for Node 2

sqlplus / as sysdba
SQL> STARTUP;
SQL> CREATE UNDO TABLESPACE UNDOTBS2
  DATAFILE '+DATA/CLNECDB/undotbs2_01.dbf' SIZE 2G AUTOEXTEND ON;
SQL> SHUTDOWN IMMEDIATE;
SQL> EXIT;

5.5 Enable Redo Thread for Node 2

sqlplus / as sysdba
SQL> STARTUP MOUNT;
SQL> ALTER DATABASE ADD LOGFILE THREAD 2
  GROUP 4 ('+REDO/CLNECDB/redo04a.log','+DATA/CLNECDB/redo04b.log') SIZE 512M,
  GROUP 5 ('+REDO/CLNECDB/redo05a.log','+DATA/CLNECDB/redo05b.log') SIZE 512M,
  GROUP 6 ('+REDO/CLNECDB/redo06a.log','+DATA/CLNECDB/redo06b.log') SIZE 512M;
SQL> ALTER DATABASE ENABLE PUBLIC THREAD 2;
SQL> ALTER DATABASE OPEN;
SQL> EXIT;

5.6 Add Database to Clusterware (srvctl)

# On clone-rac1 as oracle:
srvctl add database -d CLNECDB \
  -o $ORACLE_HOME \
  -p +DATA/CLNECDB/spfileCLNECDB.ora \
  -r PRIMARY \
  -s OPEN \
  -t IMMEDIATE \
  -i CLNECDB1,CLNECDB2

srvctl add instance -d CLNECDB -i CLNECDB1 -n clone-rac1
srvctl add instance -d CLNECDB -i CLNECDB2 -n clone-rac2
srvctl start database -d CLNECDB
srvctl status database -d CLNECDB

5.7 Register PDB with Clusterware

srvctl add service -d CLNECDB -s EBSPDB_SERVICE \
  -pdb CLNEPDB \
  -preferred CLNECDB1,CLNECDB2 \
  -failovertype SELECT \
  -failovermethod BASIC \
  -failoverretry 5 \
  -failoverdelay 3

srvctl start service -d CLNECDB -s EBSPDB_SERVICE
srvctl status service -d CLNECDB -s EBSPDB_SERVICE

6. EBS 12.2 Application Tier Clone

6.1 Prerequisites — Database Side

Step	Task	Command / Action	Run As
1	Connect to EBS PDB	ALTER SESSION SET CONTAINER=CLNEPDB;	oracle/sqlplus
2	Unlock APPS schema (if locked)	ALTER USER APPS ACCOUNT UNLOCK;	oracle/sqlplus
3	Reset APPS password	EXEC APPS.FND_USER_PKG.CHANGE_USER_NAME('APPS','APPS');	oracle/sqlplus
4	Verify APPLSYS	SELECT username, account_status FROM dba_users WHERE username='APPLSYS';	oracle/sqlplus
5	Check concurrent manager tables	SELECT COUNT(*) FROM apps.fnd_concurrent_queues;	oracle/sqlplus

6.2 Source Application Tier — Pre-Clone Steps

Run on prod-app1 as applmgr:

# Source the EBS environment
source /u01/app/ebsapps/EBSapps.env run

# Prepare source for cloning
perl $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier

6.3 Copy Application Tier to Target Nodes

From prod-app1, copy the entire EBS base to both target app nodes:

# Copy filesystem to clone-app1
tar czf - /u01/app/ebsapps | ssh clone-app1 "tar xzf - -C /"

# Copy filesystem to clone-app2
tar czf - /u01/app/ebsapps | ssh clone-app2 "tar xzf - -C /"

6.4 Configure Application Tier on clone-app1

Run on clone-app1 as applmgr:

cd $COMMON_TOP/clone/bin
perl adcfgclone.pl appsTier

When prompted, provide:

6.5 Configure Application Tier on clone-app2

Run on clone-app2 as applmgr:

cd $COMMON_TOP/clone/bin
perl adcfgclone.pl appsTier

Provide the same parameters as clone-app1 but with clone-app2 as the hostname.

6.6 Post-Clone EBS Configuration

Step	Task	Command / Action	Run As
1	Source env on clone-app1	source /u01/app/ebsapps/EBSapps.env run	applmgr
2	Start Apache/OHS	adapcctl.sh start	applmgr
3	Start OPMN	adopmnctl.sh start	applmgr
4	Start Node Manager	adnodemgrctl.sh start	applmgr
5	Start WLS Admin Server	adstrtal.sh	applmgr
6	Start all Managed Servers	adadminsrvctl.sh start all	applmgr
7	Start Concurrent Manager	adcmctl.sh start apps/<apps_pwd>	applmgr
8	Verify EBS URL	http://clone-app1.domain.com:8000/OA_HTML/AppsLocalLogin.jsp	Browser

7. Post-Clone Tasks

7.1 EBS Database Profile Updates

-- Connect to EBS PDB:
ALTER SESSION SET CONTAINER=CLNEPDB;

-- Update application node hostname in FND_NODES
UPDATE apps.fnd_nodes SET node_name='clone-app1'
  WHERE node_name='prod-app1';
UPDATE apps.fnd_nodes SET node_name='clone-app2'
  WHERE node_name='prod-app2';
COMMIT;

-- Update server host in WF_AGENTS
UPDATE apps.wf_agents SET address=REPLACE(address,'prod-app1','clone-app1')
  WHERE address LIKE '%prod-app1%';
COMMIT;

7.2 Disable/Modify Email Notifications

-- Disable all concurrent program email notifications on clone
UPDATE apps.fnd_svc_components
  SET component_status='STOPPED'
  WHERE component_type='WF_MAILER';
COMMIT;

7.3 Autoconfig

Run Autoconfig on both app nodes and the DB node to synchronize configuration:

# On clone-app1 and clone-app2 as applmgr:
perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

# On clone-rac1 as oracle (DB tier autoconfig):
perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

7.4 ETCC Verification

Run the EBS Technology Codelevel Checker to confirm patch levels:

perl $AD_TOP/patch/115/bin/adpatch.pl options='nocopypfile' \
  patchtop=$PATCH_TOP appspass=$APPS_PWD \
  workers=8 logfile=etcc.log

7.5 Verify ADOP Cycle is Clean

-- Check for stuck ADOP sessions:
SELECT session_id, prepare_status, apply_status,
       finalize_status, cutover_status, cleanup_status
FROM apps.ad_adop_sessions
WHERE session_id = (SELECT MAX(session_id) FROM apps.ad_adop_sessions);

8. Adding a New Node to Existing RAC Cluster

8.1 Overview

This section covers extending an existing 2-node RAC cluster by adding a third node (clone-rac3). The same process applies to adding a fourth node.

8.2 Prepare the New Node

Step	Task	Command / Action	Run As
1	Install same OS version	Match patch level with existing nodes	root
2	Configure kernel parameters	/etc/sysctl.conf — match existing nodes	root
3	Create OS users/groups	oracle, grid, oinstall, dba, asmadmin etc.	root
4	Configure /etc/hosts	Add all node hostnames + VIPs + SCAN	root
5	Set up SSH equivalence	ssh-keygen; ssh-copy-id to all existing nodes	oracle/grid
6	Verify ASM disks visible	ls /dev/oracleasm/disks/	root
7	Configure NTP	Same NTP server as other nodes	root

8.3 Extend Grid Infrastructure to New Node

Run from an existing node (clone-rac1) as grid:

cd $GRID_HOME/addnode
./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}" \
  "CLUSTER_NEW_VIRTUAL_HOSTNAMES={clone-rac3-vip}"

8.4 Verify Clusterware on New Node

# Verify from any node:
crsctl stat res -t

# Verify from new node:
crsctl check cluster -all
olsnodes -n

8.5 Extend Oracle Database Software to New Node

Run from clone-rac1 as oracle:

cd $ORACLE_HOME/addnode
./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}"

8.6 Add New Instance to Database

# Add instance 3 as oracle on clone-rac1:
srvctl add instance -d CLNECDB -i CLNECDB3 -n clone-rac3

# Configure instance-specific parameters:
sqlplus / as sysdba
SQL> ALTER SYSTEM SET instance_number=3 SID='CLNECDB3' SCOPE=SPFILE;
SQL> ALTER SYSTEM SET thread=3 SID='CLNECDB3' SCOPE=SPFILE;
SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS3' SID='CLNECDB3' SCOPE=SPFILE;
SQL> EXIT;

# Create undo tablespace for node 3:
sqlplus / as sysdba
SQL> CREATE UNDO TABLESPACE UNDOTBS3
  DATAFILE '+DATA/CLNECDB/undotbs3_01.dbf' SIZE 2G AUTOEXTEND ON;

# Add redo thread for node 3:
SQL> ALTER DATABASE ADD LOGFILE THREAD 3
  GROUP 7 ('+REDO/CLNECDB/redo07a.log','+DATA/CLNECDB/redo07b.log') SIZE 512M,
  GROUP 8 ('+REDO/CLNECDB/redo08a.log','+DATA/CLNECDB/redo08b.log') SIZE 512M,
  GROUP 9 ('+REDO/CLNECDB/redo09a.log','+DATA/CLNECDB/redo09b.log') SIZE 512M;
SQL> ALTER DATABASE ENABLE PUBLIC THREAD 3;
SQL> EXIT;

# Start instance on new node:
srvctl start instance -d CLNECDB -i CLNECDB3
srvctl status database -d CLNECDB

8.7 Add New EBS App Node

In standard EBS 12.2 deployments, the application filesystem (APPL_TOP, COMMON_TOP, OA_HTML) is on shared NFS/NAS storage mounted by all app nodes. Adding a new app node does NOT require copying files — the new node mounts the existing shared filesystem. Only INST_TOP (instance-specific config, logs, temp) is local to each node.

Run on clone-app3 as applmgr:

# Step 1: Mount shared EBS filesystems on new node (as root):
mount -t nfs nfs-server:/exports/ebsapps /u01/app/ebsapps
mount -t nfs nfs-server:/exports/common  /u01/app/common
# Add permanent entries to /etc/fstab

# Step 2: Create local INST_TOP directory:
mkdir -p /u01/app/inst/apps/CLNE_clone-app3

# Step 3: Source environment from existing node context:
source /u01/app/ebsapps/EBSapps.env run

# Step 4: Create node-specific context file using adclonectx.pl:
perl $AD_TOP/bin/adclonectx.pl addnode \
  contextfile=$CONTEXT_FILE \
  pairsfile=/tmp/clone-app3_pairs.txt

The pairs file (/tmp/clone-app3_pairs.txt) contains node-specific overrides:

s_hostname=clone-app3
s_local_hostname=clone-app3
s_webentryhost=clone-app3
s_active_webport=8000
s_tools_oracle_home=/u01/app/ebsapps/fs1/EBSapps/10.1.2

# Step 5: Run Autoconfig on new node using new context file:
perl $AD_TOP/bin/adconfig.pl contextfile=/u01/app/inst/apps/CLNE_clone-app3/appl/admin/CLNE_clone-app3.xml

# Step 6: Start services on new node:
adapcctl.sh start
adopmnctl.sh start
adnodemgrctl.sh start
adstrtal.sh

# Step 7: Register new app node in FND_NODES (if not auto-registered):
INSERT INTO apps.fnd_nodes (node_name, server_id, support_cp,
  support_forms, support_web, platform_code)
VALUES ('clone-app3', apps.fnd_nodes_s.nextval,
  'Y', 'Y', 'Y', 'LINUX');
COMMIT;

9. Troubleshooting Common Issues

9.1 RMAN Duplicate Failures

Symptom	Resolution
ORA-17628: Oracle error 19505	ASM directory for target does not exist. Create dirs with asmcmd mkdir.
RMAN-05001: auxiliary channel not started	Auxiliary listener not running or TNS not reachable. Check tnsnames.ora and listener.ora on target.
ORA-12154: TNS could not resolve	Add PRODCDB and CLNECDB_AUX to tnsnames.ora on source and target.
Duplicate hangs after datafile copy	Check network bandwidth. Use fewer parallel channels if bandwidth limited.

9.2 RAC Startup Issues

Symptom	Resolution
ORA-29701: unable to connect to Cluster Manager	Clusterware not running. Run: crsctl start cluster -all
Instance stuck in MOUNT state	Redo thread not enabled. ALTER DATABASE ENABLE PUBLIC THREAD N;
UNDO errors on node 2	Wrong undo tablespace assigned. Check SPFILE parameter undo_tablespace for SID.
CSS daemon not starting on new node	OCR or voting disk not accessible. Check ASM disk group visibility.

9.3 EBS Clone Issues

Symptom	Resolution
adcfgclone.pl fails on context file	Run adpreclone.pl on source first. For node addition use adclonectx.pl addnode instead of adcfgclone.pl.
WLS Admin Server not starting	Check $DOMAIN_HOME/servers/AdminServer/logs/. Often Java heap or port conflict.
Login page not loading after clone	Run Autoconfig. Check Apache/OHS config in $INST_TOP/ora/10.1.3/Apache/Apache/conf/.
FND_NODES mismatch	Update fnd_nodes with new hostnames. Run Autoconfig after.
Concurrent Manager not starting	Check ICM log in $APPLCSF/$APPLLOG. Usually APPS password or FND_NODES issue.

10. Post-Clone Validation Checklist

✓	Task	Command / Action	Run As
✓	DB instances running on both nodes	srvctl status database -d CLNECDB	oracle
✓	All PDBs open	SELECT name, open_mode FROM v$pdbs;	oracle/sqlplus
✓	SCAN listener working	tnsping EBSPDB_SERVICE	oracle
✓	EBS login page accessible	http://clone-app1:8000/OA_HTML/AppsLocalLogin.jsp	Browser
✓	Concurrent Manager running	adcmctl.sh status apps/<pwd>	applmgr
✓	No invalid objects	SELECT COUNT(*) FROM dba_objects WHERE status='INVALID';	oracle/sqlplus
✓	Archive log mode verified	ARCHIVE LOG LIST;	oracle/sqlplus
✓	Email notifications disabled	Check fnd_svc_components WF_MAILER status	oracle/sqlplus
✓	Autoconfig completed on all nodes	Check autoconfig log for errors	applmgr
✓	ADOP session clean (no stuck sessions)	Check ad_adop_sessions	oracle/sqlplus
✓	Tablespace usage normal	Check dba_tablespace_usage_metrics	oracle/sqlplus
✓	Alert log clean on all nodes	tail -100 alert_CLNECDB*.log	oracle