Documentation · Database Operations

# Oracle EBS 12.2 + 19c RAC Clone Using RMAN Duplicate

An end-to-end runbook to clone an Oracle E-Business Suite 12.2 environment backed by a 2-node Oracle 19c RAC database using RMAN Active Database Duplicate within the same datacenter — including single-instance-to-RAC conversion, EBS Rapid Clone, post-clone tasks, and adding a new RAC node.

Version 2.0 · Oracle 19c + EBS 12.2.x · Same-datacenter clone

## 1. Overview and Architecture

This runbook covers the end-to-end procedure to clone an Oracle E-Business Suite 12.2 environment backed by a 2-node Oracle 19c Real Application Clusters (RAC) database using RMAN Duplicate within the same datacenter. It also covers the process of adding a new RAC node to an existing cluster.

### 1.1 Scope

- Source: 2-node Oracle 19c RAC (CDB with EBS PDB), ASM storage, EBS 12.2 application tier (2 app nodes)
- Target: New 2-node RAC cluster + 2 new EBS app tier nodes — all within the same datacenter
- Database clone method: RMAN Duplicate (Active Database Duplicate — no backup required)
- EBS clone method: Rapid Clone (`adcfgclone.pl`) after DB duplicate
- New node addition: Oracle Clusterware + Grid Infrastructure extension to a third/fourth node

### 1.2 Environment Reference

Source DB Cluster
prod-rac1, prod-rac2
Target DB Cluster
clone-rac1, clone-rac2
Source EBS App Nodes
prod-app1, prod-app2
Target EBS App Nodes
clone-app1, clone-app2
Oracle DB Version
19c (19.x.x.x)
EBS Version
12.2.x (with latest patches)
DB Name (CDB)
PRODCDB → CLNECDB
EBS PDB Name
EBSPDB → CLNEPDB
ASM Disk Groups
+DATA, +REDO, +FRA
Grid Home
/u01/app/19.0.0/grid
Oracle Home
/u01/app/oracle/product/19.0.0/dbhome_1
EBS Base
/u01/app/ebsapps
OS User (DB)
oracle / oinstall, dba
OS User (Grid)
grid / oinstall, asmadmin

## 2. Pre-Clone Checklist

### 2.1 Hardware and OS Requirements

- Identical CPU architecture (x86_64)
- Same OS version and patch level as source nodes
- Sufficient RAM: minimum 32GB per DB node (same as source)
- ASM disk groups provisioned and visible on target nodes
- Network: Public, Private (interconnect), and SCAN IPs configured
- SSH equivalence set up between all source and target nodes (oracle and grid users)
- NTP synchronized across all nodes

### 2.2 Oracle Software Prerequisites

- Oracle Grid Infrastructure 19c installed and configured on target cluster (clusterware up, SCAN listener running)
- Oracle Database 19c software installed on target nodes (`ORACLE_HOME` same path as source)
- ASM disk groups created on target: +DATA, +REDO, +FRA (same names as source, size ≥ source usage)
- Oracle Inventory consistent across all nodes

### 2.3 Network Prerequisites

Public IPs
One per node (clone-rac1, clone-rac2)
Virtual IPs (VIP)
One per node (clone-rac1-vip, clone-rac2-vip)
SCAN Name
clone-scan (resolves to 3 IPs via DNS/GNS)
Private Interconnect
bond0 (10GbE recommended, MTU 9000)
/etc/hosts
All nodes must resolve each other by hostname

### 2.4 Source Database Checks

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Check DB open mode | `SELECT name, open_mode, db_unique_name FROM v$database;` | oracle/sqlplus
2 | Check CDB/PDB status | `SELECT name, open_mode FROM v$pdbs;` | oracle/sqlplus
3 | Check archive log mode | `ARCHIVE LOG LIST;` | oracle/sqlplus
4 | Check FRA space | `SELECT * FROM v$recovery_file_dest;` | oracle/sqlplus
5 | Check RMAN config | `SHOW ALL;` | rman target /
6 | Verify ASM disk groups | `SELECT name, state, total_mb, free_mb FROM v$asm_diskgroup;` | grid/sqlplus
7 | Check active sessions on source | `SELECT count(*) FROM v$session WHERE status='ACTIVE';` | oracle/sqlplus

⚠ NOTE: Place the source database in archive log mode if it is not already. EBS 12.2 requires archive log mode for online patching (ADOP).

## 3. Target Cluster Preparation

### 3.1 Grid Infrastructure Verification

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Verify clusterware status | `crsctl stat res -t` | grid
2 | Check SCAN listener | `srvctl status scan_listener` | grid
3 | Verify ASM disk groups | `asmcmd lsdg` | grid
4 | Check OCR and Voting Disk | `ocrcheck` · `crsctl query css votedisk` | root
5 | Verify interconnect | `oifcfg getif` | grid

### 3.2 ASM Disk Group Preparation

Shell — clone-rac1 as grid
# On clone-rac1 as grid: asmcmd ASMCMD> lsdg ASMCMD> mkdir +DATA/CLNECDB ASMCMD> mkdir +REDO/CLNECDB ASMCMD> mkdir +FRA/CLNECDB ASMCMD> exit

### 3.3 Oracle Password File on Target

Shell — as oracle
# On clone-rac1 as oracle: orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB1 password=<sys_password> entries=10 # On clone-rac2 as oracle: orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB2 password=<sys_password> entries=10

### 3.4 Static Listener Configuration on Target

Create/update `$TNS_ADMIN/listener.ora` on both target nodes:

listener.ora
LISTENER_CLNECDB = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = clone-rac1-vip)(PORT = 1521)))) SID_LIST_LISTENER_CLNECDB = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = CLNECDB) (ORACLE_HOME = /u01/app/oracle/product/19.0.0/dbhome_1) (SID_NAME = CLNECDB1))) # Start the listener: lsnrctl start LISTENER_CLNECDB

### 3.5 Auxiliary Init Parameter File

Create `/tmp/init_aux.ora` on clone-rac1:

/tmp/init_aux.ora
db_name=CLNECDB db_unique_name=CLNECDB enable_pluggable_database=TRUE sga_target=8G pga_aggregate_target=2G control_files='+DATA/CLNECDB/control01.ctl','+REDO/CLNECDB/control02.ctl' db_file_name_convert='+DATA/PRODCDB','+DATA/CLNECDB','+REDO/PRODCDB','+REDO/CLNECDB' log_file_name_convert='+REDO/PRODCDB','+REDO/CLNECDB' db_recovery_file_dest='+FRA' db_recovery_file_dest_size=500G audit_file_dest='/u01/app/oracle/admin/CLNECDB/adump' diagnostic_dest='/u01/app/oracle' cluster_database=FALSE instance_number=1 thread=1
⚠ NOTE: Set `cluster_database=FALSE` initially. It will be set to TRUE after the duplicate completes and RAC is configured.

## 4. RMAN Active Database Duplicate

### 4.1 Start Auxiliary Instance

Shell — clone-rac1 as oracle
# On clone-rac1 as oracle: export ORACLE_SID=CLNECDB1 mkdir -p /u01/app/oracle/admin/CLNECDB/adump sqlplus / as sysdba SQL> STARTUP NOMOUNT PFILE='/tmp/init_aux.ora'; SQL> EXIT;

### 4.2 Configure RMAN Channels and Start Duplicate

Run the following RMAN script from clone-rac1 as oracle. This performs an Active Database Duplicate — no backup needed, data streams directly from source to target:

RMAN — connect
rman target sys/<sys_pwd>@PRODCDB auxiliary sys/<sys_pwd>@CLNECDB_AUX

Then at the RMAN prompt:

RMAN — DUPLICATE
DUPLICATE TARGET DATABASE TO CLNECDB FROM ACTIVE DATABASE USING COMPRESSED BACKUPSET NOFILENAMECHECK SPFILE PARAMETER_VALUE_CONVERT 'PRODCDB','CLNECDB', '+DATA/PRODCDB','+DATA/CLNECDB', '+REDO/PRODCDB','+REDO/CLNECDB' SET db_unique_name='CLNECDB' SET db_recovery_file_dest='+FRA' SET db_recovery_file_dest_size='500G' SET cluster_database='FALSE' SET instance_number='1' SET thread='1' ALLOCATE AUXILIARY CHANNEL c1 DEVICE TYPE DISK ALLOCATE AUXILIARY CHANNEL c2 DEVICE TYPE DISK ALLOCATE AUXILIARY CHANNEL c3 DEVICE TYPE DISK ALLOCATE AUXILIARY CHANNEL c4 DEVICE TYPE DISK;
⚠ NOTE: The duplicate may take several hours depending on source database size. Monitor the alert log on the target: `tail -f $ORACLE_BASE/diag/rdbms/clnecdb/CLNECDB1/trace/alert_CLNECDB1.log`

### 4.3 Post-Duplicate Verification

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Verify DB is open | `SELECT name, open_mode FROM v$database;` | oracle/sqlplus
2 | Verify PDB status | `SELECT name, open_mode FROM v$pdbs;` | oracle/sqlplus
3 | Open EBS PDB | `ALTER PLUGGABLE DATABASE CLNEPDB OPEN;` | oracle/sqlplus
4 | Check datafiles | `SELECT name FROM v$datafile;` | oracle/sqlplus
5 | Verify redo logs | `SELECT group#, status FROM v$log;` | oracle/sqlplus
6 | Check temp tablespace | `SELECT name FROM v$tempfile;` | oracle/sqlplus

## 5. Convert Single Instance to 2-Node RAC

### 5.1 Create SPFILE in ASM

SQL — clone-rac1 as oracle
# On clone-rac1 as oracle: sqlplus / as sysdba SQL> CREATE SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora' FROM MEMORY; SQL> SHUTDOWN IMMEDIATE; SQL> EXIT;

### 5.2 Create RAC-Enabled PFILE

Create `/tmp/init_rac.ora` on clone-rac1:

/tmp/init_rac.ora
SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora'

### 5.3 Modify SPFILE for RAC

SQL
sqlplus / as sysdba SQL> STARTUP PFILE='/tmp/init_rac.ora'; SQL> ALTER SYSTEM SET cluster_database=TRUE SCOPE=SPFILE; SQL> ALTER SYSTEM SET instance_number=1 SID='CLNECDB1' SCOPE=SPFILE; SQL> ALTER SYSTEM SET instance_number=2 SID='CLNECDB2' SCOPE=SPFILE; SQL> ALTER SYSTEM SET thread=1 SID='CLNECDB1' SCOPE=SPFILE; SQL> ALTER SYSTEM SET thread=2 SID='CLNECDB2' SCOPE=SPFILE; SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS1' SID='CLNECDB1' SCOPE=SPFILE; SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS2' SID='CLNECDB2' SCOPE=SPFILE; SQL> SHUTDOWN IMMEDIATE; SQL> EXIT;

### 5.4 Create Undo Tablespace for Node 2

SQL
sqlplus / as sysdba SQL> STARTUP; SQL> CREATE UNDO TABLESPACE UNDOTBS2 DATAFILE '+DATA/CLNECDB/undotbs2_01.dbf' SIZE 2G AUTOEXTEND ON; SQL> SHUTDOWN IMMEDIATE; SQL> EXIT;

### 5.5 Enable Redo Thread for Node 2

SQL
sqlplus / as sysdba SQL> STARTUP MOUNT; SQL> ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 4 ('+REDO/CLNECDB/redo04a.log','+DATA/CLNECDB/redo04b.log') SIZE 512M, GROUP 5 ('+REDO/CLNECDB/redo05a.log','+DATA/CLNECDB/redo05b.log') SIZE 512M, GROUP 6 ('+REDO/CLNECDB/redo06a.log','+DATA/CLNECDB/redo06b.log') SIZE 512M; SQL> ALTER DATABASE ENABLE PUBLIC THREAD 2; SQL> ALTER DATABASE OPEN; SQL> EXIT;

### 5.6 Add Database to Clusterware (srvctl)

Shell — clone-rac1 as oracle
# On clone-rac1 as oracle: srvctl add database -d CLNECDB \ -o $ORACLE_HOME \ -p +DATA/CLNECDB/spfileCLNECDB.ora \ -r PRIMARY \ -s OPEN \ -t IMMEDIATE \ -i CLNECDB1,CLNECDB2 srvctl add instance -d CLNECDB -i CLNECDB1 -n clone-rac1 srvctl add instance -d CLNECDB -i CLNECDB2 -n clone-rac2 srvctl start database -d CLNECDB srvctl status database -d CLNECDB

### 5.7 Register PDB with Clusterware

Shell
srvctl add service -d CLNECDB -s EBSPDB_SERVICE \ -pdb CLNEPDB \ -preferred CLNECDB1,CLNECDB2 \ -failovertype SELECT \ -failovermethod BASIC \ -failoverretry 5 \ -failoverdelay 3 srvctl start service -d CLNECDB -s EBSPDB_SERVICE srvctl status service -d CLNECDB -s EBSPDB_SERVICE

## 6. EBS 12.2 Application Tier Clone

### 6.1 Prerequisites — Database Side

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Connect to EBS PDB | `ALTER SESSION SET CONTAINER=CLNEPDB;` | oracle/sqlplus
2 | Unlock APPS schema (if locked) | `ALTER USER APPS ACCOUNT UNLOCK;` | oracle/sqlplus
3 | Reset APPS password | `EXEC APPS.FND_USER_PKG.CHANGE_USER_NAME('APPS','APPS');` | oracle/sqlplus
4 | Verify APPLSYS | `SELECT username, account_status FROM dba_users WHERE username='APPLSYS';` | oracle/sqlplus
5 | Check concurrent manager tables | `SELECT COUNT(*) FROM apps.fnd_concurrent_queues;` | oracle/sqlplus

### 6.2 Source Application Tier — Pre-Clone Steps

Run on prod-app1 as applmgr:

Shell — prod-app1 as applmgr
# Source the EBS environment source /u01/app/ebsapps/EBSapps.env run # Prepare source for cloning perl $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier
⚠ NOTE: `adcfgclone.pl` is used for full environment clones (new environment, different hostname/instance name). For adding a node to an existing EBS installation, use `adclonectx.pl addnode` instead — see Section 8.7.

### 6.3 Copy Application Tier to Target Nodes

From prod-app1, copy the entire EBS base to both target app nodes:

Shell — prod-app1
# Copy filesystem to clone-app1 tar czf - /u01/app/ebsapps | ssh clone-app1 "tar xzf - -C /" # Copy filesystem to clone-app2 tar czf - /u01/app/ebsapps | ssh clone-app2 "tar xzf - -C /"
⚠ NOTE: This may take significant time depending on EBS installation size (typically 100GB–300GB). Consider using `rsync` for subsequent syncs.

### 6.4 Configure Application Tier on clone-app1

Run on clone-app1 as applmgr:

Shell — clone-app1 as applmgr
cd $COMMON_TOP/clone/bin perl adcfgclone.pl appsTier

When prompted, provide:

Target System Hostname
clone-app1
Target Instance Name
CLNE (new environment name)
Target DB Host
clone-scan (SCAN hostname)
Target DB Port
1521
Target DB Service Name
EBSPDB_SERVICE (or PDB service)
Target ORACLE_SID (CDB)
CLNECDB
APPS Password
<apps_password>
SYSTEM Password
<system_password>
WebLogic Admin Password
<wls_password>

### 6.5 Configure Application Tier on clone-app2

Run on clone-app2 as applmgr:

Shell — clone-app2 as applmgr
cd $COMMON_TOP/clone/bin perl adcfgclone.pl appsTier

Provide the same parameters as clone-app1 but with `clone-app2` as the hostname.

### 6.6 Post-Clone EBS Configuration

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Source env on clone-app1 | `source /u01/app/ebsapps/EBSapps.env run` | applmgr
2 | Start Apache/OHS | `adapcctl.sh start` | applmgr
3 | Start OPMN | `adopmnctl.sh start` | applmgr
4 | Start Node Manager | `adnodemgrctl.sh start` | applmgr
5 | Start WLS Admin Server | `adstrtal.sh` | applmgr
6 | Start all Managed Servers | `adadminsrvctl.sh start all` | applmgr
7 | Start Concurrent Manager | `adcmctl.sh start apps/<apps_pwd>` | applmgr
8 | Verify EBS URL | `http://clone-app1.domain.com:8000/OA_HTML/AppsLocalLogin.jsp` | Browser

## 7. Post-Clone Tasks

### 7.1 EBS Database Profile Updates

SQL — connect to EBS PDB
-- Connect to EBS PDB: ALTER SESSION SET CONTAINER=CLNEPDB; -- Update application node hostname in FND_NODES UPDATE apps.fnd_nodes SET node_name='clone-app1' WHERE node_name='prod-app1'; UPDATE apps.fnd_nodes SET node_name='clone-app2' WHERE node_name='prod-app2'; COMMIT; -- Update server host in WF_AGENTS UPDATE apps.wf_agents SET address=REPLACE(address,'prod-app1','clone-app1') WHERE address LIKE '%prod-app1%'; COMMIT;

### 7.2 Disable/Modify Email Notifications

SQL
-- Disable all concurrent program email notifications on clone UPDATE apps.fnd_svc_components SET component_status='STOPPED' WHERE component_type='WF_MAILER'; COMMIT;
⚠ NOTE: Always disable email notifications on cloned/non-production environments to prevent accidental emails to real users.

### 7.3 Autoconfig

Run Autoconfig on both app nodes and the DB node to synchronize configuration:

Shell
# On clone-app1 and clone-app2 as applmgr: perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE # On clone-rac1 as oracle (DB tier autoconfig): perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

### 7.4 ETCC Verification

Run the EBS Technology Codelevel Checker to confirm patch levels:

Shell
perl $AD_TOP/patch/115/bin/adpatch.pl options='nocopypfile' \ patchtop=$PATCH_TOP appspass=$APPS_PWD \ workers=8 logfile=etcc.log

### 7.5 Verify ADOP Cycle is Clean

SQL
-- Check for stuck ADOP sessions: SELECT session_id, prepare_status, apply_status, finalize_status, cutover_status, cleanup_status FROM apps.ad_adop_sessions WHERE session_id = (SELECT MAX(session_id) FROM apps.ad_adop_sessions);
⚠ NOTE: If the source had an in-progress ADOP session, it will be replicated to the clone. Abort it on the clone: `adop phase=abort`.

## 8. Adding a New Node to Existing RAC Cluster

### 8.1 Overview

This section covers extending an existing 2-node RAC cluster by adding a third node (clone-rac3). The same process applies to adding a fourth node.

### 8.2 Prepare the New Node

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Install same OS version | Match patch level with existing nodes | root
2 | Configure kernel parameters | `/etc/sysctl.conf` — match existing nodes | root
3 | Create OS users/groups | oracle, grid, oinstall, dba, asmadmin etc. | root
4 | Configure /etc/hosts | Add all node hostnames + VIPs + SCAN | root
5 | Set up SSH equivalence | `ssh-keygen; ssh-copy-id` to all existing nodes | oracle/grid
6 | Verify ASM disks visible | `ls /dev/oracleasm/disks/` | root
7 | Configure NTP | Same NTP server as other nodes | root

### 8.3 Extend Grid Infrastructure to New Node

Run from an existing node (clone-rac1) as grid:

Shell — clone-rac1 as grid
cd $GRID_HOME/addnode ./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}" \ "CLUSTER_NEW_VIRTUAL_HOSTNAMES={clone-rac3-vip}"
⚠ NOTE: This will copy Grid Infrastructure to the new node, configure CSS/CRS/OCR on the new node, and start clusterware. Monitor progress via `$GRID_HOME/cfgtoollogs/addNode/`.

### 8.4 Verify Clusterware on New Node

Shell
# Verify from any node: crsctl stat res -t # Verify from new node: crsctl check cluster -all olsnodes -n

### 8.5 Extend Oracle Database Software to New Node

Run from clone-rac1 as oracle:

Shell — clone-rac1 as oracle
cd $ORACLE_HOME/addnode ./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}"

### 8.6 Add New Instance to Database

Shell + SQL — clone-rac1
# Add instance 3 as oracle on clone-rac1: srvctl add instance -d CLNECDB -i CLNECDB3 -n clone-rac3 # Configure instance-specific parameters: sqlplus / as sysdba SQL> ALTER SYSTEM SET instance_number=3 SID='CLNECDB3' SCOPE=SPFILE; SQL> ALTER SYSTEM SET thread=3 SID='CLNECDB3' SCOPE=SPFILE; SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS3' SID='CLNECDB3' SCOPE=SPFILE; SQL> EXIT; # Create undo tablespace for node 3: sqlplus / as sysdba SQL> CREATE UNDO TABLESPACE UNDOTBS3 DATAFILE '+DATA/CLNECDB/undotbs3_01.dbf' SIZE 2G AUTOEXTEND ON; # Add redo thread for node 3: SQL> ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 7 ('+REDO/CLNECDB/redo07a.log','+DATA/CLNECDB/redo07b.log') SIZE 512M, GROUP 8 ('+REDO/CLNECDB/redo08a.log','+DATA/CLNECDB/redo08b.log') SIZE 512M, GROUP 9 ('+REDO/CLNECDB/redo09a.log','+DATA/CLNECDB/redo09b.log') SIZE 512M; SQL> ALTER DATABASE ENABLE PUBLIC THREAD 3; SQL> EXIT; # Start instance on new node: srvctl start instance -d CLNECDB -i CLNECDB3 srvctl status database -d CLNECDB

### 8.7 Add New EBS App Node

In standard EBS 12.2 deployments, the application filesystem (APPL_TOP, COMMON_TOP, OA_HTML) is on shared NFS/NAS storage mounted by all app nodes. Adding a new app node does NOT require copying files — the new node mounts the existing shared filesystem. Only INST_TOP (instance-specific config, logs, temp) is local to each node.

⚠ NOTE: If your EBS is deployed on local disk (non-shared, e.g. DMZ environments), you will need to `rsync`/`tar` the filesystem to the new node first. For standard shared NFS deployments, skip directly to Step 1 below.

Run on clone-app3 as applmgr:

Shell — clone-app3
# Step 1: Mount shared EBS filesystems on new node (as root): mount -t nfs nfs-server:/exports/ebsapps /u01/app/ebsapps mount -t nfs nfs-server:/exports/common /u01/app/common # Add permanent entries to /etc/fstab # Step 2: Create local INST_TOP directory: mkdir -p /u01/app/inst/apps/CLNE_clone-app3 # Step 3: Source environment from existing node context: source /u01/app/ebsapps/EBSapps.env run # Step 4: Create node-specific context file using adclonectx.pl: perl $AD_TOP/bin/adclonectx.pl addnode \ contextfile=$CONTEXT_FILE \ pairsfile=/tmp/clone-app3_pairs.txt

The pairs file (`/tmp/clone-app3_pairs.txt`) contains node-specific overrides:

/tmp/clone-app3_pairs.txt
s_hostname=clone-app3 s_local_hostname=clone-app3 s_webentryhost=clone-app3 s_active_webport=8000 s_tools_oracle_home=/u01/app/ebsapps/fs1/EBSapps/10.1.2
Shell — clone-app3
# Step 5: Run Autoconfig on new node using new context file: perl $AD_TOP/bin/adconfig.pl contextfile=/u01/app/inst/apps/CLNE_clone-app3/appl/admin/CLNE_clone-app3.xml # Step 6: Start services on new node: adapcctl.sh start adopmnctl.sh start adnodemgrctl.sh start adstrtal.sh # Step 7: Register new app node in FND_NODES (if not auto-registered): INSERT INTO apps.fnd_nodes (node_name, server_id, support_cp, support_forms, support_web, platform_code) VALUES ('clone-app3', apps.fnd_nodes_s.nextval, 'Y', 'Y', 'Y', 'LINUX'); COMMIT;

## 9. Troubleshooting Common Issues

### 9.1 RMAN Duplicate Failures

Symptom | Resolution
--- | ---
ORA-17628: Oracle error 19505 | ASM directory for target does not exist. Create dirs with `asmcmd mkdir`.
RMAN-05001: auxiliary channel not started | Auxiliary listener not running or TNS not reachable. Check `tnsnames.ora` and `listener.ora` on target.
ORA-12154: TNS could not resolve | Add PRODCDB and CLNECDB_AUX to `tnsnames.ora` on source and target.
Duplicate hangs after datafile copy | Check network bandwidth. Use fewer parallel channels if bandwidth limited.

### 9.2 RAC Startup Issues

Symptom | Resolution
--- | ---
ORA-29701: unable to connect to Cluster Manager | Clusterware not running. Run: `crsctl start cluster -all`
Instance stuck in MOUNT state | Redo thread not enabled. `ALTER DATABASE ENABLE PUBLIC THREAD N;`
UNDO errors on node 2 | Wrong undo tablespace assigned. Check SPFILE parameter `undo_tablespace` for SID.
CSS daemon not starting on new node | OCR or voting disk not accessible. Check ASM disk group visibility.

### 9.3 EBS Clone Issues

Symptom | Resolution
--- | ---
adcfgclone.pl fails on context file | Run `adpreclone.pl` on source first. For node addition use `adclonectx.pl addnode` instead of `adcfgclone.pl`.
WLS Admin Server not starting | Check `$DOMAIN_HOME/servers/AdminServer/logs/`. Often Java heap or port conflict.
Login page not loading after clone | Run Autoconfig. Check Apache/OHS config in `$INST_TOP/ora/10.1.3/Apache/Apache/conf/`.
FND_NODES mismatch | Update `fnd_nodes` with new hostnames. Run Autoconfig after.
Concurrent Manager not starting | Check ICM log in `$APPLCSF/$APPLLOG`. Usually APPS password or FND_NODES issue.

## 10. Post-Clone Validation Checklist

✓ | Task | Command / Action | Run As
--- | --- | --- | ---
✓ | DB instances running on both nodes | `srvctl status database -d CLNECDB` | oracle
✓ | All PDBs open | `SELECT name, open_mode FROM v$pdbs;` | oracle/sqlplus
✓ | SCAN listener working | `tnsping EBSPDB_SERVICE` | oracle
✓ | EBS login page accessible | `http://clone-app1:8000/OA_HTML/AppsLocalLogin.jsp` | Browser
✓ | Concurrent Manager running | `adcmctl.sh status apps/<pwd>` | applmgr
✓ | No invalid objects | `SELECT COUNT(*) FROM dba_objects WHERE status='INVALID';` | oracle/sqlplus
✓ | Archive log mode verified | `ARCHIVE LOG LIST;` | oracle/sqlplus
✓ | Email notifications disabled | Check `fnd_svc_components` WF_MAILER status | oracle/sqlplus
✓ | Autoconfig completed on all nodes | Check autoconfig log for errors | applmgr
✓ | ADOP session clean (no stuck sessions) | Check `ad_adop_sessions` | oracle/sqlplus
✓ | Tablespace usage normal | Check `dba_tablespace_usage_metrics` | oracle/sqlplus
✓ | Alert log clean on all nodes | `tail -100 alert_CLNECDB*.log` | oracle

After a clone, run a TuneVault health check to validate the environment automatically — invalid objects, tablespace usage, ADOP state, listener health, and EBS operations. Add a connection →

Related: Privilege setup · Oracle setup guide · Blog version of this runbook · support@tunevault.app