← All articles
Runbook · 18 min read · Kiran Kumar Ale

# Oracle EBS 12.2 + 19c RAC Clone Using RMAN Duplicate — Complete Runbook

A production-tested, end-to-end procedure to clone an Oracle E-Business Suite 12.2 environment backed by a 2-node Oracle 19c RAC database using RMAN Active Database Duplicate within the same datacenter — including single-instance-to-RAC conversion, EBS Rapid Clone, post-clone tasks, and adding a new RAC node.

oracle-rac ebs-clone rman oracle-19c

## 1. Overview and Architecture

This runbook covers the end-to-end procedure to clone an Oracle E-Business Suite 12.2 environment backed by a 2-node Oracle 19c Real Application Clusters (RAC) database using RMAN Duplicate within the same datacenter. It also covers the process of adding a new RAC node to an existing cluster.

### 1.1 Scope

- Source: 2-node Oracle 19c RAC (CDB with EBS PDB), ASM storage, EBS 12.2 application tier (2 app nodes)
- Target: New 2-node RAC cluster + 2 new EBS app tier nodes — all within the same datacenter
- Database clone method: RMAN Duplicate (Active Database Duplicate — no backup required)
- EBS clone method: Rapid Clone (`adcfgclone.pl`) after DB duplicate
- New node addition: Oracle Clusterware + Grid Infrastructure extension to a third/fourth node

### 1.2 Environment Reference

Source DB Cluster
prod-rac1, prod-rac2
Target DB Cluster
clone-rac1, clone-rac2
Source EBS App Nodes
prod-app1, prod-app2
Target EBS App Nodes
clone-app1, clone-app2
Oracle DB Version
19c (19.x.x.x)
EBS Version
12.2.x (with latest patches)
DB Name (CDB)
PRODCDB → CLNECDB
EBS PDB Name
EBSPDB → CLNEPDB
ASM Disk Groups
+DATA, +REDO, +FRA
Grid Home
/u01/app/19.0.0/grid
Oracle Home
/u01/app/oracle/product/19.0.0/dbhome_1
EBS Base
/u01/app/ebsapps
OS User (DB)
oracle / oinstall, dba
OS User (Grid)
grid / oinstall, asmadmin

## 2. Pre-Clone Checklist

### 2.1 Hardware and OS Requirements

- Identical CPU architecture (x86_64)
- Same OS version and patch level as source nodes
- Sufficient RAM: minimum 32GB per DB node (same as source)
- ASM disk groups provisioned and visible on target nodes
- Network: Public, Private (interconnect), and SCAN IPs configured
- SSH equivalence set up between all source and target nodes (oracle and grid users)
- NTP synchronized across all nodes

### 2.2 Oracle Software Prerequisites

- Oracle Grid Infrastructure 19c installed and configured on target cluster (clusterware up, SCAN listener running)
- Oracle Database 19c software installed on target nodes (`ORACLE_HOME` same path as source)
- ASM disk groups created on target: +DATA, +REDO, +FRA (same names as source, size ≥ source usage)
- Oracle Inventory consistent across all nodes

### 2.3 Network Prerequisites

Public IPs
One per node (clone-rac1, clone-rac2)
Virtual IPs (VIP)
One per node (clone-rac1-vip, clone-rac2-vip)
SCAN Name
clone-scan (resolves to 3 IPs via DNS/GNS)
Private Interconnect
bond0 (10GbE recommended, MTU 9000)
/etc/hosts
All nodes must resolve each other by hostname

### 2.4 Source Database Checks

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Check DB open mode | `SELECT name, open_mode, db_unique_name FROM v$database;` | oracle/sqlplus
2 | Check CDB/PDB status | `SELECT name, open_mode FROM v$pdbs;` | oracle/sqlplus
3 | Check archive log mode | `ARCHIVE LOG LIST;` | oracle/sqlplus
4 | Check FRA space | `SELECT * FROM v$recovery_file_dest;` | oracle/sqlplus
5 | Check RMAN config | `SHOW ALL;` | rman target /
6 | Verify ASM disk groups | `SELECT name, state, total_mb, free_mb FROM v$asm_diskgroup;` | grid/sqlplus
7 | Check active sessions on source | `SELECT count(*) FROM v$session WHERE status='ACTIVE';` | oracle/sqlplus

⚠ NOTE: Place the source database in archive log mode if it is not already. EBS 12.2 requires archive log mode for online patching (ADOP).

## 3. Target Cluster Preparation

### 3.1 Grid Infrastructure Verification

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Verify clusterware status | `crsctl stat res -t` | grid
2 | Check SCAN listener | `srvctl status scan_listener` | grid
3 | Verify ASM disk groups | `asmcmd lsdg` | grid
4 | Check OCR and Voting Disk | `ocrcheck` `crsctl query css votedisk` | root
5 | Verify interconnect | `oifcfg getif` | grid

### 3.2 ASM Disk Group Preparation

    # On clone-rac1 as grid:
    asmcmd
    ASMCMD> lsdg
    ASMCMD> mkdir +DATA/CLNECDB
    ASMCMD> mkdir +REDO/CLNECDB
    ASMCMD> mkdir +FRA/CLNECDB
    ASMCMD> exit

### 3.3 Oracle Password File on Target

    # On clone-rac1 as oracle:
    orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB1 password=<sys_password> entries=10

    # On clone-rac2 as oracle:
    orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB2 password=<sys_password> entries=10

### 3.4 Static Listener Configuration on Target

Create/update `$TNS_ADMIN/listener.ora` on both target nodes:

    LISTENER_CLNECDB =
      (DESCRIPTION_LIST =
        (DESCRIPTION =
          (ADDRESS = (PROTOCOL = TCP)(HOST = clone-rac1-vip)(PORT = 1521))))

    SID_LIST_LISTENER_CLNECDB =
      (SID_LIST =
        (SID_DESC =
          (GLOBAL_DBNAME = CLNECDB)
          (ORACLE_HOME = /u01/app/oracle/product/19.0.0/dbhome_1)
          (SID_NAME = CLNECDB1)))

    # Start the listener:
    lsnrctl start LISTENER_CLNECDB

### 3.5 Auxiliary Init Parameter File

Create `/tmp/init_aux.ora` on clone-rac1:

    db_name=CLNECDB
    db_unique_name=CLNECDB
    enable_pluggable_database=TRUE
    sga_target=8G
    pga_aggregate_target=2G
    control_files='+DATA/CLNECDB/control01.ctl','+REDO/CLNECDB/control02.ctl'
    db_file_name_convert='+DATA/PRODCDB','+DATA/CLNECDB','+REDO/PRODCDB','+REDO/CLNECDB'
    log_file_name_convert='+REDO/PRODCDB','+REDO/CLNECDB'
    db_recovery_file_dest='+FRA'
    db_recovery_file_dest_size=500G
    audit_file_dest='/u01/app/oracle/admin/CLNECDB/adump'
    diagnostic_dest='/u01/app/oracle'
    cluster_database=FALSE
    instance_number=1
    thread=1

⚠ NOTE: Set `cluster_database=FALSE` initially. It will be set to TRUE after the duplicate completes and RAC is configured.

## 4. RMAN Active Database Duplicate

### 4.1 Start Auxiliary Instance

    # On clone-rac1 as oracle:
    export ORACLE_SID=CLNECDB1
    mkdir -p /u01/app/oracle/admin/CLNECDB/adump
    sqlplus / as sysdba
    SQL> STARTUP NOMOUNT PFILE='/tmp/init_aux.ora';
    SQL> EXIT;

### 4.2 Configure RMAN Channels and Start Duplicate

Run the following RMAN script from clone-rac1 as oracle. This performs an Active Database Duplicate — no backup needed, data streams directly from source to target:

    rman target sys/<sys_pwd>@PRODCDB auxiliary sys/<sys_pwd>@CLNECDB_AUX

Then at the RMAN prompt:

    DUPLICATE TARGET DATABASE TO CLNECDB
      FROM ACTIVE DATABASE
      USING COMPRESSED BACKUPSET
      NOFILENAMECHECK
      SPFILE
        PARAMETER_VALUE_CONVERT 'PRODCDB','CLNECDB',
                                '+DATA/PRODCDB','+DATA/CLNECDB',
                                '+REDO/PRODCDB','+REDO/CLNECDB'
        SET db_unique_name='CLNECDB'
        SET db_recovery_file_dest='+FRA'
        SET db_recovery_file_dest_size='500G'
        SET cluster_database='FALSE'
        SET instance_number='1'
        SET thread='1'
      ALLOCATE AUXILIARY CHANNEL c1 DEVICE TYPE DISK
      ALLOCATE AUXILIARY CHANNEL c2 DEVICE TYPE DISK
      ALLOCATE AUXILIARY CHANNEL c3 DEVICE TYPE DISK
      ALLOCATE AUXILIARY CHANNEL c4 DEVICE TYPE DISK;

⚠ NOTE: The duplicate may take several hours depending on source database size. Monitor the alert log on the target:
`tail -f $ORACLE_BASE/diag/rdbms/clnecdb/CLNECDB1/trace/alert_CLNECDB1.log`

### 4.3 Post-Duplicate Verification

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Verify DB is open | `SELECT name, open_mode FROM v$database;` | oracle/sqlplus
2 | Verify PDB status | `SELECT name, open_mode FROM v$pdbs;` | oracle/sqlplus
3 | Open EBS PDB | `ALTER PLUGGABLE DATABASE CLNEPDB OPEN;` | oracle/sqlplus
4 | Check datafiles | `SELECT name FROM v$datafile;` | oracle/sqlplus
5 | Verify redo logs | `SELECT group#, status FROM v$log;` | oracle/sqlplus
6 | Check temp tablespace | `SELECT name FROM v$tempfile;` | oracle/sqlplus

## 5. Convert Single Instance to 2-Node RAC

### 5.1 Create SPFILE in ASM

    # On clone-rac1 as oracle:
    sqlplus / as sysdba
    SQL> CREATE SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora' FROM MEMORY;
    SQL> SHUTDOWN IMMEDIATE;
    SQL> EXIT;

### 5.2 Create RAC-Enabled PFILE

Create `/tmp/init_rac.ora` on clone-rac1:

    SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora'

### 5.3 Modify SPFILE for RAC

    sqlplus / as sysdba
    SQL> STARTUP PFILE='/tmp/init_rac.ora';
    SQL> ALTER SYSTEM SET cluster_database=TRUE SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET instance_number=1 SID='CLNECDB1' SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET instance_number=2 SID='CLNECDB2' SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET thread=1 SID='CLNECDB1' SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET thread=2 SID='CLNECDB2' SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS1' SID='CLNECDB1' SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS2' SID='CLNECDB2' SCOPE=SPFILE;
    SQL> SHUTDOWN IMMEDIATE;
    SQL> EXIT;

### 5.4 Create Undo Tablespace for Node 2

    sqlplus / as sysdba
    SQL> STARTUP;
    SQL> CREATE UNDO TABLESPACE UNDOTBS2
      DATAFILE '+DATA/CLNECDB/undotbs2_01.dbf' SIZE 2G AUTOEXTEND ON;
    SQL> SHUTDOWN IMMEDIATE;
    SQL> EXIT;

### 5.5 Enable Redo Thread for Node 2

    sqlplus / as sysdba
    SQL> STARTUP MOUNT;
    SQL> ALTER DATABASE ADD LOGFILE THREAD 2
      GROUP 4 ('+REDO/CLNECDB/redo04a.log','+DATA/CLNECDB/redo04b.log') SIZE 512M,
      GROUP 5 ('+REDO/CLNECDB/redo05a.log','+DATA/CLNECDB/redo05b.log') SIZE 512M,
      GROUP 6 ('+REDO/CLNECDB/redo06a.log','+DATA/CLNECDB/redo06b.log') SIZE 512M;
    SQL> ALTER DATABASE ENABLE PUBLIC THREAD 2;
    SQL> ALTER DATABASE OPEN;
    SQL> EXIT;

### 5.6 Add Database to Clusterware (srvctl)

    # On clone-rac1 as oracle:
    srvctl add database -d CLNECDB \
      -o $ORACLE_HOME \
      -p +DATA/CLNECDB/spfileCLNECDB.ora \
      -r PRIMARY \
      -s OPEN \
      -t IMMEDIATE \
      -i CLNECDB1,CLNECDB2

    srvctl add instance -d CLNECDB -i CLNECDB1 -n clone-rac1
    srvctl add instance -d CLNECDB -i CLNECDB2 -n clone-rac2
    srvctl start database -d CLNECDB
    srvctl status database -d CLNECDB

### 5.7 Register PDB with Clusterware

    srvctl add service -d CLNECDB -s EBSPDB_SERVICE \
      -pdb CLNEPDB \
      -preferred CLNECDB1,CLNECDB2 \
      -failovertype SELECT \
      -failovermethod BASIC \
      -failoverretry 5 \
      -failoverdelay 3

    srvctl start service -d CLNECDB -s EBSPDB_SERVICE
    srvctl status service -d CLNECDB -s EBSPDB_SERVICE

## 6. EBS 12.2 Application Tier Clone

### 6.1 Prerequisites — Database Side

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Connect to EBS PDB | `ALTER SESSION SET CONTAINER=CLNEPDB;` | oracle/sqlplus
2 | Unlock APPS schema (if locked) | `ALTER USER APPS ACCOUNT UNLOCK;` | oracle/sqlplus
3 | Reset APPS password | `EXEC APPS.FND_USER_PKG.CHANGE_USER_NAME('APPS','APPS');` | oracle/sqlplus
4 | Verify APPLSYS | `SELECT username, account_status FROM dba_users WHERE username='APPLSYS';` | oracle/sqlplus
5 | Check concurrent manager tables | `SELECT COUNT(*) FROM apps.fnd_concurrent_queues;` | oracle/sqlplus

### 6.2 Source Application Tier — Pre-Clone Steps

Run on prod-app1 as applmgr:

    # Source the EBS environment
    source /u01/app/ebsapps/EBSapps.env run

    # Prepare source for cloning
    perl $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier

⚠ NOTE: `adcfgclone.pl` is used for full environment clones (new environment, different hostname/instance name). For adding a node to an existing EBS installation, use `adclonectx.pl addnode` instead — see Section 8.7.

### 6.3 Copy Application Tier to Target Nodes

From prod-app1, copy the entire EBS base to both target app nodes:

    # Copy filesystem to clone-app1
    tar czf - /u01/app/ebsapps | ssh clone-app1 "tar xzf - -C /"

    # Copy filesystem to clone-app2
    tar czf - /u01/app/ebsapps | ssh clone-app2 "tar xzf - -C /"

⚠ NOTE: This may take significant time depending on EBS installation size (typically 100GB–300GB). Consider using `rsync` for subsequent syncs.

### 6.4 Configure Application Tier on clone-app1

Run on clone-app1 as applmgr:

    cd $COMMON_TOP/clone/bin
    perl adcfgclone.pl appsTier

When prompted, provide:

Target System Hostname
clone-app1
Target Instance Name
CLNE (new environment name)
Target DB Host
clone-scan (SCAN hostname)
Target DB Port
1521
Target DB Service Name
EBSPDB_SERVICE (or PDB service)
Target ORACLE_SID (CDB)
CLNECDB
APPS Password
<apps_password>
SYSTEM Password
<system_password>
WebLogic Admin Password
<wls_password>

### 6.5 Configure Application Tier on clone-app2

Run on clone-app2 as applmgr:

    cd $COMMON_TOP/clone/bin
    perl adcfgclone.pl appsTier

Provide the same parameters as clone-app1 but with `clone-app2` as the hostname.

### 6.6 Post-Clone EBS Configuration

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Source env on clone-app1 | `source /u01/app/ebsapps/EBSapps.env run` | applmgr
2 | Start Apache/OHS | `adapcctl.sh start` | applmgr
3 | Start OPMN | `adopmnctl.sh start` | applmgr
4 | Start Node Manager | `adnodemgrctl.sh start` | applmgr
5 | Start WLS Admin Server | `adstrtal.sh` | applmgr
6 | Start all Managed Servers | `adadminsrvctl.sh start all` | applmgr
7 | Start Concurrent Manager | `adcmctl.sh start apps/<apps_pwd>` | applmgr
8 | Verify EBS URL | `http://clone-app1.domain.com:8000/OA_HTML/AppsLocalLogin.jsp` | Browser

## 7. Post-Clone Tasks

### 7.1 EBS Database Profile Updates

    -- Connect to EBS PDB:
    ALTER SESSION SET CONTAINER=CLNEPDB;

    -- Update application node hostname in FND_NODES
    UPDATE apps.fnd_nodes SET node_name='clone-app1'
      WHERE node_name='prod-app1';
    UPDATE apps.fnd_nodes SET node_name='clone-app2'
      WHERE node_name='prod-app2';
    COMMIT;

    -- Update server host in WF_AGENTS
    UPDATE apps.wf_agents SET address=REPLACE(address,'prod-app1','clone-app1')
      WHERE address LIKE '%prod-app1%';
    COMMIT;

### 7.2 Disable/Modify Email Notifications

    -- Disable all concurrent program email notifications on clone
    UPDATE apps.fnd_svc_components
      SET component_status='STOPPED'
      WHERE component_type='WF_MAILER';
    COMMIT;

⚠ NOTE: Always disable email notifications on cloned/non-production environments to prevent accidental emails to real users.

### 7.3 Autoconfig

Run Autoconfig on both app nodes and the DB node to synchronize configuration:

    # On clone-app1 and clone-app2 as applmgr:
    perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

    # On clone-rac1 as oracle (DB tier autoconfig):
    perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

### 7.4 ETCC Verification

Run the EBS Technology Codelevel Checker to confirm patch levels:

    perl $AD_TOP/patch/115/bin/adpatch.pl options='nocopypfile' \
      patchtop=$PATCH_TOP appspass=$APPS_PWD \
      workers=8 logfile=etcc.log

### 7.5 Verify ADOP Cycle is Clean

    -- Check for stuck ADOP sessions:
    SELECT session_id, prepare_status, apply_status,
           finalize_status, cutover_status, cleanup_status
    FROM apps.ad_adop_sessions
    WHERE session_id = (SELECT MAX(session_id) FROM apps.ad_adop_sessions);

⚠ NOTE: If the source had an in-progress ADOP session, it will be replicated to the clone. Abort it on the clone: `adop phase=abort`.

## 8. Adding a New Node to Existing RAC Cluster

### 8.1 Overview

This section covers extending an existing 2-node RAC cluster by adding a third node (clone-rac3). The same process applies to adding a fourth node.

### 8.2 Prepare the New Node

Step | Task | Command / Action | Run As
--- | --- | --- | ---
1 | Install same OS version | Match patch level with existing nodes | root
2 | Configure kernel parameters | `/etc/sysctl.conf` — match existing nodes | root
3 | Create OS users/groups | oracle, grid, oinstall, dba, asmadmin etc. | root
4 | Configure /etc/hosts | Add all node hostnames + VIPs + SCAN | root
5 | Set up SSH equivalence | `ssh-keygen; ssh-copy-id` to all existing nodes | oracle/grid
6 | Verify ASM disks visible | `ls /dev/oracleasm/disks/` | root
7 | Configure NTP | Same NTP server as other nodes | root

### 8.3 Extend Grid Infrastructure to New Node

Run from an existing node (clone-rac1) as grid:

    cd $GRID_HOME/addnode
    ./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}" \
      "CLUSTER_NEW_VIRTUAL_HOSTNAMES={clone-rac3-vip}"

⚠ NOTE: This will copy Grid Infrastructure to the new node, configure CSS/CRS/OCR on the new node, and start clusterware. Monitor progress via `$GRID_HOME/cfgtoollogs/addNode/`.

### 8.4 Verify Clusterware on New Node

    # Verify from any node:
    crsctl stat res -t

    # Verify from new node:
    crsctl check cluster -all
    olsnodes -n

### 8.5 Extend Oracle Database Software to New Node

Run from clone-rac1 as oracle:

    cd $ORACLE_HOME/addnode
    ./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}"

### 8.6 Add New Instance to Database

    # Add instance 3 as oracle on clone-rac1:
    srvctl add instance -d CLNECDB -i CLNECDB3 -n clone-rac3

    # Configure instance-specific parameters:
    sqlplus / as sysdba
    SQL> ALTER SYSTEM SET instance_number=3 SID='CLNECDB3' SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET thread=3 SID='CLNECDB3' SCOPE=SPFILE;
    SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS3' SID='CLNECDB3' SCOPE=SPFILE;
    SQL> EXIT;

    # Create undo tablespace for node 3:
    sqlplus / as sysdba
    SQL> CREATE UNDO TABLESPACE UNDOTBS3
      DATAFILE '+DATA/CLNECDB/undotbs3_01.dbf' SIZE 2G AUTOEXTEND ON;

    # Add redo thread for node 3:
    SQL> ALTER DATABASE ADD LOGFILE THREAD 3
      GROUP 7 ('+REDO/CLNECDB/redo07a.log','+DATA/CLNECDB/redo07b.log') SIZE 512M,
      GROUP 8 ('+REDO/CLNECDB/redo08a.log','+DATA/CLNECDB/redo08b.log') SIZE 512M,
      GROUP 9 ('+REDO/CLNECDB/redo09a.log','+DATA/CLNECDB/redo09b.log') SIZE 512M;
    SQL> ALTER DATABASE ENABLE PUBLIC THREAD 3;
    SQL> EXIT;

    # Start instance on new node:
    srvctl start instance -d CLNECDB -i CLNECDB3
    srvctl status database -d CLNECDB

### 8.7 Add New EBS App Node

In standard EBS 12.2 deployments, the application filesystem (APPL_TOP, COMMON_TOP, OA_HTML) is on shared NFS/NAS storage mounted by all app nodes. Adding a new app node does NOT require copying files — the new node mounts the existing shared filesystem. Only INST_TOP (instance-specific config, logs, temp) is local to each node.

⚠ NOTE: If your EBS is deployed on local disk (non-shared, e.g. DMZ environments), you will need to `rsync`/`tar` the filesystem to the new node first. For standard shared NFS deployments, skip directly to Step 1 below.

Run on clone-app3 as applmgr:

    # Step 1: Mount shared EBS filesystems on new node (as root):
    mount -t nfs nfs-server:/exports/ebsapps /u01/app/ebsapps
    mount -t nfs nfs-server:/exports/common  /u01/app/common
    # Add permanent entries to /etc/fstab

    # Step 2: Create local INST_TOP directory:
    mkdir -p /u01/app/inst/apps/CLNE_clone-app3

    # Step 3: Source environment from existing node context:
    source /u01/app/ebsapps/EBSapps.env run

    # Step 4: Create node-specific context file using adclonectx.pl:
    perl $AD_TOP/bin/adclonectx.pl addnode \
      contextfile=$CONTEXT_FILE \
      pairsfile=/tmp/clone-app3_pairs.txt

The pairs file (`/tmp/clone-app3_pairs.txt`) contains node-specific overrides:

    s_hostname=clone-app3
    s_local_hostname=clone-app3
    s_webentryhost=clone-app3
    s_active_webport=8000
    s_tools_oracle_home=/u01/app/ebsapps/fs1/EBSapps/10.1.2

    # Step 5: Run Autoconfig on new node using new context file:
    perl $AD_TOP/bin/adconfig.pl contextfile=/u01/app/inst/apps/CLNE_clone-app3/appl/admin/CLNE_clone-app3.xml

    # Step 6: Start services on new node:
    adapcctl.sh start
    adopmnctl.sh start
    adnodemgrctl.sh start
    adstrtal.sh

    # Step 7: Register new app node in FND_NODES (if not auto-registered):
    INSERT INTO apps.fnd_nodes (node_name, server_id, support_cp,
      support_forms, support_web, platform_code)
    VALUES ('clone-app3', apps.fnd_nodes_s.nextval,
      'Y', 'Y', 'Y', 'LINUX');
    COMMIT;

## 9. Troubleshooting Common Issues

### 9.1 RMAN Duplicate Failures

Symptom | Resolution
--- | ---
ORA-17628: Oracle error 19505 | ASM directory for target does not exist. Create dirs with `asmcmd mkdir`.
RMAN-05001: auxiliary channel not started | Auxiliary listener not running or TNS not reachable. Check `tnsnames.ora` and `listener.ora` on target.
ORA-12154: TNS could not resolve | Add PRODCDB and CLNECDB_AUX to `tnsnames.ora` on source and target.
Duplicate hangs after datafile copy | Check network bandwidth. Use fewer parallel channels if bandwidth limited.

### 9.2 RAC Startup Issues

Symptom | Resolution
--- | ---
ORA-29701: unable to connect to Cluster Manager | Clusterware not running. Run: `crsctl start cluster -all`
Instance stuck in MOUNT state | Redo thread not enabled. `ALTER DATABASE ENABLE PUBLIC THREAD N;`
UNDO errors on node 2 | Wrong undo tablespace assigned. Check SPFILE parameter `undo_tablespace` for SID.
CSS daemon not starting on new node | OCR or voting disk not accessible. Check ASM disk group visibility.

### 9.3 EBS Clone Issues

Symptom | Resolution
--- | ---
adcfgclone.pl fails on context file | Run `adpreclone.pl` on source first. For node addition use `adclonectx.pl addnode` instead of `adcfgclone.pl`.
WLS Admin Server not starting | Check `$DOMAIN_HOME/servers/AdminServer/logs/`. Often Java heap or port conflict.
Login page not loading after clone | Run Autoconfig. Check Apache/OHS config in `$INST_TOP/ora/10.1.3/Apache/Apache/conf/`.
FND_NODES mismatch | Update `fnd_nodes` with new hostnames. Run Autoconfig after.
Concurrent Manager not starting | Check ICM log in `$APPLCSF/$APPLLOG`. Usually APPS password or FND_NODES issue.

## 10. Post-Clone Validation Checklist

✓ | Task | Command / Action | Run As
--- | --- | --- | ---
✓ | DB instances running on both nodes | `srvctl status database -d CLNECDB` | oracle
✓ | All PDBs open | `SELECT name, open_mode FROM v$pdbs;` | oracle/sqlplus
✓ | SCAN listener working | `tnsping EBSPDB_SERVICE` | oracle
✓ | EBS login page accessible | `http://clone-app1:8000/OA_HTML/AppsLocalLogin.jsp` | Browser
✓ | Concurrent Manager running | `adcmctl.sh status apps/<pwd>` | applmgr
✓ | No invalid objects | `SELECT COUNT(*) FROM dba_objects WHERE status='INVALID';` | oracle/sqlplus
✓ | Archive log mode verified | `ARCHIVE LOG LIST;` | oracle/sqlplus
✓ | Email notifications disabled | Check `fnd_svc_components` WF_MAILER status | oracle/sqlplus
✓ | Autoconfig completed on all nodes | Check autoconfig log for errors | applmgr
✓ | ADOP session clean (no stuck sessions) | Check `ad_adop_sessions` | oracle/sqlplus
✓ | Tablespace usage normal | Check `dba_tablespace_usage_metrics` | oracle/sqlplus
✓ | Alert log clean on all nodes | `tail -100 alert_CLNECDB*.log` | oracle

### Run a health check on your cloned environment

After a RAC clone, TuneVault runs 200+ automated checks — invalid objects, tablespace usage, ADOP session state, listener health, and EBS operations — so you can validate the clone in minutes instead of eyeballing SQL by hand.

Run Your Free Health Check →
KKA
Kiran Kumar Ale

20+ years Oracle EBS/DBA/Cloud. 23 Oracle & PM certifications across DBA, EBS, Cloud Infrastructure, and Security tracks. Built TuneVault to encode two decades of production Oracle expertise into an automated health check tool.

linkedin.com/in/kirankumarale
hello@tunevault.app
linuxappsdba.blogspot.com