🏗️ EBS Upgrade

# EBS Upgrade / Technology Stack Upgrade Triage Runbook

⏱ 2–24 hrs (upgrade scope dependent) EBS 12.1 / 12.2 Updated 2026-05-15
🔴 First: determine where the failure happened
The fix for an ADOP failure is completely different from an AutoConfig failure or a WebLogic startup failure. Before acting, identify which phase failed and read its log. Acting on the wrong phase wastes hours.

## 1. EBS 12.1 → 12.2 Upgrade Path Overview

EBS 12.1→12.2 is a major upgrade involving both the application technology stack (WebLogic replaces Oracle AS, Forms 11g, ADOP replaces adpatch) and the patching model (online patching with dual filesystem). It is not a patch — it is a full upgrade requiring a maintenance window of typically 8–24 hours.

ℹ️ AD/TXK delta baseline is mandatory
Before beginning the 12.2 upgrade, you must apply the latest AD and TXK minipack on the 12.1 system. Check MOS Note 1315333.1 for the current baseline requirement. Skipping this causes immediate upgrade failures.

Phase | What happens | Duration (typical)
--- | --- | ---
Pre-upgrade steps | AD/TXK delta, DB upgrade to 12c/19c, preinstall patches | 2–4 hrs
Upgrade start | Run u12201061.drv (12.2 upgrade driver) — schema changes, data migration | 4–12 hrs
Technology stack install | Install WebLogic, Forms 11g, OHS; configure domains | 1–2 hrs
AutoConfig | First AutoConfig under new tech stack | 15–30 min
ADOP cutover model | Switch to dual-filesystem (run/patch editions) | 30 min
Post-upgrade patches | Mandatory post-upgrade patches per MOS 1594222.1 | 1–3 hrs

## 2. ADOP Phase Failures

ADOP (AD Online Patching) replaced adpatch in EBS 12.2. It operates in five sequential phases. A failure in any phase requires targeted recovery — not a full restart.

    -- Check the status of the current ADOP cycle
    SELECT
      patch_run_id,
      phase,
      status,
      start_date,
      end_date,
      workers,
      log_file
    FROM ad_adop_session_patches
    ORDER BY patch_run_id DESC
    FETCH FIRST 10 ROWS ONLY;

    -- Check current ADOP session
    SELECT
      adop_session_id,
      node_name,
      status,
      start_date,
      end_date
    FROM ad_adop_sessions
    ORDER BY start_date DESC
    FETCH FIRST 5 ROWS ONLY;

### prepare phase failure

    -- prepare creates the patch edition and synchronizes filesystems
    -- Log: $APPL_TOP/../log/adop_<session>/prepare_<node>.log

    -- Common causes: stale patch edition from prior failed cycle
    -- Fix: cleanup first, then re-prepare
    adop phase=cleanup
    adop phase=prepare

### apply phase failure

    -- apply runs patch workers; a failed worker does NOT stop the whole apply
    -- Find failed workers in log directory
    -- ls $APPL_TOP/../log/adop_<session>/apply_<node>/

    -- Restart a failed apply from where it left off
    adop phase=apply patches=<patch_number> restart=yes

    -- If a specific worker keeps failing, check its log
    -- cat $APPL_TOP/../log/adop_<session>/apply_<node>/adworker_<N>.log

    -- Check for worker errors in DB
    SELECT
      w.worker_id,
      w.status,
      w.log_file,
      j.unit_id,
      j.action_code,
      j.message
    FROM ad_parallel_workers w
    JOIN ad_parallel_jobs j ON w.worker_id = j.worker_id
    WHERE j.status IN ('F', 'R')
    ORDER BY j.last_update_date DESC;

### finalize phase failure

    -- finalize compiles invalid objects and runs post-patch tasks
    -- Log: $APPL_TOP/../log/adop_<session>/finalize_<node>.log

    -- Re-run finalize (safe to re-run)
    adop phase=finalize

    -- If finalize fails on a specific object compilation, find it in DB
    SELECT owner, object_name, object_type, status
    FROM dba_objects
    WHERE status = 'INVALID'
      AND owner NOT IN ('SYS', 'SYSTEM')
    ORDER BY owner, object_type, object_name;

### cutover phase failure

    -- cutover switches the run edition to the patched edition
    -- It requires all users to be out of the system (check active sessions)

    -- Check active DB sessions before cutover
    SELECT username, COUNT(*) AS sessions
    FROM v$session
    WHERE username IS NOT NULL
      AND username NOT IN ('SYS', 'SYSTEM', 'DBSNMP')
    GROUP BY username
    ORDER BY sessions DESC;

    -- Run cutover
    adop phase=cutover

    -- If cutover fails midway: check the log, restart is NOT supported
    -- You must abort and cleanup, then restart the entire patch cycle
    -- See Section 3 for abort recovery

### cleanup phase failure

    -- cleanup drops the old patch edition objects and removes stale filesystem content
    -- Safe to re-run if it fails
    adop phase=cleanup

    -- If cleanup hangs on edition drop (large schemas take time)
    -- Check for blocking locks
    SELECT
      s.sid, s.serial#, s.username, s.status,
      l.type, l.request, l.block
    FROM v$session s
    JOIN v$lock l ON s.sid = l.sid
    WHERE l.block = 1;

    -- Force cleanup if it's been stuck > 2 hours
    adop phase=cleanup mode=full

## 3. Failed Cutover Recovery

🔴 Failed cutover is the worst-case scenario
If cutover fails mid-execution, EBS is in an inconsistent state — neither the old nor the new edition is fully active. The only safe path is `adop phase=abort`, then a fresh patch cycle. Do NOT attempt to restart cutover.

    -- Step 1: Abort the failed adop session
    adop phase=abort

    -- Step 2: If abort also fails (stuck sessions), kill them in DB
    SELECT sid, serial#, username, status, schemaname
    FROM v$session
    WHERE schemaname LIKE '%_PATCH%'
       OR schemaname LIKE '%_NEXT%';

    ALTER SYSTEM KILL SESSION '&sid,&serial' IMMEDIATE;

    -- Step 3: Run cleanup to remove the failed patch edition
    adop phase=cleanup

    -- Step 4: Verify no patch editions remain
    SELECT edition_name, usable
    FROM dba_editions
    ORDER BY edition_name;
    -- Should show only one usable edition (the run/base edition)

    -- Step 5: Verify filesystem is clean
    -- ls $APPL_TOP_NE  (should be minimal/empty after cleanup)

    -- Step 6: Now safe to start a new patch cycle
    adop phase=prepare

### Cleaning stuck ADOP sessions in the database

    -- ADOP session state in DB (check ad_adop_sessions)
    SELECT
      adop_session_id,
      node_name,
      status,
      start_date
    FROM ad_adop_sessions
    WHERE status NOT IN ('C', 'F')
    ORDER BY start_date DESC;

    -- Mark stuck session as aborted (last resort if adop phase=abort fails)
    UPDATE ad_adop_sessions
    SET status = 'F'  -- F = Failed/Aborted
    WHERE status = 'R'  -- R = Running
      AND adop_session_id = &session_id;
    COMMIT;

## 4. Concurrent Manager Not Starting Post-Upgrade

    -- Check CM status
    SELECT
      concurrent_queue_name,
      manager_type,
      running_processes,
      max_processes,
      target_node,
      last_refresh_date
    FROM fnd_concurrent_queues_vl
    WHERE enabled_flag = 'Y'
    ORDER BY concurrent_queue_name;

### ICM (Internal Concurrent Manager) won't start

    -- Check APPS password is correct in FND_USER
    SELECT user_name, encrypted_user_password, last_logon_date
    FROM fnd_user
    WHERE user_name = 'APPS';

    -- Reset APPS password if needed (requires SYSADMIN login)
    -- Navigate to: Security → User → Define → APPS user → change password
    -- OR use FNDCPASS utility:
    FNDCPASS apps/apps_password 0 Y system/manager_password ALLORACLE APPS new_password

    -- Verify $FND_TOP/bin/FNDLIBR is executable
    -- ls -la $FND_TOP/bin/FNDLIBR

    -- Check service startup log
    -- cat $APPLCSF/$APPLLOG/STARTCM.log (11.5) or check admin/log/

    -- Force CM restart
    -- As applmgr: adcmctl.sh start apps/<password>

### CM starts but workers fail immediately

    -- Check for environment issues — AutoConfig may not have run after upgrade
    -- Verify correct ORACLE_HOME and ORACLE_SID are set in the Apps tier env
    -- source $APPL_TOP/envshell.env (or APPS<SID>_<host>.env)

    -- Check for stale NLS_CHARACTERSET mismatches
    SELECT parameter, value
    FROM nls_database_parameters
    WHERE parameter IN ('NLS_CHARACTERSET', 'NLS_NCHAR_CHARACTERSET');

    -- Verify FND_INSTALL_PROCESSES table is clean
    SELECT * FROM fnd_install_processes
    WHERE status IN ('I', 'R')
    ORDER BY last_updated_date DESC;

## 5. AutoConfig Failures During Technology Stack Patching

    -- AutoConfig log location
    -- $APPL_TOP/admin/<SID>/log/<date>/adconfig.txt
    -- For apps tier: $INST_TOP/admin/log/<date>/adconfig.txt

    -- Run AutoConfig manually (as applmgr)
    perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

    -- If AutoConfig fails on database tier
    perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE run=DB

### Common AutoConfig failure: context file out of date

    -- After major upgrade, context file may have missing or wrong values
    -- Run adbldxml.pl to rebuild context file
    perl $AD_TOP/bin/adbldxml.pl \
      appspass=apps_password \
      contextfile=$CONTEXT_FILE \
      xmlfile=/path/to/old_context.xml

    -- Validate context file
    perl $AD_TOP/bin/adchkutl.pl contextfile=$CONTEXT_FILE

### AutoConfig fails on WebLogic domain config

    -- Check MW_HOME and WL_HOME are set correctly in context
    -- grep 'MW_HOME\|WL_HOME' $CONTEXT_FILE

    -- WebLogic domain config backup before AutoConfig
    -- cp -r $INST_TOP/ora/10.1.3/ $INST_TOP/ora/10.1.3.bak/

    -- If AutoConfig corrupted domain: restore from backup and re-run
    -- rm -rf $INST_TOP/ora/10.1.3/
    -- cp -r $INST_TOP/ora/10.1.3.bak/ $INST_TOP/ora/10.1.3/
    perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

## 6. WebLogic / Forms / OHS Upgrade Issues

### AdminServer won't start after upgrade

    -- Check AdminServer log
    -- tail -100 $DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log
    -- tail -100 $DOMAIN_HOME/servers/AdminServer/logs/AdminServer-diagnostic.log

    -- Common cause: port conflict (AdminServer default: 7001)
    -- lsof -i :7001 (or netstat -tlnp | grep 7001)

    -- Common cause: boot.properties missing or corrupted
    -- Recreate boot.properties:
    -- mkdir -p $DOMAIN_HOME/servers/AdminServer/security/
    -- cat > $DOMAIN_HOME/servers/AdminServer/security/boot.properties <<EOF
    -- username=weblogic
    -- password=<plain_text_password>
    -- EOF
    -- WLS will encrypt it on next start

    -- Start AdminServer manually
    -- $DOMAIN_HOME/bin/startWebLogic.sh > /tmp/admin.log 2>&1 &

### NodeManager failures

    -- NodeManager manages Managed Server lifecycle
    -- Log: $MW_HOME/oracle_common/common/nodemanager/nodemanager.log

    -- Check if NodeManager is running
    -- ps -ef | grep NodeManager

    -- Restart NodeManager
    -- cd $MW_HOME/oracle_common/common/bin
    -- ./startNodeManager.sh > /tmp/nm.log 2>&1 &

    -- If NodeManager refuses to enroll a machine
    -- Check nodemanager.properties: ListenPort, ListenAddress
    -- cat $WL_HOME/server/lib/nodemanager.properties | grep Listen

### OHS (Oracle HTTP Server) won't start

    -- OHS log location
    -- $INST_TOP/ora/10.1.3/Apache/Apache/logs/error_log

    -- Check port conflicts (default OHS: 8000 / 443)
    -- netstat -tlnp | grep -E '8000|443|7777'

    -- Restart OHS
    $ADMIN_SCRIPTS_HOME/adopmnctl.sh start
    $ADMIN_SCRIPTS_HOME/adohactl.sh start

    -- If OHS config is corrupt after AutoConfig: check httpd.conf
    -- diff $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf \
    --      $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf.bkXXXX

## 7. EBS R12.2 Online Patching Troubleshooting

EBS 12.2 uses an online patching model (Edition-Based Redefinition) — patches apply against the patch edition while users work in the run edition. The two editions share the same data but have separate code objects.

    -- Check current run and patch editions
    SELECT
      property_name,
      property_value
    FROM fnd_product_installations
    WHERE property_name LIKE '%EDITION%';

    -- Check edition objects are in sync
    SELECT
      object_name, object_type,
      edition_name,
      status
    FROM dba_edition_obj_audit_trail
    WHERE status = 'INVALID'
    ORDER BY object_name;

### Edition drift — run edition has invalid objects

    -- Check run edition objects for invalids
    SELECT owner, object_name, object_type, status
    FROM dba_objects
    WHERE status = 'INVALID'
      AND owner IN ('APPS', 'APPLSYS')
    ORDER BY object_type, object_name;

    -- Force recompile in APPS schema
    EXEC DBMS_UTILITY.COMPILE_SCHEMA(schema=>'APPS', compile_all=>FALSE);
    EXEC DBMS_UTILITY.COMPILE_SCHEMA(schema=>'APPLSYS', compile_all=>FALSE);

## 8. AD Utilities — Common Failure Modes

### adadmin — Maintain Applications Database Objects

    -- adadmin log: $APPL_TOP/admin/<SID>/log/adadmin<N>.log

    -- Run adadmin to recompile invalid objects (menu option)
    -- adadmin -> Maintain Applications Database Objects ->
    --           Compile Invalid Objects in the APPS Schema

    -- adadmin fails with ORA-01017 (invalid username/password)
    -- Usually APPS schema password needs reset
    -- alter user apps identified by <password>;

    -- Check adadmin worker log for specific errors
    -- ls $APPL_TOP/admin/<SID>/log/adwork*.log | xargs grep "Error\|ORA-" | tail -20

### adpatch — applying patches on 12.1 or older 12.2

    -- adpatch log: $APPL_TOP/admin/<SID>/log/adpatch.log

    -- Check if previous adpatch session needs to be completed
    SELECT
      patch_run_id,
      phase,
      status,
      start_date
    FROM ad_patch_runs
    ORDER BY start_date DESC
    FETCH FIRST 5 ROWS ONLY;

    -- If adpatch stopped mid-run, restart it
    -- adpatch will detect the incomplete run and offer to restart
    adpatch

    -- Check for failed workers in a previous run
    SELECT
      w.worker_id, w.status,
      j.action_code, j.message
    FROM ad_parallel_workers w
    JOIN ad_parallel_jobs j ON w.worker_id = j.worker_id
    WHERE j.status = 'F'
    ORDER BY j.last_update_date DESC;

## 9. Post-Upgrade Validation

    -- 1. Verify FND_INSTALL_PROCESSES shows successful upgrade steps
    SELECT
      install_group_num,
      status,
      last_update_date,
      product_short_name,
      creation_date
    FROM fnd_install_processes
    WHERE status NOT IN ('S')
    ORDER BY last_update_date DESC
    FETCH FIRST 20 ROWS ONLY;

    -- 2. Verify AD_BUGS for applied patches
    SELECT
      bug_number,
      bug_name,
      last_update_date
    FROM ad_bugs
    WHERE bug_number IN (
      '12345678'  -- replace with your critical patch numbers
    )
    ORDER BY last_update_date;

    -- 3. Verify key product versions post-upgrade
    SELECT
      product_short_name,
      release_name,
      patch_level,
      install_status
    FROM fnd_product_installations
    WHERE install_status != 'N'
    ORDER BY product_short_name;

    -- 4. Check AD patch history for missing drivers
    SELECT
      patch_name,
      patch_type,
      creation_date
    FROM ad_applied_patches
    ORDER BY creation_date DESC
    FETCH FIRST 20 ROWS ONLY;

    -- 5. Verify AutoConfig ran with no errors after upgrade
    -- grep "ERROR\|FAILED" $INST_TOP/admin/log/*/adconfig.txt | tail -20

## 10. Apps Tier Cloning Issues Post-Upgrade

    -- After upgrading, cloning requires updated clone scripts

    -- Step 1: On source (post-upgrade) system, run adpreclone
    -- perl $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier
    -- perl $ADMIN_SCRIPTS_HOME/adpreclone.pl dbTier

    -- Step 2: Copy filesystem to target

    -- Step 3: On target, run adcfgclone
    -- perl $CLONE_TOP/bin/adcfgclone.pl appsTier /path/to/context_template.xml

    -- Common post-upgrade clone failure: outdated $CLONE_TOP scripts
    -- The clone scripts in $AD_TOP/clone must be updated to EBS 12.2 versions
    -- Verify: grep 'version' $AD_TOP/clone/bin/adcfgclone.pl | head -3

    -- If adcfgclone fails on context file: rebuild context first
    perl $AD_TOP/bin/adbldxml.pl \
      contextfile=$NEW_CONTEXT_FILE \
      templatefile=$AD_TOP/admin/template/<template>.xml

## 11. Common Errors

### AD-A-01022: Unable to determine the version of file

    -- adop/adpatch can't determine a file's version
    -- Root cause: file header missing, binary file, or encoding issue

    -- Identify the problem file from adpatch/adop log
    -- grep "AD-A-01022" $APPL_TOP/admin/<SID>/log/adpatch.log

    -- Option 1: Force-apply by skipping version check
    -- adop phase=apply patches=<N> options=nofileversioncheck
    -- adpatch: use 'options=nofileversioncheck' flag

    -- Option 2: Manually update the file version in FND tables
    SELECT * FROM fnd_application_files
    WHERE file_name = '&filename';

    -- Option 3: Copy clean file from another server running same patch level

### ADZDCLN errors during cleanup

    -- ADZDCLN is the PL/SQL cleanup package for edition objects
    -- Errors usually mean dependent objects are blocking edition drop

    -- Check for objects dependent on the patch edition
    SELECT
      owner, name, type,
      referenced_owner, referenced_name, referenced_type
    FROM dba_dependencies
    WHERE referenced_owner LIKE '%_NEXT'
       OR referenced_owner LIKE '%_PATCH'
    FETCH FIRST 20 ROWS ONLY;

    -- If cleanup keeps failing on the same object, check MOS for a bug
    -- Search: "ADZDCLN [object_name] 12.2" on support.oracle.com

    -- Force cleanup with verbose logging
    adop phase=cleanup loglevel=statement

### OUI-67076: OPatch failed — inventory not found

    -- OPatch can't find the Oracle Inventory
    -- Check inventory pointer
    -- cat /etc/oraInst.loc

    -- Verify inventory exists
    -- ls -la $ORACLE_INVENTORY/ContentsXML/inventory.xml

    -- Re-attach Oracle home to inventory
    $ORACLE_HOME/oui/bin/runInstaller -silent -ignoreSysPrereqs \
      -attachHome \
      ORACLE_HOME=$ORACLE_HOME \
      ORACLE_HOME_NAME=OraDB19Home1

    -- Validate the home is now visible to OPatch
    $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME

✅ Post-upgrade EBS checklist

- AutoConfig completed with no errors on all tiers
- Internal Concurrent Manager running and processing requests
- ADOP cleanup completed — single edition in dba_editions
- FND_INSTALL_PROCESSES shows no FAILED steps
- Invalid objects in APPS/APPLSYS schemas = 0 after adadmin compile
- WebLogic AdminServer, OHS, and all Managed Servers started
- Applications URL accessible and login works
- Run a sample concurrent request end-to-end to validate CM

### Know your EBS patch level and ADOP cycle status in one click

TuneVault health checks surface ADOP cycle state, failed workers, CM status, invalid objects, and missing mandatory patches — before they become upgrade blockers.

Run a Free Health Check →