메뉴 건너뛰기

Korea Oracle User Group

Backup & Recovery

12C RAC, OS Fault 로 다운된 1번 Node 재구성

Talros 2020.08.28 16:53 조회 수 : 1117

 

1.  구성 환경 

 

 OS : 리눅스  Redhat 7.2

 Oracle Version : 12.1

 Type : 2 Node RAC

 

 

2. 장애 상황 

 

- 1번 노드의 OS Fault 로 복구가 불가능 해짐

 

- 2번 노드는 정상 적으로 살아 있음

 

- GRID , ORACLE  Engine 이 설치된 볼륨은 복구를 해서 재사용 가능

 

- 재구성 방안

  

  1. GRID 만 Relink 로 재구성

  

  2.  1번 재구성 실패시 GRID만 삭제 후 인스톨 , 백업 받은 OCR 로 리소스 복구

 

 

 

3-1 . OCR 백업 

 

[root@testsvr2 ocr_backup]# 

[root@testsvr2 ocr_backup]# /oracle/grid/bin/ocrconfig -export /home/oracle/DBA/work/ocr_backup/ocr_backup_20200810.bak

[root@testsvr2 ocr_backup]# ls -l

total 92

-rw------- 1 root root 93966 Aug 10 10:52 ocr_backup_20200810.bak

[root@testsvr2 ocr_backup]# 

 

 

3-2. 2번 노드 서비스 정지

 

# 2번 노드 서비스 종료 ( 1번 노드 relink 에 영향은 없음  ) 

 

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> srvctl stop database -d datadbL

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> crsctl stat res -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details       

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.CRS.dg

               ONLINE  ONLINE       testsvr2                 STABLE

ora.DATA.dg

               ONLINE  ONLINE       testsvr2                 STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       testsvr2                 STABLE

ora.asm

               ONLINE  ONLINE       testsvr2                 Started,STABLE

ora.net1.network

               ONLINE  ONLINE       testsvr2                 STABLE

ora.ons

               ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.MGMTLSNR

      1        OFFLINE OFFLINE                               STABLE

ora.testdbl.db

      1        ONLINE  OFFLINE                               STABLE

      2        ONLINE  ONLINE       testsvr2                 Open,STABLE

ora.datadbl.db

      1        OFFLINE OFFLINE                               STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.cvu

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.oc4j

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.testsvr1.vip

      1        ONLINE  INTERMEDIATE testsvr2                 FAILED OVER,STABLE

ora.testsvr2.vip

      1        ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> ps -ef|gre psmon

bash: gre: command not found...

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> ps -ef|grep smon

oracle   44864     1  0 Jan03 ?        01:09:35 ora_smon_testdbL2

oracle   49246     1  0  2019 ?        00:09:42 asm_smon_+ASM2

root     49272     1  1  2019 ?        3-14:38:44 /oracle/grid/bin/osysmond.bin

oracle   61254 40816  0 14:40 pts/0    00:00:00 grep --color=auto smon

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup>  srvctl stop database -d testdbL -o immediate

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup>  srvctl stop database -d datadbl -o immediate

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> crsctl stat res -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details       

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.CRS.dg

               ONLINE  ONLINE       testsvr2                 STABLE

ora.DATA.dg

               ONLINE  ONLINE       testsvr2                 STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       testsvr2                 STABLE

ora.asm

               ONLINE  ONLINE       testsvr2                 Started,STABLE

ora.net1.network

               ONLINE  ONLINE       testsvr2                 STABLE

ora.ons

               ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.MGMTLSNR

      1        OFFLINE OFFLINE                               STABLE

ora.testdbl.db

      1        OFFLINE OFFLINE                               STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.datadbl.db

      1        OFFLINE OFFLINE                               STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.cvu

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.oc4j

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       testsvr2                 STABLE

ora.testsvr1.vip

      1        ONLINE  INTERMEDIATE testsvr2                 FAILED OVER,STABLE

ora.testsvr2.vip

      1        ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> 



# root 유저로 접속 GRID Service shutdown 



[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> su -

Password: 

Last login: Mon Aug 10 10:52:26 KST 2020 on pts/0

[root@testsvr2 ~]# 

[root@testsvr2 ~]# 

[root@testsvr2 ~]# 

[root@testsvr2 ~]# /oracle/grid/bin/crsctl stop crs

[root@testsvr2 bin]# /oracle/grid/bin/crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testsvr2'

CRS-2673: Attempting to stop 'ora.crsd' on 'testsvr2'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'testsvr2'

CRS-2673: Attempting to stop 'ora.CRS.dg' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.testsvr2.vip' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.cvu' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.oc4j' on 'testsvr2'

CRS-2677: Stop of 'ora.cvu' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.testsvr2.vip' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.testsvr1.vip' on 'testsvr2'

CRS-2677: Stop of 'ora.CRS.dg' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'testsvr2'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'testsvr2'

CRS-2677: Stop of 'ora.testsvr1.vip' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'testsvr2'

CRS-2677: Stop of 'ora.asm' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.oc4j' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'testsvr2'

CRS-2677: Stop of 'ora.ons' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'testsvr2'

CRS-2677: Stop of 'ora.net1.network' on 'testsvr2' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'testsvr2' has completed

CRS-2677: Stop of 'ora.crsd' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.storage' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testsvr2'

CRS-2677: Stop of 'ora.storage' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.ctssd' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.evmd' on 'testsvr2'

CRS-2673: Attempting to stop 'ora.asm' on 'testsvr2'

CRS-2677: Stop of 'ora.ctssd' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.crf' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.evmd' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'testsvr2' succeeded

CRS-2677: Stop of 'ora.asm' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'testsvr2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'testsvr2'

CRS-2677: Stop of 'ora.cssd' on 'testsvr2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'testsvr2'

CRS-2677: Stop of 'ora.gipcd' on 'testsvr2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testsvr2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@testsvr2 bin]# 

[root@testsvr2 bin]# ps -ef|grep ohas

root     30184     1  0  2019 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

root     63242 62147  0 14:45 pts/0    00:00:00 grep --color=auto ohas

[root@testsvr2 bin]# 

 

 

4. 1번 노드 Relink 

 # 복구 해야할 1번 노드로 root 로 접속

 

[root@testsvr1 grid]# pwd

/oracle/grid

[root@testsvr1 grid]# cd /oracle/grid/crs/install

[root@testsvr1 install]# ls

appcluster.pl            crska.pm       dropdb.pl           oracle-ohasd.service        s_crsconfig_defs

cmdllroot.sh             crsoc4j.pm     inittab             oracss.pm                   s_crsconfig_testsvr1_env.txt

crsconfig_addparams.sbs  crsohasd.pm    install.excl        paramfile.crs               s_crsutils.pm

crsconfig_params         crspatch.pm    install.incl        ParentDirPerm_testsvr1.txt  tfa_setup

crsconfig_params.sbs     crstfa.pm      installRemove.excl  rootcrs.pl

crsdeconfig.pm           crsupgrade.pm  oraacfs.pm          rootcrs.sh

crsgpnp.pm               crsutils.pm    oraafd.pm           roothas.pl

crsinstall.pm            dropdb         oracle-ohasd.conf   roothas.sh

 

# grid home unlock

 


[root@testsvr1 install]# export ORACLE_HOME=/oracle/grid

[root@testsvr1 install]# perl rootcrs.pl -unlock -crshome /oracle/grid

Using configuration parameter file: ./crsconfig_params

2020/08/10 15:18:35 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.



2020/08/10 15:20:14 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.



2020/08/10 15:20:14 CLSRSC-347: Successfully unlock /oracle/grid



[root@testsvr1 install]# 
 

 

 

# oracle user execute relink

 


[+ASM1:oracle@testsvr1]export ORACLE_HOME=/oracle/grid

[+ASM1:oracle@testsvr1] /oracle/grid/bin> /oracle/grid/bin/relink all

writing relink log to: /oracle/grid/install/relink.log

[+ASM1:oracle@testsvr1] /oracle/grid/bin> 
 

 

 

# root user 로 접속 Relink 작업 마무리

 

[root@testsvr1 install]# cd /oracle/grid/rdbms/install

[root@testsvr1 install]# pwd

/oracle/grid/rdbms/install

[root@testsvr1 install]# ls

clonedb.pl          filemap.ora   rdbms               rootadd_rdbms.sh   sbs

config_filemap.sbs  install.excl  rootadd_filemap.sh  s7x_workflow.akwf  setup_processor_group.sh

[root@testsvr1 install]# 

[root@testsvr1 install]# ./rootadd_rdbms.sh

[root@testsvr1 install]# 



[root@testsvr1 install]# cd /oracle/grid/crs/install

[root@testsvr1 install]# pwd

/oracle/grid/crs/install

[root@testsvr1 install]# 

[root@testsvr1 install]# ls

appcluster.pl            crsgpnp.pm     crsupgrade.pm  installRemove.excl    ParentDirPerm_testsvr1.txt    s_crsutils.pm

cmdllroot.sh             crsinstall.pm  crsutils.pm    oraacfs.pm            rootcrs.pl                    tfa_setup

crsconfig_addparams.sbs  crska.pm       dropdb         oraafd.pm             rootcrs.sh

crsconfig_params         crsoc4j.pm     dropdb.pl      oracle-ohasd.conf     roothas.pl

crsconfig_params.saved   crsohasd.pm    inittab        oracle-ohasd.service  roothas.sh

crsconfig_params.sbs     crspatch.pm    install.excl   oracss.pm             s_crsconfig_defs

crsdeconfig.pm           crstfa.pm      install.incl   paramfile.crs         s_crsconfig_testsvr1_env.txt

[root@testsvr1 install]# perl rootcrs.pl -patch

Using configuration parameter file: ./crsconfig_params

[root@testsvr1 install]# 
 

 

 

5.  CRS de-config , 기존 Engine 에 남아 있는 crs 설치 정보 강제 삭제

 

   # OS 영역에 init 정보와 같은 설치 정보가 없어 relink 만으로는 실행이 안됨

 

[root@testsvr1 install]# /oracle/grid/crs/install/rootcrs.pl -deconfig -force

Using configuration parameter file: /oracle/grid/crs/install/crsconfig_params

Usage: srvctl <command> <object> [<options>]

    commands: enable|disable|start|stop|status|add|remove|modify|update|getenv|setenv|unsetenv|config|upgrade|downgrade

    objects: database|service|asm|diskgroup|listener|home|ons

For detailed help on each command and object and its options use:

  srvctl <command> -help [-compatible] or

  srvctl <command> <object> -help [-compatible]

PRKO-2012 : nodeapps object is not supported in Oracle Restart

CRS-4046: Invalid Oracle Clusterware configuration.

CRS-4000: Command Stop failed, or completed with errors.

2020/08/10 15:46:24 CLSRSC-463: The deconfiguration or downgrade script could not stop current Oracle Clusterware stack.



2020/08/10 15:46:24 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.



2020/08/10 15:46:52 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.



Failure in execution (rc=-1, 0, No such file or directory) for command /etc/init.d/ohasd deinstall

2020/08/10 15:46:53 CLSRSC-557: Oracle Clusterware stack on this node has been successfully deconfigured. There were some errors which can be ignored.



[root@testsvr1 install]# 

 

6. CRS 데몬 재구성  , root.sh 실행

 

[root@testsvr1 grid]# ls

addnode      crf     deinstall    gnsd           javavm  log      olap         osysmond  racg            slax      utl

assistants   crs     demo         gpnp           jdbc    md       ologgerd     oui       rdbms           sqlpatch  wlm

auth         css     diagnostics  has            jdk     mdns     OPatch       owm       relnotes        sqlplus   wwg

bin          ctss    dmu          hs             jlib    network  opmn         perl      rest            srvm      xag

cdata        cv      eons         install        JRE     nls      oracore      plsql     root.sh         suptools  xdk

cfgtoollogs  dbs     evm          instantclient  ldap    oc4j     oraInst.loc  precomp   rootupgrade.sh  ucp

clone        dc_ocm  gipc         inventory      lib     ohasd    ord          QOpatch   scheduler       usm

[root@testsvr1 grid]# pwd

/oracle/grid

[root@testsvr1 grid]# sh ./root.sh

Performing root user operation.



The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /oracle/grid



Enter the full pathname of the local bin directory: [/usr/local/bin]: 

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.



Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /oracle/grid/crs/install/crsconfig_params

2020/08/10 15:47:54 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.



2020/08/10 15:48:19 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.



2020/08/10 15:48:19 CLSRSC-363: User ignored prerequisites during installation



OLR initialization - successful

2020/08/10 15:48:45 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'



CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-2672: Attempting to start 'ora.evmd' on 'testsvr1'

CRS-2672: Attempting to start 'ora.mdnsd' on 'testsvr1'

CRS-2676: Start of 'ora.mdnsd' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.evmd' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'testsvr1'

CRS-2676: Start of 'ora.gpnpd' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testsvr1'

CRS-2672: Attempting to start 'ora.gipcd' on 'testsvr1'

CRS-2676: Start of 'ora.cssdmonitor' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.gipcd' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'testsvr1'

CRS-2672: Attempting to start 'ora.diskmon' on 'testsvr1'

CRS-2676: Start of 'ora.diskmon' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.cssd' on 'testsvr1' succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testsvr1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'testsvr1'

CRS-2673: Attempting to stop 'ora.evmd' on 'testsvr1'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'testsvr1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testsvr1'

CRS-2677: Stop of 'ora.drivers.acfs' on 'testsvr1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'testsvr1' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'testsvr1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'testsvr1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'testsvr1'

CRS-2677: Stop of 'ora.cssd' on 'testsvr1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'testsvr1'

CRS-2677: Stop of 'ora.gipcd' on 'testsvr1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testsvr1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'testsvr1'

CRS-2672: Attempting to start 'ora.evmd' on 'testsvr1'

CRS-2676: Start of 'ora.mdnsd' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.evmd' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'testsvr1'

CRS-2676: Start of 'ora.gpnpd' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'testsvr1'

CRS-2676: Start of 'ora.gipcd' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testsvr1'

CRS-2676: Start of 'ora.cssdmonitor' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'testsvr1'

CRS-2672: Attempting to start 'ora.diskmon' on 'testsvr1'

CRS-2676: Start of 'ora.diskmon' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.cssd' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'testsvr1'

CRS-2672: Attempting to start 'ora.ctssd' on 'testsvr1'

CRS-2676: Start of 'ora.ctssd' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'testsvr1'

CRS-2676: Start of 'ora.asm' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'testsvr1'

CRS-2676: Start of 'ora.storage' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'testsvr1'

CRS-2676: Start of 'ora.crf' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'testsvr1'

CRS-2676: Start of 'ora.crsd' on 'testsvr1' succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

CRS-2672: Attempting to start 'ora.cvu' on 'testsvr1'

CRS-2672: Attempting to start 'ora.scan1.vip' on 'testsvr1'

CRS-2672: Attempting to start 'ora.testsvr1.vip' on 'testsvr1'

CRS-2672: Attempting to start 'ora.testsvr2.vip' on 'testsvr1'

CRS-6017: Processing resource auto-start for servers: testsvr1

CRS-2672: Attempting to start 'ora.ons' on 'testsvr1'

CRS-2672: Attempting to start 'ora.oc4j' on 'testsvr1'

CRS-2676: Start of 'ora.cvu' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.ons' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'testsvr1' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'testsvr1'

CRS-2676: Start of 'ora.testsvr1.vip' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.testsvr2.vip' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'testsvr1' succeeded

CRS-2676: Start of 'ora.oc4j' on 'testsvr1' succeeded

CRS-6016: Resource auto-start has completed for server testsvr1

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2020/08/10 15:54:10 CLSRSC-343: Successfully started Oracle Clusterware stack



PRKO-2190 : VIP exists for node testsvr1, VIP name testsvr1-vip

PRKZ-1072 : SCAN name "sigdbqa-scan" is already registered on network 1

PRCS-1028 : Single Client Access Name (SCAN) listener resources already exist on network 1

PRCN-3004 : Listener MGMTLSNR already exists

PRCR-1086 : resource ora.cvu is already registered

PRCC-1014 : scan1 was already running

PRCR-1004 : Resource ora.scan1.vip is already running

PRCR-1079 : Failed to start resource ora.scan1.vip

CRS-5702: Resource 'ora.scan1.vip' is already running on 'testsvr1'

PRCC-1014 : LISTENER_SCAN1 was already running

PRCR-1004 : Resource ora.LISTENER_SCAN1.lsnr is already running

PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr

CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'testsvr1'

PRCC-1014 : cvu was already running

PRCR-1004 : Resource ora.cvu is already running

PRCR-1079 : Failed to start resource ora.cvu

CRS-5702: Resource 'ora.cvu' is already running on 'testsvr1'

Preparing packages...

cvuqdisk-1.0.9-1.x86_64

PRCN-3004 : Listener LISTENER already exists

2020/08/10 15:54:54 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded



[root@testsvr1 grid]# 

 

정상적으로 CRS 데몬 동작

 

7. 2번 노드 재시작

 

 # 2번 노드 클러스터 시작

 

[root@testsvr2 bin]# ./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@testsvr2 bin]# ls
 

 

 # oracle user 1번 노드에서 디비 start

 

 

[+ASM1:oracle@testsvr1] /home/oracle> srvctl start instance -db testdbL -instance testdbL1

[+ASM1:oracle@testsvr1] /home/oracle> crsctl stat res -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details       

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.CRS.dg

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.DATA.dg

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.asm

               ONLINE  ONLINE       testsvr1                 Started,STABLE

               ONLINE  ONLINE       testsvr2                 Started,STABLE

ora.net1.network

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.ons

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.MGMTLSNR

      1        OFFLINE OFFLINE                               STABLE

ora.testdbl.db

      1        ONLINE  ONLINE       testsvr1                 Open,STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.datadbl.db

      1        OFFLINE OFFLINE                               STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.cvu

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.oc4j

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.testsvr1.vip

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.testsvr2.vip

      1        ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

[+ASM1:oracle@testsvr1] /home/oracle> srvctl start instance -db datadbL -instance datadbL1

[+ASM1:oracle@testsvr1] /home/oracle> crsctl stat res -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details       

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.CRS.dg

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.DATA.dg

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.asm

               ONLINE  ONLINE       testsvr1                 Started,STABLE

               ONLINE  ONLINE       testsvr2                 Started,STABLE

ora.net1.network

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

ora.ons

               ONLINE  ONLINE       testsvr1                 STABLE

               ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.MGMTLSNR

      1        OFFLINE OFFLINE                               STABLE

ora.testdbl.db

      1        ONLINE  ONLINE       testsvr1                 Open,STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.datadbl.db

      1        ONLINE  ONLINE       testsvr1                 Open,STABLE

      2        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.cvu

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.oc4j

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.testsvr1.vip

      1        ONLINE  ONLINE       testsvr1                 STABLE

ora.testsvr2.vip

      1        ONLINE  ONLINE       testsvr2                 STABLE

--------------------------------------------------------------------------------

[+ASM1:oracle@testsvr1] /home/oracle> 
 

 

RAC 의 경우 OS를 새로 설치를 한 상황 임에도 기존 Engine 만 보존이 되 있으면 큰 어려움 없이 노드 복구가 가능 했습니다.

 

 

 

위로