1. 구성 환경
OS : 리눅스 Redhat 7.2
Oracle Version : 12.1
Type : 2 Node RAC
2. 장애 상황
- 1번 노드의 OS Fault 로 복구가 불가능 해짐
- 2번 노드는 정상 적으로 살아 있음
- GRID , ORACLE Engine 이 설치된 볼륨은 복구를 해서 재사용 가능
- 재구성 방안
1. GRID 만 Relink 로 재구성
2. 1번 재구성 실패시 GRID만 삭제 후 인스톨 , 백업 받은 OCR 로 리소스 복구
3-1 . OCR 백업
[root@testsvr2 ocr_backup]# /oracle/grid/bin/ocrconfig -export /home/oracle/DBA/work/ocr_backup/ocr_backup_20200810.bak
[root@testsvr2 ocr_backup]# ls -l
total 92
-rw------- 1 root root 93966 Aug 10 10:52 ocr_backup_20200810.bak
[root@testsvr2 ocr_backup]#
3-2. 2번 노드 서비스 정지
# 2번 노드 서비스 종료 ( 1번 노드 relink 에 영향은 없음 )
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE testsvr2 STABLE
ora.DATA.dg
ONLINE ONLINE testsvr2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE testsvr2 STABLE
ora.asm
ONLINE ONLINE testsvr2 Started,STABLE
ora.net1.network
ONLINE ONLINE testsvr2 STABLE
ora.ons
ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testsvr2 STABLE
ora.MGMTLSNR
1 OFFLINE OFFLINE STABLE
ora.testdbl.db
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE testsvr2 Open,STABLE
ora.datadbl.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE testsvr2 STABLE
ora.oc4j
1 ONLINE ONLINE testsvr2 STABLE
ora.scan1.vip
1 ONLINE ONLINE testsvr2 STABLE
ora.testsvr1.vip
1 ONLINE INTERMEDIATE testsvr2 FAILED OVER,STABLE
ora.testsvr2.vip
1 ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> ps -ef|gre psmon
bash: gre: command not found...
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> ps -ef|grep smon
oracle 44864 1 0 Jan03 ? 01:09:35 ora_smon_testdbL2
oracle 49246 1 0 2019 ? 00:09:42 asm_smon_+ASM2
root 49272 1 1 2019 ? 3-14:38:44 /oracle/grid/bin/osysmond.bin
oracle 61254 40816 0 14:40 pts/0 00:00:00 grep --color=auto smon
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> srvctl stop database -d testdbL -o immediate
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> srvctl stop database -d datadbl -o immediate
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE testsvr2 STABLE
ora.DATA.dg
ONLINE ONLINE testsvr2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE testsvr2 STABLE
ora.asm
ONLINE ONLINE testsvr2 Started,STABLE
ora.net1.network
ONLINE ONLINE testsvr2 STABLE
ora.ons
ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testsvr2 STABLE
ora.MGMTLSNR
1 OFFLINE OFFLINE STABLE
ora.testdbl.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.datadbl.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE testsvr2 STABLE
ora.oc4j
1 ONLINE ONLINE testsvr2 STABLE
ora.scan1.vip
1 ONLINE ONLINE testsvr2 STABLE
ora.testsvr1.vip
1 ONLINE INTERMEDIATE testsvr2 FAILED OVER,STABLE
ora.testsvr2.vip
1 ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup>
# root 유저로 접속 GRID Service shutdown
[+ASM2:oracle@testsvr2] /home/oracle/DBA/work/ocr_backup> su -
Password:
Last login: Mon Aug 10 10:52:26 KST 2020 on pts/0
[root@testsvr2 ~]#
[root@testsvr2 ~]#
[root@testsvr2 ~]#
[root@testsvr2 ~]# /oracle/grid/bin/crsctl stop crs
[root@testsvr2 bin]# /oracle/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testsvr2'
CRS-2673: Attempting to stop 'ora.crsd' on 'testsvr2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'testsvr2'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.testsvr2.vip' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.cvu' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'testsvr2'
CRS-2677: Stop of 'ora.cvu' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.testsvr2.vip' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.testsvr1.vip' on 'testsvr2'
CRS-2677: Stop of 'ora.CRS.dg' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'testsvr2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'testsvr2'
CRS-2677: Stop of 'ora.testsvr1.vip' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'testsvr2'
CRS-2677: Stop of 'ora.asm' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'testsvr2'
CRS-2677: Stop of 'ora.ons' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'testsvr2'
CRS-2677: Stop of 'ora.net1.network' on 'testsvr2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'testsvr2' has completed
CRS-2677: Stop of 'ora.crsd' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testsvr2'
CRS-2677: Stop of 'ora.storage' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.evmd' on 'testsvr2'
CRS-2673: Attempting to stop 'ora.asm' on 'testsvr2'
CRS-2677: Stop of 'ora.ctssd' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.crf' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'testsvr2' succeeded
CRS-2677: Stop of 'ora.asm' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'testsvr2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'testsvr2'
CRS-2677: Stop of 'ora.cssd' on 'testsvr2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'testsvr2'
CRS-2677: Stop of 'ora.gipcd' on 'testsvr2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testsvr2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@testsvr2 bin]#
[root@testsvr2 bin]# ps -ef|grep ohas
root 30184 1 0 2019 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
root 63242 62147 0 14:45 pts/0 00:00:00 grep --color=auto ohas
[root@testsvr2 bin]#
4. 1번 노드 Relink
# 복구 해야할 1번 노드로 root 로 접속
/oracle/grid
[root@testsvr1 grid]# cd /oracle/grid/crs/install
[root@testsvr1 install]# ls
appcluster.pl crska.pm dropdb.pl oracle-ohasd.service s_crsconfig_defs
cmdllroot.sh crsoc4j.pm inittab oracss.pm s_crsconfig_testsvr1_env.txt
crsconfig_addparams.sbs crsohasd.pm install.excl paramfile.crs s_crsutils.pm
crsconfig_params crspatch.pm install.incl ParentDirPerm_testsvr1.txt tfa_setup
crsconfig_params.sbs crstfa.pm installRemove.excl rootcrs.pl
crsdeconfig.pm crsupgrade.pm oraacfs.pm rootcrs.sh
crsgpnp.pm crsutils.pm oraafd.pm roothas.pl
crsinstall.pm dropdb oracle-ohasd.conf roothas.sh
# grid home unlock
[root@testsvr1 install]# export ORACLE_HOME=/oracle/grid
[root@testsvr1 install]# perl rootcrs.pl -unlock -crshome /oracle/grid
Using configuration parameter file: ./crsconfig_params
2020/08/10 15:18:35 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2020/08/10 15:20:14 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2020/08/10 15:20:14 CLSRSC-347: Successfully unlock /oracle/grid
[root@testsvr1 install]#
# oracle user execute relink
[+ASM1:oracle@testsvr1]export ORACLE_HOME=/oracle/grid
[+ASM1:oracle@testsvr1] /oracle/grid/bin> /oracle/grid/bin/relink all
writing relink log to: /oracle/grid/install/relink.log
[+ASM1:oracle@testsvr1] /oracle/grid/bin>
# root user 로 접속 Relink 작업 마무리
[root@testsvr1 install]# pwd
/oracle/grid/rdbms/install
[root@testsvr1 install]# ls
clonedb.pl filemap.ora rdbms rootadd_rdbms.sh sbs
config_filemap.sbs install.excl rootadd_filemap.sh s7x_workflow.akwf setup_processor_group.sh
[root@testsvr1 install]#
[root@testsvr1 install]# ./rootadd_rdbms.sh
[root@testsvr1 install]#
[root@testsvr1 install]# cd /oracle/grid/crs/install
[root@testsvr1 install]# pwd
/oracle/grid/crs/install
[root@testsvr1 install]#
[root@testsvr1 install]# ls
appcluster.pl crsgpnp.pm crsupgrade.pm installRemove.excl ParentDirPerm_testsvr1.txt s_crsutils.pm
cmdllroot.sh crsinstall.pm crsutils.pm oraacfs.pm rootcrs.pl tfa_setup
crsconfig_addparams.sbs crska.pm dropdb oraafd.pm rootcrs.sh
crsconfig_params crsoc4j.pm dropdb.pl oracle-ohasd.conf roothas.pl
crsconfig_params.saved crsohasd.pm inittab oracle-ohasd.service roothas.sh
crsconfig_params.sbs crspatch.pm install.excl oracss.pm s_crsconfig_defs
crsdeconfig.pm crstfa.pm install.incl paramfile.crs s_crsconfig_testsvr1_env.txt
[root@testsvr1 install]# perl rootcrs.pl -patch
Using configuration parameter file: ./crsconfig_params
[root@testsvr1 install]#
5. CRS de-config , 기존 Engine 에 남아 있는 crs 설치 정보 강제 삭제
# OS 영역에 init 정보와 같은 설치 정보가 없어 relink 만으로는 실행이 안됨
Using configuration parameter file: /oracle/grid/crs/install/crsconfig_params
Usage: srvctl <command> <object> [<options>]
commands: enable|disable|start|stop|status|add|remove|modify|update|getenv|setenv|unsetenv|config|upgrade|downgrade
objects: database|service|asm|diskgroup|listener|home|ons
For detailed help on each command and object and its options use:
srvctl <command> -help [-compatible] or
srvctl <command> <object> -help [-compatible]
PRKO-2012 : nodeapps object is not supported in Oracle Restart
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Stop failed, or completed with errors.
2020/08/10 15:46:24 CLSRSC-463: The deconfiguration or downgrade script could not stop current Oracle Clusterware stack.
2020/08/10 15:46:24 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2020/08/10 15:46:52 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
Failure in execution (rc=-1, 0, No such file or directory) for command /etc/init.d/ohasd deinstall
2020/08/10 15:46:53 CLSRSC-557: Oracle Clusterware stack on this node has been successfully deconfigured. There were some errors which can be ignored.
[root@testsvr1 install]#
6. CRS 데몬 재구성 , root.sh 실행
addnode crf deinstall gnsd javavm log olap osysmond racg slax utl
assistants crs demo gpnp jdbc md ologgerd oui rdbms sqlpatch wlm
auth css diagnostics has jdk mdns OPatch owm relnotes sqlplus wwg
bin ctss dmu hs jlib network opmn perl rest srvm xag
cdata cv eons install JRE nls oracore plsql root.sh suptools xdk
cfgtoollogs dbs evm instantclient ldap oc4j oraInst.loc precomp rootupgrade.sh ucp
clone dc_ocm gipc inventory lib ohasd ord QOpatch scheduler usm
[root@testsvr1 grid]# pwd
/oracle/grid
[root@testsvr1 grid]# sh ./root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/grid/crs/install/crsconfig_params
2020/08/10 15:47:54 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2020/08/10 15:48:19 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2020/08/10 15:48:19 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
2020/08/10 15:48:45 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'testsvr1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'testsvr1'
CRS-2676: Start of 'ora.mdnsd' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.evmd' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'testsvr1'
CRS-2676: Start of 'ora.gpnpd' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testsvr1'
CRS-2672: Attempting to start 'ora.gipcd' on 'testsvr1'
CRS-2676: Start of 'ora.cssdmonitor' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'testsvr1'
CRS-2672: Attempting to start 'ora.diskmon' on 'testsvr1'
CRS-2676: Start of 'ora.diskmon' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.cssd' on 'testsvr1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testsvr1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'testsvr1'
CRS-2673: Attempting to stop 'ora.evmd' on 'testsvr1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'testsvr1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testsvr1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'testsvr1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'testsvr1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'testsvr1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'testsvr1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'testsvr1'
CRS-2677: Stop of 'ora.cssd' on 'testsvr1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'testsvr1'
CRS-2677: Stop of 'ora.gipcd' on 'testsvr1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testsvr1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'testsvr1'
CRS-2672: Attempting to start 'ora.evmd' on 'testsvr1'
CRS-2676: Start of 'ora.mdnsd' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.evmd' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'testsvr1'
CRS-2676: Start of 'ora.gpnpd' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'testsvr1'
CRS-2676: Start of 'ora.gipcd' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testsvr1'
CRS-2676: Start of 'ora.cssdmonitor' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'testsvr1'
CRS-2672: Attempting to start 'ora.diskmon' on 'testsvr1'
CRS-2676: Start of 'ora.diskmon' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.cssd' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'testsvr1'
CRS-2672: Attempting to start 'ora.ctssd' on 'testsvr1'
CRS-2676: Start of 'ora.ctssd' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'testsvr1'
CRS-2676: Start of 'ora.asm' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'testsvr1'
CRS-2676: Start of 'ora.storage' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'testsvr1'
CRS-2676: Start of 'ora.crf' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'testsvr1'
CRS-2676: Start of 'ora.crsd' on 'testsvr1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2672: Attempting to start 'ora.cvu' on 'testsvr1'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'testsvr1'
CRS-2672: Attempting to start 'ora.testsvr1.vip' on 'testsvr1'
CRS-2672: Attempting to start 'ora.testsvr2.vip' on 'testsvr1'
CRS-6017: Processing resource auto-start for servers: testsvr1
CRS-2672: Attempting to start 'ora.ons' on 'testsvr1'
CRS-2672: Attempting to start 'ora.oc4j' on 'testsvr1'
CRS-2676: Start of 'ora.cvu' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.ons' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'testsvr1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'testsvr1'
CRS-2676: Start of 'ora.testsvr1.vip' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.testsvr2.vip' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'testsvr1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'testsvr1' succeeded
CRS-6016: Resource auto-start has completed for server testsvr1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2020/08/10 15:54:10 CLSRSC-343: Successfully started Oracle Clusterware stack
PRKO-2190 : VIP exists for node testsvr1, VIP name testsvr1-vip
PRKZ-1072 : SCAN name "sigdbqa-scan" is already registered on network 1
PRCS-1028 : Single Client Access Name (SCAN) listener resources already exist on network 1
PRCN-3004 : Listener MGMTLSNR already exists
PRCR-1086 : resource ora.cvu is already registered
PRCC-1014 : scan1 was already running
PRCR-1004 : Resource ora.scan1.vip is already running
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5702: Resource 'ora.scan1.vip' is already running on 'testsvr1'
PRCC-1014 : LISTENER_SCAN1 was already running
PRCR-1004 : Resource ora.LISTENER_SCAN1.lsnr is already running
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr
CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'testsvr1'
PRCC-1014 : cvu was already running
PRCR-1004 : Resource ora.cvu is already running
PRCR-1079 : Failed to start resource ora.cvu
CRS-5702: Resource 'ora.cvu' is already running on 'testsvr1'
Preparing packages...
cvuqdisk-1.0.9-1.x86_64
PRCN-3004 : Listener LISTENER already exists
2020/08/10 15:54:54 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@testsvr1 grid]#
정상적으로 CRS 데몬 동작
7. 2번 노드 재시작
# 2번 노드 클러스터 시작
CRS-4123: Oracle High Availability Services has been started.
[root@testsvr2 bin]# ls
# oracle user 1번 노드에서 디비 start
[+ASM1:oracle@testsvr1] /home/oracle> crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.DATA.dg
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.asm
ONLINE ONLINE testsvr1 Started,STABLE
ONLINE ONLINE testsvr2 Started,STABLE
ora.net1.network
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.ons
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testsvr1 STABLE
ora.MGMTLSNR
1 OFFLINE OFFLINE STABLE
ora.testdbl.db
1 ONLINE ONLINE testsvr1 Open,STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.datadbl.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE testsvr1 STABLE
ora.oc4j
1 ONLINE ONLINE testsvr1 STABLE
ora.scan1.vip
1 ONLINE ONLINE testsvr1 STABLE
ora.testsvr1.vip
1 ONLINE ONLINE testsvr1 STABLE
ora.testsvr2.vip
1 ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
[+ASM1:oracle@testsvr1] /home/oracle> srvctl start instance -db datadbL -instance datadbL1
[+ASM1:oracle@testsvr1] /home/oracle> crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.DATA.dg
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.asm
ONLINE ONLINE testsvr1 Started,STABLE
ONLINE ONLINE testsvr2 Started,STABLE
ora.net1.network
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
ora.ons
ONLINE ONLINE testsvr1 STABLE
ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testsvr1 STABLE
ora.MGMTLSNR
1 OFFLINE OFFLINE STABLE
ora.testdbl.db
1 ONLINE ONLINE testsvr1 Open,STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.datadbl.db
1 ONLINE ONLINE testsvr1 Open,STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.cvu
1 ONLINE ONLINE testsvr1 STABLE
ora.oc4j
1 ONLINE ONLINE testsvr1 STABLE
ora.scan1.vip
1 ONLINE ONLINE testsvr1 STABLE
ora.testsvr1.vip
1 ONLINE ONLINE testsvr1 STABLE
ora.testsvr2.vip
1 ONLINE ONLINE testsvr2 STABLE
--------------------------------------------------------------------------------
[+ASM1:oracle@testsvr1] /home/oracle>
RAC 의 경우 OS를 새로 설치를 한 상황 임에도 기존 Engine 만 보존이 되 있으면 큰 어려움 없이 노드 복구가 가능 했습니다.
댓글 0
번호 | 제목 | 글쓴이 | 날짜 | 조회 수 |
---|---|---|---|---|
9 | DB 컨트롤 파일 추가하기 | 명품관 | 2023.03.27 | 1045 |
8 | data pump 모니터링 | 명품관 | 2023.01.31 | 1073 |
7 | Rman 복구 중 Netbackup 이 연결이 안되는 에러 [1] | Talros | 2022.11.24 | 132 |
» | 12C RAC, OS Fault 로 다운된 1번 Node 재구성 | Talros | 2020.08.28 | 1117 |
5 | alter database begin backup 사용을 권해 드립니다. [1] | 우뽕 | 2020.01.22 | 14358 |
4 | Rman 을 이용해 ASM 에 컨트롤 파일 복구 하기 [1] | Talros | 2019.07.16 | 2942 |
3 | Data Pump Query 파라미터 사용법 [1] | Talros | 2016.01.28 | 4530 |
2 | Data pump 사용시 Warnings Like DBMS_AW_EXP: SYS.AW$EXPRESS: OLAP not enabled | Talros | 2016.01.08 | 1988 |
1 | 비정상적인 복구를 수행하게 되는 경우(Redo log 유실, Archive log 파일 부재, Resetlog open 에러) | 명품관 | 2015.12.22 | 6911 |