Oracle RAC RU rolling patching from 19.3 to 19.9.0.0.201020(Patch 31750108)
1. 패치에 포함된 README 탐독
패치를 하기 전 항상 패치에 포함된 Readme 파일을 꼼꼼히 읽어 봐야 한다.
이번 패치의 경우 rolling 방식의 패치가 가능하다고 문서에 기재되어 있다.
해당 패치에 대한 문서를 확인하려면
Doc 2694903.1 Oracle Database 19c RU/RUR Oct 2020 Known Issues 를 참조하시기 바랍니다.
패치에 포함된 readme.html 문서는 파일 첨부해 두도록 하겠습니다.
2. 패치전 점검 사항
1) opatch 버전 점검
이번 패치의 경우 opatch 버전이 최소 12.2.0.19 이상이여야 한다.
|
1
2
3
4
|
mprac1.localdomain@oracle:MPRAC1:/home/oracle> opatch version
OPatch Version: 12.2.0.1.17
OPatch succeeded.
|
cs |
위와 같이 버전이 낮기 때문에 버전을 업그레이드 해야한다.
아래 링크에서 받아서 설치하면 된다.
https://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=6880880
아래와 같이 Grid Home과 Oracle Home에 파일을 로드하고
Grid owner와 Oracle owner 계정으로 unzip 수행 후 버전을 확인하다.(2번 노드에서도 수행해야함)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
--Zip file upload
mprac1@root:/root> cp p6880880_200000_Linux-x86-64.zip /u01/app/19.0.0/grid/
mprac1@root:/root> chown grid:oinstall /u01/app/19.0.0/grid/p6880880_200000_Linux-x86-64.zip
mprac1@root:/root> cp p6880880_200000_Linux-x86-64.zip /u01/app/oracle/product/19.0.0/db_1/
mprac1@root:/root> chown oracle:oinstall /u01/app/oracle/product/19.0.0/db_1/p6880880_200000_Linux-x86-64.zip
--Grid Home opatch upgrade
mprac1@root:/root> cd /u01/app/19.0.0/grid
mprac1@root:/u01/app/19.0.0/grid> mv OPatch OPatch_bak
mprac1@root:/u01/app/19.0.0/grid> unzip p6880880_200000_Linux-x86-64.zip
...
...
mprac1@root:/u01/app/19.0.0/grid> chown -R grid:oinstall OPatch
mprac1@root:/u01/app/19.0.0/grid> su - grid
mprac1@grid:+ASM1:/home/grid> opatch version
OPatch Version: 12.2.0.1.23
OPatch succeeded.
--Oracle Home opatch upgrade
mprac1@root:/root> cd /u01/app/oracle/product/19.0.0/db_1/
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> mv OPatch OPatch_bak
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> unzip p6880880_200000_Linux-x86-64.zip
...
...
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> chown -R oracle:oinstall OPatch
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> opatch version
OPatch Version: 12.2.0.1.23
OPatch succeeded.
|
cs |
2) Oracle Inventory 정보 확인 및 저장
패치 수행 전에 현재 정보를 저장해 두자.
간혹 이슈 발생으로 확인이 필요해지는 경우가 생기기도 한다.(2번 노드에서도 체크해야 함)
|
1
2
3
4
5
6
|
--grid 계정에서 확인
mprac1@grid:+ASM1:/home/grid> opatch lsinventory -detail -oh $ORACLE_HOME > before_patch_lsinventory.txt
--oracle 계정에서 확인
mprac1@oracle:MPRAC1:/home/oracle> opatch lsinventory -detail -oh $ORACLE_HOME > before_patch_lsinventory.txt
|
cs |
생성된 파일 before_patch_lsinventory.txt 내용을 확인해 둔다.
3) Opatch Conflict Check
패치 적용 전 항상 Conflict 체크를 해야한다.
만약 충돌 패치가 있다면 해결 방안을 만들고 해결하고 수행해야 한다.
먼저 패치 zip 파일을 unzip 한다.
|
1
2
3
4
5
6
|
--patch 파일 unzip
mprac1@root:/root> mv p31750108_190000_Linux-x86-64.zip /tmp/
mprac1@root:/root> unzip /tmp/p31750108_190000_Linux-x86-64.zip -d /tmp
...
...
mprac1@root:/root> chown -R grid:oinstall /tmp/31750108/
|
cs |
패치 문서상으로는 아래와 같이 확인하라고 되어 있다.
|
1
2
3
4
5
6
7
8
9
|
For Grid Infrastructure Home, as home user:
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31771877
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31772784
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31773437
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31780966
For Database home, as home user:
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31771877
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31772784
|
cs |
그래서 Grid Home owner인 grid 계정에서 위 5개의 디렉토리로 체크를 해 보면 /tmp/31750108 체크에서 아래와 같이 에러가 발생한다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
|
mprac1@root:/root> su - grid
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31771877
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-30-31PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31772784
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-31-16PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31780966
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-32-27PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-33-12PM_1.log
This command doesn't support System Patch.
OPatch failed with error code 21
|
cs |
이는 무시해도 되는 에러이다.
Readme 파일에서 체크할 필요가 없는 패치 디렉토리를 체크하라고 잘못 기재한 내용이다.
This command doesn't support System Patch 문구의 경우
오라클 메타링크에서 조회해 보면 잘못된 디렉토리 지정에 대한 에러라는 내용이 많다
로그 파일의 마지막 부분 로그를 확인하면 아래와 같으며
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
[Dec 8, 2020 1:33:13 PM] [INFO] PREREQ session
[Dec 8, 2020 1:33:13 PM] [INFO] CUP_LOG: Trying to load HomeOperations object
[Dec 8, 2020 1:33:13 PM] [INFO] CUP_LOG: HomeOperations object created. CUP1.0 is enabled
[Dec 8, 2020 1:33:13 PM] [INFO] OPatch invoked as follows: 'prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/ -invPtrLoc /u01/app/19.0.0/grid/oraInst.loc '
[Dec 8, 2020 1:33:13 PM] [INFO] Runtime args: [-javaagent:/u01/app/19.0.0/grid/OPatch/jlib/oracle.opatch.classpath.jar, -Xmx3072m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/u01/app/19.0.0/grid/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=prereq, -DOPatch.ORACLE_HOME=/u01/app/19.0.0/grid, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=/u01/app/19.0.0/grid/OPatch, -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
[Dec 8, 2020 1:33:13 PM] [INFO] Heap in use : 30 MB
Total memory: 146 MB
Free memory : 115 MB
Max memory : 2731 MB
[Dec 8, 2020 1:33:13 PM] [INFO] Oracle Home : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
OUI location : /u01/app/19.0.0/grid/oui
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-33-12PM_1.log
[Dec 8, 2020 1:33:13 PM] [INFO] Patch history file: /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch_history.txt
[Dec 8, 2020 1:33:13 PM] [INFO] [OPSR-TIME] Loading raw inventory
[Dec 8, 2020 1:33:13 PM] [INFO] [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 23 (MB)
[Dec 8, 2020 1:33:13 PM] [INFO] [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 23 (MB)
[Dec 8, 2020 1:33:13 PM] [INFO] [OPSR-TIME] Raw inventory loaded successfully
[Dec 8, 2020 1:33:19 PM] [INFO] This command doesn't support System Patch.
[Dec 8, 2020 1:33:19 PM] [INFO] Finishing PrereqSession at Tue Dec 08 13:33:19 KST 2020
[Dec 8, 2020 1:33:19 PM] [INFO] Stack Description: java.lang.RuntimeException: This command doesn't support System Patch.
at oracle.opatch.PrereqSession.process(PrereqSession.java:184)
at oracle.opatch.OPatchSession.process(OPatchSession.java:2780)
at oracle.opatch.OPatch.process(OPatch.java:853)
at oracle.opatch.OPatch.main(OPatch.java:910)
|
cs |
점검 대상 디렉토리를 확인해 보면 딱히 점검할 파일이나 내용이 존재하지 않는다.
|
1
2
3
4
5
6
7
8
9
10
11
12
|
mprac1@oracle:MPRAC1:/home/oracle> ls -al /tmp/31750108/
total 136
drwxr-x--- 7 grid oinstall 143 Oct 19 21:09 .
drwxrwxrwt. 32 root root 4096 Dec 8 13:49 ..
drwxr-x--- 5 grid oinstall 81 Oct 19 21:10 31771877
drwxr-x--- 5 grid oinstall 62 Oct 19 21:13 31772784
drwxr-x--- 5 grid oinstall 62 Oct 19 21:09 31773437
drwxr-x--- 4 grid oinstall 48 Oct 19 21:14 31780966
drwxr-x--- 2 grid oinstall 4096 Oct 19 21:10 automation
-rw-rw-r-- 1 grid oinstall 5054 Oct 20 06:36 bundle.xml
-rw-r--r-- 1 grid oinstall 122266 Oct 20 06:20 README.html
-rw-r--r-- 1 grid oinstall 0 Oct 19 21:10 README.txt
|
cs |
이번에는 Oracle Home owner인 oracle 계정으로 Oracle Home 대상 패치에 대해서 패치 충돌 점검을 해 본다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
|
mprac1@grid:+ASM1:/home/grid> exit
logout
mprac1@root:/root> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31771877
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.0.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/db_1/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-12-08_13-35-24PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
mprac1@oracle:MPRAC1:/home/oracle> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31772784
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.0.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/db_1/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-12-08_13-35-55PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
|
cs |
위와 같이 충돌 이슈가 존재하지 않는다
이로서 Grid Home 과 Oracle Home 패치에 대한 패치 충돌은 없는 상태로 확인할 수 있다.
4) Opatch SystemSpace Check
패치에 필요한 공간이 적절하게 남아 있는지를 점검해 본다.
먼저 아래와 같은 내용의 patch_list_gihome.txt 파일을 생성한다.
|
1
2
3
4
5
6
|
mprac1@root:/root> cat /tmp/patch_list_gihome.txt
/tmp/31750108/31771877
/tmp/31750108/31772784
/tmp/31750108/31773437
/tmp/31750108/31780966
mprac1@root:/root> chown grid:oinstall /tmp/patch_list_gihome.txt
|
cs |
grid 계정에서 아래와 같이 필요한 공간에 대해 점검해 본다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
mprac1@root:/root> su - grid
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_15-32-56PM_1.log
Invoking prereq "checksystemspace"
Prereq "checkSystemSpace" passed.
OPatch succeeded.
|
cs |
이슈 사항이 없는 것을 확인할 수 있다.
이제 oracle 계정으로 점검해 보자.
마찬가지로 아래 내용을 담은 patch_list_dbhome.txt 파일을 먼저 생성한다.
|
1
2
3
4
|
mprac1@root:/root> cat /tmp/patch_list_dbhome.txt
/tmp/31750108/31771877
/tmp/31750108/31772784
mprac1@root:/root> chown oracle:oinstall /tmp/patch_list_dbhome.txt
|
cs |
아래와 같이 점검해 보자
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
mprac1@root:/root> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.0.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/db_1/oraInst.loc
OPatch version : 12.2.0.1.23
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-12-08_15-34-57PM_1.log
Invoking prereq "checksystemspace"
Prereq "checkSystemSpace" passed.
OPatch succeeded.
|
cs |
위와 같이 이슈가 없다.
3. Grid Home 패치 수행(발생하는 이슈가 포함되어 있으므로 바로 따라하지 말고 내용 충분히 본 후 진행하기 바랍니다)
Readme 문서에 rolling fashion이 가능하다고 하여 opatchauto 를 사용하여
rolling 방식으로 진행했지만 에러가 발생하게 된다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
mprac1@root:/root> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> srvctl stop instance -db MPRAC -instance MPRAC1 -stopoption immediate
mprac1@oracle:MPRAC1:/home/oracle> exit
logout
mprac1@root:/root> export PATH=$PATH:/u01/app/19.0.0/grid/OPatch
mprac1@root:/root> opatchauto apply /tmp/31750108
OPatchauto session is initiated at Tue Dec 8 16:13:14 2020
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_04-13-17PM.log.
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_04-14-03PM.log
The id for this session is 51ED
Shared home /u01/app/19.0.0/grid can only be patched in nonrolling mode.
OPATCHAUTO-72030: Execution mode invalid.
OPATCHAUTO-72030: Cannot execute in rolling mode, as CRS home is shared.
OPATCHAUTO-72030: Execute in non-rolling mode.
OPatchAuto failed.
OPatchauto session completed at Tue Dec 8 16:14:26 2020
Time taken to complete the session 1 minute, 12 seconds
opatchauto failed with error code 42
|
cs |
opatchauto의 경우 crs를 기동한 상태에서 진행하는 형태인데 이렇게 진행했을 때 에러가 발생해
crs를 내려놓고 진행을 시도해 봤다. 하지만 역시나 같은 에러가 발생했다.
이는 Grid Home이 shared 상태라서 그렇다는 것인데
1, 2번 노드 모드 로컬 디스크에 Grid Home을 설치했기 때문에 shared 상태는 아니다.
아마도 이부분은 버그 같다는 생각이 든다.
rolling 방식으로 패치를 진행하면 위와 같은 에러가 지속적으로 발생한다
해당 원인에 대해 깊이있게 확인을 해 보지 못하고 nonrolling 방식으로 진행하기로 변경했다.
1번 노드의 crs만 내려놓고 nonrolling 모드로 진행한다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
|
mprac1@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME -nonrolling
OPatchauto session is initiated at Tue Dec 8 20:17:43 2020
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_08-17-48PM.log.
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_08-17-53PM.log
The id for this session is ERE4
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid
Patch applicability verified successfully on home /u01/app/19.0.0/grid
Executing patch validation checks on home /u01/app/19.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.0.0/grid
Bringing down CRS service on home /u01/app/19.0.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/mprac1/crsconfig/crs_prepatch_apply_inplace_mprac1_2020-12-08_08-18-35PM.log
CRS service brought down successfully on home /u01/app/19.0.0/grid
Start applying binary patch on home /u01/app/19.0.0/grid
Binary patch applied successfully on home /u01/app/19.0.0/grid
Starting CRS service on home /u01/app/19.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/mprac1/crsconfig/crs_postpatch_apply_inplace_mprac1_2020-12-08_08-34-11PM.log
Failed to start CRS service on home /u01/app/19.0.0/grid
Execution of [GIStartupAction] patch action failed, check log for more details. Failures:
Patch Target : mprac1->/u01/app/19.0.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.0.0/grid, host: mprac1.
Command failed: /u01/app/19.0.0/grid/perl/bin/perl -I/u01/app/19.0.0/grid/perl/lib -I/u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/crs/install -I/u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/xag /u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/crs/install/rootcrs.pl -postpatch -nonrolling
Command failure output:
Using configuration parameter file: /u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/mprac1/crsconfig/crs_postpatch_apply_inplace_mprac1_2020-12-08_08-34-11PM.log
2020/12/08 20:34:32 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'mprac1'
CRS-2672: Attempting to start 'ora.evmd' on 'mprac1'
CRS-2676: Start of 'ora.mdnsd' on 'mprac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'mprac1'
CRS-2676: Start of 'ora.gpnpd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'mprac1'
CRS-2676: Start of 'ora.gipcd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'mprac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'mprac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'mprac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'mprac1'
CRS-2676: Start of 'ora.diskmon' on 'mprac1' succeeded
CRS-2676: Start of 'ora.crf' on 'mprac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'mprac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'mprac1'
CRS-2676: Start of 'ora.ctssd' on 'mprac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'mprac1'
CRS-2676: Start of 'ora.asm' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'mprac1'
CRS-2676: Start of 'ora.storage' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'mprac1'
CRS-2676: Start of 'ora.crsd' on 'mprac1' succeeded
Cluster registry server error. Server error stack -
PROC-00053: The configured patch level, release patch level and the patch level on the other active cluster nodes are not all the same.
- end of server error stackCRS-2883: Resource 'ora.crsd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
PRVH-0116 : Path "/u01/app/19.0.0/grid/crs/install/cmdllroot.sh" with permissions "rw-r--r--" does not have execute permissions for the owner, file's group, and others on node "mprac1".
PRVG-2031 : Owner of file "/u01/app/19.0.0/grid/crs/install/cmdllroot.sh" did not match the expected value on node "mprac1". [Expected = "grid(1001)" ; Found = "root(0)"]
PRVG-2032 : Group of file "/u01/app/19.0.0/grid/crs/install/cmdllroot.sh" did not match the expected value on node "mprac1". [Expected = "oinstall(1001)" ; Found = "root(0)"]
CRS-4000: Command Start failed, or completed with errors.
2020/12/08 20:35:38 CLSRSC-117: Failed to start Oracle Clusterware stack from the Grid Infrastructure home /u01/app/19.0.0/grid
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Tue Dec 8 20:35:41 2020
Time taken to complete the session 17 minutes, 58 seconds
opatchauto failed with error code 42
|
cs |
위와 같이 패치에 실패하게 된다.
문제를 해결하고 opatchauto resume 으로 다시 진행하라는 내용이다.
문제는 위 내용 중 아래 내용을 보면 알 수 있다.
|
1
2
3
4
|
Cluster registry server error. Server error stack -
PROC-00053: The configured patch level, release patch level and the patch level on the other active cluster nodes are not all the same.
- end of server error stackCRS-2883: Resource 'ora.crsd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
|
cs |
1번 노드의 패치가 끝나고 1번 노드의 crs를 올리려고 하니
현재 2번에 기동되어 있는 crs와 패치 레벨이 맞지 않는다는 내용이다.
그래서 2번 노드의 crs를 내리고 resume을 진행해 준다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
|
mprac1.localdomain@root:/root> opatchauto resume
OPatchauto session is initiated at Tue Dec 8 20:41:46 2020
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_08-41-50PM.log
Resuming existing session with id ERE4
Starting CRS service on home /u01/app/19.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/mprac1/crsconfig/crs_postpatch_apply_inplace_mprac1_2020-12-08_08-42-09PM.log
CRS service started successfully on home /u01/app/19.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:mprac1
CRS Home:/u01/app/19.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/31750108/31771877
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
Patch: /tmp/31750108/31772784
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
Patch: /tmp/31750108/31773437
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
Patch: /tmp/31750108/31780966
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
OPatchauto session completed at Tue Dec 8 20:47:54 2020
Time taken to complete the session 6 minutes, 10 seconds
|
cs |
1번 노드의 Grid Home 패치가 성공적으로 완료됐다.
grid 계정으로 패치 인벤토리를 확인해 본다.
|
1
2
3
4
5
6
|
mprac1@grid:+ASM1:/home/grid> opatch lsinventory | grep "Patch description"
Patch description: "TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)"
Patch description: "ACFS RELEASE UPDATE 19.9.0.0.0 (31773437)"
Patch description: "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
Patch description: "Database Release Update : 19.9.0.0.201020 (31771877)"
mprac1@grid:+ASM1:/home/grid>
|
cs |
위와 같이 잘 적용된 것을 확인할 수 있다.
현재 상태는 1번 노드의 crs는 기동되어 있는 상태이며 2번 노드의 crs는 내려가 있는 상태이다.
2번 노드의 Grid Home 패치하기 전에 1번 노드의 crs를 내리고
패치가 안된 2번 노드의 crs를 올려 놓은 상태로 만들어 놓아야 한다.
이렇게 하는 이유는 opatchauto의 경우 crs가 떠 있는 상태에서 진행하는 방식이라서
2번 노드에서 opatchauto 명령어로 진행하면 2번 노드의 crs를 직접 올릴려고 하는데
앞서 발생한 에러와 마찬가지로 1, 2번 노드의 crs 패치 레벨이 맞지 않아
2번 노드 crs의 기동시 에러가 발생하여 패치 진행이 되지 않는다.
그래서 1번 노드의 crs를 내리고 2번 노드의 crs를 기동시켜 놓은 상태에서 opatchauto를 수행한다.
아래 내용은 위에 설명한 상황에서의 에러 내용이다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
mprac2@root:/root> export PATH=$PATH:/u01/app/19.0.0/grid/OPatch
mprac2@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME -nonrolling
OPatchauto session is initiated at Tue Dec 8 20:53:38 2020
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_08-53-47PM.log.
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_08-53-55PM.log
The id for this session is 8FKQ
OPATCHAUTO-72132: Grid is not running on the local host.
OPATCHAUTO-72132: Cannot start a new apply or rollback session when the local grid is not running.
OPATCHAUTO-72132: Please start grid service on the local host to start patching.
OPatchAuto failed.
OPatchauto session completed at Tue Dec 8 20:54:20 2020
Time taken to complete the session 0 minute, 43 seconds
opatchauto failed with error code 42
|
cs |
이제 1번 노드의 crs를 내리고 2번 노드의 crs를 올리고 진행해 보자
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
mprac2@root:/root> crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
mprac2@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME -nonrolling
OPatchauto session is initiated at Tue Dec 8 21:03:13 2020
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_09-03-18PM.log.
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_09-03-51PM.log
The id for this session is S7SL
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid
Patch applicability verified successfully on home /u01/app/19.0.0/grid
Executing patch validation checks on home /u01/app/19.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.0.0/grid
Bringing down CRS service on home /u01/app/19.0.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/mprac2/crsconfig/crs_prepatch_apply_inplace_mprac2_2020-12-08_09-04-36PM.log
CRS service brought down successfully on home /u01/app/19.0.0/grid
Start applying binary patch on home /u01/app/19.0.0/grid
Binary patch applied successfully on home /u01/app/19.0.0/grid
Starting CRS service on home /u01/app/19.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/mprac2/crsconfig/crs_postpatch_apply_inplace_mprac2_2020-12-08_09-20-11PM.log
CRS service started successfully on home /u01/app/19.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:mprac2
CRS Home:/u01/app/19.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/31750108/31771877
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
Patch: /tmp/31750108/31772784
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
Patch: /tmp/31750108/31773437
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
Patch: /tmp/31750108/31780966
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
OPatchauto session completed at Tue Dec 8 21:26:10 2020
Time taken to complete the session 22 minutes, 57 seconds
|
cs |
2번 노드 Grid Home도 정상적으로 패치가 완료되었다.
4. Oracle Home 패치 수행(발생하는 이슈가 포함되어 있으므로 바로 따라하지 말고 내용 충분히 본 후 진행하기 바랍니다)
현재 2번 Grid Home 패치를 진행해 1번의 crs는 내려가 있고 2번 노드의 crs만 올라와 있는 상태이다.
현재 상태에서 1번 노드의 Oracle Home 패치를 진행해 본다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
|
mprac1@root:/root> echo $ORACLE_HOME
/u01/app/oracle/product/19.0.0/db_1
mprac1@root:/root> which opatch
/u01/app/oracle/product/19.0.0/db_1/OPatch/opatch
mprac1@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME
OPatchauto session is initiated at Wed Dec 9 10:18:08 2020
System initialization log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchautodb/systemconfig2020-12-09_10-18-22AM.log.
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-12-09_10-18-29AM.log
The id for this session is TMKA
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
Executing patch validation checks on home /u01/app/oracle/product/19.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/db_1
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/db_1
Performing postpatch operation on home /u01/app/oracle/product/19.0.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
Preparing home /u01/app/oracle/product/19.0.0/db_1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:mprac1
RAC Home:/u01/app/oracle/product/19.0.0/db_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /tmp/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/31750108/31771877
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_10-19-13AM_1.log
Patch: /tmp/31750108/31772784
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_10-19-13AM_1.log
OPatchauto session completed at Wed Dec 9 10:27:58 2020
Time taken to complete the session 9 minutes, 52 seconds
|
cs |
정상적으로 완료됐다.
2번 노드의 Oracle Home 패치를 진행해 본다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
mprac2@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME
OPatchauto session is initiated at Wed Dec 9 11:20:37 2020
System initialization log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchautodb/systemconfig2020-12-09_11-20-53AM.log.
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-12-09_11-21-58AM.log
The id for this session is 1FAA
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
Executing patch validation checks on home /u01/app/oracle/product/19.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/db_1
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
Successfully prepared home /u01/app/oracle/product/19.0.0/db_1 to bring down database service
Bringing down database service on home /u01/app/oracle/product/19.0.0/db_1
Database service successfully brought down on home /u01/app/oracle/product/19.0.0/db_1
Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Failed while applying binary patches on home /u01/app/oracle/product/19.0.0/db_1
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : mprac2->/u01/app/oracle/product/19.0.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.0.0/db_1, host: mprac2.
Command failed: /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto apply /tmp/31750108 -oh /u01/app/oracle/product/19.0.0/db_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.0.0/db_1/oraInst.loc -jre /u01/app/oracle/product/19.0.0/db_1/OPatch/jre -persistresult /u01/app/oracle/product/19.0.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_mprac2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.0.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_mprac2_rac_2.ser
Command failure output:
==Following patches FAILED in apply:
Patch: /tmp/31750108/31771877
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_11-22-52AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Wed Dec 9 11:32:30 2020
Time taken to complete the session 11 minutes, 56 seconds
opatchauto failed with error code 42
|
cs |
위의 내용과 같이 에러가 발생한다.
/u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_11-22-52AM_1.log 로그 파일을 통해
내용을 확인해 보면 에러 내용은 아래와 같다.
|
1
2
3
4
5
6
|
[Dec 9, 2020 1:54:32 PM] [INFO] ENTERING METHOD: NApplyReportForPrereqs(OPatchPatch[] patches,OPatchNApplyOptions options)
[Dec 9, 2020 1:54:32 PM] [INFO] [OPSR-TIME] Loading cooked inventory
[Dec 9, 2020 1:54:32 PM] [INFO] [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 83 (MB)
[Dec 9, 2020 1:54:32 PM] [INFO] Stack Description: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
Possible causes are:
ORACLE_HOME/inventory/oneoffs/31771877 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31771877/etc/config/actions" or "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31771877/etc/config/inventory" does not exist.
|
cs |
위 로그로 확인해 보니 /u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs 이 디렉토리 밑에 31771877 디렉토리가 존재하지 않았다.
1번 노드에서 scp를 통해서 내용을 아래와 같이 복사해 온다.
|
1
2
3
|
mprac1@oracle:MPRAC1:/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs> scp -r ./31771877 mprac2:/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/
inventory.xml 100% 319KB 20.2MB/s 00:00
actions.xml 100% 2372KB 95.4MB/s 00:00
|
cs |
2번 노드에서 다시 패치를 진행한다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
|
mprac2@root:/root> opatchauto resume
OPatchauto session is initiated at Wed Dec 9 14:06:27 2020
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-12-09_02-06-28PM.log
Resuming existing session with id 3QLX
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
Executing patch validation checks on home /u01/app/oracle/product/19.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/db_1
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
Successfully prepared home /u01/app/oracle/product/19.0.0/db_1 to bring down database service
Bringing down database service on home /u01/app/oracle/product/19.0.0/db_1
Database service successfully brought down on home /u01/app/oracle/product/19.0.0/db_1
Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/db_1
Performing postpatch operation on home /u01/app/oracle/product/19.0.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
Starting database service on home /u01/app/oracle/product/19.0.0/db_1
Database service successfully started on home /u01/app/oracle/product/19.0.0/db_1
Preparing home /u01/app/oracle/product/19.0.0/db_1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/db_1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:mprac2
RAC Home:/u01/app/oracle/product/19.0.0/db_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /tmp/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/31750108/31771877
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_14-06-45PM_1.log
Reason: /tmp/31750108/31771877 is not required to be applied to oracle home /u01/app/oracle/product/19.0.0/db_1
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/31750108/31772784
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_14-07-21PM_1.log
Patching session reported following warning(s):
_________________________________________________
[WARNING] The database instance 'MPRAC2' from '/u01/app/oracle/product/19.0.0/db_1', in host'mprac2' is not running. SQL changes, if any, will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.
OPatchauto session completed at Wed Dec 9 14:08:35 2020
Time taken to complete the session 2 minutes, 9 seconds
|
cs |
위 WARNING의 경우는 datapatch를 진행하지 않아서 진행하라는 의미이다.
1번과 2번에 적용된 패치 정보를 인벤토리를 통해서 확인해 보자
|
1
2
3
4
5
6
7
|
mprac1@oracle:MPRAC1:/home/oracle> opatch lsinventory | grep "Patch description"
Patch description: "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
Patch description: "Database Release Update : 19.9.0.0.201020 (31771877)"
mprac2@oracle:MPRAC2:/home/oracle> opatch lsinventory | grep "Patch description"
Patch description: "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
Patch description: "Database Release Update : 19.9.0.0.201020 (31771877)"
|
cs |
위와 같이 잘 적용되어 있는 것을 확인할 수 있다.
5. Post-Installation
1) datapatch 적용
먼저 인스턴스를 기동하고 pluggable DB를 모두 open 시켜준다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
mprac1@oracle:MPRAC1:/home/oracle> sqlss
SQL*Plus: Release 19.0.0.0.0 - Production on 수 12월 9 15:17:19 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
휴지 인스턴스에 접속되었습니다.
SQL> startup
ORACLE 인스턴스가 시작되었습니다.
Total System Global Area 3221224552 bytes
Fixed Size 9141352 bytes
Variable Size 822083584 bytes
Database Buffers 2382364672 bytes
Redo Buffers 7634944 bytes
데이터베이스가 마운트되었습니다.
데이터베이스가 열렸습니다.
SQL> alter pluggable database all open;
플러그인할 수 있는 데이터베이스가 변경되었습니다.
SQL> quit
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0에서 분리되었습니다.
|
cs |
datapatch를 수행한다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
mprac1@oracle:MPRAC1:/home/oracle> cd $ORACLE_HOME/OPatch
mprac1@oracle:MPRAC1:/u01/app/oracle/product/19.0.0/db_1/OPatch> ./datapatch -verbose
SQL Patching tool version 19.9.0.0.0 Production on Wed Dec 9 15:26:33 2020
Copyright (c) 2012, 2020, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_846331_2020_12_09_15_26_33/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
No interim patches found
Current state of release update SQL patches:
Binary registry:
19.9.0.0.0 Release_Update 200930183249: Installed
PDB CDB$ROOT:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 20/12/07 15:34:33.745914
PDB MPPDB:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 20/12/07 15:56:03.693390
PDB PDB$SEED:
Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 20/12/07 15:56:03.693390
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED MPPDB
No interim patches need to be rolled back
Patch 31771877 (Database Release Update : 19.9.0.0.201020 (31771877)):
Apply from 19.3.0.0.0 Release_Update 190410122720 to 19.9.0.0.0 Release_Update 200930183249
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 3
Validating logfiles...done
Patch 31771877 apply (pdb CDB$ROOT): WITH ERRORS (RETRYABLE)
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_CDBROOT_2020Dec09_15_31_37.log (errors)
-> Error at line 475: script rdbms/admin/standard.sql
- ORA-00060: 자원 대기중 교착 상태가 검출되었습니다
Patch 31771877 apply (pdb PDB$SEED): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_PDBSEED_2020Dec09_15_58_50.log (no errors)
Patch 31771877 apply (pdb MPPDB): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_MPPDB_2020Dec09_15_58_49.log (no errors)
Adding patches to retry installation queue...done
Retry installation queue:
For the following PDBs: CDB$ROOT
No interim patches need to be rolled back
Patch 31771877 (Database Release Update : 19.9.0.0.201020 (31771877)):
Apply from 19.3.0.0.0 Release_Update 190410122720 to 19.9.0.0.0 Release_Update 200930183249
No interim patches need to be applied
Installing patches...
Patch installation complete. Total patches installed: 1
Validating logfiles...done
Patch 31771877 apply (pdb CDB$ROOT): SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_CDBROOT_2020Dec09_16_21_26.log (no errors)
SQL Patching tool complete on Wed Dec 9 16:39:08 2020
|
cs |
위와 같이 정상적으로 완료되었다.
이제 invalid object 체크 및 컴파일을 위해 utlrp를 수행해 준다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
mprac1@oracle:MPRAC1:/home/oracle> cd $ORACLE_HOME/rdbms/admin
mprac1@oracle:MPRAC1:/u01/app/oracle/product/19.0.0/db_1/rdbms/admin> sqlss
SQL*Plus: Release 19.0.0.0.0 - Production on 수 12월 9 16:45:43 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
다음에 접속됨:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
SQL> @utlrp.sql
|
cs |
이제 모든 서비스를 정상적으로 기동한다.
6. 적용된 사항 확인
패치 적용된 부분들을 Query 해 본다.
|
1
2
3
4
5
|
mprac1@grid:+ASM1:/home/grid> crsctl query crs releasepatch
Oracle Clusterware release patch level is [376483838] and the complete list of patches [31771877 31772784 31773437 31780966 ] have been applied on the local node. The release patch string is [19.9.0.0.0].
mprac2@grid:+ASM2:/home/grid> crsctl query crs releasepatch
Oracle Clusterware release patch level is [376483838] and the complete list of patches [31771877 31772784 31773437 31780966 ] have been applied on the local node. The release patch string is [19.9.0.0.0].
|
cs |
sqlplus로 조회해 본다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
mprac1@oracle:MPRAC1:/home/oracle> sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on 수 12월 9 16:51:44 2020
Version 19.9.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
다음에 접속됨:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
SQL> SELECT CON_ID
,TO_CHAR (ACTION_TIME, 'YYYY-MM-DD') AS ACTION_TIME
,PATCH_ID
,PATCH_TYPE
,ACTION
,DESCRIPTION
,SOURCE_VERSION
,TARGET_VERSION
FROM CDB_REGISTRY_SQLPATCH
ORDER BY CON_ID, ACTION_TIME, PATCH_ID;
CON_ID ACTION_TIM PATCH_ID PATCH_TYPE ACTION DESCRIPTION SOURCE_VER TARGET_VER
---------- ---------- ---------- ---------- ---------- ------------------------------------------------------------ ---------- ----------
1 2020-12-07 29517242 RU APPLY Database Release Update : 19.3.0.0.190416 (29517242) 19.1.0.0.0 19.3.0.0.0
1 2020-12-09 31771877 RU APPLY Database Release Update : 19.9.0.0.201020 (31771877) 19.3.0.0.0 19.9.0.0.0
1 2020-12-09 31771877 RU APPLY Database Release Update : 19.9.0.0.201020 (31771877) 19.3.0.0.0 19.9.0.0.0
2 2020-12-07 29517242 RU APPLY Database Release Update : 19.3.0.0.190416 (29517242) 19.1.0.0.0 19.3.0.0.0
2 2020-12-09 31771877 RU APPLY Database Release Update : 19.9.0.0.201020 (31771877) 19.3.0.0.0 19.9.0.0.0
3 2020-12-07 29517242 RU APPLY Database Release Update : 19.3.0.0.190416 (29517242) 19.1.0.0.0 19.3.0.0.0
3 2020-12-09 31771877 RU APPLY Database Release Update : 19.9.0.0.201020 (31771877) 19.3.0.0.0 19.9.0.0.0
7 행이 선택되었습니다.
SQL>
|
cs |
7. 총평
여러 에러를 겪으면서 완료했는데 충분히 준비하고 진행한 상황에서도 에러를 겪게된다.
에러 중에는 Grid Home을 shared라고 오인식 하는 버그 같은 내용도 있고
readme 문서의 내용으로 진행해도 정상적으로 진행되지 않는 등 readme 문서가 부실하다는 느낌을 받게 된다.
이전에도 경험하고 느낀 점이지만 오라클 패치의 readme 문서는 잘못 기재한 경우도 많고
설치시에 내용을 세세히 다 다루지 않는 거 같다.
매번 느끼는 거지만 오라클의 경우 패치 적용하는 과정에 대해 문서가 패치를 진행하는 엔지니어에게나
패치를 고려하고 있는 사용자에게는 불친절하다라는 느낌을 받게 된다.
이 부분은 오라클에서도 고쳐야할 부분인거 같다.
마지막으로 현재 작성한 내용이 길어졌는데 유저 입장에서 일일이 세세한 부분까지 확인하면서
읽을 수 있도록 하기 위해 패치 설치시에 발생한 내용을 다 넣어서 그렇게 되었다.ㅏ
이런 부분도 참고가 될 수 있기를 바란다.
에러 내용까지 확인하고 픽스하는 부분까지 넣다보니 글이 길어졌습니다.
하지만 에러를 확인하고 픽스하는 과정에 대해서 담겨 있어 참고할 만한 부분이 될 수 있다고 생각이 들어서 길어도 넣어 봤습니다.
감사합니다.