中煙

unfinished BPP ZYBPCPRD2:172.16.33.180

ZYEDEV cannot backup

Incremental checkpoint up to RBA [0x546a.bf56.0], current log tail at RBA [0x546a.c0f5.0]
Thu Jun 30 02:39:18 2016
Shutting down instance: further logons disabled
Thu Jun 30 02:39:18 2016
Shutting down instance (immediate)
License high water mark = 45
Thu Jun 30 02:39:23 2016
Process OS id : 14011 alive after kill
Errors in file 
Thu Jun 30 02:44:26 2016
Active call for process 21958 user 'oraecd' program 'oracleECD@ZYEDEV'
SHUTDOWN: waiting for active calls to complete.
Thu Jun 30 03:05:43 2016
Incremental checkpoint up to RBA [0x546a.c1d3.0], current log tail at RBA [0x546a.c1d6.0]

ZYEDEV:oraecd 22> ps -ef | grep 21958
oraecd 21958     1 249  Jun 13  ?        24746:59 oracleECD (LOCAL=NO)
oraecd 11167  8864  0 10:44:32 pts/ta    0:00 grep 21958

   INST_ID        SID    SERIAL# SPID       USERNAME   PROGRAM
   ---------- ---------- ---------- ---------- ---------- ---------------------------------------------
        1        170      40653 29492      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        145         58 29494      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        150       4095 13938      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        177      11006 2058       SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        135       6113 29498      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        184      21011 8343       SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        158      28697 5156       SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        176          3 21958      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        189       9169 10270      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        179      24371 29490      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)
        1        191      13656 14013      SAPSR3     disp+work@ZYEDEV (TNS V1-V3)

Reson

Often Oracle (SMON or PMON depending on whether Shared Server is used) will wait for the OS to terminate the process(es) associated with the session. If the OS never returns, or fails to terminate them, then the instance shutdown will hang with this message (Shutdown Waiting for Active Calls to Complete)

Action

ZYEDEV:oraecd 41> ps -ef | grep 21958
oraecd 21958     1 234  Jun 13  ?        24779:17 oracleECD (LOCAL=NO)
oraecd 12195  8864  0 11:16:53 pts/ta    0:00 grep 21958
ZYEDEV:oraecd 42> netstat -ni
Name      Mtu  Network         Address         Ipkts              Ierrs Opkts              Oerrs Coll
lo0      32808 127.0.0.0       127.0.0.1       1400525337         0     1400525907         0     0   
lan900    1500 172.16.33.0     172.16.33.137   8902009490         0     4927298885         0     0   
ZYEDEV:oraecd 43> kill 21958
ZYEDEV:oraecd 44> ps -ef | grep 21958
oraecd 12222  8864  0 11:18:42 pts/ta    0:00 grep 21958
ZYEDEV:oraecd 45>