Oracle Fusion Middleware 12c: 11/18/14

Tuesday 18 November 2014

Unable to start the Weblogic AdminServer 12c: Server subsystem failed. Reason: java.lang.NullPointerException

Scenario:

Users were unable to start the AdminServer ( weblogic 12c). The startup failed with following error

####<Jul 31, 2013 4:47:04 AM PDT> <Critical> <WebLogicServer> <host02> <AdminServer> <main> <<WLS Kernel>> <> <> <1375271224766> <BEA-000386> <Server subsystem failed. Reason: java.lang.NullPointerException
java.lang.NullPointerException at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:571) at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:556) at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:575) at weblogic.management.internal.PendingDirectoryManager.find(PendingDirectoryManager.java:556) at weblogic.management.internal.PendingDirectoryManager.getAllFiles(PendingDirectoryManager.java:413) at weblogic.management.provider.internal.EditAccessImpl.<init>(EditAccessImpl.java:165)

Reason:

One user stopped and restarted the Admin service as "root" user instead of the S/W owner (which was oracle). He also tried to create datasources for the application after starting up the weblogic Admin server.

Impact:

The action modified the ownership and permissions on many of the domain's folders to root. This was stopping the Admin server from coming up.

Tried few work around like renaming the AdminServer folder and then starting up, removing the replicas.prop file under AdminServer/data/ldap/conf directory and renaming WLS_DIAGNOSTICS000000.DAT file under AdminServer/data/store/diagnostics location. But nothing worked in this case.

Solution:

Check for the permission of "pending" folder under $DOMAIN_HOME. If it is under root move it back to oracle (or the original WLS S/W owner).

Also check whether we have any files under pending. If so we can rename them too

chown -R oracle:oinstall pending

This did the trick. After this I was able to start up the AdminServer.

Error while starting WebLogic Server : java.lang.NumberFormatException: null

I was working on WebLogic Server where Admin Server failed to start because Disk was 100% full. Admin Server failed to start after clearing some unwanted files from server with below error message

Error message in Admin Server log file $DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log

__________________
<aug 13, 2010 2:49:51 PM BST> <Critical> <WebLogicServer> <BEA-000386>
<Server subsystem failed. Reason: java.lang.NumberFormatException: null
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at weblogic.ldap.EmbeddedLDAP.validateVDEDirectories(EmbeddedLDAP.java:1035)
at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:212)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
_________________

Since error was point to embedded LDAP server for weblogic so I compared files related to embedded ldap server ($DOMAIN_HOME/ servers/ AdminServer/ data/ ldap/) with one of our running weblogic instance and found that size ofreplicas.prop file was 0 byte.

$DOMAIN_HOME/servers/AdminServer/data/ldap/conf

-rw-r—– 1 oraowc dba 568 Aug 13 09:17 vde.prop
-rw-r—– 1 oraowc dba 0 Aug 13 09:17 mapping.cfg
-rw-r—– 1 oraowc dba 0 Aug 13 14:33 replicas.prop

Fix:

Option 1 - Open

$DOMAIN_HOME/servers/AdminServer/data/ldap/conf/replicas.prop and add line
replica.num=0

Option 2- Move file $DOMAIN_HOME/servers/AdminServer/data/ldap/conf/replicas.prop to replicas.prop.bak (On Restart WebLogic Server will recreate this file)

Unable to start Admin server BEA-000386 (Server subsystem failed. Reason: java.lang.AssertionError: java.lang.reflect.InvocationTargetException)

Problem: Not able to start Admin server with BEA-000386 Error.

Symptom:
Weblogic Server fail with below error:

<Apr 30, 2014 5:50:48 PM IST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: java.lang.reflect.In
ocationTargetException
java.lang.AssertionError: java.lang.reflect.InvocationTargetException
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:175)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1092)
at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:737)
Truncated. see log file for complete stacktrace
Caused By: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173)
Truncated. see log file for complete stacktrace
Caused By: weblogic.security.internal.encryption.EncryptionServiceException
at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:139)
at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:187)
at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
Truncated. see log file for complete stacktrace
>

Solution:
The file SerializedSystemIni.dat has become corrupted. Admin Server will not start because it is not able to decrypt the credential and perform authentication properly.

Take backup of config.xml file and replace all the encrypted values with plain text, like below:

<credential-encrypted>{AES}HLsT8dQxvGB0z6NBItIlMaiX/kYyI/d7SNqhwnqL/xot2gUAj7EglBJeM2mkynJDZUbwCe7cN6hwVRU5jZkLOZLA3C/WUqlc6voRI9mKiPTfaqMvqNSJsny+gas8V8cO</credential-encrypted>

<credential-encrypted>SimpleText</credential-encrypted>

Once you restart the Admin Server, all the plain text values will get encrypted again.

Error: Server subsystem failed. Reason: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 while Starting Weblogic managed server !!

Issue details :- we got below exception during weblogic manged server and UI showing status FAILED TO START SERVER. xxx_server.out file saw below exception.. ..

--------------- Exception in .OUT logs

<Jul 30, 2014 8:33:57 PM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

at java.util.ArrayList.RangeCheck(ArrayList.java:547)

at java.util.ArrayList.get(ArrayList.java:322)

at weblogic.rmi.cluster.BasicReplicaList.getPrimary(BasicReplicaList.java:90)

at weblogic.rmi.cluster.ClusterableRemoteRef.getPrimaryRef(ClusterableRemoteRef.java:155)

at weblogic.rmi.cluster.ClusterableRemoteRef.add(ClusterableRemoteRef.java:474)

Truncated. see log file for complete stacktrace

<Jul 30, 2014 8:33:57 PM EDT> <Warning> <oracle.soa.services.notification> <BEA-000000> <<.> With the current setting, only Email notifications will be sent; Notifications via voice, SMS or IM will not be sent. If you would like to enable them, please configure corresponding sdpmessaging driver. Then modify the accounts and set NotificationMode attribute to ALL in workflow-notifica tion-config.xml>

[TopLink Info]: 2014.07.30 20:33:58.166--ServerSession(301646737)--deferred_session logout successful

[TopLink Info]: 2014.07.30 20:33:58.166--ServerSession(304814922)--tracking_session logout successful

---------------------------- OR ----------------------------------

Solution :- Please verify Disk space availabe in System[if yes the clean unnecessary files and make sure available disk space should be 20-40% of total disk space] then shutdown whole domain including node manager[NM] {FYI- I believe there is no command available to stop NM. Please kill it kill -9 [PID] }

NOTE:- Rolling restart doesn't help. must be stop whole domain and start

Step1:- Start NM then start Admin servers.

Step2:- Before start the Managed servers Clean tmp and cache folders under @DOMAIN_HOME/servers/xxx_server1/

it's worked for our environment all time[Please let me know if issue is not resolved..then will update this blog...]

Create WebLogic 12C Datasource using WLST Scripting Tool

Creating and configuring various resources in WebLogic Server using WLST tool, where we used properties file data as an input to the resource configuration. Here is an example in which we can see How to Configure WebLogic DataSource using WLST tool

In bellow sample, use a properties file to avoid hard coding. This script will be same for various DataSource configuration and need to change the Properties files entry to make a new DataSource.

Step1). Create a Directory (Based on space availability) in your file system:

”C:WLST_DS_Creation” and place a properties file like below in this directory with name “Datasource.properties”

Datasource.properties:

Location of Properties:- C:\Users\krushnavenus\Desktop\Datasource\datasource.properties

Datasource.py:

Step2). Create Datasource in different directory by following WLST script

Create “Datasource.py”:

Location of Datasource.py:- C:\Users\krushnavenus\Desktop\Datasource\datasource.py

# ############### Connecting to Start #################################

def connectAdmin() :

try:

connect(CONUSR,CONPWD, CONURL)

print('Successfully connected')

except:

print 'Unable to find admin server...'

exit()

################### Configuring Connection Pool #############################

def connPool(DSnam) :

DRVPARM='/JDBCSystemResources/'+DSnam+'/JDBCResource/'+DSnam+'/JDBCDriverParams/'+DSnam

cd(DRVPARM)

set('Url',DBURL)

set('DriverName',DBDRV)

set('Password',DBPASS)

cd(DRVPARM+'/Properties/'+DSnam)

cmo.createProperty('user')

cd(DRVPARM+'/Properties/'+DSnam+'/Properties/user')

set('Value',DBUSR)

############ Creating Data source ###############################

def createDS() :

print('Naming the datasource')

DSnam = DSName

cmo.createJDBCSystemResource(DSnam)

RESOURCE='/JDBCSystemResources/'+DSnam+'/JDBCResource/'+DSnam

cd(RESOURCE)

set('Name',DSnam)

#Setting JNDI name

cd(RESOURCE+'/JDBCDataSourceParams/'+DSnam)

print RESOURCE+'/JDBCDataSourceParams/'+DSnam

set('JNDINames',jarray.array([String(JNDIName)], String))

connPool(DSnam)

#Set Connection Pool specific parameters

cd(RESOURCE+'/JDBCConnectionPoolParams/'+DSnam)

cmo.setTestConnectionsOnReserve(true)

cmo.setTestTableName('SQL SELECT 1 FROM DUAL')

cmo.setConnectionReserveTimeoutSeconds(25)

cmo.setMaxCapacity(15)

cmo.setConnectionReserveTimeoutSeconds(10)

cmo.setTestFrequencySeconds(120)

cd(RESOURCE+'/JDBCDataSourceParams/'+DSnam)

cmo.setGlobalTransactionsProtocol('TwoPhaseCommit')

cd('/JDBCSystemResources/'+DSnam)

# targets the DS to Servers(Cluster or Server)

#targetType=raw_input('Target to (C)luster or (S)erver: ')

#if targetType in ('C','c') :

if targetType == 'true':

#clstrNam=raw_input('Cluster Name: ')

set('Targets',jarray.array([ObjectName('com.bea:Name='+clstrNam+',Type=Cluster')], ObjectName))

else:

#servr=raw_input('Server Name: ')

set('Targets',jarray.array([ObjectName('com.bea:Name='+servr+',Type=Server')], ObjectName))

############### Main Script #####################################

if __name__== "main":

print('This will enable you to create or update a Datasource')

connectAdmin()

edit()

startEdit()

# Create a new JDBC resource)

cd('/')

createDS()

save()

activate()

dumpStack()

disconnect()

####################################

execution command:-

syntax:

$MW_HOME/$WL_HOME/common/bin/wlst.sh –loadProperties datasource.properties datasource.py location

Execution command:-

D:\u02\app\Oracle\Middleware\Oracle_Home\wlserver\common\bin>wlst.cmd -loadProperties C:\Users\krushnavenus\Desktop\Datasource\datasource.propertiesC:\Users\krushnavenus\Desktop\Datasource\datasource.py

OUT PUT OF DATASOURCES:-

A JDBC data source is an object bound to the JNDI tree that provides database connectivity through a pool of JDBC connections. Applications can look up a data source on the JNDI tree and then borrow a database connection from a data source.

This Screen shot summarizes the JDBC data source objects that have been created in this domain.

Tuning JRockit

Tuning the Heap Size:

A large heap decreases the garbage collection frequency but may take slightly longer to garbage collect.
Typically a heap should be at least twice the size of the live objects in the heap.
Should set the heap as large as the PHYSICAL memory in your system will allow, as long as it doesn’t cause paging.
-Xms:, which sets the initial and minimum heap size.
-Xmx:, which sets the maximum heap size.
Set the initial/minimum heap size (-Xms) to the same value as the maximum heap size (-Xmx).

Tuning the Garbage Collection

JRockit offers three garbage collection modes and a number of static garbage collection strategies. You can tune the garbage collection to suit your application’s needs.
-XgcPrio:throughput: Defines that the garbage collection should be optimized for application throughput. This is the default garbage collection mode.
-XgcPrio:pausetime : Defines that the garbage collection should be optimized for short garbage collection pauses.
-XgcPrio:deterministic {Only with JRockit Real-Time}: Defines that GC should be optimized for very short and deterministic garbage collection pauses.

Tuning the Pause Target

This parameter is used to tune JVMs which host ‘time critical applications’ (Ex: A banking application that normally take 100 ms to complete and times out after 400 ms, cannot afford a GC pause time of 500ms)
The pause time mode uses a pause target for optimizing the pause times; pause target should be as high as your application can tolerate
This parameter is coupled and used with ‘-XgcPrio:pausetime’ (ex: java -XgcPrio:pausetime -XpauseTarget:300ms weblogic.Server); default ‘pauseTarget’ is not specified is 500ms

# Below given parameters should not be changes until you

feel a real need to tune your Weblogic server; incorrect

 tuning may lead to uneven performance degradation.

Tuning Compaction

Whenever GC(old collection) runs it defragments the heap, making object allocation easier for the JVM this process is called heap compaction; compaction may lead to long pauses in GC because it takes up the CPU time to arrange and defragment the heap
-XXcompactRatio:centage> : this specifies the percentage of heap which will be defragmented when GC runs; a good value for the compact ratio is usually between 1 and 20; a too low value can also invite trouble as it slows down defragmentation and increases the amount of dark matter(“Dark matter” is wasted heap memory and fragments the heap)
-XXcompactSetLimit:<references> : When compaction has moved objects, the references to these objects must be updated. JVM has to do this before threads have access to those memory references; this parameter defines how many references GC can compact within the compaction area (Ex: java -XXcompactSetLimit:20000 MyApplication)

Tuning the TLA (Thread Local Area)

It is the chunk of memory reserved in the heap for a thread for its exclusive use.
When this TLA gets filled up we see the ‘java.lang.OutOfMemoryError: nativeGetNewTLA’, to fix this we can tune the ‘-XXtlaSize:min=e>,preferred=’ parameter increase the TLA (Ex: -XXtlasize:min=8k,preferred=512k)
Increasing the TLA size is beneficial for multi threaded applications; however increasing it too much may result in more fragmentation and more frequent garbage collections.

Basic Unix commands

DOS Command	Unix Command	Descriptions
CD	cd	Change directory
CHKDSK	du	Disk usage
CLS	clear	Clear the current screen
COPY	cp	Copying files
DEL	rm	Removing files or directories
DIR	ls	File listing of directories
MD	mkdir	Create a directory
MORE	more	Type out a file with paging
PRINT	lpr	Print out a file
RD	rmdir	Remove a directory
RENAME	mv	Moving files around
TYPE	cat	Type out files

Unix Command	Descriptions
rm file	Remove a file
rm file1 file2 …	Remove a number of files
rm -r directory	Remove a directory include the sub-directory
rmdir directory	Remove a directory

Unix Command	Descriptions
man cp	Display on-line manual for the "cp" command
man -k keyword	Display manual help file related to the keyword
passwd	Change your login password
pwd	Display the path name of where you are
uptime	Tell you how long the machine has been up and running
users	Tell you who is logging in
who	Tell you who is logging-in in detail
w	Tell you who is logging in and doing what!
whoami	Show you the owner of this account
finger user	Find out the personal information of a user
finger name	Try to find the person's info. by his/her name
finger email-address	Try to find the person's info across the network
write user	Write a message on somebody's screen
talk user	Talk to the person logging in the same system with you
talk email-address	Talk to somebody logging in the network
date	Display today's time and date
cal year	Display the calendar of the specified year (e.g. 1997)