Unable to Activate Changes in EBS Weblogic Domain after making a change from Admin Console/WLST

[Su]

We are facing a strange issue.

We are having EBS 12.2.8 and bundled WLS server 10.3.6. When all MAnaged Servers(OACORE,OAFM,FORMS,Accessgate etc) are ip and running and we are trying to Do some Domain Level changes(e.g. Chnage Server Start Parameter, Increase Message Timeout etc etc)  from WLS Admin Console/WLST  the Active Changes never gets completes.

If we do same by Shutting down all managed servers leaving Admin up only changes succeed. We did lots of follow up with Oracle Support. We have environment with JAN 2020 WLS PSU,Overlay applied or Old PSU applied. No patch was able to fix this.

Taking thread dumps can only show as follows:






2020-04-17 08:58:26
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.241-b09 mixed mode):

"[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71a000f800 nid=0x55e1 in Object.wait() [0x00007f70eb2eb000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cc8c38e8> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000cc8c38e8> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71a000d800 nid=0x55e0 in Object.wait() [0x00007f70eb3ec000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cc8e1a90> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000cc8e1a90> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71a000b800 nid=0x55df in Object.wait() [0x00007f70eb4ec000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f208e440> (a weblogic.rjvm.ResponseImpl)
        at weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:95)
        - locked <0x00000000f208e440> (a weblogic.rjvm.ResponseImpl)
        at weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:135)
        at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:110)
        at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:228)
        at javax.management.remote.rmi.RMIConnectionImpl_1036_WLStub.fetchNotifications(Unknown Source)
        at weblogic.management.remote.common.RMIConnectionWrapper$26.run(ClientProviderBase.java:1447)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
        at weblogic.security.Security.runAs(Security.java:61)
        at weblogic.management.remote.common.RMIConnectionWrapper.fetchNotifications(ClientProviderBase.java:1445)
        at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1341)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:588)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
        at weblogic.management.mbeanservers.domainruntime.internal.JMXExecutor$1.timerExpired(JMXExecutor.java:98)
        at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:284)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71a0009800 nid=0x55de in Object.wait() [0x00007f70eb5ee000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cc8d38e8> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000cc8d38e8> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71a0008800 nid=0x55dd in Object.wait() [0x00007f70eb6ef000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cc8e4d70> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000cc8e4d70> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71a0007000 nid=0x55dc in Object.wait() [0x00007f7162f9a000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cc8d2798> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000cc8d2798> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"SocketTimeout" daemon prio=10 tid=0x00007f7190152800 nid=0x5512 waiting on condition [0x00007f70eb7f0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at HTTPClient.SocketTimeout.run(StreamDemultiplexor.java:1240)

"DynamicListenThread[Default]" daemon prio=10 tid=0x00007f719004d800 nid=0x5469 runnable [0x00007f70eb8f1000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:403)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
        at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
        at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
        at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
        at java.lang.Thread.run(Thread.java:745)

"weblogic.GCMonitor" daemon prio=10 tid=0x00007f70dc014000 nid=0x5468 in Object.wait() [0x00007f70eb9f2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cbf2ac60> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        - locked <0x00000000cbf2ac60> (a java.lang.ref.ReferenceQueue$Lock)
        at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88)
        at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64)

"CacheCleaner" daemon prio=10 tid=0x00007f7197e33000 nid=0x5463 in Object.wait() [0x00007f70ebaf3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c9a0f778> (a oracle.ias.cache.TimeQ)
        at oracle.ias.cache.TimeQ.waitForWork(TimeQ.java:178)
        - locked <0x00000000c9a0f778> (a oracle.ias.cache.TimeQ)
        at oracle.ias.cache.CacheCleaner.run(CacheCleaner.java:116)

"Worker-2" daemon prio=10 tid=0x00007f7197e4f000 nid=0x5462 in Object.wait() [0x00007f70ebbf4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
        - locked <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
        - locked <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)

"Worker-1" daemon prio=10 tid=0x00007f7197e4e800 nid=0x5461 in Object.wait() [0x00007f70ebdf6000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
        - locked <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
        - locked <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)

"Worker-0" daemon prio=10 tid=0x00007f7197e74000 nid=0x5460 in Object.wait() [0x00007f70ebcf5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
        - locked <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
        - locked <0x00000000c99ec6b0> (a oracle.ias.cache.TaskQ)
        at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)

"RepolessTargetManager" daemon prio=10 tid=0x00007f7197b01800 nid=0x545c in Object.wait() [0x00007f716010d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c8483280> (a oracle.sysman.emdrep.repoless.RepolessTargetManager)
        at oracle.sysman.emdrep.repoless.RepolessTargetManager.waitNewAgent(RepolessTargetManager.java:483)
        - locked <0x00000000c8483280> (a oracle.sysman.emdrep.repoless.RepolessTargetManager)
        at oracle.sysman.emdrep.repoless.RepolessTargetManager.run(RepolessTargetManager.java:920)
        at java.lang.Thread.run(Thread.java:745)

"HealthMonitor" daemon prio=10 tid=0x00007f7197ab7800 nid=0x545b in Object.wait() [0x00007f71604e2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c8713010> (a oracle.sysman.util.healthMonitor.HealthMonitor)
        at oracle.sysman.util.healthMonitor.HealthMonitor.run(HealthMonitor.java:282)
        - locked <0x00000000c8713010> (a oracle.sysman.util.healthMonitor.HealthMonitor)
        at java.lang.Thread.run(Thread.java:745)

"Timer-5" daemon prio=10 tid=0x00007f71979d8000 nid=0x5457 in Object.wait() [0x00007f71607e5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c84fe718> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000000c84fe718> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"EmChartCacheWorker" daemon prio=10 tid=0x00007f719799d800 nid=0x5456 in Object.wait() [0x00007f71606e4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c8537de0> (a oracle.sysman.emSDK.chart.EmChartCacheWorker)
        at java.lang.Object.wait(Object.java:503)
        at oracle.sysman.emSDK.chart.EmChartCacheWorker.run(EmChartCacheManager.java:393)
        - locked <0x00000000c8537de0> (a oracle.sysman.emSDK.chart.EmChartCacheWorker)

"Timer-4" daemon prio=10 tid=0x00007f7197852000 nid=0x544c in Object.wait() [0x00007f71608e6000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c7fc9978> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000c7fc9978> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Java2D Disposer" daemon prio=10 tid=0x00007f71977f4000 nid=0x544b in Object.wait() [0x00007f71605e3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c811af10> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        - locked <0x00000000c811af10> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at sun.java2d.Disposer.run(Disposer.java:145)
        at java.lang.Thread.run(Thread.java:745)

"DmsThread-1" daemon prio=10 tid=0x00007f7197146000 nid=0x5437 waiting on condition [0x00007f71609e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c6c64948> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1075)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at oracle.dms.aggregator.DmsThreadFactory$1$1.run(DmsThreadFactory.java:56)
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.dms.aggregator.DmsThreadFactory$1.run(DmsThreadFactory.java:51)
        at java.lang.Thread.run(Thread.java:745)

"[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f7100001000 nid=0x5423 in Object.wait() [0x00007f716224c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0e75ba0> (a weblogic.management.provider.internal.ActivateTaskImpl)
        at weblogic.management.provider.internal.ActivateTaskImpl.waitForCompletion(ActivateTaskImpl.java:398)
        - locked <0x00000000f0e75ba0> (a weblogic.management.provider.internal.ActivateTaskImpl)
        at weblogic.management.provider.internal.ActivateTaskImpl.waitForTaskCompletion(ActivateTaskImpl.java:377)
        at weblogic.management.provider.internal.EditAccessImpl.activateChangesAndWaitForCompletion(EditAccessImpl.java:963)
        at weblogic.management.mbeanservers.edit.internal.ConfigurationManagerMBeanImpl.activate(ConfigurationManagerMBeanImpl.java:371)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:437)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.edit.internal.RecordingInterceptor.invoke(RecordingInterceptor.java:199)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.edit.internal.EditLockInterceptor.invoke(EditLockInterceptor.java:112)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1488)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1329)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1428)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:849)
        at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
        at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:190)
        at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:228)
        at javax.management.remote.rmi.RMIConnectionImpl_1036_WLStub.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1021)
        at weblogic.management.jmx.MBeanServerInvocationHandler.doInvoke(MBeanServerInvocationHandler.java:545)
        at weblogic.management.jmx.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:381)
        at com.sun.proxy.$Proxy130.activate(Unknown Source)
        at com.bea.console.utils.MBeanChangeManager.distributeChanges(MBeanChangeManager.java:335)
        at com.bea.console.actions.changemgmt.ActivateChangesAction.execute(ActivateChangesAction.java:40)
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
        at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.access$201(PageFlowRequestProcessor.java:97)
        at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor$ActionRunner.execute(PageFlowRequestProcessor.java:2044)
        at org.apache.beehive.netui.pageflow.interceptor.action.internal.ActionInterceptors.wrapAction(ActionInterceptors.java:91)
        at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.processActionPerform(PageFlowRequestProcessor.java:2116)
        at com.bea.console.internal.ConsolePageFlowRequestProcessor.processActionPerform(ConsolePageFlowRequestProcessor.java:269)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:237)
        at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.processInternal(PageFlowRequestProcessor.java:556)
        at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:853)
        at org.apache.beehive.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:631)
        at org.apache.beehive.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:158)
        at com.bea.console.internal.ConsoleActionServlet.process(ConsoleActionServlet.java:262)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:416)
        at com.bea.console.internal.ConsoleActionServlet.doGet(ConsoleActionServlet.java:134)
        at org.apache.beehive.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1199)
        at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:686)
        at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:142)
        at com.bea.portlet.adapter.scopedcontent.StrutsStubImpl.processAction(StrutsStubImpl.java:76)
        at com.bea.portlet.adapter.NetuiActionHandler.raiseScopedAction(NetuiActionHandler.java:111)
        at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:181)
        at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:167)
        at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:225)
        at com.bea.netuix.nf.ControlLifecycle$2.visit(ControlLifecycle.java:180)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:324)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)
        at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:130)
        at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:395)
        at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:361)
        at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:352)
        at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:184)
        at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:159)
        at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:388)
        at com.bea.netuix.servlets.manager.UIServlet.doPost(UIServlet.java:258)
        at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:199)
        at com.bea.netuix.servlets.manager.SingleFileServlet.service(SingleFileServlet.java:251)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at com.bea.console.utils.MBeanUtilsInitSingleFileServlet.service(MBeanUtilsInitSingleFileServlet.java:64)
        at weblogic.servlet.AsyncInitServlet.service(AsyncInitServlet.java:130)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3812)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3778)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2336)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2235)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1530)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71964e7000 nid=0x539b in Object.wait() [0x00007f716154b000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c6386e90> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000c6386e90> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f7108001800 nid=0x5396 in Object.wait() [0x00007f7161742000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cbf609a8> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:449)
        - locked <0x00000000cbf609a8> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:227)
        at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:275)
        at javax.management.remote.rmi.RMIConnectionImpl$4.run(RMIConnectionImpl.java:1290)
        at javax.management.remote.rmi.RMIConnectionImpl$4.run(RMIConnectionImpl.java:1288)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1296)
        at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
        at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
        at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
        at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
        at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

"[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f7190017800 nid=0x5395 in Object.wait() [0x00007f71619de000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c639c698> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000c639c698> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"Thread-43" daemon prio=10 tid=0x00007f7170019800 nid=0x5394 runnable [0x00007f7161d89000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000000c639d310> (a java.io.BufferedInputStream)
        at com.sun.jndi.ldap.Connection.run(Connection.java:886)
        at java.lang.Thread.run(Thread.java:745)

"Thread-42" daemon prio=10 tid=0x00007f7170018800 nid=0x5393 runnable [0x00007f7162456000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000000c639dd58> (a java.io.BufferedInputStream)
        at com.sun.jndi.ldap.Connection.run(Connection.java:886)
        at java.lang.Thread.run(Thread.java:745)

"Thread-41" daemon prio=10 tid=0x00007f7170018000 nid=0x5392 runnable [0x00007f7162557000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000000c63be840> (a java.io.BufferedInputStream)
        at com.sun.jndi.ldap.Connection.run(Connection.java:886)
        at java.lang.Thread.run(Thread.java:745)

"Thread-40" daemon prio=10 tid=0x00007f7170017000 nid=0x5391 runnable [0x00007f71b812b000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000000c63bf288> (a java.io.BufferedInputStream)
        at com.sun.jndi.ldap.Connection.run(Connection.java:886)
        at java.lang.Thread.run(Thread.java:745)

"LogFlusher" daemon prio=10 tid=0x00007f715c2bc800 nid=0x5390 waiting on condition [0x00007f716295b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)

"LogFlusher" daemon prio=10 tid=0x00007f7195504800 nid=0x538c waiting on condition [0x00007f7162153000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)

"oracle.dfw.incident.FloodController" daemon prio=10 tid=0x00007f71954ef800 nid=0x5387 in Object.wait() [0x00007f7162355000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c497e7d0> (a java.lang.Object)
        at oracle.dfw.impl.incident.FloodController.run(FloodController.java:136)
        - locked <0x00000000c497e7d0> (a java.lang.Object)
        at java.lang.Thread.run(Thread.java:745)

"oracle.dfw.incident.IncidentCreatorThread" daemon prio=10 tid=0x00007f71954eb800 nid=0x5384 in Object.wait() [0x00007f7162658000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c3a39020> (a java.util.ArrayList)
        at java.lang.Object.wait(Object.java:503)
        at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$IncidentCreatorThread.run(DiagnosticsDataExtractorImpl.java:2189)
        - locked <0x00000000c3a39020> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:745)

"oracle.dfw.impl.incident.ADRHelper - purge incidents" daemon prio=10 tid=0x00007f71954ea800 nid=0x5382 in Object.wait() [0x00007f716285a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c3a35400> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000000c3a35400> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"oracle.dfw.impl.incident - ADR Executor (created: Fri Apr 17 08:38:55 BST 2020)" daemon prio=10 tid=0x00007f71954e9000 nid=0x5380 waiting on condition [0x00007f7162759000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c3a36838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1075)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:745)

"DFW Diagnostic Dump Sampling - JavaClassHistogram" daemon prio=10 tid=0x00007f7194efd000 nid=0x537b in Object.wait() [0x00007f7162a95000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c3a97cd0> (a [B)
        at oracle.dfw.sampling.DumpSampling.run(DumpSampling.java:549)
        - locked <0x00000000c3a97cd0> (a [B)

"DFW Diagnostic Dump Sampling - JVMThreadDump" daemon prio=10 tid=0x00007f7194efb800 nid=0x537a in Object.wait() [0x00007f7162b96000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c3a96228> (a [B)
        at oracle.dfw.sampling.DumpSampling.run(DumpSampling.java:549)
        - locked <0x00000000c3a96228> (a [B)

"LogFlusher" daemon prio=10 tid=0x00007f719545c000 nid=0x5377 waiting on condition [0x00007f7162c97000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)

"LogFlusher" daemon prio=10 tid=0x00007f719545a000 nid=0x5376 waiting on condition [0x00007f7162d98000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)

"LogFlusher" daemon prio=10 tid=0x00007f7195457800 nid=0x5375 waiting on condition [0x00007f7162e99000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)

"Thread-16" prio=10 tid=0x00007f7194e36800 nid=0x536e waiting on condition [0x00007f716309b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c3b3f4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
        at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676)
        at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751)
        at java.lang.Thread.run(Thread.java:745)

"Timer-2" daemon prio=10 tid=0x00007f7194a93000 nid=0x536a in Object.wait() [0x00007f716399e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c317d158> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000c317d158> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Runtime Authorization Event Notifier" daemon prio=10 tid=0x00007f7194aa9800 nid=0x5369 in Object.wait() [0x00007f7163a9f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c3485d00> (a oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl)
        at java.lang.Object.wait(Object.java:503)
        at oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl.run(RuntimeEventNotifierImpl.java:230)
        - locked <0x00000000c3485d00> (a oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl)
        at java.lang.Thread.run(Thread.java:745)

"JMX Framework document pooling thread" daemon prio=10 tid=0x00007f715c021000 nid=0x5368 waiting on condition [0x00007f7163ba0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at oracle.as.config.notification.filesystem.WatchingDocumentChangeNotifier.run(WatchingDocumentChangeNotifier.java:252)
        at java.lang.Thread.run(Thread.java:745)

"JPS FileChangeNotifier<clean-only>" daemon prio=10 tid=0x00007f7194947800 nid=0x5366 waiting on condition [0x00007f71b832d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c290f7f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1075)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:745)

"DoSManager" daemon prio=10 tid=0x00007f716400b800 nid=0x5365 waiting on condition [0x00007f71b822c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.octetstring.vde.DoSManager.run(DoSManager.java:433)

"VDE Replication Thread" daemon prio=10 tid=0x00007f7194372000 nid=0x5363 in Object.wait() [0x00007f71b842e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2a53f10> (a com.octetstring.vde.replication.Replication)
        at com.octetstring.vde.replication.Replication.wait30sec(Replication.java:384)
        - locked <0x00000000c2a53f10> (a com.octetstring.vde.replication.Replication)
        at com.octetstring.vde.replication.Replication.run(Replication.java:350)

"VDE Transaction Processor Thread" daemon prio=10 tid=0x00007f719439e000 nid=0x5362 in Object.wait() [0x00007f71b852f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2ab3518> (a com.octetstring.vde.backend.standard.TransactionProcessor)
        at java.lang.Object.wait(Object.java:503)
        at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)
        - locked <0x00000000c2ab3518> (a com.octetstring.vde.backend.standard.TransactionProcessor)
        at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x00007f719411b800 nid=0x5360 waiting for monitor entry [0x00007f71b8630000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
        - waiting to lock <0x00000000c2b2fe08> (a weblogic.socket.PosixSocketMuxer$1)
        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x00007f719411a000 nid=0x535f runnable [0x00007f71b8731000]
   java.lang.Thread.State: RUNNABLE
        at weblogic.socket.PosixSocketMuxer.poll(Native Method)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102)
        - locked <0x00000000c2b2fe08> (a weblogic.socket.PosixSocketMuxer$1)
        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x00007f7194130000 nid=0x535e waiting for monitor entry [0x00007f71b8832000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95)
        - waiting to lock <0x00000000c2b2fe08> (a weblogic.socket.PosixSocketMuxer$1)
        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x00007f719411c800 nid=0x535d waiting for monitor entry [0x00007f71b8933000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
        - waiting to lock <0x00000000c2b2fe08> (a weblogic.socket.PosixSocketMuxer$1)
        at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
        at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

"NM Reaper" daemon prio=10 tid=0x00007f7194096800 nid=0x534c in Object.wait() [0x00007f71b8b37000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2b447b0> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000000c2b447b0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Thread-7" prio=10 tid=0x00007f7194099800 nid=0x534b waiting on condition [0x00007f71b8d3b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c26c8950> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
        at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676)
        at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751)
        at java.lang.Thread.run(Thread.java:745)

"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71f5f3c800 nid=0x534a in Object.wait() [0x00007f71b9041000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c26f3568> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
        - locked <0x00000000c26f3568> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:233)

"weblogic.timers.TimerThread" daemon prio=10 tid=0x00007f71f5f32000 nid=0x5349 in Object.wait() [0x00007f71b9142000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2c1cf90> (a java.lang.Object)
        at weblogic.timers.internal.TimerThread.doWait(TimerThread.java:114)
        - locked <0x00000000c2c1cf90> (a java.lang.Object)
        at weblogic.timers.internal.TimerThread.access$000(TimerThread.java:22)
        at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:290)

"weblogic.time.TimeEventGenerator" daemon prio=10 tid=0x00007f71f5f1c800 nid=0x5348 in Object.wait() [0x00007f71b9344000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2902aa8> (a weblogic.time.common.internal.TimeTable)
        at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)
        - locked <0x00000000c2902aa8> (a weblogic.time.common.internal.TimeTable)
        at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117)
        at java.lang.Thread.run(Thread.java:745)

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f71f5ee4000 nid=0x5347 in Object.wait() [0x00007f71b964d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000cc09dc40> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:449)
        - locked <0x00000000cc09dc40> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
        at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:227)
        at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:275)
        at javax.management.remote.rmi.RMIConnectionImpl$4.run(RMIConnectionImpl.java:1290)
        at javax.management.remote.rmi.RMIConnectionImpl$4.run(RMIConnectionImpl.java:1288)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1296)
        at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
        at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
        at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
        at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
        at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

"Timer-1" daemon prio=10 tid=0x00007f71f5ed5000 nid=0x5346 in Object.wait() [0x00007f71b974f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2b41db8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000000c2b41db8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Timer-0" daemon prio=10 tid=0x00007f71f57af800 nid=0x5341 in Object.wait() [0x00007f71b98c3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2890128> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000c2890128> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Service Thread" daemon prio=10 tid=0x00007f71f41ca800 nid=0x533d runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f71f41c8000 nid=0x533c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f71f41c5000 nid=0x533b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f71f41bc800 nid=0x533a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f71f41a5800 nid=0x5339 in Object.wait() [0x00007f71f08f7000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c265c280> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        - locked <0x00000000c265c280> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" daemon prio=10 tid=0x00007f71f41a1000 nid=0x5338 in Object.wait() [0x00007f71f09f8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2b423e0> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:503)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:180)
        - locked <0x00000000c2b423e0> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:141)

"main" prio=10 tid=0x00007f71f4011800 nid=0x532e in Object.wait() [0x00007f71fb95e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000c2902860> (a weblogic.t3.srvr.T3Srvr)
        at java.lang.Object.wait(Object.java:503)
        at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:994)
        - locked <0x00000000c2902860> (a weblogic.t3.srvr.T3Srvr)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:494)
        at weblogic.Server.main(Server.java:71)

"VM Thread" prio=10 tid=0x00007f71f419a800 nid=0x5337 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f71f4027000 nid=0x532f runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f71f4029000 nid=0x5330 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f71f402b000 nid=0x5331 runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f71f402d000 nid=0x5332 runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f71f402f000 nid=0x5333 runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f71f4030800 nid=0x5334 runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f71f4032800 nid=0x5335 runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f71f4034800 nid=0x5336 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f71f41d5000 nid=0x533e waiting on condition

JNI global references: 451

Heap
 PSYoungGen      total 300032K, used 125583K [0x00000000eaa80000, 0x0000000100000000, 0x0000000100000000)
  eden space 252416K, 48% used [0x00000000eaa80000,0x00000000f21d0d30,0x00000000fa100000)
  from space 47616K, 7% used [0x00000000fa100000,0x00000000fa4530b8,0x00000000fcf80000)
  to   space 47104K, 0% used [0x00000000fd200000,0x00000000fd200000,0x0000000100000000)
 ParOldGen       total 699392K, used 226619K [0x00000000bff80000, 0x00000000eaa80000, 0x00000000eaa80000)
  object space 699392K, 32% used [0x00000000bff80000,0x00000000cdccef18,0x00000000eaa80000)
 PSPermGen       total 188416K, used 188113K [0x00000000a7f80000, 0x00000000b3780000, 0x00000000bff80000)
  object space 188416K, 99% used [0x00000000a7f80000,0x00000000b37344d8,0x00000000b3780000)

[ErmanArslansOracleBlog]

What is your Weblogic PSU version?
In EBS 12.2.8, it should be at least 10.3.6.0.180417 .

Please check that.

The recommended one is WebLogic PSU 10.3.6.0.200114.
Current minimum recommended version is 10.3.6.0.181016

Ref : Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes (Doc ID 1594274.1)

Also there are some config related corruptions reported. These kinds of issue may result that problem as well -> Timeout While Activating Changes in Admin Console - "java.lang.RuntimeException: Timed out waiting for completion" (Doc ID 2431037.1)

[Su]

Hi Erman,

Thanks for reply.:-)

Yes, we are on Latest WebLogic PSU 10.3.6.0.200114 and have all latest Overlay Patches applied already from 1594274.1.

Also,
we tried clearing config.xml from pending folder, removing .lok files from domain home and config folders.
Even validated RDA output to check ulimit, EBS Storage specific issues etc. All are set correctly.

No solution yet . :-(

Btw, these are Exalogic vServers but i guess that may not made a difference.

[ErmanArslansOracleBlog]

1)that note says, "remove the config file" .. (it doesn't instruct you to clear it) .. Maybe taking a backup and removing it may help .

Timeout While Activating Changes in Admin Console - "java.lang.RuntimeException: Timed out waiting for completion" (Doc ID 2431037.1)

2) ensure you have no time synch issues on your apps servers. (I m talking about server time.. ntp and clock sync.)

3)Check your EBS aware admin server created scripts.. Ensure the ADMIN URL url matches the value of the Admin Server Listen Address.. This may be communication issue between managed servers and admin server (over http)
Also connect to the he WLS Console, make sure the Admin Server has the Listen Address defined.  Environment -> Servers -> AdminServer..  Ensure WLS side of your EBS is configured accordingly..

4) Check your topology and network.. Ensure there is nothing that may prevent the communication between the Admin Server and Managed Servers.  (in case you have a multi node conf..)

5)Enable debug logs for weblogic admin server, reproduce the issue and check the log again..
This may help us see the reason behind.

[Su]

Thanks Erman.

The issue was fixed after setting this autoconfig variable.

<proxybypassdomain oa_var="s_proxybypassdomain" customized="yes"> [our ebs servers domain]</proxybypassdomain>

Post this the Communication from Admin to Managed Servers becomes instantaneous and also console  changes started to take effect.

[ErmanArslansOracleBlog]

Good for you Su :)