Weblogic JMS client threads stuck while refreshing connection

By : cheekoo
Source: Stackoverflow.com
Question!

Greetings,

We are using Spring to integrate with Weblogic JMS server using wlfullclient.jar. Every once in a while, the jmsListenerthreads get stuck. We have observed that this happens while refreshing the connection.

Versions used: wlfullClient.jar : 10.0.1 Spring : 3.0.3

Stack trace for a couple of stuck threads:

    Name: jmsListenerContainer-1770
State: WAITING on [email protected]
Total blocked: 3,681  Total waited: 3,664

Stack trace: 
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
weblogic.jms.client.JMSSession.waitForState(JMSSession.java:4239)
weblogic.jms.client.JMSSession.close(JMSSession.java:1259)
weblogic.jms.client.JMSConnection.closeSessions(JMSConnection.java:1076)
weblogic.jms.client.JMSConnection.mergedCloseAndOnException(JMSConnection.java:950)
   - locked [email protected]
weblogic.jms.client.JMSConnection.close(JMSConnection.java:763)
weblogic.jms.client.ReconnectController.close(ReconnectController.java:426)
org.springframework.jms.connection.ConnectionFactoryUtils.releaseConnection(ConnectionFactoryUtils.java:80)
org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:385)
   - locked [email protected]
org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:856)
org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:838)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:969)
   - locked [email protected]
java.lang.Thread.run(Thread.java:619)

Name: jmsListenerContainer-1755
State: TIMED_WAITING on [email protected]
Total blocked: 3,588  Total waited: 3,605

Stack trace: 
java.lang.Object.wait(Native Method)
weblogic.jms.client.ReconnectController.waitForStateInternal(ReconnectController.java:309)
weblogic.jms.client.ReconnectController.analyzeExceptionAndReconnect(ReconnectController.java:525)
weblogic.jms.client.ReconnectController.computeJMSConsumer(ReconnectController.java:652)
weblogic.jms.client.WLConsumerImpl.receive(WLConsumerImpl.java:166)
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:405)
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1056)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1048)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
java.lang.Thread.run(Thread.java:619)

Name: jmsListenerContainer-1732
State: BLOCKED on [email protected] owned by: jmsListenerContainer-1770
Total blocked: 3,586  Total waited: 3,584

Stack trace: 
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1096)
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:958)
java.lang.Thread.run(Thread.java:619)

Spring configurations:

<bean id="messageListener"
  class="org.springframework.jms.listener.adapter.MessageListenerAdapter">
  <constructor-arg>
   <bean class="com.eharmony.matching.mnssender.jms.DefaultDtoDelegate" />
  </constructor-arg>
  <property name="defaultListenerMethod" value="receive" />
  <property name="messageConverter">
   <bean class="com.eharmony.matching.mnssender.jms.Converter" />
  </property>
 </bean>

 <bean id="jmsListenerContainer"
  class="org.springframework.jms.listener.DefaultMessageListenerContainer">
  <property name="concurrentConsumers" value="${MNSClient.jms.concurrent.consumers}" />
  <property name="connectionFactory" ref="jmsQueueConnectionFactory" />
  <property name="destination" ref="queue" />
  <property name="sessionTransacted" value="true" />
  <property name="messageListener" ref="messageListener" />
 </bean>

 <bean id="jmsQueueConnectionFactory"
  class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter">
  <property name="targetConnectionFactory">
   <ref bean="internalJmsQueueConnectionFactory" />
  </property>
  <property name="username">
   <value>${MNSClient.receive.jms.user}</value>
  </property>
  <property name="password">
   <value>${MNSClient.receive.jms.password}</value>
  </property>
 </bean>

 <bean id="internalJmsQueueConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
  <property name="jndiTemplate">
   <ref bean="jndiTemplate" />
  </property>
  <property name="jndiName">
   <value>${MNSClient.receive.jms.connectionFactoryName}</value>
  </property>
 </bean>

 <!-- RECEIVING JMS MESSAGE -->
 <bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
  <property name="environment">
   <props>
    <prop key="java.naming.factory.initial">${MNSClient.receive.jms.initialFactoryName}</prop>
    <prop key="java.naming.provider.url">${MNSClient.receive.jms.providerUrl}</prop>
   </props>
  </property>
 </bean>
 <bean id="queue" class="org.springframework.jndi.JndiObjectFactoryBean">
  <property name="jndiTemplate">
   <ref bean="jndiTemplate" />
  </property>
  <property name="jndiName">
   <value>${MNSClient.receive.jms.queueName}</value>
  </property>
 </bean>

Any pointers will be highly appreciated!

Regards.

By : cheekoo


Answers

Are both those threads calling methods on the same Session (or Consumer/Producer derived from the same session)? Session and anything derived from it (Producer, Consumer, etc) are not thread-safe, so you need to protect against multiple threads simultaneously operating on them. Either by serializing access through your own locking or by creating different Sessions for each Producer/Consumer.

Don't forget the implicit thread involved in async message processing (onMessage() use) when counting whether you're accessing a session from multiple threads.

If you're violating that rule, expect bad things (like deadlocks).

By : John M


Ok, with some more googling, I found the answer. We had hit a bug with Spring JMS integration: https://jira.springframework.org/browse/SPR-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel#issue-tabs

This has been fixed now, and we would update to latest version of Spring.

By : cheekoo


This video can help you solving your question :)
By: admin