JUnit, DBUnit and Oracle

As previously blogged, DBUnit is a powerful addition to your unit test armoury. Having used it with Oracle, there are a few nuances which are worth writing about and therefore remembering!

When creating a connection to an Oracle database, DBUnit will populate a Map of table names. By default these will not be prefixed with the schema name. As a result, you can get duplicate table names. An example of an exception raised for the duplicate table WWV_FLOW_DUAL100 is shown below:

org.dbunit.database.AmbiguousTableNameException: WWV_FLOW_DUAL100 at org.dbunit.dataset.OrderedTableNameMap.add(OrderedTableNameMap.java:198)
 at org.dbunit.database.DatabaseDataSet.initialize(DatabaseDataSet.java:231)
 at org.dbunit.database.DatabaseDataSet.getTableMetaData(DatabaseDataSet.java:281)
 at org.dbunit.operation.AbstractOperation.getOperationMetaData(AbstractOperation.java:80)
 at org.dbunit.operation.AbstractBatchOperation.execute(AbstractBatchOperation.java:140)

To resolve this, set the database config property FEATURE_QUALIFIED_TABLE_NAMES to be true. This will make sure all tables in the Map are unique. As a consequence of this, the table names in the XML data file will need to be prefixed with the schema name.

Another useful database config property is PROPERTY_DATATYPE_FACTORY. In the XML data file, if there are dates with the time element set then the time element will be ignored unless the database config property PROPERTY_DATATYPE_FACTORY is set to new Oracle10DataTypeFactory() (or the equivalent data factory for the version of Oracle being used.)

An example of setting these values in a @BeforeClass annotated JUnit method is shown below:

@BeforeClass
public static void loadDataset() throws Exception {

    // database connection
    ResourceCache resourceCache = ResourceCache.getInstance();
    String driverClassString = resourceCache.getProperty("datasource.driver.class.name");
    String databaseURL = resourceCache.getProperty("datasource.url");
    String username = resourceCache.getProperty("test.datasource.username");
    String password = resourceCache.getProperty("test.datasource.password");

    Class driverClass = Class.forName(driverClassString);
    Connection jdbcConnection = DriverManager.getConnection(databaseURL, username, password);              
    connection = new DatabaseConnection(jdbcConnection);
    DatabaseConfig config = connection.getConfig();
    config.setProperty(DatabaseConfig.FEATURE_QUALIFIED_TABLE_NAMES, true);
    config.setProperty(DatabaseConfig.PROPERTY_DATATYPE_FACTORY, new Oracle10DataTypeFactory());         
        
    FlatXmlDataSetBuilder flatXmlDataSetBuilder = new FlatXmlDataSetBuilder();
    flatXmlDataSetBuilder.setColumnSensing(true);
    dataset = flatXmlDataSetBuilder.build(Thread.currentThread()
        .getContextClassLoader()
        .getResourceAsStream("Test-dataset.xml"));
 }

Using Spring's StoredProcedure class with Oracle Spatial JGeometry

The Spring framework provides a neat wrapper class you can extend when you want to call a stored procedure. Sometimes though you need to extend to use a native connection because, for example, you need to pass an ORACLE Geometry type to the stored procedure.

The below code wraps a Create_Geometry_Line stored procedure which takes a geometry object. The calling code will use the executeMap method to pass in a Map of in parameters. It converts the list of doubles (which represent the points of the line) to an object of type STRUCT.

package uk.co.city81.persistence.dao.geometry;

import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Types;
import java.util.Map;

import oracle.jdbc.OracleTypes;
import oracle.spatial.geometry.JGeometry;
import oracle.sql.STRUCT;

import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.SqlOutParameter;
import org.springframework.jdbc.core.SqlParameter;
import org.springframework.jdbc.object.StoredProcedure;


public class CreateGeometryLineDAO extends StoredProcedure {

    private Connection nativeConn = null;

    public static final String CREATE_GEOMETRY_LINE = "geometry_pkg.Create_Geometry_Line";

    // Constructor
    public CreateGeometryLineDAO(JdbcTemplate jdbcTemplate) {
        setJdbcTemplate(jdbcTemplate);
        setFunction(true);
        setSQL();
        declareParameter(new SqlOutParameter("l_Return", Types.INTEGER));
        declareInParameters();
        declareOutParameters();
        compile();
    }

    // Get the native connection
    protected Connection getNativeConn() throws SQLException {  
        if ((nativeConn == null) || nativeConn.isClosed()) {
            nativeConn = this.getJdbcTemplate().getNativeJdbcExtractor()
                .getNativeConnection(this.getJdbcTemplate()
                .getDataSource().getConnection());
        }
        return nativeConn;   
    }

    protected void declareInParameters() {
        declareParameter(new SqlParameter("p_geom_data",OracleTypes.STRUCT)); 
    }

    protected void declareOutParameters() {
        // declare out params
    }

    protected void setSQL() {
        setSql(CREATE_GEOMETRY_LINE);
    }

    /**
     * Execute the stored procedure
     * 
     * @param inParamMap a map of the stored procedure parameters
     */
    public Map executeMap(Map inParamMap) {
        Map outParamMap = null;
        try {  
            if (inParamMap.get("p_geom_data") != null) {
                java.util.List<Double> ordinates 
                    = (java.util.List<Double>) inParamMap.get("p_geom_data");
                double[] ordinateDoubles = new double[ordinates.size()];
                int count = 0;
                for (Double ordinate : ordinates) {
                    ordinateDoubles[count] = ordinate.doubleValue();
                    count++;
                }

                int[] elemInfo = new int[]{1,2,1};
                JGeometry j_geom = new JGeometry(
                    JGeometry.GTYPE_CURVE,8307, elemInfo,ordinateDoubles);        
                STRUCT obj = JGeometry.store(j_geom, getNativeConn());
                inParamMap.put("p_geom_data", obj);
            }

            outParamMap = super.execute(inParamMap);

        } catch (DataAccessException daex) {
            // throw exception
        } catch (SQLException e) {
            // throw exception
        }

        return outParamMap;
    }

}

The jdbcTemplate can be injected into the DAO's constructor. The extracts from the context xml are below:

    <bean id="datasource" class="org.springframework.jdbc.datasource.DriverManagerDataSource" 
        destroy-method="close">

        <property name="driverClassName">
            <value>oracle.jdbc.OracleDriver</value>
        </property>
        <property name="url">
            <value>jdbc:oracle:thin:@server:port:db</value>
        </property>
        <property name="username">
            <value>username</value>
        </property>
        <property name="password">
            <value>password</value>
        </property>
    </bean>

    <bean id="nativeJdbcExtractor" class="org.springframework.jdbc.support.nativejdbc.SimpleNativeJdbcExtractor" 
      lazy-init="true"/>

    <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
        <property name="dataSource">
            <ref bean="datasource" />
        </property>
        <property name="nativeJdbcExtractor">
            <ref bean="nativeJdbcExtractor" />
        </property>
    </bean>

    <bean id="createGeometryLineDAO" class="uk.co.city81.persistence.dao.geometry.CreateGeometryLineDAO">
        <constructor-arg ref="jdbcTemplate" />
    </bean>

JAAS, EJB Security and Glassfish

With EJBs you can specify security by using annotations from the javax.annotation.security package. The below article describes how to setup security on a bean and access the methods of the bean via an annotation or via JAAS which has been setup on Glassfish.

The below bean class only allows the MANAGER role access to use the services exposed. In this case, findCustomerByAccountNumber. (The AccessRoles.MANAGER resolves to a string so if the string changes the CustomerServiceBean doesn't have to change.)

@RolesAllowed(AccessRoles.MANAGER)
@Stateless(mappedName = JndiResourceName.CUSTOMER_SERVICE)
@Remote(CustomerService.class)
public class CustomerServiceBean implements CustomerService {

    @Override
    public Customer findCustomerByAccountNumber(String accountNumber) {
        Customer customer = null;
        // do stuff to find customer
        return customer;
    } 

}

To call the findCustomerByAccountNumber, the code can use the @RunAs annotation as described below:

    @RunAs(AccessRoles.MANAGER)
    public void verifyCustomer(String accountNumber) {
        // do stuff 
        Customer customer = 
            customerService.findCustomerByAccountNumber(String accountNumber);
        // do more stuff 
    }

But what if the roles calling the verify method can vary ie MANAGERS, OPERATORS, ADMIN. In this scenario, we would want to authenticate the 'caller' before accessing the findCustomerByAccountNumber method. A solution to this would be to use JAAS (Java Authentication and Authorization Service.)

The principal of this is to create realms and have users and groups in the realm. There are a few steps involved which are described as follows:

Firstly, the app server (in this example Glassfish) needs to create a realm and the below describes how to do this using the command line. It assumes connection pools, user and group database tables have been created and populated, and that the flexiblejdbcrealm-0.4.jar is in the Glassfish lib dir:

asadmin --host  delete-auth-realm customer-realm

asadmin --host  create-auth-realm --classname=org.wamblee.glassfish.auth.FlexibleJdbcRealm --property="jaas.context=customerJdbcRealm:datasource.jndi=
jdbc/Customer:sql.password=select password from customeruser where username\=?:sql.groups=select g.groupname from customergroup g inner join user_group ug on g.id\=ug.group_id inner join customeruser u on ug.user_id\=u.id where u.username\=?:password.digest=MD5:password.encoding=BASE64" customer-realm

asadmin --host  --user admin set server-config.security-service.activate-default-principal-to-role-mapping=true

asadmin --host  set-log-level javax.enterprise.system.core.security=INFO

asadmin --host  set-log-level org.wamblee.glassfish.auth=INFO

The login.conf needs to have the below added to it:

customerJdbcRealm {com.mypackage.auth.CustomerLoginModule required;}

The CustomerLoginModule class extends FlexibleJdbcLoginModule and gives us the ability to intercept the login/authentication calls if we so wish. In this case, any login exceptions are being logged:

public class CustomerLoginModule extends FlexibleJdbcLoginModule implements LoginModule {

    private static final Logger SECURITY_LOGGER = Logger.getLogger("com.mypackage");

    @Override
    protected void authenticate() throws javax.security.auth.login.LoginException {

        try {
            super.authenticate();
        } catch(LoginException le){
            SECURITY_LOGGER.error("Authentication failed for " 
                + _username + ". " + le.getMessage());
            throw le;
        }

    }

}

We can now change the verify method to authenticate before calling the 'secure'  findCustomerByAccountNumber method:

    private ProgrammaticLoginInterface programmaticLogin = 
        new ProgrammaticLogin();

    public void verifyCustomer(String accountNumber) {
        Customer customer = null;
        boolean loginSuccessful = programmaticLogin.login("manager", "password", "customer-realm", true);  

        if (loginSuccessful) { 
            customer = customerService.findCustomerByAccountNumber(String accountNumber);
        } else {
            // throw exception 
        }

    }

The call to the ProgrammaticLogin instance attempts to use the supplied name and password directly to login to the current realm. If successful, a security context is created for that user and is used by the EJB when checking what roles are allowed to call it.

For example purposes, the above verifyCustomer method has the name and password hard coded but in reality these values could be obtained from a login web page or other such authentication mechanisms.

Portable Global JNDI Names and Maven

With the advent of the EJB 3.1 specification, the JNDI names for session beans have become portable via the java:global namespace.

As described on the Glassfish EJB FAQ page (http://glassfish.java.net/javaee5/ejb/EJB_FAQ.html#What_is_the_syntax_for_portable_global_) the syntax for the global namespace is:

java:global[/<app-name>]/<module-name>/<bean-name>

where the app-name is the ear name (if the app is deployed as an ear), the module-name the ejb jar or war name and the bean-name the session bean name. Note that names are unqualified and minus the extension.

Given the syntax an example of using the portable JNDI name using the @EJB annotation would be:

@EJB(lookup=
    "java:global/customer-ear/customer-1-0-0/CustomerServiceBean")

The JNDI name may now be portable but the above code would not be reusable if the customer jar name changed. It is quite often the case that jars will have versions in their name eg customer-1-0-0.jar and you would not want to change your code everytime there was a new release of the customer jar so you need to change the module name in the lookup.

One option would be to rename the jar when building the ear file so the module name defaults to the jar name. To do this in Maven would be to explicitly add the ejbModule to the ear plugin as shown below. The name of the jar is what is specified in the bundleFileName tag.

    <plugin>
        <artifactId>maven-ear-plugin</artifactId>
        <version>2.4</version>
        <configuration>
            <generateApplicationXml>true</generateApplicationXml>
            <archive>
                <manifest>
                    <addClasspath>true</addClasspath>
                </manifest>
            </archive>
            <version>6</version>
            <defaultLibBundleDir>lib</defaultLibBundleDir>
            <earSourceDirectory>resources</earSourceDirectory>
            <modules>
                <ejbModule>
                    <groupId>com.mybank.customer</groupId>
                    <artifactId>customer-server</artifactId>
                    <bundleFileName>customer.jar</bundleFileName>
                </ejbModule>
            </modules>
        </configuration>
    </plugin>

The lookup would then be as below:

@EJB(lookup=
    "java:global/customer-ear/customer/CustomerServiceBean")

This means the code containing the injection point doesn't have to change with the names of the jars.

Java Collections and Threading

In the java.util.concurrent package, there exist several collection classes which aid the development of thread safe code. Below is a brief overview of the five most useful classes and a comparison of them with the some collection that exist in the java.util package.


ConcurrentHashMap<K,V>

java.util.HashMap is a collection which holds key value pairs but it is not synchronised. If multiple threads are accessing such a collection, the keys and values being put into the collection may not be visible to those threads calling get. A solution would be to create a synchronised version by calling the static method Collections.synchronizedMap() but this would 'lock' all operations on the Map.

Another solution would be to use ConcurrentHashMap<K,V>. The idea behind this class is that only the bucket that holds the data being accessed is 'locked'. This allows (more often than not) the get operation to work without blocking. Note that using an iterator obtained from the ConcurrentHashMap will not reflect map modifications since getting the iterator. Therefore the iterator will not throw ConcurrentModificationException.


ConcurrentSkipListMap<K,V>

java.util.concurrent.ConcurrentSkipListMap offers similar functionality to the above class but it maintains a sorted order (either the natural order or the order based on using a Comparator.) It also provides methods like ceilingKey, lowerEntry and tailMap. Because of the ordering, insertion can be slower than that of a ConcurrentHashMap but iterating over the collection would be faster.


ConcurrentSkipListSet<E>

Similar to the above but is a NavigableSet.


CopyOnWriteArrayList<E>

java.util.ArrayList is a collection which holds a resizable list of entries but it is not synchronised. As with ConcurrentHashMap, a solution would be to create a synchronised version by calling the static method Collections.synchronizedList() but this would 'lock' all operations on the List. Another solution would be to use CopyOnWriteArrayList<E> and this is best suited to lists where reads are frequent and writes are infrequent. It works by making a change to a copy of the list and then replacing the existing list with the modified copy..


CopyOnWriteArraySet<E>

Similar to the above but is a Set.

Thread-safe and lock free variables

The java.util.concurrent.atomic package contains classes which aim to ease concurrency concerns in multithreaded applications. They offer better performance than using synchronisation because its operations on fields do not require locking.

The classes AtomicBoolean, AtomicInteger, AtomicLong and AtomicReference enable the primitives boolean, int, long and an object reference to be atomically updated. They all provide utility methods like compareAndSet which takes an expected value and the update value and returns false if the expected value doesn't equal (==) the current value. The AtomicInteger and AtomicLong classes also provide atomic pre and post decrement/increment methods like getAndIncrement or incrementAndGet.

An example use of one of the 'scalar' atomic classes is shown below:

public class IdentifierGenerator {

    private AtomicLong seed = new AtomicLong(0);

    public long getNextIdentifier() {
        return seed.getAndIncrement();
    }

} 

The int and long primitives, and also object references can be held in arrays where the elements are atomically updated namely AtomicIntegerArray, AtomicLongArray and AtomicReferenceArray.

There also exists field updater classes namely AtomicIntegerFieldUpdater, AtomicLongFieldUpdater and AtomicReferenceFieldUpdater. With these classes you can still use the methods on the class but use the updater class to manage the field that requires atomic access via methods like compareAndSet. Note that because of this, atomcity isn't guaranteed.

These are abstract classes and the below (contrived!) example shows how an instance would be created:

    public void setBalance(Customer customer, int exisitngBalance, 
        int newBalance) {

        AtomicIntegerFieldUpdater<Customer> balanceAccessor 
            = AtomicIntegerFieldUpdater.newUpdater(
        Customer.class, "balance"); 
        balanceAccessor.compareAndSet(customer, exisitngBalance, newBalance);

    }

Weak and Soft References and the Garbage Collector

Java uses the Garbage Collector to manage memory. It does this by freeing up memory no longer needed by the application ie memory used by objects that are not referenced anymore, in other words, objects that are ‘unreachable’.

Typically, objects are strongly referenced so they remain in memory until no longer reachable,

Customer customer = new Customer();

but there are certain scenarios where you’d want to keep references to objects and also allow them to be garbage collected if needs be eg when memory is at a premium. A solution to this problem is to use Reference Objects.

There are three concrete implementations of the java.lang.ref.Reference<T> class: WeakReference, SoftReference and PhantomReference.

Objects held only by a WeakReference are deemed to be weakly reachable and are therefore eligible for garbage collection. An example of creating a WeakReference is below:

WeakReference<Customer> weakCustomer = new

    WeakReference<Customer>(customer);

To obtain the customer object held by the weak reference, you would call weakCustomer.get(). This will always return the customer object unless the garbage collecter has deemed the object to be weakly reachable and therefore has freed it from memory. In that case, calling the get method will return null;

If you want to hold a collection of the objects that can be garbage collected, then you can use a WeakHashMap. This collection uses weak references as keys, so when the key becomes garbage collected, the entry for that key in the WeakHashMap gets removed. This collection class isn’t synchronized so use Collections.synchronizedMap for a synchronized version, but be beware that whether synchronized or not, the size of the Map may decrease over time as the garbage collector removes entries, so an iterator may return ConcurrentModificationException.

A SoftReference holds onto the object more strongly than a WeakReference and if memory is not a problem the garbage collector will not free up the memory used by objects held by a soft reference.

You can also create another type of reference to an object, a PhantomReference. Calling the get method on such a reference will always return null though thereby preventing you from ‘resurrecting’ the object.  It’s only real use is when you pass a ReferenceQueue into the PhantomReference constructor.

You can pass a ReferenceQueue into the constructor of all the Reference types. It enables you to keep track of objects which have been garbage collected. When the object’s finalize method is called, the object will be placed on the reference queue. This process is called ‘enqueuing’. You then know when it was removed from memory.

In the case of a PhantomReference, passing in a null ReferenceQueue would be a pointless exercise, rendering the creation of the PhantomReference useless.