Category Archives: lab49

Java concurrency bug patterns

Rather randomly and on a legacy note – here’s a series of links on the subject of java concurrency bug patterns from both IBM DeveloperWorks and from Alex Miller at Pure Danger Tech both for my reference and yours. Most readers, I expect will know all of this, but if you don’t the list is worth reading. It’s also good to raise awareness of such concurrency bug patterns in general as a surprisingly large number of interview candidates fail to answer questions correctly in this area.

In general, although most people tend to consider this optional or are blissfully unaware of it, documenting the thread safety of your code is very good practice and essential if you consider that FindBugs detects them and validates your assertions.

Tech arms race in the Tron landscape

On the train, back to the real world, this evening for another year at work I came across a fascinating article in the New York Times titled ‘Electronic Trading Creates A New Trading Landscape – The New Speed Of Money Reshaping Markets‘. For the duration of that journey I was wholly engrossed in the article and the radial thought processes it triggered effortlessly and constantly on technology and finance. It was an inspiring read and one that made me glad and relieved that I happened to work in this industry.

Predominantly it talked about how, over time, smaller exchanges (such as Direct Edge) had reclaimed the overwhelming dominance and market share of the historic exchange duopoly of NASDAQ-NYSE and how, during that process, New Jersey had been transformed to become ‘the heart of Wall St’ through the placements of data centres within it that now host and operate some of the largest stock exchanges in the US. The charming reference to a ‘Tron landscape’ was made in the article based on the likeness of the blue phosphorescent lighting used in the datacentres for illumination to that in the film.

More interesting to me, however, was the story of how this progression had been driven from its core by the breakneck speed and sheer extent of technological automation, advancement and innovation leaving traders, regulators and the market struggling to keep up in its trail. So where are we now? Exchanges are distributed, leaner and more competitive. Through colocation, software advancement, closer proximity to targets and with new fibre optic pathways constantly being laid between critical geographic exchange data routes trading is faster. Through high frequency trading, dark pools and strategic algorithms – trading is more intelligent allowing arbitrage and price exploitation through micro trading under stealth.

What have been the consequences of such advancements over time however? The use of HFT to place a very large bulk order in small increments was found to be the root cause of a market crash last May when this particular HFT algorithm continued placing trades as part of a large order despite prices sinking part way through. As a result the SEC and the exchanges introduced a halt to trading on individual stocks if the price fell more than ten percent in a five minute period. Dark pools have been in the spotlight for being opaque and exempt from public scrutiny. And there is talk of regulation not only of data centres and colocation but of perhaps technology’s greatest achievement of all – speed. The unattended and perahps ill-considered advancement of technology for mostly selfish motives has resulted in a disproportionate loss of control, transparency and ethical considerations away from human discretion and towards machine code. Can technology continue to dominate this industry progression at its core to its advantage or will it become the very victim of its own success? I wonder where we go from here. What do you think?

Java still #1

OracleTechNet have just posted: “New Tiobe Index is out – Java still #1, C# on the march”. Cool. It’s obvious that the first five are major players but the one that makes me curious is Python (Tiobe). I wonder what areas it’s gaining its popularity and adoption in. Objective C also appears to be making major strides but the single reason for that is obvious and this will be nothing but a steep upward trend.

VirtualBox: Sharing folders between host and guest

Recently, having finally refused to surrender to windows, I installed Ubuntu virtualised as a guest on Windows as a host using Oracle’s recently released VirtualBox. Here’s a tip on how to share folders between guest and host in the official way.

On the guest VM virtualbox menu open ‘Shared folders’.

Open Shared Folders

Open Shared Folders

On the top right of the dialogue box that comes up click the ‘+’ icon. Fill in the dialogue by adding a name and location.

Add share

Add share

After that you should have a shares dialogue as below.

Shares Dialogue

Shares Dialogue

Next, as root, mount manually.

mkdir /mnt/share
mount -t vboxsf virtual-box-ubuntu-share /mnt/share/

And, finally, add the following entry into /etc/fstab for future boots.

virtual-box-ubuntu-share /mnt/share vboxsf defaults 0 0

Done. Ubuntu on VirtualBox running as guest on a Windows host is by far the best and most compelling complement to your development environment if you are forced into using Windows as a host. VirtualBox even supports seamless mode which means that you can have Linux and Windows windows intermingled on the windows desktop. Superb. And best of all – both VirtualBox and Ubuntu being completely free.

Update: Great news. VirtualBox 4.0 is out. Here’s what’s new.

Augmented reality real time text translation

I normally don’t post about iphone applications but this turned out to be revolutionary – a symphony amidst the noise. Point your phone camera at a picture and it not only translates the text in the view of the camera but superimposes it onto the camera image itself transforming in real time what you see. And all without an internet connection or network costs. Simply phenomenal. Just when you think the community has done all it can with the iOS SDK once in a while something like this happens. Check it out. Quoted from MacRumors:

Word Lens Offers Real Time Language Translation. QuestVisual has released an eye-catching iOS app called Word Lens. Word Lens is an augmented reality app for the iPhone and iPod Touch (with video camera) which offers real time translation of text. You simply point your device’s video camera at a sign and the program translates and superimposes the translated text onto the video in real time. The demo video shows it in action.

Adobe launches SDP demonstration prototype

I’m not sure if this old news but I’ve only just come across it. Adobe has released a demonstration small scale SDP-like prototype both in Flash and AIR that they call a real time trader desktop. It has many faces and rotates like a cube which is quite a nice effect and makes a nice change from flat panes and tabs which are the default paradigm. Perhaps, as a result of the flurry of SDPs having been worked upon and released of late in the financial industry using Flash and AIR, Adobe feel they no longer need a middleman to invest in this area of consulting work and have done their own albeit as a demonstration. Although I think there’s more to consulting in this area particularly on the server side – I quite like the way they’ve published and marketed their prototype as an open product on the web with great attention to visual detail. Food for thought. Via.

java.lang.IllegalAccessException upon reflective class instantiation

This evening I got the following exception when reflectively instantiating a class.

Caused by: java.lang.IllegalAccessException: Class foo.Baz can not access a member of class foo.Bar with modifiers ""
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65) [na:1.6.0_16]
	at java.lang.Class.newInstance0(Class.java:349) [na:1.6.0_16]
	at java.lang.Class.newInstance(Class.java:308) [na:1.6.0_16]
	at foo.bar(Baz.java:126) [build/:na]
	... 90 common frames omitted

If you get this all you need to do is to make sure that the constructor of the class you are trying to instantiate is visible to the class that is doing the reflective instantiation. The empty modifier string in the exception message is confusing at first but simply means that your constructor is package private and therefore has no access modifier. If your constructor has an access modifier it will show within that string.

Credits: Troubleshooting the reflective API.

EhCache and Hibernate talk by Greg Luck

There is a free event upcoming in London on Tuesday 29th June 2010 titled ‘In the brain of Greg Luck, the new EhCache 2.0.0 and Hibernate caching SPI provider – boost hibernate performance by 10x’. Although I’m not a big fan of Hibernate – EhCache is a very prevalent open source caching library and given that this talk is by the creator of EhCache it certainly seems worthwhile attending. What has made EhCache more interesting of late is that it has been taken over by Terracotta – the network attached memory vendor. On an unrelated note there also seems to be a new meetup group for Google App Engine.

Closures in JDK7 stagnate

Lately, on the lambda-dev mailing list, dedicated to discussion on closures in jdk7, there’s been an eerie silence. With the feature complete deadline looming in two weeks members of the list have expressed concern at the lack of activity and questioned Oracle’s commitment to the cause. Alex Buckley from Sun has not been forthcoming in his responses which has only served to heighten their concern. Here are a few representative emails the last of which was sent earlier today.

Judging by the lack of activity, the secrecy on the official status and progress of the project, the feature complete deadline looming and the apparent failure to reach consensus on the list despite frenetic discussion on the topic it certainly seems to me that as time goes on it is becoming less and less likely that jdk7 will receive closures in any form whatsoever. If they do not make it into the next version of java it will indeed be a shame as closures were the feature the community was looking forward to the most.

My personal opinion is that in the light of the recent takeover of Sun by Oracle and with the release of JDK7 eagerly awaited by the community (for almost three and a half years I might add) Oracle should, at the very least, offer a degree of transparency on what they are doing and what their plans are for JDK7. As Neal Gafter points out in his most recent post to the list whatever the outcome silence sends the wrong message and I couldn’t agree more.

Update: David Flannagan has published a similar post.

Update: It seems the discussion has once again kicked off with a new proposal by Brian Goetz. However, as Stephen Coulbourne points out, the thought process that Oracle went through to reject other options and come up with this new proposal has not been transparent.

Update: There seems to have been a comment posted on the above linked blog by a member of the Oracle staff saying “There are several resources that are working on closures at Sun/Oracle. We are reviewing schedules for Java 7. As soon as we get done, we will publish those milestones as part of OpenJDK. Java is alive an kicking and so are closures in Java.”. This, if true, is great news.

Update: For those of you coming through from dzone this post is a little out of date. Discussions on closures are in full swing now on the lambda-dev mailing list so there’s less reason for concern.

Creation, dynamic loading and instrumentation with javaagents

Referrers: ReverseEngineeringStackExchange, StackOverflow.

Sometime back I had to delve into the depths of how one could dynamically and programmatically load a javaagent at runtime – in other words how a javaagent could be attached to a running process. Since then I’ve been meaning to blog my findings. Finally, after quite some time, I’ve got around to it but here I take the opportunity to expand the scope of the article to a complete treatment of how to create a javaagent, load it statically at startup or dynamically at runtime and also how to perform instrumentation of a simple class. As always the code will reside in an eclipse maven project that will be made available for download.

Introduction

Java agents are self contained components through which application classes pass at the level of byte code instructions in the form of byte arrays. They were introduced in java5 along with the powerful java.lang.instrument package. Java agents can be loaded statically at startup or dynamically (programmatically) at runtime to attach to a running process in a fail-safe fashion.

Lifecycle of a javaagent

The most important thing to understand is the lifecycle of a javaagent and its behaviour in relation to the application itself. The lifecycle hook points are as follows.

  • PreMain – The PreMain method is invoked first prior to running the main method. This is the hookpoint for loading the javaagent statically at startup.
  • Main – The main method is always invoked after the PreMain invocation.
  • AgentMain – The AgentMain method can be invoked at any time before or after the Main method. This hookpoint is for loading the javaagent dynamically at runtime and attaching to a running process.

Creation of a javaagent

In order to create a java agent you must first define your chosen hook points from the lifecycle above as below.

package name.dhruba.javaagent;

import java.lang.instrument.Instrumentation;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyJavaAgent {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgent.class);

    private static Instrumentation instrumentation;

    /**
     * JVM hook to statically load the javaagent at startup.
     * 
     * After the Java Virtual Machine (JVM) has initialized, the premain method
     * will be called. Then the real application main method will be called.
     * 
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void premain(String args, Instrumentation inst) throws Exception {
        logger.info("premain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * JVM hook to dynamically load javaagent at runtime.
     * 
     * The agent class may have an agentmain method for use when the agent is
     * started after VM startup.
     * 
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void agentmain(String args, Instrumentation inst) throws Exception {
        logger.info("agentmain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * Programmatic hook to dynamically load javaagent at runtime.
     */
    public static void initialize() {
        if (instrumentation == null) {
            MyJavaAgentLoader.loadAgent();
        }
    }

}

Once you’ve defined your hook points you must make the JVM aware by putting them in a manifest file.

Main-Class: name.dhruba.javaagent.MyMainClass
Agent-Class: name.dhruba.javaagent.MyJavaAgent
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Premain-Class: name.dhruba.javaagent.MyJavaAgent

In the example project I’m doing so using maven and the maven assembly plugin which also packages the project as a single all inclusive uber executable jar for ease of testing.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-assembly-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>attached</goal>
      </goals>
      <phase>package</phase>
      <configuration>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
        <archive>
          <manifest>
            <mainClass>name.dhruba.javaagent.MyMainClass</mainClass>
          </manifest>
          <manifestEntries>
            <Premain-Class>name.dhruba.javaagent.MyJavaAgent</Premain-Class>
            <Agent-Class>name.dhruba.javaagent.MyJavaAgent</Agent-Class>
            <Can-Redefine-Classes>true</Can-Redefine-Classes>
            <Can-Retransform-Classes>true</Can-Retransform-Classes>
          </manifestEntries>
        </archive>
      </configuration>
    </execution>
  </executions>
</plugin>

Instrumenting classes

Within the javaagent hook points one can define java.lang.instrument.ClassFileTransformer implementations that are invoked for all classes being loaded within an application. These receive classes as byte arrays and have the option of redefining them also in terms of byte arrays. Here I provide an example class file transformation.

The following User pojo returns ‘foo’ as its name.

package name.dhruba.user;

public class MyUser {

    public String getName() {
        return "foo";
    }

}

The following class file transformer (using ASM) redefines the User pojo to return ‘bar’ instead.

package name.dhruba.javaagent;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.security.ProtectionDomain;

import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.Label;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyClassFileTransformer implements ClassFileTransformer, Opcodes {

    static final Logger logger = LoggerFactory.getLogger(MyClassFileTransformer.class);

    @Override
    public byte[] transform(ClassLoader loader, String className, Class classBeingRedefined,
            ProtectionDomain protectionDomain, byte[] classfileBuffer)
            throws IllegalClassFormatException {
        logger.info("class file transformer invoked for className: {}", className);

        if (className.equals("name/dhruba/user/MyUser")) {

            ClassWriter cw = new ClassWriter(0);
            MethodVisitor mv;

            cw.visit(V1_6, ACC_PUBLIC + ACC_SUPER, "name/dhruba/user/MyUser", null,
                    "java/lang/Object", null);

            cw.visitSource(null, null);

            {
                mv = cw.visitMethod(ACC_PUBLIC, "", "()V", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(3, l0);
                mv.visitVarInsn(ALOAD, 0);
                mv.visitMethodInsn(INVOKESPECIAL, "java/lang/Object", "", "()V");
                mv.visitInsn(RETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            {
                mv = cw.visitMethod(ACC_PUBLIC, "getName", "()Ljava/lang/String;", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(6, l0);
                mv.visitLdcInsn("bar");
                mv.visitInsn(ARETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            cw.visitEnd();

            return cw.toByteArray();
        }

        return classfileBuffer;
    }

}

Note that this is not standard AOP or proxy logic. The class file transformer is literally redefining the bytecode instructions incrementally in the form of byte arrays.

Static loading of a javaagent at startup

Static loading of a javaagent is done by using the -javaagent=/path/to/file.jar=options command line option to the java executable as below.

$ java -javaagent:target/javaagent-examples-jar-with-dependencies.jar=foobarbaz name.dhruba.javaagent.MyMainClass foo bar baz
21:37:50.783 [main] INFO  name.dhruba.javaagent.MyJavaAgent - premain method invoked with args: foobarbaz and inst: sun.instrument.InstrumentationImpl@1786e64
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/javaagent/MyMainClass
21:37:50.789 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: [foo, bar, baz]
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
21:37:50.800 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Above, the javaagent lifecycle is clearly visible. The premain (and not the agentmain) method is invoked first. The class file transformer is passed classes as they are loaded. The transformer chooses to redefine the User object prior to its use. Subsequently the classes loaded at shutdown also pass through the transformer.

Dynamic loading of a javaagent at runtime

Dynamic loading of a javaagent at runtime can be done quite easily in a programmatic fashion but requires the sun tools jar to be present on the classpath. Certain libraries like jmockit have avoided this by opting to absorb the relevant classes from the sun tools jar into its library under the same package names. In Maven the tools jar can be added to the classpath very easily.

<dependency>
  <groupId>com.sun</groupId>
  <artifactId>tools</artifactId>
  <version>1.6.0</version>
  <scope>system</scope>
  <systemPath>${java.home}/../lib/tools.jar</systemPath>
</dependency>

The sun tools jar api can then be used to load the java agent simply by provided the path to the jar file as follows.

package name.dhruba.javaagent;

import java.lang.management.ManagementFactory;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.sun.tools.attach.VirtualMachine;

public class MyJavaAgentLoader {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgentLoader.class);

    private static final String jarFilePath = "/home/dhruba/.m2/repository/"
            + "javaagent-examples/javaagent-examples/1.0/"
            + "javaagent-examples-1.0-jar-with-dependencies.jar";

    public static void loadAgent() {
        logger.info("dynamically loading javaagent");
        String nameOfRunningVM = ManagementFactory.getRuntimeMXBean().getName();
        int p = nameOfRunningVM.indexOf('@');
        String pid = nameOfRunningVM.substring(0, p);

        try {
            VirtualMachine vm = VirtualMachine.attach(pid);
            vm.loadAgent(jarFilePath, "");
            vm.detach();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

}

As you can see – the code above is querying the pid of the running process and attaching the javaagent to the process. If any kind of failure occurs at this point it is ignored silently whereas if an agent is loaded on startup failures result in termination of startup.

We can now initialise the java agent prior to invoking our main method using the MyJavaAgent.initialize() hookpoint we declared earlier.

package name.dhruba.javaagent;

import java.util.Arrays;

import name.dhruba.user.MyUser;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyMainClass {

    static final Logger logger = LoggerFactory.getLogger(MyMainClass.class);

    static {
        MyJavaAgent.initialize();
    }

    /**
     * Main method.
     * 
     * @param args
     */
    public static void main(String[] args) {
        logger.info("main method invoked with args: {}", Arrays.asList(args));
        logger.info("userName: {}", new MyUser().getName());
    }

}

The output is very similar but with a subtely different path through the code.

20:58:50.923 [main] INFO  n.dhruba.javaagent.MyJavaAgentLoader - dynamically loading javaagent
20:58:51.249 [Attach Listener] INFO  name.dhruba.javaagent.MyJavaAgent - agentmain method invoked with args:  and inst: sun.instrument.InstrumentationImpl@c2ff5
20:58:51.266 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: []
20:58:51.267 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
20:58:51.276 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Note that this time the agentmain method was invoked instead of the premain method. The rest of the output is the same.

Conclusion

Javaagents along with the java.lang.instrument package are a powerful feature set of the Java language that allow you complete control over the classes of any given application. Instrumentation also has far more liberty with its capabilities than proxying or pointcut weaving. This gives complete dynamic capability to an otherwise very static language. It has allowed development of powerful non-intrusive tools like JMockit which immediately gains numerous advantages over other mocking tools largely because it is based on instrumentation. I look forward to further exploring the possibilities going forward.

Resources

ASM, ByteCodeOutline Plugin for Eclipse (Eclipse 3.5 update site), java.lang.instrument, JMockit.

Update [2/7/2011]: It’s nice to see this featured on StackOverflow.

Thanks to Anton Arhipov, Software Engineer at JRebel, for kindly linking to this post.