Tag Archives: twitter

Creation, dynamic loading and instrumentation with javaagents

Referrers: ReverseEngineeringStackExchange, StackOverflow.

Sometime back I had to delve into the depths of how one could dynamically and programmatically load a javaagent at runtime – in other words how a javaagent could be attached to a running process. Since then I’ve been meaning to blog my findings. Finally, after quite some time, I’ve got around to it but here I take the opportunity to expand the scope of the article to a complete treatment of how to create a javaagent, load it statically at startup or dynamically at runtime and also how to perform instrumentation of a simple class. As always the code will reside in an eclipse maven project that will be made available for download.

Introduction

Java agents are self contained components through which application classes pass at the level of byte code instructions in the form of byte arrays. They were introduced in java5 along with the powerful java.lang.instrument package. Java agents can be loaded statically at startup or dynamically (programmatically) at runtime to attach to a running process in a fail-safe fashion.

Lifecycle of a javaagent

The most important thing to understand is the lifecycle of a javaagent and its behaviour in relation to the application itself. The lifecycle hook points are as follows.

  • PreMain – The PreMain method is invoked first prior to running the main method. This is the hookpoint for loading the javaagent statically at startup.
  • Main – The main method is always invoked after the PreMain invocation.
  • AgentMain – The AgentMain method can be invoked at any time before or after the Main method. This hookpoint is for loading the javaagent dynamically at runtime and attaching to a running process.

Creation of a javaagent

In order to create a java agent you must first define your chosen hook points from the lifecycle above as below.

package name.dhruba.javaagent;

import java.lang.instrument.Instrumentation;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyJavaAgent {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgent.class);

    private static Instrumentation instrumentation;

    /**
     * JVM hook to statically load the javaagent at startup.
     * 
     * After the Java Virtual Machine (JVM) has initialized, the premain method
     * will be called. Then the real application main method will be called.
     * 
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void premain(String args, Instrumentation inst) throws Exception {
        logger.info("premain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * JVM hook to dynamically load javaagent at runtime.
     * 
     * The agent class may have an agentmain method for use when the agent is
     * started after VM startup.
     * 
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void agentmain(String args, Instrumentation inst) throws Exception {
        logger.info("agentmain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * Programmatic hook to dynamically load javaagent at runtime.
     */
    public static void initialize() {
        if (instrumentation == null) {
            MyJavaAgentLoader.loadAgent();
        }
    }

}

Once you’ve defined your hook points you must make the JVM aware by putting them in a manifest file.

Main-Class: name.dhruba.javaagent.MyMainClass
Agent-Class: name.dhruba.javaagent.MyJavaAgent
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Premain-Class: name.dhruba.javaagent.MyJavaAgent

In the example project I’m doing so using maven and the maven assembly plugin which also packages the project as a single all inclusive uber executable jar for ease of testing.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-assembly-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>attached</goal>
      </goals>
      <phase>package</phase>
      <configuration>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
        <archive>
          <manifest>
            <mainClass>name.dhruba.javaagent.MyMainClass</mainClass>
          </manifest>
          <manifestEntries>
            <Premain-Class>name.dhruba.javaagent.MyJavaAgent</Premain-Class>
            <Agent-Class>name.dhruba.javaagent.MyJavaAgent</Agent-Class>
            <Can-Redefine-Classes>true</Can-Redefine-Classes>
            <Can-Retransform-Classes>true</Can-Retransform-Classes>
          </manifestEntries>
        </archive>
      </configuration>
    </execution>
  </executions>
</plugin>

Instrumenting classes

Within the javaagent hook points one can define java.lang.instrument.ClassFileTransformer implementations that are invoked for all classes being loaded within an application. These receive classes as byte arrays and have the option of redefining them also in terms of byte arrays. Here I provide an example class file transformation.

The following User pojo returns ‘foo’ as its name.

package name.dhruba.user;

public class MyUser {

    public String getName() {
        return "foo";
    }

}

The following class file transformer (using ASM) redefines the User pojo to return ‘bar’ instead.

package name.dhruba.javaagent;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.security.ProtectionDomain;

import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.Label;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyClassFileTransformer implements ClassFileTransformer, Opcodes {

    static final Logger logger = LoggerFactory.getLogger(MyClassFileTransformer.class);

    @Override
    public byte[] transform(ClassLoader loader, String className, Class classBeingRedefined,
            ProtectionDomain protectionDomain, byte[] classfileBuffer)
            throws IllegalClassFormatException {
        logger.info("class file transformer invoked for className: {}", className);

        if (className.equals("name/dhruba/user/MyUser")) {

            ClassWriter cw = new ClassWriter(0);
            MethodVisitor mv;

            cw.visit(V1_6, ACC_PUBLIC + ACC_SUPER, "name/dhruba/user/MyUser", null,
                    "java/lang/Object", null);

            cw.visitSource(null, null);

            {
                mv = cw.visitMethod(ACC_PUBLIC, "", "()V", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(3, l0);
                mv.visitVarInsn(ALOAD, 0);
                mv.visitMethodInsn(INVOKESPECIAL, "java/lang/Object", "", "()V");
                mv.visitInsn(RETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            {
                mv = cw.visitMethod(ACC_PUBLIC, "getName", "()Ljava/lang/String;", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(6, l0);
                mv.visitLdcInsn("bar");
                mv.visitInsn(ARETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            cw.visitEnd();

            return cw.toByteArray();
        }

        return classfileBuffer;
    }

}

Note that this is not standard AOP or proxy logic. The class file transformer is literally redefining the bytecode instructions incrementally in the form of byte arrays.

Static loading of a javaagent at startup

Static loading of a javaagent is done by using the -javaagent=/path/to/file.jar=options command line option to the java executable as below.

$ java -javaagent:target/javaagent-examples-jar-with-dependencies.jar=foobarbaz name.dhruba.javaagent.MyMainClass foo bar baz
21:37:50.783 [main] INFO  name.dhruba.javaagent.MyJavaAgent - premain method invoked with args: foobarbaz and inst: sun.instrument.InstrumentationImpl@1786e64
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/javaagent/MyMainClass
21:37:50.789 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: [foo, bar, baz]
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
21:37:50.800 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Above, the javaagent lifecycle is clearly visible. The premain (and not the agentmain) method is invoked first. The class file transformer is passed classes as they are loaded. The transformer chooses to redefine the User object prior to its use. Subsequently the classes loaded at shutdown also pass through the transformer.

Dynamic loading of a javaagent at runtime

Dynamic loading of a javaagent at runtime can be done quite easily in a programmatic fashion but requires the sun tools jar to be present on the classpath. Certain libraries like jmockit have avoided this by opting to absorb the relevant classes from the sun tools jar into its library under the same package names. In Maven the tools jar can be added to the classpath very easily.

<dependency>
  <groupId>com.sun</groupId>
  <artifactId>tools</artifactId>
  <version>1.6.0</version>
  <scope>system</scope>
  <systemPath>${java.home}/../lib/tools.jar</systemPath>
</dependency>

The sun tools jar api can then be used to load the java agent simply by provided the path to the jar file as follows.

package name.dhruba.javaagent;

import java.lang.management.ManagementFactory;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.sun.tools.attach.VirtualMachine;

public class MyJavaAgentLoader {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgentLoader.class);

    private static final String jarFilePath = "/home/dhruba/.m2/repository/"
            + "javaagent-examples/javaagent-examples/1.0/"
            + "javaagent-examples-1.0-jar-with-dependencies.jar";

    public static void loadAgent() {
        logger.info("dynamically loading javaagent");
        String nameOfRunningVM = ManagementFactory.getRuntimeMXBean().getName();
        int p = nameOfRunningVM.indexOf('@');
        String pid = nameOfRunningVM.substring(0, p);

        try {
            VirtualMachine vm = VirtualMachine.attach(pid);
            vm.loadAgent(jarFilePath, "");
            vm.detach();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

}

As you can see – the code above is querying the pid of the running process and attaching the javaagent to the process. If any kind of failure occurs at this point it is ignored silently whereas if an agent is loaded on startup failures result in termination of startup.

We can now initialise the java agent prior to invoking our main method using the MyJavaAgent.initialize() hookpoint we declared earlier.

package name.dhruba.javaagent;

import java.util.Arrays;

import name.dhruba.user.MyUser;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyMainClass {

    static final Logger logger = LoggerFactory.getLogger(MyMainClass.class);

    static {
        MyJavaAgent.initialize();
    }

    /**
     * Main method.
     * 
     * @param args
     */
    public static void main(String[] args) {
        logger.info("main method invoked with args: {}", Arrays.asList(args));
        logger.info("userName: {}", new MyUser().getName());
    }

}

The output is very similar but with a subtely different path through the code.

20:58:50.923 [main] INFO  n.dhruba.javaagent.MyJavaAgentLoader - dynamically loading javaagent
20:58:51.249 [Attach Listener] INFO  name.dhruba.javaagent.MyJavaAgent - agentmain method invoked with args:  and inst: sun.instrument.InstrumentationImpl@c2ff5
20:58:51.266 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: []
20:58:51.267 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
20:58:51.276 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Note that this time the agentmain method was invoked instead of the premain method. The rest of the output is the same.

Conclusion

Javaagents along with the java.lang.instrument package are a powerful feature set of the Java language that allow you complete control over the classes of any given application. Instrumentation also has far more liberty with its capabilities than proxying or pointcut weaving. This gives complete dynamic capability to an otherwise very static language. It has allowed development of powerful non-intrusive tools like JMockit which immediately gains numerous advantages over other mocking tools largely because it is based on instrumentation. I look forward to further exploring the possibilities going forward.

Resources

ASM, ByteCodeOutline Plugin for Eclipse (Eclipse 3.5 update site), java.lang.instrument, JMockit.

Update [2/7/2011]: It’s nice to see this featured on StackOverflow.

Thanks to Anton Arhipov, Software Engineer at JRebel, for kindly linking to this post.

O2 offers free Twitter SMS integration

Although I saw this at announcement time it’s taken me this long to blog it but anyway for anyone on O2 who hasn’t seen this already O2 are offering free SMS replies and direct messages on Twitter August onwards. My reaction: awesome. In addition to unlimited data and unlimited free access to wifi hot spots around the country now we can add integration with the hottest micro blogging provider to the list. I think this is particularly significant in that networks are beginning to put customers first and also realising just how powerful an enabling feature it can be to simply blur the lines between a hardware device and the internet. Thanks. Sources: (O2 Twitter, O2 Blog).

Fixing Java UnknownHostException on Ubuntu

On Ubuntu, on fresh installs, I frequently tend to get UnknownHostException(s) while running Java such as below.

INFO: I/O exception (java.net.UnknownHostException) caught when processing request: repository.springsource.com.s3.amazonaws.com

This normally happens for me when running 32-bit Java on 64-bit Ubuntu. The solution is simply to install lib32nss-mdns as follows and then retry.

sudo apt-get install lib32nss-mdns

Sources (ubuntu bug, ehow).

O2 forbids iPhone 3GS upgrade for existing customers

After the great elation of hearing about the release of the new iPhone 3GS by Apple at the WWDC 2009 now comes the heartbreaking and shocking news that UK residents simply cannot own one. In essence this means that if you were one of the keen purchasers (on a monthly contract) of the iPhone 3G early on when it was released then you must now pay the penalty of being ineligible for an upgrade. This has caused a public outroar on the web and enraged and frustrated many. Twitter is flooding almost in real time with messages of outrage and complaint, a twitter trend #o2fail has appeared and direct dialogue has been established with O2 through their very own twitter account. There is even an online petition (twitition) for the cause.  News websites (1,2) and blogs are also expressing their dismay and reporting on that of others. Despite this frenzy of activity O2’s policy remains and that feeling of lack of control and an inability to get around these restrictions can be difficult to channel and vent.

My situation is dire. In order to upgrade to the new iphone I must pay £555 in total: £280 for the remaining term of my existing contract (8×25) plus £275 to get the 32gb version of the new iphone on a £35/month tariff. This is equivalent to buying the iphone 3gs outright on pay as you go for £540. Either is a serious amount of money. Many factors have given rise to this incredibly difficult situation. Apple has released a new phone within a year of their previous release and that too with a brand new operating system and new hardware specifications. Secondly O2 has for some time had exclusive rights to sell the iphone in the UK which means that they have now acquired a captive audience that must continue their subscriptions simply because there are no alternative provider opportunities. Lastly, whether you wish to believe this or not, the truth is that the iphone is unlike any other and is therefore evoking a reaction unlike any other.

I understand where O2 are coming from – when people bought the iphone 3g last year, like myself, they agreed to heed the terms of their 18 month contract. At the time we were happy to have the new iPhone that utilised the fastest available data network in the UK and also it certainly didn’t seem likely that Apple would re-release as soon as they have done especially given that data networks were not expected to leap in speed anytime soon. O2 also set their own trend by allowing upgrades from iPhone Edge to iPhone 3G. However things change and when Apple is involved you have to expect the unexpected.

Regardless of what the contracts say there is one simple yet crucial fact that O2 have failed to realise – for innumerable reasons the iPhone has become an exception to all rules and for that reason must be treated as one. The iPhone is unlike any other phone and has an appeal unlike that of any other phone. At this point it is no longer a phone – it is whatever you want it to be at all times. It has become an ingrained part of peoples’ lifestyles and the two are virtually inseparable and indistinguishable. It would be redundant to go through why it has become such an exclusive item and an object or desire as that should be more than obvious by now but simply to say that it has and it will only become more so. Given that – anyone who stops to consider the dynamics of the iphone’s appeal to the mass public whether a lay person or a business entity should realise that you simply cannot deprive the public of a new iPhone release and get away with it unscarred. O2 is facing the rebellion assault and will have made many enemies over time – even more so than previously.

The lack of competition i.e .a monopoly for such reasons is a very bad thing. There are rumors that the exclusivity contract between O2 and Apple is coming to an end in the last quarter of this year. Let’s hope that’s the case. The shifts in mobile phone contracts is worrying to say the least – initially we moved from 12 month to 18 month and now O2 is beginning to offer 24 month contracts. Expecting consumers to keep the same phone for that amount of time is hardly realistic. There will be new releases and consumers will want a change after some time. The feeling of lock in is not a pleasant one – it can be incredibly frustrating. On top of that O2 is beginning to charge around £15/month for iphone network tethering despite the fact that the phone has unlimited data. The average consumer needs the internet but is the average consumer able to afford such exorbitant monthly fees?

So now what? O2 customers will either fork out £500-£600 to upgrade from their existing contracts and grow bitter over time in the company of their new iPhone about the amount of money they’ve spent or they’ll forego the purchase and grow bitter over time about having been made to watch the iphone 3gs revolution from the sidelines. Either way the outcome for O2 is not good in terms of public opinion and loyalty.

Apple WWDC 2009 over

The Apple WWDC 2009 is now over. I had been following it this evening on twitter in real time courtesy of macrumors live. Below is a summary extract from the macrumors live twitter account. Points of note that are relevant to me – I’ll be upgrading to Snow Leopard for $29 in September and I’ll also be getting the 32GB iPhone 3GS June 19th onwards or at the soonest possible opportunity (assuming they allow premature upgrades). If I am not able to get my hands on the new iPhone 3GS I’ll be upgrading to the new v3.0 OS on June 17th for free. As much as I’d like the new 15″ macbook pro it’ll be a while before I can afford it. Apple have now updated their website. Overall great news.

  • iPhone 3GS available June 19th in US, UK, and 6 others. More come a week later, others this summer. 8 GB iPhone for $99 available today.42 minutes ago from web
  • iPhone 3GS: $199/16 GB and $299/32 GB for new and qualifying AT&T customers. Available in black and white. Current 8GB avail for $99.about 1 hour ago from web
  • iPhone 3GS: VoiceOver accessibility, Nike+ support, data encryption, improved battery life (up to 50% on Wi-Fi), eco-friendlyabout 1 hour ago from web
  • iPhone 3GS: API allows devs to integrate video capture; Voice control for call dialing, iPod playback; Digital compass integrates with Maps.about 1 hour ago from web
  • iPhone 3GS camera has still and video modes. Video mode: 30 fps VGA with audio, auto-focus, auto-exposure. Can trim video right on iPhone.about 1 hour ago from web
  • iPhone 3GS supports 7.2Mbps HSPDA data, has 3 MP auto-focus camera with “tap to focus” and improved low-light sensitivityabout 1 hour ago from web
  • iPhone 3GS announced (“S” for “speed”). 2-3x faster at tasks. Looks essentially identical to iPhone 3G. No front-facing camera.about 1 hour ago from web
  • iPhone OS 3.0 available June 17th. Free to all iPhone users, $9.95 for iPod touch users.about 1 hour ago from web
  • More Zipcar: iPhone app can honk vehicle’s horn and unlock car; Line6/Planet Waves: add guitar effects from iPhoneabout 1 hour ago from web
  • Ngmoco: Star Defense; Pasco: hardware accessories for scientific experiments for children; Zipcar: car-sharing reservationsabout 1 hour ago from web
  • TomTom: turn-by-turn GPS directions (apps and hardware). Car kit will hold iPhone on suction mount and integrate charging.about 1 hour ago from web
  • Dev Demos: Gameloft (Asphalt 5 racer), AirStrip Tech (Critical Care medical data app), ScrollMotion (digital books via in-app purchasing)about 1 hour ago from web
  • Turn-by-turn directions apps supported. Push notifications: alerts, sounds, badges. Now developer demos.about 2 hours ago from web
  • Reviewing OS 3.0 APIs: in-app purchasing, peer-to-peer connectivity, hardware accessory support, Google Maps available for use by devsabout 2 hours ago from web
  • “Find My iPhone”: Allows MobileMe customers to locate lost iPhones, send alerts, and remote wipe.about 2 hours ago from web
  • Safari for iPhone: HTTP audio/video streaming support, autofill, JavaScript 3x faster than OS 2.2.1., HTML 5 supportabout 2 hours ago from web
  • Tethering via USB or Bluetooth. Supported on 22 carriers, but not on AT&T at launch.about 2 hours ago from web
  • Cut, copy, paste, undo. Search. MMS in 3.0, but AT&T not supporting until later this summer. iTunes movie rentals/purchases from iPhone.about 2 hours ago from web
  • Video of developers and customers talking about the revolutionary platform and apps. Now moving on to talk about iPhone OS 3.0.about 2 hours ago from web
  • Now shifting to iPhone: 1,000,000 SDK downloads, 50,000 apps, 40 million iPhones/iPod touches, 1 billion+ appsabout 2 hours ago from web
  • Snow Leopard available in September for all Intel Macs. $29 for Leopard users, $49 family pack.about 2 hours ago from web
  • More Snow Leopard: Grand Central Station/OpenCL harness the processing power of GPUs, Demoing Exchange support for Mail, iCal, Address Bookabout 2 hours ago from web
  • Safari 4 final version released today on all platforms. Also showing demos of new QuickTime (brand-new interface), Stacks and Expose in SLabout 2 hours ago from web
  • Previewing Snow Leopard: Streamlined installation, Expose built into Dock icons, Preview and Mail 2x faster.about 2 hours ago from web
  • MacBook Air update: now starting at $1499, $1799 with 256 GB SSDabout 2 hours ago from web
  • 13″ MacBook now called MacBook Pro: starts at $1199. Same RAM and HD options as 15″ MBP, backlit KB, FW800. All models available today.about 3 hours ago from web
  • 17″ MBP updated: starting at $2499, retains ExpressCard slot, CPU and HD spec bumpsabout 3 hours ago from web
  • New 15″ MacBook Pro: non-removable 7 hr battery, improved display, SD card slot, up to 3.06Ghz, 8 GB RAM, and 500 GB HD/256GB SSD – $1699about 3 hours ago from web