Maximum Zeal ~ Emphatic prose on indulged fascinations

Creation, dynamic loading and instrumentation with javaagents

Referrers: ReverseEngineeringStackExchange, StackOverflow.

Sometime back I had to delve into the depths of how one could dynamically and programmatically load a javaagent at runtime – in other words how a javaagent could be attached to a running process. Since then I’ve been meaning to blog my findings. Finally, after quite some time, I’ve got around to it but here I take the opportunity to expand the scope of the article to a complete treatment of how to create a javaagent, load it statically at startup or dynamically at runtime and also how to perform instrumentation of a simple class. As always the code will reside in an eclipse maven project that will be made available for download.

Introduction

Java agents are self contained components through which application classes pass at the level of byte code instructions in the form of byte arrays. They were introduced in java5 along with the powerful java.lang.instrument package. Java agents can be loaded statically at startup or dynamically (programmatically) at runtime to attach to a running process in a fail-safe fashion.

Lifecycle of a javaagent

The most important thing to understand is the lifecycle of a javaagent and its behaviour in relation to the application itself. The lifecycle hook points are as follows.

  • PreMain – The PreMain method is invoked first prior to running the main method. This is the hookpoint for loading the javaagent statically at startup.
  • Main – The main method is always invoked after the PreMain invocation.
  • AgentMain – The AgentMain method can be invoked at any time before or after the Main method. This hookpoint is for loading the javaagent dynamically at runtime and attaching to a running process.

Creation of a javaagent

In order to create a java agent you must first define your chosen hook points from the lifecycle above as below.

package name.dhruba.javaagent;

import java.lang.instrument.Instrumentation;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyJavaAgent {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgent.class);

    private static Instrumentation instrumentation;

    /**
     * JVM hook to statically load the javaagent at startup.
     * 
     * After the Java Virtual Machine (JVM) has initialized, the premain method
     * will be called. Then the real application main method will be called.
     * 
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void premain(String args, Instrumentation inst) throws Exception {
        logger.info("premain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * JVM hook to dynamically load javaagent at runtime.
     * 
     * The agent class may have an agentmain method for use when the agent is
     * started after VM startup.
     * 
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void agentmain(String args, Instrumentation inst) throws Exception {
        logger.info("agentmain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * Programmatic hook to dynamically load javaagent at runtime.
     */
    public static void initialize() {
        if (instrumentation == null) {
            MyJavaAgentLoader.loadAgent();
        }
    }

}

Once you’ve defined your hook points you must make the JVM aware by putting them in a manifest file.

Main-Class: name.dhruba.javaagent.MyMainClass
Agent-Class: name.dhruba.javaagent.MyJavaAgent
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Premain-Class: name.dhruba.javaagent.MyJavaAgent

In the example project I’m doing so using maven and the maven assembly plugin which also packages the project as a single all inclusive uber executable jar for ease of testing.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-assembly-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>attached</goal>
      </goals>
      <phase>package</phase>
      <configuration>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
        <archive>
          <manifest>
            <mainClass>name.dhruba.javaagent.MyMainClass</mainClass>
          </manifest>
          <manifestEntries>
            <Premain-Class>name.dhruba.javaagent.MyJavaAgent</Premain-Class>
            <Agent-Class>name.dhruba.javaagent.MyJavaAgent</Agent-Class>
            <Can-Redefine-Classes>true</Can-Redefine-Classes>
            <Can-Retransform-Classes>true</Can-Retransform-Classes>
          </manifestEntries>
        </archive>
      </configuration>
    </execution>
  </executions>
</plugin>

Instrumenting classes

Within the javaagent hook points one can define java.lang.instrument.ClassFileTransformer implementations that are invoked for all classes being loaded within an application. These receive classes as byte arrays and have the option of redefining them also in terms of byte arrays. Here I provide an example class file transformation.

The following User pojo returns ‘foo’ as its name.

package name.dhruba.user;

public class MyUser {

    public String getName() {
        return "foo";
    }

}

The following class file transformer (using ASM) redefines the User pojo to return ‘bar’ instead.

package name.dhruba.javaagent;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.security.ProtectionDomain;

import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.Label;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyClassFileTransformer implements ClassFileTransformer, Opcodes {

    static final Logger logger = LoggerFactory.getLogger(MyClassFileTransformer.class);

    @Override
    public byte[] transform(ClassLoader loader, String className, Class classBeingRedefined,
            ProtectionDomain protectionDomain, byte[] classfileBuffer)
            throws IllegalClassFormatException {
        logger.info("class file transformer invoked for className: {}", className);

        if (className.equals("name/dhruba/user/MyUser")) {

            ClassWriter cw = new ClassWriter(0);
            MethodVisitor mv;

            cw.visit(V1_6, ACC_PUBLIC + ACC_SUPER, "name/dhruba/user/MyUser", null,
                    "java/lang/Object", null);

            cw.visitSource(null, null);

            {
                mv = cw.visitMethod(ACC_PUBLIC, "", "()V", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(3, l0);
                mv.visitVarInsn(ALOAD, 0);
                mv.visitMethodInsn(INVOKESPECIAL, "java/lang/Object", "", "()V");
                mv.visitInsn(RETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            {
                mv = cw.visitMethod(ACC_PUBLIC, "getName", "()Ljava/lang/String;", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(6, l0);
                mv.visitLdcInsn("bar");
                mv.visitInsn(ARETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            cw.visitEnd();

            return cw.toByteArray();
        }

        return classfileBuffer;
    }

}

Note that this is not standard AOP or proxy logic. The class file transformer is literally redefining the bytecode instructions incrementally in the form of byte arrays.

Static loading of a javaagent at startup

Static loading of a javaagent is done by using the -javaagent=/path/to/file.jar=options command line option to the java executable as below.

$ java -javaagent:target/javaagent-examples-jar-with-dependencies.jar=foobarbaz name.dhruba.javaagent.MyMainClass foo bar baz
21:37:50.783 [main] INFO  name.dhruba.javaagent.MyJavaAgent - premain method invoked with args: foobarbaz and inst: sun.instrument.InstrumentationImpl@1786e64
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/javaagent/MyMainClass
21:37:50.789 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: [foo, bar, baz]
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
21:37:50.800 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Above, the javaagent lifecycle is clearly visible. The premain (and not the agentmain) method is invoked first. The class file transformer is passed classes as they are loaded. The transformer chooses to redefine the User object prior to its use. Subsequently the classes loaded at shutdown also pass through the transformer.

Dynamic loading of a javaagent at runtime

Dynamic loading of a javaagent at runtime can be done quite easily in a programmatic fashion but requires the sun tools jar to be present on the classpath. Certain libraries like jmockit have avoided this by opting to absorb the relevant classes from the sun tools jar into its library under the same package names. In Maven the tools jar can be added to the classpath very easily.

<dependency>
  <groupId>com.sun</groupId>
  <artifactId>tools</artifactId>
  <version>1.6.0</version>
  <scope>system</scope>
  <systemPath>${java.home}/../lib/tools.jar</systemPath>
</dependency>

The sun tools jar api can then be used to load the java agent simply by provided the path to the jar file as follows.

package name.dhruba.javaagent;

import java.lang.management.ManagementFactory;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.sun.tools.attach.VirtualMachine;

public class MyJavaAgentLoader {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgentLoader.class);

    private static final String jarFilePath = "/home/dhruba/.m2/repository/"
            + "javaagent-examples/javaagent-examples/1.0/"
            + "javaagent-examples-1.0-jar-with-dependencies.jar";

    public static void loadAgent() {
        logger.info("dynamically loading javaagent");
        String nameOfRunningVM = ManagementFactory.getRuntimeMXBean().getName();
        int p = nameOfRunningVM.indexOf('@');
        String pid = nameOfRunningVM.substring(0, p);

        try {
            VirtualMachine vm = VirtualMachine.attach(pid);
            vm.loadAgent(jarFilePath, "");
            vm.detach();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

}

As you can see – the code above is querying the pid of the running process and attaching the javaagent to the process. If any kind of failure occurs at this point it is ignored silently whereas if an agent is loaded on startup failures result in termination of startup.

We can now initialise the java agent prior to invoking our main method using the MyJavaAgent.initialize() hookpoint we declared earlier.

package name.dhruba.javaagent;

import java.util.Arrays;

import name.dhruba.user.MyUser;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyMainClass {

    static final Logger logger = LoggerFactory.getLogger(MyMainClass.class);

    static {
        MyJavaAgent.initialize();
    }

    /**
     * Main method.
     * 
     * @param args
     */
    public static void main(String[] args) {
        logger.info("main method invoked with args: {}", Arrays.asList(args));
        logger.info("userName: {}", new MyUser().getName());
    }

}

The output is very similar but with a subtely different path through the code.

20:58:50.923 [main] INFO  n.dhruba.javaagent.MyJavaAgentLoader - dynamically loading javaagent
20:58:51.249 [Attach Listener] INFO  name.dhruba.javaagent.MyJavaAgent - agentmain method invoked with args:  and inst: sun.instrument.InstrumentationImpl@c2ff5
20:58:51.266 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: []
20:58:51.267 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
20:58:51.276 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Note that this time the agentmain method was invoked instead of the premain method. The rest of the output is the same.

Conclusion

Javaagents along with the java.lang.instrument package are a powerful feature set of the Java language that allow you complete control over the classes of any given application. Instrumentation also has far more liberty with its capabilities than proxying or pointcut weaving. This gives complete dynamic capability to an otherwise very static language. It has allowed development of powerful non-intrusive tools like JMockit which immediately gains numerous advantages over other mocking tools largely because it is based on instrumentation. I look forward to further exploring the possibilities going forward.

Resources

ASM, ByteCodeOutline Plugin for Eclipse (Eclipse 3.5 update site), java.lang.instrument, JMockit.

Update [2/7/2011]: It’s nice to see this featured on StackOverflow.

Thanks to Anton Arhipov, Software Engineer at JRebel, for kindly linking to this post.

19 Responses to Creation, dynamic loading and instrumentation with javaagents

  1. Great end-to-end tutorial! I can’t wait to give this technology a spin.

    I imagine using this VirtualMachine.loadAgent()/TransformClassFile mechanism to wholesale replace an existing class implementation with another. For example, say i want to simply comment out or add a couple lines to a method in the class. It seems like this sort of “wholesale dynamic class patching” would be conceptually easier than bytecode manipulation via ASM or whatever.

    I envision a tool with the usage of:

    jpatch

    Are you aware of such a tool? Is there some fundamental roadblock I’m overlooking that would prevent such a tool from working as expected? (Granted I can imagine that weird things would happen if the bytecode change was too radical.)

    Also, when using loadAgent() to inject the agent into a running JVM, does the JVM automatically cycle the class file transfer through all already loaded classes? This doesn’t appear to be clear from what I’ve read of the documentation so far. Guess I’ll find out for myself when I give this stuff a try.

  2. Thanx for the help.. However i dont know how to load java agnet dynamically… I hav seen the result of static loading …and how the result of dynamic loading comes?is it by entering command similar to staticloading?

  3. I copied your project and it almost works :) What about this exception:

    Exception in thread “main” java.lang.NoSuchMethodError: user.MyUser: method ()V not found
    at javaagent.MyMainClass.main

    ?

  4. Thanks for publishing the article. I couldn’t get the example to work though.

    $ java -javaagent:/home/hdara/src/javaagent-examples/target/javaagent-examples-jar-with-dependencies.jar foo
    Exception in thread “main” java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:343)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:358)
    Caused by: java.lang.Error: Unresolved compilation problem:
    Logger cannot be resolved to a type

    at name.dhruba.javaagent.MyJavaAgent.premain(MyJavaAgent.java:25)
    … 6 more
    FATAL ERROR in native method: processing of -javaagent failed
    Aborted

    I checked that Logger is there in the jar, so why is it failing? Some other dependencies that Logger depends on are missing?

  5. I modified the code to use the java.util.logging classes and now it fails on Opscode class, which is again there in the same jar. I am using JDK1.6.0_21x64 on ubuntu.

  6. I guess using java.util.logging package causes java.util.StackOverflowError, which is probably why you used an alternative logging implementation. Still not sure why it doesn’t find the classes though.

  7. Hi DHRUBA,

    First of all, really nice article ! It has helped me out from some issues… but I have a question about it because I still have an issue with the sun class agent loaded (VirtualMachine).

    I use Spring AOP and I need to pass a JVM parameter (java agent) on start up so the agent can be loaded with JVM.

    In Eclipse, I put -javaagent:spring-instrument-3.1.0.RELEASE.jar as argument and agent is loaded so all ok. But when I build my application, the agent loaded class gives an error (an annoying one, cause the meta inf is correct…)

    error:
    Error occurred during initialization of VM
    agent library failed to init: instrument
    Error opening zip file or JAR manifest missing : spring-instrument-3.1.0.RELEASE.jar

    So I skip your part of making a custom java agent because I want to use the one of Spring…

    Meta-INF Spring:

    Manifest-Version: 1.0
    Ant-Version: Apache Ant 1.8.2
    Created-By: 1.6.0_29-b11-402-11M3527 (Apple Inc.)
    Export-Package: org.springframework.instrument;version=”3.1.0.RELEASE”
    Bundle-Vendor: SpringSource
    Bundle-Version: 3.1.0.RELEASE
    Tool: Bundlor 1.0.0.RELEASE
    Bundle-Name: Spring Instrument
    Bundle-ManifestVersion: 2
    Premain-Class: org.springframework.instrument.InstrumentationSavingAge
    nt
    Bundle-SymbolicName: org.springframework.instrument
    Bundle-Creator: cbeams
    Implementation-Title: org.springframework.instrument
    Implementation-Version: 3.1.0.RELEASE

    any idea what the problem could be ??

    Grtz,

    Sikke

  8. Hi Dhruba,

    I would like to know a bit more on Java agent. I want to attach a Java agent to a running VM instance not the one that will help in loading the Java agent. How can this be done, Let me be more clear how does these Java monitoring tool attach an agent to a running JVM instance. I tried with a process id of a Java process running on windows system but the agent does not perform any operation as it does if I do like mentioned in the tutorial Kindly suggest the reason

    Thanks

  9. Sanyam. Attaching an agent to a running process is exactly what this post talks about. It has everything you need and I can personally vouch that it works on all operating systems. I’m sorry but I have no idea why it doesn’t work for you. Thanks for stopping by.

    • Hi Dhruba
      The following statements find the pid of the process to which Java agent attaches

        String nameOfRunningVM = ManagementFactory.getRuntimeMXBean().getName();
              int p = nameOfRunningVM.indexOf('@');
              String pid = nameOfRunningVM.substring(0, p);  // taken from your tutorial
      

      These statements attach the Java agent to the VM obtained using pid.

       try {
                  VirtualMachine vm = VirtualMachine.attach(pid);
                  vm.loadAgent(jarFilePath, "");
                  vm.detach();
              } catch (Exception e) {
                  throw new RuntimeException(e);
              }
      

      This works fine, but here is the problem I am talking about,

      in place of pid if I give id by selecting from windows task manager it fails throwing some error and if successful the agent does not do any instrumentation to the classes of that VM instance.

    • Hi Dhruba,

      Suppose I want to attach my Java agent to a Java application running on my system ( say netbeans). In this case What I am doing is, from windows task manager I get the PID of netbeans. and when I execute my dynamic agent attach program in which PID is hardcoded (of netbeans already running) as obtained from the Task manager, nothing is attaching to netbeans application.
      PS. I am running this Dynamic agent loading program seperately i.e., independent of netbeans IDE running.

  10. Okay so you are running two processes instead of one. The post above relates only to one. You could probably find a way to call the netbeans main method from your process and then it would be in the same one. I’ve not tried two process attachment so I cannot comment on whether it works or why it fails sorry. Though I have a feeling that it has to be in the same process to work.

  11. Hi
    Nice article.
    After loading java agents dynamically, it stays in memory and Transformer runs with every class call.

    How can it be later on Unloaded , through code ofcourse, without restating the JVM?

    Thanks,
    Tal

Leave a Reply