DEXing Deelang – First steps in SVN

If you’re one of the few who’ve already taken a look at Deelang, you’ll know that it’s a simple scripting language that compiles to a custom bytecode format, which then runs in a simple virtual machine. It can run anywhere, but is targeted mostly for embedded devices, and especially for Android.

All this is fine, and for the most part it doesn’t perform too badly. Unfortunately though it does mean that it’s most common operation (calling out to Java methods) is relatively slow, because under the hood reflection is used everywhere. This has the potential to be a problem since in Dee, everything is a method – you can’t define your own classes, methods or functions in scripts, but instead rely on a (Java-side) API provided by whoever embedded the language in their application. Even arithmetic and conditionals are done with methods. Reflection (especially on Android) is just too slow for this kind of usage.

To address this, I’ve been thinking for some time about a implementing a native (i.e. Dex) compiler for Deelang on Android, and over the past few days I’ve finally made a start. The architecture is pretty well mapped out in a first-cut kind of way (well, in my mind at least) but the implementation is only just taking it’s first baby steps. Hardly anything is supported right now – in fact, the only thing it’s possible to actually compile is literals and simple, direct function calls (what the Dee VM calls SELFCALLs). But it’s a start, and it’s in Subversion on Googlecode now if you feel like taking a look. All the work is being done in a branch, which you can browse or check-out at:

http://deelang.googlecode.com/svn/branches/DEXCOMPILER/deelang

What can it do?

As mentioned above, it’s very limited at the moment. In fact, about the most exciting thing it can do is take code such as:

foo(1,2)

and compile it to native Dex bytecode like:

//class:0000  access:0x0001
public class DexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f extends com.roscopeco.deelang.runtime.CompiledScript

//method:0000  access:0x0001
//LDexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f;.run(Ldee/lang/DeelangObject;Lcom/roscopeco/deelang/runtime/Binding;)V
public V run(dee.lang.DeelangObject,com.roscopeco.deelang.runtime.Binding)
                this:v3   //DexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f
                    :v4   //dee.lang.DeelangObject
                    :v5   //com.roscopeco.deelang.runtime.Binding
CONST               |     |v1=0x00000001  // int:1   float:0.000000
NEW_INSTANCE        |     |v0=NEW Ldee/lang/DeelangInteger;
INVOKE_DIRECT       |     |v0.(v5,v1)  //Ldee/lang/DeelangInteger;.(Lcom/roscopeco/deelang/runtime/Binding;I)V
CONST               |     |v1=0x00000002  // int:2   float:0.000000
NEW_INSTANCE        |     |v2=NEW Ldee/lang/DeelangInteger;
INVOKE_DIRECT       |     |v2.(v5,v1)  //Ldee/lang/DeelangInteger;.(Lcom/roscopeco/deelang/runtime/Binding;I)V
INVOKE_VIRTUAL      |     |v4.foo(v0,v2)  //Lcom/roscopeco/deelang/compiler/dex/CompilerFuncTestBase$Foo;.foo(Ldee/lang/DeelangInteger;Ldee/lang/DeelangInteger;)V
RETURN_VOID         |     |return

Development is moving quite quickly though, so over the next few days expect some support for most of the current (VM-based) capabilities.

Why go native?

As mentioned, the current VM architecture makes heavy use of reflection, which in some cases just doesn’t have the performance we need. Compiling ‘natively’ to Dex bytecode will eliminate all reflection in the generated code and core runtime, and will make compiled scripts fully-fledged members of your application. There won’t be any need to carry around the VM and it’s runtime, and instead you’ll only need a lightweight core runtime package that provides the standard implementations of the arithmetic operators, the if and or operators, and so on.

The trade off is flexibility. In the VM, all binding is done at runtime, with the ability to swap out any bound object (including the self reference) at any time. In the compiler, binding has to be static, so some of that flexibility is lost. But in 90% of cases (at least in our code) this doesn’t matter, as scripts are run against a fixed binding anyway (that’s what provides the ‘API’ for users to script against).

In any event, this isn’t about replacing the current VM setup – the new compiler is actually just a new back-end for the existing com.roscopeco.deelang.compiler.Compiler class. To compile the script above, you use the standard compiler with the new backend, like so:

Compiler c = new Compiler();
byte[] dex = c.compile(new DexCompilationUnit(c, "<no file>", Foo.class),
                 Parser.staticParse("foo(1,2)")).getCode();

You can still use the old (now called DVM) compiler and VM runtime as before (although there have been some slightly incompatible API changes during implementation of the new architecture, notably the moving of the deelang.* namespace to dee.vm.lang.*, so you’ll need to tweak your code a bit) if you feel it fits your needs better. There are currently no plans to drop the VM (although it may stop being the default at some point).