Deelang 0.20 released!

After about a month of coding (on and off), The Deelang DEX compiler is now feature complete, and what better way to celebrate than by releasing the code? It’s always been available in Subversion of course, but now you can download all the new stuff from the downloads page as either a source package or ready-made Jar, without all that messing around with Subversion, finding the right branch, or any of that. It’s right there, on the downloads page, just waiting for you to grab and go!

I’ve blogged a bit about the new features in this version (for example, here, here and here), and if you missed all that and are now wondering what on Earth I’m on about, I’ve also blogged generally about Deelang (e.g. here). The short version is that Deelang is a compiled script language designed for embedded devices (especially Android) that allows developers to give their users a simple way to extend their apps by writing small scripts.

Now that there’s a file release available, we’re hoping to get some bug reports and open a discussion about where Deelang could/should go. Other than the DEX compiler (which has been about a month of solid effort in OSS terms) development is a bit haphazard – it works for us, so we leave it. If we need something new, we implement it, then we leave it at that. I really want to get a community going around this thing, to make it more generally useful for everyone…

So why not grab the code, play around with it, file issues if you find any bugs, and come over and join the mailing list?

DEXing Deelang – On a real device!

Further to my previous post, I’ve been working this week on the new native compiler for Deelang on Android. This is a drop-in replacement for the standard custom bytecode compiler and VM that compiles Deelang scripts directly to Dalvik Executable  (DEX) bytecode for use on the Dalvik VM used in Android.

Development has moved fairly quickly, and there is now support for some of the basic features of the language. Not everything is supported at present (there’s no field access, no method blocks, and no arithmetic support) but the basics of function/method calls are implemented, along with local variable assignment and access, and a lot of behind the scenes support that make implementing the missing features a lot easier.

The compiler is being written largely test-first, so there’s good test coverage of new features as they go in (using Dex2Jar to generate readable dumps of the generated code, which is then compared against the expected code), but this doesn’t prove a crucial fact – that we can load the code into a running device, and execute it. To this end, there’s a small sample app that lets us write code on a device, compile it and run it against a simple binding we provide.

The screenshot shows this simple (and, if I’m honest, ugly) app running on an HTC One X. Compiling the code is quick enough given the early stage of development, but it’s actually running it where the native compiler shines – because the code is a standard Java class (it’s implemented as a subclass of the com.roscopeco.deelang.runtime.CompiledScript class, which is also a java.lang.Runnable) there’s no discernible difference between running a script and calling any other Java method.

We’re not benchmarking this in any serious way at this early stage in the project, but it’s nice to get a feel for how things are going from a performance perspective.

If you’re interested in taking a look at the work so far, you can get the sample app from SVN at the following repository URL:

http://deelang.googlecode.com/svn/branches/DEXCOMPILER/DeeSample/

The sample includes a snapshot Deelang jar, so you don’t need the actual project to try it out. Of course, if you want to play around with the code you can grab it from:

http://deelang.googlecode.com/svn/branches/DEXCOMPILER/deelang/

Updating the snapshot is as simple as running ‘ant jar‘ in the deelang directory, and then copying out the new snapshot over to the DeeSample libs/ directory.

DEXing Deelang – First steps in SVN

If you’re one of the few who’ve already taken a look at Deelang, you’ll know that it’s a simple scripting language that compiles to a custom bytecode format, which then runs in a simple virtual machine. It can run anywhere, but is targeted mostly for embedded devices, and especially for Android.

All this is fine, and for the most part it doesn’t perform too badly. Unfortunately though it does mean that it’s most common operation (calling out to Java methods) is relatively slow, because under the hood reflection is used everywhere. This has the potential to be a problem since in Dee, everything is a method – you can’t define your own classes, methods or functions in scripts, but instead rely on a (Java-side) API provided by whoever embedded the language in their application. Even arithmetic and conditionals are done with methods. Reflection (especially on Android) is just too slow for this kind of usage.

To address this, I’ve been thinking for some time about a implementing a native (i.e. Dex) compiler for Deelang on Android, and over the past few days I’ve finally made a start. The architecture is pretty well mapped out in a first-cut kind of way (well, in my mind at least) but the implementation is only just taking it’s first baby steps. Hardly anything is supported right now – in fact, the only thing it’s possible to actually compile is literals and simple, direct function calls (what the Dee VM calls SELFCALLs). But it’s a start, and it’s in Subversion on Googlecode now if you feel like taking a look. All the work is being done in a branch, which you can browse or check-out at:

http://deelang.googlecode.com/svn/branches/DEXCOMPILER/deelang

What can it do?

As mentioned above, it’s very limited at the moment. In fact, about the most exciting thing it can do is take code such as:

foo(1,2)

and compile it to native Dex bytecode like:

//class:0000  access:0x0001
public class DexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f extends com.roscopeco.deelang.runtime.CompiledScript

//method:0000  access:0x0001
//LDexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f;.run(Ldee/lang/DeelangObject;Lcom/roscopeco/deelang/runtime/Binding;)V
public V run(dee.lang.DeelangObject,com.roscopeco.deelang.runtime.Binding)
                this:v3   //DexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f
                    :v4   //dee.lang.DeelangObject
                    :v5   //com.roscopeco.deelang.runtime.Binding
CONST               |     |v1=0x00000001  // int:1   float:0.000000
NEW_INSTANCE        |     |v0=NEW Ldee/lang/DeelangInteger;
INVOKE_DIRECT       |     |v0.(v5,v1)  //Ldee/lang/DeelangInteger;.(Lcom/roscopeco/deelang/runtime/Binding;I)V
CONST               |     |v1=0x00000002  // int:2   float:0.000000
NEW_INSTANCE        |     |v2=NEW Ldee/lang/DeelangInteger;
INVOKE_DIRECT       |     |v2.(v5,v1)  //Ldee/lang/DeelangInteger;.(Lcom/roscopeco/deelang/runtime/Binding;I)V
INVOKE_VIRTUAL      |     |v4.foo(v0,v2)  //Lcom/roscopeco/deelang/compiler/dex/CompilerFuncTestBase$Foo;.foo(Ldee/lang/DeelangInteger;Ldee/lang/DeelangInteger;)V
RETURN_VOID         |     |return

Development is moving quite quickly though, so over the next few days expect some support for most of the current (VM-based) capabilities.

Why go native?

As mentioned, the current VM architecture makes heavy use of reflection, which in some cases just doesn’t have the performance we need. Compiling ‘natively’ to Dex bytecode will eliminate all reflection in the generated code and core runtime, and will make compiled scripts fully-fledged members of your application. There won’t be any need to carry around the VM and it’s runtime, and instead you’ll only need a lightweight core runtime package that provides the standard implementations of the arithmetic operators, the if and or operators, and so on.

The trade off is flexibility. In the VM, all binding is done at runtime, with the ability to swap out any bound object (including the self reference) at any time. In the compiler, binding has to be static, so some of that flexibility is lost. But in 90% of cases (at least in our code) this doesn’t matter, as scripts are run against a fixed binding anyway (that’s what provides the ‘API’ for users to script against).

In any event, this isn’t about replacing the current VM setup – the new compiler is actually just a new back-end for the existing com.roscopeco.deelang.compiler.Compiler class. To compile the script above, you use the standard compiler with the new backend, like so:

Compiler c = new Compiler();
byte[] dex = c.compile(new DexCompilationUnit(c, "<no file>", Foo.class),
                 Parser.staticParse("foo(1,2)")).getCode();

You can still use the old (now called DVM) compiler and VM runtime as before (although there have been some slightly incompatible API changes during implementation of the new architecture, notably the moving of the deelang.* namespace to dee.vm.lang.*, so you’ll need to tweak your code a bit) if you feel it fits your needs better. There are currently no plans to drop the VM (although it may stop being the default at some point).

Deelang – First file release available to download

After a few months in Subversion, with a few folks playing with the code and seeing what use they can make of it, the first file release of Deelang is now available from the project’s downloads page. This release is still a very early public release (as denoted by the 0.18 version number) but is already mostly functional (with the notable exception of first-class regular expressions, which may never make it in anyway) and ready for testing.

Like most of our Android stuff, Deelang started life as internal code for a (closed source) third-party app, that ended up being deemed “possibly generally-useful enough” for open-source. So, at the moment, it mostly does the things we needed it to do when it was written. And, as with ORMDroid, the intention here isn’t to just dump the code into Google Code and forget about it – I intent to keep developing these things. What I really need is for folks to pick up the code, start using it, and either come back to me and tell me what cool new things they want to see it do, or (even better!) send over a patch with the aforementioned cool new things.

In case you’ve read this far in the hope of finding out what the heck Deelang is, the project page summary puts it thus:

Deelang is a lightweight, embeddable, dynamically-typed scripting language for Java that is intended for use in user-scriptable applications in limited resource environments, and especially for Android devices.

Basically it’s a very simple scripting environment that allows you to provide some manner of customization to your users, or to others deploying your apps. You define an API (it comes with almost none, beyond a basic object hierarchy for ‘primitives’) for code to run against, and Deelang provides a compiler, and a virtual machine in which to run scripts against this API. To quote (again) from the project site on google, it allows you/your users/someone else to write scripts like:

pic = Camera.take_picture()
pic.save_to_media()

Facebook.share(pic) {
caption = "I just took this picture!"
  album = "Random pics I took"
} or {
  post_status("took a pic but couldn't upload it :(")
}

(Obviously, assuming you provide the appropriate Camera and Facebook implementations). These scripts can then be compiled and run on the fly, or compiled and cached for later, perhaps to run in response to some broadcast intent or whatever.

So anyway, if you have a minute, grab the code and take a look, have a look at the (sparse) wiki, see if Deelang might be useful in your own projects, and if you think of a killer feature that you could really, really use, get in touch via the comments and we’ll talk…