Deelang 0.22 released on GitHub!

So I spent a bit of time today getting together the overdue release of Deelang ready, and finally pushed the 0.22 binary releases to Google Code, with source releases staying on GitHub. The Google Code project has now been fully migrated over to GitHub, and remains only to host binary downloads for those that want to get up and running quickly.

This release of Deelang is largely a maintenance release, with a number of bugfixes to the internals of the Dex compiler, the runtime library and a few other places. It does however include a few new features, notably:

  • Or Blocks are now implemented in the DEX compiler, and are used exactly like as they are in the Dee VM. 
  • Comparison operators are fully implemented.
  • Method chaining on parenthesized expressions works as it always should have!

This release doesn’t yet include the new AST rewriting I previously blogged about (that’s planned for 0.30 at the moment, as it’s a big change) but does address a lot of the issues that will eventually fix in a quick-fix kinda way – mostly as short-term fixups that will go away once the migration to the new AST layout is complete.

 

Advertisements

Deelang & ORMDroid: Moving to GitHub

Image representing GitHub as depicted in Crunc...

After spending all their lives so far on Google Code, I’ve decided it’s time to move both Deelang and ORMDroid over to GitHub. The projects have moved away from the old Subversion VCS and are now using Git, and the actual moving of the wiki pages, issues and everything else is well underway.

Why the move? Well, GitHub makes collaboration much more pleasant (Forks and pull-requests vs. patches isn’t much of a contest), and crucially for an Open Source project, a lot more people are comfortable with Git these days than SVN (in the OSS world, at least). Until fairly recently the whole Git thing had passed me by to an extent, but it’s easy to pick up and I have to say it’s now my preferred VCS.

On top of all that, I just prefer the GitHub interface!

Here are the new links to the projects. Please update your references if you have them.

Rewriting Vs. Rewriting

As previously mentioned, I planned to spend yesterday rewriting Deelang‘s parser with the aim of fixing some long-term issues and revisiting some design decisions that get in the way of the DEX compiler. As it turned out, I didn’t have as much time as I wanted to spend on this (as usual!) but I did get some time to play with it, and it turned out that my decision to rewrite much of the parser was, well, somewhat premature.

The main aim of the rewrite was to change the way the AST is generated, to remove the imaginary CHAIN nodes that proliferate the current AST. CHAIN nodes are used as the LHS in field accesses and method calls, as well as one or two other places, and denote that the receiver is the result of the previous operation. They work great when compiling for a stack machine (such as the JVM or the original Dee VM) – they translate simply as “pop the top object from the stack” – but are less intuitive when compiling for a register architecture, such as Dalvik.

Continue reading

Biting the bullet – time for a rewrite

In response to a feature request from some of the guys here, my current work on Deelang focusses on implementing proper equality and comparison operators as part of the language. The current release has no support for operators beyond basic arithmetic – equality and comparison have traditionally been implemented as methods hacked on top of the standard library. This results in code such as:

1.eql(2)  => true
5.lt(10)  => true
10.gt(20) => false

Leaving aside for a moment that the above isn’t actually that bad (in my opinion at least), implementing this as operators is actually quite simple for both the dex and deevm compilers. However, while putting it together I ran up against an old elephant that’s been sitting quietly in the corner for some time now – the current parser is a mess, and it might just be time to rewrite the grammar.

The problem is not new, but I’ve ignored it for a while. Basically, it boils down to the fact that the current parser cannot handle code such as:

(1+2).foo()

Not only does this not work, but it fails miserably with nothing more specific than a MismatchedSetException at the terminator, after a lot of backtracking and ultimately ignoring the method call completely. The above code parses to the following tree:

Parse Tree Notice all the abandoned trees (in red), before the (erroneous) final parse, and the MismatchSetException up there on the right. The method call gets parsed at one point, but that tree was then abandoned in favour of a tree in which the call is quietly ignored. This is clearly one confused parser, and all over something as simple as (1+2).foo()! Clearly, this needs fixing.

As I say, I’ve ignored this for a while. It should be relatively simple to fix (and indeed it is) with a bit of rejigging in the grammar. However, this problem is actually symptomatic of something deeper – namely, that the Deelang grammar is a mess. In the past, as problems such as this have cropped up, they’ve been fixed by adding to the grammar. New productions, imaginary tokens, and syntactic predicates have all been added to cope with a specific case, with no real wider plan.As long as the tests still passed at the end, the additions stayed.

The result of all this is that things that should be handled in a unified way are actually handled in a variety of ways. My personal favourite example of this is the way chained method calls are handled – I won’t illustrate it here, but if you’re interested just debug something like “foo().bar().baz().quux()” in ANTLRWorks. Trust me, it’s not pretty. It’s inefficient, it’s inconsistent with other parts of the grammar, and it requires the compiler to jump through some pretty awkward hoops to keep track of who is using what target register. It worked well when the only target was the (stack-based) Dee vm, but as requirements have grown it’s become cumbersome – the only reason it still works this way is inertia.

With all this in mind, I’ve decided that now is the ideal time to rewrite the grammar to get rid of these issues. I’ve never been a big believer in planning to throw one away, but in this case it looks like I will, anyhow. To be fair, I’m not planning a complete rewrite – large parts of the grammar are fine as they are (literals, for example). But the meat of it – from atoms through function calls to method calls – will be rewritten in a way that’s more consistent, cleaner, and hopefully requires a lot less backtracking. I’m also aiming to reduce the lookahead where I can, although some of the actual language design makes this quite difficult.

Unfortunately this grand plan must remain just that for now – I’m very short on time to actually work on this at the moment. Since it’s not an actual key requirement I can’t allocate any actual work time to it (even though it will make things easier and save time down the line). So provisionally, I’ve set aside Sunday for the rewrite.

Now I just need to hope nothing more pressing crops up between now and then. Wish me luck!

Deelang 0.20 released!

After about a month of coding (on and off), The Deelang DEX compiler is now feature complete, and what better way to celebrate than by releasing the code? It’s always been available in Subversion of course, but now you can download all the new stuff from the downloads page as either a source package or ready-made Jar, without all that messing around with Subversion, finding the right branch, or any of that. It’s right there, on the downloads page, just waiting for you to grab and go!

I’ve blogged a bit about the new features in this version (for example, here, here and here), and if you missed all that and are now wondering what on Earth I’m on about, I’ve also blogged generally about Deelang (e.g. here). The short version is that Deelang is a compiled script language designed for embedded devices (especially Android) that allows developers to give their users a simple way to extend their apps by writing small scripts.

Now that there’s a file release available, we’re hoping to get some bug reports and open a discussion about where Deelang could/should go. Other than the DEX compiler (which has been about a month of solid effort in OSS terms) development is a bit haphazard – it works for us, so we leave it. If we need something new, we implement it, then we leave it at that. I really want to get a community going around this thing, to make it more generally useful for everyone…

So why not grab the code, play around with it, file issues if you find any bugs, and come over and join the mailing list?

Dex vs Dee VM – A microbenchmark

During a recent discussion, I had the need to show some numbers demonstrating why Deelang needs the new DEX compiler. As I said in that thread, it’s all about removing reflection from the compiled classes.

We all know that “reflection is slow” (for some value of slow) and especially on Android, that it isn’t generally a good solution for general use in method invocation. Looking up methods by reflection is slow, as is performing the actual invoke. But how slow? Could I quantify the difference?

Since the Deelang DEX compiler is still being actively developed, I’ve never paid much attention to performance before, and have certainly never benchmarked it. We knew that the Dee VM wasn’t fast enough for some of the things we were using it for, and I knew from experience that it would be faster without reflection, and we left it at that. This discussion got me thinking though, and I decided it would be nice to have some idea of the actual difference between the Dee VM and scripts compiled to DEX.

Despite the fact that microbenchmarks are often a bad idea, and are a bit of a minefield at the best of times, I wrote one in this instance. After all, I only want some idea of the comparative performance for the sake of discussion – I won’t actually be basing any design or implementation decisions on these results.

I expected a decent performance boost in the DEX compiled scripts, but I honestly wasn’t quite prepared for how much of a boost:

09-27 10:45:27.525: I/BENCHMARK(15853): Rehearsal
09-27 10:45:37.665: I/BENCHMARK(15853): DeeVM completed : 10000 runs : 10107ms (10.107 seconds)
09-27 10:45:37.725: I/BENCHMARK(15853): Dex completed   : 10000 runs : 31ms (0.031 seconds)
09-27 10:45:37.725: I/BENCHMARK(15853): Real
09-27 10:45:46.695: I/BENCHMARK(15853): DeeVM completed : 10000 runs : 8938ms (8.938 seconds)
09-27 10:45:46.745: I/BENCHMARK(15853): Dex completed   : 10000 runs : 25ms (0.025 seconds)

(This is logcat from the benchmark running on-device on a HTC One X)

That’s right – almost 9 seconds for the reflection-based Dee VM, compared to 0.025 seconds for the DEX compiled version. The difference is so marked that I at first didn’t believe the results. I changed the script to actually produce some output to logcat, to make sure that it was actually running each implementation properly, and it was. It turns out that, in this case, reflection has a much bigger impact that I expected.

In case you’re interested, you can see the benchmark code at the this page on the project wiki on Google Code.

DEXing Deelang – On a real device!

Further to my previous post, I’ve been working this week on the new native compiler for Deelang on Android. This is a drop-in replacement for the standard custom bytecode compiler and VM that compiles Deelang scripts directly to Dalvik Executable  (DEX) bytecode for use on the Dalvik VM used in Android.

Development has moved fairly quickly, and there is now support for some of the basic features of the language. Not everything is supported at present (there’s no field access, no method blocks, and no arithmetic support) but the basics of function/method calls are implemented, along with local variable assignment and access, and a lot of behind the scenes support that make implementing the missing features a lot easier.

The compiler is being written largely test-first, so there’s good test coverage of new features as they go in (using Dex2Jar to generate readable dumps of the generated code, which is then compared against the expected code), but this doesn’t prove a crucial fact – that we can load the code into a running device, and execute it. To this end, there’s a small sample app that lets us write code on a device, compile it and run it against a simple binding we provide.

The screenshot shows this simple (and, if I’m honest, ugly) app running on an HTC One X. Compiling the code is quick enough given the early stage of development, but it’s actually running it where the native compiler shines – because the code is a standard Java class (it’s implemented as a subclass of the com.roscopeco.deelang.runtime.CompiledScript class, which is also a java.lang.Runnable) there’s no discernible difference between running a script and calling any other Java method.

We’re not benchmarking this in any serious way at this early stage in the project, but it’s nice to get a feel for how things are going from a performance perspective.

If you’re interested in taking a look at the work so far, you can get the sample app from SVN at the following repository URL:

http://deelang.googlecode.com/svn/branches/DEXCOMPILER/DeeSample/

The sample includes a snapshot Deelang jar, so you don’t need the actual project to try it out. Of course, if you want to play around with the code you can grab it from:

http://deelang.googlecode.com/svn/branches/DEXCOMPILER/deelang/

Updating the snapshot is as simple as running ‘ant jar‘ in the deelang directory, and then copying out the new snapshot over to the DeeSample libs/ directory.

DEXing Deelang – First steps in SVN

If you’re one of the few who’ve already taken a look at Deelang, you’ll know that it’s a simple scripting language that compiles to a custom bytecode format, which then runs in a simple virtual machine. It can run anywhere, but is targeted mostly for embedded devices, and especially for Android.

All this is fine, and for the most part it doesn’t perform too badly. Unfortunately though it does mean that it’s most common operation (calling out to Java methods) is relatively slow, because under the hood reflection is used everywhere. This has the potential to be a problem since in Dee, everything is a method – you can’t define your own classes, methods or functions in scripts, but instead rely on a (Java-side) API provided by whoever embedded the language in their application. Even arithmetic and conditionals are done with methods. Reflection (especially on Android) is just too slow for this kind of usage.

To address this, I’ve been thinking for some time about a implementing a native (i.e. Dex) compiler for Deelang on Android, and over the past few days I’ve finally made a start. The architecture is pretty well mapped out in a first-cut kind of way (well, in my mind at least) but the implementation is only just taking it’s first baby steps. Hardly anything is supported right now – in fact, the only thing it’s possible to actually compile is literals and simple, direct function calls (what the Dee VM calls SELFCALLs). But it’s a start, and it’s in Subversion on Googlecode now if you feel like taking a look. All the work is being done in a branch, which you can browse or check-out at:

http://deelang.googlecode.com/svn/branches/DEXCOMPILER/deelang

What can it do?

As mentioned above, it’s very limited at the moment. In fact, about the most exciting thing it can do is take code such as:

foo(1,2)

and compile it to native Dex bytecode like:

//class:0000  access:0x0001
public class DexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f extends com.roscopeco.deelang.runtime.CompiledScript

//method:0000  access:0x0001
//LDexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f;.run(Ldee/lang/DeelangObject;Lcom/roscopeco/deelang/runtime/Binding;)V
public V run(dee.lang.DeelangObject,com.roscopeco.deelang.runtime.Binding)
                this:v3   //DexCompiledScript531a8036-8e20-4965-8148-e87dfb51283f
                    :v4   //dee.lang.DeelangObject
                    :v5   //com.roscopeco.deelang.runtime.Binding
CONST               |     |v1=0x00000001  // int:1   float:0.000000
NEW_INSTANCE        |     |v0=NEW Ldee/lang/DeelangInteger;
INVOKE_DIRECT       |     |v0.(v5,v1)  //Ldee/lang/DeelangInteger;.(Lcom/roscopeco/deelang/runtime/Binding;I)V
CONST               |     |v1=0x00000002  // int:2   float:0.000000
NEW_INSTANCE        |     |v2=NEW Ldee/lang/DeelangInteger;
INVOKE_DIRECT       |     |v2.(v5,v1)  //Ldee/lang/DeelangInteger;.(Lcom/roscopeco/deelang/runtime/Binding;I)V
INVOKE_VIRTUAL      |     |v4.foo(v0,v2)  //Lcom/roscopeco/deelang/compiler/dex/CompilerFuncTestBase$Foo;.foo(Ldee/lang/DeelangInteger;Ldee/lang/DeelangInteger;)V
RETURN_VOID         |     |return

Development is moving quite quickly though, so over the next few days expect some support for most of the current (VM-based) capabilities.

Why go native?

As mentioned, the current VM architecture makes heavy use of reflection, which in some cases just doesn’t have the performance we need. Compiling ‘natively’ to Dex bytecode will eliminate all reflection in the generated code and core runtime, and will make compiled scripts fully-fledged members of your application. There won’t be any need to carry around the VM and it’s runtime, and instead you’ll only need a lightweight core runtime package that provides the standard implementations of the arithmetic operators, the if and or operators, and so on.

The trade off is flexibility. In the VM, all binding is done at runtime, with the ability to swap out any bound object (including the self reference) at any time. In the compiler, binding has to be static, so some of that flexibility is lost. But in 90% of cases (at least in our code) this doesn’t matter, as scripts are run against a fixed binding anyway (that’s what provides the ‘API’ for users to script against).

In any event, this isn’t about replacing the current VM setup – the new compiler is actually just a new back-end for the existing com.roscopeco.deelang.compiler.Compiler class. To compile the script above, you use the standard compiler with the new backend, like so:

Compiler c = new Compiler();
byte[] dex = c.compile(new DexCompilationUnit(c, "<no file>", Foo.class),
                 Parser.staticParse("foo(1,2)")).getCode();

You can still use the old (now called DVM) compiler and VM runtime as before (although there have been some slightly incompatible API changes during implementation of the new architecture, notably the moving of the deelang.* namespace to dee.vm.lang.*, so you’ll need to tweak your code a bit) if you feel it fits your needs better. There are currently no plans to drop the VM (although it may stop being the default at some point).

Deelang – First file release available to download

After a few months in Subversion, with a few folks playing with the code and seeing what use they can make of it, the first file release of Deelang is now available from the project’s downloads page. This release is still a very early public release (as denoted by the 0.18 version number) but is already mostly functional (with the notable exception of first-class regular expressions, which may never make it in anyway) and ready for testing.

Like most of our Android stuff, Deelang started life as internal code for a (closed source) third-party app, that ended up being deemed “possibly generally-useful enough” for open-source. So, at the moment, it mostly does the things we needed it to do when it was written. And, as with ORMDroid, the intention here isn’t to just dump the code into Google Code and forget about it – I intent to keep developing these things. What I really need is for folks to pick up the code, start using it, and either come back to me and tell me what cool new things they want to see it do, or (even better!) send over a patch with the aforementioned cool new things.

In case you’ve read this far in the hope of finding out what the heck Deelang is, the project page summary puts it thus:

Deelang is a lightweight, embeddable, dynamically-typed scripting language for Java that is intended for use in user-scriptable applications in limited resource environments, and especially for Android devices.

Basically it’s a very simple scripting environment that allows you to provide some manner of customization to your users, or to others deploying your apps. You define an API (it comes with almost none, beyond a basic object hierarchy for ‘primitives’) for code to run against, and Deelang provides a compiler, and a virtual machine in which to run scripts against this API. To quote (again) from the project site on google, it allows you/your users/someone else to write scripts like:

pic = Camera.take_picture()
pic.save_to_media()

Facebook.share(pic) {
caption = "I just took this picture!"
  album = "Random pics I took"
} or {
  post_status("took a pic but couldn't upload it :(")
}

(Obviously, assuming you provide the appropriate Camera and Facebook implementations). These scripts can then be compiled and run on the fly, or compiled and cached for later, perhaps to run in response to some broadcast intent or whatever.

So anyway, if you have a minute, grab the code and take a look, have a look at the (sparse) wiki, see if Deelang might be useful in your own projects, and if you think of a killer feature that you could really, really use, get in touch via the comments and we’ll talk…