Generate less bytecode with default methods

Java’s default methods (introduced back in Java 8) are one of those features that solve the intended problem reasonably well, while at the same time allowing all kinds of nasty code and weird inheritance stuff when used in a general-purpose kind of way. Indeed, they’ve been the subject of a ton of posts and the way they (fail to) work still surprises people. They’ve been around for a while now, and there’s a lot of good advice out there on how they should and shouldn’t be used, so that’s not what I’m going to talk about in this post.

What I am going to talk about, however, is the way I’m using them in Moxy, as a way to do a lot less work in runtime-generated bytecode.

A bit of background

Moxy is a mock framework, and it does it’s job by generating mock classes at runtime. These mock classes naturally use a fair bit of bytecode generation, replacing real method implementations with new methods that implement the mock behaviour.

These mock methods do the standard mock thing of recording their invocation, and then they implement behaviour that has been set-up on the mock beforehand.

The problem

In order to implement this behaviour, mock objects need access to all kinds of state that’s specific to the instance – stubbed return values, exceptions they should throw, and more. They also need to have easy access to the mock engine that created them (so they can record invocations, for example).

This is often implemented via static methods on some class somewhere whose job it is to store state. The generated code will typically be peppered with a ton of invokestatic instructions that calls out to methods on those classes in order to get things done.

This works, but it has a few problems:

  • Some kind of mapping between instances and their state must be maintained.
  • It can make it difficult to swap out mock strategies in a clean fashion.
  • It can make testing (of the framework) difficult.

The Moxy approach

Instead of storing state statically, Moxy stores it right there in the mock, and all interaction with the state is done with instance methods. These methods can’t be inherited from a superclass (since mocks usually subclass the mocked class), and I really don’t want to generate them (I like to keep generated code to a minimum, as it’s harder to maintain and test that regular Java), so instead Moxy abuses default methods.

It works like this:

  • All mocks implement an interface, ASMMockSupport.
  • This interface defines one abstract method, __moxy_asm_ivars(). This method is implemented in generated code (since interfaces don’t have instance variables, the mocks hold the variables and expose them via this method)*.
  • The support interfaces defines a ton of default methods that do various kinds of work based on those instance variables.
  • The actual generated code for mocked methods uses invokeinterface calls on this to get stuff done as needed.
  • Other parts of the framework that interact with the mocks (e.g. stubbers) simple cast the mock instance to ASMMockSupport and call the methods they need.

* Yes, that’s a strange name for a method. Moxy does this to lessen the chances of its built-in methods clashing with methods being mocked.

 

The default methods on the support interface handle things like finding the engine that created the mock, adding stubbing, finding appropriate stubbing for a given invocation, running delegate methods and actions, and determining whether the current thread is currently in what Moxy calls a monitored invocation (used for recording purposes only, when mock behaviour is disabled).

This has the benefit that a whole lot of fairly-complex code that would otherwise be either generated, or stuffed into static methods, is now implemented, in Java, as instance methods. They’re easy to unit-test, and they’re easy to maintain. The added bonus is the generated bytecode is also easier to maintain, since it’s as short as possible.

Is this the intended use of default methods? Not at all. Is it correct, or appropriate? I’ll leave that to you to decide. I firmly believe, however, that in this very limited use-case it’s a good solution to a tricky problem, all things considered.

If you’re interested in looking at the code, you’ll find it here.

Advertisements

Mocking v2.0

We all love to use mocks in our Java tests, right? Add the usual mock framework to your test dependencies, sprinkle a few mock, when and thenWhatever calls in to your code, and happy days. You’re able to test things in isolation, and as an added bonus ensure your code is actually calling the stuff it’s supposed to be calling by verifying the mock afterwards. The sun is shining, life is good, and your code is tested to perfection.

But then the clouds roll in, and you have to test a final class. Or you need to mock out a static method in some legacy code. Or, horror, some code that directly calls new on a class you want to mock.

You could hit up StackOverflow, where you’ll find that all these things are possible with existing frameworks, assuming you’re prepared to add in another dependency, use a ton of boilerplate, and refactor your code a bit. Depending on which framework you’re currently using there are a ton of different ways you can kind of achieve what you need.

By now you’ve sunk another hour or two, and it’s still not quite working.

What if there was another way?

The other way

What if I told you there was a framework out there that let you take a final class, and just do this?

mockClasses(FinalClass.class);

FinalClass sc = new FinalClass();

assertThat(sc.returnHello()).isNull();

assertMock(() -> sc.returnHello()).wasCalledOnce();

when(() -> sc.returnHello()).thenReturn("Goodbye");

assertThat(sc.returnHello()).isEqualTo("Goodbye");

assertMock(() -> sc.returnHello()).wasCalledTwice();

What if it also did statics, with the same API, and no extra dependencies?

mockClasses(ClassWithStatic.class);

assertThat(ClassWithStatic.returnHello()).isNull();

when(() -> ClassWithStatic.returnHello()).thenReturn("Goodbye");

assertThat(ClassWithStatic.returnHello()).isEqualTo("Goodbye");

What if it even let you mock out constructors?

final RuntimeException rte = new RuntimeException("MARKER");

mockClasses(FinalClass.class);

when(() -> new FinalClass()).thenThrow(rte);

assertThatThrownBy(() -> new FinalClass()).isSameAs(rte);

What if we put all our arguments about whether we should be doing this kind of mocking to one side for now, because sometimes in life, you’re faced with a nail.

And when you’re faced with a nail, you need a hammer.

There’s one here: https://github.com/roscopeco/moxy

Or grab it with Maven:

<dependency>
	<groupId>com.roscopeco.moxy</groupId>
	<artifactId>moxy-core</artifactId>
	<version>0.90.1</version>
</dependency>

Curb FFI – FFI port of Curb

I believe I mentioned that I’m back at work on the Curb project. In case you missed that, Curb is a ruby extension that provides bindings to the libcurl library. I started it way back, left to concentrate on other things (having kids, making money, etc) and now I’m involved again, working with Todd Fisher who took over maintaining the project when I bowed out. I’ll be fixing bugs, answering questions and generally helping out.

In the meantime, I’ve also started work on porting (if that’s the right word) Curb to FFI, with a view to moving away from the existing C code. The motivations are manifold:

  • As it stands, Curb is pretty much tied to MRI. In the modern Ruby world, where you’ve got JRuby and Rubinius and who-knows-what-next, this is recognised as a bad thing.
  • It’s a nightmare to get it working on Windows. This is because, and I can speak with some authority here as someone who develops on Windows every day, Windows sucks for development. Unless you’re using all-Microsoft tooling, in which case it’s pretty awesome. But for interoperability with portable code, and libraries targeted primarily at other platforms, it sucks.
  • FFI is probably the right way to do these things these days. 10+ years ago, when Curb was first hacked together in about six hours, C extensions were the shizz. Now, not so much. Unless you really need the level of hardware hackery and performance you can get with C, things are better off in Ruby code.

So to sum up, this port is about future-proofing Curb, making it easier to develop, easier to use cross-platform, and (in the long run) safer, probably more performant, and ensuring it can run on all Rubies, including whatever whizz-bang next-gen thing comes out next week (my bet is it’ll be written in Rust. Or Go. Or something…).

Check it out (or clone it, as the cool kids say nowadays) at https://github.com/roscopeco/curb/tree/ffi.

Building Curb on Windows

man with laptop over his head
At first I was like…

As I mentioned in my previous post, I’m now working on Curb again, and currently on Windows. Back in 2006 when I wrote the original Curb, I wouldn’t go near a Windows box for religious reasons, and the project has never officially supported the platform. There are plenty of bugs and posts around about people having mixed success with getting it to work.

Since then, my views have mellowed, partly I think because I’ve come to appreciate that there’s no room for religion in software engineering – you use the right tool for the job. Partly that, and partly by my desire to, you know, get paid and own stuff and whatnot. These days I do as much work on Windows as I do on *nix, and my bank balance is all the better for it. So anyway, back to the point of this post.

Building Curb on Windows is a bit of a nightmare. It took me a fair while to figure it out, and I wrote the damn thing. If it got me feeling like the guy in the picture above, then how must the everyday user trying to get the gem going feel? If they’re just trying to install it and write some code, I’m guessing they immediately give up and go with another library. But what if they are trying to use something else that depends on the Curb gem? I’m guessing that’s when they get so frustrated they file shouty bug reports with the project. And I don’t blame them. It would have been nice to have someone to shout at while I was trying to get it working.

In the end, I did get it going, but it required me to build libcurl from source. I posted an answer to the bug report I mentioned above with the how-to of it, and the poor person who filed it reported back that it worked. You could click through, but I’ll reproduce the steps below to save your finger.

Most likely, your compiled libcurl is incompatible with the ruby devkit compiler. You can verify this by running (this is on my machine, obviously change your paths):


> c:\Ruby200Devkit\mingw\bin\gcc.exe -L. -lcurl
# c:/[...]/ld.exe: skipping incompatible ./libcurl.dll when searching for -lcurl
# c:/[...]/ld.exe: skipping incompatible ./libcurl.dll when searching for -lcurl
# c:/[...]/bin/ld.exe: cannot find -lcurl
# collect2.exe: error: ld returned 1 exit status

Notice the lines about “Skipping incompatible” before the final “cannot find lcurl”. Try grabbing the curl source from https://curl.haxx.se/download/curl-7.54.0.tar.gz , extract it somewhere, and then open up a terminal and try the following:


> c:\path-to-your-ruby-devkit\devkitvars.sh
> which gcc
# important - verify here that the gcc is the one from your ruby devkit!
> cd 
> sh configure --with-winssl     # and whatever other options you require
> make
> make install

It’s very important that you don’t have any other mingw/msys/cygwin compilers in your path, and that you do this in a windows terminal, not msys/cygwin sh. Otherwise, you’ll probably just build another incompatible library.

You should now have libcurl-4.dll in c:\path-to-your-ruby-devkit\local\bin. Now it gets a bit messy, so hang in there. This part is probably a bug in the build, but easy to work around.


> cd c:\path-to-your-ruby-devkit\local\bin
> copy libcurl-4.dll libcurl.dll

The build needs libcurl.dll (it won’t compile with libcurl-4.dll, but the extension won’t run without the -4 suffix, so we’ll have to manually copy the dll later on).


> gem install curb -- --with-curl-lib=c:\path-to-your-ruby-devkit\local\bin --with-curl-include=c:\path-to-your-ruby-devkit\local\include

Assuming the build goes well (it should, though it might take a while), you now just need to copy the curl dll to the appropriate place:


> copy c:\path-to-your-ruby-devkit\local\bin\libcurl-4.dll C:\Ruby22-x64\lib\ruby\gems\2.2.0\gems\curb-0.9.3\ext

Now run irb -rcurl and, all being well, you should be good to go.

I’m back on the Curb team!

curl / libcurl
curl lives at https://curl.haxx.se/

Way back in 2006, after reading on ruby-talk about people’s problems using libcurl with Ruby, I put together a basic C extension to replace the (by-then-unmaintained) bindings that already existed. Little did I know then that it would end up being some of the most popular open-source code I’ve written.

In time, pressures of family, work, and life in general meant I didn’t have a lot of time to devote to my open-source works, and so Todd Fisher took over as maintainer of the project, and did some awesome work improving it, building the multi interface code, and generally making Curb more and more useful. The project is still going strong over ten years later.

I came back to Curb via some work I was doing on a Windows-based system, and after a few hiccups building the library I found myself patching things up a bit to make them work on Windows (which isn’t officially a supported platform for Curb), and a couple of pull requests later Todd got in touch and asked if I’d like to be added to the project on Github. I thought this over for, oh, about ten seconds, and then said yes. I have more time these days for Open Source work, and I don’t have anything currently on the front-burner, so I’m now back on Curb. Todd’s still the boss (after all, he has more code in there than I do!) but I’ll be happily working away, fixing bugs where I can, and trying to make life easier for people out there wanting to use Curb on Windows.

Here’s the original announcement of Curb, from November of 2006.

From: Ross Bamford
Date: November 17, 2006 1:25:05 PM CST
To: ruby-talk_at_ruby-lang.org (ruby-talk ML)
Subject: [ANN] Curb 0.0.1 – New ruby libcurl bindings
Reply-To: ruby-talk_at_ruby-lang.org

Curb 0.0.1 is now available from http://curb.rubyforge.org. Curb
provides nice, easy to use bindings to the libcurl URL transfer
library.

This is the first release, and Curb is still a work-in-progress.
Currently, it supports the curl_easy interface, and can handle the
most common usages for libcurl.

The project is now seeking user feedback, testers on various
platforms, and requests for the features you need the most.

Blog Reboot

Just a quick note to say that the blog is now back online, but sadly missing quite a few posts that were lost by my old webhost.

The last post I’ve been able to restore is from almost two years ago, when I was just starting out on the drone project. Here’s a quick summary of what’s been happening since then:

  • The drone project is still unfinished, and has fallen by the wayside a bit if I’m honest. I still plan to get it going, someday.
  • I’ve recently found time to resume work on my minimal experimental kernel, Mink. (Find it on github if you’re interested).
  • I have a new day-job – I’m now working for a small non-IT company, putting together a brand new bespoke MRP system for them using Java EE, with a Vaadin front end. In between times, I’m also the entire IT department, which is kinda stressful but also fun.

So all in all you’ve not missed a great deal, and now the blog has a new home, I’ll be regularly updating here with new posts.

The Drone Project

IOIO Board
Because I like experimenting with things, and because I love cool toys, I’ve decided the time is right to build myself a drone quadcopter. 

I’ve played around with RC ‘copters before, and my electronics knowledge is fair (if a little rusty), so I’m not too worried about the actual build (he said, naively). The bit I’m really looking forward to is the software.

I’m still very much in the planning stages (so far I’ve bought one bit of hardware – see pic right). The main goal for this drone is that it will not be radio controlled. I want it to be totally autonomous, without any onboard RC gear at all. That’s right – I even want it to handle take-off and landing all by itself. A tad over-ambitious? I guess only time will tell!

That’s all for now on the drone front, but stay tuned for updates as the build progresses. I’ll be posting here as things move on, and hopefully some day soon I’ll post a video of the maiden flight!

The sharp-eyed amongst you might already have realised how I’m planning to control my drone, but if not, then consider this: Where might I find a device that runs code, and has accelerometers/gyros, a compass, GPS, a camera and wireless network capability, all built in as standard?

Spritesheets Revisited – Game sprites with Inkscape + Gimp

I’ve lately been doing some work with the (awesome) Unity game engine, prototyping a 2D platformer using version 4.3’s new 2D mode, and as part of that I needed to create some sprite-sheets for game-related animations and the like.

Inkscape Screen Shot
Spritesheet fodder!

As I’ve mentioned before, my tools of choice for graphical work are Inkscape and The GIMP, so I needed to  put together a workflow that would allow me to draw vectors in Inkscape and then export pixels to Gimp, where I would create suitable spritesheets for import into Unity.

I don’t want delve too deeply into the Inkscape/Gimp interaction in this post (I’m still stuck on Windows at the moment, so I just followed this post to get Inkscape’s “Save as XCF with layers” feature working – thanks to Chris Hillbig for the instructions). So, skipping over that part, what I ended up with was a GIMP image, made up of layers, each layer being an animation frame. I now needed to make this into a flat spritesheet, suitable for PNG export to Unity.

If all this sounds a bit familiar, that’s because it’s very reminiscent of this post from September 2012 (if you’ve read that post, then thanks for sticking around for this long!), when I was busy putting retroify.me together and needed to easily create spritesheets, in GIMP, from layered images. Of course, I initially just used the GIMP script I developed then to create my sheets, and while it did work, it wasn’t perfect. Why? Well, to explain that, I think a little background is in order.

The way retroify.me works, it’s easy (and efficient) for us to use very wide spritesheets with a single row of sprites, and so the script I wrote way back in 2012 does just that – it creates a sheet that’s arbitrarily wide, with the overall height being the height of a single sprite. For Unity, however, this is non-optimal. Like most game-engines, and anything else that works directly with video hardware to provide high-performance graphics, Unity prefers its assets to be sized by powers of two. While it will happily chomp away on an image that’s 5376 pixels by 256, and even handle the slicing for me, this can result in some odd visual artifacts, and will prevent proper optimisation of the graphics in the game by the engine. I really needed to revisit my old script, and add in the ability to create images with multiple rows, to allow my spritesheets to fit neatly into power-of-two-sized images.

And so, it was back to Scheme. As I mentioned in the old post, I’m not a Scheme programmer, but it’s always fun to play around in an unfamiliar language. I spent a little while hacking around, re-reading the GIMP script API, and generally having fun, and the end result is a new script that does everything I need. In the spirit of the previous post, here is the updated code:

; Script to convert image layers to a sprite sheet.
; Copyright (c) 2012-2014 Ross Bamford.
; http://www.apache.org/licenses/LICENSE-2.0.html
(define (script-fu-unityspritesheet img sheet-size-opt)
  (let* (
      (layer-count          (car    (gimp-image-get-layers img)))
      (layer-ids            (cadr   (gimp-image-get-layers img)))
      (sprite-width         (car    (gimp-image-width img)))
      (sprite-height        (car    (gimp-image-height img)))
      (spritesheet-width    (expt 2 (- 12 sheet-size-opt)))
      (sprites-per-row      (/ spritesheet-width sprite-width))   
      (rows-needed          (ceiling(/ layer-count sprites-per-row)))

      ; Create new image with appropriate size
      (spritesheet-image (car (gimp-image-new 
                                      spritesheet-width
                                      spritesheet-width 
                                      RGB)
                          )
      )
    )

    ; Copy and move each source layer appropriately
    (let (
        (i 0)
        (y 0)
        (x 0)
      )

      (while (< i layer-count)
        (let (
            ; Copy the layer...
            (new-layer (car(gimp-layer-new-from-drawable 
                                      (aref layer-ids (- layer-count (+ i 1))) 
                                      spritesheet-image)
                        )
            )                     
          )

          ; Insert, position and make it visible
          (gimp-image-insert-layer spritesheet-image new-layer 0 -1)        
          (gimp-layer-translate new-layer x y)
          (gimp-item-set-visible new-layer TRUE)
        )

        ; Update i/x/y for next iteration...
        (set! i (+ i 1))
        (set! x (+ x sprite-width))
        
        (if (>= x spritesheet-width)
            (begin
                (set! x 0)
                (set! y (+ y sprite-height))
            )
        )
      )
    )

    ; Merge layers and rename
    (gimp-image-merge-visible-layers spritesheet-image CLIP-TO-IMAGE)
    (gimp-item-set-name (aref (cadr (gimp-image-get-layers spritesheet-image)) 0) "spritesheet")

    ; Create view for new image
    (gimp-display-new spritesheet-image)
  )
)

(script-fu-register "script-fu-unityspritesheet"
                    "Create Unity spritesheet"
                    "Create a Unity-compatible spritesheet from current image layers"
                    "Ross Bamford"
                    "Copyright (c)2012-2014 Ross Bamford"
                    "2012-2014"
                    "*"
                    SF-IMAGE    "Image"         0
                    SF-OPTION   "Size"          '("4096" "2048" "1024" "512")
                )

(script-fu-menu-register "script-fu-unityspritesheet"
                         "<Image>/Sprites")

(Once again, thanks for the folks at ToHTML.com for the syntax highlighting 🙂 )

Random headache #57 – Rails 4 and Nokogiri on Windows

Facepalm

So I’m currently working on a rails-based internal app for a company I occasionally do some work for (and often, wish I didn’t), and for one reason or another I’m stuck developing on a Windows box. I won’t go into the reasons for this right now (suffice it to say that I blame Broadcom. Or Sony. I’m not sure…) but it is what it is – I’m stuck with Windows on x86-64.

The stack in use here is Rails 4.0 on Ruby 2.0, which is apparently known to be problematic on 64-bit Windows, and it seems there have been some previous problems with Nokogiri added into the mix. I’ve spent a few hours now chasing this one down, and googling around has turned up quite a few people experiencing similar issues. The problem is that I could not get Bundler to use the (already installed) fat binary Nokogiri 1.6.1. No matter what I tried at the (braindead) windows command prompt, it would always download the source gem and try to compile it. Which didn’t work. If you’ve ever tried to get a working libxml2 build going on Windows, you’ll understand why I wasn’t prepared to push it. I just wanted to use the pre-built binary provided by the nice folks at Nokogiri.

After putting up with an annoying develop/test cycle involving pushing every change over to an old (Linux) laptop, running tests, and then coming back to the Windows machine for further coding, I finally found a way to make it work on Windows, thanks to a read through the Gemfile manual. It turns out I can unpack the binary gem into a local path within my app, using the :path option. So I unpacked the binary gem into a new directory, and then modified the app’s Gemfile thusly:

# Workaround windows issue (quelle suprise) with nokogiri on win64
# Note: We don't actually require nokogiri (which is why there's no platform agnostic
# gem line for it) - it's only required by the other dependencies...
#
# Also, since we wouldn't dream of deploying production on a windows box, this is
# a development/test only dependency...
group :development, :test do
  gem 'nokogiri', '1.6.1', path: File.join('.', 'localgems', 'nokogiri-1.6.1.beta.1.mingw.1-x64-mingw32'), platforms: :x64_mingw
end

With this in place, I was finally able to run bundle install all the way to the end. I did run up against one other problem (an execjs runtime error, which pops up because Windows doesn’t support v8, and so therubyracer is incompatible too), which I patched up by following these instructions (this is for execjs 2.02 – YMMV with other versions, obviously). After that, I was able to run my specs, and even start the rails server locally.

Job done! Now I can get back to writing some actual code…

Edit: After sorting this, I did a bundle upgrade and moved to rails 4.0.1. Predictably enough, everything broke again, with all kinds of exceptions about missing timezone info. When the obvious quick-fix didn’t work, I gave up and moved back to 4.0.0. I’ve banged my head against this particular brick wall enough for one day!