One-to-many in ORMDroid – at last!

I’m pleased to announce that as of this morning, ORMDroid now has some basic support for one-to-many relationships between entities, thanks to Jacob Ferrero and as hinted at in a previous post. Jacob actually put this together a few months ago, the only hold-up has been me getting around to merging the code in, so it’s great that it’s finally done. It’s not in a release yet (later this week is the plan on that) but is available in Git on the master branch.

Jacob explains the changes better than I can, so here’s a quote directly from his original pull request:

Basic usage is like this:

Person.java

class Person extends Entity {
    public String name;

    @Column(inverse = "person")
    public ArrayList<Hobby> hobbies;
}

Hobby.java

class Hobby extends Entity {
    public String type;
    public Person person;
}

And then you would just use it like normal:

Person person = Entity.query(Person).where(eql("name", "Christ")).execute();

for(Hobby hobby : person.hobbies)
    Log.d(TAG, "Hobby found: " + hobby.type);

It’s worth noting that this change does break the API slightly, the most common case being that if you’ve previously used @Column(inverse = true) then you’ll need to migrate to the new form which names the foreign column.

In the unlikely case that you’ve implemented your own type mappings, you’ll also need to make some changes as the public API has changed slightly (there’s a new parameter, precursors, which if you’re not currently using you can safely ignore in your code) and there have been some minor changes under the hood (e.g. in EntityMapping#load) that you’ll need to update your code for.

All in all, some long-overdue positive movement in ORMDroid. We’re definitely moving toward a new release this week, and I promise to be less tardy in the future!

 

 

Advertisements

Just checking in…

Wow, so my last post was September? That’s just not good blogging, and for that I apologise… There’s been a lot going on, but this is no time for excuses – I’m just posting this to let you all know that I’m still here, still writing code (lately, not as much as I’d like, but there you go) and I’m soon going to be making a start on the huge list of Open Source stuff that’s been building up over the past few months. Here are a few highlights of what’s in store:

  • Some awesome new stuff will be going into ORMDroid very soon, with some great contributions from Jacob Ferrero including one-to-many support. I can only apologise to Jacob for the time it’s taking to get these reviewed and merged!
  • Speaking or ORMDroid, it must be about time for that 1.0 release…
  • I’m planning to finally finish up Deelang. It still needs the new parser architecture sorting out, and some bug-fixes, plus some decent documentation. It’s been sitting there for too long, it needs to be pushed out the door!
  • I have some new code to put into Mink, including some rudimentary user-mode bootstrap code, along with a basic round-robin scheduler and other basic bits that will allow development to continue. A lot of this code is written, but still shockingly untested and even-more-shockingly out of version control!

Outside of Open Source, I still have to do something with retroify.me, a fact I was reminded of the other day when the domain name came up for renewal. The annoying thing there is that it’s done – I just need a reasonably-priced swag supplier who can print up cool T-shirts and the like for people to buy (any suggestions much appreciated!). We’ve also got an exciting new venture on the verge of private beta, which we hope to be launching as a commercial product later in the year.

So, all in all, exciting times. Now I just have to actually get on and do all this stuff.

Watch this space.

 

The “Scoring Predictions” kata

Official Ruby logo
Official Ruby logo (Photo credit: Wikipedia)

While idly surfing around earlier, I caught this post on coderbits, announcing that some of the Practicing Ruby archives are available under a free documentation license. When I saw that one of the released articles was written by James Edward Gray II, I just had to take a look.

In case you don’t know, JEGII is a “name” in the Ruby world, and for several years he ran the weekly ruby quiz on ruby-talk. At the time I was quite active on the list, and used to really look forward to the quiz as it always provided an interesting diversion from whatever I was working on that week. I even helped out by setting the occasional quiz myself (and that one won me a signed copy of James’ book).

Anyway, his article for Practicing Ruby was a great read, and proposed a problem that I’ve not specifically come across before. Just reading the intro re-awakened the old ruby-quizzer in me, so I had to have a go. Before I read any further than the problem definition, I wrote up some code. Here’s my first version:

def score(guesses, winners, scores = [15, 10, 5, 3, 1])
  guesses.zip(winners).inject([0, scores]) do |cscores, pair|
    guess, winner, cur_score, scores_ary = *pair, *cscores
    
    cur_score += if guess.eql? winner
      scores_ary.first
    elsif winners.include? guess
      1
    end || 0

    [cur_score, scores_ary[1..-1]]
  end.first
end

This works, but it feels funky in places. Using an array to pass multiple variables through the inject loop, for example, isn’t good coding. I don’t write anywhere near as much Ruby as I once did so I tend to forget things. Sure enough, reading the rest of the article reminded me that zip can take more than one argument!

This allowed me to refine my code a bit:

def score(guesses, winners, scores = [15, 10, 5, 3, 1])
  guesses.zip(winners, scores).inject(0) do |cur_score, (guess, winner, score)|
    cur_score + if guess.eql? winner
      score
    elsif winners.include? guess
      1
    end || 0
  end
end

This is also using another little trick I’d forgotten about (Mathias Meyer had a reminder for me in this post) – tuple arguments. Inject passes in the current score, plus the current three-element array from the zipped array. By surrounding the three argument names with parenthesis, Ruby splats the array out into three variables automatically for me. Sweet!

This fun little exercise reminded why I love Ruby so much. It’s the little things: zip and inject, ‘if’ as an expression (notice the ‘end || 0’ line at the end of the if block), tuple arguments and so much more.

Right now my day job project is a rails-based proprietary webapp, where I’m obviously writing Ruby, but somehow it’s a lot less fun than little snippets like this. My new resolution is to spend a little time every week seeking out these little problems – if Ruby Quiz no longer comes to me, then I must go to it!

 

ORMDroid 0.40 released

ORMDroid 0.40 is now available from the releases area on Github. This is the first packaged release since 0.20 almost eleven months ago, and incorporates new features, reworked old features, and bugfixes. There’s a full announcement at the link above.

As well as packaging this release, we’ve now also closed the (legacy) Google Code site for ORMDroid – we’re now 100% Github!

This release marks the last call for major new features for the 1.0 series. If you’ve got any burning issues you’d like us to look at, discuss in the comments or post an issue on Github and we’ll take a look.

Enjoy!

Creating a bootable hard-disk image with Grub2

Hard Disk Platter
Image courtesy of Wikimedia Commons.

Creating a Qemu hard-disk image that boots with Grub2 is, as it turns out, not too difficult. Unfortunately though, the steps you need to get it working are not completely obvious, and documentation is still pretty sparse (probably because Grub2 is still quite new). In this post I’ll run through how I got it working, and how I integrated it into my build.

I mentioned before how I had decided to start playing with OS development with Mink (which is now on Github, by the way). I’ve got a very simple kernel that does almost exactly nothing, and obviously I need to boot it up so I can test it as I add things. The obvious (and accepted) way to do this is to use some kind of virtual machine software to emulate the target computer (in my case, at the moment, that’s an x86 PC). I’m using Qemu.

Originally, I was using grub-mkrescue to create CDROM ISO images, and this worked fine for a while. Now, though, I’m getting to the point where I want to start implementing a disk driver, and filesystem support. To get started, I just want to implement simple IDE (i.e. PATA) support in the kernel – I don’t want to do ATAPI (which I’d need for the CDROM based setup) and I really don’t want to push off with an ISO9660 filesystem driver. I want read-write capability, so I’ve chosen to switch from the ISO build to a hard-disk image file, on which I’m using an ext2 filesystem.

So I spent much of today trying to get it to work. Googling the subject turned up a few links, but sadly nothing worked immediately, possibly because Grub2 is still being developed and has changed since the various tutorials were written. In the end I hacked up a solution based on bits from many different places, using the loopback device to mount the disk image, and then grub-install to actually install the bootloader. The configuration file I’d already put together for the ISO boot still works fine, and the final script I came up with is so short and simple, you’d never guess it took me the best part of a day to write!

Firstly, here’s mkimage.sh, which is the script that does the actual image creation:

# Create the actual disk image - 20MB
dd if=/dev/zero of=mink.img count=20 bs=1048576

# Make the partition table, partition and set it bootable.
parted --script mink.img mklabel msdos mkpart p ext2 1 20 set 1 boot on

# Map the partitions from the image file
kpartx -a mink.img

# sleep a sec, wait for kpartx to create the device nodes
sleep 1

# Make an ext2 filesystem on the first partition.
mkfs.ext2 /dev/mapper/loop0p1

# Make the mount-point
mkdir -p build/tmp/p1

# Mount the filesystem via loopback
mount /dev/mapper/loop0p1 build/tmp/p1

# Copy in the files from the staging directory
cp -r build/img/* build/tmp/p1

# Create a device map for grub
echo "(hd0) /dev/loop0" > build/tmp/device.map

# Use grub2-install to actually install Grub. The options are:
#   * No floppy polling.
#   * Use the device map we generated in the previous step.
#   * Include the basic set of modules we need in the Grub image.
#   * Install grub into the filesystem at our loopback mountpoint.
#   * Install the MBR to the loopback device itself.
grub2-install --no-floppy                                                      \
              --grub-mkdevicemap=build/tmp/device.map                          \
              --modules="biosdisk part_msdos ext2 configfile normal multiboot" \
              --root-directory=build/tmp/p1                                    \
              /dev/loop0

# Unmount the loopback
umount build/tmp/p1

# Unmap the image
kpartx -d mink.img

# hack to make everything owned by the original user, since it will currently be
# owned by root...
LOGNAME=`who am i | awk '{print $1}'`
LOGGROUP=`groups $LOGNAME | awk '{print $3}'`
chown $LOGNAME:$LOGGROUP -R build mink.img

Take away the comments, and that’s 15 lines… Just under two lines per hour – good thing I don’t work for IBM!

The comments pretty well explain what’s going on line-by-line, so I won’t go into much more depth here. The only things to note are that I use Fedora – on other system’s grub2-install is just called grub-install – and that on most systems, this will need to be run as root, so I added a small hack at the end to set the file ownership back to the original user once the image is created.

To integrate this into the build, I simply call it from the Makefile (I’ve included the older ISO build target here as well so you can see how it all works together):

# Generates the image staging area under build/img. This is
# the directory layout that is used to make the ISO or hard-disk
# images for emulators.
.PHONY: image-staging
image-staging: mink.bin grub2/grub.cfg
	$(MKDIR) build/img/boot/grub2
	$(CP) --parents $^ build/img/boot

# Generates the Grub2-based ISO - see bochsrc.txt for info.
# Requires grub2-mkrescue and xorriso be installed!
mink.iso: image-staging
	grub2-mkrescue -o mink.iso build/img# Generates a hard-disk image. See mkimage.sh.
mink.img: image-staging
	sudo sh mkimage.sh

(Thanks to tohtml.com for the syntax highlighting).

This should be pretty self-explanatory, so I won’t bore you with further details.

So there you have it – an easy way to make a Grub2-bootable hard-disk image as part of your build process. Of course you can do it manually by booting your virtual machine from a Linux live CD and installing grub to the hard drive, but if you want to automate it, feel free to use this (it’s under the MIT license, so please give credit where it’s due – see here).

Finally, in the event that Grub changes as it continues to evolve, please let me know in the comments and I’ll try to keep this post up-to-date.

Reasons why my code is late #173: The Weather

For the UK, this is damn fine weather...
For the UK, this is damn fine weather. And the “Partly cloudy” just isn’t true – clear blue skies!

I currently have quite a lot of work outstanding on various Open Source projects, and I’ve been really busy lately what with one thing and another, but I’ve deliberately kept today free so I could catch up with things a bit. There’s a lot of (long overdue) work on Deelang (the current milestone is at least a couple of months overdue), and ORMDroid has had some great patches this week including private field mapping (thanks to Germán Valencia and Machinarius), some fixes to the SQL generation, and fixes for some spurious exceptions with unknown types (this one also adds a mapping for java.util.Date, and removes the old default mapping functionality – I’ll post more about the API changes soon). All this is in Git, but needs packaging up into a release.

There’s also some stuff I wanted to clean up in Rote, and I’ve lately been working on an operating system kernel (for learning purposes of course – I think at some point every programmer decides it’s time to write one) and was quite excited about getting some time to work on the scheduler this weekend.

But sadly, all of this is going to have to wait. It’ll be late. The reason? Well, here in the UK, we have a heatwave – and that’s a rare-enough thing that when it does happen, you just have to take advantage of it!

I guess it’s a British thing. Some would say we’re lucky – we don’t get any kind of extreme weather. Hurricanes, cyclones, all that stuff is something you see on TV. But what we do get lots of is mediocre weather. We spend much of our time keeping out of the rain. For the last two or three years, summer has been a washout. Most of the time, in the UK, the Sun is a (sort-of) newspaper, not something that makes the outdoors hot.

So I’m sorry guys, but all those updates are going to be a little while longer. This weather won’t last, so I have to make the most of it while I can. Yesterday I sat in the garden, drank a few beers, fell asleep and got sunburned. I’ve been out there all morning and guess what? That’s right – as soon as I hit “post” – I’m going back for more of the same.

As for the updates – we’ll have to take a raincheck.

And for your next project…

We’ve all been there – you’re at a bit of a loose end, you want to write some cool software, but try as you might you just can’t get inspired. There’s that vague idea you’ve been toying with for months, you know – the one that involves dependency injection, HTML Canvas and/or Haxe – but you’re not ready to make a start on that yet. You want something new.

If this sounds familiar, check this out (Warning: Maybe NSFW, depending on your employers). Boom – all the inspiration you could ever possibly want!

That’s all for now, I’m off to start work on my new Pascal wrapper for lightweight Lingo Blackberry applications that utilises the Singleton Pattern.

Android and the Form Post Download problem…

Only 338 points *and* no POST download???
Only 338 points *and* no POST download???

Another day, another workaround… This time, it was for this rather annoying issue that’s been open for over four years (maybe because it’s in as an RFE and not an actual bug, who knows?). It basically describes an unexpected behaviour in the stock Android browser, whereby it cannot handle form POST requests that result in a file download.

This was picked up by one of the testers on Retroify.me (thanks Natalie!) where we’re using form post to facilitate saving image data from an HTML 5 canvas to the user’s computer. This is something that some browsers can handle without server involvement, but support varies and the various browsers implement filename selection in different ways, so we went with the solution of posting the Base64-encoded image data back to the server, which then initiates a download with disposition: attachment, allowing us to give the file whatever name we like.

This works fine on every browser we’re targeting, except the stock Android browser in everything from Gingerbread to Jelly Bean. Clicking the “Download” button on that browser resulted in “Download unsuccessful” 100% of the time.

This took some tracking down, and after sanity checking the appropriate code to make sure we were returning the right content-type and so forth, I turned to the logs to see if I could see anything out of the ordinary. Sure enough, enlightenment ensued:

Started POST "/download" for 192.168.0.4 at 2013-04-28 20:17:28 +0100
<<Base64 parameters elided>>
Processing by AppController#download as HTML
Rendered text template (0.0ms)
Sent data retroified.me.png (1.5ms)
Completed 200 OK in 3ms (Views: 1.3ms | ActiveRecord: 0.0ms) 

Started GET "/download" for 192.168.0.4 at 2013-04-28 20:17:29 +0100
Processing by AppController#download as HTML
Completed 500 Internal Server Error in 2ms (Views: 1.1ms | ActiveRecord: 0.0ms)

Immediately after sending the POST, Android follows up with a GET to the same endpoint. After some thinking (read: Googling) I decided that this must be the download manager trying to grab the file – the browser simply ignores the first (successful) response, and instead launches the download manager to grab the file via a GET. The problem is, somewhere along the way, the form post data is discarded.

In our particular case even if the post data wasn’t discarded, GET would never work because the Base64-encoded image data is simply too big for a GET request – Our Heroku dyno would simply refuse to touch it. Even if the browser insists on using a download manager in this way, my expected behaviour would be for it to pass along the entire request to it. However, on Android that might mean stuffing a large amount of Base64 into an intent, which could cause other problems.

It’s at this point that I found the aforementioned “RFE” 1780 and realised that this is what I was up against. Given the previous paragraph I can allow that this isn’t as simple a fix as it might first appear, but still – four years and no love?

In the end, I decided the only way forward was simply to work-around the problem by doing the image download a different way on the stock Android browser. Fortunately, I had already implemented server-side image generation for use with the Facebook integration, so all I needed to do was make the download button behave differently in Android stock. Here’s the Javascript that does that:

/* Download image from canvas */
function downloadImage(canvas) {
  var data = canvas.toDataURL("image/png");

  // Stock android browser can't download from post.
  // See: https://code.google.com/p/android/issues/detail?id=1780
  var ua = navigator.userAgent.toLowerCase();
  var isStockAndroid = ua.indexOf("android") > -1 && ua.indexOf("chrome") == -1 && ua.indexOf("dolphin") == -1;

  if(isStockAndroid || data.substr(0,6) == "data:,") {
    // redirect to server-side download
    window.location = getGeneratedPictureURL() + "?d=1&f=retroified.me.png";
  } else {
    post_to_url("/download", { data: data });
  }
}

(Note: This also works around an issue with pre-Gingerbread, where canvas.toDataURL is not actually implemented).

What this actually does is check if the user-agent reports Android, but also that it doesn’t specify any of the browsers that are known to work. I had to go this way since the stock browser’s user agent doesn’t easily allow us to do single-step sniffing. If the stock browser is detected, the form post is skipped, and server-side generation is used instead (getGeneratedPictureURL() returns the server-side URL for the currently-displayed image, and we just add some parameters to force disposition: attachment and set the filename).

This works, but in my opinion it’s not ideal. The user’s browser has already done the work of generating this image, but that load is then pushed back onto the server when the Download button is clicked. This is not a trivial thing – the image generation (using ImageMagick behind the scenes) is a relatively slow operation. Of course we cache the generated images, but with 20M+ possible combinations we don’t keep them cached forever.

All we can hope at this point is that 1780 gets some attention at some point soon. If you’re so inclined, you might even want to head over there and star it to let the AOSP guys know it’s still alive…

It’s been a while!

Wow, looks like I’ve not blogged since February! Doesn’t seem that long, but there you go…

I’ve actually been really busy lately, with work and other things. Mostly I’ve been spending some time getting our new app into beta (check it out at http://retroify.me). It’s been a bit of a learning curve for me (it’s the first time I’ve really used Javascript in anger, and the first FB platform integration I’ve done for anything commercial) but it’s been (and continues to be) lots of fun.

This has also meant that things have fallen behind a bit on Deelang and ORMDroid, but I’ve got plans to get back on track with both projects very soon.

Anyway, I’ll just leave this here for now, with a promise that now the main development of the app is over, normal blogging services will be resumed in the near future 🙂

Savon – SOAP on Ruby

For reasons best left unsaid, I’ve recently had occasion to access BetFair‘s SOAP-based API from a rails app. Although SOAP is widely touted as yesterday’s technology (indeed, BetFair are currently in the process of replacing theirs with a JSON based setup) it’s still widely used in the enterprise space and something I’ve come into contact with a fair bit in the past, though mostly from Java.

Googling around for a decent Ruby SOAP client library lead me to Savon, the “Heavy Metal SOAP client”, which has been “Abstracting the craziness since 2009”.

I was impressed with the API offered by Savon (although less so with that offered by BetFair) and within a very short time was up and running. The library seems stable, fairly speedy and best of all, it’s actively developed right now.

In a more general sense, this kind of thing is what I love about Ruby and it’s wealth of great open-source libraries. In just over a week I’ve gone from initial spec to working webapp thanks in large part to the work of those who’ve gone before me. Good times!