Android and the Form Post Download problem…

Only 338 points *and* no POST download???
Only 338 points *and* no POST download???

Another day, another workaround… This time, it was for this rather annoying issue that’s been open for over four years (maybe because it’s in as an RFE and not an actual bug, who knows?). It basically describes an unexpected behaviour in the stock Android browser, whereby it cannot handle form POST requests that result in a file download.

This was picked up by one of the testers on Retroify.me (thanks Natalie!) where we’re using form post to facilitate saving image data from an HTML 5 canvas to the user’s computer. This is something that some browsers can handle without server involvement, but support varies and the various browsers implement filename selection in different ways, so we went with the solution of posting the Base64-encoded image data back to the server, which then initiates a download with disposition: attachment, allowing us to give the file whatever name we like.

This works fine on every browser we’re targeting, except the stock Android browser in everything from Gingerbread to Jelly Bean. Clicking the “Download” button on that browser resulted in “Download unsuccessful” 100% of the time.

This took some tracking down, and after sanity checking the appropriate code to make sure we were returning the right content-type and so forth, I turned to the logs to see if I could see anything out of the ordinary. Sure enough, enlightenment ensued:

Started POST "/download" for 192.168.0.4 at 2013-04-28 20:17:28 +0100
<<Base64 parameters elided>>
Processing by AppController#download as HTML
Rendered text template (0.0ms)
Sent data retroified.me.png (1.5ms)
Completed 200 OK in 3ms (Views: 1.3ms | ActiveRecord: 0.0ms) 

Started GET "/download" for 192.168.0.4 at 2013-04-28 20:17:29 +0100
Processing by AppController#download as HTML
Completed 500 Internal Server Error in 2ms (Views: 1.1ms | ActiveRecord: 0.0ms)

Immediately after sending the POST, Android follows up with a GET to the same endpoint. After some thinking (read: Googling) I decided that this must be the download manager trying to grab the file – the browser simply ignores the first (successful) response, and instead launches the download manager to grab the file via a GET. The problem is, somewhere along the way, the form post data is discarded.

In our particular case even if the post data wasn’t discarded, GET would never work because the Base64-encoded image data is simply too big for a GET request – Our Heroku dyno would simply refuse to touch it. Even if the browser insists on using a download manager in this way, my expected behaviour would be for it to pass along the entire request to it. However, on Android that might mean stuffing a large amount of Base64 into an intent, which could cause other problems.

It’s at this point that I found the aforementioned “RFE” 1780 and realised that this is what I was up against. Given the previous paragraph I can allow that this isn’t as simple a fix as it might first appear, but still – four years and no love?

In the end, I decided the only way forward was simply to work-around the problem by doing the image download a different way on the stock Android browser. Fortunately, I had already implemented server-side image generation for use with the Facebook integration, so all I needed to do was make the download button behave differently in Android stock. Here’s the Javascript that does that:

/* Download image from canvas */
function downloadImage(canvas) {
  var data = canvas.toDataURL("image/png");

  // Stock android browser can't download from post.
  // See: https://code.google.com/p/android/issues/detail?id=1780
  var ua = navigator.userAgent.toLowerCase();
  var isStockAndroid = ua.indexOf("android") > -1 && ua.indexOf("chrome") == -1 && ua.indexOf("dolphin") == -1;

  if(isStockAndroid || data.substr(0,6) == "data:,") {
    // redirect to server-side download
    window.location = getGeneratedPictureURL() + "?d=1&f=retroified.me.png";
  } else {
    post_to_url("/download", { data: data });
  }
}

(Note: This also works around an issue with pre-Gingerbread, where canvas.toDataURL is not actually implemented).

What this actually does is check if the user-agent reports Android, but also that it doesn’t specify any of the browsers that are known to work. I had to go this way since the stock browser’s user agent doesn’t easily allow us to do single-step sniffing. If the stock browser is detected, the form post is skipped, and server-side generation is used instead (getGeneratedPictureURL() returns the server-side URL for the currently-displayed image, and we just add some parameters to force disposition: attachment and set the filename).

This works, but in my opinion it’s not ideal. The user’s browser has already done the work of generating this image, but that load is then pushed back onto the server when the Download button is clicked. This is not a trivial thing – the image generation (using ImageMagick behind the scenes) is a relatively slow operation. Of course we cache the generated images, but with 20M+ possible combinations we don’t keep them cached forever.

All we can hope at this point is that 1780 gets some attention at some point soon. If you’re so inclined, you might even want to head over there and star it to let the AOSP guys know it’s still alive…

It’s been a while!

Wow, looks like I’ve not blogged since February! Doesn’t seem that long, but there you go…

I’ve actually been really busy lately, with work and other things. Mostly I’ve been spending some time getting our new app into beta (check it out at http://retroify.me). It’s been a bit of a learning curve for me (it’s the first time I’ve really used Javascript in anger, and the first FB platform integration I’ve done for anything commercial) but it’s been (and continues to be) lots of fun.

This has also meant that things have fallen behind a bit on Deelang and ORMDroid, but I’ve got plans to get back on track with both projects very soon.

Anyway, I’ll just leave this here for now, with a promise that now the main development of the app is over, normal blogging services will be resumed in the near future 🙂

Savon – SOAP on Ruby

For reasons best left unsaid, I’ve recently had occasion to access BetFair‘s SOAP-based API from a rails app. Although SOAP is widely touted as yesterday’s technology (indeed, BetFair are currently in the process of replacing theirs with a JSON based setup) it’s still widely used in the enterprise space and something I’ve come into contact with a fair bit in the past, though mostly from Java.

Googling around for a decent Ruby SOAP client library lead me to Savon, the “Heavy Metal SOAP client”, which has been “Abstracting the craziness since 2009”.

I was impressed with the API offered by Savon (although less so with that offered by BetFair) and within a very short time was up and running. The library seems stable, fairly speedy and best of all, it’s actively developed right now.

In a more general sense, this kind of thing is what I love about Ruby and it’s wealth of great open-source libraries. In just over a week I’ve gone from initial spec to working webapp thanks in large part to the work of those who’ve gone before me. Good times!

Deelang 0.22 released on GitHub!

So I spent a bit of time today getting together the overdue release of Deelang ready, and finally pushed the 0.22 binary releases to Google Code, with source releases staying on GitHub. The Google Code project has now been fully migrated over to GitHub, and remains only to host binary downloads for those that want to get up and running quickly.

This release of Deelang is largely a maintenance release, with a number of bugfixes to the internals of the Dex compiler, the runtime library and a few other places. It does however include a few new features, notably:

  • Or Blocks are now implemented in the DEX compiler, and are used exactly like as they are in the Dee VM. 
  • Comparison operators are fully implemented.
  • Method chaining on parenthesized expressions works as it always should have!

This release doesn’t yet include the new AST rewriting I previously blogged about (that’s planned for 0.30 at the moment, as it’s a big change) but does address a lot of the issues that will eventually fix in a quick-fix kinda way – mostly as short-term fixups that will go away once the migration to the new AST layout is complete.

 

Deelang & ORMDroid: Moving to GitHub

Image representing GitHub as depicted in Crunc...

After spending all their lives so far on Google Code, I’ve decided it’s time to move both Deelang and ORMDroid over to GitHub. The projects have moved away from the old Subversion VCS and are now using Git, and the actual moving of the wiki pages, issues and everything else is well underway.

Why the move? Well, GitHub makes collaboration much more pleasant (Forks and pull-requests vs. patches isn’t much of a contest), and crucially for an Open Source project, a lot more people are comfortable with Git these days than SVN (in the OSS world, at least). Until fairly recently the whole Git thing had passed me by to an extent, but it’s easy to pick up and I have to say it’s now my preferred VCS.

On top of all that, I just prefer the GitHub interface!

Here are the new links to the projects. Please update your references if you have them.

Jelly Bean on One X – Finally!

HTC One X

After what seems like an eternity since the announcement that HTC would be updating the One X to Jelly Bean, and after a couple of months seeing it happen in other parts of the world, HTC finally got around yesterday to releasing the update to UK users, myself included.

“Been a long time coming” according to Fudzilla, and I’d tend to agree!

This initial update is for phones that aren’t network locked – it will depend now on network when others get the update (obviously, Orange et al. will need time to add their own software, themes and the like) – and you may have to do a manual update check if you didn’t get the update yet. As usual, the update comes in two parts – a small (1.5mb) pre-update, which is then followed by a bigger (~400mb) OS update.

Initial impressions are unsurprising, given that we’ve seen Jelly Bean on many other devices at this point, but it’s definitely a good progression for the One X. Personally I’ve often felt that the UI experience with ICS has always been a little disappointing on the device, given its powerful processor and RAM profile, and this has been addressed to a reasonable extent in the update. It still lags a little in places (orientation changes still aren’t as smooth as I’d like) but it’s definitely better. Coupled with the myriad small UI updates that Jelly Bean brings, this update has made my One X that little bit better – just in time for Christmas!

Had your update yet? Let me know in the comments!

 

Scratch: Teaching kids to think in blocks

The Scratch Cat
The Scratch Cat

Over the weekend I spent a few very rewarding hours with my ten-year-old son playing with Scratch, MIT’s drag-and-drop educational programming system. I’ve often wanted to “get him into” programming (I was a little younger than he is now when I started, and at ten I was beginning to get really interested in writing code on the old 8-bit machines) but haven’t really known where to begin. It turns out the ideal place was Scratch!

Although I’ve been aware of Scratch for a while (it comes pre-installed on the Raspberry Pi I got a few months ago, which I’ve not really done anything with yet) I only really looked at it properly last week, when a friend of mine asked me for a little help with an introduction to CS course he was doing, in which the first problem set was to create something in Scratch (I helped him make a Space Invaders clone). It was while working on that I realised I really should introduce Jake (that’s my son) to Scratch. After I mentioned it to him, it turned out he’d looked at it a little at school, and so it was decided – we’d spend a bit of time on Sunday playing around with it.

Continue reading “Scratch: Teaching kids to think in blocks”

Rewriting Vs. Rewriting

As previously mentioned, I planned to spend yesterday rewriting Deelang‘s parser with the aim of fixing some long-term issues and revisiting some design decisions that get in the way of the DEX compiler. As it turned out, I didn’t have as much time as I wanted to spend on this (as usual!) but I did get some time to play with it, and it turned out that my decision to rewrite much of the parser was, well, somewhat premature.

The main aim of the rewrite was to change the way the AST is generated, to remove the imaginary CHAIN nodes that proliferate the current AST. CHAIN nodes are used as the LHS in field accesses and method calls, as well as one or two other places, and denote that the receiver is the result of the previous operation. They work great when compiling for a stack machine (such as the JVM or the original Dee VM) – they translate simply as “pop the top object from the stack” – but are less intuitive when compiling for a register architecture, such as Dalvik.

Continue reading “Rewriting Vs. Rewriting”

Biting the bullet – time for a rewrite

In response to a feature request from some of the guys here, my current work on Deelang focusses on implementing proper equality and comparison operators as part of the language. The current release has no support for operators beyond basic arithmetic – equality and comparison have traditionally been implemented as methods hacked on top of the standard library. This results in code such as:

1.eql(2)  => true
5.lt(10)  => true
10.gt(20) => false

Leaving aside for a moment that the above isn’t actually that bad (in my opinion at least), implementing this as operators is actually quite simple for both the dex and deevm compilers. However, while putting it together I ran up against an old elephant that’s been sitting quietly in the corner for some time now – the current parser is a mess, and it might just be time to rewrite the grammar.

The problem is not new, but I’ve ignored it for a while. Basically, it boils down to the fact that the current parser cannot handle code such as:

(1+2).foo()

Not only does this not work, but it fails miserably with nothing more specific than a MismatchedSetException at the terminator, after a lot of backtracking and ultimately ignoring the method call completely. The above code parses to the following tree:

Parse Tree Notice all the abandoned trees (in red), before the (erroneous) final parse, and the MismatchSetException up there on the right. The method call gets parsed at one point, but that tree was then abandoned in favour of a tree in which the call is quietly ignored. This is clearly one confused parser, and all over something as simple as (1+2).foo()! Clearly, this needs fixing.

As I say, I’ve ignored this for a while. It should be relatively simple to fix (and indeed it is) with a bit of rejigging in the grammar. However, this problem is actually symptomatic of something deeper – namely, that the Deelang grammar is a mess. In the past, as problems such as this have cropped up, they’ve been fixed by adding to the grammar. New productions, imaginary tokens, and syntactic predicates have all been added to cope with a specific case, with no real wider plan.As long as the tests still passed at the end, the additions stayed.

The result of all this is that things that should be handled in a unified way are actually handled in a variety of ways. My personal favourite example of this is the way chained method calls are handled – I won’t illustrate it here, but if you’re interested just debug something like “foo().bar().baz().quux()” in ANTLRWorks. Trust me, it’s not pretty. It’s inefficient, it’s inconsistent with other parts of the grammar, and it requires the compiler to jump through some pretty awkward hoops to keep track of who is using what target register. It worked well when the only target was the (stack-based) Dee vm, but as requirements have grown it’s become cumbersome – the only reason it still works this way is inertia.

With all this in mind, I’ve decided that now is the ideal time to rewrite the grammar to get rid of these issues. I’ve never been a big believer in planning to throw one away, but in this case it looks like I will, anyhow. To be fair, I’m not planning a complete rewrite – large parts of the grammar are fine as they are (literals, for example). But the meat of it – from atoms through function calls to method calls – will be rewritten in a way that’s more consistent, cleaner, and hopefully requires a lot less backtracking. I’m also aiming to reduce the lookahead where I can, although some of the actual language design makes this quite difficult.

Unfortunately this grand plan must remain just that for now – I’m very short on time to actually work on this at the moment. Since it’s not an actual key requirement I can’t allocate any actual work time to it (even though it will make things easier and save time down the line). So provisionally, I’ve set aside Sunday for the rewrite.

Now I just need to hope nothing more pressing crops up between now and then. Wish me luck!

Tech Filled Fantasy

So, Nokia have advertised the life out of the new Lumia range and the Windows software it will be running, but now the guys over at HTC have released some specs and pictures of their new HTC 8 range.

Im not going to spend all of this article talking about Windows Phone as I’ve already shown my excitement for it previously and wrote a few articles that concern it, as well as the Lumia 920 article, each of these can be found here ..

http://techfilledfantasy.wordpress.com/2012/09/06/nokia-lumia-920-are-the-boots-of-apple-google-shaking/

http://techfilledfantasy.wordpress.com/2012/08/11/270/

These articles will give you a bit of an overview of Windows Phone 8, and what you can expect from it.

The HTC 8X and HTC 8S will both run the Windows OS. Just like the HTC One X & S, you know more or less what to expect from each of these models that each will have its own selling points.

Taking the…

View original post 569 more words

PJ5 TTL CPU

Blog about the TTL based CPU design we're making

Don Charisma

because anything is possible with Charisma

Tech Filled Fantasy

One stop shop for all your tech-filled pleasures!