Random headache #57 – Rails 4 and Nokogiri on Windows

Facepalm

So I’m currently working on a rails-based internal app for a company I occasionally do some work for (and often, wish I didn’t), and for one reason or another I’m stuck developing on a Windows box. I won’t go into the reasons for this right now (suffice it to say that I blame Broadcom. Or Sony. I’m not sure…) but it is what it is – I’m stuck with Windows on x86-64.

The stack in use here is Rails 4.0 on Ruby 2.0, which is apparently known to be problematic on 64-bit Windows, and it seems there have been some previous problems with Nokogiri added into the mix. I’ve spent a few hours now chasing this one down, and googling around has turned up quite a few people experiencing similar issues. The problem is that I could not get Bundler to use the (already installed) fat binary Nokogiri 1.6.1. No matter what I tried at the (braindead) windows command prompt, it would always download the source gem and try to compile it. Which didn’t work. If you’ve ever tried to get a working libxml2 build going on Windows, you’ll understand why I wasn’t prepared to push it. I just wanted to use the pre-built binary provided by the nice folks at Nokogiri.

After putting up with an annoying develop/test cycle involving pushing every change over to an old (Linux) laptop, running tests, and then coming back to the Windows machine for further coding, I finally found a way to make it work on Windows, thanks to a read through the Gemfile manual. It turns out I can unpack the binary gem into a local path within my app, using the :path option. So I unpacked the binary gem into a new directory, and then modified the app’s Gemfile thusly:

# Workaround windows issue (quelle suprise) with nokogiri on win64
# Note: We don't actually require nokogiri (which is why there's no platform agnostic
# gem line for it) - it's only required by the other dependencies...
#
# Also, since we wouldn't dream of deploying production on a windows box, this is
# a development/test only dependency...
group :development, :test do
  gem 'nokogiri', '1.6.1', path: File.join('.', 'localgems', 'nokogiri-1.6.1.beta.1.mingw.1-x64-mingw32'), platforms: :x64_mingw
end

With this in place, I was finally able to run bundle install all the way to the end. I did run up against one other problem (an execjs runtime error, which pops up because Windows doesn’t support v8, and so therubyracer is incompatible too), which I patched up by following these instructions (this is for execjs 2.02 – YMMV with other versions, obviously). After that, I was able to run my specs, and even start the rails server locally.

Job done! Now I can get back to writing some actual code…

Edit: After sorting this, I did a bundle upgrade and moved to rails 4.0.1. Predictably enough, everything broke again, with all kinds of exceptions about missing timezone info. When the obvious quick-fix didn’t work, I gave up and moved back to 4.0.0. I’ve banged my head against this particular brick wall enough for one day!

Advertisements

Android NDK + Eclipse + Windows = Headache!

Whew! Just spent an “interesting” hour or two trying to get some native android code working under Eclipse on Windows… Although this isn’t the first time I’ve used the NDK, it’s the first time I’ve tried to get it working with Eclipse on the Windows (+Cygwin) platform. To borrow an unrelated phrase from elsewhere, it’s been emotional – I’m guessing the problems I’ve had are a combination of my lack of knowledge of Windows, and Windows lack of knowledge of, well, anything really.

No flames necessary, by the way. My position on Windows as a development platform are well known around these parts, and I don’t think there’s really any need to go into all that again. But Windows is what I have to hand at the moment, so it’s what I must work with.

The background is that I had an existing Android project that had some cross-compiled code (actually, the Quake III engine) that required a bit of work. The engine itself is a separate (non-Android) project, so I started by grabbing the source from subversion, and did the cross compile using the latest NDK (r8b). This went without a hitch once I’d updated a few things in the Makefile to adjust for the local setup.

Cool“, thought I. “Now to get going on the Android side of things, where I actually need to make changes”.

The way this project is set up is fairly convoluted, but it works. The android side of things has a bit of C that exposes the game engine to Java via JNI. The engine itself is (as mentioned above) cross-compiled separately, and there’s a simple Ant build that takes care of copying in the required shared libraries during the build.

To cut a long story short, it didn’t work. I checked the project out, and Eclipse couldn’t resolve the platform includes. I checked it out as a new (basic Android) project. Still no includes. I checked it out as a new C++ project, and manually added the appropriate nature and settings for an ADT project (Yes, I was getting desperate). Predictably enough, still no dice. I deleted the whole thing. I checked it out from a terminal and fiddled with various things. I shouted a bit. I went away and watched TV for a while (“A touch of cloth“, a new spoof detective show on Sky 1, by the way, is hilarious!). I googled incessantly. I checked paths, and re-checked them. Still, Eclipse couldn’t find the includes.

The maddening thing in all this was that the ndk-build command was working fine, and building the JNI glue without any problems. Even during the Eclipse build, it was fine. But I couldn’t actually deploy the app from within Eclipse because the IDE itself couldn’t see the includes, and so wouldn’t let me launch the app because it contained “errors”. Googling had turned up quite a few people with similar problems, but none of the proffered solutions worked for me. The include paths in the Eclipse C/C++ build properties were obviously correct, but still, no includes. I added more paths. I changed what I could. I created a brand-new project with a basic JNI test as a sanity check. And guess what? That’s right. No includes.

Eventually, I decided that there must be some kind of problem with the Android tool chain provided by the ADT. In every case so far I’d gone the “right-click -> Android Tools -> Add native support” route, so I decided to get back to basics and do things manually. For anyone else who encounters similar problems, these are the steps I took:

  • I closed the project in eclipse, and deleted .cproject from a terminal.
  • I re-opened the project in the C/C++ perspective.
  • I went to File->New->Convert to a C/C++ project (Adds C/C++ Nature).
  • I selected “Makefile project“, and “–Other toolchain–” (obviously I tried the “Android GCC” option but ended up with the same problems).
  • I then fired up the Project Properties, and went into the Builders page. Here, I unchecked “Use default build command” and entered the following instead: bash -c 'c:/opt/android-ndk-r8b/ndk-build':Eclipse Builders Properties
  • I then switched to the “Behaviour” tab in this same dialog, and removed the ‘all’ argument to the incremental build command, and completely disabled the “clean” behaviour since I didn’t need it when using ndk-build manually:Eclipse Builders "Behaviour" dialog
  • Finally, I manually entered the include path into the “C/C++ General/Paths and Symbols” settings. In my case, the path I added was “NDK_HOME\platforms\android-14\arch-arm\usr\include“. I made sure that both the “Add to all configurations” and “Add to all languages” checkboxes were checked (I’m on Eclipse Juno, these checkboxes may not show up in earlier versions). So that the include paths ended up looking like:Eclipse include paths dialog
  • I applied these settings, and okayed the dialog.
  • Finally, for good measure, I closed the project, re-opened it, and ran a clean followed by a build, and all was good!

So that was it. The includes were all showing up (I actually had to prove this to myself by Ctrl+Clicking on an #include <jni.h> line!), and I could happily deploy and run the project from Eclipse. ndk-build was running (as it always had) but there were no more complaints from Eclipse. Needless to say, I’m happy. I can now get on with the actual work I set out to do in the first place.

I’m still not convinced this is a bug, so I haven’t filed any reports anywhere. But when I get chance to run down exactly what’s going on here in more depth I will either realise where I went wrong in the first place, or I’ll file bugs appropriately.

Final note: If you’re having similar problems, and decide to follow the steps I took, keep in mind that depending on your project you may need additional include paths, to ensure Eclipse can see the platform-specific headers from the NDK. In my case this wasn’t necessary, but if it is for you then just take a look at the standard paths in an NDK project, and copy them across as necessary.