Curb FFI – FFI port of Curb

I believe I mentioned that I’m back at work on the Curb project. In case you missed that, Curb is a ruby extension that provides bindings to the libcurl library. I started it way back, left to concentrate on other things (having kids, making money, etc) and now I’m involved again, working with Todd Fisher who took over maintaining the project when I bowed out. I’ll be fixing bugs, answering questions and generally helping out.

In the meantime, I’ve also started work on porting (if that’s the right word) Curb to FFI, with a view to moving away from the existing C code. The motivations are manifold:

  • As it stands, Curb is pretty much tied to MRI. In the modern Ruby world, where you’ve got JRuby and Rubinius and who-knows-what-next, this is recognised as a bad thing.
  • It’s a nightmare to get it working on Windows. This is because, and I can speak with some authority here as someone who develops on Windows every day, Windows sucks for development. Unless you’re using all-Microsoft tooling, in which case it’s pretty awesome. But for interoperability with portable code, and libraries targeted primarily at other platforms, it sucks.
  • FFI is probably the right way to do these things these days. 10+ years ago, when Curb was first hacked together in about six hours, C extensions were the shizz. Now, not so much. Unless you really need the level of hardware hackery and performance you can get with C, things are better off in Ruby code.

So to sum up, this port is about future-proofing Curb, making it easier to develop, easier to use cross-platform, and (in the long run) safer, probably more performant, and ensuring it can run on all Rubies, including whatever whizz-bang next-gen thing comes out next week (my bet is it’ll be written in Rust. Or Go. Or something…).

Check it out (or clone it, as the cool kids say nowadays) at https://github.com/roscopeco/curb/tree/ffi.

Advertisements

Building Curb on Windows

man with laptop over his head
At first I was like…

As I mentioned in my previous post, I’m now working on Curb again, and currently on Windows. Back in 2006 when I wrote the original Curb, I wouldn’t go near a Windows box for religious reasons, and the project has never officially supported the platform. There are plenty of bugs and posts around about people having mixed success with getting it to work.

Since then, my views have mellowed, partly I think because I’ve come to appreciate that there’s no room for religion in software engineering – you use the right tool for the job. Partly that, and partly by my desire to, you know, get paid and own stuff and whatnot. These days I do as much work on Windows as I do on *nix, and my bank balance is all the better for it. So anyway, back to the point of this post.

Building Curb on Windows is a bit of a nightmare. It took me a fair while to figure it out, and I wrote the damn thing. If it got me feeling like the guy in the picture above, then how must the everyday user trying to get the gem going feel? If they’re just trying to install it and write some code, I’m guessing they immediately give up and go with another library. But what if they are trying to use something else that depends on the Curb gem? I’m guessing that’s when they get so frustrated they file shouty bug reports with the project. And I don’t blame them. It would have been nice to have someone to shout at while I was trying to get it working.

In the end, I did get it going, but it required me to build libcurl from source. I posted an answer to the bug report I mentioned above with the how-to of it, and the poor person who filed it reported back that it worked. You could click through, but I’ll reproduce the steps below to save your finger.

Most likely, your compiled libcurl is incompatible with the ruby devkit compiler. You can verify this by running (this is on my machine, obviously change your paths):


> c:\Ruby200Devkit\mingw\bin\gcc.exe -L. -lcurl
# c:/[...]/ld.exe: skipping incompatible ./libcurl.dll when searching for -lcurl
# c:/[...]/ld.exe: skipping incompatible ./libcurl.dll when searching for -lcurl
# c:/[...]/bin/ld.exe: cannot find -lcurl
# collect2.exe: error: ld returned 1 exit status

Notice the lines about “Skipping incompatible” before the final “cannot find lcurl”. Try grabbing the curl source from https://curl.haxx.se/download/curl-7.54.0.tar.gz , extract it somewhere, and then open up a terminal and try the following:


> c:\path-to-your-ruby-devkit\devkitvars.sh
> which gcc
# important - verify here that the gcc is the one from your ruby devkit!
> cd 
> sh configure --with-winssl     # and whatever other options you require
> make
> make install

It’s very important that you don’t have any other mingw/msys/cygwin compilers in your path, and that you do this in a windows terminal, not msys/cygwin sh. Otherwise, you’ll probably just build another incompatible library.

You should now have libcurl-4.dll in c:\path-to-your-ruby-devkit\local\bin. Now it gets a bit messy, so hang in there. This part is probably a bug in the build, but easy to work around.


> cd c:\path-to-your-ruby-devkit\local\bin
> copy libcurl-4.dll libcurl.dll

The build needs libcurl.dll (it won’t compile with libcurl-4.dll, but the extension won’t run without the -4 suffix, so we’ll have to manually copy the dll later on).


> gem install curb -- --with-curl-lib=c:\path-to-your-ruby-devkit\local\bin --with-curl-include=c:\path-to-your-ruby-devkit\local\include

Assuming the build goes well (it should, though it might take a while), you now just need to copy the curl dll to the appropriate place:


> copy c:\path-to-your-ruby-devkit\local\bin\libcurl-4.dll C:\Ruby22-x64\lib\ruby\gems\2.2.0\gems\curb-0.9.3\ext

Now run irb -rcurl and, all being well, you should be good to go.

Random headache #57 – Rails 4 and Nokogiri on Windows

Facepalm

So I’m currently working on a rails-based internal app for a company I occasionally do some work for (and often, wish I didn’t), and for one reason or another I’m stuck developing on a Windows box. I won’t go into the reasons for this right now (suffice it to say that I blame Broadcom. Or Sony. I’m not sure…) but it is what it is – I’m stuck with Windows on x86-64.

The stack in use here is Rails 4.0 on Ruby 2.0, which is apparently known to be problematic on 64-bit Windows, and it seems there have been some previous problems with Nokogiri added into the mix. I’ve spent a few hours now chasing this one down, and googling around has turned up quite a few people experiencing similar issues. The problem is that I could not get Bundler to use the (already installed) fat binary Nokogiri 1.6.1. No matter what I tried at the (braindead) windows command prompt, it would always download the source gem and try to compile it. Which didn’t work. If you’ve ever tried to get a working libxml2 build going on Windows, you’ll understand why I wasn’t prepared to push it. I just wanted to use the pre-built binary provided by the nice folks at Nokogiri.

After putting up with an annoying develop/test cycle involving pushing every change over to an old (Linux) laptop, running tests, and then coming back to the Windows machine for further coding, I finally found a way to make it work on Windows, thanks to a read through the Gemfile manual. It turns out I can unpack the binary gem into a local path within my app, using the :path option. So I unpacked the binary gem into a new directory, and then modified the app’s Gemfile thusly:

# Workaround windows issue (quelle suprise) with nokogiri on win64
# Note: We don't actually require nokogiri (which is why there's no platform agnostic
# gem line for it) - it's only required by the other dependencies...
#
# Also, since we wouldn't dream of deploying production on a windows box, this is
# a development/test only dependency...
group :development, :test do
  gem 'nokogiri', '1.6.1', path: File.join('.', 'localgems', 'nokogiri-1.6.1.beta.1.mingw.1-x64-mingw32'), platforms: :x64_mingw
end

With this in place, I was finally able to run bundle install all the way to the end. I did run up against one other problem (an execjs runtime error, which pops up because Windows doesn’t support v8, and so therubyracer is incompatible too), which I patched up by following these instructions (this is for execjs 2.02 – YMMV with other versions, obviously). After that, I was able to run my specs, and even start the rails server locally.

Job done! Now I can get back to writing some actual code…

Edit: After sorting this, I did a bundle upgrade and moved to rails 4.0.1. Predictably enough, everything broke again, with all kinds of exceptions about missing timezone info. When the obvious quick-fix didn’t work, I gave up and moved back to 4.0.0. I’ve banged my head against this particular brick wall enough for one day!

The “Scoring Predictions” kata

Official Ruby logo
Official Ruby logo (Photo credit: Wikipedia)

While idly surfing around earlier, I caught this post on coderbits, announcing that some of the Practicing Ruby archives are available under a free documentation license. When I saw that one of the released articles was written by James Edward Gray II, I just had to take a look.

In case you don’t know, JEGII is a “name” in the Ruby world, and for several years he ran the weekly ruby quiz on ruby-talk. At the time I was quite active on the list, and used to really look forward to the quiz as it always provided an interesting diversion from whatever I was working on that week. I even helped out by setting the occasional quiz myself (and that one won me a signed copy of James’ book).

Anyway, his article for Practicing Ruby was a great read, and proposed a problem that I’ve not specifically come across before. Just reading the intro re-awakened the old ruby-quizzer in me, so I had to have a go. Before I read any further than the problem definition, I wrote up some code. Here’s my first version:

def score(guesses, winners, scores = [15, 10, 5, 3, 1])
  guesses.zip(winners).inject([0, scores]) do |cscores, pair|
    guess, winner, cur_score, scores_ary = *pair, *cscores
    
    cur_score += if guess.eql? winner
      scores_ary.first
    elsif winners.include? guess
      1
    end || 0

    [cur_score, scores_ary[1..-1]]
  end.first
end

This works, but it feels funky in places. Using an array to pass multiple variables through the inject loop, for example, isn’t good coding. I don’t write anywhere near as much Ruby as I once did so I tend to forget things. Sure enough, reading the rest of the article reminded me that zip can take more than one argument!

This allowed me to refine my code a bit:

def score(guesses, winners, scores = [15, 10, 5, 3, 1])
  guesses.zip(winners, scores).inject(0) do |cur_score, (guess, winner, score)|
    cur_score + if guess.eql? winner
      score
    elsif winners.include? guess
      1
    end || 0
  end
end

This is also using another little trick I’d forgotten about (Mathias Meyer had a reminder for me in this post) – tuple arguments. Inject passes in the current score, plus the current three-element array from the zipped array. By surrounding the three argument names with parenthesis, Ruby splats the array out into three variables automatically for me. Sweet!

This fun little exercise reminded why I love Ruby so much. It’s the little things: zip and inject, ‘if’ as an expression (notice the ‘end || 0’ line at the end of the if block), tuple arguments and so much more.

Right now my day job project is a rails-based proprietary webapp, where I’m obviously writing Ruby, but somehow it’s a lot less fun than little snippets like this. My new resolution is to spend a little time every week seeking out these little problems – if Ruby Quiz no longer comes to me, then I must go to it!

 

Savon – SOAP on Ruby

For reasons best left unsaid, I’ve recently had occasion to access BetFair‘s SOAP-based API from a rails app. Although SOAP is widely touted as yesterday’s technology (indeed, BetFair are currently in the process of replacing theirs with a JSON based setup) it’s still widely used in the enterprise space and something I’ve come into contact with a fair bit in the past, though mostly from Java.

Googling around for a decent Ruby SOAP client library lead me to Savon, the “Heavy Metal SOAP client”, which has been “Abstracting the craziness since 2009”.

I was impressed with the API offered by Savon (although less so with that offered by BetFair) and within a very short time was up and running. The library seems stable, fairly speedy and best of all, it’s actively developed right now.

In a more general sense, this kind of thing is what I love about Ruby and it’s wealth of great open-source libraries. In just over a week I’ve gone from initial spec to working webapp thanks in large part to the work of those who’ve gone before me. Good times!