Installing Java 1.4.x on Snow Leopard

OneSwarm’s wiki (link) covers some easy steps for installing Java 1.4 on Snow Leopard. I needed it to get the MPowerPlayer SDK running so I could compile some code for Java-enabled mobile phones. Thanks to wheremydogs.at for pointing out that MPowerPlayer relies on some Java 1.4 classes.

cd /tmp/
wget http://www.cs.washington.edu/homes/isdal/snow_leopard_workaround/java.1.4.2-leopard.tar.gz
tar -xvzf java.1.4.2-leopard.tar.gz
sudo mv 1.4.2 /System/Library/Frameworks/JavaVM.framework/Versions/1.4.2-leopard
cd /System/Library/Frameworks/JavaVM.framework/Versions/
sudo ln -s 1.4.2-leopard 1.4
open "/Applications/Utilities/Java Preferences.app"

For those of you also working with MPowerPlayer, the error I was getting is:

  $ ant
  Buildfile: build.xml
  
  compile:
  
  preverify:
       [exec] Error preverifying class java.lang.Class
       [exec]     VERIFIER ERROR java/lang/Class.newInstance0()Ljava/lang/Object;:
       [exec] Illegal type in constant pool
       [exec] Result: 1
  
  package:
  
  BUILD SUCCESSFUL
  Total time: 1 second

But now that I have Java 1.4, it’s compiling fine.

Get Rails working again after installing MacPorts

Mac Rails developers: I installed MacPorts to get RMagick, and it installed a new version of Ruby, without RubyGems. This meant that when I typed script/server, it claimed that:

“Rails requires RubyGems >= . Please install RubyGems and try again: http://rubygems.rubyforge.org”

MacPorts had added some crap to my PATH variable. If you edit your ~/.profile you’ll see this line:

export PATH=/opt/local/bin:/opt/local/sbin:$PATH

Just change that to:

export PATH=$PATH:/opt/local/bin:/opt/local/sbin

Then open a NEW shell prompt, i.e. a new Terminal window (this reloads the PATH variable)…

…and it’ll find your OLD version of Rails before the new MacPorts one, and things will work normally. You can confirm by typing “which ruby” in the terminal, and you should see “/usr/bin/ruby” NOT “/opt/local/bin/ruby”.

Amazon s3 storage search interface

I use Amazon S3 for backup, and find that it’s hard to use because there’s no search function, so I can’t see what I’ve backed up.

So, I extended s3cmd.rb (comes with the WONDERFUL s3cmd.rb rsync clone for Amazon S3) with a search command.

Now I can use s3sync.rb to rsync my files up, then type:

s3sync.rb search bucket_name:path/to/restrict/search search_term

and I get, for example:

backup/2009/spatial/jeff-spatial-vertical.key
backup/2009/spatial/spatial-assignment-1-jeff.pdf
backup/2009/spatial/spatial-assignment-1.key
backup/2009/spatial/spatial-assignment-1.key.zip
backup/2009/spatial/spatial-interface
backup/2009/spatial/spatial-lecture-1
backup/2009/spatial/spatial-lecture-1.key
backup/2009/spatial/spatial-lecture-1.key.zip
backup/2009/spatial/spatial-lecture-1/spatial-lecture-1.key
backup/2009/spatial/spatial-lecture-1/spatial-lecture-1.pdf
backup/2009/spatial/spatial-lecture-3.key
backup/2009/spatial/spatial-lecture-3.mov
backup/2009/spatial/spatial-lecture-3.pdf
backup/2009/spatial/spatial-uploads
backup/2009/spatial/spatial-wayfinding.key
you just spent 6.0e-05 cents

The last line is because the price of making a list request in s3 is $0.01 per 1000 requests in the US; so I thought it’d be a good idea to tell people how much they’re spending.

I’ll try to commit this to the main s3sync.net codebase, but for now you can just download it here: s3sync

Composition of a typical OpenStreetMap API response

I’ve been wondering how much of the data transferred by the OpenStreetMap API is actual geometric data as opposed to timestamp and author data. I ran some rough numbers on a typical API response (in JSON, not XML, though these are relative measurements, so it shouldn’t matter too much). The file I examined is here: dr5ru0.json but my count is by number of characters and I did include formatting.

osm-composition

Native JSON parsing… hooray, with benchmarks!

I just tested the native JSON parser in Safari (this is also implemented in Firefox now, and IE8) and it dramatically speeds up the parsing of Cartagen’s JSON data files! Take a look at these times for parsing six typical data plots in milliseconds:

prototype.js’s .evalJSON():
parsed: 43
parsed: 7
parsed: 47
parsed: 63
parsed: 37
parsed: 53
Native JSON.parse():
parsed: 8
parsed: 6
parsed: 6
parsed: 9
parsed: 5
parsed: 7

That’s a sixfold improvement!

And for Firefox 3.5:

prototype.js’s .evalJSON():
parsed: 174
parsed: 157
parsed: 198
parsed: 349
parsed: 155
parsed: 455
parsed: 323
parsed: 276
parsed: 272
Native JSON.parse():
parsed: 13
parsed: 13
parsed: 14
parsed: 20
parsed: 12
parsed: 25
parsed: 20
parsed: 18
parsed: 18

That’s more than 15 times the speed!

Adjustment for Rails ‘find()’ method

Rails performs an SQL SELECT statement like “SELECT * FROM table_name WHERE table_name.id IN (1,2,3,4,5)” when its .find() method is passed an array of IDs. But it fails if one of those ids is not found. Sometimes it’s useful to find all _available_ records, but not fail if not all are found. I adjusted the search for this scenario in my working copy of the OpenStreetMap Rails port:

# @nodes += Node.find(nodes_to_fetch, :include => :node_tags)
@nodes += Node.find(:all,:conditions => ["id IN ("+nodes_to_fetch.join(',')+")"], :include => :node_tags)

Google News library for Ruby

I was surprised that Google News has no API, so I built my own: Using HTTParty, I wrote a small Ruby library for Google News. Put this in a ‘googlenews.rb’ file in the /lib/ directory of your Rails application to grab google news stories and topic lists.

Haven’t figured out how to get more than 10 items… Safari seems to be able to with feed://news.google.com?output=rss… but I’m not sure how.

gem "httparty"
require "httparty"

class Googlenews
  include HTTParty
  base_uri 'news.google.com'
  
  def self.items
    options = { :query => { :output => 'rss' } }
    features = self.get('/news', options)
    features['rss']['channel']['item']
  end
  
  def self.extract_topic(item)
    # ncl=duPfj30A5WoxBHMupoOtlciO1jY1M&
    item['description'].match(/ncl=([a-zA-Z0-9]+)&/)[1]
  end

  def self.topic(item)
    options = { :query => { :output => 'rss', :ncl => self.extract_topic(item) } }
    features = self.get('/news/more', options)
    features['rss']['channel']['item']
  end
  

  # store in database (unused)
  # def self.story(item)
    # item['category']
    # item['title']
    # item['guid']
    # item['description']
    # item['link']
    # item['pubDate']

    # t.string :category, :default => ''
    # t.string :title, :default => ''
    # t.string :guid, :default => ''
    # t.text :description, :default => ''
    # t.string :link, :default => ''
  # end

end

Once you’ve dropped this in /lib/, you can use it in a Rails controller like this (I needed JSON, to work with the feed in JavaScript):

class DataController < ApplicationController

  def latest
    render :json => Googlenews.items
  end

end

Update: to get more than 10 items, add a GET parameter ‘num=20’ to the feed URL. This works both for topic and for the main feed. I haven’t confirmed this but I think it’s still limited to 30 items. Again, Safari manages to get more, but perhaps it does repeated queries. Still investigating. Anyways, that makes it:

 def self.items
    options = { :query => { :output => 'rss', :num => '20' } }
    features = self.get('/news', options)
    features['rss']['channel']['item']
  end 

Update: This now has a real home on Github, where it’s called ruby-googlenews.

SVG vs. Canvas

Well, SVG doesn’t scale well to large numbers of objects, but Canvas doesn’t scale well to large screens:

Performance with many objects

Here are the results of the first fruitful experiment, which clearly shows that SVG performance degrades quickly (exponentially on Safari?) in the number of objects, but Canvas performance remains at a near-constant low. This makes sense, since Canvas is just a bitmap buffer, while SVG has to maintain additional references to each object that it renders. Also, though not pictured, note that performance in clearing an SVG element also decreases in the number of drawn objects.

Performance on a large rendering context

Via Boris Smus

Serving ‘OSM-JSON’ alongside XML from the OpenStreetMap Rails port

OpenStreetMap.org‘s RESTful API allows anyone to access data on their continually growing collaborative map of the world… in XML. This is great for most applications, but if you’re working in JavaScript (as we are), XML might as well be greek. We need JSON.

To offer OSM-JSON along with of OSM-XML, we added a route to accept a “.format” suffix, and split up the render call based on the params[:format] part of the route:

# /config/routes.rb:46-50

map.connect "api/#{API_VERSION}/geohash/:geohash.:format", :controller => 'api', :action => 'geohash'
map.connect "api/#{API_VERSION}/geohash/:geohash", :controller => 'api', :action => 'geohash'

map.connect "api/#{API_VERSION}/map.:format", :controller => 'api', :action => 'map'
map.connect "api/#{API_VERSION}/map", :controller => 'api', :action => 'map'

Notice we also added a ‘geohash’ route. Whereas the /map call requires a bbox parameter (‘bbox=min_lon,min_lat,max_lon,max_lat’), we can use a geohash (Geohash in JavaScript, Geohash in Rails) which defines a bounding box as a sequence of letters and numbers. This fits Cartagen’s needs well, and since it doesn’t require any parameters, we can page cache it in Rails. (Remember that page caching bypasses Rails entirely, letting Apache handle these cached files at high speed – that saved us when we were on BoingBoing).

Continue reading “Serving ‘OSM-JSON’ alongside XML from the OpenStreetMap Rails port”