RealityForge.org

Multi-value LIKE parameters in SQL

2010-09-05T00:00:00+10:00

Thinking in SQL, thinking in terms of sets took me a while to achieve. In the beginning I often lapsed into procedural thinking and relied heavily on stored procedures. Often I find myself of learning one more feature that makes me re-evaluate how I have written SQL in the past.

Learning about recursive common table expressions made me change the way I queried hierarchial data. Table valued functions had a similar impact particularly when using CROSS APPLY . However sometimes it is just the combination of simple features that produce something interesting.

I was tasked with creating a report searching for people based on a search string that a user supplies. The search string is space separated and all the terms in the search need to be matched. So if a user supplies the search phrase “Jam Wa” then the query must return return a person named “Walker, James” and “James Walker” but not “Brian Walker”.

The solution is presented below. Once I realized that the rvalue of the LIKE operator could be a column expression then it was simple to use a common table expression MatchSet, a table of like clauses. The only trick was ensuring that the number of rows returned joined against MatchSet was equal to the number of rows in MatchSet. This ensured that all terms in the query were matched.

DECLARE @SearchString VARCHAR(50)
SET @SearchString = 'Jam Wal'

WITH MatchSet(Match) AS (
 SELECT
   '%' + Value + '%' as Match
 FROM
   Warehouse.fnSplit(' ', @SearchString)
 WHERE Value != ''
)
SELECT * FROM tblPhysicalUnit WHERE ID IN (
SELECT
 P.ID
FROM
 tblPhysicalUnit P
JOIN MatchSet M ON P.DisplayString LIKE M.Match
GROUP BY P.ID
HAVING COUNT(*) = (SELECT COUNT(*) FROM MatchSet)
)

HTML 5 Canvas Sheet Fire Effect

2010-05-19T00:00:00+10:00

I had a few hours to kill last Saturday so I was looking to try out some neat little programming hack … possibly in a new programming language. I settled on playing with the HTML 5 Canvas component in javascript.

Initially I built the traditional “Hello World!” program of the graphics world … the Starfield effect! I was working on a 2d canvas but projecting from a 3D particle system. The maths are relatively easy to get right even if you are out of practice. However this was too easy and I grew bored even though I did create a fireworks system that looked pretty.

I decided to move onto pixel based simulation and that is when I remembered the sheet fire effect as above. I used to love putting the effect in assembly demos back in the day. It is a simple system where each pixel is assigned a heat. The heat of a pixel is a function of the average heat of the surrounding pixels in the previous time frame with more weight given to the pixels below the current pixel. The heat decays each time step. (i.e. heat rises and dissipates over time). The bottom row of pixels has randomly generated heat. The heat value is then mapped to a pixel value. In the example above I used a linear mapping between heat and intensity of the red color. A more interesting mapping could map heat values to a spectrum of colors such as red → yellow → blue → white.

While it was interesting to spend a few hours learning about the canvas component, I have decided that I really dislike javascript as a language. It has the significant advantage that it is ubiquitous and every Joe can almost be guaranteed to have it installed but as a developer the experience is somewhat lacking. Even with all the fancy debugging plugins provided with FireFox it was less than fun. I really felt the lack of static typing which is not something I have felt even in languages like Ruby.

I put the sheet fire effect on one of the http error pages so that I don’t lose it. It will be interesting to see if the javascript still runs in a year or two or if it rots as the HTML 5 standard evolves.

Blog Rebooted

2010-05-07T00:00:00+10:00

Over the last few years I have blogged, wiki-ied and word-smithed in all sorts of random locations. While most of the stuff has been about programming in one form or another – I have also spent a lot of time writing about beginner photography, image processing, rock climbing, caving and other things.

Every time I start writing again it seems to be a new blog, in a new location with no reference to that which came before. Unfortunately as my interest wanes the site inevitably gets forgotten and passes into the void. Rather than seeing all the articles disappear again, I have decided to centralize my writing on this site. Rather than lock the articles away in some database, I have decided to store them in source control and generate a static site with Jekyll . I have imported the articles from the old realityforge.org blog and one of the blogger.com blogs that I once used. Content from my other blogs may appear in time (Or may not!).

I suspect that I will get back to writing again once the post PhD haze has died down. When that time comes expect to see articles posted here.

MMTk Tutorial

2007-08-15T00:00:00+10:00

I was a little stressed earlier on today so I decided to relax by doing some coding and I have yet to really play down in the guts of MMTk so I decided to have a look at the tutorial. The tutorial available from the website is a little out of date but relatively easy to follow even if you have little knowledge about garbage collection literature. MMTk is designed to provide automatic memory management for different languages in different environments. Initially some of the abstractions within the toolkit seem somewhat academic but when viewed in this context it makes much more sense.

For example, ObjectReference is used to refer to the client objects that MMTk is managing. Tasks (i.e. processes , threads and fibers ) are not directly represented but the data required for those entities are represented by CollectorContext and MutatorContext. In gc literature a collector is the task responsible for identifying and reclaiming the space used by client objects when they are no longer live. The collector is for most parts a virtual machine managed entity that is triggered by various events. The mutators are the tasks representing application functionality and generally responsible for allocating client objects.

At the current time, in the Jikes RVM the VM_Processor are both collectors and mutators at various times. This is enforced by MM_ProcessorContext that includes both the CollectorContext and MutatorContext classes. MM_ProcessorContext extends the mutator, rather than composes the mutator as the information in the MutatorContext is used very frequently. Every allocation needs to access mutator specific methods. By extending it rather than composing it the methods can be accessed directly rather than indirectly. This increases performance but I have never actually evaluated how much of an impact this has.

In VMs that use native threads I would expect that mutators would be separate threads from collectors. In a StopTheWorld plan the collector threads would likely sit idle until a collection is triggered and all the mutator threads became idle. The mutator threads would most likely need to get to a gc safe location. GC safe locations are easy to do within the RVM’s green threads but more “interesting” in native threads I would hazard to guess.

Another trick that MMTk uses to improve performance is to separate out operations on global resources vs collector local resources or mutator local resources. For example, each mutator is likely to grab a block of memory from a global resource (synchronized) and allocate from that block (unsynchronized) until the local block runs out, at which point it would access the global resource again.

The tutorial predates the separation of CollectorContext from MutatorContext (previously it seems to have been combined in a single PlanLocal class). Once I figured that out it was relatively easy to translate the tutorials instructions into the new MMTk. Especially if you peek at some of the other classes in MMTk plan sub-packages.

I think my next little experiment will be decorating objects with some before and after data. In theory it should be relatively easy and I guess I could look at refcount plans for hints – I assume they have to store data around the object.

Anyhoo – I wonder if anyone else finds it relaxing to muck about coding up a garbage collector ? :)

Fixed maximum space sizes

2007-07-28T00:00:00+10:00

A few days ago I started to look into what it would take to get the Jikes RVM running as a server with a large number of threads. Oh and by large, I mean somewhere in the vicinity of 80,000 threads on a modern desktop machine with 4GB of memory available. Almost every thread would maintain an open network connection (with an 8KB kernel buffer) and intermittently transmit a message.

At the time I did not think it was an unreasonable demand. Erlang can reportedly create a software isolated process in 3-400 bytes and Haskell can create a thread in about 1KB. I have not investigated these claims but lets assume that these claims include runtime structures required to implement scheduling and garbage collection.

I wanted to get each Jikes RVM thread down to about 1KB for runtime structures, 4KB for the call stack, 4KB for the thread local allocator and another 8KB for each network connection. This would mean that I was looking at about 1.3GB “overhead” to create the threads before I added in any application logic. Not ideal but not too bad. I expected that that big challenge I was going to face was implementing a scheduler that had a decent response time and reasonable throughput. Unfortunately I hit a few problems before even getting to that step.

When creating the stacks the Jikes RVM allocates them into the Large Object Space (or LOS). The LOS is used as the runtime attempts to allocate a byte array in the default space. The allocation request first checks whether the allocation exceeds a threshold and if it does attempts to allocate the object in the LOS.

Currently spaces in MMTk are assigned an address range at build time. The address range is either based on a specific size (i.e. 32MB for the Immortal Space) or based on a proportion of the available space (i.e. 3% for the LOS). The available space is from the end of the code in the bootimage (i.e. 0×57000000 + 24<<20) to the maximum mappable address (0xB0000000 on ia32-linux by default). Why it is not from the end of the reference map in the bootimage I am not entirely sure. Based on this the maximum size of the LOS is around 40MB. Nowhere near enough for my purposes.

Luckily it looks like MMTk will undergo some evolution in the near future. RVM-157 describes the proposed changes. It looks like the management of address ranges and the management of memory chunks will be separated. Address ranges for spaces may still be defined at build time but it will be possible for spaces to have overlapping address ranges. So the set of memory chunks allocated to a space can grow or shrink based on actual usage rather than restricted by some artificial limitations.

Some Spaces may still need to have exclusive ownership of an address range for performance reasons. If a Space exclusively owns an address range the cost for checking if an address is within a Space is just comparing an address to a start and end address. Non-exclusive ownership of an address range means that MMTk must maintain a map between memory chunks and the owning Space and the cost of checking if an address is within a Space is much higher. In a generational collector the (frequently called) write barriers must check whether an address is within the nursery space and thus the nursery space should have exclusive ownership of an address range to maintain an acceptable performance level.

It was my first time looking at the MMTk code and in many ways it seems much nicer than the rest of the Jikes RVM codebase and much more consistent. There is still a few uglies there and a bunch of undocumented assumptions but I guess that is going to be the case in any large codebase. There is a bunch of strange terminology in it but it could just be because I am unfamiliar with gc literature. i.e. A Space defines; (1) an address range, (2) a set of allocated memory chunks and (3) a management policy. Personally I would have used the term area as space seems to easy to confuse with other aspects. Ohwell!

I had planned to do most of the work to enable this myself but my first commit that I believed was largely performance neutral and was just a first step in refactoring seemed to raise the hackles of the ANU contingent. Not surprising given they didn’t really know where I was going with it and I am a self confessed ignoramus regarding garbage collection. They seem to have come up with a much more thought out plan than I had and Daniel may even tackle it in a few weeks. If not I will have to start hacking at it at a later stage.

In the meantime I attempted to change the RVM so that stacks are allocated in the Primitive Large Object Space (PLOS) as they are basically a large byte array and the PLOS can grow to 7% of the available space. Unfortunately that caused complete failure on the 64-bit build target, most likely due to alignment issues. I don’t have any hardware available to test on so I reverted to allocating in the LOS. Ugh.

The last week I have been out of action due to illness and next week it looks like other parts of my PhD will need to be tackled. But hopefully I will get back to trying to enable large numbers of threads soon!

Testing Helpers in Rails

2007-06-14T00:00:00+10:00

It can be useful at times to unit test helpers to make sure they generate correct html. It is not obvious how to do this at first. So far I have been testing my helper by defining a class “MyClass” at the top of my unit test and including all the appropriate modules. I also need to define a url_for method if I ever want to test helpers that generate links.

The code follows (Replace MyHelper with your appropriate helper class);

class MyClass
  include ERB::Util
  include ActionView::Helpers::TagHelper
  include ActionView::Helpers::UrlHelper
  include MyHelper

  def url_for(options)
    ActionController::Routing::Routes.reload if ActionController::Routing::Routes.empty?
    generated_path, extra_keys = ActionController::Routing::Routes.generate_extras(options, {})
    generated_path
  end
end

Then in my tests I do something like;

def test_revision_link
  assert_equal(
    "<a href=\"http://svn.sourceforge.net/viewvc/jikesrvm?view=rev&amp;revision=22\">22</a>", 
    MyClass.new.revision_link(22))
end

Seems easy enough to do in retrospect but things usually do.

Enhanced Rails Plugin to Validate (X)HTML and CSS

2007-05-09T00:00:00+10:00

It feels like forever ago that I was last working with ruby on rails. But I am back hacking away at a few rails and I slowly getting more familiarity with ruby itself rather than just using it as part of the rails platform.

Something I have wanted to do for a long time is automagically validate content such as (X)HTML, CSS, atom feeds etc that is generated by the web application. A while ago I put together the assert-valid-asset plugin that allowed you to assert (in functional tests) that the content generated is valid. However you still had to explicitly call the assert.

So recently I enhanced the plugin so that it can automatically validate generated content when configured to do so. To configure auto validation you need to set a class variable in Test::Unit::TestCase via code such as;

class Test::Unit::TestCase
  self.auto_validate = true
end

Then anytime content is generated in tests (such as via get and post methods) it will check the mime type of the content. If the content has a mime type of ‘text/html’ or ‘text/xhtml’ it will pass it to the ‘assert_valid_markup’ method. If the content is ‘text/css’ then it will be validated by the ‘assert_valid_css’ method.

Of course you may have tests that generate invalid (X)HTML or CSS (to work with specific unnamed browsers) and you may want to exclude these tests from the automatic content validation. This can be done by adding the test symbol to the exclude list or alternatively by adding the desired test symbols to an include list. Both of the following examples have identical behavior;

Example 1:

class FooControllerTest < Test::Unit::TestCase
  ...
  self.auto_validate_excludes = [:test_foo, :test_bar]

  def test_foo; ... ; end
  def test_bar; ... ; end
  def test_baz; ... ; end
end

Example 2:

class FooControllerTest < Test::Unit::TestCase
  ...
  self.auto_validate_includes = [:test_baz]

  def test_foo; ... ; end
  def test_bar; ... ; end
  def test_baz; ... ; end
end

Update 12th of May, 2010

The plugin is now available on GitHub. See the GitHub project page

Smuggle

2006-11-14T00:00:00+11:00

I needed to mirror some galleries from one smugmug account to another. I thought about doing it manually but that idea seemed like way too much work.

I found a python script by rutt that did a little bit of what I wanted and went to work hacking. This resulted in smuggle.py that may be useful if you ever have to programatically interact with the smugmug APIs. It will probably evolve as my needs change and as I actually start to learn python.

~~-The source is available from subversion.~~-

Update 12th of May, 2010

The subversion repository holding the original version of this has since gone away but in more recent times it has become easier to achieve this as there is a myriad of libraries out there. Below is an example in ruby of how easy it is to download images from a smugmug account. Upload is left as an exercise for the reader.

require 'smugmug'
require 'fileutils'
require 'net/http'

Username = 'MyUser'
Password = 'MyPassword'
AlbumIDs = [1234]

puts "Logging into #{Username}"
sm = SmugMug::SmugMug.new(Username,Password)

sm.each do |a|
  puts "#{a.title} - #{a.album_id}"
  if AlbumIDs.include?( a.album_id )
    FileUtils.mkdir_p "#{a.album_id}"
    a.each do |image|
       puts "#{image.file_name} has url #{image.original_url}"
       filename = "#{a.album_id}/#{image.image_id}#{File.extname(image.file_name)}"
       File.open(filename,'w') do |f|
         f.write Net::HTTP.get(URI.parse("#{image.original_url}"))
       end unless File.exist?( filename )
    end
  end
end

Moving from Eyelook to Smugmug

2006-07-03T00:00:00+10:00

I needed to host photos from a recent trip to New Zealand. I started modifying eyelook to make it easier to do what I wanted such as geocoding, bulk upload, hierarchial grouping, different sizes, different layouts, themes etc. It was taking a bit of time so I decided to look around at existing services but none of them suited my needs.

Then I found SmugMug and I was hooked. This is the best photo-sharing service I have seen. It seems to be aimed at professional photographers – there is no free version but you get what you pay for. It is far from perfect but there is no way I would consider anything else at this stage.

Luckily it has a referral system that gets you $5 off your first year if you indicate an account that got you onto the system so I got mine a little cheaper. I also helped out the friend that referred me by making it a little cheaper for them next year. If anyone wants to get same deal, it’s simple: enter my email address ( peter at realityforge dot org ) or my coupon ( EqCTz7hz7wkW2 ) in the ‘Referred by’ field on the signup form and you get $5 off.

Best photo sharing service yet!

Eyelook: Rails photo gallery

2006-04-21T00:00:00+10:00

Eyelook is a photo gallery I created over a weekend when I was home sick. It is amazing how productive rails can make you because within 8 hours I had the code in place and had set it up on my personal web server.

Very little of that time was spent coding – most of it was spent working on re-arranging the xhtml+css or figuring out how to go about Loading Binary Data into Rails Fixtures.

Every eyelook application has a set of users who have a list of albums with pictures. The image data is stored in the database and cached on the filesystem on demand. This way a backup of the database backs up the complete system.

An example of users page that lists galleries is

You can select one of these galleries and it will bring up a list of images such as

From there you can either download the original or can view a larger lightbox style image

The admin section is not as sexy but it is functional.

Update 12th of May, 2010

The subversion repository holding the original version of this has since gone away but the code has been imported into GitHub. See the GitHub Project Page for further details.