Copious Free Time

Managing a Gem Repository with Stickler

2008-10-09T22:19:06-06:00

2010-09-20:

Stickler has been completely rewritten and is now at version 2.0.1. Please see the latest documentation

Managing a Gem Repository with Stickler (2008-10-09)

One of the items Fernand and I mentioned in our Lone Star talk was Stickler. This was a project I had originally started on back in June and it lay dormant for a little while.

The first public version (0.1.0) is now released.

Why should I use Stickler?

One of the problems that we have had at work is inadvertently upgrading a gem on a production system. For example, you are adding a gem to a system that depends on activerecord, >= 2.0.2. This is great, until active_record 2.1.1 comes out and on your production system you type gem install somegem and it updates your active record installation, as a side effect. Yes, you can just quickly uninstall it, but it is annoying.

Another issue that some installations have, is production boxes are unable to talk directly to the Internet. This is a problem when doing a gem installation using rubygems. You either have to run your own gem server, or install the gems locally.

Installation and Setup

Stickler currently has a basic workflow. First, install and setup your Stickler repository.

jeremy@aramis:~ % sudo gem install stickler
  ============================================================

  Thank you for installing Stickler!

  * Create a new stickler repository:
      stickler setup /path/to/repo

  * Look at the help:
      stickler help

  ============================================================
Successfully installed stickler-0.1.0
1 gem installed
Installing ri documentation for stickler-0.1.0...
Installing RDoc documentation for stickler-0.1.0...

jeremy@aramis:~ % stickler setup /tmp/stickler
created repository root /tmp/stickler
created directory /tmp/stickler/gems
created directory /tmp/stickler/log
created directory /tmp/stickler/specifications
created directory /tmp/stickler/dist
created directory /tmp/stickler/cache
copied in default configuration to /tmp/stickler/stickler.yml
Setting up sources
 * loading http://gems.rubyforge.org/

A Stickler repository is a specific directory that holds all the gem files that are managed by Stickler for your personal distribution. In this case, Stickler is now installed as a gem and we have setup a repository in /tmp/stickler.

Populating the Repository

A repository is great, and completely useless unless it is populated with gems to distribute. So that is what we will do. The best way to operate on a Stickler repository is to be in it. You can also use the --directory commandline option on every command, but that can get old. Therefore, we'll operate from the root of the repository.

cd /tmp/stickler

First lets look at the state of the repository using the info command.

jeremy@aramis:/tmp/stickler % stickler info
Setting up sources
 * loading http://gems.rubyforge.org/

Upstream Sources
----------------

  http://gems.rubyforge.org/ : 16651 gems available
                             : 0 gems existing

Configured gems (in stickler.yml)
---------------------------------


Existing gems
-------------

Yes, that definitely is empty. We do a fair number of merb apps, so lets add merb to our repository. It is a 2 step process to add a gem to the repository via the commandline. We first pick the requirement operator, and then the version. In our case, we want to stick with a specific version of merb so we'll add merb, = 0.9.8 to our repository

jeremy@aramis:/tmp/stickler % stickler add gem merb
Setting up sources
 * loading http://gems.rubyforge.org/

You need to pick the merb Requirement to configure Stickler.
This involves picking one of the following Requirement operators
See http://docs.rubygems.org/read/chapter/16#page74 for operator info.

You need to (1) pick an operator and (2) pick a requirement.
The most common operators are >=, > and ~>
1. =  Equals version
2. != Not equal to version
3. >  Greater than version
4. <  Less than version
5. >= Greater than or equal to
6. <= Less than or equal to
7. ~> Approximately greater than
(1) Pick an operator ? 1

Now to pick a requirement.  Based upon your chosen operator '=',
These are the available version of the merb gem.
1. = 0.9.8   7. = 0.9.2   13. = 0.4.1  19. = 0.3.0  25. = 0.0.6
2. = 0.9.7   8. = 0.5.3   14. = 0.4.0  20. = 0.2.0  26. = 0.0.5
3. = 0.9.6   9. = 0.5.2   15. = 0.3.7  21. = 0.1.0  27. = 0.0.4
4. = 0.9.5   10. = 0.5.1  16. = 0.3.4  22. = 0.0.9  28. = 0.0.3
5. = 0.9.4   11. = 0.5.0  17. = 0.3.3  23. = 0.0.8  29. = 0.0.2
6. = 0.9.3   12. = 0.4.2  18. = 0.3.1  24. = 0.0.7  30. = 0.0.1
(2) Pick a requirement ? 1

Resolving gem dependencies for merb (= 0.9.8, runtime) ...
Adding ParseTree-2.1.1-x86-mswin32
Adding ruby2ruby-1.1.9
Adding RubyInline-3.7.0
Adding ParseTree-2.2.0
Adding merb-action-args-0.9.8
Adding merb-assets-0.9.8
Adding highline-1.4.0
Adding diff-lcs-1.1.2
Adding templater-0.3.2
Adding merb-gen-0.9.8
Adding haml-2.0.3
Adding merb-haml-0.9.8
Adding builder-2.1.2
Adding merb-builder-0.9.8
Adding mailfactory-1.4.0
Adding merb-mailer-0.9.8
Adding merb-parts-0.9.8
Adding merb-cache-0.9.8
Adding merb-slices-0.9.8
Adding merb-jquery-0.9.8
Adding extlib-0.9.7
Adding abstract-1.0.0
Adding erubis-2.6.2
Adding json_pure-1.1.3
Adding rubyforge-1.0.0
Adding rake-0.8.3
Adding hoe-1.8.0
Adding rspec-1.1.8
Adding rack-0.4.0
Adding mime-types-1.15
Adding hpricot-0.6.161-java
Adding hpricot-0.6.161
Adding hpricot-0.6-x86-mswin32
Adding thor-0.9.6
Adding merb-core-0.9.8
Adding merb-helpers-0.9.8
Adding merb-more-0.9.8
Adding mongrel-1.1.5-java
Adding mongrel-1.1.1-java
Adding daemons-1.0.10
Adding fastthread-1.0.1
Adding fastthread-1.0.1-x86-mswin32
Adding mongrel-1.1.5
Adding mongrel-1.1.3-x86-mswin32
Adding mongrel-1.1.5-x86-mswin32-60
Adding mongrel-1.1.2-x86-mswin32
Adding gem_plugin-0.2.3
Adding cgi_multipart_eof_fix-2.5.0
Adding mongrel-1.1.5-x86-mingw32
Adding merb-0.9.8

Wow, that's a lot of gems added with for merb. And this is more than would be normally installed if you did a gem install merb. Stickler assumes you are redistributing gems, and as such gets every available platform for a gem when you add it to the repository. And then it recurses through all runtime and development dependencies. And now when we look at the repository state we see:

Setting up sources
 * loading http://gems.rubyforge.org/

Upstream Sources
----------------

  http://gems.rubyforge.org/ : 16652 gems available
                             : 50 gems existing

Configured gems (in stickler.yml)
---------------------------------

merb : = 0.9.8

Existing gems
-------------

ParseTree-2.1.1-x86-mswin32
ParseTree-2.2.0
RubyInline-3.7.0
abstract-1.0.0
builder-2.1.2
cgi_multipart_eof_fix-2.5.0
daemons-1.0.10
diff-lcs-1.1.2
erubis-2.6.2
extlib-0.9.7
fastthread-1.0.1
fastthread-1.0.1-x86-mswin32
gem_plugin-0.2.3
haml-2.0.3
highline-1.4.0
hoe-1.8.0
hpricot-0.6-x86-mswin32
hpricot-0.6.161
hpricot-0.6.161-java
json_pure-1.1.3
mailfactory-1.4.0
merb-0.9.8
merb-action-args-0.9.8
merb-assets-0.9.8
merb-builder-0.9.8
merb-cache-0.9.8
merb-core-0.9.8
merb-gen-0.9.8
merb-haml-0.9.8
merb-helpers-0.9.8
merb-jquery-0.9.8
merb-mailer-0.9.8
merb-more-0.9.8
merb-parts-0.9.8
merb-slices-0.9.8
mime-types-1.15
mongrel-1.1.1-java
mongrel-1.1.2-x86-mswin32
mongrel-1.1.3-x86-mswin32
mongrel-1.1.5
mongrel-1.1.5-java
mongrel-1.1.5-x86-mingw32
mongrel-1.1.5-x86-mswin32-60
rack-0.4.0
rake-0.8.3
rspec-1.1.8
ruby2ruby-1.1.9
rubyforge-1.0.0
templater-0.3.2
thor-0.9.6

There is a difference between configured gems and existing gems. Configured gems are those you have specifically requested to be in the repository. Existing gems are those that are added to support the configured gems and the configured gems themselves.

Adding Additional Sources

Part of the reason for Stickler is to merge upstream gem repositories into your own internal repository. Say you use the github repository and want to add a gem or two from it to your internal system.

jeremy@aramis:/tmp/stickler % stickler add source http://gems.github.com
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/
http://gems.github.com added to sources

That was easy enough. Now lets add a gem from github and watch the dependencies roll in.

jeremy@aramis:/tmp/stickler % stickler add gem aniero-tire_swing
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/

You need to pick the aniero-tire_swing Requirement to configure Stickler.
This involves picking one of the following Requirement operators
See http://docs.rubygems.org/read/chapter/16#page74 for operator info.

You need to (1) pick an operator and (2) pick a requirement.
The most common operators are >=, > and ~>
1. =  Equals version
2. != Not equal to version
3. >  Greater than version
4. <  Less than version
5. >= Greater than or equal to
6. <= Less than or equal to
7. ~> Approximately greater than
(1) Pick an operator ? =

Now to pick a requirement.  Based upon your chosen operator '=',
These are the available version of the aniero-tire_swing gem.
1. = 0.0.3  2. = 0.0.2
(2) Pick a requirement ? 1

Resolving gem dependencies for aniero-tire_swing (= 0.0.3, runtime) ...
Adding polyglot-0.2.3
Adding treetop-1.2.4
Adding attributes-5.0.1
Adding activesupport-2.1.1
Adding aniero-tire_swing-0.0.3

You can see that aneiro-tire_swing was added from github, and it also resolved the dependencies to gems that reside on rubyforge. Looking at info shows us:

jeremy@aramis:/tmp/stickler % stickler info
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/

Upstream Sources
----------------

  http://gems.rubyforge.org/ : 16652 gems available
                             : 54 gems existing
     http://gems.github.com/ : 2463 gems available
                             : 1 gems existing

Configured gems (in stickler.yml)
---------------------------------

aniero-tire_swing : = 0.0.3
merb : = 0.9.8
...

Batch Addition

It can get repetitive to manually add gem after gem from the commandline. When you know exactly what you need, crack open the stickler.yml file and add a few gems. For instance, lets say that we are in the process of updating all our rails apps to 2.1. In the meantime we want to make sure we do not inadvertently update our production systems to 2.1. They are still at 2.0.2. So we add all the rails gems at the 2.0.2 level to the stickler.yml file in the repository.

---
downstream_source: http://gems.example.com/
sources:
- http://gems.rubyforge.org/
- http://gems.github.com/
gems:
  aniero-tire_swing:
  - = 0.0.3
  merb:
  - = 0.9.8
  activerecord: = 2.0.2
  activesupport: = 2.0.2
  actionmailer: = 2.0.2
  actionpack: = 2.0.2
  activeresource: = 2.0.2
  rails: = 2.0.2

We can now use the sync command to add all of them in at once.

jeremy@aramis:/tmp/stickler % stickler sync
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/

Making sure that all gems listed in configuration are available

Resolving gem dependencies for rails (= 2.0.2, runtime) ...
Adding activerecord-2.0.2
Adding actionpack-2.0.2
Adding actionmailer-2.0.2
Adding activesupport-2.0.2
Adding activeresource-2.0.2
Adding rails-2.0.2
Resolving gem dependencies for activerecord (= 2.0.2, runtime) ...
Resolving gem dependencies for activeresource (= 2.0.2, runtime) ...
Resolving gem dependencies for aniero-tire_swing (= 0.0.3, runtime) ...
Resolving gem dependencies for actionpack (= 2.0.2, runtime) ...
Resolving gem dependencies for actionmailer (= 2.0.2, runtime) ...
Resolving gem dependencies for merb (= 0.9.8, runtime) ...
Resolving gem dependencies for activesupport (= 2.0.2, runtime) ...

You can see that it added in all the gems in and then it also made sure that aneiro-tire_swing and merb were also synced up. If you need to rebuild the entire repository from scratch, use stickler sync --rebuild and it will wipe out the current gems and specifications in the repo and download them again.

Distribution of Gems

Populating the repository is only good if you can get your internal systems to utilize the repo. This is where the generate commands come into play. The generate index command does exactly what a gem generate_index does, but it uses your repository as the base.

Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/
Generating rubygems index in /private/tmp/stickler/dist
Loading 61 gems from /private/tmp/stickler/dist
...............WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/fastthread-1.0.1-x86-mswin32.gem => fastthread-1.0.1-x86-mswin32 (fastthread-1.0.1-mswin32)
.....WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/hpricot-0.6-x86-mswin32.gem => hpricot-0.6-x86-mswin32 (hpricot-0.6-mswin32)
WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/hpricot-0.6.161-java.gem => hpricot-0.6.161-java (hpricot-0.6.161-jruby)
..................WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/mongrel-1.1.1-java.gem => mongrel-1.1.1-java (mongrel-1.1.1-jruby)
WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/mongrel-1.1.2-x86-mswin32.gem => mongrel-1.1.2-x86-mswin32 (mongrel-1.1.2-mswin32)
WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/mongrel-1.1.3-x86-mswin32.gem => mongrel-1.1.3-x86-mswin32 (mongrel-1.1.3-i386-mswin32)
....WARNING:  Skipping misnamed gem: /private/tmp/stickler/dist/gems/ParseTree-2.1.1-x86-mswin32.gem => ParseTree-2.1.1-x86-mswin32 (ParseTree-2.1.1-i386-mswin32)
............
Loaded all gems
Generating quick index gemspecs for 54 gems
......................................................
Complete
Generating specs index
Generating latest specs index
Generating quick index
Generating latest index
Generating Marshal master index
Generating YAML master index for 54 gems (this may take a while)
......................................................
Complete
Compressing indicies

The generated, distributable gem repository based upon your stickler repository is generated in the dist sub directory in your repository. This directory can be rsynced to a web server in your infrastructure, or run a web server on this machine and point its document root, or a directory alias to the dist directory.

An extra item you may do is globally configure your rubygems installations to automatically use your internal gem server as the canonical source instead of http://gems.rubyforge.org/. Stickler provides a quick command to generate this top level configuration, and the file contains the information on where to put it.

jeremy@aramis:/tmp/stickler % stickler generate sysconfig > gemrc
Setting up sources
 * loading http://gems.rubyforge.org/
 * loading http://gems.github.com/
Generating configuration to stdout

jeremy@aramis:/tmp/stickler % cat gemrc
#
# This is the system wide configuration to be used by
# rubygem clients that install gems from the repository
# located at :
#
#   http://gems.example.com/
#
# On Unix like machines install in
#
#   /etc/gemrc
#
# On Windows machines install in
#
#   C:\Documents and Settings\All Users\Application Data\gemrc
#
---
:sources:
- http://gems.example.com/

Conclusions

Hopefully you can find Stickler useful in your infrastructure. Please let me know about features and bugs.

Comments (View)

Packaging an Application With Crate

2008-11-30T18:44:34-07:00

2009-01-18:

Updated tutorial with new versions of crate (0.2.1) and amalgalite (0.7.1)

Packaging an Application With Crate (2008-11-30)

Crate is way to package up your ruby applications as statically compiled binaries. I was lucky enough to talk about Crate at RubyConf '08.

This is a small tutorial that expands upon my RubyConf talk and demonstrates how to package gem application as a standalone statically compiled executable.

Step 1 — Install Crate

To get started, you'll need to install crate. It is distributed mainly as a gem from rubyforge and installs in the standard way.

gem install crate

Crate is part of my copiousfreetime rubyforge project and its git repository is available on github. Patches are always welcome.

crate on github
public clone url - <git://github.com/copiousfreetime/crate.git>

Step 2 — Create a new Crate Application

In this tutorial we are going to package up the htpassword-ruby commandline application in my htauth gem as a standalone executable.

% crate -v
Crate 0.2.1

% crate htpassword-ruby
[16:39:00]  INFO: creating htpasswd-ruby
[16:39:00]  INFO: creating htpasswd-ruby/Rakefile
[16:39:00]  INFO: creating htpasswd-ruby/crate_boot.c
[16:39:00]  INFO: creating htpasswd-ruby/recipes/amalgalite
[16:39:00]  INFO: creating htpasswd-ruby/recipes/amalgalite/amalgalite.rake
[16:39:00]  INFO: creating htpasswd-ruby/recipes/arrayfields
[16:39:00]  INFO: creating htpasswd-ruby/recipes/arrayfields/arrayfields.rake
[16:39:00]  INFO: creating htpasswd-ruby/recipes/configuration
[16:39:00]  INFO: creating htpasswd-ruby/recipes/configuration/configuration.rake
[16:39:00]  INFO: creating htpasswd-ruby/recipes/openssl
[16:39:00]  INFO: creating htpasswd-ruby/recipes/openssl/openssl.rake
[16:39:00]  INFO: creating htpasswd-ruby/recipes/ruby
[16:39:00]  INFO: creating htpasswd-ruby/recipes/ruby/ext-extmk.rb.patch
[16:39:00]  INFO: creating htpasswd-ruby/recipes/ruby/ruby.rake
[16:39:00]  INFO: creating htpasswd-ruby/recipes/rubygems
[16:39:00]  INFO: creating htpasswd-ruby/recipes/rubygems/rubygems.rake
[16:39:00]  INFO: creating htpasswd-ruby/recipes/zlib
[16:39:00]  INFO: creating htpasswd-ruby/recipes/zlib/zlib.rake

% cd htpassword-ruby

This created a new directory structure to do the building of our standalone htpassword-ruby application. Change into the new htpassword-ruby directory now and run rake -T to see the full set of tasks that are available. Most of them you will not use. The two you want to pay attention to are default and ruby.

Step 3 — Integrate HTAuth Dependency Targets

The way all of this is integrated together is a final single binary with the name htpassword-ruby which is an embedded ruby interpreter wrapped up in a thin C application. The C application is in the crate_boot.c file you'll see in the top level of the project directory.

The pure ruby code involved in the whole system is stored in SQLite databases. This will included the ruby standard library and your application code. As part of a Crate application, the require statement is overwritten to load from rows in an SQLite database instead of the file system. This feature is all from the amalgalite gem and anyone case use it for other projects if they so choose.

In the mean time, we now have a build system setup to build ruby, but not our application. There is no facility as of yet in Crate to automatically add build targets for a gem, but they are in the works for a later version. In the mean time we'll need to roll our own, its not that hard. Currently I'm not completely satisfied with the way this integrates and will most likely change it in a future release.

The HTAuth gem is pretty self contained, it only has one dependency, highline. We're going to add two new recipes, one for highline and one for htauth.

% mkdir recipes/highline
% vi recipes/highline/highline.rake

We edit it to be the following:

#
# The recipe for integrating highline into the ruby build
#
Crate::GemIntegration.new("highline", "1.5.0") do |t|
  t.upstream_source = "http://rubyforge.org/frs/download.php/46328/highline-1.5.0.gem"
end

And we do the same for htauth:

% mkdir recipes/htauth
% vi recipes/htauth/htauth.rake

#
# The recipe for integrating htauth into the ruby build
#
Crate::GemIntegration.new("htauth", "1.0.2") do |t|
  t.upstream_source = "http://rubyforge.org/frs/download.php/47663/htauth-1.0.2.gem"
end

And finally we we integrate these two new build targets into the final ruby build. Edit the recipes/ruby/ruby.rake file and add the two noted lines.

Crate::Ruby.new( "ruby", "1.8.6-p287") do |t| 
  t.depends_on( "openssl" )
  t.depends_on( "zlib" )

  t.integrates( "amalgalite" )
  t.integrates( "arrayfields" )
  t.integrates( "configuration" )

  t.integrates( "highline" )    # Add this line
  t.integrates( "htauth" )      # and this line

  t.upstream_source  = "ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p287.tar.gz"
  t.upstream_md5     = "f6cd51001534ced5375339707a757556"

  ENV["CPPFLAGS"]= "-I#{File.join( t.install_dir, 'usr', 'include')}"
  ENV["LDFLAGS"] = "-L#{File.join( t.install_dir, 'usr', 'lib' )}" 

  def t.build
    # put the .a files from the fakeroot/usr/lib directory into the package
    # directory so the compilation can use them
    %w[ libz.a libcrypto.a libssl.a ].each do |f| 
      FileUtils.cp File.join( install_dir, "usr", "lib", f ), pkg_dir
    end 
    sh "./configure --disable-shared --prefix=#{File.join( '/', 'usr' )}" 
    sh "make"
  end

  t.install_commands << "make install DESTDIR=#{t.install_dir}"

end

Now we are setup to do the two big build steps.

Step 4 — Building Ruby as a Static Library

In order to build a static ruby based application we first need to build ruby itself as a static library. In doing so we also need to build all the binary dependencies of ruby statically. In this case, that includes zlib, openssl and amalgalite.

Fortunately, all that is taken care of for you in the rake tasks. All you need to do at this point is run the ruby rake task. It will download, unpack and build all the various components. This may take a while, so take a break and get something to drink.

% rake ruby
# snip lots of output, look in project.log for the full output.  
....
23:54:58  INFO: Bulding zlib 1.2.3
....
23:55:30  INFO: Bulding openssl 0.9.8i
...
00:01:22  INFO: Integrating highline into ruby's tree
...
00:01:24  INFO: Integrating htauth into ruby's tree
...
00:01:24  INFO: Bulding ruby 1.8.6-p287
...
00:02:59  INFO: ruby 1.8.6-p287 is installed

You should now have a ruby executable in fakeroot/usr/bin/ruby. You can run this script to prove that it is statically compiled. Notice that there are no require statements. This script will not run on a normal ruby, it will only work with one that is statically compiled.

% cat versions.rb
puts "zlib    : #{Zlib::zlib_version}"
puts "OpenSSL : #{OpenSSL::OPENSSL_VERSION}"
puts "SQLite  : #{Amalgalite::SQLite3::Version}"

% /opt/local/bin/ruby versions.rb
ruby versions.rb
versions.rb:1: uninitialized constant Zlib (NameError)

% fakeroot/usr/bin/ruby versions.rb
zlib    : 1.2.3
OpenSSL : OpenSSL 0.9.8j 07 Jan 2009
SQLite  : 3.6.10

The really important products of this build process are all the .a files in the ruby build directory that will be used to build the final httpassword-ruby executable.

% ls -1 build/ruby/ruby-1.8.6-p287/*.a
build/ruby/ruby-1.8.6-p287/libcrypto.a
build/ruby/ruby-1.8.6-p287/libruby-static.a
build/ruby/ruby-1.8.6-p287/libssl.a
build/ruby/ruby-1.8.6-p287/libz.a

Additionally the highline and htauth ruby code was integrated into the ruby stdlib directory structure.

Step 5 — Building the final executable

The final step is to create out final htpasswd-ruby executable. The output of this step is:

a single executable
multiple SQLite databases holding all the ruby source code

At this point we need to update the top level Rakefile in your build system. We need to set the executable name and tell the crate build system how to launch the application.

In our case, we are going to wrap only the htpasswd-ruby application that is part of htauth. For this we look at that script which comes with the htauth gem. This script boils down to the single line:

HTAuth::Passwd.new.run(ARGV, ENV)

Which, it turns out, is exactly the way that crate likes to have applications launched. At this point I should sidestep and say how a crate based application is launched.

Crate Bootstrap Process

A Crate built application is fundamentally an embedded ruby interpreter that runs a single script. The file crate_boot.c at the top of your build system is

initialize the ruby interpreter
set ARGV for ruby
initialize all statically compiled extensions
bootstrap the Amalgalite driver
remove all filesystem directories from the $LOAD_PATH
switch to using Amalgalite backed require
require the file in the C constant CRATE_MAIN_FILE
instantiate a single instance of the class named in the C constant CRATE_MAIN_CLASS
invoke run( ARGV, ENV) on the newly instantiated class
exit

Now we dive into the Rakefile and put the final touches on our build to make everything come out the way we want. Currently crate only supports launching an application from a top level class. That means no Module::Class to use for the main class.

require 'crate'

PROJ_NAME = "htpasswd-ruby"
Crate::Project.new( PROJ_NAME ) do |crate|

  # setting these will set the appropriate C constants
  crate.main_file  = "application"
  crate.main_class = "App"
  crate.run_method = "run"

  # make sure 'application.rb' is packed into the databases
  crate.packing_lists << Crate::PackingList.new( Dir.glob("*.rb") )

end

We then have the application.rb file at the top of our build system directory. It is very simple and gives you an idea of how to put a thin ruby wrapper around any existing ruby application.

class App
  def run( argv, env )
    require 'htauth'
    HTAuth::Passwd.new.run( argv, env )
  end
end

Run the default rake task and take a look at the output.

% rake default
(in /Users/jeremy/tmp/htpasswd-ruby)
mkdir -p /Users/jeremy/tmp/htpasswd-ruby/dist
16:59:27  INFO: Packing amalgalite into /Users/jeremy/tmp/htpasswd-ruby/dist/lib.db
16:59:28  INFO: Packing ruby standard lib into /Users/jeremy/tmp/htpasswd-ruby/dist/lib.db
16:59:29  INFO: Packing ruby extension libs into /Users/jeremy/tmp/htpasswd-ruby/dist/lib.db
16:59:31  INFO: Packing project packing lists lists into /Users/jeremy/tmp/htpasswd-ruby/dist/app.db
16:59:32  INFO: Build htpasswd-ruby

% file dist/*
dist/app.db:        SQLite database (Version 3)
dist/htpasswd-ruby: Mach-O executable i386
dist/lib.db:        SQLite database (Version 3)

Our final result, a self-contained set of 3 files containing a statically compiled ruby interpreter and and app to run in it. In this case, we can deploy these 3 files to any i386 Mac OS X machine and run it, it will not use the system ruby. All the ruby libs are in the SQLite3 databases. And just to show that it works.

% cd dist/
% ./htpasswd-ruby -c test.db jjh
Adding password for jjh.
        New password: *****************
Re-type new password: *****************
% cat test.db 
jjh:GtPOQTplES6BE

Closing Thoughts

Well, there you have it; self-contained, statically-built, shippable ruby applications. There are definite improvements that can be made, and I would love to hear what people would like to do with Crate.

Some specific features that I will be adding in no particular order and as my copious free time allows:

cratify an existing gem for one-stop packaging
cross compilation targets so you can build, from one library, any executable for any target you have a cross compiler
installer wrappers, so you can ship
- .dmg for Mac OS X
- setup.exe for Windows
- rpm for CentOS/RHEL/Fedora/etc
- shar for general UNIX
smart dependency tracking so only those portions of the ruby stdlib that your application needs is packaged

Comments (View)

FFI Talk at MountainWest RubyConf 2009

2009-03-14T09:56:51-06:00

FFI Talk at MountainWest RubyConf 2009 (2009-03-14)

I presented my talk on FFI at MountainWest RubyConf yesterday. Lots of good comments from folks, thank you all. Here's all the links from my talk and a copy of the presentation on SlideShare.

Comments (View)

Writing SQL Functions in Ruby

2009-01-10T18:28:00-07:00

Writing SQL Functions in Ruby (2009-01-10)

I just released Amalgalite 0.6.0 with a couple of major enhancements. The biggest of which is the ability to write your own scalar and aggregate SQL functions in Ruby. You may not know them by that name, but that's the proper terminology according to SQLite.

I'm going to do a brief overview of what these function types are and then show how to write a scalar and an aggregate SQL function for SQLite using Amalgalite.

Scalar SQL Functions

Scalar SQL Functions are the functions used to do some sort of data transformation, or some test on a piece of data. They return a one value, based upon the input parameter(s). In the following statement the length() function is a scalar function.

SELECT name, length( name ) FROM country;

These pages give the complete list of built-in scalar functions in SQLite.

There are lots of functions in there that you probably recognize from general SQL usage: hex(), coalesce(), lower(), trim(), etc.

Aggregate SQL Functions

Aggregate SQL Functions are the functions used to do, well, aggregation of data across rows. They return a single value, based upon multiple rows of data. You use aggregate functions when you use GROUP BY SQL clauses. In this code example, the sum() function is an aggregate function.

SELECT country, sum( population ) FROM subcountry GROUP BY country;

Aggregate Functions lists the built-in aggregate functions for SQLite.

Writing a Scalar SQL Function

Sometimes you get to a point doing database work that you want your own scalar function, or you know of a scalar function in some other database that you want to use with Amalgalite. For instance the sha1() function is available in PostgreSQL and MySQL, so lets add it for use in our SQLite databases.

Functions are defined with respect to database instances in Amalgalite. They must be defined each time the database is opened.

require 'rubygems'
require 'amalgalite'
#
# Create a database and a contrived test table
#
db = Amalgalite::Database.new( ":memory:" )
db.execute( "CREATE TABLE test( filename, line_number, data)" )

#
# add some data
#
require 'open-uri'
db.prepare("INSERT INTO test( filename, line_number, data ) VALUES( ?, ?, ? )") do |stmt|
  lines = open("http://redmine.ruby-lang.org/repositories/entry/ruby-18/README.EXT?format=raw").readlines
  lines.each_with_index do |line, i|
    stmt.execute( "README.EXT", i + 1, line )
  end
end

#
# Define the sha1 SQL function
#
require 'digest/sha1'
db.define_function( 'sha1' ) do |x|
  Digest::SHA1.hexdigest( x.to_s )
end

And that is it. We just defined the sha1(t) SQL function for use in this database. We told SQLite that there is now a function available sha1 that takes a single argument x. Let's use it.

rows = db.execute("SELECT sha1('I love ruby')")
puts rows.first[0] # => d729b72eceebbc2c690ca9b586edd895d8bd4405

And use it on some actual table data too.

db.execute("SELECT data, sha1( data ) as sha1 FROM test") do |row|
  print "#{row['sha1']}: #{row['data']}"
end

Also look at the rdoc for Amalgalite::Database#define_function and Amalgalite::Function for other ways to implement a scalar function.

And a more complete example ships as examples/define_function.rb with Amalgalite.

Writing an Aggregate SQL Function

Aggregate functions are possibly more rare to write, but no less effective. An implementation of an aggregate function is different in that it has 2 methods and needs to save state. Therefore it is implemented as a class. Lets implement a wordcount(s) aggregate that counts the number of words.

# Implementation of the wordcount( s ) SQL aggregate
class WordCount < Amalgalite::Aggregate
    def initialize
      @name = 'wordcount'
      @arity = 1
      @word_count = 0
    end

    def step( str )
      @word_count += str.strip.split(/\s+/).size
    end

    def finalize
      return @word_count
    end
end

# define the aggregate in the database
db.define_aggregate( 'wordcount', WordCount )

We always define an aggregate as a class. The rules regarding the class are listed in the rdoc for Amalgalite::Aggregate.

The way it works is that for each row that is grouped into a single output row has the step() method called once, and then the finalize() method is called one time at the end. A new instance of the class implementing the aggregate is instantiated every time the aggregate is used in a statement.

row = db.execute("SELECT filename, wordcount( data ) as wc  FROM test GROUP BY filename").first
puts "#{row['filename']}: #{row['wc']}"  # =>  README.EXT: 4736
db.close

Also read the rdoc for Amalgalite::Aggregate and look at another example examples/define_aggregate.rb that ships with Amalgalite.

Now go define some crazy useful functions.

Comments (View)

Amalgalite Pthread Error

2008-07-05T23:58:52-06:00

Amalgalite Pthread Error (2008-07-05)

If you happen to get an error like the following when running amalgalite, it was a bug in how amalgalite was built. I wasn't matching the thread support in amalgalite with the threadsupport of the installed ruby.

amalgalite-0.2.0/ext/amalgalite3.so: undefined symbol: pthread_mutexattr_init

For future reference this is how you can see how ruby was compiled on my iMac

>> puts "#{RUBY_VERSION} #{RUBY_PATCHLEVEL} (#{RUBY_RELEASE_DATE}) #{RUBY_PLATFORM}"
1.8.6 114 (2008-03-03) i686-darwin9.2.2
>> require 'rbconfig'
>> puts Config::CONFIG['configure_args']
 '--prefix=/opt/local' '--enable-shared' '--mandir=/opt/local/share/man'
 '--enable-pthread' '--without-tk' 'CC=/usr/bin/gcc-4.0' 'CFLAGS=-O2'
 'LDFLAGS=-L/opt/local/lib' 'CPPFLAGS=-I/opt/local/include'
 'CPP=/usr/bin/cpp-4.0'

versus my OpenBSD server:

>> puts "#{RUBY_VERSION} #{RUBY_PATCHLEVEL} (#{RUBY_RELEASE_DATE}) #{RUBY_PLATFORM}"
1.8.6 110 (2007-09-23) x86_64-openbsd4.2
>> require 'rbconfig'
>>puts Config::CONFIG['configure_args']
 '--enable-shared' '--enable-ipv6' '--with-dbm-type=bogus'
 '--with-opt-dir=/usr/local' '--with-tcl-include=/usr/local/include/tcl8.4'
 '--with-tk-include=/usr/local/include/tk8.4' '--with-X11-dir=/usr/X11R6'
 '--prefix=/usr/local' '--sysconfdir=/etc' '--mandir=/usr/local/man'
 '--infodir=/usr/local/info' 'CC=cc' 'CFLAGS=-O2 -pipe'
 'CPPFLAGS=-DOPENSSL_NO_STATIC_ENGINE' 'YACC=yacc'

You can see that on my iMac ruby was compiled with @--enable-pthread@ but on my OpenBSD machine that configuration option is missing. I didn't take this into account when building Amalgalite.

Version 0.2.1 was just released which fixes this bug.

gem install amalgalite

Comments (View)