Fabric (py orchestration util) issue with my FreeBSD node, and Easy Fix

Recently I started implementing a Fabfile to deal with 7-8 different types of linux/bsd distributions at a time.

Now for the folks new to term "Fabfile", it's the key what-to-do guide for Fabric. Fabric is a fine Orchestration utility capable of commanding numerous remote machines in parallel. For the ruby community folks, you can call it Capistrano's Python sibling.

The way Fabric works is whatever you ask it to "run(<cmd>)" over remote nodes, it runs that command by passing it to " bash -l -c '<cmd>' ".

Now as the issue was with my FreeBSD machine is it didn't had "bash" in the first place. So, it was failing.

It can be fixed in 2 ways

First way,

add a line like following to make Fabric use the SHELL present on the remote node, as FreeBSD node had 'C Shell'... so

env.shell = '/bin/csh -c'

Second way,
install 'bash' on the remote nodes and make sure it's reachable via path "/bin/bash"

pkg_add -r bash 

ln -sf /usr/local/bin/bash /bin/bash

varnish ~ sometimes fails but don't tell

Varnish-Cache

We were playing around with the automation around famous Varnish-Cache service and stumbled upon something... the silent killer style of Varnish Cache.

The automated naming was such that it was appending different specific service-node host-names and service-names to form backends and then load balance onto them using "director ... round-robin" configuration. It was creating names that way to avoid name collision for same service running on different nodes load-balanced by Varnish-Cache.

We checked for the configuration correctness

$ varnishd -C -f /my/varnish/config/file

It passed.

We started the Varnish service

$ /etc/init.d/varnish start

It started.

We tried accessing the services via Varnish.

It failed saying there no http service running at Varnish machine:port.

Now what. Can't figure out anything wrong in configuration VCL. So, start looking at logs.
Tried starting varnishlog service

$ service varnishlog start

This failed giving error about _.vsm being not present, which was actually present and with right user permissions.

Then a colleague of mine suggests he has faced such issue before due to extremely long backend names.

I extremely shortened the backend name and it started working.

So, the length there does effect but the VCL gives no error when starting Varnish-Cache.

BTW, from the checks performed... the maximum character backend name working for the configuration was 44 Character long.

is 'Splunk' eating up your disk space

if you have been using default set of splunk configurations, soon you could also face your entire disk space being filled with splunk's database.....

you can keep a check over that by lowering down its upper-limit over database indices size from several 100s 0f 1000s MBs ((default maxTotalDataSizeMB per index is 500Gigabytes)) to the desired/affordable Size in MBs.


 
File: /var/ebs/splunk/etc/system/local/indexes.conf

maxTotalDataSizeMB = 3000

managing index size in Splunk is better covered at
http://blogs.splunk.com/2011/01/03/managing-index-sizes-in-splunk/

[dev-ruby] 'gherkin' need a native install AND I ported entire jruby dir

When I started working on 'derailed_cuke', I was newbie at 'Cucumber' and it's dependency list included 'Gherkin'... I was using JRuby.

[[ derailed_cuke ]] 
this project is a simple way to use cucumber tests independent of any Rails using Ruby's Rake.

Due to some reasons... I copied this project on another machine along-wth entire JRuby-Pack from older machine.
Now, this JRuby-Pack of mine from older machine already had all the gems installed in it and was kin'of a portable setup with script declaring environment variables for '$JRUBY_HOME' & '$JRUBY_HOME\bin' to be added to $PATH.

I setup JRuby-Pack & started the application 'derailed_cuke'. There was the error for gem 'gherkin' of not being present.
But, I had it installed on earlier machine when I got Cucumber installed.

I called in 'gem install gherkin' again to make-up for whatever have been missing.
Installation Successful. But still a 'gherkin' error was there. Running 'gem list' showed that now I had two versions of gherkin installed.

Looking a more into issue, resolved it as:

[*] the newest version just installed wasn't desired by Cucumber,

[*] the version desired by Cucumber was copied from another box and,

[*] 'gherkin' requires a 'Native Install' 

*** thus do a clean 'bundle install' to avoid such newbie mistakes

So, doing a native install of specific 'gherkin' gem was required.