Ruby/Rails utilizing Solr in Master/Slave setup

Apache Solr is a high-performance enterprise grade search server being interacted with a REST-like API, documents are provided to as xml, json or binary. It extends Lucene search library.

Ruby/Rails communicates with this awesome search server using Sunspot (sunspot, sunspot_rails gem) library, to do a full-text search. Nice tutorial to use Solr in your Rails project using Sunspot.

Solr can perform as a standalone search server and even as master/slave instances collaborating with each other, with slaves polling master to sync-in their data.

Solr instances can be configured like Slave and as a Master.
Configuration file 'solrconfig.xml' needs to be edited with a new RequestHandler for Replication configured:

 

{for Slave} :: to either poll at their Master's machine address at regular intervals

{for Master} :: or commit the changes and clone the mentioned configuration files when Slave asked for it 

for detailed reference:http://wiki.apache.org/solr/SolrReplication 

for optimizaed ways using ssh/rsync based replication:
http://wiki.apache.org/solr/CollectionDistribution

Now, here you can even use a single configuration file with 'Replication' node having fields for both Master and Slave with an 'enable' child-node with possible values 'true|false' set as per requirement for Master & Slave nodes.

Sunspot dealing with a single Solr instance, gets $ProjectRoot/config/sunspot.yml

production:
  solr:
    hostname: standaloneSolr.mydomain.com
    port: 8983
    log_level: WARNING

Sunspot dealing with a master/slave Solr set-up, gets $ProjectRoot/config/sunspot.yml

production:
  solr:
    hostname: slaveSolr.mydomain.com
    port: 8983
    master_hostname: masterSolr.mydomain.com
    master_port: 8983
    log_level: WARNING
  master_solr:
    hostname: masterSolr.mydomain.com
    port: 8983
    log_level: WARNING

If you have more than one slaves, they need to be handled by a load-balancer and the DNS Entry for that Load Balancer comes here in slave's hostname field.

Also, the fields 'master_hostname' and 'master_port' below 'solr:' are not mandatory and supposed to be referred from 'master_solr:' block. But, it has been observed in some cases that mentioning them explicitly avoids non-picking of configuration.

By default, Sunspot configures Ruby/Rails application to Write-Only to Master and Read-Only from Slave.

explicit update SASS in Rails Application

Sass (Syntactically Awesome StyleSheets)?
Available @ http://sass-lang.com/
Is a fine way to produce clean and easy CSS code using a meta-scripting-language. It has feature for developer ease like variables, nesting, mixins, and selector inheritance.

If you are somehow using Sass in your Ruby-On-Rails application and using the default Webrick to serve the web content, it automatically updates the "public/stylesheets/*.css" files from your styling meta-script in "public/stylesheets/sass/*.css".

But, if you are using any other web-server (like, thin)..... in that case you'll have to get your CSS files to be explicitly updated using 'sass --update' action.

$ bundle exec sass --update 
--watch public/stylesheets/sass:public/stylesheets


So, include it in your deploy Rake Tasks and get updated stylesheets everytime without fail.

mysql-server retained old credentials, got work-around but no solution

While trying re-configuration of an existing  mysql-server over CentOS, tried 'yum remove' the mysql-server and then again 'yum install' it.

When I tried setting up a new password for 'root' using 'mysqladmin', it raised an error. Some random troubleshooting showed it still had earlier-installation's root credential working for it.

Trying some more stuff, I manually set 'old_password=0' in '/etc/my.cnf' and then tried re-installing. It still had the earlier password working for it.

For time being, got a work-around fixing the problem... 

$ yum erase mysql mysql-server
$ rm -rf /var/lib/mysql
$ yum install mysql mysql-server
$ service mysqld restart 

The location of user's information i.e. the user table data resides in '/var/lib/mysql/mysql/user.MDY'; but just removing that only file wouldn't work because it don't get recreated on re-installation if entire directory structure is present.

A service restart is required to create a vanilla copy of '/var/lib/mysql'. 

Still... the issue exists of why to do it manually.

The incidence is under discussion at the link below. If you have a non-hackish solution for the problem or could spot the actual mistake, please reply there.

http://stackoverflow.com/questions/6606869/mysql-retains-password-of-earlier-installation-even-after-proper-yum-remove

cucumber's selenium test failed and firefox ran blank..... OOOPS, version in-compatibility

I faced my cuke test failing a while back, and recently saw some of my colleagues re-checking their environment settings, cucumber installations and tweaking their cuke-features in different ways..... and I thought I missed spreading a major sports-page news for all headline cuke readers which might have been wasting a lot of time of a lot of people just because the word is not well spread.

Newer Versions of Firefox

are not friendly with 

in-trend Selenium Drivers

So, if you have direct selenium tests failing, or write cucumber/capybara behavioral tests but seek the cause for failure of correct behavior... lookout for a version downgrade for your firefox from whatever 4.x, 5.x or higher Beta versions you have to any of 3.x versions.
And in case of being the victim of same in-compatibility, you would see your tests GREEN again.

Tech-Xpress-Guide so far... from learnings of year 2010

GitHub Repo ~ https://github.com/abhishekkr/a-techXpress-guide

As in what I picked up nicely last year, I did Uploaded few How-To Express Guide in past few months on my SlideShare page ~
http://www.slideshare.net/AbhishekKr/documents

Tech-Xpress-Guide so far ~

[] using SNMP  for secure remote resource monitoring ~
http://www.slideshare.net/AbhishekKr/an-express-guide-ltlt-snmp-for-secure-rremote-resource-monitoring

[] using Nagios for quick & efficient IT Infrastructure monitoring
http://www.slideshare.net/AbhishekKr/an-express-guide-ltlt-nagios-for-it-infrastructure-monitoring

[] using Cacti for IT Infrastructure monitoring with nice analytic graphing
http://www.slideshare.net/AbhishekKr/an-express-guide-cacti-for-it-infrastructure-monitoring-graphing

[] using Zabbix for IT Infrastructure monitoring with easy-to-use Web UI
http://www.slideshare.net/AbhishekKr/an-express-guide-zabbix-for-it-monitoring

[] using DummyNet to mock different Network latencies & bandwidth for testing or other purpose
http://www.slideshare.net/AbhishekKr/an-express-guide-dummynet-for-tweaking-network-latencies-bandwidth

[] creating and using Solaris Native Zones {similar to Linux LXC but much efficient}
http://www.slideshare.net/AbhishekKr/a-tech-xpressguidesolariszonesnativenlxbranded

[] Ethernet Bonding in *Nix for Load Balancing
http://www.slideshare.net/AbhishekKr/a-tech-xpressguideethernetbondingfornics

[] setting up Squid Cache Proxy service
http://www.slideshare.net/AbhishekKr/a-tech-xpressguidesquidforloadbalancingncacheproxy

[] setting Syslog Centralization for *nix machines
http://www.slideshare.net/AbhishekKr/a-tech-xpressguidesyslogcentralizationloggingwithwindows

[dev-ruby] "JRubyOnRails Blames ActiveRecord" *ing: culprit Cucumber, inspector JConsole

I was working on JRubyOnRails project, where we used Cucumber in providing a web-app feature that required functional testing of some web portals.

Scenario: 
     In this Rails application, 'Cucumber' tests for different web-portals are invoked every scheduled interval using 'derailed_cuke' and then the test results are processed and saved in a MySQL database.

Gems Used:
     #for Rails3
          'rails', '3.0.3'
     #for MySQL ActiveRecord
          'activerecord-jdbcmysql-adapter', '1.0.2'
          'activerecord-jdbc-adapter', '1.0.2'
          'jdbc-mysql', '5.0.4'
     #for Cucumber
          'cucumber', '0.10.0'
          'cucumber-rails', '0.3.2'

          'gherkin', '2.3.3'

     #with JRuby version 1.5.5

Exception:

• After starting Rails Server and waiting for some non-specific time, an exception started arising mentioning ActiveRecord and an Heap OutOfMemory Exception being raised.
• I checked for my database services, schema & content; all was as expected and even working fine from 'rails console'.

Action:

• My bad old debug-style... the first thing I did was inserted 'print STATUS_MSG' at all suspected locations for ActiveRecord.
• It worked :), and the location sorted out was the place where cucumber-tests' processed results was saved in database.
• So now, 'Cucumber' was under suspicion radar... I google-d over recent incidents including both 'OutOfMemory Exception' and 'Cucumber', and there were few blogposts about similar problem of 'memory leakage' in Cucumber.

     I had a JConsole window already fired-up and profiling the Memory level changes
     in my JRubyOnRails appication (one of the major benefit of using JRuby).

• So, I started tweaking the Cucumber test's schedule interval to a higher and lower value.
  This clearly showed it's impact over the Memory Profiling by JConsole and the relativity of duration in which the Exception was being raised.

Solution:

     Until the Cucumber flaw is corrected, the work-around used was

        [] pushing in some more RAM
        [] using JVM Parameters to increase the Heap Size used by Rails application
         #jruby -J-Xmx2500m -J-Xms2500m -J-Xmn512m -S rails s
             here,
                  -J-Xmx2500m ~ means maximum Java Heap Size increased to 2500m
                  -J-Xms2500m ~ means initial Java Heap Size increased to 2500m
                  -J-Xmn2500m ~ means eden Java Heap Size increased to 500m
                  for server-side applications, Xmx & Xms is preferred to be kept same

[dev-ruby] 'gherkin' need a native install AND I ported entire jruby dir

When I started working on 'derailed_cuke', I was newbie at 'Cucumber' and it's dependency list included 'Gherkin'... I was using JRuby.

[[ derailed_cuke ]] 
this project is a simple way to use cucumber tests independent of any Rails using Ruby's Rake.

Due to some reasons... I copied this project on another machine along-wth entire JRuby-Pack from older machine.
Now, this JRuby-Pack of mine from older machine already had all the gems installed in it and was kin'of a portable setup with script declaring environment variables for '$JRUBY_HOME' & '$JRUBY_HOME\bin' to be added to $PATH.

I setup JRuby-Pack & started the application 'derailed_cuke'. There was the error for gem 'gherkin' of not being present.
But, I had it installed on earlier machine when I got Cucumber installed.

I called in 'gem install gherkin' again to make-up for whatever have been missing.
Installation Successful. But still a 'gherkin' error was there. Running 'gem list' showed that now I had two versions of gherkin installed.

Looking a more into issue, resolved it as:

[*] the newest version just installed wasn't desired by Cucumber,

[*] the version desired by Cucumber was copied from another box and,

[*] 'gherkin' requires a 'Native Install' 

*** thus do a clean 'bundle install' to avoid such newbie mistakes

So, doing a native install of specific 'gherkin' gem was required.