sphinx-doc failing for manpage builder

Sphinx-doc (http://sphinx-doc.org/latest/index.html) is a utility to create intelligent and friendly documentation with existing support for Python and C/C++ projects.

It can produce documentation in varied formats like html, LaTeX, epub, Texinfo, manpages and plaintext. It has lots of other features and multiple community created extensions to hyper-activate its usability.

To give it a quick try "Sphinx - First Steps" tutorial can be used, or a bit more with "sampledoc tutorial".

Here we will be discussing issue faced whilst creating manpages from a sphinx-documented project raising issue for "man" builder not being supported.

The solution is very simple but being a beginner (or external user) makes one doubt it.

System package version of Sphinx-doc for some of the distros is still not above v1.0 and the manpage support is just found in those.
So, to counter this error just don't use system package for it but instead use "pip install sphinx".

If you don't have pip already, follow this article: https://pip.pypa.io/en/latest/installing.html .

xopen : shell function to have verbose xdg-open : Mac's "open" alternative for Linux

I have been using xdg-open for sometime now, it's a utility similar to "open" utility popular among MacOSx users.

It enables you to open any file in the default "open with" program assigned to it's type. So, just passing any type of file to this utility would let you open it in the program it's supposed to open.

What is xdg-open?
source[1]: https://wiki.archlinux.org/index.php/xdg-open
source[2]: http://linux.die.net/man/1/xdg-open

There is just this little shell function, that makes your xdg-open usage a bit more verbose in case of errors (bad-syntax/ghost-file/ghost-program/open-failure) faced about the reason for it. And also shortens the access util name obviously.

xopen ()
{
    xdg-open "$@";
    _TMP_EXITCODE=$?;
    if [ "${_TMP_EXITCODE}" == "1" ]; then
        echo "[ERROR:] Error in command line syntax";
    else
        if [ "${_TMP_EXITCODE}" == "2" ]; then
            echo "[ERROR:] One of the files passed on the command line did not exist";
        else
            if [ "${_TMP_EXITCODE}" == "3" ]; then
                echo "[ERROR:] A required tool could not be found";
            else
                if [ "${_TMP_EXITCODE}" == "4" ]; then
                    echo "[ERROR:] The action failed";
                fi;
            fi;
        fi;
    fi;
    return $_TMP_EXITCODE
}

Fabric (py orchestration util) issue with my FreeBSD node, and Easy Fix

Recently I started implementing a Fabfile to deal with 7-8 different types of linux/bsd distributions at a time.

Now for the folks new to term "Fabfile", it's the key what-to-do guide for Fabric. Fabric is a fine Orchestration utility capable of commanding numerous remote machines in parallel. For the ruby community folks, you can call it Capistrano's Python sibling.

The way Fabric works is whatever you ask it to "run(<cmd>)" over remote nodes, it runs that command by passing it to " bash -l -c '<cmd>' ".

Now as the issue was with my FreeBSD machine is it didn't had "bash" in the first place. So, it was failing.

It can be fixed in 2 ways

First way,

add a line like following to make Fabric use the SHELL present on the remote node, as FreeBSD node had 'C Shell'... so

env.shell = '/bin/csh -c'

Second way,
install 'bash' on the remote nodes and make sure it's reachable via path "/bin/bash"

pkg_add -r bash 

ln -sf /usr/local/bin/bash /bin/bash

perform performance test using multi-mechanize

gave multi-mechanize a try recently, when in-need of behavior analysis for an application on massive concurrent requests being sent to it

What? multi-mechanize is an OpenSource load testing framework written and configurable in Python. So either make calls to any web service or simple import and utilize any python-accessible API/service.
It's successor of an old Python load testing framework called Pylot.

Developed and maintained @GitHub

Install available through 'pip' too as
   $ pip install multi-mechanize
It also requires matplotlib library if you wanna view Graphs in the generated HTML report (or otherwise), BUT it's not mandatory as the framework would perform all tests fine with an import error for matplotlib.

This also follows a bit convention oriented run structure. We'll see how.

Before starting any performance check user is supposed to create a project using

$ multimech-newproject project_name

This creates a dir 'project_name' at place of execution with following dir-structure
./project_name
├── config.cfg
└── test_scripts
    └── v_user.py

here './project_name/config.cfg' mainly contains global configuration around the duration to run,

[global]
run_time = 30                # second duration for test to run, required field
rampup = 0                   # second duration for users to ramp-up, required field
results_ts_interval = 10  # second time series interval for result analysis, required field
progress_bar = on         # console progress bar on/off, default=on
console_logging = off    # console logging to standard ouput, default=off
xml_report = off            # xml/jtl report generation, default=off
# results_database = sqlite:///results.db ## optional component to push results to DB
# post_run_script = do_whatever.py ## to run any script to do anything on test completion 

[user_group-1]
threads = 3                  # number of threads to run the following script in
script = v_user.py        # the actual script that will be run to test, give any 

## similar more users with same/different threads and script values can be added with names like user_group-ANYTHING.

Now, to run this prepared project

$ multimech-run ./project_name

As I already mentioned it picks certain implementations by convention, most important to notice is the way user-group-script is to be prepared.
Basically it's to be a python script that can utilize all Python magic you know or install. But it need to have a 'Transaction' class with 'run' method.
When you run your project using multi-mechanize, it instantiates 'Transaction' class and keeps calling 'run' method in a loop...

Example user-script to test load at your 'python -m SimpleHTTPServer'

# v_user_server.py
import httplib


class Transaction(object):
    def run(self):
        conn = httplib.HTTPConnection('127.0.0.1:8000')
        conn.request('GET', '/')
        resp = conn.getresponse()
        conn.close() 

        assert ((resp.status / 400) == 0), 'BadResponse: HTTP %s' % resp.status
# finito file

Find the detailed scripting guide with plenty good examples here : http://testutils.org/multi-mechanize/scripts.html

pycallgraph : usage example

'pycallgraph' is a fine python utility enabling developers to prepare call graph for any python code piece... it also notifies the number of times a call being made and total time spent in it

Homepage: http://pycallgraph.slowchop.com/pycallgraph/wiki
Codebase: https://github.com/gak/pycallgraph

it require 'graphviz' (http://www.graphviz.org/) to be present on the same machine for image generation from the analyzed calls data...

install : $ pip install pycallgraph

Usage#1 Selective graphing

wrap the code to be graphed as follows...

import pycallgraph
pycallgraph.start_trace()
fetch() # logic to be graphed
pycallgraph.stop_trace()
just_log() # logic NOT to be graphed
pycallgraph.start_trace()
process() # logic to be graphed
pycallgraph.stop_trace()
pycallgraph.make_dot_graph('path_to_graph.png')

Usage#2 Decorator

Import the decorator method and place the decorator over any method where call graph is required...
import this file wherever you require the @callgraph decorator and use it

Decorator Python code. Sample code with decorator usage and selective trace usage are at end of this blog... from this gist: https://gist.github.com/abhishekkr/5592520



pycallgraph.start_trace() method lets you pass a filter function to not graph some modules on purpose, say in nova I don't wanna graph all the dependency libraries magical calls...


STATUTORY WARNING: Graph will be huge for a very high level call and probably unreadable due to overworked image generator. Use it wisely on burnt areas.


Call Graph images for code sample in Gists

for call with decorator



for call with selective trace

console and json

recently Alan posted a very nice aticle around prettifying json, reminded me of <this draft>... he posted 2 out of 3 utilities I was gonna mention... so here is 3rd and the shell profile way to use first 2

$ sudo wget -c -O /etc/profile.d/a.json.sh https://raw.github.com/abhishekkr/tux-svc-mux/master/shell_profile/a.json.sh

it contains 2 functions available at shell

# usage example:
# $ json_me 'echo {"a": 1, "b": 2}'
# $ json_me 'curl http://127.0.0.1/my.json'
json_me(){
bash -c $@ | python -mjson.tool
}

# requirement: $ pip install pjson
# usage example:
# $ pjson_me 'echo {"a": 1, "b": 2}'
# $ pjson_me 'curl http://127.0.0.1/my.json'
pjson_me(){
bash -c $@ | pjson
}

3rd utility is 'jq', an awesome utility which prettifies an perform sed-like operations on json. This tutorial describes all its magical powers.

make rvmsudo using running user's env values

I was automating my personal linux box setup in opscode chef, to ease my life hence-forth and reach practice what I preach state...

While setting-up the 'development' tools recipe, automating the 'nodejs' set-up... due to missing reliable yum repository, decided on using 'nvm' (it's for nodejs as rvm is for ruby).

to utilize 'nvm' to install/manage 'nodejs', need to install it first

$ curl https://raw.github.com/creationix/nvm/master/install.sh | sh

How install.sh here works is, it calculates NVM_TARGET="$HOME" to handle the location for '.nvm' directory cloned from the nvm git repo, where the entire nodejs environment lives.

I had the install command as execute with the user value referred to the dev-user of my choice.

While running 'rvmsudo chef-solo ....', it picks desired user but because of $HOME inference in the 'nvm/install.sh' still the HOME value got picked for '/root'. It messed up the situation.

To fix it, and any certain issues around permission which might occur... using

$ rvmsudo USER=$USER HOME=$HOME chef-solo -j....

it's working with desired output.

Though, I need to correct my nodejs set-up by pushing the required RPM to my yum reposiotry.

But, this 'rvmsudo' trick can be used for any similar scenarios where you need sudo privileges for rvm but desire to provide current users' environment values.

Apache httpd VirtualHosts : one gets default, unknown faults

Recently faced a situation where even after removing a VirtualHost, its ServerName was giving HTTP 200 response. It was all because of missed RTFM agenda.

When VirtualHosts get applied in Apache HTTPD server configuration, the first definition encountered by Apache Controller gets selected as the default route logic selected if the ServerName doesn't match any provided.

To get an explicit _default_ provider, one of the vhost definitions need to be told so... as in last configuration file piece.

~

~

foodcritic rake task ~ one that works for me

FoodCritic (a lint tool for your OpsCode Chef cookbooks)
$ gem install foodcritic --no-ri --no-rdoc

Rake Task to get the FoodCritic rolling at your cookbooks
(here cookbook reside in dir 'cookbooks' at the root of directory with Rakefile)
the one at wiki doesn't work for me, but this does

_

varnish ~ sometimes fails but don't tell

Varnish-Cache

We were playing around with the automation around famous Varnish-Cache service and stumbled upon something... the silent killer style of Varnish Cache.

The automated naming was such that it was appending different specific service-node host-names and service-names to form backends and then load balance onto them using "director ... round-robin" configuration. It was creating names that way to avoid name collision for same service running on different nodes load-balanced by Varnish-Cache.

We checked for the configuration correctness

$ varnishd -C -f /my/varnish/config/file

It passed.

We started the Varnish service

$ /etc/init.d/varnish start

It started.

We tried accessing the services via Varnish.

It failed saying there no http service running at Varnish machine:port.

Now what. Can't figure out anything wrong in configuration VCL. So, start looking at logs.
Tried starting varnishlog service

$ service varnishlog start

This failed giving error about _.vsm being not present, which was actually present and with right user permissions.

Then a colleague of mine suggests he has faced such issue before due to extremely long backend names.

I extremely shortened the backend name and it started working.

So, the length there does effect but the VCL gives no error when starting Varnish-Cache.

BTW, from the checks performed... the maximum character backend name working for the configuration was 44 Character long.

Project::Inception.include? Infrastructure['base_architecture']

requirements change.....

Requirements are supposed to change, else there will be no market.

But when you are involved with the client to formulate the Project which has to be 'developed and released', you try to cover all the important specs related to 'development and release'.

In this era, mostly people can be seen to opt for IaaS, PaaS solutions for their production release (and for development depending on budget).
There are lots of Organizations, who have had their own SysAdmin Team for years for years and have more faith on them for cheaper, better, secure, and/or conrolled environment.
Then there are also projects that need to be on internal organizational networks due to some internal boxe-d service or company policies.

Either way, initially there can be two situations. They will have an idea about where they want it implemented, or not.

If they decide, you know better. It's better. You come up with the best suited solution respective to projected requirement. And get a yes from them.

If they decide to drive the decision. Even more better. Now analyze the solution, and project the respective pros, cons and the patch-y aid that will be required. Get them to agree with approach that wouldn't make the project-life a hell.

It needed to be done at start, once you start with project..... these will only slow you down and 'slum your time' once you are deciding it post-project-inception.

Creating RPM of MCollective for Ruby1.9

Note: With little tweaks comes major pre-requisite checks.

Clone the latest branch from GitHub repo for MCollective
$ git clone git://github.com/puppetlabs/marionette-collective.git

Removing Ruby 1.8 version specification from RedHat Spec
$ sed -i 's/.*ruby.abi.*//g' ext/redhat/mcollective.spec

from the changes 11/Jun/2011 : marionette-collective/commit/ba86f7762d

the lines removed by above command are

BuildRequires: ruby(abi) = 1.8

Requires: ruby(abi) = 1.8

Removing Rubygem related specification from RedHat Spec
$ sed -i 's/.*rubygem.*//g' ext/redhat/mcollective.spec

from the changes 11/Jun/2011 : marionette-collective/commit/ba86f7762d

the lines removed by above command are

Requires: rubygems

Requires: rubygem(stomp)

create the RPMs, generating this would require you to be on a RedHat-base system with packages like ruby, rubygem rake, redhat-lsb (& rpm-build, I think) installed

$ rake rpm

The checks you need:
On the machine where you install the created rpms..... remember to get ruby1.9.3 and rubygem stomp already installed.

Can get older created MCollective 1.3.2 RPMs friendly with Ruby 1.9 at http://yum-my.appspot.com/flat_web/index.htm .

The latest branch will give you version 2.0.1 .

quick PuppetMaster Service Script for gem installed puppet set-up

On easy rubygem-way `gem install puppet` installation method for Puppet doesn't get you the *nix platforms service script... so here is one allowing you to perform Start||Stop||Restart||Status service task for puppetmaster.

• Save the file below as '/etc/init.d/puppetmaster'
  $ sudo curl -L -o /etc/init.d/puppetmaster https://gist.github.com/raw/2479100/15f79c68be3f6f6bf516adf385aac1f29f802a45/gistfile1.rb
• and turn on its eXecutable bit
  $ sudo chmod +x /etc/init.d/puppetmaster
• Now you can use it as
  $ service puppetmaster status

PuppetMaster Service Script

[Puppet] Exported Resources is a beautiful thing..... thing to use and improvise

Lately, I've been real blunt towards Puppet because of the soup that leaked some specific scenario flaws in very busy times.
So, it's my duty to applause if I like something which is a novel and beautiful concept.

For a well organized auto-magically managed set-up apart from a fine infrastructure and its configuration management mechanism, a very important part is for the monitoring and logging solution to spread across the infrastructure in a similar seamless and scalable fashion.


Puppet enables it very finely with the use of exporting and collecting resources.

Exported Resources are super virtual resources.
Once exported with the storeconfig setting being true, the puppetmaster consumes and keeps these virtual resources out available for all hosts.

Read in detail: http://docs.puppetlabs.com/guides/exported_resources.html

An example from the link above, using a simple File resource

node a {
  @@file { "/tmp/foo":
      content => "fjskfjs\n",
      tag => "foofile",
  }
}
node b {
  File <<| tag == 'foofile' |>>
}

It'll have its flaws (as all softwares do, mine and others)..... just hope they don't interfere in my use-cases.

MCollective can't handle Puppet ~ just like psychotic love stories

Some puppet-izers didn't like what I said in my last post... still a mess is a mess no matter how worth it might be.
Past 2 months, I've been in pain due to the psychotic love story of MCollective and Puppet.

YES, they are very helpful products to automate configuration management and orchestrate metadata-based multicast-ed actions.

YES, they are now under the same organization PuppetLabs which is whole-heartedly working to improve them so they could retain their status in the started-to-glamour-izing DevOps domain. So, both of them will improve a lot.

But, first of all.
If you don't properly test your corporate-aiming projects over Ruby1.9.x; please do post a big notice on your projects page or at least on first page of your amazing Doc.
My story for past few weeks:
I start using a project..... oh it's failing, debug..... oh it's still failing..... debug.
Ahhhhh, ok ask at IRC.
WHAT? I'm using Ruby1.9.x so its possible I might be facing *some* issues.
Is this is a Joke, I'm trying to stabilize an important project over here.
At least be truthful, and don't behave like TV Commercials with small+quick Disclaimers.

And Again.
I'm managing MCollective over all my instances via Puppet. Obviously, because that's what Puppet is for..... managing state of instances.
I've a CI which instigates MCollective to orchestrate different actions over my infrastructure. That's why I took the pain to get the changing-but-not-releasing MCollective over my infrastructure.


Now, this also involves MCollective firing up Puppet in non-daemonizing mode.
This used to raise an Exception in MCollective due to un-handled 'nil' return value failing up my entire build.
I go on IRC, again trying to get a solution..... I get to know that even this has been fixed later in the git-repo since I rpm-ed it last time. But I also get an answer that this wouldn't work as Puppet no-daemon mode has problems. And if I wanna follow this approach, its my decision.
First of all, what the hell should I do when there is just one straight approach left. Except for if I start using something else to manage MCollective.

Though, I took the latest pull..... rpm-ed it and updated all my nodes. And the Puppet non-daemon problem has not yet been raised.
But, what kind of response is it.
If it wasn't for the team decision..... I'd have changed it overnight x(

Puppet ain't sweet anymore & Marionette-Collective hopeful

its a fine piece of utility working mostly fine, but my luck..... mostly end up getting blocked in my tasks by the flaws in library/utility/framework :(

Puppet ain't sweet.

Puppet is one of the most famous automated configuration management, built upon ruby enforcing just it's DSL to be used to use the power underneath.

(new to it, wanna know more http://puppetlabs.com/puppet/how-puppet-works/)

MCollective is a novel concept with good technical design (which can be still matured, though)  for parallel server orchestration with a filter-enabled broadcast of request distribution.

(new to it, get a glimpse at http://puppetlabs.com/mcollective/introduction/)

Why? What happened? Puppet is Good.
Puppet has been there in the wild for a long time (at-least more than 5yrs) now as compared to the closest competitor to it 'Chef'.
Saying it out-loud the feel I'm getting using them lately ~ as if it is acting like an old pop-artist trying to put out a lot at-once to maintain their superiority over upcoming rockstars.
I've used their older service before which wasn't much glossy (at design level) but a lot more stable and composed at implementation..... which is more important from the perspective of a system-maintaining utility.

When you are always in pace of bringing changes and improving upon your current set-up, you don't want to keep on getting blocked by quickly adapted & recently obsoleted features.

Why do you want me to place 'pluginsync = true' on all the boxes to get this new feature working. Why can't enabling it over PuppetMaster should take care of it. In what kind of consistent system does you require custom-resource-type to be not recognizable over all clients. If you don't wanna it get used over any node, you wouldn't use it there. Plain Simple *hit.

You set-up a new puppet-master at Client side and find out your old, working, correct manifest from tested set-up suddenly start failing. Why? Because the freaking parser is broken and not able interpret the symbol notation anymore. Move onto value-convention and it is working.
Ok, you smashed your head and got that working. Now say you have learnt your lesson of not trusting the newer versions and Gemfile-d it for the bundler to handle. Suddenly it starts failing 'cuz you find out the newer version exists no-more. It was yanked off.

Some of there new suggested approaches in custom facter require Facter v 1.7.0; the gem available up their is still v1.6.5. What? Legacy fact distribution is not supported in the new distributions and is obsolete.

There are also several other design-level implementation which I don't agree and sometime don't agree at all. But, not considering them all, there is too much chaos in Puppet right-now.

What do they expect, to keep on pulling the latest build from repository..... building its rpm/gem and then using it. In what freaking world do you expect this from a platform which has been built to sustain a deterministic state of machine.

MCollective is mostly good, a light of hope that Puppet adopted to lit its gloomy days. But to find out its not properly tested over Ruby1.9.x gets me worried.

resque and WildCard usage

While testing we were starting resque workers as

 rake resque:work QUEUE=* 

this used to start the workers listening on all the Resque's queues and never exit.

So hoping Resque understands wildcards well, we made

 rake resque:work QUEUE=q01.* 

a part of our automated configuration tasks to start all our queues within Resque's queue starting with name q01 like q01.rss and q01.atom.

These queues were getting started fine and displayed well on Resque's WebUI.
But on scheduling a task for these queues, the worker started with q01.* wasn't able to pick up the queued task for neither q01.rss nor q01.atom.

Investigating the Resque codebase the understanding we developed was just plain '*' is separately handled out to return entire array of all available queues and is acted upon. But it doesn't handle Wildcards.

So, we started it as

 rake resque:work QUEUE=q01.rss 
rake resque:work QUEUE=q01.atom

and it worked fine.

Don't do the same mistake.

if you don't already know what is redis and resque:

Resque : a famous opensource ruby library to create background workers on multiple queues and processing them when any task is available
Redis : it's a popular opensource project for in-memory key value data-store 
(why were you reading then actually)

is 'Splunk' eating up your disk space

if you have been using default set of splunk configurations, soon you could also face your entire disk space being filled with splunk's database.....

you can keep a check over that by lowering down its upper-limit over database indices size from several 100s 0f 1000s MBs ((default maxTotalDataSizeMB per index is 500Gigabytes)) to the desired/affordable Size in MBs.


 
File: /var/ebs/splunk/etc/system/local/indexes.conf

maxTotalDataSizeMB = 3000

managing index size in Splunk is better covered at
http://blogs.splunk.com/2011/01/03/managing-index-sizes-in-splunk/

[OpsCode Chef] when chef's changes can be re-edited but un-available to Search

[DevOps:///OpsCode.Chef]

ate a weird meal last evening, chef was angry I think.....

When I created an AWS instance in same way (by swiss-'knife ec2 server create...' toolset) using same old boot-up script to get that insance auto-configured as chef-client; instance got created and was visible in the instance list  but not available to my recipes trying to search for it using its applied role and other tags.

The same procedure has worked successfully for all previous time, and with no change it suddenly started failing.

I logged-in to the freshly created instance and exec 'chef-client --once' again, it had a successful run but still the recipe failed to grab the node by its role and tags.
So it was a perfectly performing instance according to its run_list, but not available to other features trying to grab its information over Chef Search mechanism.

What could be the problem?

The answer to it is hidden in knowing how Chef Search function.

Chef Search uses Solr Indexing service, which an enterprise-grade OpenSource search server based upon Apache Lucene.


It has a system service 'chef-solr' which  acts as a wrapper around Solr to be run for Chef.
Though, this service was fine as I was able to search all earlier created nodes..... just the new additions were escaping the act.

Now, if Chef-Solr was working fine then we need to look its source for data. Obviously, it was working fine for already indexed data but didn't happen to index anything newly added.

Chef-Solr gets data passed on from the message queue of RabbitMQ, placed before to ease the load of several concurrent requests for Solr.
Chef-Exapander is a system-level service that fetches data from RabbitMQ's queue and formats them before passing on to chef-solr.

This combination of chef-exapnder and chef-solr acts as final feature of Chef Indexer.

So, I looked for chef-expander service.
It was in failing mode, restarted the service and bam!
We are back up and the new aditions are visible in Chef Search.

am an MVB..... now, officially

have been walking along-with the technology and blogging about it.....

few days back, I got a mail inviting to become a part of MVB (Most Valued Blogger) program at DZone..... we finished the formalities and now, officially I'm an MVB.

feels good after having a look at some of other members from MVB http://www.dzone.com/page/mvbs