[DevOps:///OpsCode.Chef]
ate a weird meal last evening, chef was angry I think.....
When I created an AWS instance in same way (by swiss-'knife ec2 server create...' toolset) using same old boot-up script to get that insance auto-configured as chef-client; instance got created and was visible in the instance list but not available to my recipes trying to search for it using its applied role and other tags.
The same procedure has worked successfully for all previous time, and with no change it suddenly started failing.
I logged-in to the freshly created instance and exec 'chef-client --once' again, it had a successful run but still the recipe failed to grab the node by its role and tags.
So it was a perfectly performing instance according to its run_list, but not available to other features trying to grab its information over Chef Search mechanism.
Chef Search uses Solr Indexing service, which an enterprise-grade OpenSource search server based upon Apache Lucene.
It has a system service 'chef-solr' which acts as a wrapper around Solr to be run for Chef.
Though, this service was fine as I was able to search all earlier created nodes..... just the new additions were escaping the act.
Now, if Chef-Solr was working fine then we need to look its source for data. Obviously, it was working fine for already indexed data but didn't happen to index anything newly added.
Chef-Solr gets data passed on from the message queue of RabbitMQ, placed before to ease the load of several concurrent requests for Solr.
Chef-Exapander is a system-level service that fetches data from RabbitMQ's queue and formats them before passing on to chef-solr.
This combination of chef-exapnder and chef-solr acts as final feature of Chef Indexer.
So, I looked for chef-expander service.
It was in failing mode, restarted the service and bam!
We are back up and the new aditions are visible in Chef Search.
ate a weird meal last evening, chef was angry I think.....
When I created an AWS instance in same way (by swiss-'knife ec2 server create...' toolset) using same old boot-up script to get that insance auto-configured as chef-client; instance got created and was visible in the instance list but not available to my recipes trying to search for it using its applied role and other tags.
The same procedure has worked successfully for all previous time, and with no change it suddenly started failing.
I logged-in to the freshly created instance and exec 'chef-client --once' again, it had a successful run but still the recipe failed to grab the node by its role and tags.
So it was a perfectly performing instance according to its run_list, but not available to other features trying to grab its information over Chef Search mechanism.
What could be the problem?
The answer to it is hidden in knowing how Chef Search function.Chef Search uses Solr Indexing service, which an enterprise-grade OpenSource search server based upon Apache Lucene.
It has a system service 'chef-solr' which acts as a wrapper around Solr to be run for Chef.
Though, this service was fine as I was able to search all earlier created nodes..... just the new additions were escaping the act.
Now, if Chef-Solr was working fine then we need to look its source for data. Obviously, it was working fine for already indexed data but didn't happen to index anything newly added.
Chef-Exapander is a system-level service that fetches data from RabbitMQ's queue and formats them before passing on to chef-solr.
This combination of chef-exapnder and chef-solr acts as final feature of Chef Indexer.
So, I looked for chef-expander service.
It was in failing mode, restarted the service and bam!
We are back up and the new aditions are visible in Chef Search.
No comments:
Post a Comment