Just the ramblings of a LAMP DevOps

blog.robert.mcfrazier.com

Category Archives: Hadoop

Cassandra and Hadoop and Hive, Oh my! (part 2)

Posted on by Posted in Cassandra, Hadoop, NoSQL 3 Comments

This is part two to my previous post about adding Cassandra, Hadoop and Hive to the infrastructure at work. We are building a new web based application and wanted to add features that the current infrastructure would not support.

We needed to get our web and database development teams trained on Cassandra, we used DataStax Training and had them onsite to not only deliver training but to answer questions and give guidance on how to setup our column families.

Training

20130430_101428
We had a onsite 2 day class delivered by Tupshin Harper from DataStax. He did an excellent
job of getting us up to speed, and answering all questions about NoSQL basics, how to attack some of the problems we are looking to solve, and how to use Hive/Pig.

Infrastructure

Our initial setup was development and QA environment single machine installations of Cassandra. This allowed us to start developing and getting used to how to work with Cassandra, CQL and Fluent Cassandra (we are a .Net shop, expanding into using more open source tools), without having to worry about maintaining a multiple instance setup.

We are now building the staging and production environments both of which have multiple nodes in the clusters. Another wrinkle we had to overcome was the staging and production environment are locked down, so I did not have internet access on those instances to perform a typical install of DataStax Enterprise. To solve this problem we created an internal YUM repository, and mirrored the RedHat, and DataStax repos. Now we can use YUM to install and solve all of the dependency issues. Doing the install this way we did have to manually edit a few files, generate node tokens by hand, and bring the cluster up first then use OpsCenter to install the opscenter-agent on each node.

Cassandra 1.2 for DataStax Enterprise

Cassandra 1.2 for DataStax Enterprise, once it is released, brings virtual nodes and cql3, which has a binary protocol, and Collection columns (collections set, list and map). Features that we are prepared to take full advantage of once they are released to DataStax Enterprise.

PHP client for Impala

Posted on by Posted in Hadoop, NoSQL, PHP 31 Comments

I was reading about Impala, a fast big data store from Cloudera, and I noticed only Ruby and a Java had clients. Why no PHP client for Impala. So, I thought I would give it a go at creating one.

After a lot of reading and studying both the Java and the Ruby clients I finally was successful at getting a PHP client to connect to the Impala service.

This is meant to be the starting blocks (library) for a more user friendly client. I am working on that currently and hope to have something to share soon. I plan on learning more about Impala and the API methods that are available via the thrift service.

Here is the GitHub for the code.

I bundled the PHP library code into a phar archive. There are a lot of files and it is easier to deal with. I also did not use the supplied Thrift classloader, I used the Zend Framework classmap_genarator script to build a classmap, and I wrote a very simple class loader to consume it.

Check it out and let me know what you think.

 

Cassandra and Hadoop and Hive, Oh my!

Posted on by Posted in Hadoop, Linux, NoSQL 1 Comment

At work we are installing a Cassandra cluster for an additional tool in the database toolbox. Hadoop and Hive are bundled with it because we are using the DataStax distribution of Cassandra. This gives us a nice platform to store data and run Hadoop data mining jobs.

We will be using Cassandra several different ways:

  • A database for information that does not need to be stored relationally.
  • A caching server for data jobs that are run on the databases that we do not want direct web traffic to make calls against.
  • A data mining platform using Hadoop and Hive/Pig.

Being a .Net shop, we chose Fluent Cassandra as the Cassandra client library, this was after a healthy Fluent Cassandra vs Aquiles debate.

We are in the process of installing and configuring the cluster now, so I’ll post again after we have the cluster up in our development environment.