Let Pig Crunch Your Data

A couple of days ago I read about this new Language called Pig Latin, which was developed by Yahoo and contributed to Apache. The Pig Wiki describes it as follows:

A language for processing data, called Pig Latin.

A set of evaluation mechanisms for evaluating a Pig Latin program. Current evaluation mechanisms include (a) local evaluation in a single JVM, (b) evaluation by translation into one or more Map-Reduce jobs, executed using Hadoop.

Since I just downloaded my confusing monthly usage report of Amazon S3, I thought I'd give it a try and see how good Pig is in bringing some light to that data. My first step was to download the tutorial, which contains among others a Jar file called pig.jar, which is essentially all you need to run Pig scripts.

Pig Latin let's query data, e.g load it from a file, filter, group or join that data and eventually apply some function to it. It reminds me of SQL, in that you are querying sets of data and applying functions to those. Pig Latin knows literals, tuples, bags, which are basically sets of tuples and maps. You can use statements to create so called relations, which have aliases that you can refer to in subsequent statements.

In case of my S3 usage report I came up with a little Pig Latin script, which accumulates my S3 usage data in a more useful manner:

raw = LOAD 'report.csv' USING PigStorage(',') AS (Service, Operation, UsageType, Resource, StartTime, EndTime, UsageValue);
clean = FILTER raw BY Service eq 'AmazonS3';
ops = GROUP clean BY (Operation, Resource);
op_counts = FOREACH ops GENERATE group, SUM(clean.UsageValue)/1000000;
dump op_counts;

The first line basically loads the data from the file 'report.csv', which is as the name implies comma separated. Each column of that file gets an identifier, so that I can refer to those columns in subsequent statements. Since the S3 usage reports come with a headline and we don't want to analyze that, we get rid of it using the FILTER statement and a condition, as shown in the second line.

Service, Operation, UsageType, Resource, StartTime, EndTime, UsageValue
AmazonS3,GetObject,EU-DataTransfer-Out-Bytes,dlinsin-archives,11/01/08 00:00:00,11/01/08 01:00:00,272
AmazonS3,GetObject,EU-DataTransfer-Out-Bytes,dlinsin-downloads,11/01/08 01:00:00,11/01/08 02:00:00,269
AmazonS3,GetObject,EU-Requests-Tier2,dlinsin-downloads,11/01/08 03:00:00,11/01/08 04:00:00,37
AmazonS3,GetObject,EU-DataTransfer-Out-Bytes,dlinsin-downloads,11/01/08 03:00:00,11/01/08 04:00:00,116276888
AmazonS3,GetObject,EU-Requests-Tier2,dlinsin-downloads,11/01/08 05:00:00,11/01/08 06:00:00,58
AmazonS3,GetObject,EU-DataTransfer-Out-Bytes,dlinsin-downloads,11/01/08 05:00:00,11/01/08 06:00:00,179015922
AmazonS3,StandardStorage,StorageObjectCount,dlinsin-archives,11/01/08 07:00:00,11/01/08 08:00:00,137
AmazonS3,StandardStorage,EU-TimedStorage-ByteHrs,dlinsin-downloads,11/01/08 07:00:00,11/01/08 08:00:00,9809386248
AmazonS3,StandardStorage,EU-TimedStorage-ByteHrs,dlinsin-archives,11/01/08 07:00:00,11/01/08 08:00:00,89743752
AmazonS3,StandardStorage,StorageObjectCount,dlinsin-downloads,11/01/08 07:00:00,11/01/08 08:00:00,47
AmazonS3,StandardStorage,StorageObjectCount,dlinsin-archives,11/02/08 07:00:00,11/02/08 08:00:00,137
AmazonS3,StandardStorage,EU-TimedStorage-ByteHrs,dlinsin-downloads,11/02/08 07:00:00,11/02/08 08:00:00,9809386248
...

I wanted to see the amount of traffic of each operation applied on each of my buckets in S3. As you can see the report look quite cluttered, therefore I grouped the data by 'Operation' and 'Resource', which is basically what line 3 does. I then applied a SUM function to each of the grouped item's 'UsageValue' field, which results in the amount of traffic in bytes. For better readability I converted the results in Megabytes. Finally the last statements send the results to stdout.

crusty:pigtmp dlinsin$ java -cp pig.jar org.apache.pig.Main -x local amazons3.pig
((DeleteObject, dlinsin-downloads), 2.0E-6)
((GetObject, dlinsin-archives), 1.785656)
((GetObject, dlinsin-downloads), 11194.995918)
((HeadBucket, dlinsin-downloads), 0.032442)
((HeadObject, dlinsin-downloads), 5.42E-4)
((ListAllMyBuckets, ), 0.00448)
((ListBucket, dlinsin-downloads), 0.108617)
((PutObject, dlinsin-downloads), 0.098595)
((StandardStorage, dlinsin-archives), 1346.158335)
((StandardStorage, dlinsin-downloads), 147140.794425)

If you are interested you can download the pig tutorial together with my script and the Amazon S3 usage report as a pre-packaged archive. Simply unzip it and start pig with

java -cp pig.jar org.apache.pig.Main -x local amazons3.pig

I think this little script already shows the potential of Pig Latin and it's evaluation mechanism, but there is more. You can not only run your scripts on a local JVM, but also in a Hadoop cluster in MapReduce style. There's also an API which let's you embed Pig in a Java application.

We have a Lucene index with about 150GB at work and I could imagine running all sorts of statistical analyzes on it. Unfortunately I couldn't find any Lucene integration, in terms of simply loading a index and accessing the Lucene documents, which are basically simple sets of tuples. I would really love to see how Pig performs, when it crunches a big bunch of data.

Compass - A Hibernate Lucene Integration

• InfoQ article
• Hibernate developer list
• Compass framework

I've been following the Hibernate Lucene integration plans of JBoss for a while now and I came across the Compass framework, which seems to be very interesting:

The aim of Compass is to simplify the integration of search functionality into any application. Compass is built on top of Lucene using a well defined search engine abstraction. Compass extends core Lucene and adds transaction support and fast updates, as well as the ability to store the index in the database. Also, most importantly, it does not try to hide Lucene's features - all of Lucene's functionality is available through Compass.

The framework consists of a core API, which is similar to the Hibernate API, handling configuration, search engine operations and transactional support. Furthermore it provides a module called Compass GPS aiming at integration with various data sources. Among others it provides intgeration with ORM tools like Hibernate.

...Compass introduces two main operations: Indexing and Mirroring. Indexing allows to automatically index the database content using both Hibernate mappings and Compass mappings. Objects that have both mappings will be automatically fetched from the database using Hibernate and saved into the search engine. Mirroring allows to automatically mirror operations done using Hibernate API into the search engine by registering event listeners with Hibernate. This allows to keep the index up to date with any changes done to the database through Hibernate API.

One thing, that I've already been worried about previously, is performance. How does this synchronization perform under heavy load? Another point are transaction boundaries, since there are two data source that need to be updated when performing changes. Does a transaction fail cause your application's Lucene index couldn't be updated?

Though I haven't checked out Compass myself, I believe it extends Lucene in a really cool way. Minimizing the boilerplate code to map your domain objects to Lucene documents is one of these great extensions. Another nice feature is the integration with Hibernate in very transparent fashion, so no additional code is necessary to keep your Lucene index snychronized.

Hibernate Lucene Integration

• Hibernate Annotations documentation
  
• Hibernate developer mailing list posting
  

I checked out Hibernate's Lucene integration and I must say it looks quite interesting. The concept it pretty straightforward: An entity is mapped to a Lucene Index, it's properties can be indexed using annotations. Hibernate is taking care of the mapping it self, all you need to do is register an EventListener which is called after a Hibernate event (e.g. an update) occurred.

So far the document only covers creating/updating Lucene indexes. I found an interesting posting on the Hibernate developer mailing list, which points out that there is more to come with Hibernate 3.3. It seems like there will be some kind of LuceneSession object that can be used, among other things, to query a Lucene index. The cool thing is that you can use the native Lucene Query Language and get a Hibernate Query object in return.

This sounds really cool, the only concerns I have are about performance. What if you have a huge index, in terms of a lot of Lucene Documents, how big is the performance impact when storing/updating/querying the index? Let's say you are performing a search with a big result set, how is Hibernate gonna handle that (caching or proxies)? Another issue, when storing/updating a Lucene Index, is how and when is the optimize() method called?

Updates will follow...

Hibernate 3.2

• http://www.artima.com/forums/flat.jsp?forum=276&thread=182271
• http://www.hibernate.org
  

A couple of days ago I read an interesting interview over at artima.com with a lead engineer of the Hibernate project. He talks about the new features and the JPA compliance of Hibernate 3.2 GA. Two points he mentioned sounded quite interesting, since I haven't heard of them, though apparently they are not new features of 3.2:

• Hibernate's Lucene integration
  

This work is being led by Emmanuel Bernard. It builds on Hibernate Annotations and the event framework of Hibernate Core. The idea is to delegate maintenance of Lucene indexes that are populated based on domain entity state to Hibernate based on persistence events.

Whenever you update or delete or insert a new entity instance, Hibernate fires events in response to these actions which the Lucene integration listeners pick up on and perform any needed actions on the Lucene indexes. The beauty of the whole integration is the simplicity of the solution: you get all this from just a few annotations on your domain classes and a few extra lines of Hibernate configuration for the listener stack.

• An Antlr-based query translator
  

...currently working on a slew of new HQL features, such as ad-hoc join syntax and UNION/MINUS queries; this is all being done on top of the Antlr-based query translator we introduced in 3.0.

I'm facing both of these issues in one of my current projects, where we have to keep a Lucene-Index synchronized with the data in a database. This is a fairly challenging problem, where Hibernate 's implementation might be worth a look at. The Antlr-based query translator is quite interesting cause I have to implement a custom search syntax for both queries against a database and queries against Lucene.

I'll might blog about my investigations on these to issues. Write me if you've got experience with either one of these topics.