kCGErrorFailure with BigSQL

So you are trying to start BigSQL on your mac and get the following error:

java[5413] <Error>: kCGErrorFailure: Set a breakpoint 
@ CGErrorBreakpoint() to catch errors as they are logged.

after trying to run the following command:

hadoop namenode -format

The issue may have to do with the combination of your OS and Java version (we saw the error with OS 10.7.4 and Java version 1.6.0_51). We fixed this issue by updating the version of Java to the latest release and changed the definition of JAVA_HOME in setENV.sh in the main BigSQL directory.


Hadoop Thrift Tutorial

At OpenSCG we have been using thrift to make a Hadoop Foreign Data Wrapper. This tool, which is already integrated into BigSQL, allows you to take advantage of the power of Hadoop from within PostgreSQL. The BigSQL Tutorial show how you can easily create a postgres table that references a Hive or Hbase table, run a parallel Hive query, and see your data returned, all in the psql terminal. Behind the scenes, the Hadoop Foreign Data Wrapper uses thrift tools to make Hive and Hbase clients in C++, then the user creates a foreign server from within PostgreSQL which the clients can then communicate to.

The goal of this tutorial is to show you a very simplified look at the making of thrift servers and clients. You will use thrift to make a non-blocking java server, a Python client and a C++ client. The clients will be able to send a structure and commands to the server, and the server will do calculations and print out the result. I made the structure and two functions described below:

  • UserProfile struct that consists of a user id, name, blurb, month, day, and year of birth
  • Repeat function that returns the user’s id, name, and blurb
  • Daysold function that returns the approximate number of days since the user was born

Setting up Thrift

Make sure that you have all of the appropriate apache thrift requirements; this includes getting the Libevent API, Boost Libraries, and language support for Java, Python and C++. Get the latest version tarball or build from source using the instructions on the Apache Thrift Download page.

Creating the Java Server

The first step to using thrift is to create the imaginary file Start.thrift in the thirft/compiler/cpp/ directory. This type of file is made up of thrift types and services that the server will implement and the client will call. Thrift allows you to make one file, no matter what language you intend on using, that will create all of the source code you need.


#!/usr/local/bin/thrift --gen java

namespace java Test

    struct UserProfile {
        1: i32 uid,
        2: string name,
        3: string blurb,
        4: i32 month,
        5: i32 day,
        6: i32 year
    service UserStorage {
        void repeat(1: UserProfile user)
        void daysold(1: UserProfile user)

Create the Java code by running the following command from the cpp directory. The -r specifies that you want to generate all the included files.

    $ thrift -r --gen java start.thrift

Make the java server from scratch add the following files to the created gen-java directories:


package Test;

import org.apache.thrift.transport.TServerSocket;
import org.apache.thrift.transport.TServerTransport;
import org.apache.thrift.server.TServer;
import org.apache.thrift.server.TServer.Args;
import org.apache.thrift.server.TSimpleServer;
import Test.UserProfile;
import Test.UserStorage;

public class UserStorageserver {
public static void StartsimpleServer(UserStorage.Processor processor) {

        try {
            TServerTransport serverTransport = new TServerSocket(9090);
            TServer server = new TSimpleServer(new Args(serverTransport).processor(processor));
            System.out.println("Starting the simple server...");
        } catch (Exception e) {

    public static void main(String[] args) {
        StartsimpleServer(new UserStorage.Processor(new UserStorageHandler()));


package Test;

import java.util.Calendar;
import Test.UserProfile;
import Test.UserStorage;

    public class UserStorageHandler implements Test.UserStorage.Iface {
        public void repeat(UserProfile user) {
            System.out.println("User's ID: " + user.uid + "\nUser's Name: " + user.name + "\nUser's Blurb: " + user.blurb+ "\n");

    public void daysold(UserProfile user){
        System.out.println("Days Old:\n");
        int month = Calendar.getInstance().get(Calendar.MONTH);
        int year = Calendar.getInstance().get(Calendar.YEAR);
        int agem;
        int agey;
            if (month>user.month){
        int age=agem+agey+user.day;
        System.out.println("User is approximately " + age +" days old.\n");

The easiest way to create the Java server is to make it in eclipse. I added the two files I had just created above, UserStorage.Java, and UserProfile.Java. Once these files are all in the correct packages, you need to add four addition .jar files to your build path (properties -> Java Build Path -> Add External Jars):

  • Libthrift-javadoc.jar
  • Libthrift.jar
  • Slf4j-api
  • Slf4j-simple

The thrift Jars can be found in the thrift/lib/java/build/ directory. The two Slf4j Jars you could get from the slf4j download, or you could just remove all of the code that has dependencies on it, it will not affect this example. Once everything is error-less in eclipse, you can run UserStorageserver.Java as a Java application, or export it as a runnable jar. You can run the jar from the command line with:

    $ Java -jar Java_server.jar

Creating the C++ Client

The next step is to make a Thrift client in C++, first you have to change the namespace from “java” to “cpp” in start.thrift. Next, generate the C++ code:

    $ thrift --gen cpp start.thrift
    $ cd gen-cpp

Compile the code using the g++ command. The arguments “-I/usr/local/include/thrift” specifies that you are include all of the header files in the thrift directory, “-Wall” indicates that you want to see any warnings, “-c” specifies to compile the .cpp file, and “-o” allows you to name the output file.

    $ g++ -Wall -I/usr/local/include/thrift -c start_constants.cpp -o constants.og++ -Wall -I/usr/local/include/thrift -c start_types.cpp -o types.o
    $ g++ -Wall -I/usr/local/include/thrift -c UserStorage.cpp -o UserStorage.o

If you’re trying this example on Ubuntu and get this error:

        /usr/local/include/thrift/protocol/TProtocol.h:646:11: error: 'uint32_t' does not name a type

All you need to do is add the following line before include thrift.h in the header file you cannot compile:

       #include <stdint.h>

Or if you have gotten this error:

    error: 'class apache::thrift::protocol::TProtocol' has no memeber named 'writeMessageEnd'
    error: 'class apache::thrift::protocol::TProtocol' has no memeber named 'writeEnd'

All you need to do is add the following line to the header file you cannot compile:

        #include <arpa/inet.h>

Create the following file in the gen-cpp directory:


#include "UserStorage.h" // One of the autogenerated files

#include <transport/TSocket.h>
#include <transport/TBufferTransports.h>
#include <protocol/TBinaryProtocol.h>

using namespace ::apache::thrift;
using namespace ::apache::thrift::protocol;
using namespace ::apache::thrift::transport;

using namespace ::Test;

int main(int argc, char **argv) {
    boost::shared_ptr<TSocket> socket(new TSocket("", 9090));
    boost::shared_ptr<TTransport> transport(new TBufferedTransport(socket));
    boost::shared_ptr<TProtocol> protocol(new TBinaryProtocol(transport));

    UserStorageClient client(protocol);


    UserProfile user;
        user.blurb="Cpp's blurb";

    printf("Sending User Info to Server\n");

    return 0;

Compile and link together all of the .o files. The “-L/usr/local/lib” argument specifies which library to use while linking.

    $ g++ -Wall -I/usr/local/include/thrift -c UserStorage_Client.cpp -o client.o
    $ g++ -L/usr/local/lib *.o -o UserStorage_Client -lthrift

If you’re working on OSX and you’ve gotten the following error:

    /usr/local/include/thrift/transport/TSocket.h:240: error: expected ',' or '...' before '*' token
    /usr/local/include/thrift/transport/TSocket.h:240: error: ISO C++ forbids declaration of 'sockaddr' with no type
    /usr/local/include/thrift/transport/TSocket.h:293: error: 'sockaddr_in' does not name a type
    /usr/local/include/thrift/transport/TSocket.h:294: error: 'sockaddr_in6' does not name a type

You need to add the following to TSocket.h in /usr/local/include/thrift/transport:

        #include <sys/socket.h>
        #include <arpa/inet.h>

Once the code compiles and links, run the client by:

    $ ./UserStorage_Client

Right now you should get an error implying that the server is not on:

        Socket::open() connect() <Host: localhost Port: 9090>Connection refused
        Abort trap

If you get the following error:

    error while loading shared libraries: libthrift-0.9.0.so: cannot open shared object file: No such file or directory

It just means that you need to add libthrift-0.9.0 to your path:

     $ export LD_LIBRARY_PATH='/usr/lib':$LD_LIBRARY_PATH

Creating the Python Client

Creating a Python client is even easier than creating the C++ client. In order to use python, you must go into the Thrift/lib/py directory and run

    $ sudo python setup.py install

If you do not do this, you will get this error when you try to run the client:

    File "./UserStorage_Client.py", line 3, in <module> from Test import UserStorage
    File "thrift-0.9.0/compiler/cpp/gen-py/Test/UserStorage.py", line 9, in <module> from
    thrift.Thrift import TType, TMessageType, TException, TApplicationException
    ImportError: No module named thrift.Thrift

Next, go to the thrift/compiler/cpp directory, change the namespace from “cpp” to “py” and generate the python code:

    $ thrift -r --gen py start.thrift

In the newly created gen-py folder, add this file:


import sys

from Test import UserStorage
from Test.ttypes import *

from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol

    # Make socket
    transport = TSocket.TSocket('localhost', 9090)

    # Buffering is critical. Raw sockets are very slow
    transport = TTransport.TBufferedTransport(transport)

    # Wrap in a protocol
    protocol = TBinaryProtocol.TBinaryProtocol(transport)

    # Create a client to use the protocol encoder
    client = UserStorage.Client(protocol)

    # Connect!

    up = UserProfile(uid=2, name="Python", blurb="Python's blurb", month=1, day=28, year=1978)
    print 'Sending User Info to Server'

    info = client.repeat(up)
    days = client.daysold(up)

    # Close!

    except Thrift.TException, tx:
    print '%s' % (tx.message)

Finally, run the client with the command:

    $ python UserStorage_Client.py

Once again, you should get an error implying that the server is not on:

    Could not connect to localhost:9090

Test Server and Clients

Start up the Java server (In eclipse- right click on UserStorageserver.Java and select “Run as Java Application”) Go to the directory where my Java_Server.jar is located and use the command:

    $ Java -jar Java_Server.jar

Execute the C++ client by going to the gen-cpp directory in a separate terminal window:

    $ ./UserStorage_Client

Execute the Python client by going to the gen-py directory:

    $ Python UserStorage_Client.py

If everything ran correctly, each client’s terminal should say “User Info Sent to Server” and the server’s terminal should show this:

Starting the simple server...
User's ID: 1
User's Name: Cpp
User's Blurb: Cpp's blurb

Days Old:
User is approximately 8146 days old.

User's ID: 2
User's Name: Python
User's Blurb: Python's blurb

Days Old:
User is approximately 12808 days old.

Announcing BigSQL 9.3 Beta 1.7

Our latest BigSQL Beta is now feature rich and stable. Powerful/simple/safe developers sandbox bundles are available for Linux and OSX. BigSQL leverages new features of PostgreSQL 9.3 to provide seamless & deep integration with Hadoop/Hive. The combined bundle gives you all the best OLTP features of standard ANSI PostgreSQL with all of the massive parallel query capabilities of Hive & Hadoop.

It’s 100% open source and 100% free. Take a look today at http://www.bigsql.org for Downloads, Release Notes, Tutorial, etc.


Announcing BenchmarkSQL 4.0

I’ve been working on this project for nine years now and it finally doesn’t suck. :-) Thanks to Cady Motyka for finishing the work to rip out the annoying Swing GUI on this TPC-C like RDBMS Benchmark. It now runs wonderfully from the command-line and is useful for testing the performance of any RDBMS OLTP server.

Look for the SourceForge.net Download link for this project in the lower right hand corner of :


A Java Based Command Line PortChecker

The below code snippet is a lovely little piece of Java magic the we use in various installation and configuration scenarios to verify if a port is available or busy

import java.io.*;
import java.net.*;

public class PortChecker {

    public static void main(String[] args) {

	int port=0;

	if (args.length == 1) {
    	  try {
	        port = Integer.parseInt(args[0]);
	    } catch (NumberFormatException e) {
	        System.err.println("INVALID ARGUMENT: must be an integer.");
      	} else {
	    System.err.println("A single integer argument is required");

	try {
	    ServerSocket srv = new ServerSocket(port);
	    srv = null;
	} catch (IOException e) {

Setting JAVA_HOME for JDK6 on Linux and OSX

Here is a useful little code snippet I use as part of BigSQL.

if [ "x$JAVA_HOME" == "x" ]; then
  if [ -d "$macosx_j6" ]; then
    export JAVA_HOME="$macosx_j6"
  elif [ -d "$ubuntu_j6" ]; then
    export JAVA_HOME="$ubuntu_j6"
  elif [ -d "$redhat_j6" ]; then
    export JAVA_HOME="$redhat_j6"
    echo "WARNING: Cannot figure out JAVA_HOME."
    exit 3