The adventures of a software architect

My Camel likes the Elephant

Sun, 20 Dec 2009 13:00:36 +0100

I’m still working on the Camel HDFS component and I think it reached a good level of maturity. I added also the consumer part, so it’s possible now to read from an HDFS file. I submitted the patch to the Camel community and it’s pending for the approval and to be integrated into the Camel trunk.
In the meanwhile if you wanna play with the latest version of this component, you can check it out from my subversion repository:

svn co http://home.davidgreco.it/svn/camel-hdfs/trunk camel-hdfs

Below the documentation of the component.

HDFS Component:

URI Format

hdfs://hostname[:port][/path][?options]

the path is treated in the following way:
1) as a consumer if it’s a file it just read the file, otherwise if it represents a directory it scans all the file under the path satisfying the configured pattern. All the files under that directory must be of the same type.
2) as a producer, if at least one split strategy is defined, the path is considered a directory and under that directory the producer creates a different file per split named seg0, seg1, seg2, etc.

Options

Name Default Value Description
overwrite true the HDFS file can be overwritten

bufferSize 4096 the buffer size used by HDFS

replication 3 the HDFS replication factor

blockSize 64MB the size of the HDFS blocks

fileType NORMAL_FILE it can be SEQUENCE_FILE,
MAP_FILE, ARRAY_FILE, or
BLOOMMAP_FILE, see Hadoop

fileSystemType HDFS it can be LOCAL for local filesystem

keyType NULL the type for the key in case of
sequence or map files. See below.

valueType TEXT the type for the key in case of
sequence or map files. See below.

splitStrategy A string describing the strategy on
how to split the file based on different
criteria. See below.

openedSuffix opened When a file is opened for reading/
writing the file is renamed with this
suffix to avoid to read it during the
writing phase.

readSuffix read Once the file has been read is
renamed with this suffix to avoid to
read it again.

initialDelay 0 For the consumer, how much to wait
before to start scanning the directory.

delay 0 Then interval between the directory
scans.

pattern * The pattern used for scanning the
directory

chunkSize 4096 When reading a normal file, this is split
into chunks producing a message per
chunk

The keyType and the valueType can be:
NULL it means that the key or the value is absent
BYTE for writing a byte, the java Byte class is mapped into a BYTE
BYTES for writing a sequence of bytes. It maps the java ByteBuffer class
INT for writing java integer
FLOAT for writing java float
LONG for writing java long
DOUBLE for writing java double
TEXT for writing java strings

BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.

Splitting Strategy
In the current version of Hadoop (0.20.1) opening a file in append mode is disabled since it’s not enough reliable. So, for the moment, it’s only possible to create new files. The Camel HDFS endpoint tries to solve this problem in this way:
• If the split strategy option has been defined, the actual file name will be <file name>0 initially
• Every time a splitting condition is met a new file is created with name <original file name>N where N is 1, 2, 3, etc.
The splitStrategy option is defined as a string with the following syntax:
splitStrategy=<ST>:<value>,<ST>:<value>,*

where <ST> can be:
BYTES a new file is created, and the old is closed when the number of written bytes is more than <value>
MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value>
IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds

for example:
hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written

P.S. (8 Jan 2010)
I moved this project to github here. There is also a wiki page under the Apache Camel documentation, here.

When a Camel encounters an Elephant

Tue, 17 Nov 2009 11:59:30 +0100

I don’t know why, but I’m still pretty linked to my HPC background, even if I spent most of my time working for different kind of enterprises and trying to solve problems on different domains, I always thought that enterprises should try to learn from the HPC community how to solve “hard” business problems. With the adjective “hard”, I mean difficult in terms of computational power needed for solving the problem.
I’m observing a sort of convergence towards a sort of cross fertilization in terms of technologies involved between the HPC community and the enterprise business community. For example, grid computing and cloud computing are technologies that used to be already popular in the HPC field, but now are getting more and more interest even in the enterprises.
In the HPC, off course the emphasis is more on the solution of numerical problems where the data set involved can be really huge. There are so many interesting problems in this area: oil exploration, weather forecast, molecular dynamics, etc. On the other end enterprises need to process huge amount of data as well, but in this case they usually need to digest huge amount of data in structured/semi-structured form for extracting something useful: analysis of web servers logs, information retrieval/extraction from a massive number of documents, indexing, etc.
In both cases, there is a trend in trying to scale the applications using many inexpensive nodes than using big computers.
Along this line, I started to give a look to a very interesting technology used for scaling the analysis of very huge quantity of data.
This technology is Hadoop. Hadoop is an open source implementation of the map/reduce algorithm proposed by Google. I would say that Google didn’t invent anything, they just applied the concept of collective computations (reductions), typically used in parallel computing, to the problem of creating reverse indexes for the web search. I used to use sort of map/reduce algorithm using MPI when I was working in a parallel computing laboratory.
I don’t want to go deep into details regarding the Hadoop ecosystem, on the Apache site it’s plenty of information. What I tried to do is to see if one of my favorites open source tools, Camel, could be useful for making the usage of Hadoop easier. Look at the picture below:

A fundamental piece in the Hadoop ecosystem is HDFS (Hadoop File System). From the Apache site: “The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets.”
So, before analyzing the data the first step is to put the data on the HDFS filesystem. HDFS provides APIs and also command line utilities for loading data into HDFS, however I thought it would be nice to take advantage of the multi-transport/multi-protocol nature of Camel for accomplishing the task of filling the Hadoop file system.
In fact, using Camel you could easily gather data for virtually any kind of system using a plethora of many different mechanisms: files, ftp, http, mail, atom, rss, xmpp, etc. So, having an HDFS Camel component you could just define routes that gather data from any kind of sources to be put directly into the Hadoop file system, this is why I wrote a Camel HDFS component. In attachment I put the zip file containing the component, youi can dig around the tests to see how it works. At the end of this post I’ll put some documentation. For the next steps I’m thinking if it makes sense to integrate Camel even with the map/reduce framework. In the meanwhile I think that using Camel only for writing data into an HDFS filesystem could really help.
Very soon I’ll try to submit this work to the official Camel project.

HDFS Component:
This component provides only a producer, it only allows writing data into the HDFS, reading from HDFS is not implemented yet. This component can be only used with the to clause.

URI Format

hdfs://hostname[:port][/path][?options]

Options

Name Default Value Description
append false the HDFS file is opened in append
mode

overwrite true the HDFS file can be overwritten

bufferSize 4096 the buffer size used by HDFS

replication 3 the HDFS replication factor

blockSize 64MB the size of the HDFS blocks

fileType NORMAL_FILE it can be SEQUENCE_FILE or
MAP_FILE, see Hadoop

fileSystemType HDFS it can be LOCAL for local filesystem

keyType NULL the type for the key in case of
sequence or map files. See below.

valueType TEXT the type for the key in case of
sequence or map files. See below.

splitStrategy A string describing the strategy on
how to split the file based on different
criteria. See below

The keyType and the valueType can be:
NULL it means that the key or the value is absent
BYTE for writing a byte, the java Byte class is mapped into a BYTE
BYTES for writing a sequence of bytes. It maps the java ByteBuffer class
INT for writing java integer
FLOAT for writing java float
LONG for writing java long
DOUBLE for writing java double
TEXT for writing java strings

BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.

Splitting Strategy
In the current version of Hadoop (0.20.1) opening a file in append mode is disabled since it’s not enough reliable. So, for the moment, it’s only possible to create new files. The Camel HDFS endpoint tries to solve this problem in this way:
• If the split strategy option has been defined, the actual file name will be <file name>0 initially
• Every time a splitting condition is met a new file is created with name <original file name>N where N is 1, 2, 3, etc.
The splitStrategy option is defined as a string with the following syntax:
splitStrategy=<ST>:<value>,<ST>:<value>,*

where <ST> can be:
BYTES a new file is created, and the old is closed when the number of written bytes is more than <value>
MESSAGES a new file is created, and the old is closed when the number of written messages is more than <value>
IDLE a new file is created, and the old is closed when no writing happened in the last <value> milliseconds

for example:
hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
it means: a new file is created either when it has been idle for more than 1 second or if more than 5 bytes have been written

Towards the real-time web

Sat, 24 Oct 2009 17:18:42 +0200

In the last couple of months I spent some time investigating different options for real-time pushing of data to the browser.
Till very recently, a common technique for simulating a sort of real-time updating of information on a web based application was through the usage of polling or long-polling. I don’t want to spend time explaining those two mechanisms, it’s plenty of information on the web.
Sometime relying on the browser polling periodically information updates it’s not enough. There are classes of applications were the latency really counts: real-time monitoring, gaming on line, etc.
So, since the browser is becoming more and more a platform providing full computing power (there is a race among the different browsers around the speed of the javascript interpreters), it’s now possible to conceive web based applications able to receive a continuous stream of informations even for latency sensitive applications.
The very first technology I experimented with has been reverse ajax also known as comet. In particular, I got very good feedbacks from a comet implementation called Bayeux protocol. Bayeux is an high level protocol developed to simplify the development of comet based applications, it’s being developed under the Dojo foundation umbrella. It seems a promising technology, it’s simple and it’s already stable, they are very close to the release of the version 1.0. Bayeux got interest also among other software providers, there are other Bayeux implementations both open and closed source: Grizzly, Weblogic, Tomcat.
From the scalability standpoint Bayeux takes advantage of the new advancement in the web container technology. The cometd implementation of Bayeux, the original one, is based on the continuation mechanism of jetty , a very powerful and popular web container. This mechanism is going to be included into the upcoming Servlet 3.0 specifications, so, pretty soon we will have a standard mechanism for developing Java based comet applications. In the context of a web container the continuation allows to break the one to one correspondence between a connection and a thread. In fact, since a comet application is based on the browser keeping one connection to the server always open, it’s obvious that this it’s not affordable, one thousand clients would mean one thousands active threads. Using jetty and its continuation mechanism you could host even thousands and thousands connections just with one single machine.
After played with Bayeux I started to look around for alternatives, this is what I discovered:
1. If you like Flash/Flex, Adobe provides BlazeDS, a server-based Java remoting and web messaging technology that enables developers to easily connect to back-end distributed data and push data in real-time. There are also other open source implementation of the protocols as defined by BlazeDS, just google a little bit and you’ll find what you are looking for.
2. XMPP, this is a popular presence and real-time communication protocol. XMPP can be also used for pushing data to the browser, this can be achieved using Flash through XMPP Flash client APIs freely available. It’s also possible to use http through the BOSH XMPP extension. Using BOSH a browser can be connected just using the HTTP protocol. There are Javascript APIs for developing BOSH based client applications, among the others I found this library Strophe pretty popular.
3. HTML5: this is the new HTML standard. A lot of new features are being introduced in the new HTML standard, among the other, W3C is working on a protocol for allowing a bi-directional communication between a browser and a remote host. This protocol is called Websockets. This new protocol is the result of the joint effort of W3C and IETF.
IETF is defining the transport protocol, you can find the RFC here where W3C is busy with the definition of the APIs to be used inside the browser for using the Websockets protocol. There is an interesting product by this company Kaazing. They provide a gateway with associated client APIs allowing to develop Websockets based applications. If a browser doesn’t directly support the Websockets protocol, the Kaazing Javascript libraries fallback to simulate the Websockets client layer. Tha Kaazing gateway provides specific Javascript libraries for connecting a bowser with popular messaging technologies like XMPP. I found the overall Kaazing stack fast and easy to use.
My personal opinion is that Javascript is already enough fast and expressive for developing rich internet applications. All the main web browsers are constantly improved in terms of Javascript processing speed and, moreover, it’s really plenty of powerful Javascript frameworks for helping people to develop nice and user friendly applications.
Javascript is also available in all the browsers on board of the most popular smart phones. So, the client side is going to be dominated by Javascript. Given this fact, I would opt for Javascript/HTTP based solutions for real time data pushing. Bayeux, in my opinion, is the first candidate, it could be a medium term solution till the Websockets will become adopted and available. I found also interesting the XMPP based approach, it could be an alternative, however it forces you to have the XMPP broker server in the middle, between the client and the data producer. Using Bayeux, the protocol listener/manager can be directly co-located with your back-end giving you a better chance for reducing the latency.
Above, I tried to describe solutions based on well defined protocols either “de iure” or “de facto” standardized.
However, during my investigations on pushing technologies, I found other platforms/solution providing this kind of capabilities.
I found these particularly interesting:
• Lift: it’s a Scala based web framework providing also a comet mechanism.
• Erlang based solution: read this article really impressive.
Sorry for this long post, it’s always so difficult to condense so many information into something not too much long and boring ;)

Using ActiveMQ and CXF for inter-webapp communication

Fri, 2 Oct 2009 13:42:58 +0200

For a project I’m working on I had the need to make two separate web applications running inside Tomcat to talk each other.
Basically, each web application represents a distinct business component, see previous post, that exposes a well defined interface.
I preferred to stick with the one business component per web app since it allows to deploy and manage the business components separately each with its own lifecycle.
Being on different deployment units the two separated business components cannot talk each other even if they reside on the same container, this is, off course, due to the class loading mechanism of the the web container.
So, the problem is, if I have two business components deployed into two different web archives how could I make one component consume services exposed by the other one ?
I decided to use CXF and ActiveMQ for achieving that goal. Using the code first approach, just annotating the component interfaces with JAX-WS annotations I can make those interfaces remotely callable just using the configuration based mechanism, based on Spring, provided by CXF.
In order to make this communication as fast as possible I decided to use ActiveMQ embedded into Tomcat using the vm connector. In this way, when the client calls the service exposed by the component on the other side, the messages are sent using the memory by passing the network stack (even the local loop). Off course, I still have to pay the price of marshaling/unmarshaling to/from SOAP/XML, but I thought it was worthwhile for preserving a clean separation among the business components.
As usual, I’m putting as an attachment (in the comment section) a simple project showing what I tried to describe on this post.
Let me know what you think.

OSGi for enterprise applications: part 3

Sun, 21 Jun 2009 17:32:59 +0200

My efforts in understanding the implication of the usage of OSGi are still continuing. With this entry I’d like to share with you my efforts to use OSGi as the foundation technology for developing component based enterprise systems. As I wrote many times, I’m really convinced on the effectiveness of the component based development for realizing large scale enterprise systems. A couple of years ago I had the chance to work with Peter Herzum one of the authors of the book Business Component Factory. This book introduced to me the concept of Business Component. Being a lazy person I don’t want to write too much about these things. very simply a business component is a unit of functionality that manages a single business abstraction. A business component is tiered, for each architectural tier it contains a distributed component:
• User Distributed Component: responsible for the user interface.
• Workspace Distributed Component: supports activities belonging to a single user.
• Enterprise/Service Distributed Component: it contains the business logic and it enforces the integrity of the shared resources managed by the resource component
• Resource Distributed Component: it provides access to the shared resources like the persistent data in a database.
This picture summarizes the relationship among the different concepts in a business component based system.

I found this model very powerful, during the analysis phase you try to discover the business components that could realize your system. Then for each DC (Distributed Component) belonging to the individuated BCs you start to define the interfaces, after this phase you could start to develop each single DC in insulation maybe developing proper mockups for the DCs you depend upon.
I thought that the combination of OSGi + SpringDM and the usage of a robust OSGi based container like ServiceMix4 could allow me implementing the business component concept. In fact, what I found dealing with OSGi for the enterprise is the lacking of a more coarse grain component model. In my opinion the bundle doesn’t provide the right level of abstraction when you want to develop a component based system. Maybe, it’s a silly idea, but OSGi can be thought as a sort of assembler for component models. Given the extreme modularity and the low level service concept that OSGi provides, this framework can be used for implementing any kind of more sophisticated component model. On the other end OSGi is being adopted as a foundation layer for other component models like JBI (with ServiceMix4) and JEE (Geronimo, Glassfish). So, as I said, OSGi, per se, provides a component model but unfortunately doesn’t provide the right level of abstraction for dealing with the development of business component based systems. As a direct consequence, it’s pretty easy to fail in managing the complexity of so many bundles with all the relationships among them. An enterprise application of moderate complexity can be made of hundreds bundles, so, without a strict discipline or a strict architectural blueprint is very probable that sooner or later you’ll loose the control, loosing also the advantages of using OSGi.
So, at least with the current status of OSGi based technologies, it’s really important to establish either an architectural metaphor or an architectural blueprint that is well suited for the development of your system/enterprise application, and then spend some time for setting up reusable templates.
Driven by this considerations, after decided to embrace OSGi for developing enterprise applications, I decided to choose an architectural metaphor: the business component concept. Then we spent a couple of weeks for developing a business component example that can be used as a template, in this way the developers should have an easier way for starting the development of a business functionality and at the same time (most important) they have a clear path to follow and clear model to adhere to.
The template we developed provides a hierarchical maven based project of a complete business components with all its distributed component.
The resource component uses JPA for dealing with data persistence. The service component uses JAX-WS based annotation and CXF for providing a remoting capable interface, The service component then uses Camel for providing an event driven connector. In fact, in our model, a distributed component should provide a remote interface and it should also able to receive and send events. At the end using SpringDM, CXF and Camel we could add the needed connectors to our components (synchronous, RPC based connector, and asynchronous event based connector), just using a configuration based approach. The template we wrote contains all the needed dependencies and import/export packages directives, the idea behind is to relieve the developer from dealing with too much details of the OSGi mechanic.
I hope to put sooner in attachment a complete example of a business component hoping it could be helpful for you.

OSGi for enterprise applications: part 2

Thu, 30 Apr 2009 10:04:38 +0200

Developing enterprise applications means dealing heavily with persistence. We constantly need to save data regarding our customers, accounts, etc. JEE solved this pretty nicely with new specifications in the broad area of the EJB3.0 standard. This new standard is JPA (Java Persistence API), and it allows annotating a POJO for driving how to map it into a relational database. I’m not an expert, but I would say that it’s pretty cool. So, I decided, JPA is the right solution for persisting data in my hypothetical OSGi based enterprise application.
So, after two weeks of hammering I found the right combination of a JPA provider and SpringDM support for persisting my data using OSGi as a container. The net result is not too bad (I hope), even if I had to really working hard to cope with the typical problems when you use this kind of tools under an OSGi environment.
The problem was due to the fact that most of the JPA providers are based on some class weaving mechanism typically at load time, this conflicts with the OSGi class loading behavior, so I had to find the JPA provider that was closer to the OSGi behavior, found it ! It’s EclipseLink.
EclipseLink is the former Oracle’s TopLink that has been donated to the Eclipse foundation and it has been explicitly meant to run on top of an OSGi container.
In attachment in the comment section I put a self contained example that shows how to run SpringDM and EclipseLink under ServiceMix4 kernel and Felix.
During this exercise I learnt a lot of things, and I refined my knowledge on the tools I decided to use for developing OSGi based enterprise applications. For example, I learnt how to use Maven for automatically generate the features descriptor. It’s pretty impressive, the maven plugin is able to generate the feature descriptor checking the packages imported and exported by the various bundles. The ServiceMix guys did such an impressive job, thanks Guillaume ;) As usual the projects contains some basic information for understanding how it works, I hope it’s enough clear, I’m too lazy for writing more.
More in the next blog entries ...

OSGi for enterprise applications: part 1

Sun, 26 Apr 2009 15:04:30 +0200

Lately I’m working hard for understanding what are the best tools available for developing OSGi based enterprise applications.
Off course, what I’m going to say reflects what are my feelings in terms of software development and supporting tools.
Firstly, these are the requirements I have in mind for an OSGi based development environment:
1. everything must be Maven based. I need an environment capable to full automatize the development life-cycle including support for continuous integration, Maven is the right tool for that.
2. I need to be able to write unit tests that simulates the bundles running inside the OSGi container. Being able to develop unit tests close to the integration tests is a powerful mechanism for improving the application’s quality.
3. I need Spring and Spring DM as a dependency injection framework and as a mechanism for simplifying the registration and looking up of OSGi services.
4. I need a powerful OSGi based container for putting in production my applications.
After some investigation I came up to the following tools:
1. pax-contruct from the open source community ops4j. This tool provides powerful scripts that helps building complete maven based OSGi projects.
2. pax-exam another impressive tool from the ops4j gang, it allows writing unit tests that run the container deploying all the bundles you want including your application’s bundles.
3. ServiceMix 4 Kernel, it’s not a secret that this is my preferred OSGi based container.
For understanding how to use those tools together I decided to see if it’s possible to deploy into an OSGi container a nice and powerful distributed key-value storage system: Voldemort.
In fact, it’s becoming increasingly popular the usage of sophisticated caching mechanism for scaling and making more fault tolerant massive web sites (Voldemort is being developed by the LinkedIn guys).
I think it could be interesting to have a look at this page where different kind of distributed/replicated caching mechanisms are compared:
http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/
So, after a while I came up to a maven based project that I have built using the pax-construct tools. This project shows how to convert plain jars into bundles and how to use pax-exam for testing an OSGi application.
First of all I converted the Voldemort’s jar into a bundle and I developed a simple Spring DM based bundle that starts a voldemort server pointing to specific configuration directory. Using pax-exam, I wrote a simple unit test that put and retrieve a key/value pair from the cache.
As usual I’m going to attach the entire project in the comments section.
Please, give a look at the README file for better understanding how it works. In the README I put also the gshell commands for running the voldemort server inside the SMX4 container.
In the next blog entries, I’m going to show and attach other projects showing how to solve common problems using SMX4/OSGi.

SMX4 + Camel + Esper = An open source CEP platform

Thu, 26 Mar 2009 20:34:13 +0100

Complex event processing is getting more and more hot in the enterprise software world. Modeling business processes in terms of event processing is a nice level of abstraction and that fits very well with the concept of real-time enterprise. Many use cases can be thought where a complex event processing platform is really suitable: fraud management, service level agreement management, intrusion detection RFID, etc.
The market provides different solutions for developing CEP based applications, it seems to me that we can roughly categorize CEP platforms in two broad categories: stream oriented, finite state machine/rules oriented. A CEP stream oriented typically provides a SQL like language that you can use for querying the stream asking, for example, how many events you received in the last ten seconds. A finite state machine/rules based platform provides a language for triggering state transitions or rules once an event of a certain type arrives.
I’m not enough expert to tell you which is the best, but just for my personal taste I like the first type, stream oriented. In the open source arena I started to really appreciate on of this kind of platforms: Esper.
I’ve already contributed a demo showing how to use Camel and Esper together using the camel-esper component developed by James Strachan, so I decided to repeat the exercise using Servicemix4 as a very powerful OSGi and SpringDM container. The result, I think, it’s pretty cool, OSGi, Camel and Esper really fit nicely together (even if I had to hammer a lot before solving all the “class not defined issues”), you can start and stop the processing of the event streams, you can add you processing routes, etc. I put the results of my efforts in the comment section as an attachment, let me know what you think.