Regu's mind trail

2012 in review

regumindtrail — Mon, 31 Dec 2012 04:43:04 +0000

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 5,300 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 9 years to get that many views.

Click here to see the complete report.

Open Source RabbitMQ connector aka. transport for Mule

regumindtrail — Fri, 24 Aug 2012 07:08:54 +0000

Code commits on Trooper (https://github.com/regunathb/Trooper/) have kept me busy the last few weeks. Trooper is an umbrella project for a number of things.

Trooper now has a RabbitMQ connector for Mule.

Is “Big Data” same as “Big Analytics”?

regumindtrail — Mon, 26 Mar 2012 12:49:09 +0000

Do a Google on Big Data and you are more likely to find people talking about two things:

How Open Source solutions like Hadoop have pioneered this space
How some companies have used these solutions to build large scale analytics solutions and business intelligence modules.

Read more and one will find mention of Map Reduce and how many of the NoSQL data stores support this useful “Data Locality” pattern – taking compute to where the data is.

Hadoop users and the creators themselves acknowledge that the technology is good for “streaming reads” and supports high throughput at the cost of latency. This constraint and the fact that Map Reduce tasks are very I/O bound, make it seemingly unsuitable for use cases that involve users waiting for a response such as in OLTP applications.

While all of the above is relevant and mostly true, is it also leading to a certain stereo-typing – that of equating Big Data to Big Analytics?

It might be useful to describe Big Data first. Gartner categorizes data build up in an enterprise as under : Volume, Variety and Velocity. Rapid growth in any of these categories or combinations thereof, results in Big Data. It might be worthwhile to note here that there is no classification under transaction processing or analytics, thereby implying that Big Data is not just Big Analytics.

Big Data solutions need not be limited to Big Analytics and may extend to low latency data access workloads as well. A few random thoughts on patterns and solutions:

Data Sharding – useful to scale low latency data stores like RDBMS to store Big Data. Sharding may be built into application code, use an intermediary between the application and data store or inherently supported by the data store using auto-sharding of data.
Data Stores by purpose – Big Data invariably means distribution and may result in data duplication; within a single store or multiple. For e.g. data extracts from a DFS like Hadoop may also be stored in a high-speed NoSQL or sharded RDBMS and accessed via secondary indices. This could lead to scenarios outlined by the CAP theorem (http://en.wikipedia.org/wiki/CAP_theorem).
Data Stores that effectively leverage CPU, RAM and disk space – Moore’s Law has been proven right the last few years and data stores like the Google Big Table (or HBase) successfully leverage the trend of abundant commodity compute, memory and storage.
Optimized Compute Patterns – Efforts like Peregrine(http://peregrine_mapreduce.bitbucket.org/) that support pipe-lined Map Reduce jobs.
Data aware Grid topologies – A compute grid where worker participation in a compute task is influenced by data available locally to the worker, usually in-memory. Note that this is different from the data locality pattern implemented in most Map Reduce frameworks.
And more…..

It may suffice to say that Big Analytics has been the most visible and commonly deployed use case on Big Data. New age companies, especially the internet based ones, have been using Big Data technologies to deliver content sharing, email, instant messaging and social platform services in near real time. Enterprises are slowly but surely warming up to this trend.

SOA Governance requires more than a Service Registry

regumindtrail — Thu, 17 Jun 2010 10:46:35 +0000

The big software vendors use the term – Service Registry synonymously with SOA Governance. In the process they inadvertently confuse a reader into thinking that setting up a Service Registry can ensure SOA Governance. I wrote an article on this subject that got published here : http://www.cioupdate.com/insights/article.php/3886106/SOA-Governance-Requires-More-than-a-Service-Registry.htm

CIOUpdate article on SOA and Cloud

regumindtrail — Tue, 23 Mar 2010 13:07:16 +0000

I wrote this article for CIOUpdate.com (http://www.cioupdate.com) a while back on SOA and its relation to the Cloud. The article tries to introduce the two concepts and compares them using different view perspectives.

You can find the original article here : http://www.cioupdate.com/reports/article.php/3853076/How-SOA-and-the-Cloud-Relate.htm

Add a B2B Gateway to your services

regumindtrail — Wed, 04 Mar 2009 09:36:15 +0000

Mention SOA or Services and most of your audience would immediately relate it to web-services – yes, the often un-intented misuse of XML over Http that gives the technology and anything related to it a bad name in the world of high-performance J2EE applications.

Two of the biggest culprits in loss of Performance are I/O and Transformation overheads. Web services has both these drawbacks – increased data transfer i.e higher I/O associated with markup overhead of well-formed XML and the CPU utilization overhead when converting XML to Java and back aka. Marshalling.

Web-services and its implementation of XML over Http is good when it is genuinely needed. For e.g. exposing services for consumption with partner organizations where consuming technologies are not known or for integration between disparate systems. However often this need for integration unfortunately leads people to stereotype services as web-services in a SOA.

The question then is : can we reap the benefits of SOA and not suffer the drawbacks of the overheads inherent in web-services? I believe, we can.

Quite a while back, I read this excellent IBM Redbook : Implementing SOA using ESB where the author recommends deploying a B2B Gateway external to the ESB. I must admit it didnot make much sense to me then. I have come to appreciate it much better these days. A B2B Gateway enables consumption of services by “third-party” . This “third-party” may be a client from a different technology platform or from an altogether different organization.

A separate B2B gateway introduces the possibility of:

Making the web-service channel independent of the service implementation and therefore a matter of choice to use (and therefore suffer) the XML over Http interface
Introducing the much required security standards(and implementations) for securing services and data managed by the services
Using third party implementations that specialize in implementing WS-* policies
Using hardware to augment the processing capability provided by software frameworks – e.g. XML appliances

The SOA runtime therefore must enable services to be written independently of XML and the WS-* specifications/constraints. The Web-service interface is then an optional channel , via a B2B Gateway, to invoke the services.

We, at MindTree, have taken this design further in our implementation of Momentum – an SOA based delivery Platform. Interfaces like JMS and web-services are optional channels provided by the Framework to invoke any deployed service. A schematic that explains this approach is shown below:

The web-service interface is therefore an optional means to invoke your service when you separate the service container and the ESB(optional) from the B2B Gateway and deploy the latter as a separate infrastructure. You can then benefit from the good of web-services without compromising on your service’s QoS.

Leveraging stateful and stateless frameworks

regumindtrail — Tue, 16 Sep 2008 09:13:50 +0000

I would not be exaggerating if I said its a war out there between the supporters of stateful and stateless frameworks and solutions in Java. Add to this melee the number of frameworks that are built on either of these two premise. An application developer is therefore spoilt for choices and often confused. The easiest way out is to align with one of the camps and trust their solutions.

Proponents of the various frameworks vouch for their solutions – a natural thing, but unfortunately it is biased due to their own business interests. I’ll take two solutions for illustration purposes – Spring and JBoss Seam. I am a big fan of Spring and am interested enough to dabble in Seam. FYI, one of my projects uses Seam extensively and I personally have designed complex systems on the JBoss platform in the past. So, there is no bias there. I’ll use an example to illustrate the comment on bias - I find the Spring Acegi framework to be a great solution for application security. I suspect the people behind Seam might think likewise but will never integrate the two. I read something about they having evaluated Acegi and decided to write their own. JBoss Seam security looks good but one is forced to write stuff for say RDBMS or LDAP based authentication while it is readily available in Spring Acegi. Other useful features are Transaction(compensation based) support for LDAP operations and the Spring AOP implementation.

I do believe the need for evolution. Hibernate came along and changed the way we wrote JDBC access. But, I would hate to see somebody rubbish stuff that others have done before. For e.g. I dont subscribe to Gavin King’s (creator of Hibernate and Seam) thought that applications are by default stateful. Maintaining state comes with a price – we have all not forgotten the problems we had with earlier versions of Stateful EJBs, Entity beans and Http session replication. I dont think Seam has cracked this entirely yet despite claims that the JBoss clustering on JGroups does fine grained replication of changes. My team members complain of huge files on the JBoss server environment that is presumably used to passivate state. I would not convince myself to write all my applications as stateful ones even if the associated framework claims greater productivity (this is debatable and the answers are unique to situations).

The best solution for an application developer like me would be to use the best of both worlds. There definitely is a case for using both Spring and Seam in your application stack. Lets look at an architectural paradigm that has generated lot of interest these days – SOA. Web services over Http and JMS are common technologies used to implement services in a SOA. Both these technologies advocate a request-response model and are mostly stateless. Spring and its integration with a framework like Mule make a compelling choice to implement the services layer. However you need more than just services to build an end application – you need a presentation layer, a mechanism to maintain conversational state (between user and application) and ability to perform data access where the nature of the use case does not require you to write it as a service. Enter Seam and other productivity enhancing web frameworks like Grails. Presentation frameworks and component libraries (Facelets, JSF component libraries, jQuery) e.t.c integrate nicely and is rarely an issue.

Such an approach allows one to leverage the scalability benefits of stateless frameworks (for the services layer) and the productivity benefits of the stateful frameworks. A combined reference architecture that demonstrates building stateful applications on a stateless service layer in the true spirit of SOA is useful. The MindTree Momentum platform achieve this – leverage the best of open source stateful and stateless frameworks.

MCA or SCEA

regumindtrail — Tue, 09 Sep 2008 10:33:25 +0000

Let me set one thing straight – I have never been a big fan of certifications. It stems from ones on programming languages that require you to remember stuff that is easily found in the API docs. And, of course the aversion to written examinations that all of us with Indian schooling know so well of and consequently loathe I have therefore never taken up any professional certifications.

So, it was with some misgiving that I responded to a colleague’s query if I would like to take up the Microsoft Certified Architect evaluation. Having used Java predominantly the last few years, my first and obvious question was – how is the MCA relevant to a Java person? I listened a bit to the process and decided to give it a try for two reasons:

The evaluation process was around a solution that one has architected i.e. it was case study based
My colleague’s persuasion that it was a worthwhile experience

I’ll leave the process of evaluation out and just say this much that I created a few documents on competency, case study and a presentation that was to be presented to the board. The initial screening round was telephonic and was done by an existing MCA.

The review board was a 4 member team that I had to meet face to face and “defend the case study” as the process put it. I was truly impressed by the end of it all – not because I got the certification but because of the manner in which it was conducted. The review by the board was the highlight of the process and I mentioned thus when asked about the motivation for taking this up.

A few impressive things about the process :

Emphasis on one’s own experience in architecting a solution and almost neglible theritical knowledge. One can in no way prepare for this evaluation overnight.
Thorough professionalism of the people involved. You work and interface with existing MCAs throughout the process. I interacted with a total of 7. This is significant if you consider that there are a little over 100 MCAs worldwide. Thats the amount of focus and attention every candidate receives.
Clear sepraration of roles being certified – Solution architect & Infrastructure architect. There is Enterprise Architect as well. I applied for Solution architect.
The feedback one receives at the end of the certification. This feedback is very personalized. For e.g. specific books that one is advised to read based on the board review.
The collective experience of the architects on the board. It is OK to admit you dont know answers to questions(you will during the review ) that come at you in quick succession. I learnt it is difficult to match the collective knowledge of all those gentlemen on the board. I am guessing the average age of a MCA on the board should be well over 40 – seasoned architects surely!
Inclusion into the fraternity of MCAs. This is a small group of like minded people. I was invited to be a panelist speaker at the “Microsoft Architecture Days” session in Bangalore. I was allowed to talk about Java when discussing the topic “Software + Services”! This might be unheard of before.
Lastly the nicely mounted certificate signed by Bill Gates

Where does the “Sun Certified Enterprise Architect” stand in comparision? Again, I was unaware of its existence until recently when I saw it on a candidate’s resume. I was impressed by the name and the source and therefore did some googling on what it took to become a SCEA. This was before I spoke to the candidate who was a SCEA. I was disappointed by what I heard and what I read about the certification:

Firstly, the evaluation process tests one’s skills at application design and at best solution architecture. I feel it is a complete misnomer to have “enterprise architect” in its name.
The process is mostly offline. I wonder how “cheating” is prevented i.e. one person defines the solution on behalf of another.
Competencies essential to an architect such as communication, leadership, process, e.t.c are not evaluated here at all.
All the benefits or highlights I experienced in the MCA are missing here.

It would really do good to make the SCEA more interactive, personal, effective and valued especially since it comes from such a credible source as Sun Microsystems. Until then I would continue to vouch for the MCA programme – true value from Microsoft.

Accessing an OSGi service from a non-OSGi client

regumindtrail — Tue, 10 Jun 2008 13:03:57 +0000

I warmed up to the idea of OSGi after reading an introduction like : “Use OSGi to service enable your JVM”. For background, we at MindTree have created a SOA based J2EE delivery platform built on open source technologies – primarily Spring. In this framework, we expose POJOs as services using Spring remoting and access them via the Mule ESB. The idea of being able to expose POJOs as services without having to remote enable them made OSGi an interesting proposition.

I chose Spring Dynamic Modules(Spring DM) for three reasons:

I hoped to leverage all of Spring’s capabilities while writing my service
Ability to manage my service POJOs as Spring beans
Easy migration path for our framework

My test was to write a POJO, declare it as a Spring bean and an OSGi service (using Spring DM) and eventually access the service from a standalone application. This completes a typical usage scenario. I used the “simple-service” sample code that comes with the Spring DM distribution (version 1.1.0).

I faced issues either because I was not able to find the right documentation or because it was simply not there. I encountered and eventually overcame the following issues:

Ability to launch an OSGi container in standalone mode and without the prompts – most samples on the internet showed me how to bring up Equinox with the “osgi>” prompt. This was easily solved though.
Ability to load all the required OSGi bundles *before* I could load my bundle
Look up a published service from the bundle I deployed and invoke it from my standalone code

A few lessons learned from this exercise are:

OSGi is very strict when it comes to class visibility and loading. Parent class loader delegation does not apply – something we are very used to in J2SE and J2EE applications
It is easy to share services and packages between bundles. Not all that easy if you want to share between a bundle and a non-OSGi client application. For e.g, you cannot simply type-cast a service you have looked up to an interface that is packaged in the bundle and available on your local classpath.
Boot path delegation (creating fragment bundles i.e extensions to the system bundle) can help address the above need but must be used selectively and carefully in order to preserve the benefits of OSGi.
All bundles are loaded asynchronously. You need to account for this before you look up any service.

I have replicated below the manifest contents from my fragment bundle (one that exports the common interfaces & classes) and the service bundle (one that imports the common interfaces and implements the service):

Fragment manifest:

——————————————————–

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: m1000350
Build-Jdk: 1.5.0
Bundle-Version: 1.0
Bundle-SymbolicName: org.springframework.osgi.samples.simpleservice.boot
Bundle-Name: Simple-Service-Sample-boot
Bundle-Vendor: Spring Framework
Fragment-Host: system.bundle <— note this extension
Export-Package: org.springframework.osgi.samples.simpleservice <— note this export
Bundle-ManifestVersion: 2

———————————————————

The manifest for my service bundle looks like:

——————————————————–

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: m1000350
Build-Jdk: 1.5.0
Bundle-Version: 1.0
Spring-Context: *;create-asynchronously=false <— Spring DM integration
Bundle-SymbolicName: org.springframework.osgi.samples.simpleservice
Bundle-Name: Simple-Service-Sample
Bundle-Vendor: Spring Framework
Bundle-ManifestVersion: 2
Import-Package: org.springframework.osgi.samples.simpleservice <— note this import

——————————————————–

I have attached the code for my sample application. Unfortunately, the complete Spring DM integration (the use of OsgiBundleXmlApplicationContext) does not work and I have posted a question on the Spring support forum :Spring DM issue.

I was however able to invoke the OSGi service from my non-OSGi client.

Disclaimer: The code is just a sample and is not validated in its approach or manner of using the OSGi platform.

OSGiTest - Rename to .java and open in your favorite editor

ESB – a logical set of capabilities?

regumindtrail — Thu, 29 May 2008 11:18:20 +0000

Today it would not raise many eye-brows if I said that an application server need not necessarily be a physical standalone entity or process and can actually be “assembled” from a set of available frameworks. Thanks to frameworks like Spring and standards like OSGi (and containers supporting it), this can be realized and not remain just on paper.

How about an ESB? The picture that most of us have seen (and reproduced in this post : Myth of the ESB ) appear to imply this big central infrastructure that acts as a conduit for all integration and message based interactions within an enterprise.

How easy or difficult is it to realize such a deployment in an organization? I have worked on SOA for a few years and more recently on deploying ESB as an infrastructure component in a SOA. Not once in these cases have we done such a deployment. Why? The answer is simple – most organizations take an incremental approach to SOA adoption and not all existing applications are amenable for migrating their business logic as services and thereafter accessed via an ESB.

Does this mean an ESB is only a promise? No, if you care to see it a little differently. I’ll go back to my earlier example of an application server. In the world of light-weight containers, components and dynamically loaded modules, the application server can be seen as a set of capabilities realized through an integrated set of components,frameworks,libraries and a runtime. The advantage here is that one can use as much of the application server as is needed.

If I were to apply the same analogy to an ESB, the first question that comes to my mind is – what are the capabilities of an ESB (or) what can an ESB provide to an application? IBM puts it nicely (and using a nice term) by defining a “Capability Model” in their Redbook on “Implementing an SOA using an ESB”. This capability model, for me, also serves as a good yard stick to measure an ESB offering – sort of a self compliance check.

A gist of the capability model is listed below:

Communication : Routing, Addressing, Response / request, Publish / subscribe, Sync/Async e.t.c
Service Interaction: Service interface definition(WSDL), Substitution of service
implementation, Service directory and discovery
Integration : Database, Legacy and application adapters, Service mapping, Protocol transformation
Quality of Service : Transactions, assured delivery
Security : Authentication, Authorization, Security standards
Service Level : Performance, Throughput, Availability
Message Processing : Content-based logic, Message and data
transformations, Validation
Management and Autonomic : Administration capability, Logging, Metering, Monitoring
Modeling : Object modeling, Common business object models
Infrastructure Intelligence : Business rules, Policy driven behavior

Now thats a huge list of capabilities and you may not need all of them – again like in an application server, you may not need JNDI, JTA, JMS and an EJB container all at once. However you can integrate these capabilities when needed. Can you do the same for an ESB? Yes, you can. Its a different post altogether how one can do that. One might ask this question : but why take the trouble?

The answers come from the cost of deployment, the set of features truly available in the fancy ESBs and the flexibility of actually using an ESB’s features but within your own runtime process. There are quite a few light-weight ESBs that help you do this. If you carefully look at the features in these ESBs, you’ll notice that they fall short of the Capability Model in more ways than one. This is true for the commercial ESBs as well. The lack of standards adds to the disconnect. In the meanwhile, considering an ESB as a logical set of capabilities and using it that way puts many things into perspective – including the value of an ESB in a SOA.