Tag: Java EE

Revisiting Global Data Consistency in Distributed (Microservice) Architectures

Back in 2015 I wrote a couple of articles about how you can piggyback a standard Java EE Transaction Manager to get data consistency across distributed services (here is the original article and here is an article about doing it with Spring Boot, Tomcat or Jetty). Last year I was fortunate enough to work on a small project where we questioned data consistency from the ground up. Our conclusion was that there is another way of getting data consistency guarantees, one that I had not considered in another article that I wrote about patterns for binding resources into transactions. This other solution is to change the architecture from a synchronous one to an asynchronous one. The basic idea is to save business data together with "commands" within a single database transaction. Commands are simply facts that other systems still need to be called.By reducing the number of concurrent transactions to just one, it is possible to guarantee that data will never be lost. Commands which have been committed are then executed as soon as possible and it is the command execution (in a new transaction) which then makes calls to remote systems. Effectively it is an implementation of the BASE consistency model, because from a global point of view, data is only eventually consistent. Imagine the situation where updating an insurance case should result in creating a task in a workflow system so that a person gets a reminder to do something, for example write to the customer. The code…

Read more

Global Data Consistency, Transactions, Microservices and Spring Boot / Tomcat / Jetty

We often build applications which need to do several of the following things together: call backend (micro-) services, write to a database, send a JMS message, etc. But what happens if there is an error during a call to one of these remote resources, for example if a database insert fails, after you have called a web service? If a remote service call writes data, you could end up in a globally inconsistent state because the service has committed its data, but the call to the database has not been committed.In such cases you will need to compensate the error, and typically the management of that compensation is something that is complex and hand written. Arun Gupta of Red Hat writes about different microservice patterns in the DZone Getting Started with Microservices Refcard. Indeed the majority of those patterns show a  microservice calling multiple other microservices. In all these cases, global data consistency becomes relevant, i.e. ensuring that failure in one of the latter calls to a microservice is either compensated, or the commital of the call is re-attempted, until all the data in all the microservices is again consistent. In other articles about microservices there is often little or no mention of data consistency across remote boundaries, for example the good article titled "Microservices are not a free lunch" where the author just touches on the problem with the statement "when things have to happen ... transactionally ...things get complex with us needing to manage ... distributed transactions to…

Read more

Several Patterns for Binding Non-transactional Resources into JTA Transactions

I recently published an article about how to bind non-transactional resources like web services / microservices into global distributed transactions so that recovery is handled automatically. Over the years I have often had to integrate "non-transactional" systems into Java EE application servers and data consistency was often a topic of discussion or even a non-functional requirement. I've put "non-transactional" into quotes because often the systems contain ways of ensuring data consistency, for example using calls to compensate, but the systems aren't what you might traditionally call transactional. There is certainly no way of configuring a Java EE application server to automatically handle recovery for such resources. The following is a list of patterns that we compiled, showing different ways to maintain consistency when faced with the task of integrating a non-transactional system. Write job to database - The common scenario whereby you want to send say an email confirmation after a sale is made. You cannot send the email and then attempt to commit the sales transaction to your database, because if the commit fails, the customer receives an email stating that they have bought something and you have no record of it. You cannot send the email after the sales transaction is committed to your database, because if the sending of the email fails (e.g. the mail server is temporarily down), the customer won't get their confirmation, perhaps with a link to the tickets that they bought. One solution is to write the fact that an email needs to…

Read more

Mysql versions prior to 5.7 do not fully support two phase commit

While doing some tests for the recently released generic JCA adapter which is capable of binding remote calls to microservices (as well as other things) into JTA transactions, I discovered a bug in Mysql 5.6 which has been around for nearly ten years. The test scenario was a crash after the "prepare" phase of the XA transaction, after both the database and the generic connector vote to commit the transaction. After crashing the database or the application server, the transaction manager will try to recover and commit the transaction in the database. But while testing the crashes with Mysql 5.6 rather than Mysql 5.7, I was having the problem that the database never actually committed the transaction, meaning that the system as a whole was in an inconsistent state. There were absolutely no problems with the generic connector, just the database. The application server continuously logged that there was an incomplete transaction but was unable to complete the transaction in the database. During the simulated database crash, the application returned a HeuristicMixedException to indicate to the user that something is not and will not ever be consistent. The error logged by JBoss was: ...WARN (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid: <...eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK ...> I spent time debugging in the Mysql driver code but eventually came across MySQL Bug #12161 which has only just been closed after being open for nearly 10 years! It is…

Read more

Global Data Consistency in Distributed (Microservice) Architectures

UPDATE: Now supports Spring and Spring Boot outside of a full Java EE sever. See new article for more details! UPDATE 2: See this new article for more details about using asynchronous calls to remote applications to guarantee global data consistency I've published a generic JCA resource adapter on Github available from Maven (ch.maxant:genericconnector-rar) with an Apache 2.0 licence. This let's you bind things like REST and SOAP web services into JTA transactions which are under the control of Java EE application servers. That makes it possible to build systems which guarantee data consistency, with as little boiler plate code as possible, very easily. Be sure to read the FAQ. Imagine the following scenario... Functional Requirements ... many pages of sleep inducing requirements... FR-4053: When the user clicks the "buy" button, the system should book the tickets, make a confirmed payment transaction with the acquirer and send a letter to the customer including their printed tickets and a receipt. ... many more requirements... Selected Non-Functional Requirements NFR-08: The tickets must be booked using NBS (the corporations "Nouvelle Booking System"), an HTTP SOAP Web Service deployed within the intranet. NFR-19: Output Management (printing and sending letters) must be done using COMS, a JSON/REST Service also deployed within the intranet. NFR-22: Payment must be done using our Partner's MMF (Make Money Fast) system, deployed in the internet and connected to using a VPN. NFR-34: The system must write sales order records to its own Oracle database. NFR-45: The data related to a…

Read more

Is Asynchronous EJB Just a Gimmick?

In previous articles (here and here) I showed that creating non-blocking asynchronous applications could increase performance when the server is under a heavy load. EJB 3.1 introduced the @Asynchronous annotation for specifying that a method will return its result at some time in the future. The Javadocs state that either void or a Future must be returned. An example of a service using this annotation is shown in the following listing: The annotation is on line 4. The method returns a Future of type String and does so on line 10 by wrapping the output in an AsyncResult. At the point that client code calls the EJB method, the container intercepts the call and creates a task which it will run on a different thread, so that it can return a Future immediately. When the container then runs the task using a different thread, it calls the EJB's method and uses the AsyncResult to complete the Future which the caller was given. There are several problems with this code, even though it looks exactly like the code in all the examples found on the internet. For example, the Future class only contains blocking methods for getting at the result of the Future, rather than any methods for registering callbacks for when it is completed. That results in code like the following, which is bad when the container is under load: This kind of code is bad, because it causes threads to block meaning that they cannot do anything useful during that…

Read more

Javascript everywhere

I can think of at least two scenarios when you might need to run the same algorithms in both a Javascript and a Java environment. A Javascript client is running in offline mode and needs to run some business logic or complex validation which the server will need to run again later when say the model is persisted and the server needs to verify the consistent valid state of the model, You have several clients which need access to the same algorithms, for example a Javascript based single page application running in the browser and web service which your business partners use which is deployed in a Java EE application server. You could build the algorithm twice: once in Java and once in Javascript but that isn't very friendly in terms of maintenance. You could build it once in just Java and make the client call the server to run the algorithm, but that doesn't work in an offline application like you can build using HTML 5 and AngularJS, and it doesn't make for a very responsive client. So why not build the algorithm just once, using Javascript, and then use the javax.script Java package which first shipped with Java SE 7 and was improved with Java SE 8, in order to execute the algorithm when you need to use it from Java? That is precisely what I asked myself, and so I set about building an example of how to do it. The first thing I considered was deployment…

Read more

Play 2.0 framework and XA transactions

XA transactions are useful and out of the box, Play 2.0 today does not have support for them.  Here I show how to add that support: First off, some examples when XA is useful: - JPA uses two physical connections if you use entities from two different persistence.xml - those two connections might need to be committed in one transaction, so XA is your only option - Committing a change in a database and at the same time committing a message to JMS.  E.g. you want to guarantee that an email is sent after you successfully commit an order to the database, asynchronously.  There are other ways, but JMS provides a transactional way to do this with little overhead in having to think about failure. - Writing to a physically different database because of any of several political reasons (legacy system, different department responsible for different database server / different budgets). - See http://docs.codehaus.org/display/BTM/FAQ#FAQ-WhywouldIneedatransactionmanager So the way I see it, XA is something Play needs to "support". Adding support is very easy.  I have created a play plugin which is based on Bitronix.  Resources are configured in the Bitronix JNDI tree (why on earth does Play use a config file rather than JNDI?! anyway...)  You start the transaction like this, "withXaTransaction":     def someControllerMethod = Action {                     withXaTransaction { ctx =>                         TicketRepository. addValidation(user.get, bookingRef, ctx)                         ValidationRepository.addValidation(bookingRef, user.get, ctx)                     }    …

Read more

JSR-299 & @Produces

I've been reading JSR-299 and I have a few thoughts. First, a quote from chapter 1: "The use of these services significantly simplifies the task of creating Java EE applications by integrating the Java EE web tier with Java EE enterprise services. In particular, EJB components may be used as JSF managed beans, thus integrating the programming models of EJB and JSF." The second sentence made my ears prick up. Do I really want EJBs to be the backing beans of web pages? To me that sounds like one is mixing up responsibilities in MVC. Sure, there are people out there who say that an MVC Controller should be skinny and the model should be fat (youtube ad). Perhaps I'm old school, but I prefer my JSF page to have a backing bean which is my view model and deals with presentation specific logic, and when it needs transaction and security support, it can call through to an EJB which deals with more businessy things. The JSR then goes on to introduce the @Produces annotation. I don't like that annotation and the rest of this blog posting is about why. When I need an object, like a service or a resource, I can get the container to inject it for me. Typically, I use the @Inject, @EJB, @Resource or @PersistenceContext annotations. In the object which has such things injected, I can write code which uses those objects. I let the container compose my object using those pieces and other beans.…

Read more

Tomcat, WebSockets, HTML5, jWebSockets, JSR-340, JSON and more

On my recent excursion into non-blocking servers I came across Comet, server push technologies and then Web Sockets. I was late arriving at the Comet party, but I think I have arrived at the Web Sockets party just in time. The final standard is still being evovled and at the time of writing, the only browser supporting it by default is Chrome. So to get started, I had a look at what was around. Initially, I wasn't going to write a blog article about it, I just wanted to learn about this new thing. My only requirement was that the server implementation was Java based. The reason is simple - my entire technology stack is based on Java EE and I want to be able to integrate all my existing stuff into any new server, without the need for an integration bus. I like being able to drop a JAR file from one project into another one and to be up and running immediately. So I had a quick look at jWebSockets, Grizzly (Glassfish), Jetty and Caucho's Resin. All similar, yet all somewhat different. Most different, was jWebSockets, because they have gone to the extent of building their own server, instead of basing their solution on an existing server. I had a good long rant about that kind of thing when I blogged about node.js, so I won't start again. But jWebSockets has a real issue to face in the coming years. JSR-340 talks about support for web technologies based…

Read more