Friday, 29 November 2013

Using Flyway with OSGi (Part One)


I spent a lot of time making flyway work in an OSGi container (felix) and I'd like to share my findings. Felix support in flyway was added not too long ago, and there is not much information to be found on the website (yet).

My main sources were this issue, this github example project and, of course, the flyway source code.

P.S. I hope you like lengthy blog posts ;) And, sorry, no pictures.


Objective

[where I tell you what I want to achieve]

What I want to create here (in the first part about flyway with OSGi) as an example is a bit more complicated than the before-mentioned github example project:

I want to have 
  • a bundle with an entity, a persistence unit and a repository
  • a couple of SQL scripts defining the database and initial data
  • an integration test case actually calling the repository. 
(Part two will add another bundle with another entity which has a manyToOne relation to the first one).


The example

[where you will find the code]

If you want to check out the glory details, you can find the code for the example here. If you use eclipse with bndtools, please create a new workspace, as a cnf folder is part of the example (and would clash with a second one).

The projects you care about are the ones in the osgi/flyway/contacts subfolder.

Import them into your IDE, and, assuming you are using bndtools, everything should be fine. (Otherwise, run "ant build" in both examples.osgi.flyway.contacts.* subdirectories).

First test

[where you can make sure it does what it should]

In eclipse, with bndtools, you can right-click on the examples.osgi.flyway.contacts.it project ("it" stands for integration test) and run it as "Bnd OSGi Test Launcher (JUnit)". On the console, you should see that the test has passed, and, after refreshing the project, you should see a new folder "exampleContactsDb" together with some new log and DDL files (ending with .jdbc).

The same can be achieved by running "ant test" in the same directory.

The example uses an embedded derby database, and you can use the usual tools to look inside the databases contents if you wish.

Some background and some impediments

[where I provide some information why I did it that way]

In order to make the example work, I had to do some investigation first (skip this section if you only want to see how it is done in code. But I do recommend reading this nevertheless ;)).

It seems like the only way to utilize flyway in OSGi is by using fragments (sigh!). So, the contacts bundle, containing the DDL and SQL scripts should be a fragment attached to the com.googlecode.flyway.core bundle. On the other hand, I need to execute some code when the bundle starts in order to initialize the flyway update or init process. For this, I need to create (or reuse) a datasource.

For proper reuse, the datasource (or the initialization properties for a new datasource) should not be defined inside the bundle, but be provided from the outside, of course (e.g. using some kind of "datasourceProvider" or by passing the properties from the config admin implementation). So, being inside an OSGi container, I should use some OSGi service to provide the needed data.
But, as the OSGi compendium states in 112.4.2 ("Service Component Header")
"A Service-Component manifest header specified in a fragment is ignored by SCR"
So, using declarative services (aka SCR) and defining a fragment will not work for me here. [A resort could be to use a blueprint service (see this stackoverflow question) - but I want to stick to SCR and not introduce another technology to manage my services (and end up with both felix SCR and some blueprint bundles)].

Solution

[where I tell you how I overcame the obstacles]

So, what to do? I need a fragment bundle and use services. Well, let's have two bundles then. The first one (contacts.core) will contain the flyway initializer, the repository to access the database and the persistence unit definition. The second one (contacts.db) will define the DDL and initial SQL scripts.

Fortunately, using bndtools, I will not have to define two projects for the two bundles: one project with two bundle descriptors will do the trick (remember, the second bundle in the example is the integration test!).

The glory details

[where you find what you need to know to do it yourself]

In this section, I will show or reference all the important bits of the example project and give you some information about why I did things the way they are. (Btw, I don't mind getting feedback telling me that things could be done in a different, more elegant way ;))

All the files mentioned here are in the examples.osgi.flyway.contacts project.

1. The persistence unit

The usual persistence.xml file is defined in the resources/META-INF folder and will be copied (by bnd) into the contacts.core bundle. There is nothing (really) special about that file, it defines the "ContactsDB" persistence unit with transaction-type Resource-Local (as recommended for OSGi JPA). It lists the entities and takes care about the database connection properties.

2. The bnd file



This is the entry point for the bnd library.

In -buildpath, I define all the bundles necessary for compilation of the contacts project (the referenced jars are provided in the cnf project). 

In -sub, I let bnd know that I want to create more the one bundle from this project: Each other *.bnd file found in the project will define the contents of a derived bundle.

3. The core.bnd file

This file defines the contents of the examples.osgi.flyway.contacts.core bundle.


Include-Resource adds the resources directory to the created bundle, and the other instructions will make sure our MANIFEST.MF file will look the way it should. Nothing else will end up in the contacts.core bundle.

Service-Component: * will instruct bnd to scan the exported packages for @Component annotations. The annotated classes will be used to define OSGi declarative services automatically. They will be created during the build and end up in the OSGI-INF directory of the generated bundle.

Important: Do not forget to add the org.osgi.framework Import-Package instruction. Without, flyway will give you a "unsupported protocol: bundle" error).

4. The db.bnd file

Similar to the core.bnd file, this file defines the contents of the examples.osgi.flyway.contacts.db bundle.


Again, some (but not all) resources of the project are copied into the generated bundle, and the fragment host is defined

5. The FlywayInit class

There is a little trick involved making flyway actually do what I want it to do (as of version 2.2.1):
Replacing the ContextClassLoader before giving control to flyway (and then, after that, reverting that change again) helps flyway finding the files it searches for.
 

Conclusion

[where I..., well, you guessed it]

As I said, I spend quite a lot (too much?) time getting this to work, and I still have Part II ahead of me (which adds some more complexity and requires a workaround which I am not really happy with).

As I didn't find any tutorials going into this depth, I had to write one myself, and I guess, if you are still with me here, you liked it. At least I hope so. Give me your feedback or your insights, and spread the word.

See you in Part II.



Saturday, 16 November 2013

Getting started with OSGi and Bnd

There are many tutorials and books about OSGi, about what it is good for, why (and when) it is important to modularize your software and how OSGi can help you with that.

Many of those tutorials start with some interface / implementation example and guide you through the steps of how to create the necessary bundles in some IDE (often eclipse) and run them.

This is definitely a good approach to teach someone about OSGi, how it looks and feels like and about what modularity is good for. But I have the impression that there is a gap or lack of tutorials and documentation when it comes to bigger, more real life examples. You do not only want to create some "Hello World" bundle and let it run inside your IDE. If you want to let your OSGi project become real, you want to automatically build and test it, you want to have test coverage and a build pipeline (even continuous delivery!), and, what was the biggest challenge for me at least, you somehow want to mange all the bundles, yours, the ones you depend on (build- and runtime), and, not to forget, you somehow have to deal with those third-party jars which are not OSGi bundles yet.

I was using maven a long time to get my bundles built; the pom.xml files contained a section like this

and a matching file like this (osgi.bnd)



(If you want to use this, please note that there is an minus sign before the name of the bnd file in the pom.xml!)

This approach was working well, but it lacked all tooling help. What I mean is, that I was in full control of everything, being able to define whatever I needed in the osgi.bnd file. I could give a package a version and the maven-bundle-plugin would reliably pick it up, create the bundle jar and a matching manifest entry.

But it was a pain to figure out which version to use for that package. OSGi has the powerful concept of semantic versioning (you can find some details about this here). Problem is, that if you make mistakes with the version you define, the concept gets much less powerful, and sooner or later you have broken backwards compatibility.

Another problem with the maven-bundle-plugin approach was integration testing. I was using pax-exam, which is, no doubt, a great tool. But my pom files became huge. And I had to write glue code to setup testing, that is to make the test find the proper bundles in my maven repository. Testing can be fun, and it should be. Writing tons of code just to get the proper initial setup is no fun. 

Enter bndtools.

Reading this book convinced me to try bndtools once more. I had tried it years ago and had run into problems I cannot recall. What I didn't like about it was that it would tie me to eclipse as it is a eclipse plugin. But, ok. And it was awesome! No issues with versioning no more (bndtools supports baselining, that is, it will inspect your code and compare it to a previous version to determine which new version numbers to give to your packages). And, OSGi integration testing became as easy as it can possibly get.

But, still... with this new approach, I would force fellow code contributors to use eclipse. And how about the headless build?

Well, no harm done - the build chain can be invoked from good old ant, and, if you were using a different IDE, you could still easily just use the ant targets to build your bundles. You would not get the nice bndtools editors, but, hey, you are using a different IDE.

Enter bnd, the "Swiss army knife of OSGi".

Reading about bndtools and checking the ant targets and the code I understood that the heavy lifting under the hood was all done by the bnd tool - the same software which is used in the maven-bundle-plugin (and some more OSGi tools) as well.

So, getting curious, I started to dig into the bnd code and decided to start a blog series about this tool.

tl;dr: If you are serious about writing OSGi applications, you should definitely know how bnd works.

There is a lot of documentation on the bnd web page, but - at least to me - it seems that you have to know a lot about OSGi and even bnd in the first place before you can really understand what the documentation talks about. This is a big impediment for beginners of OSGi and I'd like to change that by writing about my insights and experiences with bnd.

If you are interested, stay tuned.