Monday, 9 December 2013

Creating a java service wrapper for an OSGi application

I admit, I have my doubts if somebody actually is interested in this.

And, probably, this is one of the worst sentences you could start your blog post with ;)

On the other hand, as you are actually reading this, you have found this page. So, you might have searched for it. Meaning, you really might be interested in learning how to create a service for an OSGi application!

First, a quick definition:

When I talk about "services" here, I mean things like "windows services" or "unix daemons", i.e. programs running in the background (which can be started automatically when the computer boots and which can react to system events).

An "OSGi application" in the context of this blog post is an executable jar file (created with bndtools). If somebody is interested in how you'd create such an executable jar file, let me know and I will write a blog post about it ;)).

Setup and Preparation

Let's assume we have the executable OSGi jar file, one you'd start from the command line like "java -jar myosgiapp.jar". And, let's assume we are on some unix machine (windows should work more or less the same way).

How to turn that into a service?

The - from my observation - most popular library for creating services is called the tanuki java service wrapper and is available under various licences (there should be one satisfying your needs).

In my case, I downloaded the community edition and unzipped it:

   tar -zxvf wrapper-linux-x86-64-3.5.23.tar.gz  

Checking out the documentation you will learn that there are four ways to create services; we will use #4 (which is about turning executable jars into services).

I create a new directory

  mkdir servicetest
 cd servicetest

and follow the instructions, that is, I create a couple of subdirectories:

  mkdir lib
  mkdir conf
  mkdir bin
  mkdir logs

and copy some files from the downloaded tanuki directory:

  cp <path_to_tanuki>/bin/wrapper bin
  cp <path_to_tanuki>/src/bin/sh.script.in bin/myosgiapp
  chmod 755 bin/myosgiapp

As you can see, I renamed sh.script.in into myosgiapp. Let's change that file a little bit. Open an editor and adjust the APP_NAME and APP_LONG_NAME to something that suits you.

Some more stuff to do:

  cp <path_to_tanuki>/lib/libwrapper.so lib
  cp <path_to_tanuki>/lib/wrapper.jar lib

and, last bit, the configuration file and the actual program

  cp <path_to_tanuki>/src/conf/wrapper.conf.in conf/wrapper.conf
  cp <path_to_osgijar>/myosgiapp.jar lib


Configuration

First thing to update in your wrapper.conf file is this property:

  wrapper.java.mainclass=...

set it to
 
  wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperJarApp

Furthermore, this here:

 
  wrapper.java.classpath (...)


should be set to
 
  wrapper.java.classpath.1=<path_to>/lib/wrapper.jar 

plus

  wrapper.java.classpath.2=<path_to>/lib/myosgiapp.jar

and (I know, we have to set the myosgiapp.jar a second time)

  wrapper.app.parameter.1==<path_to>/lib/myosgiapp.jar

Enter... the service

Now, as a reward for all this messing-around-with-files-and-dirs, you should be able to run 

  bin/myosgiapp start 

Your application should now have started as a service, giving you some PID on the console as an indication that everything went fine. Run

  bin/myosgiapp stop 

to shutdown the application and check the log under logs/wrapper.conf to see it things happened the way they should have happened. Omitting the start or stop instruction will give you a list of available command options.

There are a lot of things to consider when shutting down an application like this... basically, what tuniki does in this simple case, is a "System.exit(0)". You should create a shutdown hook to make your application disappear more gracefully.

And now?

Congrats! You are the first one ever to read this line!

Just kidding ;) Thing is, that I don't really know how many use cases there are to do a thing like this. In my case, what I want to achieve is the following:

  1. commit new code in some bundle participating in my OSGi app
  2. let jenkins build the changed bundle and run the tests
  3. if "still green", let jenkins build, test and deploy the whole app
  4. if "still green", let jenkins shut down the running demo app and restart it
The goal is to create a reliable continuous delivery chain for an OSGi application.

As usual, feedback is welcome ;)





Friday, 29 November 2013

Using Flyway with OSGi (Part One)


I spent a lot of time making flyway work in an OSGi container (felix) and I'd like to share my findings. Felix support in flyway was added not too long ago, and there is not much information to be found on the website (yet).

My main sources were this issue, this github example project and, of course, the flyway source code.

P.S. I hope you like lengthy blog posts ;) And, sorry, no pictures.


Objective

[where I tell you what I want to achieve]

What I want to create here (in the first part about flyway with OSGi) as an example is a bit more complicated than the before-mentioned github example project:

I want to have 
  • a bundle with an entity, a persistence unit and a repository
  • a couple of SQL scripts defining the database and initial data
  • an integration test case actually calling the repository. 
(Part two will add another bundle with another entity which has a manyToOne relation to the first one).


The example

[where you will find the code]

If you want to check out the glory details, you can find the code for the example here. If you use eclipse with bndtools, please create a new workspace, as a cnf folder is part of the example (and would clash with a second one).

The projects you care about are the ones in the osgi/flyway/contacts subfolder.

Import them into your IDE, and, assuming you are using bndtools, everything should be fine. (Otherwise, run "ant build" in both examples.osgi.flyway.contacts.* subdirectories).

First test

[where you can make sure it does what it should]

In eclipse, with bndtools, you can right-click on the examples.osgi.flyway.contacts.it project ("it" stands for integration test) and run it as "Bnd OSGi Test Launcher (JUnit)". On the console, you should see that the test has passed, and, after refreshing the project, you should see a new folder "exampleContactsDb" together with some new log and DDL files (ending with .jdbc).

The same can be achieved by running "ant test" in the same directory.

The example uses an embedded derby database, and you can use the usual tools to look inside the databases contents if you wish.

Some background and some impediments

[where I provide some information why I did it that way]

In order to make the example work, I had to do some investigation first (skip this section if you only want to see how it is done in code. But I do recommend reading this nevertheless ;)).

It seems like the only way to utilize flyway in OSGi is by using fragments (sigh!). So, the contacts bundle, containing the DDL and SQL scripts should be a fragment attached to the com.googlecode.flyway.core bundle. On the other hand, I need to execute some code when the bundle starts in order to initialize the flyway update or init process. For this, I need to create (or reuse) a datasource.

For proper reuse, the datasource (or the initialization properties for a new datasource) should not be defined inside the bundle, but be provided from the outside, of course (e.g. using some kind of "datasourceProvider" or by passing the properties from the config admin implementation). So, being inside an OSGi container, I should use some OSGi service to provide the needed data.
But, as the OSGi compendium states in 112.4.2 ("Service Component Header")
"A Service-Component manifest header specified in a fragment is ignored by SCR"
So, using declarative services (aka SCR) and defining a fragment will not work for me here. [A resort could be to use a blueprint service (see this stackoverflow question) - but I want to stick to SCR and not introduce another technology to manage my services (and end up with both felix SCR and some blueprint bundles)].

Solution

[where I tell you how I overcame the obstacles]

So, what to do? I need a fragment bundle and use services. Well, let's have two bundles then. The first one (contacts.core) will contain the flyway initializer, the repository to access the database and the persistence unit definition. The second one (contacts.db) will define the DDL and initial SQL scripts.

Fortunately, using bndtools, I will not have to define two projects for the two bundles: one project with two bundle descriptors will do the trick (remember, the second bundle in the example is the integration test!).

The glory details

[where you find what you need to know to do it yourself]

In this section, I will show or reference all the important bits of the example project and give you some information about why I did things the way they are. (Btw, I don't mind getting feedback telling me that things could be done in a different, more elegant way ;))

All the files mentioned here are in the examples.osgi.flyway.contacts project.

1. The persistence unit

The usual persistence.xml file is defined in the resources/META-INF folder and will be copied (by bnd) into the contacts.core bundle. There is nothing (really) special about that file, it defines the "ContactsDB" persistence unit with transaction-type Resource-Local (as recommended for OSGi JPA). It lists the entities and takes care about the database connection properties.

2. The bnd file



This is the entry point for the bnd library.

In -buildpath, I define all the bundles necessary for compilation of the contacts project (the referenced jars are provided in the cnf project). 

In -sub, I let bnd know that I want to create more the one bundle from this project: Each other *.bnd file found in the project will define the contents of a derived bundle.

3. The core.bnd file

This file defines the contents of the examples.osgi.flyway.contacts.core bundle.


Include-Resource adds the resources directory to the created bundle, and the other instructions will make sure our MANIFEST.MF file will look the way it should. Nothing else will end up in the contacts.core bundle.

Service-Component: * will instruct bnd to scan the exported packages for @Component annotations. The annotated classes will be used to define OSGi declarative services automatically. They will be created during the build and end up in the OSGI-INF directory of the generated bundle.

Important: Do not forget to add the org.osgi.framework Import-Package instruction. Without, flyway will give you a "unsupported protocol: bundle" error).

4. The db.bnd file

Similar to the core.bnd file, this file defines the contents of the examples.osgi.flyway.contacts.db bundle.


Again, some (but not all) resources of the project are copied into the generated bundle, and the fragment host is defined

5. The FlywayInit class

There is a little trick involved making flyway actually do what I want it to do (as of version 2.2.1):
Replacing the ContextClassLoader before giving control to flyway (and then, after that, reverting that change again) helps flyway finding the files it searches for.
 

Conclusion

[where I..., well, you guessed it]

As I said, I spend quite a lot (too much?) time getting this to work, and I still have Part II ahead of me (which adds some more complexity and requires a workaround which I am not really happy with).

As I didn't find any tutorials going into this depth, I had to write one myself, and I guess, if you are still with me here, you liked it. At least I hope so. Give me your feedback or your insights, and spread the word.

See you in Part II.



Saturday, 16 November 2013

Getting started with OSGi and Bnd

There are many tutorials and books about OSGi, about what it is good for, why (and when) it is important to modularize your software and how OSGi can help you with that.

Many of those tutorials start with some interface / implementation example and guide you through the steps of how to create the necessary bundles in some IDE (often eclipse) and run them.

This is definitely a good approach to teach someone about OSGi, how it looks and feels like and about what modularity is good for. But I have the impression that there is a gap or lack of tutorials and documentation when it comes to bigger, more real life examples. You do not only want to create some "Hello World" bundle and let it run inside your IDE. If you want to let your OSGi project become real, you want to automatically build and test it, you want to have test coverage and a build pipeline (even continuous delivery!), and, what was the biggest challenge for me at least, you somehow want to mange all the bundles, yours, the ones you depend on (build- and runtime), and, not to forget, you somehow have to deal with those third-party jars which are not OSGi bundles yet.

I was using maven a long time to get my bundles built; the pom.xml files contained a section like this

and a matching file like this (osgi.bnd)



(If you want to use this, please note that there is an minus sign before the name of the bnd file in the pom.xml!)

This approach was working well, but it lacked all tooling help. What I mean is, that I was in full control of everything, being able to define whatever I needed in the osgi.bnd file. I could give a package a version and the maven-bundle-plugin would reliably pick it up, create the bundle jar and a matching manifest entry.

But it was a pain to figure out which version to use for that package. OSGi has the powerful concept of semantic versioning (you can find some details about this here). Problem is, that if you make mistakes with the version you define, the concept gets much less powerful, and sooner or later you have broken backwards compatibility.

Another problem with the maven-bundle-plugin approach was integration testing. I was using pax-exam, which is, no doubt, a great tool. But my pom files became huge. And I had to write glue code to setup testing, that is to make the test find the proper bundles in my maven repository. Testing can be fun, and it should be. Writing tons of code just to get the proper initial setup is no fun. 

Enter bndtools.

Reading this book convinced me to try bndtools once more. I had tried it years ago and had run into problems I cannot recall. What I didn't like about it was that it would tie me to eclipse as it is a eclipse plugin. But, ok. And it was awesome! No issues with versioning no more (bndtools supports baselining, that is, it will inspect your code and compare it to a previous version to determine which new version numbers to give to your packages). And, OSGi integration testing became as easy as it can possibly get.

But, still... with this new approach, I would force fellow code contributors to use eclipse. And how about the headless build?

Well, no harm done - the build chain can be invoked from good old ant, and, if you were using a different IDE, you could still easily just use the ant targets to build your bundles. You would not get the nice bndtools editors, but, hey, you are using a different IDE.

Enter bnd, the "Swiss army knife of OSGi".

Reading about bndtools and checking the ant targets and the code I understood that the heavy lifting under the hood was all done by the bnd tool - the same software which is used in the maven-bundle-plugin (and some more OSGi tools) as well.

So, getting curious, I started to dig into the bnd code and decided to start a blog series about this tool.

tl;dr: If you are serious about writing OSGi applications, you should definitely know how bnd works.

There is a lot of documentation on the bnd web page, but - at least to me - it seems that you have to know a lot about OSGi and even bnd in the first place before you can really understand what the documentation talks about. This is a big impediment for beginners of OSGi and I'd like to change that by writing about my insights and experiences with bnd.

If you are interested, stay tuned.


 

Wednesday, 30 October 2013

Felix GoGo inspect command (OSGi R4.3)

I don't really know why I was not able to find a more prominent location on the web for the details of the Apache Felix inspect command in an OSGi shell, but at least I found something:

We've got the mailing list and a corresponding issue, from which I want to summarize the details in this post.

In the Felix gogo command bundle version 0.12.0 the inspect shell command was changed to the following signature:

inspect ('capability' | 'requirement') <namespace> [<bundles>]

Before OSGi R4.3 the namespace was one of  bundle, package, host and service, which now changed to one of  
  • osgi.wiring.bundle
  • osgi.wiring.package
  • osgi.wiring.host and 
  • service

"capability" and "requirement" can be abbreviated with "cap" and "req" respectively, furthermore you can use wildcards for the namespace.

Let' give some examples:

In an OSGi shell, executing a command like

  inspect cap service <bundleId>

might give you something like

The output gives you all the details about the service capabilities of the referenced bundle.

  inspect req service <bundleId>

on the contrary, lists all the service requirements:

What about the other namespaces?

Let's try

  inspect cap osgi.wiring.package <bundleId>

This is the list of all the provided packages of that bundle which comes in handy when trying to understand the package relations if a bundle doesn't want to resolve.

A last tip:

In the gogo shell you can grep the output like this

  inspect cap * 54 | grep pid

This will query all the capabilities of bundle 54 and then grep for the string "pid".

Tuesday, 29 October 2013

Debugging OSGi services

This post is a collection of tips of how to "debug" into OSGi services, grouped by the providers I am using. I will keep updating this whenever I find something new.

Apache Felix Declarative Services

Setup:

Active     |    5|Apache Felix Log Service (1.0.1)
Active     |    5|Apache Felix Declarative Services (1.6.2)
Active     |    5|Apache Felix EventAdmin (1.3.2)
Active     |    3|Apache Felix Configuration Admin Service (1.4.0)
Active     |    3|Apache Felix File Install (3.2.4)
Active     |    1|Apache Felix Gogo Command (0.12.0)
Active     |    1|Apache Felix Gogo Runtime (0.10.0)
Active     |    1|Apache Felix Gogo Shell (0.10.0)

Hint: I didn't get the scr shell commands with Declarative Services 1.6.0. Also, not all of the listed bundles might be necessary, this is just my setup.

scr:list will give you something like this:

[   1] [active       ] de.twenty11.skysail.server.config.ServerConfiguration
[   4] [active       ] de.twenty11.skysail.server.webapp.internal.WebappApplication
[   3] [active       ] de.twenty11.skysail.server.internal.Events
[   0] [active       ] de.twenty11.skysail.common.config.ConfigurationProvider
[   2] [active       ] de.twenty11.skysail.server.internal.Configuration


Now pick the id of the component you are interested in and type, for example,

scr:info 2

You will get all the details of the component, together with the information if all the requirements are satisfied (so that the component is actually active):

(...)
Reference: ApplicationProvider
    Satisfied: satisfied
    Service Name: de.twenty11.skysail.server.services.ApplicationProvider
    Multiple: multiple
    Optional: optional
    Policy: dynamic
    Policy option: reluctant
    Bound to:        [de.twenty11.skysail.server.services.ApplicationProvider]
        [de.twenty11.skysail.server.services.ApplicationProvider]
        [de.twenty11.skysail.server.services.ApplicationProvider]
        [de.twenty11.skysail.server.services.ApplicationProvider]
        [de.twenty11.skysail.server.services.ApplicationProvider]
Reference: MenuProvider
    Satisfied: satisfied
    Service Name: de.twenty11.skysail.server.services.MenuProvider
    Multiple: multiple
    Optional: optional
    Policy: dynamic
    Policy option: reluctant
    Bound to:        [de.twenty11.skysail.server.services.MenuProvider]
        [de.twenty11.skysail.server.services.MenuProvider]


help, as always, gives you the available commands:

scr:config
scr:disable
scr:enable
scr:info
scr:list

Reference: Felix Declarative Services Documentation

Sunday, 20 October 2013

converting an ordinary jar into an OSGi bundle (with maven and bnd)

Sooner or later, when dealing with OSGi, you'll find the need to utilize a third-party library which - unfortunately - was not written with OSGi in mind.

Depending on what exactly this library does, it might not be easy to convert it into an OSGi bundle. For example, if it depends a lot on Class.forName(...) constructs, it might be better to look for an alternative implementation, if that exists. Or, if that library depends heavily on further libraries and jars, you might have to convert all of those into OSGi bundles themselves, transitively... which is no fun, especially as you will not know if it will work at all until the end.

Having said that, sometimes it can be easy. A well-written library with low dependency profile, not relying on some fancy class-finding tricks can be converted into an OSGi bundle very easily.

For the following to work, I assume you are using maven.

For our example, I will pick a library I am using myself, which is not OSGi-compatible by default, stringtemplate. You can download the stringtemplate jar, and a quick look at its MANIFEST.MF files content will show you that it is not an OSGi bundle. In fact, with a manifest file not bigger that 122 bytes, you don't even need to open that file to know that you are not dealing with an OSGi bundle.

A simple pom.xml, like the one below, together with a minimal instruction file, will convert the referenced library into an OSGi bundle.

Please note, that the second file, osgi.bnd is referenced in the first one with a leading minus "-" sign.

Update: It seems like the pom setup does not work with the newest version of maven-bundle-plugin (2.4.0), you have to use 2.3.7 to make it work. I was not able yet to figure out why this happens. Thanks to Thomas Newman for the feedback!

Running "mvn clean install" now will create an OSGi bundle that you can deploy to a repository and use in your OSGi framework. Depending on how you use the library, you might have to tweak the contents of the osgi.bnd file to adjust the bundles requirements and capabilities to your context. After all, the original jar version of the library never dealt with OSGi-related concepts, so there might be some work left for you.

From my personal experience I have to add that I was using this approach quite often when starting using OSGi - I was tempted to use a lot of code which was not OSGi-aware. Over the years, for some reason, this need to utilize something from outside the OSGi world became less and less. I guess that, by now, many libraries became OSGi-aware (which is great), and, that - if you look for it - you will find OSGi-compatibly alternatives to many of your needs.

References:

  1. bnd - the tool behind the magic
  2. maven-bundle-plugin - the tool making bnd work inside maven
  3. example project on github - check out the code yourself and try it.

Thursday, 17 October 2013

skysail notes project

[originally published in skysail notes blog 29/9/2013; now moved to its parent blog]

One thing everybody seems to need is to handle notes.

What's in a note? Well, some text. And that's it - you make a note, and when you want to remember what's in that note, you search it, you find it, you read it, maybe  you act on it. Maybe you change it, and it starts all over.

Doesn't seem like a hard thing to do... I mean to create a program for.

Let's see... Unless you physically write a note and stick it to your refrigerator I mean, you will write it on your computer using some application. The question is, from where will your note be available in the future, after initial writing?

From your local machine only? The one you are sitting at right now, maybe? The one the hard drive of which will maybe fail in a couple of months? Well, maybe, why not, if the notes contents are not particularly valuable, why not. If you loose a recipe, the cell number of an old friend, maybe a license key, probably not much harm done, things can be restored or information regained somehow.

Of course, there's your fridge, your local PC or laptop, and there's "the cloud" (aka the internet).

So why not use one of the many services available to store your notes online (let's use "your notes" as a synonym for "your data" for now). Great, you get an account (easy!), you get started (easy again), you create your first notes (eas...), and, you can access it from everywhere, that's a big difference! You can even share your notes with someone if you want to! Your provider (hopefully) will care for backups, so you don't have to worry at all.

Your fridge is online, gets backuped, and you can share it, great.

But it is a potentially public fridge now. Your provider knows what's stuck on it, the NSA knows too (they seem to care), and if something goes wrong, somebody else might get access to it - or just see it, if you yourself make a mistake.

Ok, it's still the recipe, the cell number of your old friend, a couple of license keys, and, wait, some ideas about a new patent, the bank account number you don't really what your wife to know about, and that doctors number who dealt with something the insurance is not supposed to know about.

Hmm... Huston, we got a conflict.

What I am aiming at is, that people trade convenience over security. They will always, it seems. The better the offer is, the less resistance there will be.

And what about notes? After all, this here is only about notes ;)

The thing is not to pass everything to the big players, host the software yourself. Get somebody hosting it for you. Get some decent trusted open-source software and install it on your own server. It's not that hard. There are people willing to help you with that. Cryptography without back doors can be applied, it can be save. It's not even expansive. It's just a little little bit less convenient than the obvious alternatives.

Your fridge will be online, backuped, and ready to share with people you trust.

It's still your fridge, it's still your notes, - it's still your data.

It's just a little bit less convenient, sorry.

And there are some open source projects trying to achieve the things I just described, like skysail notes (the name you read in the title).

We are working on it (but already, there are solutions, you might have to search for them).


 




Sunday, 6 October 2013

launching skysail server website

Now, after my initial post yesterday, the coming out of skysail, I am ready to publish the location of the skysail.server website ;)

http://skysail.server.twentyeleven.de



Being based on OSGi, there will be quite a few "landing pages" for skysail, one for each bundle. These pages will describe the technical aspects of skysail are are meant for developers. Once I have the first products running (based on skysail) I'll have to start with end-user documentation as well, but this might take a while.

For now, for those interested, you can check out the overview pages, that is the
10.000 and 1.000 feet view, and the pages about Routing and Resources, business logic execution and SkysailResponses.


Saturday, 5 October 2013

the coming out of skysail

I've been playing with this little pet project of mine for quite a while now. I did learn a lot, mainly by making a huge number of mistakes ;)

By now, things start becoming a little bit more stable (well, we're still in pre-alpha to be honest). Nevertheless, I started with some documentation and I am ready to share my ideas, hoping for feedback.

The coming out of skysail


For starters, I want to provide a 10.000 feet view of skysail:




Looks like the typical input-process-output chain - sure, it's IT after all ;)

So, what's special about this? First of all, the skysail server deals with RESTful requests only (please note that this is not about restful web services). What seems like a restriction really adds value, as explained later.

The request will get processed (skysail will take care of authentication, authorization, logging, persistence and the like in a generic way) and return a response.

This response is a representation of the answer to that request. The specific format of the answer (that is, the response representation) depends on the request itself - asking for a JSON-formatted response will give you a JSON string. This powerful feature is build into the http protocol itself, called content negotiation.

An example

Let's assume you have a skysail server installation at myserver.org, running on port 8080, and this installation provides a RESTful "notes" application, i.e. the following paths are mapped to resources dealing with the following requests:

 /notes          -> mapped to NotesResource
 /notes/{noteId} -> mapped to NoteResource

Now, by typing http://myserver.org:8080/notes/27 in your browser, what you do is to initiate a GET request to that URL, which is dealt with by your skysail installation and mapped to the second resource, NoteResource.

NoteResource reacts to the request according to the business logic defined in the notes application. Typically, assuming you've been authenticated and have the proper permissions, the application will get the note with id 27 from its persistence store, format it according to the request, and deliver it:

{
    "message": "Note #27",
    "success": true,
    "data": [
        {
            "pid": 27,
            "parent": null,
            "title" : "myNote",
            "content" : "some content",
            "owner": {
               ...
            },
        }
    ],
}


Please note that skysail server itself is not capable of dealing with notes or any other business entities, this is the task of dedicated OSGi-bundles deployed into skysail server. Using OSGi is the only way to properly deal with all the requirements (in terms of modularization and shared services).

Summary

As described, skysail is a RESTful "business server" focusing on the business logic. This is, IMHO, a reasonable abstraction, as it really separates the representation of entities (the GUI) from the logic performed in the back-end.

Like this, it is very easy to develop various clients (think about a web application, a rich client and multiple mobile apps), all communicating with the same business server. Those clients could be written in any language, too. The only thing they'd have to suport is the HTTP protocol (and understand json).

So, the restriction on dealing with RESTful requests only gives you a chance to really decouple clients and server and provide the most generic interface, using the standard HTTP methods like GET, POST, PUT and DELETE.