Vert.x for World Domination

---

In this article I am going to show how I put Vert.x on my Raspberry Pis and my Mindstorm EV3.

The whole thing is the product of a long night of hacking so expect some updates to this article.

The associated project is on GitHub.

---

The core of my presentation Vert.x for World Domination was the setup I used to demonstrate the distribution capabilities of Vert.x.

What I wanted to do was a demo using “real life” conditions. No VMs or other tricks. I got the basic idea from the setup Michael uses to demo Hazelcast (see some photos here)

The idea of a portable Raspberry Pi cluster stuck with me and I built my somewhat smaller version.

The Brain

And here it comes, the portable brain:

 

Parts list

• 2 x Raspberry Pi Model B (inside the red lego-thing)
• 2 x Loglink WL0084A Wifi-adapters
• TP-Link TL-WR702N (the small white/blue box on the left, a very compact Wifi-router)
• EasyAcc 5A USB power adapter (the white box on the right, make sure to user the iPad-connectors to power the Pis)

Setup

Connect the Pis to the Wifi.

Slap Raspbian on the Pis, install the embedded JDK and put Vert.x on them.

The brain is good to go.

Robby

My fearsome battle robot Robby is up next.

It’s a custom EV3-bot. Go crazy on how you want it to look like but follow my instructions on how to get Vert.x up and running. That part took me a while.

 

 

 

Parts list

• The EV3 base package
• TP-Link TL-WN725N USB-Wifi (Michael reported some issues on his EV3, works fine on mine)
• A micro SD card

LejOS

With EV3 Lego provided everything to make it as easy as possible to get a custom OS on your mindstorm. The LejOS project is one of those. It is aimed at Java developers and provides everything needed to get going.

Simply follow the instructions from their Wiki to get everything set up. It’s that easy.

After connecting it to Wifi you should see something like this on the display:

Vert.x

Here comes the tricky part. First of all: You will have to use a fat jar as LejOS doesn’t provide a Bash and renders the provided vert.x-startup-skripts useless.Copy the generated jar to your EV3 (ahem, the root-password is “”, so just hit enter when asked):

scp <fatjar>.jar root@<ip-of-ev3>:~

Copy the cluster.xml found in your local Vert.x-installation:

scp <vertx-dir>/conf/cluster.xml root@<ip-of-ev3>:~

Afterwards log into the EV3.

There’s two issues we will be solving:

1. Vert.x won’t bind to the Wifi-address using the -cluster-host parameter
2. Multicast is not working

Get a Vert.x instance running on your local machine, note down the IP-address and log into your EV3:

ssh root@<ip-of-ev3>

Now open the cluster.xml for editing (vi is installed and my weapon of choice).
Disable multicast and enable tcp-ip. Add the address of your local machine (192.168.2.20 in the example).
Uncomment the interfaces block and add the address of your EV3-Wifi (the one you used to log into it)

Afterwards the config should look something like this:

…
<join>
<multicast enabled=”false”>
<multicast-group>224.2.2.3</multicast-group>
<multicast-port>54327</multicast-port>
</multicast>
<tcp-ip enabled=”true”>
<interface>192.168.2.20</interface>
</tcp-ip>
<aws enabled=”false”>
<access-key>my-access-key</access-key>
<secret-key>my-secret-key</secret-key>
<region>us-east-1</region>
</aws>
</join>
<interfaces enabled=”true”>
<interface>192.168.2.101</interface>
</interfaces>
…

Now we need to modify the start skript. Add a new file called run.sh with the following content:

#! /bin/sh
. /etc/default/lejos
export LD_LIBRARY_PATH=${LEJOS_HOME}/libjna/usr/lib/arm-linux-gnueabi/:${LEJOS_HOME}/libjna/usr/lib/jni/
/home/root/lejos/ejre1.7.0_51/bin/java -jar <nameof-fat>.jar -cp . -cluster -cluster-host 192.168.2.101


Replace the IP-Adress with the one of your EV3-Wifi (the one you used to log into it) and replace <nameof-fat> with the name of your fat-jar.

Do a chmod +x run.sh

You are ready to go.

Use ./run.sh to start Vert.x and wait. It takes around 3 minutes for everything to come up so be patient.

This is the log-output I get when running everything ( => "Succeeded in deploying module"): 


root@EV3:~# ./run.sh
Apr 07, 2014 4:53:29 PM org.vertx.java.core.logging.impl.JULLogDelegate info
INFO: Starting clustering...
Apr 07, 2014 4:54:11 PM com.hazelcast.impl.AddressPicker
INFO: Interfaces is enabled, trying to pick one address matching to one of: [192.168.2.101]
Apr 07, 2014 4:54:12 PM com.hazelcast.impl.AddressPicker
INFO: Prefer IPv4 stack is true.
Apr 07, 2014 4:54:13 PM com.hazelcast.impl.AddressPicker
INFO: Picked Address[192.168.2.101]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Apr 07, 2014 4:54:35 PM com.hazelcast.system
INFO: [192.168.2.101]:5701 [dev] Hazelcast Community Edition 2.6.7 (20140210) starting at Address[192.168.2.101]:5701
Apr 07, 2014 4:54:35 PM com.hazelcast.system
INFO: [192.168.2.101]:5701 [dev] Copyright (C) 2008-2013 Hazelcast.com
Apr 07, 2014 4:54:36 PM com.hazelcast.impl.LifecycleServiceImpl
INFO: [192.168.2.101]:5701 [dev] Address[192.168.2.101]:5701 is STARTING
Apr 07, 2014 4:54:38 PM com.hazelcast.impl.TcpIpJoiner
INFO: [192.168.2.101]:5701 [dev] Connecting to possible member: Address[192.168.2.20]:5701
Apr 07, 2014 4:54:38 PM com.hazelcast.impl.TcpIpJoiner
INFO: [192.168.2.101]:5701 [dev] Connecting to possible member: Address[192.168.2.20]:5702
Apr 07, 2014 4:54:39 PM com.hazelcast.impl.TcpIpJoiner
INFO: [192.168.2.101]:5701 [dev] Connecting to possible member: Address[192.168.2.20]:5703
Apr 07, 2014 4:54:40 PM com.hazelcast.nio.ConnectionManager
INFO: [192.168.2.101]:5701 [dev] 59397 accepted socket connection from /192.168.2.20:5701
Apr 07, 2014 4:54:40 PM com.hazelcast.nio.ConnectionManager
INFO: [192.168.2.101]:5701 [dev] 44524 accepted socket connection from /192.168.2.20:5702
Apr 07, 2014 4:54:40 PM com.hazelcast.nio.ConnectionManager
INFO: [192.168.2.101]:5701 [dev] 33806 accepted socket connection from /192.168.2.20:5703
Apr 07, 2014 4:54:44 PM com.hazelcast.cluster.ClusterManager
INFO: [192.168.2.101]:5701 [dev]

Members [4] {
Member [192.168.2.20]:5701
Member [192.168.2.20]:5702
Member [192.168.2.20]:5703
Member [192.168.2.101]:5701 this
}

Apr 07, 2014 4:55:20 PM com.hazelcast.impl.LifecycleServiceImpl
INFO: [192.168.2.101]:5701 [dev] Address[192.168.2.101]:5701 is STARTED
Apr 07, 2014 4:55:36 PM com.hazelcast.util.HealthMonitor
INFO: [192.168.2.101]:5701 [dev] memory.used=5.6M, memory.free=3.3M, memory.total=8.9M, memory.max=29.0M, memory.used/total=63.02% memory.used/max=19.29% load.process=92.00%, load.system=100.00%, load.systemAverage=250.00% q.packet.size=2, q.processable.size=1, q.processablePriority.size=0, thread.count=19, thread.peakCount=19, q.query.size=0, q.mapLoader.size=0, q.defaultExecutor.size=0, q.asyncExecutor.size=0, q.eventExecutor.size=0, q.mapStoreExecutor.size=0
Apr 07, 2014 4:56:06 PM com.hazelcast.util.HealthMonitor
INFO: [192.168.2.101]:5701 [dev] memory.used=8.0M, memory.free=4.2M, memory.total=12.2M, memory.max=29.0M, memory.used/total=65.18% memory.used/max=27.42% load.process=93.00%, load.system=100.00%, load.systemAverage=281.00% q.packet.size=1, q.processable.size=1, q.processablePriority.size=0, thread.count=21, thread.peakCount=21, q.query.size=0, q.mapLoader.size=0, q.defaultExecutor.size=0, q.asyncExecutor.size=0, q.eventExecutor.size=0, q.mapStoreExecutor.size=0
Apr 07, 2014 4:56:37 PM com.hazelcast.util.HealthMonitor
INFO: [192.168.2.101]:5701 [dev] memory.used=9.9M, memory.free=2.3M, memory.total=12.2M, memory.max=29.0M, memory.used/total=81.41% memory.used/max=34.25% load.process=94.00%, load.system=100.00%, load.systemAverage=288.00% q.packet.size=0, q.processable.size=1, q.processablePriority.size=1, thread.count=21, thread.peakCount=21, q.query.size=0, q.mapLoader.size=0, q.defaultExecutor.size=0, q.asyncExecutor.size=0, q.eventExecutor.size=0, q.mapStoreExecutor.size=0
Apr 07, 2014 4:57:04 PM org.vertx.java.core.logging.impl.JULLogDelegate info
INFO: Succeeded in deploying module

 

Advertisement

MongoDB Media Server

Code available on GitHub

The included code is a show case, not a full media server! This is also a somewhat older project, if I missed something feel free to add an issue on GitHub.

So I’ve been going through playground folder on my hard disk. That’s the place I store all my projects I did for showcasing or simply to play around with new things.
And I found a little something I did about a year ago for the Senacor DevCon. It’s a small media server project based on Spring Data and MongoDB.

I am releasing it because I think it’s a great example for the simplicity of Spring Data combined with SpringMVC.
NOTE: There’s a a python script included to import your MP3s into the database. You will need Mutagen on your machine to use it.

All the interesting things happen in the LibraryRessource. Let’s take a look at the most important parts.
The things we nee to work with are the SongRepository (a simple Spring Data repository), MongoTemplate (here be CRUD) and the GridFsTemplate, which is required for documents >16 MB (see GridFS).

@Controller
public class LibraryRessource {

    @Autowired
    private MongoTemplate mongoTemplate;

    @Autowired
    private GridFsTemplate gridFsTemplate;

    @Autowired
    private SongRepository songRepository;

Most of the @RequestMapping-annotated classes should be self explaining so I will foxus on the “tricky” ones.
getSongs(..)_ is responsible for assembling the Song-JSON documents which contain a link to the GridFS-file to start streaming. To be able to assemble I have to git hold of the MD5-checksum to reference it.

    @RequestMapping(value = "/library/songs", method = RequestMethod.GET, produces = "application/json")
    public @ResponseBody Songs getSongs(HttpServletRequest request) {
        Songs songs = new Songs();
        for (Song song: songRepository.findAll()) {
            GridFSDBFile file = gridFsTemplate.findOne(query(where("_id").is(song.getFileId())));
            String requestBase = request.getRequestURL().toString().split("song")[0];
            song.setFileHref(requestBase + "files/" + file.getMD5());
            songs.addSong(song);
        }
        return songs;
    }

The actual streaming is done in getFile(..). Thanks to the fluent-API of GridFsTemplate it turned out that the trickiest part of my application could be done in a single line of code. The only thing to be aware of is that getting hold of the HttpServletResponse used by your request is by adding it to the method signature.

    @RequestMapping(value = "/library/files/{md5}", method = RequestMethod.GET)
    public void getFile(HttpServletResponse response, @PathVariable("md5") String md5) throws IOException {
        gridFsTemplate.findOne(query(where("md5").is(md5))).writeTo(response.getOutputStream());
    }

The generated URLs can be accessed using VLC.

Using Spring Security 3 with Wicket 6-AuthRoles and JavaConfig and a little Servlet 3

Spring Security is a great framework. It unifies a lot of different authentication mechanisms into a pretty decent package. The only part that wasn’t really up to current Spring-standards was the lack of full java config support. In July they finally released the first fully java config enabled version.

In this blog post I’d like to show how to integrate this new version with Wicket 6-AuthRoles.

The project

I put all the code up on GitHub, clone it and you are ready to go. It shouldn’t be a big surprise that you have to use a Servlet 3 container. I tested the application on Tomcat 7 and Jetty 9, worked without a hitch.

How Spring and Servlet 3 Interact

Servlet 3 provides one of the most fundamental changes. It made web.xml optional. To achieve this the container will scan each war for an implementation of javax.servlet.ServletContainerInitializer to get everything going. Spring provides such a implementation with the SpringServletContainerInitializer. When it is discovered by the container it will then scan the class path for implementations of WebApplicationInitializer. And that’s where we do our stuff.

Servlet 3 Bootstrap

The class AppInitializer contains the whole code required to bootstrap the application. The only really interesting part is how to actually register the Wicket-filter. That cost me some time to figure out but the following snipped does the trick:

        WicketFilter wicketFilter = new WicketFilter(new WicketApplication()) {
            @Override
            public void init(boolean isServlet, FilterConfig filterConfig) throws ServletException {
                setFilterPath("");
                super.init(isServlet, filterConfig);
            }
        };
        FilterRegistration.Dynamic wicketFilterReg = servletContext.addFilter("wicketFilter", wicketFilter);

Auth Roles

Wicket-AuthRoles provide a complete set of annotations to cover all scenarios of authorization. In my demo application I only use @AuthorizeInstantiation.

One important thing to note is that Spring Security prepends ROLE_ to all role names. So instead of writing @AuthorizeInstantiation(Roles.USER) you will have to use @AuthorizeInstantiation(“ROLE_USER”).

Getting AuthRoles to work you need to provide a custom WebSession-implementation:

public class UserAuthenticatedWebSession extends AuthenticatedWebSession {    
    public UserAuthenticatedWebSession(Request request) {
        super(request);
    }

    @Override
    public boolean authenticate(String username, String password) {
        throw new UnsupportedOperationException("You are supposed to use Spring-Security!!");
    }

    @Override
    public Roles getRoles() {
        Roles roles = new Roles();
        Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
        for (GrantedAuthority authority : authentication.getAuthorities()) {
            roles.add(authority.getAuthority());
        }
        return roles;
    }

}

The authenticate method is not required as Spring Security is going to take care of that. The only thing we have to do in here is to provide a getRoles-implementation which uses the SecurityContextHolder

Now we have to change the WebApplication to extend AuthenticatedWebApplication and override init, getWebSessionClass and getSignInPageClass:

@Override
public void init() {
    super.init();
    getComponentInstantiationListeners().add(new SpringComponentInjector(this));
    getSecuritySettings().setAuthorizationStrategy(new AnnotationsRoleAuthorizationStrategy(this));
    mountPage("/home", HomePage.class);
    mountPage("/login", SignInPage.class);
}

We also have to specify the /login-mountpoint this should point to a Wicket-page with the following content:

<form method="POST">
    <input type="text" id="username" name="username"/>
    <input type="password" id="password" name="password"/>
    <input type="submit" value="submit"/>
</form>

The form-submit will NOT be handled by Wicket but by the Spring Security filter!

Spring Security

The final piece of the puzzle is the Spring Security configuration.

In AppInitializer I added the following code to bootstrap the Spring context and the security filter:

    AnnotationConfigWebApplicationContext root = new AnnotationConfigWebApplicationContext();
    root.register(SpringSecurityConfiguration.class);
    FilterRegistration.Dynamic springSecurityFilterChainReg = servletContext.addFilter("springSecurityFilterChain", DelegatingFilterProxy.class);
    springSecurityFilterChainReg.addMappingForUrlPatterns(EnumSet.of(DispatcherType.ERROR, DispatcherType.REQUEST), false, "/*");
    servletContext.addListener(new ContextLoaderListener(root));

The actual configuration happens in SpringSecurityConfiguration:

@Configuration
@EnableWebSecurity
public class SpringSecurityConfiguration extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.authorizeUrls()
                .antMatchers("/favicon.ico")
                    .permitAll()
                .antMatchers("/**")
                    .hasRole("USER")
                    .and()
                    .formLogin()
                        .loginPage("/login")
                        .permitAll();
    }

    @Override
    protected void registerAuthentication(AuthenticationManagerBuilder auth) throws Exception {
        auth
            .inMemoryAuthentication()
                .withUser("testuser")
                    .password("password")
                    .roles("USER");
    }
}

I use configure to tell spring security to allow unrestricted access to favicon.ico. Then I tell it ti restrict access to all other URLs to users with the role USER (reminder: this translates to ROLE_USER when used in AuthRoles-annotations). The last step is to add a custom formLogin to use the mount point from before.

In registerAuthentication I add a simple in-memory-authentication. Here you can now add all the other fancy stuff.

Get it while it’s hot: https://github.com/codepitbull/wicket-servlet3-springsecurity

Have fun.

Vert.X 2.0.0 in IntelliJ IDEA

Well, that was fast. After posting my last post I saw that vert.x 2.0.0.final came out last night. Tough luck but sitting on the train I got time enough to provide an update.

Project setup

Follow these to create a basic gradle-project with an example verticle. After doing a gradle clean build idea you can now import the project into Idea.

Running

The examples provided only show how to run this project from the command line. I am a lazy ass and like to run my stuff from the IDE.

To achieve this add a new java-module using File>New Module. Open Open Module Settings, select the newly created module and add contents of the lib-directory from your freshly downloaded vert.x-2.0.0-final-archive in the Dependencies-tab.

Create a new Run-Config and:

• set Main Class to org.vertx.java.platform.impl.cli.Starter
• set Program arguments to runmod com.yourcompany~your-module~1.0 -cp <path-to-your-project>/out/production/<module-name>
• set Use classpath of module to the module we just created and added the vert.x-libs to

Hit run.

Enjoy.

Vert.X in IntelliJ IDEA

Well, that was fast. Outdated within minutes of posting 🙂
This post is for vert.x 1.3.1. 

I just wrote an update on how to do this with vert.x 2.0.0.final

---

I took a look at Disruptor back when it was released. It was an interesting piece of software but I never got beyond playing around. Then came Node.js. I have been fighting with myself for quite a while to use it but I am kind of biased when it comes to JavaScript… Well, then came vert.x and I finally had no excuse left to get into this single-threade-thingy-stuff. Playing around with vert.x was quite a fun experience I will write about later. Today I want to show you how to get it running in my favorite IDE.

The Problem

Being polyglot apparently also means to abandon tested deployment strategies. So instead of dumping a JAR/WAR or whatever else into the vert.x-container you will have to do some special magic. And this special magic also involves some tinkering with Idea.

Basic Project

The following (very basic) build.gradle gives you a simple vert.x project.

apply plugin:'java'

configurations {
    provided
    provided.extendsFrom(compile)
}

repositories {
    mavenCentral()
    mavenLocal()
    mavenRepo url: "https://repository.apache.org/content/repositories/snapshots/"
    mavenRepo url: "http://source.mysema.com/maven2/releases/"
    mavenRepo url: "http://repo.maven.apache.org/maven2"
}

dependencies{
        compile "org.vert-x:vertx-core:1.3.1.final"
    compile "org.vert-x:vertx-lang-java:1.3.1.final"
    compile "org.vert-x:vertx-platform:1.3.1.final"
}

Import it and add a vertical like the following:

package de.codepitbull.vertx;

import org.vertx.java.core.Handler;
import org.vertx.java.core.http.HttpServerRequest;
import org.vertx.java.deploy.Verticle;

import java.util.Map;

/**
* @author Jochen Mader
*/
public class HttpVerticle extends Verticle{
@Override
public void start() throws Exception {
vertx.createHttpServer().requestHandler(new Handler() {
public void handle(HttpServerRequest req) {
StringBuilder sb = new StringBuilder();
for (Map.Entry<String, String> header : req.headers().entrySet()) {
sb.append(header.getKey()).append(": ").append(header.getValue()).append("\n");
}
req.response.putHeader("content-type", "text/plain");
req.response.end(sb.toString());
}
}).listen(8087);
}
}

And, due to a bug in 1.3.1, you also need to add a langs.properties to your project with the following content:

java=org.vertx.java.deploy.impl.java.JavaVerticleFactory
class=org.vertx.java.deploy.impl.java.JavaVerticleFactory
js=org.vertx.java.deploy.impl.rhino.RhinoVerticleFactory
coffee=org.vertx.java.deploy.impl.rhino.RhinoVerticleFactory
rb=org.vertx.java.deploy.impl.jruby.JRubyVerticleFactory
groovy=org.vertx.groovy.deploy.impl.groovy.GroovyVerticleFactory
py=org.vertx.java.deploy.impl.jython.JythonVerticleFactory
default=org.vertx.java.deploy.impl.java.JavaVerticleFactory

Setup

To get this whole thing running you will need to download the vert.x tar and untag it to your file system. Next select File>New Module and create a new Java-Module using the wizard. After you are done open Open Module Settings and select your newly created module. Go to the Dependencies-tab and add the content of your vert.x-installations lib-directory. Next add the folder containing langs.properties. Almost there.

Start Config

Now that we got our little dummy-project up and going we need to create a start config. So create a new Application-Startconfig. – Set the Main-class to org.vertx.java.deploy.impl.cli.Starter – Set Program arguments to run de.codepitbull.vertx.HttpVerticle -cp where the path is the place where Idea puts your compiled classes from the gradle-project. – Set Use class path of module to the module we just configured. That’s it.

Hit run and go to home sweet home

Serving static content with Jetty 9

I am currently (again) doing a lot of JavaScript related stuff. My favorite IDE (IntelliJ IDEA) has stellar support for this crappy language. The only thing I needed was a SMALL web server to play around with static html and some JavaScripts. The quickest way was to use Jetty 9 as it can be found on all of my development machines.

Just add a Jetty9-RunConfig in IDEA (or Eclipse, if you have to) and create static.xml with the following content in $JETTY_HOME/webapps.

<?xml version="1.0"  encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.eclipse.org/configure.dtd">

<Configure class="org.eclipse.jetty.server.handler.ContextHandler">
  <Set name="contextPath">/static</Set>
  <Set name="resourceBase">/path/to/your/document/root/</Set>
  <Set name="handler">
    <New class="org.eclipse.jetty.server.handler.ResourceHandler">
      <Set name="cacheControl">no-cache</Set>
    </New>
  </Set>
</Configure>

Don’t forget to adjust resourceBase to your liking.

Note: I also disabled caching in this example.

Bootstrapping Neo4j With Spring-Data – Without XML

With the maturing of Spring-Data I started porting all my personal projects to use Spring Data for bootstrapping. I also wanted to get rid of XML, which proved a little more tricky as I expected.

Dependencies

Let’s start with the required dependencies:

Update: Added a missing validator-dependency

 <properties>
 <spring-core.version>3.2.0.RELEASE</spring-core.version>
 </properties>
 <dependencies>
 <dependency>
 <groupId>org.springframework.data</groupId>
 <artifactId>spring-data-neo4j</artifactId>
 <version>2.2.1.RELEASE</version>
 <exclusions>
 <exclusion>
 <groupId>org.springframework</groupId>
 <artifactId>spring-asm</artifactId>
 </exclusion>
 </exclusions>
 </dependency>
 <dependency>
 <groupId>org.neo4j.app</groupId>
 <artifactId>neo4j-server</artifactId>
 <version>1.8.1</version>
 </dependency>
 <dependency>
 <groupId>org.neo4j.app</groupId>
 <artifactId>neo4j-server</artifactId>
 <classifier>static-web</classifier>
 <version>1.8.1</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-core</artifactId>
 <version>${spring-core.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-context</artifactId>
 <version>${spring-core.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-aop</artifactId>
 <version>${spring-core.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-aspects</artifactId>
 <version>${spring-core.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-beans</artifactId>
 <version>${spring-core.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-expression</artifactId>
 <version>${spring-core.version}</version>
 </dependency>
 <dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-tx</artifactId>
 <version>${spring-core.version}</version>
 </dependency>
 <dependency>
 <groupId>cglib</groupId>
 <artifactId>cglib</artifactId>
 <version>2.2.2</version>
 </dependency>
 </dependencies>
 <dependency>
 <groupId>org.hibernate</groupId>
 <artifactId>hibernate-validator</artifactId>
 <version>4.3.0.Final</version>
 </dependency>

We need/want to override the spring dependencies pulled in via Spring Data with a newer one. We also need to explicitly remove spring-asm as newer versions of Spring don’t need it.

The neo4j-dependencies are required to give us the Neo4j-WebServer.

Repositories

I created a small demo-Entity based on the Neo4j-Annotation provided by Spring-Data.

 @NodeEntity
 public class Component {
    @GraphId Long id;
    @Indexed(unique = true) String simpleName;
    @RelatedTo(type = “Component”) Set<Component> relatedTo = new HashSet<Component>();
 …
 …
 }

Next I added a repository to handle the class (oh, the beauty of Spring-Data-Repositories).

public interface ComponentRepository extends GraphRepository<Component>{
}

That’s it.

Configuration

To get it all up and running the only thing required is @Configuration-Annotated class which extends Neo4jConfiguration.

@EnableTransactionManagement
@Configuration
@EnableNeo4jRepositories(basePackages = “de.codepitbull.neo4j”)
public class CustomNeo4jConfig extends Neo4jConfiguration {
    private static final String DB_PATH = "target/neo4j";
    @Bean
    public EmbeddedGraphDatabase graphDatabaseService() {
        return new EmbeddedGraphDatabase(DB_PATH);
    }
    @Bean
    public WrappingNeoServerBootstrapper neo4jWebServer() {
        WrappingNeoServerBootstrapper server = new WrappingNeoServerBootstrapper(graphDatabaseService());
        server.start();
        return server;
    }
}

Using this config we are getting not only an embedded Neo4J instance but also a nice query-interface (including the Neo4J-Shell) running at localhost:7474.
Nice, that’s it 🙂

Create a main-Class, fire up the context and see the magic happen.

public class Main {
     public static void main(String[] args) {
         AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(CustomNeo4jConfig.class);
         context.start();
         context.getBean(WrappingNeoServerBootstrapper.class);
         context.registerShutdownHook();
     }
}

Using Converters

After playing around with Neo4J you will run into the need for using converters (everything that goes into the graph has to be converted to a String). This proved to be a little more tricky than I had expected.

Normally you only have to register a new ConversionServiceFactoryBean to provide additional back and forth conversions. Registering this been in the same @Configuration we just created will be followed by a “circular dependency”-problem.

I am not sure why this is happening but I simply moved it into a separate @Configuration _ which uses @Import_ to include the Neo4J-Configuration and things started working.

Obviously you will also have to replace CustomNeo4jConfig.class in your Main-class with MainConfiguration.class.

@Configuration
@Import(CustomNeo4jConfig.class)
public class MainConfiguration {

@Bean
public ConversionServiceFactoryBean conversionService() {
    Set converters = Sets.newHashSet();

    converters.add(new ConstructorListConverter());

    ConversionServiceFactoryBean bean = new ConversionServiceFactoryBean();
    bean.setConverters(converters);
    return bean;
}
}

Conclusion

I love it, nothing more to say.

Wicket: A slightly better “Open Session in View”

We recently participated at the The Plat_forms contest (more on that in a few days). Coding a full application in less than 30 hours is quite the task and there’s no room for wasting time.

Sadly, we wasted time. A lot. On the persistence layer.

Wasted may be too hard of a word but we spent too much time building the actual business services needed to populate the view layer.

You might wonder why it took so much time.

Well, because each use case needs a different service method which provides an entity and all it’s relations initialized as deep as the view layer will use it.

Now you might give me a confused look and ask “Why the hell did you bother building those and didn’t use Open Session in View?” and I would have answered with tirade on why I consider it an anti pattern.

But sometimes you have to rethink opinions you held dear for a long time.

Sometimes verbose is better

Before I continue to dive into Open Session in View (OSiV) a little deeper, let’s take a look why I still prefer the verbose approach. Building a relational persistence layer using an ORM can be a tricky thing.

There’s a lot of things that can go wrong. Especially if JOINs are involved. Any access to a collection or a referenced entity can cause havoc on your application performance.

By avoiding Open Session in View this problem is easily avoided as the developer has to think about every single JOIN as he will have to build the queries to resolve them.

There will also be integration tests covering individual queries and the possibility to do load tests based on those.

The moment you start using OSiV all these advantages disappear.

Sometimes terse is better

A contest (or building a prototype) has different rules. The only thing that counts is “getting it done”.

That’s where the verbose approach starts to fail as it demands a lot of code being written and tested.

Enter Open Session in View

Open Session in View

Using OSiV the database session is opened and closed through a Servlet Filter.

As the session stays open in the view layer you are free to navigate the entity-tree as you please.

This convenience comes at a price and, asides the problems I already mentioned, there are two key disadvantages:

N+1 Select problem

The most dreaded problem is the N+1-Select-Problem. Accessing collections without appropriate annotations will cause the ORM to do a Select for each entry in the collection.

So you will get 1 initial select plus N (=size of collection) subsequent selects. Using todays query-statistics-tools (every good ORM has a set of these) they are easy to find if they occur.

They are easily introduced and hard to find during development. With a good project setup they should be discovered during load testing.

If you are actually doing load tests …

Exception in the filter

ORMs do a lot of things when a database session is closed. Doing a lot of things also means that a lot of things can go wrong.

With OSiV the session is closed after the web request has been processed by the web framework. There is no way you can react on these exceptions in a meaningful way without putting significant logic into the ServletFilter. Not really a good idea.

Wicket

There is no way solving N+1 in a generic way. But the exception problem is easily solved. Well, if Wicket is your view layer. Wicket provides hooks to every step of request processing through

IRequestCycleListener. The following OpenSessionInRequestCycleListener shows how to build a Hibernate based OSiV for Wicket.

import org.apache.wicket.MetaDataKey;

import org.apache.wicket.request.IRequestHandler;

import org.apache.wicket.request.cycle.AbstractRequestCycleListener;

import org.apache.wicket.request.cycle.RequestCycle;

import org.hibernate.FlushMode;

import org.hibernate.HibernateException;

import org.hibernate.Session;

import org.hibernate.SessionFactory;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

import org.springframework.beans.factory.annotation.Configurable;

import org.springframework.dao.DataAccessResourceFailureException;

import org.springframework.orm.hibernate4.SessionFactoryUtils;

import org.springframework.orm.jpa.EntityManagerFactoryUtils;

import org.springframework.orm.jpa.EntityManagerHolder;

import org.springframework.transaction.support.TransactionSynchronizationManager;

import javax.persistence.EntityManager;

import javax.persistence.EntityManagerFactory;

import javax.persistence.PersistenceException;

import javax.persistence.PersistenceUnit;

@Configurable

public final class OpenEntityManagerInRequestCycleListener extends

        AbstractRequestCycleListener {

    private static final Logger LOG = LoggerFactory.getLogger(OpenEntityManagerInRequestCycleListener.class);

    @SuppressWarnings("serial")

    static final MetaDataKey<Boolean> PARTICIPATE = new MetaDataKey<Boolean>() {};

    @PersistenceUnit

    private EntityManagerFactory emf;

    @Override

    public void onBeginRequest(RequestCycle cycle) {

        cycle.setMetaData(PARTICIPATE, TransactionSynchronizationManager.hasResource(emf));

        if(!cycle.getMetaData(PARTICIPATE)) {

            try {

                LOG.debug("OPENING NEW ENTITY MANAGER FOR THIS REQUEST.");

                EntityManager em = emf.createEntityManager();

                TransactionSynchronizationManager.bindResource(emf, new EntityManagerHolder(em));

            }

            catch (PersistenceException ex) {

                throw new DataAccessResourceFailureException("Could not create JPA EntityManager", ex);

            }

        }       

    }

    @Override

    public void onRequestHandlerExecuted(RequestCycle cycle, IRequestHandler handler) {

        if (!cycle.getMetaData(PARTICIPATE)) {

            try {

                LOG.debug("CLOSING ENTITY MANAGER FOR THIS REQUEST.");

                EntityManagerHolder emHolder = (EntityManagerHolder)

                        TransactionSynchronizationManager.unbindResource(emf);

                EntityManagerFactoryUtils.closeEntityManager(emHolder.getEntityManager());

            }

            catch (WhateverExceptionYouAreInterstedIn e) {

                //DOSTUFF

            }

        }

    }

    @Override

    public IRequestHandler onException(RequestCycle cycle, Exception ex) {

        return super.onException(cycle, ex);    //To change body of overridden methods use File | Settings | File Templates.

    }

    protected Session openSession(SessionFactory sessionFactory) throws DataAccessResourceFailureException {

        try {

            Session session = SessionFactoryUtils.openSession(sessionFactory);

            session.setFlushMode(FlushMode.MANUAL);

            return session;

        }

        catch (HibernateException ex) {

            throw new DataAccessResourceFailureException("Could not open Hibernate Session", ex);

        }

    }

}

And the same using JPA2:

import org.apache.wicket.MetaDataKey;

import org.apache.wicket.request.IRequestHandler;

import org.apache.wicket.request.cycle.AbstractRequestCycleListener;

import org.apache.wicket.request.cycle.RequestCycle;

import org.hibernate.FlushMode;

import org.hibernate.HibernateException;

import org.hibernate.Session;

import org.hibernate.SessionFactory;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.beans.factory.annotation.Configurable;

import org.springframework.dao.DataAccessResourceFailureException;

import org.springframework.orm.hibernate4.SessionFactoryUtils;

import org.springframework.orm.hibernate4.SessionHolder;

import org.springframework.transaction.support.TransactionSynchronizationManager;

@Configurable

final class OpenSessionInRequestCycleListener extends

        AbstractRequestCycleListener {

    @SuppressWarnings("serial")

    static final MetaDataKey<Boolean> PARTICIPATE = new MetaDataKey<Boolean>() {};

    @Autowired

    private SessionFactory sessionFactory;

    @Override

    public void onBeginRequest(RequestCycle cycle) {

        cycle.setMetaData(PARTICIPATE, TransactionSynchronizationManager.hasResource(sessionFactory));

        if(!cycle.getMetaData(PARTICIPATE)) {

            Session session = openSession(sessionFactory);

            TransactionSynchronizationManager.bindResource(sessionFactory, new SessionHolder(session));

        }       

    }

    @Override

    public void onRequestHandlerExecuted(RequestCycle cycle, IRequestHandler handler) {

        if (!cycle.getMetaData(PARTICIPATE)) {

            try {

                SessionHolder sessionHolder =

                        (SessionHolder) TransactionSynchronizationManager.unbindResource(sessionFactory);

                SessionFactoryUtils.closeSession(sessionHolder.getSession());

            }

            catch (WhateverExceptionYouAreInterstedIn e) {

                //DOSTUFF

            }

        }

    }

    protected Session openSession(SessionFactory sessionFactory) throws DataAccessResourceFailureException {

        try {

            Session session = SessionFactoryUtils.openSession(sessionFactory);

            session.setFlushMode(FlushMode.MANUAL);

            return session;

        }

        catch (HibernateException ex) {

            throw new DataAccessResourceFailureException("Could not open Hibernate Session", ex);

        }

    }

}

Both use the same mechanism. Overwrite onRequestHandlerExecuted and go wild. Throw a RestartResponseException or recover gracefully, it’s up to you.

End

There’s only one question left: Would I use it?

Well, to be honest: It depends.

If I ever had to do something like Plat_forms again or a project prototype I would definitely go for it.

For a mission critical application I still prefer to go the save route and know each JOIN by its first name.

Back on the Blog, again

As you might have noticed: It’s been pretty quiet around my blog.

I took an offer to write a book about my favorite web framework, Apache Wicket.

Together with a bunch of articles I did for both it-republik (Wicket 1.5 Wicket 6 and the German Java Magazin_ (1, 2, 3, 4) I basically spent all my time writing about and coding Wicket. A few weeks ago I finally hit the last milestone of this year by attending the Plat_forms-Contest in Berlin, an experience I am going to write about soon.

Let’s keep it short:

• I am again back on the blog.
• I got a ton of notes I need to turn into posts.

Time to get going.

P.S.: I will do a few more articles in German. Looks like there is no good way to provide an article in two different languages on wordpress. This might lead to a few more changes to this blog.

Wicket und CSRF (deutsch)

Nach einer Diskussion die ich vor Kurzem zum Thema CSRF (Cross Site Request Forgery) hatte möchte ich kurz auf die Möglichkeiten zum Umgang mit diesem Angriff in Wicket eingehen. Tatsächlich handelt es sich hier um einen oft unterschätzten Angriffsvektor der eine Schwäche des Browsers und nicht der eigentlichen Webanwendung ausnutzt.

Der Angriff

Der Ablauf eines CSRF-Angriffs ist schnell erklärt.

1. Ein Benutzer authentifiziert sich in einer Webanwendung
2. Eine entsprechende Sitzung wird eröffnet und der Browser erhält ein Cookie
3. Der Benutzer navigiert, ohne sich abzumelden, auf eine andere Seite (oder öffnet ein weiteres Tab/Browser-Window).
4. Auf dieser Seite hat unser Angreifer ein entsprechendes JavaScript oder eine verstecktes Frame eingebaut mit dessen Hilfe der Browser dazu veranlasst wird eine URL innerhalb der anderen Webanwendung zu öffnen.
5. Da das Cookie immer noch gültig ist wird dies automatisch mitgeschickt und der Angreifer hat Zugriff.

Natürlich versuchen Browserhersteller Möglichkeiten für diesen Angriff zu minimieren.

Grundsätzlich sollte man als Webentwickler aber davon ausgehen, dass diese Mechanismen nicht funktionieren.

So manchem (Jemand fühlt sich gleich angesprochen 😉 ) zuerst die Möglichkeit via AJAX und XmlHTTPRequest in den Sinn. Allerdings gibt es hier eine wichtige Beschränkung die meines Wissens in allen Browsern korrekt umgesetzt ist. Ein XmlHTTPRequest kann nur auf den Ursprungsserver abgesetzt werden. Somit sind extern nachgeladene Skripten außen vor. Hat man natürlich eine Cross-Site-Scripting-Schwäche in der Anwendung …

Die Verteidigung

Grundsätzlich ist Wicket, wie jedes andere Webframework, anfällig für diese Angriffe. Allerdings liefert es auch alles was benötigt wird um diesen Angriff deutlich zu erschweren oder ihn sogar unmöglich zu machen.

In Wicket 1.4 gibt es hierfür die CryptedUrlWebRequestCodingStrategy. Mit folgendem Code-Snipplet kann sie in der Application registriert werden.

...
    protected IRequestCycleProcessor newRequestCycleProcessor() {
        return new WebRequestCycleProcessor() {
        protected IRequestCodingStrategy newRequestCodingStrategy() {
        return new CryptedUrlWebRequestCodingStrategy(new WebRequestCodingStrategy());
    }
...
}

Wicket 1.5 bietet die gleiche Funktionalität im CryptoMapper. Der wie folgt registriert wird:

...
    @Override
    public void init() {
        super.init();
        setRootRequestMapper(new CryptoMapper(getRootRequestMapper(), this));
    }
...

In beiden Fällen wird von un an für jede neu erzeugte Session ein entsprechender Schlüssel erzeugt und in ihr hinterlegt. Im Folgenden wird jede URL, bevor sie ausgeliefert wird, mit diesem verschlüsselt.

Beim Aufruf einer URL in der Anwendung (via AJAX oder dem Klicken eines Links) versucht Wicket diese zu entschlüsseln. Funktioniert dies nicht, was auf alle URLs zutrifft die nicht von Wicket ausgeliefert wurden, wird eine Fehlermeldung erzeugt.

Nachteile

Alles hat seinen Preis, auch der Schutz vor CSRF-Angriffen. Man verliert die Möglichkeit BookmarkablePages zu nutzen. Auch ihre URLs werden entsprechend verschlüsselt. Da der Schlüssel für jede Session neu erzeugt wird sind diese Links spätestens nach dem Ausloggen ungültig.

Durch die Nutzung von PageMounts kann man dieses Problem abmildern da deren URLs immer verfügbar sind.

Auch mag es stören eine Schlüssel pro Session zu erzeugen da dies die Existenz einer Session voraussetzt. Via Application.getSecuritySettings().setCryptFactory() lässt sich hier ein eigenes CryptFactory registrieren welches einen applikationsweiten Schlüssel erzeugt.

Fazit

Trotz der erwähnten Nachteile sehe ich die CryptedUrlWebRequestCodingStrategy als das sicherste und bequemste Mittel zur Abwehr von CSRF-Angriffen. Wer sich Sorgen um die zusätzliche Last durch die Verschlüsselung macht wird hier demnächst noch eine kleine Lastanalyse finden.