Thursday, December 27, 2012

Reading https url from a self-signed cert

Groovy has made fetching data from URL a snap:
println(new URL("http://www.google.com").text)
But have you ever try to get data from an https of an site that's using a self signed certificate? A browser will prompt you a risk warning, but it let you trust it and still continue. But it's much more hassle if we want to fetch the data programmatically. For example, try fetching https://www.pcwebshop.co.uk and you will see that it failed miserably:
Caught: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1762)
    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
    at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
    at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:958)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1230)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1214)
    at test.run(test.groovy:2)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1185)
    ... 8 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    ... 12 more
This is due to Java API won't let you get data if your certs is not in the keystore or "trusted". Now if you have your own testing site and did a self-signed like above, then sure you just want to trust it temporary and simply want to fetch the data. However configuring the Java API to do such simple task is a nightmare.
Today, I found a library that allow you to perform exactly this: https://github.com/kevinsawicki/http-request Trying this out using Groovy is pretty sweet:
@Grab('com.github.kevinsawicki:http-request:3.1')
import com.github.kevinsawicki.http.*
def req = HttpRequest.get("https://www.pcwebshop.co.uk")
req.trustAllCerts()
req.trustAllHosts()
println(req.body())
And Voila! We read the URL context without the pesky exception. This http-request is awesome! Now I wish JDK let you do simple thing like this. This is extremely useful for testing.

Friday, December 21, 2012

A strange case of Java generic and inheritage parameter passing

I came a cross some strange Java code and I would like to share it here. Take a look few of classes I have here:

// file: AFoo.java
package atest.deng;
public abstract class AFoo<T> {
}

// file: Foo.java
package atest.deng;
public class Foo extends AFoo<String> {
}

// file: FooProcessor.java
package atest.deng;
public class FooProcessor<T> {
    public void process(Class<AFoo<?>> cls) {
        System.out.println(cls);
    }
}

// file: FooMain.java
package atest.deng;
public class FooMain {
    public static void main(String[] args) {
        new FooProcessor().process(Foo.class);
    }
}

bash> mvn compile
bash> [INFO] BUILD SUCCESS

I tried this with JDK6 + Maven and it compiles without problem. But try to remove the <T> part from FooProcessor and it will fail to compile!

// file: FooProcessor.java
package atest.deng;
public class FooProcessor {
    public void process(Class<AFoo<?>> cls) {
        System.out.println(cls);
    }
}

bash> mvn clean compile
bash> [ERROR] java-demo\src\main\java test\deng\FooMain.java:[4,26] process(java.lang.Class<atest.deng.AFoo<?>>) in atest.deng.FooProcessor cannot be applied to (java.lang.Class<atest.deng.Foo>)

Without the <T> the code won't compile, and yet we are not even using it in this case. How and why <T> affects the method parameters invocation?

Now, we can improve the FooProcessor in this way so that the presence of <T> doesn't have any affect.

package atest.deng;
public class FooProcessor {
    public void process(Class<? extends AFoo<?>> cls) {
        System.out.println(cls);
    }
}

That's a more proper way to write the generic parameter anyway. But despite a better solution, the puzzle is that the original code compiled under the compiler, but only with the <T> presented, and yet it's not used. Wouldn't you consider this as a Java compiler bug?

Wednesday, December 19, 2012

A simple Groovy issue tracker using file system

It will be a chaos not to track bugs and feature requests when you developing software. Having a simple issue tracker would make managing the project much more successful. Now I like simple stuff, and I think for small project, having this tracker right inside the source control (especially with DSVC like Mercurial/Git etc) repository is not only doable, but very convenient as well. You don't have to go crazy with all the fancy features, but just enough to track issues are fine. I would like to propose this layout for you.

Let's say you have a project that looks like this

project
 +- src/main/java/Hello.java
 +- issues/issue-001.md
 +- pom.xml

All I need is a simple directory issues to get going. Now I have a place to track my issue! First issue issue-000.md should be what your project is about. For example:

/id=issue-001
/createdon=2012-12-16 18:07:08
/type=bug
/status=new
/resolution=
/from=Zemian
/to=
/found=
/fixed=
/subject=A simple Java Hello program

# Updated on 2012-12-16 18:07:08

We want to create a Maven based Hello world program. It should print "Hello World."

I choose .md as file extension for intending to write comments in Markdown format. Since it's a text file, you do what you want. To be more structured, I have added some headers metadata for issue tracking. Let's define some here. I would propose to use these and formatting:


 /id=issue-<NUM>
 /createdon=<TIMESTAMP>
 /type=feature|bug|question
 /status=new|reviewing|working|onhold|testing|resolved
 /resolution=fixed|rejected|duplicated
 /from=<REPORTER_FROM_NAME>
 /to=<ASSIGNEE_TO_NAME>
 /found=<VERSION_FOUND>
 /fixed=<VERSION_FIXED>

That should cover most of the bug and feature development issues. It's not cool to write software without a history of changes, including these issues created. So let's use a source control. I highly recommend you to use Mercurial hg. You can create and initialize a new repository like this.

bash> cd project
bash> hg init
bash> hg add
bash> hg commit -m "My hello world project"

Now your project is created and we have a place to track your issues. Now it's simple text file, so use your favorite text editor and edit away. However, creating new issue with those header tags is boring. It will be nice to have a script that manage it a little. I have a Groovy script issue.groovy (see at the end of this article) that let you run reports and create new issues. You can add this script into your project/issues directory and you can instantly creating new issue and querying reports! Here is an example output on my PC:

bash> cd project
bash> groovy scripts/issue.groovy

Searching for issues with /status!=resolved
Issue: /id=issue-001 /status=new /subject=A simple Java Hello program
1 issues found.

bash> groovy scripts/issue.groovy --new /type=feature /subject='Add a unit test.'

project/issues/issue-002.md created.
/id=issue-002
/createdon=2012-12-16 19:10:00
/type=feature
/status=new
/resolution=
/from=Zemian
/to=
/found=
/fixed=
/subject=Add a unit test.

bash> groovy scripts/issue.groovy

Searching for issues with /status!=resolved
Issue: /id=issue-000 /status=new /subject=A simple Java Hello program
Issue: /id=issue-002 /status=new /subject=Add a unit test.
2 issues found.

bash> groovy scripts/issue.groovy --details /id=002

Searching for issues with /id=002
Issue: /id=issue-002
  /createdon=2012-12-16 19:10:00 /found= /from=Zemian /resolution= /status=new /type=feature
  /subject=Add a unit test.
1 issues found.

bash> groovy scripts/issue.groovy --update /id=001 /status=resolved /resolution=fixed 'I fixed this thang.'
Updating issue /id=issue-001
Updating /status=resolved
Updating /resolution=fixed

Update issue-001 completed.

The script give you some quick and consistent way to create/update/search issues. But they are just plain text files! You can just as well fire up your favorite text editor and change any any thing you want. Save and even commit it into your source repository. All will not lost.

I hope this issue tracking script can get your next project started quickly. Let me know what you do you think!

Enjoy!

Zemian

And here is my issue.groovy script.


Oh, and of course I eat my own *dog food. Here are few issues that I started to track the issue.grooy itself.

Saturday, December 15, 2012

Getting started with Quartz Scheduler on MySQL database

Here are some simple steps to get you fully started with Quartz Scheduler on MySQL database using Groovy. The script below will allow you to quickly experiment different Quartz configuration settings using an external file.

First step is to setup the database with tables. Assuming you already have installed MySQL and have access to create database and tables.

bash> mysql -u root -p

sql> create database quartz2;
sql> create user 'quartz2'@'localhost' identified by 'quartz2123';
sql> grant all privileges on quartz2.* to 'quartz2'@'localhost';
sql> exit;

bash> mysql -u root -p quartz2 < /path/to/quartz-dist/docs/dbTables/tables_mysql.sql
The tables_mysql.sql can be found from Quartz distribution download, or directly from their source here. Once the database is up, you need to write some code to start up the Quartz Scheduler. Here is a simply Groovy script quartzServer.groovy that will run as a tiny scheduler server.
// Run Quartz Scheduler as a server
// Author: Author: Zemian Deng, Date: 2012-12-15_16:46:09
@GrabConfig(systemClassLoader=true)
@Grab('mysql:mysql-connector-java:5.1.22')
@Grab('org.slf4j:slf4j-simple:1.7.1')
@Grab('org.quartz-scheduler:quartz:2.1.6')
import org.quartz.*
import org.quartz.impl.*
import org.quartz.jobs.*

config = args.length > 0 ? args[0] : "quartz.properties"
scheduler = new StdSchedulerFactory(config).getScheduler()
scheduler.start()

// Register shutdown
addShutdownHook {
  scheduler.shutdown()
}

// Quartz has its own thread, so now put this script thread to sleep until
// user hit CTRL+C
while (!scheduler.isShutdown()) {
 Thread.sleep(Long.MAX_VALUE)
}
And now you just need a config file quartz-mysql.properties that looks like this:
# Main Quartz configuration
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = DatabaseScheduler
org.quartz.scheduler.instanceId = NON_CLUSTERED
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.dataSource = quartzDataSource
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 5

# JobStore: JDBC jobStoreTX
org.quartz.dataSource.quartzDataSource.driver = com.mysql.jdbc.Driver
org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/quartz2
org.quartz.dataSource.quartzDataSource.user = quartz2
org.quartz.dataSource.quartzDataSource.password = quartz2123
org.quartz.dataSource.quartzDataSource.maxConnections = 8
You can run the Groovy script as usual
bash> groovy quartzServer.groovy quartz-mysql.properties
Dec 15, 2012 6:20:26 PM com.mchange.v2.log.MLog 
INFO: MLog clients using java 1.4+ standard logging.
Dec 15, 2012 6:20:27 PM com.mchange.v2.c3p0.C3P0Registry banner
INFO: Initializing c3p0-0.9.1.1 [built 15-March-2007 01:32:31; debug? true; trace:10]
[main] INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
[main] INFO org.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
[main] INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.1.6 created.
[main] INFO org.quartz.core.QuartzScheduler - JobFactory set to: org.quartz.simpl.SimpleJobFactory@1a40247
[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Using thread monitor-based data access locking (synchronization).
[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - JobStoreTX initialized.
[main] INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.1.6) 'DatabaseScheduler' with instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 5 threads.
  Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is not clustered.

[main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'DatabaseScheduler' initialized from the specified file : 'quartz-mysql.properties' from the class resource path.
[main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.1.6
Dec 15, 2012 6:20:27 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts
-> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1hge16k8r18mveoq1iqtotg|1486306, debugUnreturnedConnectionStackTraces -> fals
e, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hge16k8r18mveoq1iqtotg|1486306, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://localhost:3306/quartz2, lastAcquisitionFailureDefaultUser -> null, maxAdministrativeTaskTime -> 0
, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 8, maxStatements -> 0, maxStatementsPerConnection -> 120, minPoolSize -> 1, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, pref
erredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]
[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Freed 0 triggers from 'acquired' / 'blocked' state.[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Recovering 0 jobs that were in-progress at the time of the last shut-down.
[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Recovery complete.
[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Removed 0 'complete' triggers.
[main] INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Removed 0 stale fired job entries.
[main] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED started.


... CTRL+C
[Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED shutting down.
[Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED paused.
[Thread-6] INFO org.quartz.core.QuartzScheduler - Scheduler DatabaseScheduler_$_NON_CLUSTERED shutdown complete.
That's a full run of above setup. Go ahead and play with different config. Read http://quartz-scheduler.org/documentation/quartz-2.1.x/configuration for more details.
Here I will post couple more easy config that will get you started in a commonly used config set.

A MySQL cluster enabled configuration. With this, you can start one or more shell terminal and run different instance of quartzServer.groovy with the same config. All the quartz scheduler instances should cluster themselve and distribute your jobs evenly.

# Main Quartz configuration
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = DatabaseClusteredScheduler
org.quartz.scheduler.instanceId = AUTO
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.dataSource = quartzDataSource
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 5

# JobStore: JDBC jobStoreTX
org.quartz.dataSource.quartzDataSource.driver = com.mysql.jdbc.Driver
org.quartz.dataSource.quartzDataSource.URL = jdbc:mysql://localhost:3306/quartz2
org.quartz.dataSource.quartzDataSource.user = quartz2
org.quartz.dataSource.quartzDataSource.password = quartz2123
org.quartz.dataSource.quartzDataSource.maxConnections = 8
Here is another config set for a simple in-memory scheduler.
# Main Quartz configuration
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = InMemoryScheduler
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 5

Now, if you need more fancy UI management of Quartz, give MySchedule a try.

Happy scheduling!

Friday, December 14, 2012

Cygwin tips: open anything with cygstart

Cygwin has this cool command called cygstart, and I usually alias it as open.
alias open='cygstart'
Here are few things I use often:
 
# bring up Windows Explore with folder open
bash> open /cygdrive/c/temp
bash> open "$USERPROFILE"
 
# bring up browser on a url
bash> open http://google.com
 
# bring up any windows program by file extension
bash> open /cygdrive/c/my.docx

Because I use a Cygwin terminal shell a lot, I have many alias shortcuts to many different directories. The open . allows me to jump to any directory and starts the Windows Explore with that directory expanded quickly. You can see more options with open --help.

Thursday, December 13, 2012

A command to remove all Windows mapped drives

If you work in a Corp env and having a Laptop, you will likely go home with it still running and it must has all the network drives still mapped. And then later when you are at home and try to use your Laptop, you will see considerable delay. The Windows Explore and many Windows application that access File Browser dialog will try relentlessly in reconnecting those drives without sucess. This also affect your Cygwin shell too.

Here is a DOS command to remove them all.

C:> net use * /delete /y
If you are in Cygwin bash, then you must quote the '*'.

Wednesday, December 12, 2012

Intellij IDEA is pretty awesome!

Did you know IntelliJ IDEA has a free Community Edition you may download? I've been test driving it, and I will post few things I learned so far. Here are few things that I usually change from out of the default installed settings.
  1. File > Settings > Editor => Uncheck "Allow placement of caret after end of line" -- This is the most annoying thing for me in IDE. It's just not for me.
  2. File > Settings > Editor > Appearance => Check "Show line numbers" -- I tend to relate errors and navigate content better with line number displayed.
  3. File > Settings > Appearance > Theme => Select "Darcula" -- This is latest version 12.0 feature. Dark screen is pretty eye pleasing compare to WHITE.
  4. File > Settings > Compiler => Check "Make project automatically" -- Another new 12.0 feature that would perform incremental compile.

Now the IDE is ready for me to work on any Java projects. If you work with Maven, then you will be happy to know that IDEA supports it out of box. You can get an existing project started with following steps:

  1. File > Import Project => Browse to your project pom.xml. Even if you have multi Maven modules, you only need to pick the top level pom.xml and it will generate the multi IDEA modules for you automatically. Then click "Ok" button.
  2. In "Import project from external model" select "Maven", and then "Next".
  3. You can accept all default settings on this prompt, and simply go "Next"
  4. Check all maven project to import
  5. Next screen is selecting a SDK for this project (meaning pick a JDK) to use. If this is your first time setup, then create a new one by click on the Grean plus icon, then browse to your JAVA_HOME directory and click OK. Click "Next" to go next step.
  6. Simply click "Next" to accept default project name. Then you are at last prompt. Simply click "Finish" button and you should have your project ready to go.

It might seems like a lot of steps to import a project, but IDEA is really nice. After your project has completely imported and compiled, you should able open any of your Java class (use CTRL+N) that has main method or any JUnit classes. Right click the content editor and select "Run ..." to execute it (or CTRL+SHIFT+F10 to run any single unit test method).

I have tried a decent size Java projects (few thousand source files) and it handles it very gracefully. The UI such as Menu items, Control buttons and tool bars are very user friendly and intuitive to use. Also, right click on most places should give you a "Context" menu with related options. Like any IDE, it supports the CTRL+Space that auto complete code as you type. There is the ALT+ENTER would suggest hints on how to fix things when you got an error with red squiggle line. Pressing CTRL+B will jump into methods or variable declarations.

Another tips I have for you is the IDEA uses many icon symbols to represent many things. These give you quick visual status on files, progress and states of your Classes etc. It's helpful to see what they mean using this http://www.jetbrains.com/idea/webhelp/symbols.html

So overall, I think IntelliJ IDEA is pretty good and awesome in many way.

Saturday, December 8, 2012

Checking DB Connection using Java

For complete sake, here is a Java version of the Groovy post to test your Oracle Database connection.
package atest;
import java.sql.*;
/**
 * Run arguments sample:
 * jdbc:oracle:thin:@localhost:1521:XE system mypassword123 oracle.jdbc.driver.OracleDriver
 */
public class DbConn {
    public static void main(String[] args) throws Exception {
        String url = args[0];
        String username = args[1];
        String password = args[2];
        String driver = args[3];

        Class.forName(driver);
        Connection conn = DriverManager.getConnection(url, username, password);
        try {
            Statement statement = conn.createStatement();
            ResultSet rs = statement.executeQuery("SELECT SYSDATE FROM DUAL");
            while(rs.next()) {
                System.out.println(rs.getObject(1));
            }
        } finally {
            conn.close();
        }
    }
}

Friday, December 7, 2012

Creating Oracle Stored Procedure using Java

Did you know you can write Oracle database stored procedure in Java? Give this a try in your sqlplus prompt.
sql> create or replace and compile java source named "MyJavaDbProcedure" as
sql> public class MyJavaDbProcedure {
sql>   public static String upcase(String text) {
sql>     return text.toUpperCase();
sql>   }
sql> };
sql> /
sql> 
sql> create or replace function upcase (s in varchar2)
sql>   return varchar2
sql> as language java
sql>   name 'MyJavaDbProcedure.upcase(java.lang.String) return java.lang.String';
sql> /
sql> 
sql> select upcase('hello') from dual;
sql> /

I let the database compile a Java source directly, but there is also the java class PL/SQL that you can load Java binary .class file as well. I am sure your DBA will fight all their might to prevent you doing stuff like this. But it's cool to see that this option is available.

Thursday, December 6, 2012

Checking DB connection using Groovy

Here is a simple Groovy script to verify Oracle database connection using JDBC.

@GrabConfig(systemClassLoader=true)
@Grab('com.oracle:ojdbc6:11g')
url= "jdbc:oracle:thin:@localhost:1521:XE"
username = "system"
password = "mypassword123"
driver = "oracle.jdbc.driver.OracleDriver"

// Groovy Sql connection test
import groovy.sql.*
sql = Sql.newInstance(url, username, password, driver)
try {
  sql.eachRow('select sysdate from dual'){ row ->
    println row
  }
} finally {
  sql.close()
}

This script should let you test connection and perform any quick ad hoc queries programmatically. However, when you first run it, it would likely failed without finding the Maven dependency for JDBC driver jar. In this case, you would need to first install the Oracle JDBC jar into maven local repository. This is due to Oracle has not publish their JDBC jar into any public Maven repository. So we are left with manually steps by installing it. Here are the onetime setup steps:

1. Download Oracle JDBC jar from their site: http://www.oracle.com/technetwork/database/features/jdbc/index-091264.html.

2. Unzip the file into C:/ojdbc directory.

3. Now you can install the jar file into Maven local repository using Cygwin.

bash> cd /cygdrive/c/ojdbc
bash> mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11g -Dpackaging=jar -Dfile=ojdbc6-11g.jar

That should make your script run successfully. The Groovy way of using Sql has many sugarcoated methods that you let you quickly query and see data on screens. You can see more Groovy feature by studying their API doc.

Note that you would need systemClassLoader=true to make Groovy load the JDBC jar into classpath and use it properly.

Oh, BTW, if you are using Oracle DB production, you will likely using a RAC configuration. The JDBC url connection string for that should look something like this:

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=MY_DB)))

Update: 12/07/2012

It appears that the groovy.sql.Sql class has a static withInstance method. This let you run onetime DB work without writing try/finally block. See this example:

@GrabConfig(systemClassLoader=true)
@Grab('com.oracle:ojdbc6:11g')
url= "jdbc:oracle:thin:@localhost:1521:XE"
username = "system"
password = "mypassword123"
driver = "oracle.jdbc.driver.OracleDriver"

import groovy.sql.*
Sql.withInstance(url, username, password, driver) { sql ->
  sql.eachRow('select sysdate from dual'){ row ->
    println row
  }
}

It's much more compact. But be aware of performance if you run it multiple times, because you will open and close the a java.sql.Connection per each call!


I have also collected couple other popular databases connection test examples. These should have their driver jars already in Maven central, so Groovy Grab should able to grab them just fine.
// MySQL database test
@GrabConfig(systemClassLoader=true)
@Grab('mysql:mysql-connector-java:5.1.22')
import groovy.sql.*
Sql.withInstance("jdbc:mysql://localhost:3306/mysql", "root", "mypassword123", "com.mysql.jdbc.Driver"){ sql -> 
  sql.eachRow('SELECT * FROM USER'){ row ->
    println row
  }
}
// H2Database
@GrabConfig(systemClassLoader=true)
@Grab('com.h2database:h2:1.3.170')
import groovy.sql.*
Sql.withInstance("jdbc:h2:~/test", "sa", "", "org.h2.Driver"){ sql -> 
  sql.eachRow('SELECT * FROM INFORMATION_SCHEMA.TABLES'){ row ->
    println row
  }
}

Wednesday, December 5, 2012

Changing Oracle XE apex webapp port

If you use Oracle XE for development, then you know it has a simple web interface through http://localhost:8080/apex. Now port 8080 is a common port used by Tomcat server, so letting Oracle XE permanently have taken it is not convenient. But you can change it like this:
 sql> select dbms_xdb.gethttpport from dual;
 sql> exec dbms_xdb.sethttpport('8081');
 sql> commit;

 bash> open http://localhost:8081/apex
Also, if you ever deleted the HR sample database on the XE installation, you can restore it like this:
 bash> cd /c/oraclexe/app/oracle/product/10.2.0/server/demo/schema/human_resources
 bash> sqlplus system < hr_main.sql

Tuesday, December 4, 2012

Inspecting your Oracle database

Some quick tips on how to inspect what you have in your Oracle database.

-- show all schemas
select distinct owner from dba_segments where owner in
     (select username from dba_users where default_tablespace not in ('SYSTEM','SYSAUX'));

-- show all tables from a schema/owner
select * from all_tables where owner = 'HR';

-- show table description
desc HR.EMPLOYEES;

-- show all users
select * from all_users;
select username, * from dba_users;


-- See who is taking up a DB lock
select c.*, b.* from v$lock a, dba_locks b, v$session c 
  where a.id1 = b.lock_id1 and b.session_id = c.sid

-- See internal SQL id
select * from v$sql

-- See Oracle latches:
select * from v$latch

-- See Library Cache of SGA (System Global Area) like buffer cache size:
select * from v$sgastat

-- See tx locks (TX) and DML locks (TM):
select * from v$lock where type in ('TX', 'TM')
select * from dba_locks where lock_type in ('Transaction', 'DML')


Thursday, November 29, 2012

A very light Groovy based web application project template

You might have heard of the project Grails is a Groovy version of Ruby on Rails like framework that let you create web application much more easier with Dynamic scripting. Despite all that power Grails provided, it is not "light" if you look under the hood. I am not saying Grails is bad or anything. Grails is actually pretty cool to write web application with. However I found myself often want something even lighter and yet still want to prototype with Groovy. So here I will show you a maven-groovy-webapp project template that I use to get start any web application development. It's very simple, light, and yet very Groovy.

How to get started

Unzip maven-webapp-groovy.zip above and you should see these few files:

bash> cd maven-webapp-groovy
bash> find .
bash> ./pom.xml
bash> ./README.txt
bash> ./src
bash> ./src/main
bash> ./src/main/java
bash> ./src/main/java/deng
bash> ./src/main/java/deng/GroovyContextListener.java
bash> ./src/main/resources
bash> ./src/main/resources/log4j.properties
bash> ./src/main/webapp
bash> ./src/main/webapp/console.gt
bash> ./src/main/webapp/health.gt
bash> ./src/main/webapp/home.gt
bash> ./src/main/webapp/WEB-INF
bash> ./src/main/webapp/WEB-INF/classes
bash> ./src/main/webapp/WEB-INF/classes/.keep
bash> ./src/main/webapp/WEB-INF/groovy
bash> ./src/main/webapp/WEB-INF/groovy/console.groovy
bash> ./src/main/webapp/WEB-INF/groovy/health.groovy
bash> ./src/main/webapp/WEB-INF/groovy/home.groovy
bash> ./src/main/webapp/WEB-INF/groovy/init.groovy
bash> ./src/main/webapp/WEB-INF/groovy/destroy.groovy
bash> ./src/main/webapp/WEB-INF/web.xml

As you can see it's a maven based application, and I have configured tomcat plugin, so you may run it like this:

bash> mvn tomcat7:run
bash> open http://localhost:8080/maven-webapp-groovy/home.groovy

And ofcourse, with maven, running package phase will let you deploy it into any real application servers when ready.

bash> mvn package
bash> cp target/maven-webapp-groovy.war $APP_SERVER_HOME/autodeploy

What's in it

You should checkout the main config in web.xml file, and you'll see that there couple built-in Groovy servlets and a custom listener.

<?xml version="1.0"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
        version="2.5">

    <description>Groovy Web Application</description>
    <welcome-file-list>
        <welcome-file>home.groovy</welcome-file>
    </welcome-file-list>

    <servlet>
        <servlet-name>GroovyServlet</servlet-name>
        <servlet-class>groovy.servlet.GroovyServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>GroovyServlet</servlet-name>
        <url-pattern>*.groovy</url-pattern>
    </servlet-mapping>

    <servlet>
        <servlet-name>TemplateServlet</servlet-name>
        <servlet-class>groovy.servlet.TemplateServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>TemplateServlet</servlet-name>
        <url-pattern>*.gt</url-pattern>
    </servlet-mapping>

    <listener>
        <listener-class>deng.GroovyContextListener</listener-class>
    </listener>
    <context-param>  
       <param-name>initScripts</param-name>
       <param-value>/WEB-INF/groovy/init.groovy</param-value>
    </context-param>
    <context-param>    
       <param-name>destroyScripts</param-name>
       <param-value>/WEB-INF/groovy/destroy.groovy</param-value>
    </context-param>

</web-app>

I've chosen to use GroovyServlet as a controller (it comes with Groovy!), and this let you use any scripts inside /WEB-INF/groovy directory. That's it, no further setup. That's about the only requirement you need to get a Groovy webapp started! See console.groovy as example and how it works. It's a groovy version of this JVM console

Now you can use Groovy to process any logic and even generate the HTML output if you like, but I find it even more easier to use TemplateServlet. This allow Groovy template files to be serve as view. It's very much like JSP, but it uses Groovy instead! And we know Groovy syntax are much shorter to write! See console.gt as exmaple and how it works.

The GroovyContextListener is something I wrote, and it's optional. This allow you to run any scripts during the webapp startup or shutdown states. I've created an empty init.groovy and destroy.groovy placeholder. So now you have all the hooks you need to prototype just about any web application you need.

Simplicity wins

This setup is just plain Java Servlet with Groovy loaded. I often think the more simple you get, then less bug and faster you code. No heavy frameworks, no extra learning curve, (other than basic Servlet API and Groovy/Java skills ofcourse), and off you go.

Go have fun with this Groovy webapp template! And let me know if you have some cool prototypes to show off after playing with this. :)

Friday, November 16, 2012

Use cygpath with -p option for your Java CLASSPATH conversion

I just noticed that Cygwin's cygpath command supports -p option. This is a real gem when writing Java wrapper script that needs to covert CLASSPATH values. A simple script can demonstrate the purpose.


#!/usr/bin/env bash
# Author: Zemian Deng, Date: 2012-11-16_06:30:00
#
# run.sh - A simple Java wrapper script for Cygwin and Unix/Linux shell. We assume 
# this script is located in a subdiretory inside the application home directory.
# Example:
#   app/bin/run.sh
#   app/config/log4j.properties
#   app/lib/log4j.jar
# Usage:
#   bash> run.sh app.Hello
#
DIR=$(cd "$(dirname $0)/.." && pwd)
CP=${CP:="$DIR/config:$DIR/lib/*"}
if [[ "$OS" == Windows* ]]; then
 CP=$(cygpath -mp $CP)
fi
java -cp "$CP" "$@"

Saturday, November 3, 2012

What does UTF-8 with BOM mean?


Believe it or not, There is no such thing as Plain Text!

All files in a modern Operating Sytems (Windows, Linux, or MacOSX) are saved with an encoding scheme! They are encoded (a table mapping of what each byte means) in such way so that other programs can read it back and understand how to get information out. It happens that US/ASCII encoding is earliest and widely used that people think it's just "Plain Tex". But even ASCII is an encoding! It uses 7 bits in mapping all US characters in saving the bytes into file. Obviously you are free to use any kind of encoding (mapping) scheme to save any files, but if you want other programs to read it back easily, then sticking to some standard ones would help a lot. Without an agreed upon encoding, programs will not able to read files and be any useful!
The most useful and practical file encoding today is "UTF-8" because it support Unicode, and it's widely used in internet.

I discovered something odd when using Eclipse and Notepadd++. In Ecilpse, if we set default encoding with UTF-8, it would use normal UTF-8 without the Byte Order Mark (BOM). But in Notepad++, it appears to support UTF-8 wihtout BOM, but it won't recoginze it when first open. You can check this by going Menu > Encoding and see which one is selected. Notepad++ seems to only recognize UTF-8 wihtout BOM with ones it converted by it's own conversion utility. Perhaps it's a bug in notepad++.

So what is BOM? The byte order mark is useless for UTF-8. They only used for UTF-16 so they know which byte order is first. But UTF-8 will allow you to save these BOM for conversion purpose... they are ineffective in encoding the doc itself. So a "normal" UTF-8, it won't have BOM, but Windows would like to use them anyway. The Windows NOTEPAD would automatically save BOM in UTF-8!

So be-aware when viewing UTF-8 without BOM encoding files in Notepad++, as it can be deceiving at first glance.

Ref: 

Sunday, October 28, 2012

A simple way to setup Java application with external configuration file

Many Java applications would deploy and run it with some kind of external configuration files. It's very typical that you would want a set of config files per environments such as DEV, QA and PROD. There many options in tackling this problem, especially in a Java app, but keeping it simple and easy to maintain would take some disciplines.
Here I would layout a simple way you may use to depploy a typical Java application. The concept is simple and you can easily apply to a standalone, web, or even a JEE application.

Use a env System Properties per environment

Java allows you to invoke any program with extra System Properties added. When launching an Java application, you should set an env property. For example:
bash> java -Denv=prod myapp.Main
This property would give your Main application to identify which environment you are running against with. You should be reading it like this inside your code:
String env = System.getProperty("env", "dev");
This way you always would have an environment to work with. Even if user doesn't supply one, it will default to use dev env.
Another useful System Property to set is app.home. You want to set this value in relative to where your application is deployed, so you may reference any files (eg: data) easily. To do this, you can use a script wrapper to automatically calculate the path. See section below for an example.

Use a config dir prefix in CLASSPATH

In stead of passing a explicit config file to your application as argument, another flexible way to load configuration file is to add an extra config folder into your CLASSPATH. For example, you can easily create a startup wrapper script myapp.sh like this:
#/usr/bin/env bash
APP_HOME=$(cd "`dirname $0`/.." && pwd)
java $JAVA_OPTS -Dapp.home=$APP_HOME -cp "$APP_HOME/config:$APP_HOME/lib/*" myapp.Main "$@"
From this, you can setup the application packaging layout this way:
myapp
    +- bin
        +- myapp.sh
    +- config
        +- dev.properties
        +- qa.properties
        +- prod.properties
    +- data
        +- myrecords.data
    +- lib
        +- myapp-1.0.0.jar
        +- slf4j-1.7.1.jar
You would typically invoke the application like this:

bash> JAVA_OPTS='-Denv=prod' myapp/bin/myapp.sh

The above will give you a good foundation to load a single config properties file per env. For example, you can read your properites file like this somewhere in your code.

// Get appHome and data dir.
String appHome = System.getProperty("app.home");
String dataDir = appHome + "/data";

// Get env value to load config parameters
String env = System.getProperty("env", "dev");
String config = env + ".properites";
Properties configProps = new Properties();
InputStream inStream = null;
try {
    inStream = getClass().getClassLoader().getResourceAsStream(config);
    configProps.load(inStream);
} finally {
    if (inStream != null)
        inStream.close();
}
// Now load any config parameters from configProps map.
Now you would have the configProps object at your disposal to read any configuration keys and values set per an environment.
NOTE: If you want a more flexible Java wrapper script, see my old post on run-java wrapper.

Do not abuse CLASSPATH

Now, just because you have setup config as part of your CLASSPATH entry, I have to caution you not to abuse it. What I mean is do not go wild on loading all your application resources in that folder! If you have that many resources that user MUST edit and configure, then you should re-think about your application design! Simple interface, or configuration in this case, is always a win. Do not bother users with complexity just because your application can support gazillion ways of configuration combination. If you can keep it as one config file, it would make users very happy.
Also, this doesn't mean you have to put the entire world inside one of prod.properites either. In the real world, an application is likely going to have only handful of user tunable parameters, and many other resources are less frequent used. I would recommand put the most frequently used parameters in these config properties only. For all other (for example most of the Spring context files in an application do not belong to a typical users config level, they are more developer level config files. In another word, changing these files would have catastrophic effect to your application!) You should put these inside as part of your myapp.jar.
You might ask, 'Oh, but what happen if I must want to override one of the resource in the jar?'. But in that very unusual case, you would still have an option to override! You have the config as prefix in CLASSPATH, remember? Even when you nested resources inside a package inside the jar, you would still able to overwrite by simply create same directory structure inside config. You probably only do this for emmergency and less frequent use anyway.

Feedback

So what are some clever ways you have seen or done with your application configuration? I hope to hear from you and share.

Thursday, October 25, 2012

Simple Variable Substitution in Java String

When I wrote about how to improve the Java Properties class using Props, I've shown a feature where you can use variable substition such as mypath=${user.home} in your config file. The implementation underneath it uses the Apache Common Lang library with org.apache.commons.lang.text.StrSubstitutor. There is nothing wrong with this, but I was curious how bad would it be to remove such dependency, so the Props can be more standalone.

Here is a quick implementation in Groovy, but you should able to translate to Java easily.

// String variable substitutions
def parseVariableNames(String text) {
    def names = []
    def pos = 0, max = text.length()
    while (pos < max) {
        pos = text.indexOf('${', pos)
        if (pos == -1)
            break
        def end = text.indexOf('}', pos + 2)
        if (end == -1)
            break
        def name = text.substring(pos + 2, end)
        names.add(name)
        pos = end + 1
    }
    return names
}
def replaceVariable(String key, String value, String text) {
    //println "DEBUG: Replacing '${key}'' with '${value}'"
    result = text.replaceAll('\\$\\{' + key + '}', value)
    return result
}

Probably not the most efficient thing, but it should work. Let's have some tests.

// Test
def map = ["name": "Zemian", "id": "1001"]
def inputs  = [
    'Hello ${name}',
    'My id is ${id}',
    '${name} is a good programmer.',
    '${name}\'s id is ${id}.'
]

result = inputs.collect{ line ->
    def names = parseVariableNames(line)
    names.each{ key ->
        line = replaceVariable(key, map.get(key), line) 
    }
    line
}
assert result == [
    'Hello Zemian',
    'My id is 1001',
    'Zemian is a good programmer.',
    'Zemian\'s id is 1001.'
]

The output should print nothing, as it passed the test. What do you think?

Tuesday, October 23, 2012

Exploring different scheduling types with Quartz 2

We often think of Cron when we want to schedule a job. Cron is very flexible in expressing an repeating occurance of an event/job in a very compact expression. However it's not answer for everything, as I often see people are asking for help in the Quartz user forum. Did you know that the popular Quartz 2 library provide many other schedule types (called Trigger) besides cron? I will show you each of the Quartz 2 built-in schedule types here within a complete, standalone Groovy script that you can run and test it out. Let's start with a simple one.

@Grab('org.quartz-scheduler:quartz:2.1.6')
@Grab('org.slf4j:slf4j-simple:1.7.1')
import org.quartz.*
import org.quartz.impl.*
import org.quartz.jobs.*

import static org.quartz.DateBuilder.*
import static org.quartz.JobBuilder.*
import static org.quartz.TriggerBuilder.*
import static org.quartz.SimpleScheduleBuilder.*

def trigger = newTrigger()
    .withSchedule(
        simpleSchedule()
        .withIntervalInSeconds(3)
        .repeatForever())
    .startNow()
    .build()
dates = TriggerUtils.computeFireTimes(trigger, null, 20)
dates.each{ println it }

This is the Quartz's SimpleTrigger, and it allows you to create a fixed rate repeating job. You can even limit to certain number of count if you like. I have imported all the nessary classes the script needs, and I use the latest Quartz 2.x builder API to create an instance of the trigger.

The quickest way to explore and test out whether a scheduling fits your need is to print out its future execution times. Hence you see me using TriggerUtils.computeFireTimes in the script. Run the above and you should get the datetimes as scheduled to be run, in this case every 3 seconds.

bash> $ groovy simpleTrigger.groovy
    Tue Oct 23 20:28:01 EDT 2012
    Tue Oct 23 20:28:04 EDT 2012
    Tue Oct 23 20:28:07 EDT 2012
    Tue Oct 23 20:28:10 EDT 2012
    Tue Oct 23 20:28:13 EDT 2012
    Tue Oct 23 20:28:16 EDT 2012
    Tue Oct 23 20:28:19 EDT 2012
    Tue Oct 23 20:28:22 EDT 2012
    Tue Oct 23 20:28:25 EDT 2012
    Tue Oct 23 20:28:28 EDT 2012
    Tue Oct 23 20:28:31 EDT 2012
    Tue Oct 23 20:28:34 EDT 2012
    Tue Oct 23 20:28:37 EDT 2012
    Tue Oct 23 20:28:40 EDT 2012
    Tue Oct 23 20:28:43 EDT 2012
    Tue Oct 23 20:28:46 EDT 2012
    Tue Oct 23 20:28:49 EDT 2012
    Tue Oct 23 20:28:52 EDT 2012
    Tue Oct 23 20:28:55 EDT 2012
    Tue Oct 23 20:28:58 EDT 2012

The most frequent used scheduling type is the CronTrigger, and you can test it out in similar way.

@Grab('org.quartz-scheduler:quartz:2.1.6')
@Grab('org.slf4j:slf4j-simple:1.7.1')
import org.quartz.*
import org.quartz.impl.*
import org.quartz.jobs.*

import static org.quartz.DateBuilder.*
import static org.quartz.JobBuilder.*
import static org.quartz.TriggerBuilder.*
import static org.quartz.CronScheduleBuilder.*

def trigger = newTrigger()
    .withSchedule(cronSchedule("0 30 08 * * ?"))
    .startNow()
    .build()
dates = TriggerUtils.computeFireTimes(trigger, null, 20)
dates.each{ println it }

The javadoc for CronExpression is very good and you should definately read it throughly to use it effectively. With the script, you can explore all the combination you want easily and verify future fire times before your job is invoked.

Now, if you have some odd scheduling needs such as run a job every 30 mins from MON to FRI and only between 8:00AM to 10:00AM, then don't try to cramp all that into the Cron expression. The Quartz 2.x has a dedicated trigger type just for this use, and it's called DailyTimeIntervalTrigger! Check this out:

@Grab('org.quartz-scheduler:quartz:2.1.6')
@Grab('org.slf4j:slf4j-simple:1.7.1')
import org.quartz.*
import org.quartz.impl.*
import org.quartz.jobs.*

import static org.quartz.DateBuilder.*
import static org.quartz.JobBuilder.*
import static org.quartz.TriggerBuilder.*
import static org.quartz.DailyTimeIntervalScheduleBuilder.*
import static java.util.Calendar.*

def trigger = newTrigger()
    .withSchedule(
        dailyTimeIntervalSchedule()
        .startingDailyAt(TimeOfDay.hourMinuteAndSecondOfDay(8, 0, 0))
        .endingDailyAt(TimeOfDay.hourMinuteAndSecondOfDay(10, 0, 0))
        .onDaysOfTheWeek(MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY)
        .withInterval(10, IntervalUnit.MINUTE))
    .startNow()
    .build()
dates = TriggerUtils.computeFireTimes(trigger, null, 20)
dates.each{ println it }

Another hidden Trigger type from Quartz is CalendarIntervalTrigger, and you would use this if you need to repeat job that's in every interval of a calendar period, such as every year or month etc, where the interval is not fixed, but calendar specific. Here is a test script for that.

@Grab('org.quartz-scheduler:quartz:2.1.6')
@Grab('org.slf4j:slf4j-simple:1.7.1')
import org.quartz.*
import org.quartz.impl.*
import org.quartz.jobs.*

import static org.quartz.DateBuilder.*
import static org.quartz.JobBuilder.*
import static org.quartz.TriggerBuilder.*
import static org.quartz.CalendarIntervalScheduleBuilder.*
import static java.util.Calendar.*

def trigger = newTrigger()
    .withSchedule(
        calendarIntervalSchedule()
        .withInterval(2, IntervalUnit.MONTH))
    .startAt(futureDate(10, IntervalUnit.MINUTE))
    .build()
dates = TriggerUtils.computeFireTimes(trigger, null, 20)
dates.each{ println it }

I hope these will help you get started on most of your scheduling need with Quartz 2. Try these out and see your future fire times before even scheduling a job into the scheduler should save you some times and troubles.

Monday, October 15, 2012

Running maven commands with multi modules project

Have you ever tried running Maven commands inside a sub-module of a multi modules Maven project, and get Could not resolve dependencies for project error msg? And you checked the dependencies that it's missing are those sister modules within the same project! So what gives? It turns out you have to give few more options to get this running correctly, and you have to remember always stays in the parent pom directory to run it!

For exmaple if you checkout the TimeMachine scheduler project, you can invoke the timemachine-hibernate module with maven commands like this:

bash> hg clone https://bitbucket.org/timemachine/scheduler
bash> cd scheduler
bash> mvn -pl timemachine-hibernate -am clean test-compile

You can start the scheduler using Maven like this (remember to stay in the parent pom directory!):

bash > mvn -pl timemachine-scheduler exec:java -Dexec.mainClass=timemachine.scheduler.tool.SchedulerServer -Dexec.classpathScope=test

I have added the -Dexec.classpathScope=test so you will see logging output, because there is an log4j.properties in the classpath for testing.

Without these, you can always run mvn install in the project root directory, then you can cd into any sub-module and run Maven commands. However you will have to keep a tab on what changed in the dependencies, even if they are in sister modules.

You can read more from this article from Sonatype.

Sunday, October 7, 2012

What's up with the JUnit and Hamcrest dependencies?

It's awesome that JUnit is recognizing the usefulness of Hamcrest, because I use these two a lot. However, I find JUnit packaging of their dependencies odd, and can cause class loading problem if you are not careful.

Let's take a closer look. If you look at junit:junit:4.10 from Maven Central, you will see that it has this dependencies graph:

+- junit:junit:jar:4.10:test
    |  - org.hamcrest:hamcrest-core:jar:1.1:test

This is great, except that inside the junit-4.10.jar, you will also find the hamcrest-core-1.1.jar content are embedded!

But why???

I suppose it's a convenient for folks who use Ant, so that they save one jar to package in their lib folder, but it's not very Maven friendly. And you also expect classloading trouble if you want to upgrade Hamcrest or use extra Hamcrest modules.

Now if you use Hamcrest long enough, you know that most of their goodies are in the second module named hamcrest-library, but this JUnit didn't package in. JUnit however chose to include some JUnit+Hamcrest extension of their own. Now including duplicated classes in jar are very trouble maker, so JUnit has a separated module junit-dep that doesn't include Hamcrest core package and help you avoid this issue. So if you are using Maven project, you should use this instead.

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit-dep</artifactId>
    <version>4.10</version>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.hamcrest</groupId>
    <artifactId>hamcrest-library</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
</dependency>

Notice that's a junit-dep, and also on how I have to exclude hamcrest from it. This is needed if you want hamcrest-library that has higher version than the one JUnit comes with, which is 1.1.

Interesting enough, Maven's dependencies in pom is order sensitive when it comes to auto resolving conflicting versions dependencies. Actually it would just pick the first one found and ignore the rest. So you can shorten above without exclusion if, only if, you place the Hamcrest bofore JUnit like this:

<dependency>
    <groupId>org.hamcrest</groupId>
    <artifactId>hamcrest-library</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>junit</groupId>
    <artifactId>junit-dep</artifactId>
    <version>4.10</version>
    <scope>test</scope>
</dependency>

This should make Maven use the following dependencies:

+- org.hamcrest:hamcrest-library:jar:1.2.1:test
|  \- org.hamcrest:hamcrest-core:jar:1.2.1:test
+- junit:junit-dep:jar:4.10:test

However I think using the exclusion tag would probably give you more stable build and not rely on Maven implicit ordering rule. And it avoid easy mistake for Maven beginer users. However I wish JUnit would do a better job at packaging and remove duplicated classes in jar. I personally think it's more productive for JUnit to also include hamcrest-libray instead of just the hamcrest-core jar.

What do you think?

Sunday, September 30, 2012

High Level Design Diagrams for TimeMachine Scheduler

Despite the advance of all the computers, I am one of those guys who still carry a pen and composition notebook around. I am far from those who has photographic memories, nor one who can digest every thing in their heads. Especially when it comes to doing software design. I still find my self scribbling on my notebook quite a bit, or even on piece of napkin if no paper is available. I am also one of those who only need high level ideas drawn out, then I can usually go directly to code, map it to some API and then write some simple implementations to get started.

That's how I started with the TimeMachine Scheduler project a while back. However, as we come to a point where we have a solid runnable code, we must also start to present it to others. Showing my composition notebook is not the prettier thing in the world, specially when I have no gift in penmenship at all. This is a time to look into some professional diagram and design tools to draw up the high level architecture.

I was contacted by Architexa to give them some review of their tool. I think this is a great opportunity for me to try it to draw up some diagrams on the TimeMachine project. Most of what I have diagrammed here are explained in details in the project reference manual, so if you want to know more, please do read it up for further reference.

TimeMachine top level packages

First, the Architexa provides their software as an Eclipse plugin, and after installing it, it can analysis an existing project. I did that and here is the package level view it presents me for the TimeMachine.

TimeMachine packages dependencies diagram

The graph is not very clear on why everything depends on schedule package, but perhaps Schedule is used through the code. Any way the TimeMachine has been carefully separating out the pakcages so it's easy for user to use. All the interfaces you need to scheduler jobs are in timemachine.scheduler. In it, there are couple static factories class: Schedules and JobTasks provide majority of the functionity you need. So the goal is you can do quiet a bit with just import timemachine.scheduler.*;; then as you need each layer functionality, you can import them explicitly the sub-packages.

TimeMachine main classes diagrams

In TimeMachine, you have few major class hierarchy that you must need to get familiar in order to write effective scheduling jobs.

A job in TimeMachine is implemented by a class with JobTask interface. The project provide few for you to get started.

TimeMachine JobTask class diagram

How often and frequent to run your job in the scheduler is provided by a Schedule. Here we provided some most used implementations, including the customizable DateListSchedule that uses a provider generator.

TimeMachine Schedule class diagram

The TimeMachine is a scheduler composed by a stackable service container. Here are some of the services that power the scheduler as a whole. The user/developer can write their own user Service and add into the container as well.

TimeMachine Service class diagram

The services above are then combined together to form the scheduler engine.

TimeMachine SchedulerEngine class diagram

TimeMachine in action diagrams

Depicting sequence diagram is pretty challenging. The Architexa plugin is pretty good in this area, especially when you already have code already done. The Architexa plugin would take advantage of existing code and Java reflection and give you a selectable choice on what and to where as action. Here I will highlight couple actions in TimeMachine.

The most basic startup of the TimeMachine sequence would look something like this.

TimeMachine Scheduler startup up sequence diagram

In the heart of the scheduler logic is in the PollingScheduleRunner service, and here are some of the actions depicted.

TimeMachine PollingScheduleRunner sequence diagram

Using Architexa

Over all, I think Architexa provided a pretty good plugin for Eclipse. I have some problems when generating the sequence diagrams. For example, I can't resize the width between two actors(classes) to display full method name. And in some actions there are no lines drawn! I suspect it's due to the tool is using code inspection, and some of the call are called by different threads.

The plugin itself seems pretty solid. You can install and uninstall without harm. They currently offer free download for small number of users, which I think it's a great way for you to explorer.

I am not too sure how effective of Architexa on builing a collabration of sharing these diagrams as platform. I treat these diagrams as supplement to the project. It certainly helps in explaining high level architeture, but it's far from the complete documentation. No documentation can get any closer to the code. I think the strength of Java being static already provided some level of documentation when just reading the code alone. I would rather Architexa to focus and perfecting the plugin that draw the diagram, because these are more important to me.

Saturday, September 29, 2012

Enhancing Spring Test Framework with beforeClass and afterClass setup

How to allow instance methods to run as JUnit BeforeClass behavior

JUnit allows you to setup methods on the class level once before and after all tests methods invocation. However, by design on purpose that they restrict this to only static methods using @BeforeClass and @AfterClass annotations. For example this simple demo shows the typical Junit setup:

package deng.junitdemo;

import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;

public class DemoTest {

    @Test
    public void testOne() {
        System.out.println("Normal test method #1.");
    }

    @Test
    public void testTwo() {
        System.out.println("Normal test method #2.");
    }

    @BeforeClass
    public static void beforeClassSetup() {
        System.out.println("A static method setup before class.");
    }

    @AfterClass
    public static void afterClassSetup() {
        System.out.println("A static method setup after class.");
    }
}

And above should result the following output:

A static method setup before class.
Normal test method #1.
Normal test method #2.
A static method setup after class.

This usage is fine for most of the time, but there are times you want to use non-static methods to setup the test. I will show you a more detailed use case later, but for now, let's see how we can solve this naughty problem with JUnit first. We can solve this by making the test implements a Listener that provide the before and after callbacks, and we will need to digg into JUnit to detect this Listener to invoke our methods. This is a solution I came up with:

package deng.junitdemo;

import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(InstanceTestClassRunner.class)
public class Demo2Test implements InstanceTestClassListener {

    @Test
    public void testOne() {
        System.out.println("Normal test method #1");
    }

    @Test
    public void testTwo() {
        System.out.println("Normal test method #2");
    }

    @Override
    public void beforeClassSetup() {
        System.out.println("An instance method setup before class.");
    }

    @Override
    public void afterClassSetup() {
        System.out.println("An instance method setup after class.");
    }
}

As stated above, our Listener is a simple contract:

package deng.junitdemo;

public interface InstanceTestClassListener {
    void beforeClassSetup();
    void afterClassSetup();
}

Our next task is to provide the JUnit runner implementation that will trigger the setup methods.

package deng.junitdemo;

import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;

public class InstanceTestClassRunner extends BlockJUnit4ClassRunner {

    private InstanceTestClassListener instanceSetupListener;

    public InstanceTestClassRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected Object createTest() throws Exception {
        Object test = super.createTest();
        // Note that JUnit4 will call this createTest() multiple times for each
        // test method, so we need to ensure to call "beforeClassSetup" only once.
        if (test instanceof InstanceTestClassListener && instanceSetupListener == null) {
            instanceSetupListener = (InstanceTestClassListener) test;
            instanceSetupListener.beforeClassSetup();
        }
        return test;
    }

    @Override
    public void run(RunNotifier notifier) {
        super.run(notifier);
        if (instanceSetupListener != null)
            instanceSetupListener.afterClassSetup();
    }
}

Now we are in business. If we run above test, it should give us similar result, but this time we are using instance methods instead!

An instance method setup before class.
Normal test method #1
Normal test method #2
An instance method setup after class.

A concrete use case: Working with Spring Test Framework

Now let me show you a real use case with above. If you use Spring Test Framework, you would normally setup a test like this so that you may have test fixture injected as member instance.

package deng.junitdemo.spring;

import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;

import java.util.List;

import javax.annotation.Resource;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration
public class SpringDemoTest {

    @Resource(name="myList")
    private List<String> myList;

    @Test
    public void testMyListInjection() {
        assertThat(myList.size(), is(2));
    }
}

You would also need a spring xml under that same package for above to run:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
     <bean id="myList" class="java.util.ArrayList">
        <constructor-arg>
            <list>
                <value>one</value>
                <value>two</value>
            </list>
        </constructor-arg>
     </bean>
</beans>

Pay very close attention to member instance List<String> myList. When running JUnit test, that field will be injected by Spring, and it can be used in any test method. However, if you ever want a one time setup of some code and get a reference to a Spring injected field, then you are in bad luck. This is because the JUnit @BeforeClass will force your method to be static; and if you make your field static, Spring injection won't work in your test!

Now if you are a frequent Spring user, you should know that Spring Test Framework already provided a way for you to handle this type of use case. Here is a way for you to do class level setup with Spring's style:

package deng.junitdemo.spring;

import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;

import java.util.List;

import javax.annotation.Resource;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.TestContext;
import org.springframework.test.context.TestExecutionListeners;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.support.AbstractTestExecutionListener;
import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;

@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners(listeners = {
        DependencyInjectionTestExecutionListener.class, 
        SpringDemo2Test.class})
@ContextConfiguration
public class SpringDemo2Test extends AbstractTestExecutionListener {

    @Resource(name="myList")
    private List<String> myList;

    @Test
    public void testMyListInjection() {
        assertThat(myList.size(), is(2));
    }

    @Override
    public void afterTestClass(TestContext testContext) {
        List<?> list = testContext.getApplicationContext().getBean("myList", List.class);
        assertThat((String)list.get(0), is("one"));
    }

    @Override
    public void beforeTestClass(TestContext testContext) {
        List<?> list = testContext.getApplicationContext().getBean("myList", List.class);
        assertThat((String)list.get(1), is("two"));
    }
}

As you can see, Spring offers the @TestExecutionListeners annotation to allow you to write any Listener, and in it you will have a reference to the TestContext which has the ApplicationContext for you to get to the injected field reference. This works, but I find it not very elegant. It forces you to look up the bean, while your injected field is already available as field. But you can't use it unless you go through the TestContext parameter.

Now if you mix the solution we provided in the beginning, we will see a more prettier test setup. Let's see it:

package deng.junitdemo.spring;

import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;

import java.util.List;

import javax.annotation.Resource;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.ContextConfiguration;

import deng.junitdemo.InstanceTestClassListener;

@RunWith(SpringInstanceTestClassRunner.class)
@ContextConfiguration
public class SpringDemo3Test implements InstanceTestClassListener {

    @Resource(name="myList")
    private List<String> myList;

    @Test
    public void testMyListInjection() {
        assertThat(myList.size(), is(2));
    }

    @Override
    public void beforeClassSetup() {
        assertThat((String)myList.get(0), is("one"));
    }

    @Override
    public void afterClassSetup() {
        assertThat((String)myList.get(1), is("two"));
    }
}

Now JUnit only allow you to use single Runner, so we must extends the Spring's version to insert what we did before.

package deng.junitdemo.spring;

import org.junit.runner.notification.RunNotifier;
import org.junit.runners.model.InitializationError;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import deng.junitdemo.InstanceTestClassListener;

public class SpringInstanceTestClassRunner extends SpringJUnit4ClassRunner {

    private InstanceTestClassListener instanceSetupListener;

    public SpringInstanceTestClassRunner(Class<?> clazz) throws InitializationError {
        super(clazz);
    }

    @Override
    protected Object createTest() throws Exception {
        Object test = super.createTest();
        // Note that JUnit4 will call this createTest() multiple times for each
        // test method, so we need to ensure to call "beforeClassSetup" only once.
        if (test instanceof InstanceTestClassListener && instanceSetupListener == null) {
            instanceSetupListener = (InstanceTestClassListener) test;
            instanceSetupListener.beforeClassSetup();
        }
        return test;
    }

    @Override
    public void run(RunNotifier notifier) {
        super.run(notifier);
        if (instanceSetupListener != null)
            instanceSetupListener.afterClassSetup();
    }
}

That should do the trick. Running the test will give use this output:

12:58:48 main INFO  org.springframework.test.context.support.AbstractContextLoader:139 | Detected default resource location "classpath:/deng/junitdemo/spring/SpringDemo3Test-context.xml" for test class [deng.junitdemo.spring.SpringDemo3Test].
12:58:48 main INFO  org.springframework.test.context.support.DelegatingSmartContextLoader:148 | GenericXmlContextLoader detected default locations for context configuration [ContextConfigurationAttributes@74b23210 declaringClass = 'deng.junitdemo.spring.SpringDemo3Test', locations = '{classpath:/deng/junitdemo/spring/SpringDemo3Test-context.xml}', classes = '{}', inheritLocations = true, contextLoaderClass = 'org.springframework.test.context.ContextLoader'].
12:58:48 main INFO  org.springframework.test.context.support.AnnotationConfigContextLoader:150 | Could not detect default configuration classes for test class [deng.junitdemo.spring.SpringDemo3Test]: SpringDemo3Test does not declare any static, non-private, non-final, inner classes annotated with @Configuration.
12:58:48 main INFO  org.springframework.test.context.TestContextManager:185 | @TestExecutionListeners is not present for class [class deng.junitdemo.spring.SpringDemo3Test]: using defaults.
12:58:48 main INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader:315 | Loading XML bean definitions from class path resource [deng/junitdemo/spring/SpringDemo3Test-context.xml]
12:58:48 main INFO  org.springframework.context.support.GenericApplicationContext:500 | Refreshing org.springframework.context.support.GenericApplicationContext@44c9d92c: startup date [Sat Sep 29 12:58:48 EDT 2012]; root of context hierarchy
12:58:49 main INFO  org.springframework.beans.factory.support.DefaultListableBeanFactory:581 | Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@73c6641: defining beans [myList,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy
12:58:49 Thread-1 INFO  org.springframework.context.support.GenericApplicationContext:1025 | Closing org.springframework.context.support.GenericApplicationContext@44c9d92c: startup date [Sat Sep 29 12:58:48 EDT 2012]; root of context hierarchy
12:58:49 Thread-1 INFO  org.springframework.beans.factory.support.DefaultListableBeanFactory:433 | Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@73c6641: defining beans [myList,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy

Obviously the output shows nothing interesting here, but the test should run with all assertion passed. The point is that now we have a more elegant way to invoking a before and after test setup that are at class level, and they can be instance methods to allow Spring injection.

Download the demo code

You may get above demo code in a working Maven project from my sandbox.

Wednesday, September 26, 2012

TimeMachine Scheduler 1.2.2 Release

This is a bug fix release for couple of issues:

 1. Fixed an issue with parsing release date string with "EDT" timezone that can cause the scheduler fail to startup

 2. Added the optional slf4j-log4j binding to view log output in war file.

You may download the latest here: https://bitbucket.org/timemachine/scheduler/downloads

Wednesday, September 19, 2012

Where did Git go?

My git is gone after I upgraded to latest MacOSX updates! What a annoying thing. Appearently it's in here now: /usr/local/git/bin/git You can edit your $PATH like it says here, or you could just relink it.
ln -s /usr/local/git/bin/git /usr/local/bin/git

Saturday, September 15, 2012

Building message based application using Camel

This is a long article that contains three separate topics:

  • Getting started with Apache Camel with Java
  • Improving startup of routes with a CamelRunner
  • Building message based application using Camel

But since I've prepared a camel-demo-1.0.0-SNAPSHOT-project.zip that has all these materials included, I thought it would easier to combine them and present it as whole.

Getting started with Apache Camel with Java

Trying out Camel with few Groovy lines is one thing, but getting a full scale project in Java is another matter. Today, I will show you how to get things started on Apache Camel with Maven based project. You may also use the provided camel-demo as project template to jump start your own Apache Camel project. You would just need to rename the Java package and rename the pom's group and artifact id's to match your need.

Preparing a Maven based project with Camel dependencies

Unzip the camel-demo project source, and you will see the basic directory layout.

camel-demo
    +- bin
    +- config
    +- data
    +- src
    +- pom.xml
    +- README.txt

What makes this demo a Camel based project is just the declaration in pom.xml. Let's take a look the file and its dependencies.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0   http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>deng.cameldemo</groupId>
    <artifactId>camel-demo</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <slf4j.version>1.6.6</slf4j.version>
        <camel.version>2.10.1</camel.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>project</descriptorRef>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>

        <!-- Unit testing lib -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit-dep</artifactId>
            <version>4.10</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-library</artifactId>
            <version>1.2.1</version>
            <scope>test</scope>
        </dependency>

        <!-- Logging lib -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>${slf4j.version}</version>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>

        <!-- Apache Commons lib -->
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.0.1</version>
        </dependency>

        <!-- Apache Camel -->
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-core</artifactId>
            <version>${camel.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-spring</artifactId>
            <version>${camel.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-groovy</artifactId>
            <version>${camel.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-jackson</artifactId>
            <version>${camel.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-mina</artifactId>
            <version>${camel.version}</version>
        </dependency>

    </dependencies>

</project>

This pom.xml decalares a Java based application and it will produce jar. It requires minimal of JDK 6 or higher. Besides the typical junit and hamcrest for unit testing, I also added slf4j for logging. I have added couple Apache's commons-lang/io to the project as well. I think these are basic settings that any Java based application should use them.

The maven-assembly-plugin I have declared is only for this demo packging purpose, and you may change or remove to suite your own project need.

For Camel dependencies, you would need minimal camel-core for routes building. And then you can add any additional components you plan to use in your project. I have added the following for building typical message based application development:

  1. The camel-spring - we want to have option to declare Camel routes in xml files as configuration. See camel-demo/config directory for samples.
  2. The camel-jackson - we want to process messaging data in our application as JSON format.
  3. The camel-mina - we want to send messaging data accross network through TCP socket.
  4. The camel-groovy - [optional] we want to be able to add dynamic scripting to route, even inside the xml config. This is great for debug and POC.

Note that since we use multiple camel components dependencies, I choose to set a Maven property ${camel.version} so that when we upgrade Camel, it's easier to maintain the pom.xml file in one place.

You should able to cd into the project directory and run mvn compile to verify that the project. It should compile without error.

Improving startup of routes with a CamelRunner

With the project pom.xml file ready, you can start creating Camel routes to handle your own business logics. Before we get too excited, let's try out a simple HelloRoute to see how it works and how we can run it first. Here is the route defnition code in src/main/java/deng/cameldemo/HelloRoute.java.

package deng.cameldemo;

import org.apache.camel.builder.RouteBuilder;

public class HelloRoute extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("timer://helloTimer?period=3000").
            to("log:" + getClass().getName());
    }
}

Take a test ride with the Camel

To see above in action, we need to add it into a CamelContext and start the context. For Java standalone program, we would write this setup code in a Main class. The Camel actually comes with a org.apache.camel.main.MainSupport abstract class that you may use to extend your own Main. However, I think it would be even nicer if Camel would provide a CamelRunner that can run like this.

$ java CamelRunner deng.cameldemo.HelloRoute

Such CamelRunner would be very user friendly and re-usable to have, so that's what I did. I wrote one like this:

package deng.cameldemo;

import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.support.FileSystemXmlApplicationContext;

/** 
 * A main program to start Camel and run as a server using RouteBuilder class names or 
 * Spring config files.
 * 
 * <p>Usage:
 * 
 * java deng.cameldemo.CamelRunner deng.cameldemo.HelloRoute
 * 
 * or
 * 
 * java -Dspring=true deng.cameldemo.CamelRunner /path/to/camel-spring.xml
 * 
 * @author Zemian Deng
 */
public class CamelRunner {
    public static void main(String[] args) throws Exception {
        CamelRunner runner = new CamelRunner();
        runner.run(args);
    }

    private static Logger logger = LoggerFactory.getLogger(CamelRunner.class);
    public void run(String[] args) throws Exception {
        if (Boolean.parseBoolean(System.getProperty("spring", "false")))
            runWithSpringConfig(args);
        else
            runWithCamelRoutes(args);

        // Wait for user to hit CRTL+C to stop the service
        synchronized(this) {
            this.wait();
        }
    }

    private void runWithSpringConfig(String[] args) {
        final ConfigurableApplicationContext springContext = new FileSystemXmlApplicationContext(args);

        // Register proper shutdown.
        Runtime.getRuntime().addShutdownHook(new Thread() { 
            @Override
            public void run() {
                try {
                    springContext.close();
                    logger.info("Spring stopped.");
                } catch (Exception e) {
                    logger.error("Failed to stop Spring.", e);
                }
            }
        });

        // Start spring
        logger.info("Spring started.");
    }

    private void runWithCamelRoutes(String[] args) throws Exception {
        final CamelContext camelContext = new DefaultCamelContext();        
        // Register proper shutdown.
        Runtime.getRuntime().addShutdownHook(new Thread() { 
            @Override
            public void run() {
                try {
                    camelContext.stop();
                    logger.info("Camel stopped for {}", camelContext);
                } catch (Exception e) {
                    logger.error("Failed to stop Camel.", e);
                }
            }
        });

        // Added RouteBuilder from args
        for (String className : args) {
            Class<?> cls = Class.forName(className);
            if (RouteBuilder.class.isAssignableFrom(cls)) {
                Object obj = cls.newInstance();
                RouteBuilder routeBuilder = (RouteBuilder)obj;
                camelContext.addRoutes(routeBuilder);
            } else {
                throw new RuntimeException("Unable to add Camel RouteBuilder " + className);
            }
        }

        // Start camel
        camelContext.start();
        logger.info("Camel started for {}", camelContext);
    }
}

To help you run the main class, I have provided a run-java wrapper script under the project's bin directory, so that you may quickly test it without having to setup classpath.

$ mvn package
$ bin/run-java deng.cameldemo.CamelRunner deng.cameldemo.HelloRoute

You will see that the program will load the HelloRoute in a DefaultCamelContext and start it as a server. The HelloRoute itself will generate a 3 seconds timer message and send it to a logger, which should be printing onto your console screen. This will continue forever until you hit CTRL+C to end it.

NOTE: You only have to invoke mvn package command once, so that it will package up all the dependencies jars in order for run-java to auto-detect them. If you are not going to use maven-assembly-plugin during package phase, then use mvn dependency:copy-dependencies command explicitly will work fine as well.

Take a test ride with the Camel, Part 2: running Camel with Spring xml configuration

The HelloRoute example above would simply provide route definition that formed by using component URI's. It will be nice if we can configure the route in a declarative manner so that we may change the route without re-compile a class file. This will be very handy especially if you are not familiar with each component's options and want to explore and try things out. Well, that's what the camel-spring is for. Beside giving you an option to load route in xml config file, it also provides a very flexible way to register custom services/processors bean in the Spring IoC container.

If you are a keen reader, you will notice in the CamelRunner code above that it has an extra runWithSpringConfig part. So the CamelRunner can actually bootstrap any Spring xml file and start a context as a server. You may use it like this:

$ bin/run-java deng.cameldemo.CamelRunner -Dspring=true config/hellocamel-spring.xml

The config/hellocamel-spring.xml is just an equivalent of our HelloRoute code but in Spring xml form:

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

    <camelContext id="helloCamel" xmlns="http://camel.apache.org/schema/spring">
        <route>
            <from uri="timer://jdkTimer?period=3000"/>
            <to uri="log://deng.cameldemo.HelloCamel"/>
        </route>
    </camelContext>

</beans>

This remove the need to compile/re-compile HelloRoute to define the Camel route to run.

Building message based application using Camel

To present you with a more practical demo, I would show you further on how to setup Camel to process message based application. In many IT shops, it's common that you would have a server to take message data as input and process them. A practical use case is to take any JSON formated message and transform it into object and process it. To do this in Camel, what you want to build is a route that will take input messages from a TCP port, and then process it in a pipeflow with any business logic you may have. You will run the route as a server, and then client may use any mean to submit the message to the TCP port. Client may even be another thin Camel client app to submit data as well. Let me show you how to get started.

Writing the server side code with Camel route

The server side would need a route to listen from a TCP port, and this is provided by camel-mina component. The first step is you need a route.

package deng.cameldemo;

import org.apache.camel.builder.RouteBuilder;

public class TcpMsgRoute extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        String port = System.getProperty("port", "12345");
        from("mina:tcp://localhost:" + port + "?sync=false").
            to("log:" + getClass().getName());
    }
}

Then the next step is ... done! No way, you mean that's all there to it for a server? Too good to be true? Well, let's try it out

$ bin/run-java deng.cameldemo.CamelRunner deng.cameldemo.TcpMsgRoute -Dport=12345
15:21:41 main INFO  org.apache.camel.impl.DefaultCamelContext:1391 | Apache Camel 2.10.1 (CamelContext: camel-1) is starting
15:21:41 main INFO  org.apache.camel.management.ManagementStrategyFactory:43 | JMX enabled.
15:21:42 main INFO  org.apache.camel.impl.converter.DefaultTypeConverter:45 | Loaded 172 type converters
15:21:42 main INFO  org.apache.camel.component.mina.MinaConsumer:59 | Binding to server address: localhost/127.0.0.1:12345 using acceptor: org.apache.mina.transport.socket.nio.SocketAcceptor@2ffad8fe
15:21:42 main INFO  org.apache.camel.impl.DefaultCamelContext:2045 | Route: route1 started and consuming from: Endpoint[mina://tcp://localhost:12345?sync=true]
15:21:42 main INFO  org.apache.camel.management.DefaultManagementLifecycleStrategy:859 | StatisticsLevel at All so enabling load performance statistics
15:21:42 main INFO  org.apache.camel.impl.DefaultCamelContext:1426 | Total 1 routes, of which 1 is started.
15:21:42 main INFO  org.apache.camel.impl.DefaultCamelContext:1427 | Apache Camel 2.10.1 (CamelContext: camel-1) started in 0.505 seconds
15:21:42 main INFO  deng.cameldemo.CamelRunner:93 | Camel started for CamelContext(camel-1)

Voila! The server is up and waiting for your users to send messages through port 12345. Not too bad for few lines of code.

Writing the client side code with Camel ProducerTemplate

Since our server expose a TCP port and take in any text content message, you can create any client that's capable writing to a TCP socket. In here, I will show you how to use Camel to write a thin client.

package deng.cameldemo.client;

import java.io.FileReader;
import org.apache.camel.CamelContext;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.impl.DefaultCamelContext;
import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class TcpMsgSender {
    public static void main(String[] args) throws Exception {
        TcpMsgSender runner = new TcpMsgSender();
        runner.run(args);
    }

    private static Logger logger = LoggerFactory.getLogger(TcpMsgSender.class);
    public void run(String[] args) throws Exception {
        String fileName = args.length > 0 ? args[0] : "data/msg.txt";
        String[] hostPort = (args.length > 1 ? args[1] : "localhost:12345").split(":");
        String host = hostPort[0];
        String port = hostPort.length > 1 ? hostPort[1] : "12345";
        logger.info("Sending tcp message {} to host={}, port={}", new Object[]{ fileName, host, port});

        String text = IOUtils.toString(new FileReader(fileName));
        logger.debug("File size={}", text.length());

        CamelContext camelContext = new DefaultCamelContext();
        ProducerTemplate producer = camelContext.createProducerTemplate();
        producer.sendBody("mina:tcp://" + host + ":" + port + "?sync=false", text);
        logger.info("Message sent.");
    }
}

This TcpMsgSender can send any text file to your server endpoint. Try this out while your server is running:

$ bin/run-java deng.cameldemo.client.TcpMsgSender data/test-msg.json localhost:12345
15:22:35 main INFO  deng.cameldemo.client.TcpMsgSender:24 | Sending tcp message data/test-msg.json to host=localhost, port=12345
15:22:35 main DEBUG deng.cameldemo.client.TcpMsgSender:27 | File size=47
15:22:35 main INFO  org.apache.camel.impl.converter.DefaultTypeConverter:45 | Loaded 172 type converters
15:22:35 main INFO  org.apache.camel.management.ManagementStrategyFactory:43 | JMX enabled.
15:22:35 main INFO  deng.cameldemo.client.TcpMsgSender:32 | Message sent.

You should able to verify from your server console output that it received the msg. The msg I sent is in data/test-msg.json, which contains this simple text:

{ "firstName" : "Zemian", "lastName" : "Deng" }

Note that our server simply receive plain text and log it. We will discuss how to process the message next.

Processing message data in JSON format with Camel and Spring xml config

You thought the server code was easy from above, guess again. You can actually replace the TcpMsgRoute with just some simple xml lines!

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

    <camelContext id="tcpMsgServer" xmlns="http://camel.apache.org/schema/spring">
        <route>
            <from uri="mina:tcp://localhost:12345?sync=false"/>
            <to uri="log://deng.cameldemo.TcpMsgServer"/>
        </route>
    </camelContext>

</beans>

Save it as config/tcpmsgserver-spring.xml. Then re-run the server, and you should get the same result as above.

$ bin/run-java deng.cameldemo.CamelRunner -Dspring=true config/tcpmsgserver-spring.xml

Now let us improve the above xml to further process the JSON message data. We will like to transform the plain text to a Java object then process by a custom bean. To do that, we first would need to add unmarshal component to the route. This is where the camel-jackson comes into play. In our demo, the unmarshalling step would convert the JSON text into a java.util.Map and then pass it to a processor bean named myMsgProcessor. Let's create a new xml file named config/tcpmsgserver-json-spring.xml as follow.

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">

    <camelContext id="tcpMsgServer" xmlns="http://camel.apache.org/schema/spring">
        <route>
            <from uri="mina:tcp://localhost:12345?sync=false"/>
            <to uri="log://deng.cameldemo.TcpMsgServer"/>
            <unmarshal>
                <json library="Jackson"/>
            </unmarshal>
            <to uri="bean:myMsgProcessor?method=process"/>
        </route>
    </camelContext>

    <bean id="myMsgProcessor" class="deng.cameldemo.MyMsgProcessor">
    </bean>

</beans>

The myMsgProcessor is an Spring bean that we provide custom logic code to process the data. At this point we have a full Java object to manipulate. The content of the processor can be any POJO with the method name specified in the URI. Here is an example one:

package deng.cameldemo;

import org.apache.camel.builder.RouteBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Map;

public class MyMsgProcessor {
    private static Logger logger = LoggerFactory.getLogger(MyMsgProcessor.class);
    public void process(Map<String, String> data) {
        logger.info("We should slice and dice the data: " + data);
    }
}

Try re-run the server with the new xml file above, and you should able to re-invoke the same client command to test it out. Here is a sample output of my server:

$ bin/run-java deng.cameldemo.CamelRunner -Dspring=true config/tcpmsgserver-json-spring.xml
17:05:25 main INFO  org.springframework.context.support.FileSystemXmlApplicationContext:456 | Refreshing org.springframework.context.support.FileSystemXmlApplicationContext@4200309: startup date [Sat Sep 15 17:05:25 EDT 2012]; root of context hierarchy
17:05:25 main INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader:315 | Loading XML bean definitions from file [/Users/zemian/projects/sandbox/camel-demo/config/tcpmsgserver-json-spring.xml]
17:05:27 main INFO  org.springframework.beans.factory.support.DefaultListableBeanFactory:557 | Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@27b75165: defining beans [template,consumerTemplate,tcpMsgServer:beanPostProcessor,tcpMsgServer,myMsgProcessor]; root of factory hierarchy
17:05:27 main INFO  org.apache.camel.spring.SpringCamelContext:1391 | Apache Camel 2.10.1 (CamelContext: tcpMsgServer) is starting
17:05:27 main INFO  org.apache.camel.management.ManagementStrategyFactory:43 | JMX enabled.
17:05:27 main INFO  org.apache.camel.impl.converter.DefaultTypeConverter:45 | Loaded 172 type converters
17:05:28 main INFO  org.apache.camel.component.mina.MinaConsumer:59 | Binding to server address: localhost/127.0.0.1:12345 using acceptor: org.apache.mina.transport.socket.nio.SocketAcceptor@5a3cae4a
17:05:28 main INFO  org.apache.camel.spring.SpringCamelContext:2045 | Route: route1 started and consuming from: Endpoint[mina://tcp://localhost:12345?sync=false]
17:05:28 main INFO  org.apache.camel.management.DefaultManagementLifecycleStrategy:859 | StatisticsLevel at All so enabling load performance statistics
17:05:28 main INFO  org.apache.camel.spring.SpringCamelContext:1426 | Total 1 routes, of which 1 is started.
17:05:28 main INFO  org.apache.camel.spring.SpringCamelContext:1427 | Apache Camel 2.10.1 (CamelContext: tcpMsgServer) started in 0.695 seconds
17:05:28 main INFO  deng.cameldemo.CamelRunner:61 | Spring started.
17:05:35 Camel (tcpMsgServer) thread #3 - MinaThreadPool INFO  deng.cameldemo.TcpMsgServer:96 | Exchange[ExchangePattern:InOnly, BodyType:String, Body:{ "firstName" : "Zemian", "lastName" : "Deng" }]
17:05:35 Camel (tcpMsgServer) thread #3 - MinaThreadPool INFO  deng.cameldemo.MyMsgProcessor:11 | We should slice and dice the data: {lastName=Deng, firstName=Zemian}

Pay attention that Camel will auto convert the data format in your route! Our client only sends the plain text as JSON format, but when server receives it, it unmarshals it using Jackson library, and then converts it into a java Map object. It then passes the map object into our processor bean. Also, in this demo, I choose to use a generic java.util.Map as processor method argument (which is output of the JSON unmarshal), but you can easily define your own business data type, such as MyCustomerData. This reveals the power of Camel, since you don't need to push the message in your flow, but only worry about writing your "processor" as a POJO. The Camel will "glue" components together to form a route and carry the message data through the pipeline flow.

On the same token, when you write your business logic in one or more processors, it's a good idea that you limit your POJO logic to be as small unit as possible. When you do this, then you can maximize the reusability of the processors. The bigger POJO you make, with many business logics mixed in, it will also make it difficult to test. So I recommend you when developing these processor beans, try to think them as LEGO pieces -- small POJO. You want to let Camel define the route and glue the LEGO pieces togther. Once you get into this habit of thiking, then you can use Camel in a more effectively way to solve many of your domain problems.

Well, that's all for today folks. I hope you enjoyed the Camel ride. Happy programming!