THIS BLOG HAS MOVED

Please follow my new blog here: http://ttddyy.github.io/

released datasource-proxy v1.3

I have released datasource-proxy version 1.3

Changes:

• new JNDI support
• JDBC4.2 (Java8) support
• fluent API builder for ProxyDataSource - “ProxyDataSourceBuilder”
• new Listeners for writing logs to System.out
• updated log format
• etc.

Please see more details in CHANGELOG

Available in maven central

dependency:
 project homepage: https://github.com/ttddyy/datasource-proxy

Spring Boot: External Config File for Container Deployed War

Spring Boot resolves external configuration file such as "application.properties" or "application.yml" in following order:

1. classpath root
2. current directory
3. classpath /config package
4. /config subdir of the current directory.

[reference spring-boot README]

This is usually good for running spring boot application with executable jar (or war).

$ java -jar my-application.jar

However, when you convert your application to build a war file which gets deployed to a servlet container and want to keep the configuration file outside of the war file, the default search path doesn't work. 

One solution for this is that you can specify the external config file location in your servlet container's "spring.config.location" system property.

For example, in tomcat:

"CATALINA_HOME/bin/setenv.sh"

export CATALINA_OPTS="-Dspring.config.location=file:/usr/local/myapp/application.properties"

* The value of system property is evaluated as spring resource style format.
So, if you are specifying a file outside of classpath, you need to put "file:" prefix.

See more detail on org.springframework.boot.context.initializer.ConfigFileApplicationContextInitializer

Library to make null-safe collections in thrift generated java object.

While I was experimenting evernote java API, I found thrift generated code for container types(list,set,map) are not really java friendly, especially for for-loop.
In thrift, when value is set to null, the transport will not send the field. But in java, when collection is null, the enhanced-for-loop throws NullPointerException.

Note note = noteStore.getNote(...);
// This throws NPE when the note doesn't have any resources
for (Resource resource : note.getResources()) {
...
}

I can put if-statement to check the collection whenever I need to access it, but that's just repetitive...

So, I wrote a util class that wraps thrift generated class to make it null-safe.

* w() is a static method in ThriftWrapper class which I wrote.

Note wrapped = w(new Note());  // wrapped
for (Resource resource : wrapped.getResources()) {
  // no more NPE. nested thrift attributes are also null-safe
}

Mechanism:

The wrapping method introspect the given thrift object and set empty mutable collections(ArrayList, HashSet, HashMap) when it finds null in collection fields. Also, it traverse child thrift classes and set empty collections as well.(foo.getBar().getBaz().getList()).

Then it returns cglib generated proxy. Since in thrift, null has a meaning when it serializes object, the proxy keeps track of collection fields which were initially null, and when "write()" method is called, it compares the current value and if it's still empty, then it will omit the field to transport.

wrapping behavior:

Note note = new Note();
note.setAttributes(new NoteAttributes());
assertThat(note.getTagGuids(), is(nullValue()));
assertThat(note.getAttributes().getClassifications(), is(nullValue()));

Note wrapped = w(note);
assertThat(wrapped.getTagGuids(), is(emptyCollectionOf(String.class)));
assertThat(wrapped.getAttributes().getClassifications(), is(notNullValue()));
assertThat(wrapped.getAttributes().getClassifications().size(), is(0));

transport behavior:

Note note = new Note();
Note wrapped = w(note);  // still a note instance  (actually a cglib proxy)
wrapped.getTagGuids().size();  // 0 (empty list)
wrapped.getResources().add(...);  // add something to collection

noteStore.createNote(wrapped);  // internally call wrapped.write(...)
// added resources are included in message, but tag-guids are not

Please reference more detailed behavior in test classes.

Code:

I pushed to my github repo.

Further:

Since I'm new to thrift, it might not be a good approach or there might be a better solution already.

If there is more request for this approach, I'll further enhance the project such:

• inline cglib to avoid dependency conflict
• make the class more spring-framework friendly bean instead of static methods
• release and push to public maven repo
• etc.

Spring Boot Actuator: How to change the endpoint base url

By default, actuator management endpoints are mapped to the top level url: "/info", "/metrics", "/beans", etc.

That may be good. But probably it would be better to have some base path. For example, "/admin/metrics" or "/manage/info".
Spring Boot provides quick and easy solution.

Configuration

In your configuration resource file (application.properties or application.yml): 

management.contextPath: /admin

Now, all the management endpoints are mapped under "/admin" base path.

"/admin/metrics", "/admin/info", "/admin/health"...

Implementation Classes

In spring boot actuator source code:

• org.springframework.boot.actuate.properties.ManagementServerProperties
• property values representation class
• org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration
• @Configuration which uses above ManagementServerProperties

How spring-boot enables actuator by adding jar dependency?

I was reading spring-boot source code, and wondered how it is auto-detecting actuator features just by adding jar dependency.

spring-boot-actuator

"spring-boot-actuator" adds management-endpoints(http) to the spring-boot application just by adding the jar file.
endpoints are metrics, health-check, spring-security auth audit, etc.

please see here for feature details.

To add actuator to your spring-boot application, add this dependencies:

    org.springframework.boot
    spring-boot-starter-actuator


    org.springframework.boot
    spring-boot-starter-web

Mechanism of auto-detecting spring-boot-actuator

So, how spring-boot application detects spring-boot-actuator?

Sample spring-boot application:

@Configuration
@EnableAutoConfiguration
@ComponentScan
public class MyApplication {

    public static void main(String[] args) {
        SpringApplication.run(MyApplication.class, args);
    }
}

"org.springframework.boot.SpringApplication" is the starting point for spring-boot application.

Auto-detection Steps

1. During "SpringApplication#run" method, it creates an application context from the given source classes("@Configuration" annotated classes). In this case "MyApplication" class.
2. When "MyApplication" class is being processed as a spring's Java-configuration, "@EnableAutoConfiguration" annotation gets interpreted. "@EnableAutoConfiguration" is a spring-boot(autoconfigure) annotation which is handled by "org.springframework.boot.autoconfigure.EnableAutoConfigurationImportSelector". (how spring handles @Enable* annotation is well documented in this blog post.)
3. In "EnableAutoConfigurationImportSelector", it uses "org.springframework.core.io.support.SpringFactoriesLoader#loadFactoryNames" from spring-core to load configurations whose key is  "org.springframework.boot.autoconfigure.EnableAutoConfiguration".
   This method reads "META-INF/spring.factories" from jar files.(multiple jar files can have "spring.factories" and when they have same key, comma delimited values will be merged.)
   spring-boot-actuator contains "META-INF/spring.factories" file. The value of the file is a comma delimited list of "@Configuration" classes under "org.springframework.boot.actuate.autoconfigure" package, which are actuator bean definition classes.

   org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
   org.springframework.boot.actuate.autoconfigure.AuditAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.EndpointAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.ErrorMvcAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.ManagementServerPropertiesAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.MetricRepositoryAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.SecurityAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.TraceRepositoryAutoConfiguration,\
   org.springframework.boot.actuate.autoconfigure.TraceWebFilterAutoConfiguration

4. These configuration classes will be imported into the main application-context, so that, spring-boot-actuator beans become available in the application.

Same mechanism is used for "spring-boot-autoconfigure". 

Its jar file contains "spring.factories" file.

Writing a Custom Spring-Boot Module

In sum, you can write a spring-boot module which will be auto-detected by spring-boot application based on the presence of the jar file:

• include "META-INF/spring.factories" file
• key: "org.springframework.boot.autoconfigure.EnableAutoConfiguration"
• value: comma delimited path to @Configuration file(s) for your module

released datasource-proxy v1.2

I released datasource-proxy version 1.2

main feature:

• query and parameter replacement
• new QueryTransformer and ParameterTransformer APIs to transform query and parameter before executing queries
• many refactoring

They are available in maven central.

dependency:

 project homepage: https://github.com/ttddyy/datasource-proxy

display ec2-name and ip-address as ssh-config or hosts file

Inspired by this instagram engineering blog, I wrote a python script to printout ec2 hostname and ip-address as ssh-config or hosts file format.

In our environment, unfortunately we don't have dns for ec2 hosts, for now. 

I use /etc/hosts. Some people use ssh-config file.

This script may help others who are in similar environment where using ssh-config or hosts file to manage ec2-host mapping.

Usage

ec2-printconfig script

util method that runs TestNG test classes and verify the result

I wrote a utility class for my project.

The method does:

• run TestNG tests programmatically
• verify the failures happened in TestNG tests run
• If there is a failure, combine all assertion messages including stack trace, then make a single AssertionError

(I'm also using google-guava)

run jetty on dependency resolved war in gradle

Background:

• test classes(such as selenium tests) are in an independent module(maven project, etc.).
• have an external dependency(mave, ivy, etc) to war file(s)
• want to run jetty on dependent war file(s)

Gradle can start up jetty on dependency resolved war file(s).

When you want to run your tests after jetty started, set "daemon=true" and make the test task depends on your new run-war task.

run multiple jetty in gradle

Background:
I have a selenium test suite for end-to-end web testing.
Currently, it is running on jenkins box but it is taking so long to finish all the tests.
So, next step is to parallelize the selenium tests.
As a first step, I need to start multiple jetty servers using different port.

With Gradle jetty plugin, I camp up how to run the multiple jetty servers before tests.

run same command on multiple directories

I've been working on applications that consist of multiple maven projects.

Many time, I have to type same commands on multiple project directories, such as "mvn install", "svn up", "git …", etc.

I wrote a bash script that executes given command(s) on multiple directories.

Multiple Directory Command:

  https://github.com/ttddyy/utils/tree/master/multi_command

Sample: (mc is the script alias I use)

  > mc svn up                            # svn up all projects

  > mc -g projA mvn install     # call "mvn install" on porjA group directories(defined in config file)

  > mc git checkout -b new_branch master  # create a new git branch to all project

  > mc git branch                      # display current git branch in all project

  > mc -g projA svn up             #  "svn up" projA directories

  > mc -g projB -c config_file du -sh  # chcek directory size of projB directories

please see more options here.

datasource-proxy framework

--
I'm writing a framework that helps monitoring and debugging query execution from app.

[Project Home]

Features:
- log all database queries with parameter values
- log each query's elapsed time
- log statistics of all database call and total query elapsed time
- provide callback to database call
- provide statistics object

Database Call Log:

Time:10, Num:1, Query:{[insert into emp ( id, name )values (?, ?);][1, foo]}
Time:1, Num:1, Query:{[select this_.id as id0_0_, this_.name as name0_0_, this_.value as value0_0_ from emp this_ where (this_.id=? and this_.name=?)][1,bar]}

Statistics Log:

DataSource:MyDatasourceA ElapsedTime:13 Call:7 Query:7 (Select:3 Insert:2 Update:1 Delete:0 Other:1)
DataSource:MyDatasourceB ElapsedTime:1 Call:1 Query:1 (Select:1 Insert:0 Update:0 Delete:0 Other:0)

If you are interested,

• project homepage http://code.google.com/p/datasource-proxy/
• download http://code.google.com/p/datasource-proxy/downloads/list

Hibernate: Association Mappings in Annotation (JPA style)

Unidirectional Association

• many to one (8.2.1) [ref]
• one to one (foreign key) (8.2.2) [ref]
• one to one (primary key) (8.2.2) [ref]
• one to many (foreign key) (8.2.3) [ref]

Unidirectional Association with Join Table

• one to many (8.3.1) [ref]
• many to one (8.3.2) [ref]
• one to one (8.3.3) [ref]
• many to many (8.3.4) [ref]

Bidirectional Association

• one to many / many to one (8.4.1) [ref]
• one to one (foreign key) (8.4.2) [ref]
• one to one (primary key) (8.4.2) [ref]

Bidirectional Association with Join Table

• one to many / many to one (8.5.1) [ref]
• one to one (8.5.2) [ref]
• many to many (8.5.3) [ref]

(*) hibernate document chapter

Reference:
- hibernate documentation
- EJB 3.0 Spec

Hibernate: Annotation many-to-many (join table)

Bidirectonal many-to-many (join table) association

Hibernate Doc (Chap 8.5.3)


::Relationship::
person(many) <-> address(many)

::DB Schema::
person(personId)
address(addressId)
personaddress(personId, addressId)

::Java Operation::
person.getAddresses();
address.getPeople();

::Annotation::

@Entity
@Table(name = "PERSON")
public class Person {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "personId")
  private int id;

  // mapping owner
  @ManyToMany
  @JoinTable(name = "PersonAddress",
    joinColumns = {
      @JoinColumn(name="personId", unique = true)           
    },
    inverseJoinColumns = {
      @JoinColumn(name="addressId")
    }
  )
  private Set<Address> addresses;
}

@Entity
@Table(name = "ADDRESS")
public class Address {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "addressId")
  private int id;

  @ManyToMany(mappedBy="addresses")  // map info is in person class
  private Set<Person> people;
}

::Generated SQL::
- person.getAddresses();
select addresses0_.personId as personId1_, addresses0_.addressId as addressId1_, address1_.addressId as addressId3_0_ from PersonAddress addresses0_ left outer join ADDRESS address1_ on addresses0_.addressId=address1_.addressId where addresses0_.personId=?

- address.getPeople();
select people0_.addressId as addressId1_, people0_.personId as personId1_, person1_.personId as personId2_0_ from PersonAddress people0_ left outer join PERSON person1_ on people0_.personId=person1_.personId where people0_.addressId=?

[Association Mapping List]

Hibernate: Annotation one-to-one (join table)

Bidirectonal one-to-one (join table) association (very unusual)

Hibernate Doc (Chap 8.5.2)


::Relationship::
person(one)(mapping owner) <-> address(one)

::DB Schema::
person(personId)
address(addressId)
personaddress(personId, addressId)

::Java Operation::
person.getAddress();
address.getPerson();

::Annotation::

@Entity
@Table(name = "PERSON")
public class Person {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "personId")
  private int id;

  @OneToOne(optional=true)
  @JoinTable(name="PersonAddress",
    joinColumns = {
      @JoinColumn(name="personId", unique = true)           
    },
    inverseJoinColumns = {
      @JoinColumn(name="addressId")
    }     
  )
  private Address address;
}

@Entity
@Table(name = "ADDRESS")
public class Address {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "addressId")
  private int id;

  @OneToOne(optional=true, mappedBy="address") // pointing to Person's address field
  private Person person;   
}

::Generated SQL::
- person.getAddress();
select person0_.personId as personId2_2_, person0_1_.addressId as addressId3_2_, address1_.addressId as addressId4_0_, address1_1_.personId as personId3_0_, person2_.personId as personId2_1_, person2_1_.addressId as addressId3_1_ from PERSON person0_ left outer join PersonAddress person0_1_ on person0_.personId=person0_1_.personId left outer join ADDRESS address1_ on person0_1_.addressId=address1_.addressId left outer join PersonAddress address1_1_ on address1_.addressId=address1_1_.addressId left outer join PERSON person2_ on address1_1_.personId=person2_.personId left outer join PersonAddress person2_1_ on person2_.personId=person2_1_.personId where person0_.personId=?

- address.getPerson();
select address0_.addressId as addressId4_2_, address0_1_.personId as personId3_2_, person1_.personId as personId2_0_, person1_1_.addressId as addressId3_0_, address2_.addressId as addressId4_1_, address2_1_.personId as personId3_1_ from ADDRESS address0_ left outer join PersonAddress address0_1_ on address0_.addressId=address0_1_.addressId inner join PERSON person1_ on address0_1_.personId=person1_.personId left outer join PersonAddress person1_1_ on person1_.personId=person1_1_.personId left outer join ADDRESS address2_ on person1_1_.addressId=address2_.addressId left outer join PersonAddress address2_1_ on address2_.addressId=address2_1_.addressId where address0_.addressId=?

[Association Mapping List]

Hibernate: Annotation one-to-many/many-to-one (join table)

Bidirectonal one-to-many/many-to-one (join table) association

Hibernate Doc (Chap 8.5.1)


::Relationship::
person(one) <-> address(one)


::DB Schema::
person(personId)
personAddress(personId, addressId)
address(addressId)

::Java Operation::
person.getAddress();
address.getPerson();

::Annotation::

@Entity
@Table(name = "PERSON")
public class Person {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "personId")
  private int id;

  @OneToMany
  @JoinTable(name = "PersonAddress",
    joinColumns = {
      @JoinColumn(name="personId", unique = true)           
    },
    inverseJoinColumns = {
      @JoinColumn(name="addressId")
    }
  )
  private Set<Address> addresses;
}

@Entity
@Table(name = "ADDRESS")
public class Address {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "addressId")
  private int id;

  @ManyToOne(optional=true)
  @JoinTable(name = "PersonAddress",
    joinColumns = {
      @JoinColumn(name="addressId")
    },
    inverseJoinColumns = {
      @JoinColumn(name="personId")
    }
  )
  private Person person;   
}

::Generated SQL::
- person.getAddresses();
select addresses0_.personId as personId2_, addresses0_.addressId as addressId2_, address1_.addressId as addressId3_0_, address1_1_.personId as personId4_0_, person2_.personId as personId2_1_ from PersonAddress addresses0_ left outer join ADDRESS address1_ on addresses0_.addressId=address1_.addressId left outer join PersonAddress address1_1_ on address1_.addressId=address1_1_.addressId left outer join PERSON person2_ on address1_1_.personId=person2_.personId where addresses0_.personId=?

- address.getPerson();
select address0_.addressId as addressId3_1_, address0_1_.personId as personId4_1_, person1_.personId as personId2_0_ from ADDRESS address0_ left outer join PersonAddress address0_1_ on address0_.addressId=address0_1_.addressId left outer join PERSON person1_ on address0_1_.personId=person1_.personId where address0_.addressId=?

[Association Mapping List]

Hibernate: Annotation one-to-one (primary-key)

Bidirectonal one-to-one (primary-key) association

Hibernate Doc (Chap 8.4.2)


::Relationship::
person(one) <-> address(one)


::DB Schema::
person(personId)
address(personId)


::Java Operation::
person.getAddress();
address.getPerson();


::Annotation::

@Entity
@Table(name="PERSON")
public class Person {
  
  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name="personId")
  private int id;
 
  @OneToOne
  @PrimaryKeyJoinColumn
  private Address address;
}

@Entity
@Table(name = "ADDRESS")
public class Address {

  @Id
  @Column(name = "personId")
  private int id;

  @OneToOne(mappedBy="address")  // inverse=true, pointnig Person's address field
  private Person person;   
}

::Generated SQL::
- person.getAddress();
select person0_.personId as personId2_2_, address1_.personId as personId3_0_, person2_.personId as personId2_1_ from PERSON person0_ left outer join ADDRESS address1_ on person0_.personId=address1_.personId left outer join PERSON person2_ on address1_.personId=person2_.personId where person0_.personId=?

- address.getPerson();
select address0_.personId as personId3_2_, person1_.personId as personId2_0_, address2_.personId as personId3_1_ from ADDRESS address0_ left outer join PERSON person1_ on address0_.personId=person1_.personId left outer join ADDRESS address2_ on person1_.personId=address2_.personId where address0_.personId=?

[Association Mapping List]

Hibernate: Annotation one-to-one (foreign-key)

Bidirectonal one-to-one (foreign-key) association

Hibernate Doc (Chap 8.4.2)


::Relationship::
person(one) <-> address(one)

::DB Schema::
person(id, addressId)
address(id)

::Java Operation::
person.getAddress();
address.getPerson();

::Annotation::

@Entity
@Table(name="PERSON")
public class Person {
   
  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name="personId")
  private int id;
 
  @ManyToOne
  @JoinColumn(name="addressId")     // inverse = false
  private Address address;
}

@Entity
@Table(name = "ADDRESS")
public class Address {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "addressId")
  private int id;

  @OneToOne(mappedBy="address")  // inverse=true, pointnig Person's address field
  private Person person;   
}

::Generated SQL::
- person.getAddress();
select person0_.personId as personId2_1_, person0_.addressId as addressId2_1_, address1_.addressId as addressId3_0_ from PERSON person0_ left outer join ADDRESS address1_ on person0_.addressId=address1_.addressId where person0_.personId=?

- address.getPerson();
select person0_.personId as personId2_1_, person0_.addressId as addressId2_1_, address1_.addressId as addressId3_0_ from PERSON person0_ left outer join ADDRESS address1_ on person0_.addressId=address1_.addressId where person0_.addressId=?

[Association Mapping List]

Hibernate: Annotation one-to-many/many-to-one

Bidirectonal one-to-many/many-to-one association

Hibernate Doc (Chap 8.4.1)

::Relationship::
person(many) <-> address(one)
- person A and person B live in address AA

::DB Schema::
person(personId, addressId)
address(addressId)


::Java Operation::
person.getAddress();
address.getPeople();

::Annotation::

@Entity
@Table(name="PERSON")
public class Person {
   
  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name="personId")
  private int id;
   
  @ManyToOne
  @JoinColumn(name="addressId")     // inverse = false
  private Address address;
}

@Entity
@Table(name = "ADDRESS")
public class Address {

  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "addressId")
  private int id;

  @OneToMany(mappedBy="address")  // pointing Person's address field
  @Column(name="personId")    // inverse=true
  private Set<Person> people;
}

::Generated SQL::

[Association Mapping List]