Tuesday 16 September 2014

Agile is a simple topic

Agile manifesto is probably one of the best ever written manifestos in software development if not the best. Simple and elegant. Good vs Bad 1 2 3 4, done. It is so simple that I am constantly disappointed by the amount of stuff that floating on the Internet about, what is agile what is not, how to do agile, Scrum, Kanban and who knows what will pop up next year claiming to to be another king of agile.

If I ever tell you we are the purist agile team and we don't have sprint, we don't have stand up meetings, we don't story board, we don't have burn down charts, we don't have planning poker cards, we don't have any of the buzzwords, most of the so called IT consultants will hang me on the spot.

Let's face it, being pure isn't about what you have, it is about what you don't! The pure gold has nothing but gold that's why it is super valuable. We should build our teams on developers, codes and business needs. The three pure ingredient of a team, any one taken away a team is no more.

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. 
Antoine de Saint-Exupery

Exactly the manifesto is saying "we value less on processes and tools" and yet we have seen all kinds of weird super imposed processes and tools everywhere. "Look, we have standups, we have sprints, we have story boards therefore we agile". NO, absolutely NOT. You can walk like a duck, quack like duck, but you are still not a duck.

But why the hype anyway?

Partly the consulting companies are to be blamed, they try to sell the buzz words to the management so that they can make $$$ by simply asking the developers to do what they already know, writing codes, but in a different way.

The biggest enemies are all the developers especially the team leaders and managers. Because they are lazy to know the developers (the people), lazy to learn the codes (the working software) the lazy to analyse the business needs. Because "in the end of the day I need to show my developers that I am doing a manager's work", "what is the shortcut?", "look, I just got this scrum from a random blog post, standups 5 mins, no problems. Poker cards, easy. Story boards, no big deal ... ". "Done, now we are scrum, now we are agile, if the things fail, it is the developers problem". Goodbye, there goes a team.

So now you question me, "you said agile is simple, why it looks so hard now?"
Any fool can make something complicated. It takes a genius to make it simple.
Woody Guthrie
People are born equal, a genius doesn't magically popup, it takes real hard work to reach that level. Let's go back to the origin, the mighty manifesto.

Get rid of all unnecessary processes and tools, and go talk to people. "What is Jimmy's strength? What can we do to make up for Sam's weakness? Is David and Carl a good pair?".

Stop typing inside Words or Excel, go read the real codes, "What can we do the enhance the clarity of the codes, how to improve the performance without too much sacrifice, what are the alternative ways to extend our software".

Stop coming up with imaginary use cases, go meet the customer "What are your point points, what are the 3 most important features that need to be enhanced and delivered. Based on our statistics, we believe if we build feature X in such a way, the business can grow in Y%, do you think we should do this?"

Stop wasting our life on keeping a useless backlog, go see the 3 biggest opportunities and threats and work on them, rise and repeat. If fact that is exactly how evolution bring our human to this stage, "eliminate the immediate threat to ensure the short term survival, and seek the opportunities for long term growth". As we all decadents of the mother nature, we are incapable of out smart her, so learn from her.

real process/methodology grows from the team not super imposed on to the team

real process/methodology does not have a name because it is unique to each team

Grow your own dream team!



Thanks for wasting your time reading my rant

Sunday 7 September 2014

Stateless Session for multi-tenant application using Spring Security

Once upon a time, I published one article explaining the principle to build Stateless Session. Coincidentally, we are working on the same task again, but this time, for a multi-tenant application. This time, instead of building the authentication mechanism ourselves, we integrate our solution into Spring Security framework.

This article will explain our approach and implementation.





Business Requirement

We need to build authentication mechanism for an Saas application. Each customer access the application through a dedicated sub-domain. Because the application will be deployed on the cloud, it is pretty obvious that Stateless Session is the preferred choice because it allow us to deploy additional instances without hassle.

In the project glossary, each customer is one site. Each application is one app. For example, site may be Microsoft or Google. App may be Gmail, GooglePlus or Google Drive. A sub-domain that user use to access the application will include both app and site. For example, it may looks like microsoft.mail.somedomain.com or google.map.somedomain.com

User once login to one app, can access any other apps as long as they are for the same site. Session will be timeout after a certain inactive period.

Background

Stateless Session

Stateless application with timeout is nothing new. Play framework has been stateless from the first release in 2007. We also switched to Stateless Session many years ago. The benefit is pretty clear. Your Load Balancer do not need stickiness; hence, it is easier to configure. As the session in on the browser, we can simply bring in new servers to boost capacity immediately. However, the disadvantage is that your session is not so big and not so confidential anymore.

Comparing to stateful application where the session is store in server, stateless application store the session in HTTP cookie, which can not grow more than 4KB. Moreover, as it is cookie, it is recommended that developers only store text or digit on the session rather than complicated data structure. The session is stored in browser and transfer to server in every single request. Therefore, we should keep the session as small as possible and avoid placing any confidential data on it. To put it short, stateless session force developer to change the way application using session. It should be user identity rather than convenient store.

Security Framework

The idea behind Security Framework is pretty simple, it helps to identify the principle that executing code, checking if he has permission to execute some services and throws exceptions if user does not. In term of implementation, security framework integrate with your service in an AOP style architecture. Every check will be done by the framework before method call. The mechanism for implementing permission check may be filter or proxy.

Normally, security framework will store principal information in the thread storage (ThreadLocal in Java). That why it can give developers a static method access to the principal anytime. I think this is somethings developers should know well; otherwise, they may implement permission check or getting principal in some background jobs that running in separate threads. In this situation, it is obviously that the security framework will not be able to find the principal.

Single Sign On

Single Sign On in mostly implemented using Authentication Server. It is independent of the mechanism to implement session (stateless or stateful). Each application still maintain their own session. On the first access to an application, it will contact authentication server to authenticate user then create its own session.

Food for Thought

Framework or build from scratch

As stateless session is the standard, the biggest concern for us is to use or not to use a security framework. If we use, then Spring Security is the cheapest and fastest solution because we already use Spring Framework in our application. For the benefit, any security framework provide us quick and declarative way to declare assess rule. However, it will not be business logic aware access rule. For example, we can define that only Agent can access the products but we can not define that one agent can only access some products that belong to him.

In this situation, we have two choices, building our own business logic permission check from scratch or build 2 layers of permission check, one is only role based, one is business logic aware. After comparing two approaches, we chose the latter one because it is cheaper and faster to build. Our application will function similar to any other Spring Security application. It means that user will be redirected to login page if accessing protected content without session. If the session exist, user will get status code 403. If user access protected content with valid role but unauthorized records, he will get 401 instead.

Authentication

The next concern is how to integrate our authentication and Authorization mechanism with Spring Security. A standard Spring Security application may process a request like below:



The diagram is simplified but still give us a raw idea how things work. If the request is login or logout, the top two filters update the server side session. After that, another filter help check access permission for the request. If the permission check success, another filter will help to store user session to thread storage. After that, controller will execute code with the properly setup environment.

For us, we prefer to create our authentication mechanism because the credential need to contain website domain. For example, we may have Joe from Xerox and Joe from WDS accessing Saas application. As Spring Security take control of preparing authentication token and authentication provider, we find it is cheaper to implement login and logout ourselves at the controller level rather than spending effort on customizing Spring Security.

As we implement stateless session, there are two works we need to implements here. At first, we need to to construct the session from cookie before any authorization check. We also need to update the session time stamp so that the session is refreshed every time browser send request to server.

Because of the earlier decision to do authentication in controller, we face a challenge here. We should not refresh the session before controller executes because we do authentication here. However, some controller methods is attached with the View Resolver that write to output stream immediately. Therefore, we have no chance to refresh cookie after controller being executed. Finally, we choose a slightly compromised solution by using HandlerInterceptorAdapter. This handler interceptor allow us to do extra processing before and after each controller method. We implement refreshing cookie after controller method if the method is for authentication and before controller methods for any other purpose. The new diagram should look like this



Cookie

To be meaningful, user should have only one session cookie. As the session always change time stamp after each request, we need to update session on every single response. By HTTP protocol, this can only be done if the cookies match name, path and domain.

When getting this business requirement, we prefer to try new way of implementing SSO by sharing session cookie. If every application are under the same parent domain and understand the same session cookie, effectively we have a global session. Therefore, there is no need for authentication server any more. To achieve that vision, we must set the domain as the parent domain of all applications.

To illustrate this global session, let come back to the earlier example where we have two applications that contain the domain name as microsoft.mail.somedomain.com or google.map.somedomain.com

For the session cookie to be global, we will set the domain as somedomain.com. Obviously, the session cookie can be seen and maintained by both applications as long as they share the same secret key to sign.

Performance

Theoretically, stateless session should be slower. Assuming that the server implementation store session table in memory, passing in JSESSIONID cookie will only trigger a one time read of object from the session table and optional one time write to update last access (for calculating session timeout). In contrast, for stateless session, we need to calculate the hash to validate session cookie, load principal from database, assigning new time stamp and hash again.

However, with today server performance, hashing should not add too much delay in server response time. The bigger concern is querying data from database, and for this, we can speed up by using cache.

In best case scenario, stateless session can perform closely enough to stateful if there is no DB call made. In stead of loading from session table, which maintained by container, the session is loaded from internal cache, which is maintained by application. In the worst case scenario, requests are being routed to many different servers and the principal object is stored in many instances. This add additional effort to load principal to the cache once per server. While the cost may be high, it occurs only once in a while.

If we apply stickiness routing to load balancer, we should be able to achieve best case scenario performance. With this, we can perceive the stateless session cookie as similar mechanism to JSESSIONID but with fall back ability to reconstruct session object.

Implementation

I have published the sample of this implementation to https://github.com/tuanngda/sgdev-blog repository. Kindly check the stateless-session project. The project requires a mysql database to work. Hence, kindly setup a schema following build.properties or modify the properties file to fit your schema.

The project include maven configuration to start up a tomcat server at port 8686. Therefore, you can simply type mvn cargo:run to start up the server.

Here is the project hierarchy:


I packed both Tomcat 7 server and the database so that it work without any other installation except MySQL. The Tomcat configuration file TOMCAT_HOME/conf/context.xml contain the DataSource declaration and project properties file.

Now, let look closer at the implementation

Session

We need two session objects, one represent the session cookie, one represent the session object that we build internally in Spring security framework:

public class SessionCookieData {
 
 private int userId;
 
 private String appId;
 
 private int siteId;
 
 private Date timeStamp;
}

and

public class UserSession {
 
 private User user;
 
 private Site site;

 public SessionCookieData generateSessionCookieData(){
  return new SessionCookieData(user.getId(), user.getAppId(), site.getId());
 }
}

With this combo, we have the objects to store session object in cookie and memory. The next step is to implement a method that allow us to build session object from cookie data.

public interface UserSessionService {
 
 public UserSession getUserSession(SessionCookieData sessionData);
}

Now, one more service to retrieve and generate cookie from cookie data.

public class SessionCookieService {

 public Cookie generateSessionCookie(SessionCookieData cookieData, String domain);

 public SessionCookieData getSessionCookieData(Cookie sessionCookie);

 public Cookie generateSignCookie(Cookie sessionCookie);
}

Up to this point, We have the service that help us to do the conversion

Cookie --> SessionCookieData --> UserSession

and

Session --> SessionCookieData --> Cookie

Now, we should have enough material to integrate stateless session with Spring Security framework

Integrate with Spring security

At first, we need to add a filter to construct session from cookie. Because this should happen before permission check, it is better to use AbstractPreAuthenticatedProcessingFilter

@Component(value="cookieSessionFilter")
public class CookieSessionFilter extends AbstractPreAuthenticatedProcessingFilter {
 
...
 
 @Override
 protected Object getPreAuthenticatedPrincipal(HttpServletRequest request) {
  SecurityContext securityContext = extractSecurityContext(request);
  
  if (securityContext.getAuthentication()!=null  
     && securityContext.getAuthentication().isAuthenticated()){
   UserAuthentication userAuthentication = (UserAuthentication) securityContext.getAuthentication();
   UserSession session = (UserSession) userAuthentication.getDetails();
   SecurityContextHolder.setContext(securityContext);
   return session;
  }
  
  return new UserSession();
 }
 ...
 
}

The filter above construct principal object from session cookie. The filter also create a PreAuthenticatedAuthenticationToken that will be used later for authentication. It is obviously that Spring will not understand this Principal. Therefore, we need to provide our own AuthenticationProvider that manage to authenticate user based on this principal.

public class UserAuthenticationProvider implements AuthenticationProvider {
@Override
  public Authentication authenticate(Authentication authentication) throws AuthenticationException {
    PreAuthenticatedAuthenticationToken token = (PreAuthenticatedAuthenticationToken) authentication;

    UserSession session = (UserSession)token.getPrincipal();

    if (session != null && session.getUser() != null){
      SecurityContext securityContext = SecurityContextHolder.getContext();
      securityContext.setAuthentication(new UserAuthentication(session));
      return new UserAuthentication(session);
    }

    throw new BadCredentialsException("Unknown user name or password");
  }
}

This is Spring way. User is authenticated if we manage to provide a valid Authentication object. Practically, we let user login by session cookie for every single request.

However, there are times that we need to alter user session and we can do it as usual in controller method. We simply overwrite the SecurityContext, which is setup earlier in the pre-authentication filter.


public ModelAndView login(String login, String password, String siteCode) throws IOException{
    
    if(StringUtils.isEmpty(login) || StringUtils.isEmpty(password)){
      throw new HttpServerErrorException(HttpStatus.BAD_REQUEST, "Missing login and password");
    }
    
    User user = authService.login(siteCode, login, password);
    if(user!=null){
      SecurityContext securityContext = SecurityContextHolder.getContext();
      UserSession userSession = new UserSession();
      userSession.setSite(user.getSite());
      userSession.setUser(user);
      securityContext.setAuthentication(new UserAuthentication(userSession));
    }else{
      throw new HttpServerErrorException(HttpStatus.UNAUTHORIZED, "Invalid login or password");
    }
    
    return new ModelAndView(new MappingJackson2JsonView());
    
  }

Refresh Session

Up to now, you may notice that we have never mentioned the writing of cookie. Provided that we have a valid Authentication object and our SecurityContext contain the UserSession, it is important that we need to send this information back to browser.

Before the HttpServletResponse is generated, we must generate and attach the session cookie to it. This new session cookie,  which has similar domain and path will replace the older session cookie that the browser is keeping.

As discussed above, refreshing session is better to be done after controller method because we implement authentication at this layer. However, there is a challenge caused by ViewResolver of Spring MVC. Sometimes, it writes to OutputStream so soon that any attempt to add cookie to response will be useless.

After consideration, we come up with a compromise solution that refresh session before controller methods for normal requests and after controller methods for authentication requests. To know whether requests is for authentication, we place an newly defined annotation at the authentication methods.

  @Override
  public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
    if (handler instanceof HandlerMethod){
      HandlerMethod handlerMethod = (HandlerMethod) handler;
      SessionUpdate sessionUpdateAnnotation = handlerMethod.getMethod().getAnnotation(SessionUpdate.class);
      
      if (sessionUpdateAnnotation == null){
        SecurityContext context = SecurityContextHolder.getContext();
        if (context.getAuthentication() instanceof UserAuthentication){
          UserAuthentication userAuthentication = (UserAuthentication)context.getAuthentication();
          UserSession session = (UserSession) userAuthentication.getDetails();
          persistSessionCookie(response, session);
        }
      }
    }
    return true;
  }

  @Override
  public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler,
      ModelAndView modelAndView) throws Exception {
    if (handler instanceof HandlerMethod){
      HandlerMethod handlerMethod = (HandlerMethod) handler;
      SessionUpdate sessionUpdateAnnotation = handlerMethod.getMethod().getAnnotation(SessionUpdate.class);
      
      if (sessionUpdateAnnotation != null){
        SecurityContext context = SecurityContextHolder.getContext();
        if (context.getAuthentication() instanceof UserAuthentication){
          UserAuthentication userAuthentication = (UserAuthentication)context.getAuthentication();
          UserSession session = (UserSession) userAuthentication.getDetails();
          persistSessionCookie(response, session);
        }
      }
    }
  }

Conclusion

The solution works well for us but we do not have the confident that this is the best practices possible. However, it is simple and does not cost us much effort to implement (around 3 days include testing).

Kindly feedback if you have any better idea to build stateless session with Spring.