My thoughts on Agile Development


I dont confess to be a authority on 'agile' , however in my last five years with various companies i have seen 'agile' becoming a buzzword  and a fad i.e. people say we do agile while they end up following water-fall model. I have learned from talking to people and understanding the challenges they faced and running scrum myself and learning from my mistakes.

sometimes people get up hung up in 'following the book to the last word' while they forget that it is "essentially" an evolution based on the current times.

what is 'agile development'

  - the only problem that matters here is that the time to market is very large for  water-fall / RUP blah blah model e.g. how would you manage your project if your time to project is 1 month or rather continuous , we can no longer afford to have requirement complete before we start design and design complete before we do development.  In other words the entire environment is continuous i.e. all stages need to react to changes as all of them are working concurrently in other words 'agile' development.

How do i get agile

  - people tend to condense the process into the small timeframe without adopting some of the principles of being 'agile'. let us consider the problem mentioned in the above paragraph and try to figure out how would we solve the problem.

• the first thing that it means that i am not going to have complete requirements before i design and develop. it means close co-operation with my product manager and my designer i.e. we collaborate on the product in an iterative process.
•  some people thing that agile means PM can give a completely new requirement 2 days before a release and expect it to get done.  to be fair to the entire "PM community" i think there are some PMs who get it and understand how it works  and some who don't get it. ( i am not going to get into why they don't get it) .
•  i can surely hear a lot of people cry out loud that what happens when it is a genuinely such a condition for e.g. an unexpected market opportunity.  to all those people out there yes agile is supposed to address this.   One of the most important features of being agile is  to instill the thought " ALL PARTIES  eng/product/qa/ops are equal stake holders in the project" , if your org does not believe in this then in my opinion you are not agile. coming back to argument , in such a case where all parties are equal parties to the project it means couple of things
• every body in the project recognizes the opportunity and willing to accept the change
• once consenus is achieved , it comes down to resource planning and scheduling and this is less of a problem in my opinion and needless to say "to put something in a bucket which is full, something needs to be taken out first"

 - make sure all the members of the project know what every member is responsible for , one cannot have mis communication when one is working on such short time projects.  have a daily scrum meeting.

• keep is small , i have been part of scrum meeting which run for one and half hours long everyday
• scrum meetings should be of the following , every team member needs to say only 3 thing

• what am i  woking today
• what i did yesterday
• am i blocked on something i am working. if there is a dependency please dont make that discussion a part of the scrum meeting.  

-  make sure you develop something, give a demo to all the stake holders of the project and then iterate on the feedback. giving early demos and collecting feedback is invaluable.

- MOST Important . RECOGNIZE that there is no single solution that fits all, a team is made of people and people are different in different teams and different orgs just because one worked in one project does not mean it will work everywhere.

• for e.g. if you successfully ran scrum in a startup with 3 engs and  qa, it  does not mean the same process applies for a large group where there are some 20 eng and 10 qa.  in this example dont hesitate in breaking up the scrum to multiple team scrums and have a representative from that team 
• understand that "agile"  means being "agile" ourselves. learn what works and does not work and tweak process till you get the desired effect.

- Lastly,  if you want to learn 'agile development' do the following

• get a book. after you read the book throw it down the drain or sell it on ebay, whatever you do don't ever touch that book again. sometimes i have a sneaking feeling that some books are ambiguous so that we can hire consultants who will explain it in lay man terms.
• understand what is the problem that you are trying to solve
• apply you analytical mind to the problem , in this way u will adapt the process to the problem instead of adapting the problem to the process.

would like to hear how other people adapted scrum to their projects.

Checking out Node.js 


I am been playing with node.js for some time and over the weekend i thought of doing a simple benchmark between node.js and jetty,  the objective was to quickly check out the two implementation in other words beat the bush and see what falls out.

I half expected the node.js performance to be less that jetty using a NIO connector , however i was pleasantly surprised to see that tps is the same while the cpu utilization of node.js was very very less.


Here is what i ended up doing

• wrote a small helloworld  httpserver in node.js , about 4 lines of code
• wrote a custom handler for jetty which says helloWorld

Used apache bench to test it for 10000 requests and 20 concurrent clients.

node.js results

=========

Server Software:      
Server Hostname:        uno
Server Port:            8124

Document Path:          /
Document Length:        14 bytes

Concurrency Level:      20
Time taken for tests:   12.896 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      780000 bytes
HTML transferred:       140000 bytes
Requests per second:    775.41 [#/sec] (mean)
Time per request:       25.793 [ms] (mean)
Time per request:       1.290 [ms] (mean, across all concurrent requests)
Transfer rate:          59.06 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2   13  29.2     10     994
Processing:     3   13  10.4     10     255
Waiting:        2   12   8.4     10     234
Total:         11   26  32.2     22    1009

Percentage of the requests served within a certain time (ms)
  50%     22
  66%     24
  75%     26
  80%     27
  90%     33
  95%     40
  98%     69
  99%    105
 100%   1009 (longest request)

Jetty results

========

Server Software:        Jetty(7.2.2.v20101205)
Server Hostname:        uno
Server Port:            8081

Document Path:          /
Document Length:        21 bytes

Concurrency Level:      20
Time taken for tests:   12.779 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1680000 bytes
HTML transferred:       210000 bytes
Requests per second:    782.54 [#/sec] (mean)
Time per request:       25.558 [ms] (mean)
Time per request:       1.278 [ms] (mean, across all concurrent requests)
Transfer rate:          128.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2   13  32.9     10    1011
Processing:     2   12  12.0     10     357
Waiting:        2   12  10.5      9     288
Total:          9   25  36.6     21    1062

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     24
  75%     25
  80%     26
  90%     32
  95%     43
  98%     98
  99%    121
 100%   1062 (longest request)

Observations:

   

1) the latency and tps between the two is very much the same  which truly surprised me so i decided to see cpu and context switches and that the number were very interesting

jetty cpu usage:

======

cpu usage : 30%

cs :  max  ~ 7200   avg ~ 5000 

node.js cpu usage:

=======

cpu usage : 4%

cs :  max 1100   avg ~ 750

jetty has a very high number of context switches (note that this is untuned server, i had expected the default config to do a better job) , this also points to the fact how engineers overlook context switches there is no way that the system can perform if it has such a high number of context switches

trying to figure out why jetty is doing so much CS and cpu is a another weekend project.

2)  because of modern cpus and because of the lean design of node.js ( which means that we are executing only a small amount of script code) the numbers are very close.

Things that i did not try or find out how to:
*) the current release does not give the ability to tune the number of worker threads. not sure how it can be tuned to different workloads.

I came away impressed with node.js and now need to look into the actual code and figure out more about its tuning parameters  as well as benchmark it against netty 

NFC why i am excited about it


During my interactions with my friends and colleagues i came to realize that  lot of people do not know about this technology and how it would make their daily experiences better. so here i am putting down my thoughts why i am excited about nfc and looking forward to it.


Consider the following use case :
Whenever is visit fry's i see a lot of people scanning the barcode and checking out the product information and as anyone knows this is a not a very fool-proof system but it works as of today, with NFC we would be able to tap our phones on the product tag on the shelf below and get all the product information.

The scope for retail is huuuge (there are a lot of other very interesting use cases but that will take me some days to talk about) and i expect google to enable the whole merchant eco system and integrate it with google checkout  and hook it up with the ad network and analytics  , i fully expect Apple will going to try the same too.

if we consider our daily interactions to consist of the following
1) me interacting with society ( let us assume for the discussion that it a web interaction )
2) me interacting with objects : buying goods (clothes/electronics/phone)  
3) my interacting with other machines at my workplace , my home , my car , my music system  using my phone

1 and 3 are possible as of today , i can control my TV/printer  through my phone , what is missing is my interactions with objects like buying a gadget, the gadget has a  bar code but i am not a bar code reader there is no way i can get information and look up what other people are talking about it and more importantly what do people in my social circle have to say about it. NFC is going to enable this and make my interaction cycle complete.

I am looking forward to the day when the technology becomes mainstream.

My web 20 experience


If i were to answer it in one line it would be "i eat my own dogfood".

I have been an enterprise guy who developed software and threw it over the wall to the QA and finally customer and all i was interested in, was in solving technical challenges. I was never responsible for keeping them running and understanding how to make them better.

As part of building and running apigee i realized the value of eating my own dogfood. One cannot build and run a website based business without understanding all different metrics associated with it. Apart from creating technology i am interested in understanding what features get used often, which part of the system is getting stressed under what conditions and the user behaviour.

In short I am much more agile, motivated and a better technologist than before. It has been hell of a ride with plenty of battle scars to go along with it and i am proud to say "I eat my own dogfood".

Application Infrastructure race in the clould

Today Intalio announced the takeover of webtide ( the maker of Jetty) me and my friend (an ex Intalio employee, and both of us being old middle ware hands) realize that this is the race for owning and proving the complete application infrastructure on the cloud.

Oracle with its acquisition of BEA and SUN in uniquely poised to provide a complete application infrastructure ( the whole enchilada) on the cloud.

I am also looking forward to when VMware would provide us with Virtual Middleware with VMware and Springsource technologies.

The Race is on ....

what is an API ?

I met a friend after a couple of years and we started talking about our work. We started talking about API Infrastructure and midway thru the discussion he asked me a very simple question. what is an API and how is it different from a Service , Why is a service Bus infrastructure different from a API Infrastructure. At that moment I realized that this is question that i should have answered first and here i am writing it down.

api definition:
- An api is reusable asset which can be invoked on the web. As a reusable asset the api can be combined/invoked from another website/mashup/application to add value for e.g. retrieving the List of Movies I have watched on Netflix over the last week and displaying it in my website.

what is the difference between a Service Infrastructure and API Infrastructure

I sometimes use services and apis in the same context and there is no difference between the two in my mind however architecturally there are two types of services

a) services which i create within my application/enterprise for business process reuse and loose coupling between the various technology stacks available. This is the soa story and i am not going there. there are enough products out there which can solve the problem

b) service or APIs which i create so that they can be consumed by my partners /consumers or even shared across departments and services which i consume from other guys.

From a technology perspective "a" requires "Service Infrastructure" while "b" is the "API Infrastructure"

• "Service Infrastructure" is about integration of services while "API Infrastructure" is about producing and consuming apis/services.
• "Service Infrastructure" is part of the middleware technology while "API Infrastructure" is more of a gateway functionality. In "Service Infrastructure" i am trying to integrate across disparate technologies how to integrate with ftp/email/jms etc while "API Infrastructure" is about exposing apis via http as POX/REST/SOAP
• the concerns of "API Infrastructure" are security,scale, sla& monitoring,visibility instead of transport mediation/data mediation/process orchestration. If i am 1800Flowers.com and exposing an api to buy flowers then i would be concerned of making it available on the web with regards to scalability and security (for e.g. how can i guard myself against XML xDos/MDos Attacks) on the other hand if i were an enterprise doing service integration i would looking at how can i expose my Messaging (JMS) backend with a HTTP front end and do not end up having duplicate messages , architecturally an API Infrastructure sits in front of the Service Infrastructure.
  

Adoption of Enterprise 2.0/N.0 == More API Traffic
-------------------------------------------------------------

Enterprise 2.0 as an idea was floated as an idea late as 2007 and my interest in Enterprise 2.0/N.0 stems only from the product engineering perspective and the new infrastructual requirements it poses.

What is my version of Enterprise 2.0 ( i can see you guys rolling your eyes but give me a hearing first...)

For me the definition of "Enterprise 2.0 is the successful adaption of social software platforms within companies, or between companies and their partners or customers". I have to say "successful adaption" because i do not think the enterprise will adapt the web 20 completely, Enterprises will never give up the control but they would like to adapt the practices of web 20 with regards to collaboration & self service and agility and adapt them into the enterprise.

How will they get manifested within the Enterprise:

I would like to see "Boeing" build its next generation "Partner Management/Tracking Platform" using mashups and social software paradigms , the idea of having the "part tracking" portlet implemented as a mashup and exposed as a widget of choice on someone's iphone, blackberry which is very salivating thought.

With Google announcing SDC and vendors providing support means only one thing i.e. we are going to see more api traffic coming from the enterprise and thats a very good thing.


End Result
more APIs /more enterprise data getting exposed/ more traffic.... how are you going to manage this ?