Manoj's Blog

2011-02-06T14:06:00.000-08:00

My thoughts on Agile Development

I dont confess to be a authority on 'agile' , however in my last five years with various companies i have seen 'agile' becoming a buzzword and a fad i.e. people say we do agile while they end up following water-fall model. I have learned from talking to people and understanding the challenges they faced and running scrum myself and learning from my mistakes.

sometimes people get up hung up in 'following the book to the last word' while they forget that it is "essentially" an evolution based on the current times.

what is 'agile development'

- the only problem that matters here is that the time to market is very large for water-fall / RUP blah blah model e.g. how would you manage your project if your time to project is 1 month or rather continuous , we can no longer afford to have requirement complete before we start design and design complete before we do development. In other words the entire environment is continuous i.e. all stages need to react to changes as all of them are working concurrently in other words 'agile' development.

How do i get agile

- people tend to condense the process into the small timeframe without adopting some of the principles of being 'agile'. let us consider the problem mentioned in the above paragraph and try to figure out how would we solve the problem.

the first thing that it means that i am not going to have complete requirements before i design and develop. it means close co-operation with my product manager and my designer i.e. we collaborate on the product in an iterative process.
some people thing that agile means PM can give a completely new requirement 2 days before a release and expect it to get done. to be fair to the entire "PM community" i think there are some PMs who get it and understand how it works and some who don't get it. ( i am not going to get into why they don't get it) .
i can surely hear a lot of people cry out loud that what happens when it is a genuinely such a condition for e.g. an unexpected market opportunity. to all those people out there yes agile is supposed to address this. One of the most important features of being agile is to instill the thought " ALL PARTIES eng/product/qa/ops are equal stake holders in the project" , if your org does not believe in this then in my opinion you are not agile. coming back to argument , in such a case where all parties are equal parties to the project it means couple of things

every body in the project recognizes the opportunity and willing to accept the change
once consenus is achieved , it comes down to resource planning and scheduling and this is less of a problem in my opinion and needless to say "to put something in a bucket which is full, something needs to be taken out first"

- make sure all the members of the project know what every member is responsible for , one cannot have mis communication when one is working on such short time projects. have a daily scrum meeting.

keep is small , i have been part of scrum meeting which run for one and half hours long everyday
scrum meetings should be of the following , every team member needs to say only 3 thing

what am i woking today
what i did yesterday
am i blocked on something i am working. if there is a dependency please dont make that discussion a part of the scrum meeting.

- make sure you develop something, give a demo to all the stake holders of the project and then iterate on the feedback. giving early demos and collecting feedback is invaluable.

- MOST Important . RECOGNIZE that there is no single solution that fits all, a team is made of people and people are different in different teams and different orgs just because one worked in one project does not mean it will work everywhere.

for e.g. if you successfully ran scrum in a startup with 3 engs and qa, it does not mean the same process applies for a large group where there are some 20 eng and 10 qa. in this example dont hesitate in breaking up the scrum to multiple team scrums and have a representative from that team
understand that "agile" means being "agile" ourselves. learn what works and does not work and tweak process till you get the desired effect.

- Lastly, if you want to learn 'agile development' do the following

get a book. after you read the book throw it down the drain or sell it on ebay, whatever you do don't ever touch that book again. sometimes i have a sneaking feeling that some books are ambiguous so that we can hire consultants who will explain it in lay man terms.
understand what is the problem that you are trying to solve
apply you analytical mind to the problem , in this way u will adapt the process to the problem instead of adapting the problem to the process.

would like to hear how other people adapted scrum to their projects.

2011-01-30T12:41:00.000-08:00

Checking out Node.js

I am been playing with node.js for some time and over the weekend i thought of doing a simple benchmark between node.js and jetty, the objective was to quickly check out the two implementation in other words beat the bush and see what falls out.

I half expected the node.js performance to be less that jetty using a NIO connector , however i was pleasantly surprised to see that tps is the same while the cpu utilization of node.js was very very less.

Here is what i ended up doing

wrote a small helloworld httpserver in node.js , about 4 lines of code
wrote a custom handler for jetty which says helloWorld

Used apache bench to test it for 10000 requests and 20 concurrent clients.

node.js results

=========

Server Software:
Server Hostname: uno
Server Port: 8124

Document Path: /
Document Length: 14 bytes

Concurrency Level: 20
Time taken for tests: 12.896 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 780000 bytes
HTML transferred: 140000 bytes
Requests per second: 775.41 [#/sec] (mean)
Time per request: 25.793 [ms] (mean)
Time per request: 1.290 [ms] (mean, across all concurrent requests)
Transfer rate: 59.06 [Kbytes/sec] received

Connection Times (ms)
   min mean[+/-sd] median max
Connect: 2 13 29.2 10 994
Processing: 3 13 10.4 10 255
Waiting: 2 12 8.4 10 234
Total: 11 26 32.2 22 1009

Percentage of the requests served within a certain time (ms)
  50% 22
  66% 24
  75% 26
  80% 27
  90% 33
  95% 40
  98% 69
  99% 105
100% 1009 (longest request)

Jetty results

========

Server Software: Jetty(7.2.2.v20101205)
Server Hostname: uno
Server Port: 8081

Document Path: /
Document Length: 21 bytes

Concurrency Level: 20
Time taken for tests: 12.779 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1680000 bytes
HTML transferred: 210000 bytes
Requests per second: 782.54 [#/sec] (mean)
Time per request: 25.558 [ms] (mean)
Time per request: 1.278 [ms] (mean, across all concurrent requests)
Transfer rate: 128.39 [Kbytes/sec] received

Connection Times (ms)
   min mean[+/-sd] median max
Connect: 2 13 32.9 10 1011
Processing: 2 12 12.0 10 357
Waiting: 2 12 10.5 9 288
Total: 9 25 36.6 21 1062

Percentage of the requests served within a certain time (ms)
  50% 21
  66% 24
  75% 25
  80% 26
  90% 32
  95% 43
  98% 98
  99% 121
100% 1062 (longest request)

Observations:

1) the latency and tps between the two is very much the same which truly surprised me so i decided to see cpu and context switches and that the number were very interesting

jetty cpu usage:

======

cpu usage : 30%

cs : max ~ 7200 avg ~ 5000

node.js cpu usage:

=======

cpu usage : 4%

cs : max 1100 avg ~ 750

jetty has a very high number of context switches (note that this is untuned server, i had expected the default config to do a better job) , this also points to the fact how engineers overlook context switches there is no way that the system can perform if it has such a high number of context switches

trying to figure out why jetty is doing so much CS and cpu is a another weekend project.

2) because of modern cpus and because of the lean design of node.js ( which means that we are executing only a small amount of script code) the numbers are very close.

Things that i did not try or find out how to:
*) the current release does not give the ability to tune the number of worker threads. not sure how it can be tuned to different workloads.

I came away impressed with node.js and now need to look into the actual code and figure out more about its tuning parameters as well as benchmark it against netty

2011-01-29T18:28:00.000-08:00

NFC why i am excited about it

During my interactions with my friends and colleagues i came to realize that lot of people do not know about this technology and how it would make their daily experiences better. so here i am putting down my thoughts why i am excited about nfc and looking forward to it.

Consider the following use case :
Whenever is visit fry's i see a lot of people scanning the barcode and checking out the product information and as anyone knows this is a not a very fool-proof system but it works as of today, with NFC we would be able to tap our phones on the product tag on the shelf below and get all the product information.

The scope for retail is huuuge (there are a lot of other very interesting use cases but that will take me some days to talk about) and i expect google to enable the whole merchant eco system and integrate it with google checkout and hook it up with the ad network and analytics , i fully expect Apple will going to try the same too.

if we consider our daily interactions to consist of the following
1) me interacting with society ( let us assume for the discussion that it a web interaction )
2) me interacting with objects : buying goods (clothes/electronics/phone)
3) my interacting with other machines at my workplace , my home , my car , my music system using my phone

1 and 3 are possible as of today , i can control my TV/printer through my phone , what is missing is my interactions with objects like buying a gadget, the gadget has a bar code but i am not a bar code reader there is no way i can get information and look up what other people are talking about it and more importantly what do people in my social circle have to say about it. NFC is going to enable this and make my interaction cycle complete.

I am looking forward to the day when the technology becomes mainstream.

2010-01-07T16:36:00.000-08:00

My web 20 experience

If i were to answer it in one line it would be "i eat my own dogfood".

I have been an enterprise guy who developed software and threw it over the wall to the QA and finally customer and all i was interested in, was in solving technical challenges. I was never responsible for keeping them running and understanding how to make them better.

As part of building and running apigee i realized the value of eating my own dogfood. One cannot build and run a website based business without understanding all different metrics associated with it. Apart from creating technology i am interested in understanding what features get used often, which part of the system is getting stressed under what conditions and the user behaviour.

In short I am much more agile, motivated and a better technologist than before. It has been hell of a ride with plenty of battle scars to go along with it and i am proud to say "I eat my own dogfood".

2009-09-01T22:44:00.000-07:00

Application Infrastructure race in the clould

Today Intalio announced the takeover of webtide ( the maker of Jetty) me and my friend (an ex Intalio employee, and both of us being old middle ware hands) realize that this is the race for owning and proving the complete application infrastructure on the cloud.

Oracle with its acquisition of BEA and SUN in uniquely poised to provide a complete application infrastructure ( the whole enchilada) on the cloud.

I am also looking forward to when VMware would provide us with Virtual Middleware with VMware and Springsource technologies.

The Race is on ....

2009-04-17T22:22:00.000-07:00

what is an API ?

I met a friend after a couple of years and we started talking about our work. We started talking about API Infrastructure and midway thru the discussion he asked me a very simple question. what is an API and how is it different from a Service , Why is a service Bus infrastructure different from a API Infrastructure. At that moment I realized that this is question that i should have answered first and here i am writing it down.

api definition:
- An api is reusable asset which can be invoked on the web. As a reusable asset the api can be combined/invoked from another website/mashup/application to add value for e.g. retrieving the List of Movies I have watched on Netflix over the last week and displaying it in my website.

what is the difference between a Service Infrastructure and API Infrastructure

I sometimes use services and apis in the same context and there is no difference between the two in my mind however architecturally there are two types of services

a) services which i create within my application/enterprise for business process reuse and loose coupling between the various technology stacks available. This is the soa story and i am not going there. there are enough products out there which can solve the problem

b) service or APIs which i create so that they can be consumed by my partners /consumers or even shared across departments and services which i consume from other guys.

From a technology perspective "a" requires "Service Infrastructure" while "b" is the "API Infrastructure"

"Service Infrastructure" is about integration of services while "API Infrastructure" is about producing and consuming apis/services.
"Service Infrastructure" is part of the middleware technology while "API Infrastructure" is more of a gateway functionality. In "Service Infrastructure" i am trying to integrate across disparate technologies how to integrate with ftp/email/jms etc while "API Infrastructure" is about exposing apis via http as POX/REST/SOAP
the concerns of "API Infrastructure" are security,scale, sla& monitoring,visibility instead of transport mediation/data mediation/process orchestration. If i am 1800Flowers.com and exposing an api to buy flowers then i would be concerned of making it available on the web with regards to scalability and security (for e.g. how can i guard myself against XML xDos/MDos Attacks) on the other hand if i were an enterprise doing service integration i would looking at how can i expose my Messaging (JMS) backend with a HTTP front end and do not end up having duplicate messages , architecturally an API Infrastructure sits in front of the Service Infrastructure.

2009-04-15T15:11:00.000-07:00

Adoption of Enterprise 2.0/N.0 == More API Traffic
-------------------------------------------------------------

Enterprise 2.0 as an idea was floated as an idea late as 2007 and my interest in Enterprise 2.0/N.0 stems only from the product engineering perspective and the new infrastructual requirements it poses.

What is my version of Enterprise 2.0 ( i can see you guys rolling your eyes but give me a hearing first...)

For me the definition of "Enterprise 2.0 is the successful adaption of social software platforms within companies, or between companies and their partners or customers". I have to say "successful adaption" because i do not think the enterprise will adapt the web 20 completely, Enterprises will never give up the control but they would like to adapt the practices of web 20 with regards to collaboration & self service and agility and adapt them into the enterprise.

How will they get manifested within the Enterprise:

I would like to see "Boeing" build its next generation "Partner Management/Tracking Platform" using mashups and social software paradigms , the idea of having the "part tracking" portlet implemented as a mashup and exposed as a widget of choice on someone's iphone, blackberry which is very salivating thought.

With Google announcing SDC and vendors providing support means only one thing i.e. we are going to see more api traffic coming from the enterprise and thats a very good thing.

End Result
more APIs /more enterprise data getting exposed/ more traffic.... how are you going to manage this ?

2009-03-13T21:08:00.000-07:00

APIs Do we need a seperate Infrastructure ?

We as an industry have been talking about apis since 2007 (when web sites become web services) and with the recent success of guys like twitter (Twitter Traffic is 10x times its site) we are starting to hear more about apis.

My colleague (Greg) recently blogged about what is an api, it gives a good definition of what an api is and raises some important questions that everybody needs to ask, whoever is looking to build an api.

As a company/user who is starting to go down the API path the subsequent question that it needs ask is do i have an existing infrastructure which can handle the requirements rasied by Greg.
Some of us might say yes, we have the infrastructure to do that or there are existing products which can help me manage my apis, but do we have the "complete and comprehensive infrastructure" to cover all the requirements.

I can possibly take an existing open source stack or some Middleware product and say it provides me the capabilities but does it also do the following

does it give analytics for e.g. Netflix launched a API and is now integrated into Facebook, Netflix would be intersted in knowing what are the top apis, who drives more traffic facebook or myspace in other words can the infrastructure provide me with information to make business decisions.
Does the infrastructure provide me with SLA and alerts so that i know when my api is down or not behaving properly.
How flexible is the deployment model of my apis so that it can change to the ever changing business needs, in other words what is the life cycle management of my apis in a ever changing world
If i am Boeing who is exposing an api to my partners can i assign different rate limits to different partners , i am paranoid about security/compliance/data security can the api infrastructure help me with that.
Does it do intelligent caching
If i am an enterprise who is consuming a lot of web Apis how can i use this infrastructure to manage and control the apis i am consuming within my enterprise (this is a discussion for another day)

On top of this do i have a robust infrastructure to scale to the requirements of the web. (Each of the requirements above probably require a dedicated blog so that's a task for some other day)

Today is the day to ask whether we need a seperate infrastructure for apis and if yes then let ask for one.

Would like to hear from people producing and consuming apis what are some of the problems you have been facing and how are you planning to solve them.