Our informal discussions around scalability inspired me to jot down some of my thoughts on this issue, and how early-stage entrepreneurs can scale their technology platform from 5 users to millions.

Some Simple Rules

There rarely exists a set of rules that, if followed, will result in nirvana – scalability is no different. Every situation is different. However, these bullets summarize my general tenets, mantras, and beliefs for scaling a web-based application or system. The rest of this post will cover these in a bit more detail.

1. Abstract your logical architecture into at least 3 tiers (web, application, and data)
2. Even if you are only using one or two servers initially, think/design as if you have 50.
3. Designing with an SOA (services oriented architecture) in mind up front will aid you in scaling down the road.
4. Don’t overlook the scalability of your network infrastructure – your servers may be running at only 25% capacity, but if you don’t have the throughput and bandwidth to handle large amounts of traffic, those souped up servers won’t help you.
5. To help remove common/special cause variation – move system services such as DNS and e-mail off of production web/app servers and onto other, dedicated servers, and disable all non-essential services.
6. Scale vertically first, wherever possible, to control costs. But only do so if the benefits are greater than the cost. Early on, putting extra resources into an existing server is probably cheaper than procuring additional servers.
7. Scale horizontally when vertical scaling begins to produce diminishing returns
8. When upgrading/replacing existing servers, replace lesser horsepower ones first – and cannibalize components wherever possible (to control burn)
9. Performance tuning is not a one-time activity – it should be an iterative process
10. If you don’t have solid system administration capabilities in-house, outsource it immediately
11. Daily reporting is a must. You can’t know when/how to scale if you don’t know the landscape at any point in time.

What is scaling (or scalability)?
Simply put, scalability refers to an system’s ability to handle increasingly heavier loads from users (activity) without fundamentally breaking the way in which it operates. In other words, as you continue to add new users and expand your business, you want your application or service to be able to easily handle the increase without slowing down, or worse, breaking down completely.

An application (or platform) is considered scalable if it can continue to service additional users, through the deployment of supplemental hardware/software/resources, without seeing any significant performance hit from the user’s standpoint. Of course, very few systems in their early prototypical states will fit this definition. Your goal is to get the product to the point where it can be scaled in this manner, while reducing the number of potential bottlenecks.

Unfortunately, for business and IT managers, there is no single way to scale an application. Ah, if only there were actually a big red “easy” button. Each situation is different, given that there are so many factors that need to be taken into account. To make matters worse, sometimes, an application seems “infinitely scalable”, only to have a major bottleneck reveal itself down the road. This doesn’t mean the end of the world – it simply means that you have to adjust accordingly. The trick is, of course, to reduce the number of “midcourse corrections” that you will have to endure.

A Bit About n-Tier Architectures
Before we dive in too deep, I should probably throw out a note or two on n-Tier architectures. If you are an IT weenie, and understand this concept, skip to the next section. Otherwise, hang with me.

In the old days, applications were deployed onto servers, and when a bottleneck was encountered, the physical resources in the machines were expanded. This was the extent of scalability. Then, some bright engineer realized that if you split a system into two “tiers”, you could distribute the workload a bit. Voila – the birth of the client/server revolution. Eventually, though, systems began to grow so large that they needed something else in order to break through the inherent bottlenecks of a 2-tier system. An even brighter engineer realized that there was no reason to stop at “2″ tiers. You could have an arbitrary number of tiers in your system. Thus, the birth of the “n-Tier” architecture (“n” representing some arbitrary number of tiers).

An “n-tier” application architecture is characterized by the functional decomposition of applications, service components, and their distributed deployment. By breaking a system down in such a manner, it provides for improved scalability, availability, manageability, and good resource utilization. A “tier” itself is nothing more than a functionally separated hardware and software component that performs a specific function. Whew – what a mouthful.

Typical “tiers” include:

• Web server tier: provides HTTP protocol support (i.e. handles web requests)
• Application server tier: provides support for web services, business logic, etc. (e.g. web services/SOA)
• Database tier: provides data storage and retrieval support (e.g. Oracle, MySQL, mainframe/VSAM, etc.)

Note: don’t confuse these “tiers” with application “layers” (presentation layer, data layer, business logic layer, etc.) Tiers are architectural in nature, whereas “layers” are generally code/library specific.

The important thing to know here is that in an “n-tier” model, a system has been broken up into various levels of functionality, each capable of some degree of horizontal scaling. Which brings me to my next point …

Horizontal vs. Vertical Scaling
When you hear people talk about “scaling horizontally” they are essentially referring to the ability to add new servers to a tier to allow it continue to provide uninterrupted service in the face of continuously increasing usage. For example, your database is chugging hard and heavy, so you can add new servers to the database tier to distribute the workload. If your web server is bogged down, you can add new web servers to do the same. This also affords you with a nice layer of failover as well. If one server experiences an issue (even to the point where it crashes), you have other servers online in that particular tier that can still provide service. This concept is becoming increasingly important as more and more systems are being deployed using SOA models (service oriented architectures).

One great example of horizontal scaling can be seen in Amazon’s new “Elastic Computing Cloud” (my old friend Chris Brown is a patent co-inventor and senior engineer on that project).

Vertical scaling, on the other hand, is where you extend/expand the physical resources in a server itself. For instance, your database server is getting way overworked, hitting the swap space early and often. You can “vertically scale” that server by simply adding more RAM, faster hard disks, better CPUs, etc.

There are benefits/pros/cons to each type of scaling. Obviously there is a cost associated with both. Horizontal scaling is theoretically infinite, whereas vertical scaling has an obvious ceiling (there is only so much horsepower that you can derive from a single server).

Horizontal scaling only makes sense if the service you are attempting to scale was designed to be extended in this manner. For many third-party applications, such as a database server, this will be the case. Of course, if you are designing the software, you will want to take this into account as you build it.

Think of hardware as simply a vehicle (perhaps a bus) for your software, your real service. If the bus gets too crowded, you add another bus to the fleet. However, not all buses will go to the same destination, so those buses need to connect together in order to get information from point A to point B within your architecture. Voila, you have the meager beginnings of a service-oriented-architecture (SOA).

A Scalability Example
Let’s set the stage with a typical early-stage example, and we’ll try to scale this application theoretically as the business scales.

John is an aspiring technology entrepreneur who has developed a really great online service called WidgetFire. WidgetFire is brand new, so there aren’t many users yet. John has built this product in his spare time, and is bootstrapping the business via his day job as a software engineer for another company. To keep his costs down, he has a single web server (built from extra parts), and he has it co-located at a local data center with a basic level of service (1U single rack space, 1Mpbs throughput, unlimited bandwidth, for probably < $100/month). On this single server, he is running Apache and MySQL together, along with the normal services (bind/DNS, sendmail, etc.)

So far, scalability is not a concern to John. But that is about to change. In a big way.

Over a period of a few months, John leverages word-of-mouth marketing and manages to aggregate 25,000 registered users for his service. The server he is running (which he’s affectionately nicknamed “Seabiscuit”) is holding up fine, but is beginning to feel the strains of all of those new sessions and database queries. Additionally, his automated e-mail notification list is starting to add to the system load, as now the server is sending out thousands of emails a day.

John “vertically scales” his server by adding some additional RAM and performing some additional performance tuning to the database. This buys him time. But not much.

Then, WidgetFire gets a mention in a prominent tech blog, and the next thing John knows, he has 100,000 registered users for his service. His server is on its knees, and practically unresponsive. Sadly, the data center staff isn’t much help – after all, he is running under a pretty basic co-located hosting plan – and they have bigger problems to deal with than the occasionally unresponsive Seabiscuit.

So John then moves to a very rudimentary n-Tier architecture. He moves his MySQL database over to a separate server, which frees up resources on the old web server. Now, the system is humming along smoothly. But after a few months, WidgetFire is quite the rage on college campuses, and John must once again address how he is going to facilitate additional traffic.

He does some vertical scaling on the database server (adds new RAM, faster disks, etc.), but it isn’t enough. The new user signups are coming too fast and furious for his 2 server setup to handle.

This particular nexus is where many startups begin to experience some tough scalability issues. The “easy” scaling options have already been exhausted (vertical scaling on one server, splitting the database off into a separate server.)

Fortunately for John, his traction has caught the interest of a handful of investors, and he is able to secure a small round of outside capital.

To get to the next level, John implements a load balancing router, an extra web server, and an extra database server. He configures the load balancer to distribute incoming web requests evenly between his two web servers. Additionally, he configures the original database server to be a “master”, and the new database server to be a “slave” server. While all database write operations occur on the master server, John realizes that most web requests that require database access will be for “reads”, so the slave can offload some of that workload from the master. Voila – he has scaled even further!

A few more months go by, and John makes the cover of Wired Magazine. VCs are clamoring to pour their cash into WidgetFire. While John basks in this glory, he doesn’t realize that his little server farm is becoming overwhelmed by the sheer success of his venture. To make matters worse, while John is putting together a plan to scale even further, his slave database server crashes, leaving only the one original database server in operation. WidgetFire is basically dead in the water.

John brings the slave server back online, but he realizes that more must be done. He adds an extra web server to the web server tier, and adds an additional slave database server to the database tier. But he doesn’t stop there. John realizes that his actual application, which is comprised of a mish-mash of Java, PHP, and Perl, is utilizing the vast majority of CPU time on the web servers themselves. John decides to move from a 2-tier model to a 3-tier model by implementing an “application services” tier. He moves this code off of the web servers and onto several new servers.

John realizes pretty quickly that his original code design really wasn’t architected for this type of model. He has to spend a couple of months retrofitting his old code to fit into more of a “web services” model. He is now surrounded by burgeoning IT costs, hosting fees, and system complexity. All of a sudden, his “Google-ready” venture is giving him a headache, and isn’t much fun anymore.

Of course, had John anticipated the steepness of his growth curve ahead of time, he could have designed his system with it in mind, and avoided at least some of the headaches.

Can’t money be thrown at the problem?
Sure. To a point. All things cost money, of course, whether it be labor (people), hardware, or software. Most systems can initially be scaled to a sufficient level by simply adding more hardware, or expanding the resources within the server. But that only gets you so far in most cases – at some point the logical architecture of the system needs to have been designed with scalability in mind. If the architecture isn’t scalable, you are either going to hit a ceiling, or spend way too much money to scale it (and even then, there are no guarantees).

It isn’t a question of whether or not money can be thrown at the problem – it all costs money. The question is how much money are you going to have to spend to scale it? Obviously, there rarely exists an endless supply of capital that can be leveraged to solve a scalability problem – especially in the startup realm. Clearly, you want to be able to control your tech spend (which is a big part of your overall burn).

The bottom line is this: If you plan for scaling initially, instead of throwing a bunch of crap together and hoping it will get you to point “B”, the less money you are going to spend as you scale.

To properly address scalability, you have to take a holistic approach, and examine your architecture, your software components, and your hardware configurations. Then you have to deploy capital in an intelligent manner. Otherwise, you could end up like John in our fictitious example – sitting on a pile of code that really wasn’t designed to split up into services across an n-tier architecture. And that, my friends, represents a serious misuse of capital.

If you are scaling by adding new hardware – that is generally a good problem to have – that hopefully means your business is expanding. However, if you are having to rewrite large amounts of code in order to scale – you’ve likely made some very serious mistakes. Too many of those mistakes, and you’ll be dead in the water. You see the latter quite often in the startup world, again, as there is so much pressure to slap something together and get it out the door.

Architectural Concerns
It all starts with the blueprint of your architecture. Note, I am not referring to your functionality matrix/map, your application requirements definition, etc. I am talking about your physical and logical architectures.

When I say physical architecture, I am referring to the physical hardware components that make up your network, application, etc. Things to consider:

• What are your network capabilities? Throughput, bandwidth, etc. Are these scalable from a cost and availability standpoint?
• What are the capabilities and specifications of your web servers, application servers, database servers, etc.?
• How many of these servers do you have at your disposal? Are they dedicated or shared?
• What are your capabilities for handling increased DNS requests, e-mail volume (in and out), etc.?

When I refer to your logical architecture, I am referring to the way in which your various software components connect with and layer into one another. Things to consider:

• Are you properly segregating functional aspects of your application into easily managed and scalable services (e.g. SOA)?
• Are you isolating those services in unique hardware or operating environments?
• Are you building for the short-term, but planning for the long-term (i.e. how extensible will this thing be down the road?)

If you are in startup mode it is very easy to fall into the trap of “getting started” – pushing code and charging up the hill. “Get-to-market” pressure from investors rarely helps. However, many such efforts are met with stiff resistance once the entrepreneur realizes that the “hill” he or she just conquered is actually but a small plateau on a mountain comprised of increasingly steeper slopes. It all starts with a good plan in place!

Having the right architecture in place doesn’t guarantee that you won’t have to eventually add more hardware or write more code. Actually, in the early stages, you may actually spend more money (especially if you are deploying web services on their own servers, etc.) However, the proper architecture allows you to get the most out of out that new hardware and software, as you will be plugging it into a framework that was built with it in mind.

A Note on System Services
Before you start diving off into creating a true services oriented architecture, do yourself a favor and split your system services off accordingly. Things like DNS services and e-mail should be move off of and away from your application’s production environment. I bring this up because more often than not, you find system services being co-located on production web servers.

It is hard to get a sense of an application’s true load on a physical machine when you have a million other things running on it. This becomes even more important if you are planning on using tools like Six Sigma to measure scaling and availability metrics, as you will need to do everything possible to remove potential causes of common/special cause variation.

The Importance of Performance Tuning
Another very important aspect of scaling is performance tuning. Performance tuning is essentially the art of tweaking and fine-tuning your various applications and services in order to maximize their operational efficiency. There are more ways to performance tune a system than you can possibly imagine. Some obvious examples would include:

• Tuning your database server to use more or less RAM (for buffering, sorts, key lookups, etc.), adding indexes on data tables to decrease query times, etc.
• Optimizing your web server’s socket configuration to improve connection times

Some not-so-obvious examples might include:

• Enabling DMA (direct memory access) for a server’s IDE drives to improve seek times (you should be using 15K RPM SCSI drives, but if you aren’t … )
• Making sure your OS kernel (for UNIX/Linux) was compiled with only the services and features you require (i.e. remove kernel bloat)
• Mounting filesystems with the “+noatime” flag, which will prevent the OS from updating the (mostly) meaningless “last access timestamp” of every file you read or write
• Partitioning your OS so that system files, logs, etc. are on their own partition (or even a separate IDE channel or SCSI bus)
• Using the “strip” command to remove debugging/profiling data from common executables to lower their memory footprint (On UNIX systems)

I have known people who have spent thousands on new hardware, only to realize later that their applications or server/OS were simply not optimized. if you don’t have the requisite skills in house to do performance tuning, then outsource this function immediately – it is that important.

Finally, I should mention that performance tuning should not be viewed as a one-time activity. You should routinely profile and tune your systems (both hardware and software).

Network Scalability
Another thing to stay on top of is your service level agreements with your data center or hosting provider. There are fundamentally three or four things to keep in mind:

First, if you are bootstrapping a startup, and you are using a low-cost, shared server, you need to move to a dedicated server solution as soon as possible. Trust me.

Next, make sure that you have the ability to quickly secure additional rackspace and bandwidth as you need it. Having your data center tell you that you are going to have to suffer major downtime because they have to move your servers to a “bigger rack” is probably not a good thing.

Third, make sure that the connections from your servers to the net are burstable. When you have those huge traffic spikes because someone put a mention of your site on digg.com, Slashdot, etc., you’ll want to be able to handle the temporary increase in traffic (without necessarily incurring a large bandwidth bill).

Finally, make sure that your throughput is not capped, or if it is capped, make sure it is capped higher than you think you’ll need. Don’t confuse bandwidth with throughput. Bandwidth refers to how much data your connection can transfer over a period of time (e.g. 100 gigabytes per month, etc.) Throughput, on the other hand, refers to how much data can be flowing through your connection at any given time (e.g. 10Mbps). Think of throughput as being the thickness of your server’s pipe – obviously, a lot more can flow through a garden hose than a soda straw. Same analogy.

To illustrate the point on throughput; a few years back we had a system that began to be wickedly unresponsive. The CPU loads on the server were only about 35-50%, so it didn’t make a lot of sense. It turns out that the connection had been capped at 10Mbps. Anything over 10Mpbs at any point in time had to basically wait in the queue. This caused perceived “slowdowns” by users. Raising the throughput cap remedied the problem, of course.

On a final note, I want to voice my support for co-locating servers rather than using a full-service hosting provider. If you have the skillset in house to maintain the boxes, using co-location can save you some money, and give you more flexibility. Again, in startup mode, every penny counts. If you are in or near Atlanta, I highly recommend my friends down at Capital Internet. I’ve co-located and hosted servers with them for 6 years now, and it has been a very enjoyable, hassle-free partnership.

Advanced Topics
There are a lot of advanced topics that come up in discussions about scaling applications. Things like caching, high availability clustering, network storage (via SANs), and distributed networks. Obviously, those are very specific areas that are beyond the scope of this already ridiculously long blog post. Suffice it to say that there are some very advanced (and expensive) toys out there that can make scaling a lot easier. However, the vast majority of applications/services can be scaled to rather massive proportions if you simply follow the bullet points at the top of this post.

As a quick refresher, in a JAD context, the functionality matrix represents a prioritized collection of the major functional components of a proposed system. These functional areas were, hopefully, agreed to by all relevant stakeholders who attended the JAD session(s) that went into its creation.

Before we go further, it is important to point out something. As much as it pains me to say this, my way is not the only way, nor is it necessarily the best way. It has worked for me consistently in the past, however. Hopefully, at a bare minimum, this article will at least give you a framework on which to build your own method(s), or at least spark your imagination a bit.

Before creating the functionality matrix, I recommend having the scribe, or some other able-bodied person, take the controls of a laptop and fire up Microsoft Excel. Excel represents a fast and easy way to collect and tabulate data. Excel will become even more important later on, when we begin doing some calculations. If at all possible, I would recommend being in a room where the laptop can be projected onto the wall. By doing so, everyone in the room can participate in the creation of the functionality matrix.

At some point during the JAD session, you, as the facilitator, will lead the group through a series of exercises in order to capture the broad requirements of the system. For many new facilitators, this part is the hardest, however, it doesn’t have to be. It really, truly is amazing what happens when you simply ask “ok, what, at a high level, does this system need to do?” The general flow will be as follows:

1. Brainstorm ideas
2. Streamline the ideas by grouping or removing
3. Go to step #1 and repeat as needed
4. Prioritize the final list of requirement categories
5. For each high level requirement area, go to step #1 and repeat as many times as necessary (to capture the detailed requirements).

When brainstorming requirements, it is important to capture each and every idea. Some ideas will be great. Some will be so-so. A few will be completely worthless in the end. But capture them all. By doing so, you are guaranteeing that every voice is behind heard; at least once. Many JAD sessions fail because consensus was not attained, or voices go unheard.

The process of brainstorming is used to quickly and efficiently capture a stream of ideas. It is not a time for discussion, and certainly not the time to argue one’s case. If you catch your participants doing this, it is imperative that you remind them of this! Capture first, discuss later.

It is practically impossible to instruct someone how to conduct a brainstorming session. The easiest way to learn is to participate in one. However, here is a small sample of the general flow of a session:

Facilitator: Ok, we are now ready to capture the high-level requirements of this new system. We are using a brainstorming session, which means no discussions, only ideas! We have 10 minutes for this, so just blurt them out. As you call them out, I will write them up here on the whiteboard. Mike, our scribe, will also capture all of the ideas on his laptop for use later on. Ready? Begin.

John: The system should offer basic RFQ functionality for our buyers.

Facilitator: Ok, good one (writes it on whiteboard).

Susan: We also need a more in-depth set of RFP functionality.

Facilitator: Ok, very good.

Roger: But aren’t those things sort of the same thing?

Facilitator: Maybe, maybe not. Let’s focus on gathering the rest of the requirements, then we’ll discuss the merits of each of them in turn.

Jane: We’ll need a news portal.

Facilitator: Got it.

Jane: And we’ll also need an online supplier directory.

Facilitator: Very good. Let’s keep going. Faster, if we can.

John: Permission marketing.

Susan: Invoicing and billing.

Randy: Broker integration.

Roger: Contract management.

Facilitator: Great! Let’s keep those ideas flowing!

and so on …

If you get the team working in a rapid-fire mode, you will be completely amazed at the sheer number (and quality!) of ideas that come forth.

As a footnote here, it may be helpful to have the team “practice” a brainstorming session first. First, pick some arbitrary inane topic. For example “the best way to hand wash a car,” or “we are building a house, let’s talk about what features this house needs to have in it.” Allocate 5 minutes to the brainstorming session, and let them go through it. This is a good ice-breaker as well. Give them some feedback on their brainstorming techniques at the end, and then move into the real session.

Once the time limit has expired (the time limit is completely arbitrary, although I recommend keeping it to 10 or 15 minutes at the most), and you have “broadly” captured the first round of requirements, it is then time to examine them in turn. Of course, anything the group feels is out of place, or nonessential, can be easily removed from the list.

Frankly, the easiest way to examine the requirements is to simply ask the group to comment openly on the generated list. Most of the time, they will logically begin funneling similar requirements into broader groupings, which is the ideal scenario. For example, RFQ, RFP, and contract management functionality might all end up in a bucket called “transactional requirements.” Things like “receiving” and “shipping” could be logically grouped into a broader “logistics” category. It is up to the group, through your leadership, to determine what the best number and type of categories to use.

Ultimately, you want to have a nice list of 5-10 (or even a dozen) high level areas, with each of them having perhaps a few sub areas within them. Once you’ve done this, you should start a second round of brainstorming. This time, however, the goal is to provide ideas around the current list of functional areas. We do this for several reasons, but the primary reason is that by seeing the list up on the wall, the participants will naturally begin to expound upon those ideas, and come up with even newer ones … or things that perhaps were overlooked in the first round of brainstorming.

Once this 2nd round has concluded, you should again lead the group through an open discussion of the requirements. Again, the priority here is to streamline the list – grouping and removing as needed. You can theoretically repeat this process any number of times, although generally speaking, 2 or 3 times is sufficient.

Once you’ve done this, you should now be in possession of the broad, high-level requirement areas for this system. You are now ready to move into the prioritization phase. We will brainstorm the detailed requirements shortly, but for now, let’s focus on the the high level requirement areas.

Remember in the previous article where I mentioned that having several thousand dollars in fake money would come in handy? Well, we’ve arrived at that discussion! There are lots of ways to lead a group through a prioritization exercise, but I am partial to what is known as the “$20 method”. In this model, the facilitator gives each participant $20 in singles. You can also do this without using actual physical fake money, but it is more fun to do it this way. Each participant can “spend” their $20 on whatever functionality they desire. The more money they spend on an area, the more important that area is to them. At its core, this is a simple exercise in “weighting”.

If John spends $8 on “logistics”, and $3 on “transactional functionality”, then that means John places a higher value on the logistics functionality within the newly proposed system. At the end of the exercise, it is interesting to see what areas people spend their money on. Often times, their areas align politically with their vested interests – so keep this in mind.

As you can see in this partial screenshot of our Excel model (click image to expand), the business side of the equation was represented by three people (John, Susan, and Mike). The technical team had 4 members present (Jeff, Roger, Jane, and Jill). They each spent $20 on the various high-level functional areas (Transaction, Workflow Management, etc.) You can also see some additional statistics to the right, such as simple summations and averages. Of note are the color coded cells to the far right, which indicate which control group favored a particular area – for instance, the technology team spent more money (on average) on Workflow Management than the business team.

I should point out that all of the functional areas are “important” if they’ve made the final list. Just because one area doesn’t get as much attention as the others, doesn’t mean it isn’t critical to the system. It is also important to point out that the final functionality matrix is not necessarily set in stone. Even though we’ve applied some basic statistical methods to capturing, grouping, and prioritizing the requirements, in the end, the business drivers may screen that a certain requirement be bumped up the list, or bumped down. Common sense, of course, should prevail. The functionality matrix is simply a streamlined viewpoint into the system requirements.

The same weighting method can be applied to the requirements under each major category. I generally use a separate Excel sheet for this. As you can see from the screenshot below, the four requirement areas under the larger “transaction” category are also prioritized.

You’ll also notice that at the top of the sheet, there is a small decision matrix. This four quad matrix compares technical complexity on the vertical axis, and business benefit on the horizontal axis. The four quadrants break down as follows:

• Upper left quadrant (high technical complexity, but low business benefit): Definitely not in phase one
• Bottom right quadrant (low technical complexity, but high business benefit): Definitely in phase one
• Lower left quadrant (low technical complexity, and low business benefit): Examine options – Risk averse approach
• Upper right quadrant (high technical complexity, and high business benefit): Examine options – Risk taker approach

Hopefully, this is self-explanatory. The idea is to figure out which quadrant each subfunctional area falls within. To do this, we need to establish some cutoff values. At the top left is a smaller version of the matrix with “cutoff” values in it. These values represent the cutoff values that affect the conditional shading of the cells in the main content area. The idea is that you should be able to quickly scan the content area to see which functional areas fall within each quadrant of the decision matrix. For example, the “contract management” functional area (under the larger “Transaction” category) has a business benefit value of 47.92%.

We got that by taking the average for that line item (10.00 + 8.75 + 10.00) and dividing that number by the total points spent (20, as in $20). That gives us a simple percentage. 47.92% of the business teams money went into the “contract management” functional area. Since our cutoff value is 30%, the shaded color is green (which according to the decision matrix, would indicate that the business folks feel that this functional area should be in phase one of the system). This area clearly has a high business benefit!

However, the technical team’s score is 45.83% – which is over the cutoff value of 25%. This means that developing this functional area will come with a high level of technical complexity (meaning risk!).

So now we have a situation where both sides of the equation (business and technology) feel that this area has a high value. This would push us to the upper right quadrant (shaded yellow). I know this all sounds a bit complicated – and it is. But hopefully this all makes some sense to you.

This post has run a bit long (okay, way long.) I will save the final bit (where we dive into the detailed requirements and finalize things) for another post. I’ll also provide some Excel templates for you to use as well.

Cheers, and happy JAD’ing!

As promised in my post on IT Facilitation Skills, this article will take a more “hands-on” look at conducting JAD workshop sessions. While there is no single “best” way to accomplish JAD objectives, I will try to present a format that has worked nicely for me in the past, in the hopes that it will give you some ideas as to how best to structure your own sessions.

JAD (Joint Application Design), is a software design process which attempts to bring together various business stakeholders (most likely users) and IT professionals in a highly focused workshop environment. Why do we prefer this format? Well, in the old days (gee, all of 10 or 15 years ago) most sets of functional requirements were cobbled up by business analysts and “tossed” over the wall to the software engineers, whereupon we were supposed to plop a handful of arcane ingredients into a big pot, mutter a few incoherent passages from Ken Thompson’s UNIX Philosophy and stir until some magic software solution poofed out of it. Alas, the road to JAD is paved with the morasses of projects past that have stemmed from poor requirements gathering.

Of course, the problem with this is obvious – business analysts are generally not technologists, and vice-versa. Business personnel are not necessarily adept at taking into account technical constraints/limitations, best practices, or the newest innovations. Likewise, technologists have not historically been overly proficient at cobbling together business requirements. It has been unequivocally proven that by bringing both sides of the house together to define the requirements that a vast majority of common problems and misunderstandings can be dealt with up front (where they cost $1,000), rather than in the 47th week of the project (where they cost $1.3M).

Make no mistake, JAD is not the panacea for software development, or even requirements gathering, for that matter. It can, however, provide a pretty good path of least resistance. Theoretically speaking, the big advantage of JAD is accuracy, or the ability to deliver a system with the functionality that is actually needed. In addition, JAD affords you with a dramatic shortening of the time it takes to complete a project (due to having tighter functional specs up front). Of course, your mileage may vary – I have found that having a very experienced facilitator goes a long way toward determining the eventual impact of the session(s). The use of JAD also improves the quality of the final product by focusing on the gathering of accurate requirements, thus reducing the likelihood of errors that are expensive to correct later on.

Much of this information is taken from “consulting 101″ – if you have some business background, a lot of the initial concepts may be familiar to you. But keep reading – we’ll get vertical soon enough. I apologize in advance for this article’s length, but I don’t believe in “blogging for blogging’s sake.” I prefer my blog entries to have substance and value. I hope you will agree!

Workshop Preparation

As any experienced facilitator will tell you, quite a bit of the work effort you will put in will occur well before the sessions even begin.

The very first thing that the facilitator needs to do is to ensure that he/she fully understands the context of the business problem being addressed. Note that I did not say that “granular industry knowledge” was necessary, nor did I mention anything about the facilitator having technical prowess. The focus here should be on understanding the context of the business problem.

What is the problem we are dealing with here? Who is affected by it? What are we trying to accomplish. What stakeholders will be affected by this system, and how many of them will attend the session(s)?

Once you have a grasp of the breadth of things, you will be in a position to direct the subject matter experts to drill down vertically into the problem. Interviewing various stakeholders is a good way to cobble up the knowledge that you need – this can usually be done pretty quickly.

Don’t be surprised, however, if you get to day one of the session and realize that you overlooked an important aspect of the group dynamic or the business problem. This can (and will) happen. Just prepare yourself as best as you can up front, and be prepared to augment that knowledge with things you pick up on-the-fly during the session.

As far as physical, tangible items goes, here is a list of items that I have found to be helpful during a JAD session:

1. Large whiteboard (preferrably mounted on wall), markers, eraser
2. Large Post-it style easel pads
3. Projector (for use with workshop slides)
4. Laptop with Microsoft Office suite (Word, Excel, etc.), or some other set of tools (Corel Office, Appleworks, Lotus Smart Suite, OpenOffice/StarOffice, etc.)
5. Several thousand dollars in fake money (I’ll explain later – ha!)

Workshop Agenda

As a starting point, I like to break down requirements gathering into two distinct phases – both phases consisting of facilitated JAD sessions. In the first JAD session, we will gather very high-level requirements and funnel them into what I call a prioritized functionality matrix. In the second session, we take that functionality matrix and generate draft use-cases from them.

For the first session, which I think is the most critical, I tend to follow a high-level agenda similar to the following:

• Introductions:
• Session Roles
• Ground Rules
• Objectives
• Reaffirm:
• Vision
• Business Drivers
• Business Context
• Baseline Functionality Matrix
• Wrap up

We will discuss each of these items in more detail below.

For the second session, the agenda would look something like this:

• Quick Review of Functionality Matrix
• Use Case Mini-Tutorial (to educate the stakeholders)
• Use Case Identification
• Draft Use Cases
• Wrap Up
• Questions
• Next Steps

NOTE: We’ll cover the detailed creation of the functionality matrix and use cases in separate articles.

Session Roles

Before beginning your session, the facilitator should work with the various stakeholder participants to identify those individuals who will fill certain critical session-specific roles. These roles are discussed briefly below, but it is incredibly important that these people be identified in advance, and are not surprised by their assignments on the day of the session!

The Facilitator: This is the easy one. This is the person leading (facilitating!) the session. For more info, see my article on IT facilitation skills.

Timekeeper: Simply put, this person is responsible for watching the clock and alerting the facilitator when sessions (or segments of a session) are approaching the pre-determined cutoff point. For example, the facilitator may decide to give the group 5 minutes to brainstorm solutions to a particular problem. The timekeeper might alert the facilitator when the clock hits 4 minutes (1 minute warning), and then once it hits the 5 minute limit. This can be done silently by a nod of the head or raising of the hand, or even verbally. Maintaining adherance to a time schedule will help keep people focused, and assist in keeping things rolling along.

Scribe: Outside of the facilitator, this person probably has the most instrumental role in the success of a session. The scribe is essentially the secretary – they keep track of all major outcomes of the session. This is similar to meeting minutes, but is more detailed. This can be done via pen and paper, but in most cases, it is probably best done on a laptop, where the person can switch around between Excel, Word, and other applications. Depending upon the makeup of the stakeholder groups in the room, this person may also be the one responsible for drafting the functionality matrix (in real-time, with input from others).

Parking Lot Attendant: Invariably, things are going to arise that are out of the scope of the session’s guidelines. In order to keep from getting off track and derailing the session, the facilitator may reserve the right to “park” those issues or questions in the “parking lot” so that they may be addressed later. I generally like to address the parking lot issues list at the end of the session (or at the end of each day, if the session will span multiple days).

SMEs/Participants/Devil’s Advocates: Everyone in the room, including the people filling the roles above. Remember, the more valued input you have, the more productive your session will be!

You may decide to come up with other ancillary responsibilities and roles – this is your discretion as the facilitator. Just be careful not to over-delegate to the participants – you want the principal parties to be focused on the task at hand, and not immersed in some other duties that will detract from this.

A Bit on Brainstorming

For those of you who may be new to the concept of brainstorming, I will simply refer you to the great entry over at WikiPedia. However, I will state that I am a big fan of whiteboards, and an even bigger fan of the big Post-It easel style pads. Those are great for capturing ideas quickly, ripping them off, sticking them on the wall, and moving along to the next idea.

Ground Rules

It always pays to lay out the ground rules up front, so that everyone is on the same page from the beginning of the session. What these rules are, in the end, is up to you as the facilitator. In general, I like to use these as a starting point:

• Stay focused
• Be honest
• Respect opinions
• Stay on time
• Use the parking lot
• Brainstorming rules
• There are no bad ideas
• No criticism of an idea
• Strive for quantity, not quality
• Strive for creativity, think out of the box
• Build on the ideas of others

After I explain these rules to the group, I generally ask if anyone else in the room has any other ideas for ground rules to add to the list for the session. Occasionally, someone will throw something out there (no checking email during the session, turn off cell phones, etc.). Calling for other ground rules is a great way to begin empowering the participants, and to set the tone for the session.

Objectives

Once the session roles and ground rules have been announced, I like to immediately state what the official “objectives” of the JAD session are. You would be amazed on the first day at how many people actually proclaim things such as “oh, I thought that JAD meant we were going to build the application together” or “I thought we were just going to talk about the need for the project, I didn’t know we were designing it!” The objectives I usually use as a starting point are:

• Define the system scope
• Draft use case models for primary scenarios
• Define the domain model/glossary

The “system scope” is going to be defined largely by our functionality matrix (deliverable from the 1st session), whereas the use case models and domain model/glossary are going to be outputs from the 2nd session.

You may feel the urge to add additional objectives to this list. My only advice to you here would be to make sure to tie any new objectives to specific deliverables that will come out of your session(s).

Reaffirmations

I like to go through a series of “reaffirmations” before I lead the group into unchartered waters. I want everyone in the room, from the lowliest coder to the most senior executive, to understand why we are building this product, who it will affect and benefit, and what value it will bring to the market or stakeholders.

Reaffirming the vision of the project is first on the list. The “vision” could be anything, really, but it should be related in some way to the project. If your organization is following a PMI style project plan, then this is probably the project charter. If you are building a software platform for a startup company, then this is probably the actual vision statement of the company itself.

Next, I like to reaffirm the business drivers and business context. Why are we doing all of this again? What business problem will this project solve? How big is the market we are talking about here? What is the value proposition?

Let’s state all of this as a matter of fact – level the playing field by educating (or-re-educating) everyone in the room.

Critical Success Factors

At this stage in the game, we really haven’t done anything toward the actual design of the system. This is a good thing. All of this groundwork will serve us well as we move further into the process.

After the reaffirmations, I like to do a quick 5-10 minute brainstorming session where the group throws out potential critical success factors (CSFs). This allows the stakeholders to identify very succinctly those things that will make or break this project or product.

Consider these examples:

• Head of Marketing: “It is imperative that we have 24×7 uptime – availability will be critical to our customers!”
• Lead Developer: “We should implement this system using an open architecture in order to easily integrate new technologies or functionality in the future.”
• Business Analyst: “It is important that we have executive buy-in and participation from our key industry design partners.”

You get the idea. Don’t camp out in this stage – this should be quick, but also helpful in setting everyone’s expectations and understandings.

Current Processes (As-Is) / Future Processes (To-Be)

Once we’ve identified the CSFs, I like to do a very brief “as-is/to-be” analysis. This may or may not be something that every project will use, but I do find them useful in two cases. If you are designing a new product (or working with a startup company), then the “as-is” will probably represent the current state of the industry. If you are doing business process reengineering, and your software product will be a part of the future processes, then the “as-is” will probably represent the current business environment.

In either event, the questions you will ask will be the same:

• What are the major flows of information?
• Who is involved?
• How are stakeholders doing business now?
• What do those processes look like?
• What technologies do they use?
• What tools do they use now?

Additionally, in either event, the “to-be” part represents the deliverables/outcomes of the project itself.

Why do we spend time on this? I have found that if all of the stakeholders, including the development team, have an understanding of where we’re going, and what things look like in the current state of affairs, the end result is a much more cohesive and functional product.

I usually spend 45 minutes on the “as-is” area, and another 45 minutes or so on the “to-be” area. I would recommend a break in between!

Assumptions

Getting weary yet? Don’t fret, as we’re nearly done with laying the groundwork! Only one last item remains. The final area that I like to touch on before leading the team into the design process is the generation of relevant assumptions. These are generated via a short (10-minute) brainstorming session. These should represent things that the project design team are assuming to be true, or in place. Generally, they represent inputs to the system, or owners of certain things. I like to brainstorm these up front, so that the group doesn’t keep disclaiming things all throughout the session.
Examples:

• “Our major design partners will be responsible for content management within the product catalog.”
• “The initial database content will be provided by Acme Corporation, one of our key partners.”
• “Marketing will provide a list of 10 qualified beta customers before we launch.”

One caveat here. If you sense that the group is starting to “haggle” or argue over the assumptions, then they apparently are not assumptions to begin with. In such cases, either choose to put the issue in the parking lot, or make a note to revisit it at a later time during the session, where it can be discussed in a more streamlined manner.

Functionality Matrix

Congratulations, you’ve made it through all of the groundwork exercises. We’re ready to begin taking all of the information from the brains of the participants and funneling them into a prioritized functionality matrix.

The functionality matrix is essentially a prioritized grid of the major functional areas within the proposed system. The matrix can be developed using whatever tools you wish, although I recommend Excel, as I find its ability to do quick calculations for me (important later on) and conditionally formatting cell colors to be handy.

However, given the sheer scope of what is involved in creating a functionality matrix, I will save it for a separate article. Don’t worry, that article has already been written – it will publish a week from the date of publication on this article, so be sure to check in then for a more detailed walkthrough. In that article, I’ll also share with you the reasons for bringing several thousand dollars in fake money to your JAD session. :)

Wrap-Up

Hopefully, the functionality matrix came together in short order. Even the most demanding set of requirements should be able to be pulled together in no more than 2 days (but most often in an afternoon). Don’t forget to address any issues that were plopped into the “parking lot” before going home!

Next steps: Use Cases

Let the participants know that all of their hard work is about to be funneled into the next phase, where the individual use-cases will be developed. These will provide the “substance” behind the functionality matrix.

My final thoughts

I know this was a lot of information, and if you’ve made it this far, then I commend you! While I think all of these ideas are a fantastic way to approach your JAD sessions, I would like to say again, however, that these are not the only areas that you can explore. Use your imagination! Every session will be different, because every project is different. Just follow your instincts, and I’m sure you’ll do fine.

In order to make things a little easier for you, I am including a bare-bones Powerpoint presentation that lays out the flow of this type of JAD session (coming later this week!).

Cheers, and happy JAD’ing!