Seeing the Forest and the Trees by Glen Kendell
"Only the spoon knows what is stirring in the pot."
-Sicilian Proverb
Production as a System
Production information systems are comprised of firewalls, servers, software,
users, data, state and more. The combination of these individual components
creates a mind-boggling array of possibilities that exist only in the
production environment.
Yes, it is complex. But more than that, when you add all those pieces together
you have something new - a system. Systems theory states that the interaction
of component parts creates an entity distinct from the combination of parts
themselves. The whole literally is greater than the sum of its parts.
One important characteristic of a system is that you cannot recreate it. Take a
forest for example. A forest is a system comprised of trees, soil, insects,
animals and so on. If you take those same components and try to recreate that
forest, you won't be able to. The climate, watersheds and environment also are
part of the forest.
Production systems are the same way. A production system cannot be fully
recreated in the test lab. Anyone who has worked with software developers knows
this from first-hand experience. The plea often heard is "My code works just
fine on my workstation" when it doesn't work in production.
Another distinguishing feature of a system is how it responds to change. Cause
and effect are not always intuitive. Even small changes may produce unexpected
results. A new species of insect in a forest may cause an explosion in the
growth of underbrush that in turn makes the forest more susceptible to fire.
Network administrators may update the firmware on the firewall to accommodate a
feature associated with a new software release. The new firmware has a security
feature that blocks certain types of users at the firewall. However, everyone
assumes the problem is with the new software release so it is rolled back, yet
the problem persists.
Points of View
When you look at a forest, what do you see? Do you see trees? Do you see shades
of green? Or do you see the beauty of a complex system at work?
Let's take a moment and pull over at a splendid scenic viewpoint overlooking
our production system. We just happened to bring a carload of coworkers along
with us. As we stretch our legs and take in the panoramic vistas, here's a
sample of what is heard from the group.
Software Developer:
"Wow. Will you look at all that software! And over there - more software! See
that one passing a file descriptor across the process boundary? I wrote that.
And see that area over there with all the tangled threads? That's the old
software that was written before I got here."
Network Designer:
"Wow. Check out all the packet flocks flying in perfect formation. I designed
the flight paths myself. Hey, what are those fragments doing over there? Let me
grab the binoculars and check out their ninth octet. Too bad, we just missed
their last hop."
Business Manager:
"Wow. You can really see the growth in recurring users from up here. If we hit
our third quarter numbers, we're going to have to increase our capital
expenditures in the next budget to keep up with the demand. Hard to imagine
there was no revenue here just a few years ago."
Customer:
"Wow. So this is where all of our critical financial data is stored. Hey, what
would happen if there were a fire?"
Seeing the Forest and the Trees
If you have a lot of information without grasping the bigger picture it could
be said that you "can't see the forest for the trees". If you understand the
gist of things without having mastered any of the details it could be said that
you "can't see the trees for the forest". These clichs apply to any system. Is
there a way to see both the forest and trees?
Add Context to Reporting
Monitoring and reporting on your production system is vital. However, reporting
on the right information can be harder than it seems.
Are you using pre-fabricated "canned" sample reports? Or have your reports been
specifically designed to capture the things that are truly of interest?
Re-visit old scheduled reporting routines. Communicate with the people actually
receiving the reports. What information is relevant? What is missing? Narrow
down reporting to focus out what is useful, not just what the reporting
software can do.
Do Modeling
Though it is impossible to completely duplicate a production system, that
doesn't mean you can't model it.
An ideal testing setup will include a test lab and a pre-release staging
platform. The servers in the test lab should be configured as close as possible
to the actual production servers. One common problem is security configuration.
Developers often write software using full root or administrator privileges,
and then attempt to add security later. Security should be integrated from the
beginning in a way consistent with how it will be on production.
The staging platform should be located on the production network. Testing
software on a pre-release staging platform in the production environment helps
pinpoint problems that crop up only in production. Operating system and
security hotfixes can also be tested on the staging server to ensure
compatibility with application software before being rolled to the production
servers.
Change Slowly
We've already learned that systems are sensitive to change. Even small changes
can cause drastic consequences.
To minimize the impact of change, minimize changes. In other words, don't go
changing a bunch of things at the same time.
How often do you hear someone say "Well, since I'm having a heart bypass I may
as well get my appendix out while I'm at it"? A major application software
upgrade over a holiday weekend is not a good time to upgrade the firewalls too.
Only change one thing at a time. By changing slowly, you allow the system a
chance to absorb the changes. If something goes wrong, you'll have a better
chance at diagnosing the problem and recovering from it.
About the Author
Glen Kendell is a network architect and owner of Release to Production. He
publishes a monthly newsletter called In-Production: Achieving True High
Availability. http://www.r2pnetworks.com
|
