HallmarcDotNet

Tue, 12 Jan 2010

Like a Sinking Ship

Scheduled Downtime

Due to the need to make changes and corrections in our datacenter, we periodically schedule a downtime or system outage. This may be a single system, a few systems, all of a particular type of system, or all systems. This last weekend, due to a proposed change to our Storage Area Network (SAN) infrastructure, we scheduled a full downtime for all information systems across our environment.

Full system downtimes are highly disruptive to the operation of our facilities, as they mean that the network, file servers, email, printers, application servers, Internet access, and other functionality is unavailable for the duration. As a consequence, they should be necessarily rare events. Moreover, due to the nature of our business (healthcare), other functions within the organization cannot simply “take the rest of the day off” when we have a downtime event planned, as patient care cannot be deferred. Neither can we calendar a downtime during off-hours, as we are open all day, every day, regardless of holiday, weather, season, or other external cause. Moreover, some of our departments actually schedule additional staff during downtimes to compensate for the additional workload created when automated systems are unavailable. We must then be both mindful of the disruption the lack of access to various technological augmentations will cause, as well as thoughtful about the budget requirement of various departments bringing in additional staff.

However, sometimes full downtimes cannot be avoided, as some kinds of upgrades or even routine maintenance require that core infrastructure be taken offline. If our hospital had fully redundant systems in every area of storage, networking, application server, etc., we could have very short downtimes or even non-disruptive downtimes by rotating primary and backup systems on and offline. However, our organization has chronically underfunded this infrastructure; as a result, very few of our systems are truly redundant, and those few generally rely on systems that are very definitely not redundant.

Bearing in mind these assorted challenges, to minimize the disruption to the other departments and reduce impact on patient care, we generally schedule our unavoidable downtimes for very early on Sunday mornings.

As you may imagine, then, advance communication about a scheduled downtime is critical across our organzation as well as internally within the Information Services department.

This last weekend’s outage, however, demonstrated how poorly this can be handled.

First, within our department, we neglected to confirm with the involved staff that we were having a downtime until the Tuesday prior to the event. This meant that several of us were completely unaware of the requirement to be at our facility at 2:00 a.m. until only a few days before the event. Further, we failed to send a reminder to the remainder of the hospital until late Wednesday morning, which meant that other departments had to scramble to find sufficient staff and to ensure that their procedures were updated prior to the event.

Second, once the downtime was brought to our attention within the department, we were neither consulted about what sorts of maintenance could or should be performed during the window, nor provided with a list of what management planned to execute during the designated time. The technical leads were essentially ignored during this stage.

Third, on the morning of the event, the part-time employee who has taken an informal leadership role in the SAN migration process and at whose behest this particular event was scheduled did not arrive until nearly 3:30 a.m., almost 90 minutes after the rest of the team was on site and ready to work (I was actually at the datacenter at 4:00 p.m. Saturday, but that was my choice to ensure that all of my systems would be ready).

Fourth, although we did eventually receive a (very short) list of the items to be reconfigured and tested, the individuals involved in the planning did not communicate the specifics of their agenda with those who would be implementing or supporting the configuration and testing. Further, those whose systems were being tested did not have printed documentation of the steps they would be executing to ensure that their systems would come back online after their initial shutdown, nor did they have documented procedures for backing out if the testing showed that the new configuration would be unworkable.

Fifth, while we did have a conference call with the various affected departments at the end of the window, the system testing had barely begun, and our secondary objectives were left unstarted. All systems were scheduled to be restarted by 7:00 a.m., but we only began to bring them back up at around 6:45 a.m. As a result, each department in turn was asking whether their systems were up well before they could possibly have been ready. This added to the confusion of the event, as we were being bombarded with requests for status updates while we were frenetically starting infrastructure, systems, databases, and application servers.

Sixth, once the systems were up, we did not perform a formalized system check to verify that functionality was working as expected. As inevitably happens in such circumstances, we did not discover at least two important glitches until late Monday afternoon.

Finally, despite management being familiar with these many issues, as well as the obvious negative fallout after the maintenance window, we have not held any kind of a meeting or less formal debriefing process to determine how the process could be improved for our next downtime.

Not only did we do it wrong, we have learned nothing as an institution from our errors.

Wed, 06 Jan 2010

That’s not very funny

Stand up Meetings

One of our teams has a daily, mandatory “stand up” meeting every morning at 8:00 a.m. Stand up meetings should be short, a self-enforcing principle because everybody participating is, after all, standing up.

The scheduling and purposes of stand up meetings vary from organization to organization; but generally they tend to be daily or twice daily, with an agenda focused around routine information sharing so that indivduals within a team understand what the rest of the group is working on and what obstacles they face. In Agile development methodologies, stand up meetings are designed to keep a team focused on the project by reducing interruptions during flow moments, as well as ensuring that problems don’t remain hidden. Because Agile methodologies require the division of a project into small, quickly achievable tasks (on the order of two weeks from start to finish), each individual’s summary should similarly be short. Within my organization, however, the daily stand up meeting has transmogrified into something entirely else.

First, the primary purpose of the meeting’s scheduling appears to be getting everybody to show up at the same time each day. Eight in the morning is not totally unreasonable, of course; but this team includes members who live up to seventy miles away, across a major metropolitan area. You do the math. When taking into consideration the requirement that team members also attend a daily 4:30 p.m. status meeting (at least it’s not called a stand up!), this makes for a long day.

Second, although still called “stand up”, these meetings now involve sitting down. This is more than a superficial difference: as previously noted, stand up meetings are supposed to be short, and sitting down permits the group leader to extend the agenda to a much greater length. In particular, the stand up meetings are at least a half-hour long. Usually longer.

Third, rather than being limited to a brief synopsis of what each team member has accomplished since the previous meeting, is working on now, and how to clear challenges lying in the path of success, the meetings have devolved into a litany of issues confronting the group managers, and a recitation of the many ways in which the project is behind schedule.

Fourth (irrespective of the meeting scope creep), rather than seeking ways to remove obstacles and promote project success, the managers spend a large fraction of the time in the meeting pointing fingers at one another and the other members of the group. As one of my colleagues recently reported about the meetings, “Yeah, that bus came gunning for me again today.”

Finally, it is unlikely that anything of significance can be achieved between the end of the 4:30 p.m. daily status meeting and the beginning of the 8:00 a.m. stand up meeting. Logically, then, this appears to be a thinly veiled method of forcing a (theoretically) highly-skilled professional staff to spend a minimum of nine hours daily on-site. Again, not an unreasonable expectation; but highly disrespectful to the individuals involved.

In other words, these stand ups are just not very funny.

Wed, 30 Dec 2009

Close the Door

Noise in the Workplace

I work in a mid-sized room with two sets of six cubicles, occupied by ten guys.

Yes, they’re all guys. There are no women on our team, which is the possible subject of a future rant.

Two sides of the roughly square room have large, square windows. A third side is a simple steel-framed drywall-sheathed wall. The fourth side holds two closets and a door to the hallway, which runs like a capital “L” through the center of our department.

Although our room is carpeted, the hallway outside is floored with industrial-grade vinyl tile, which means that sound carries easily. Additionally, directly across the five-foot-wide corridor from the door to our work area is the entrance to the department conference room, also floored in the same hard tile. The hallway turns a corner right next to the conference room door.

As a result of this acoustical arrangement, if the door into our work area is ajar even an inch, sound from the hallway and conference room bounces directly into my cubicle, which sits directly in the path of the angle of incidence -> angle of reflection for all sound in the department. Crowds of up to a dozen people often congregate there while waiting for a meeting to end, or after leaving a meeting.

Justifiably, then, I prefer that the door to our work area remain closed.

Unfortunately, although the door is solid wood but well-balanced, there is no automatic mechanism to close the door. It has to be closed by careful application of force, lightly enough that there’s no echoing bang through the department, but firmly enough that it fully latches.

More unfortunately, I don’t have a line of sight on the door, so I can’t just look up, identify, and give the evil eye to the offender. As a partial result, management, executives, and other teams within the department remain blissfully unaware of how disruptive it is to deal with the noise — All of which leads us to today’s anecdote.

Our CIO periodically wanders through the department (logically), although his office is four floors up and one tower over. Typically, he’s on his way to or from a meeting in the conference room across the hall or in the director’s office. Frequently on one of his peregrinations he will drop by our team’s work area. This wouldn’t be a problem, except he always leaves the door open.

Yesterday, he paid one of his typical visits. As usual, the first time he entered the room and exited again, he left the door open. I got up, not knowing who had left the door open, and closed it.

There have been days when I would have stood up and said, “Do you have some sort of a messiah complex? You believe you can walk on water, and you were born in a barn?” However, yesterday was not one of those days.

Fortunately.

Not a full minute later, the CIO walked in, again leaving the door wide open. I again stood, walking to the door. There, I encountered the Stubble-headed One exiting another (absent) worker’s cube.

“Oh,” I said. “Are you here for a meeting with other high muck-a-mucks? Are you waiting for the other grand pooh-bahs to arrive?”

“No,” he replied. “It’s just a normal, productive meeting.”

As he said this, my mind raced a bit. First, who thinks of meetings as “productive”? Apparently, our CIO does, although why he would feel a need to announce this remains a mystery. Second, how do I explain why I’m standing between him and the exit, blocking his intended egress? Third, should I ask him to remember to close the door next time? Fourth, how would he react to such a request? Fifth, now that he’s moving toward the exit, how do I gracefully shut the door (I’m holding the handle, so I can’t just walk away) without making it look like I want him to leave? Finally, now that my flow is completely disrupted, how will I return to a productive state?

“Uh,” I said, succinctly. “Well, thanks for stopping by.” I flattened myself against the open door to allow him plenty of space to move out into the corridor.

We stood there, looking blankly at one another, then avoiding eye contact. What was he thinking? A full thirty seconds elapsed. Finally, he turned toward the hallway and stepped out. I gently closed the door behind him.

A close call.

Mon, 28 Dec 2009

Management by Dashboard

Determining Metrics and Graphing Datasets

For about three weeks, one of the software products running on hardware I support has been performing poorly. Users have been complaining that it takes ten or fifteen minutes to do simple things like generate a pre-defined report.

This is bad, of course, for a variety of reasons.

The perceived slowdown began at about the same time as we migrated the storage (think virtualized hard disks) for the server this product runs on to some new (and on paper at least, much faster) hardware. This is called a SAN migration. The correlation suggests a relationship between the SAN migration and the poor performance. Therefore, it is my responsibility to tweak various configuration parameters, reallocate storage, etc., to improve the sytem performance to an acceptable level.

Last week, after exchanging various email messages and phone calls with the software vendor’s support team, we concluded that the performance problem is almost certainly due to the SAN migration. Further, we concluded that changing the storage configuration to include three logical volumes (again, think virtualized hard disks) instead of the one logical volume we configured during the SAN migration and changing the size of the datastream the operating system accesses each time it reads from or writes to the logical volume would improve performance.

As doing this wrong would have really ruined the Christmas holiday for a large number of people (not least me), management agreed that the change could happen today. Thus far, everything was good: we had a reasonable expectation of when the change should be made; the changes could be made while the system remained up and available; and the modifications, although time consuming, were relatively simple.

After a few minor hiccups this morning, I began the reconfiguration and all seemed to be progressing according to plan. I emailed the management concerned with the change to let them know this, and received this in reply:

	

	From: [REDACTED] 

	Sent: Monday, December 28, 2009 11:17 AM

	To: Hall, Marc; [REDACTED]; [REDACTED]; [REDACTED]

	Subject: RE: [REDACTED] follow-up
	
	Marc, [REDACTED], and [REDACTED]-





	Thank you.  I will let [REDACTED] know this migration is in process.
	Are there any metrics that could be graphed to show [REDACTED] -before
	and after migration?  Thank you.
	

Superficially, this is not a difficult or complex request. Except the data I have been collecting on the system since the first performance complaint came in is not in a format that lends itself to simple spreadsheet analysis. In fact, it will require significant massaging to even become spreadsheet data. Additionally, these metrics will be, essentially, meaningless to someone unfamiliar with how data input and output operations are measured and collated. Further, had I known that management would want graphical data, for the last three weeks I could have been working on a suitable parser to change the raw stats into something management-digestible. Although I am busy, this would have been a much less stressful way to approach the problem.

Finally, management has been balking for years at the idea of buying appropriate automation tools to do precisely this sort of analysis; it gets brought up budget cycle after budget cycle, but the funds are always cut when it comes time to actually make a purchase.

Thus my reply:

	There are OS-level IO metrics, but they're not easily spreadsheet-able.
	I'd guess seven or eight uninterrupted hours to massage it into 
	something graphable. Seven or eight uninterrupted hours isn't something
	I have lying around right now. In any event, shouldn't users be the 
	judges of whether [REDACTED]'s perceived performance is acceptable? 
	

A little overly-direct, perhaps; but I was not pleased by the request’s apparent thoughtlessness.

Management’s response:

	From: [REDACTED]

	Sent: Monday, December 28, 2009 1:00 PM

	To: Hall, Marc

	Cc: [REDACTED]; [REDACTED]

	Subject: RE: [REDACTED] follow-up
	
	Marc,





	Thank you.  Yes, the end user should be able to verify an improved 
	performance. I would like to be able to support the view with system 
	metrics, though, in the near future. Thanks.
	

So she wants pictures useful to non-technical management. I am, frankly, not certain how to proceed. As this blog entry left out huge swaths of relevant information in the hope of keeping it readable, you must imagine the complexity involved. Providing a graphical representation of the appropriate metrics is a major undertaking, simply from the perspective of deciding what to leave out; and I haven’t been a spreadsheet jockey for over a decade.

Wed, 23 Dec 2009

Peace and quiet

The Purpose of a Reception Desk

Today on my way to lunch with a few of my co-workers, we stopped by a neighboring department to pick up a colleague. Upon opening the door, I noticed an unfamiliar person sitting behind the reception desk. As the previous regular occupant of that chair had only recently been terminated under mysterious circumstances, my initial thought was that the new person must, logically, be the new department administrative assistant.

Naturally (being the thoughtful, outgoing, and friendly person I am), I acknowledged her and asked whether she was the replacement. Before she could answer, I followed up with a running joke about whether an announcement of her arrival had been sent to the entire Exchange Users group (the Exchange Users being everyone in my workplace — from the CEO on down to the lowest-paid toilet cleaner — with an email address, a list which has lately seen a spamorific number of “Buy this!” and “This tiny little department will be closed on Christmas Day!” -type messages).

Unfortunately, my initial impression of this woman’s role was completely and utterly wrong.

“I’m [REDACTED], and I’ve been here consulting for about three years,” she said, referring to her fairly senior-level role as an outside advisor to our CIO.

“Oh,” I said. “They don’t let me out much. Are you proctoring?” The last familiar person I’d witnessed sitting in that spot had been ensuring that some certification-test-takers remained untainted.

“No,” she said. “I’m here so I won’t be disturbed.”

“Oops!” I said. I did not apologize. What does one say to a senior executive, sitting at the reception desk, who thinks that’s a good place to remain uninterrupted?

The mind boggles.

Tue, 22 Dec 2009

Out-of-office

Moral Authority

One common failing of management is to assume that titular authority is all that leadership requires. However, a job title and position description really aren’t enough, by themselves, to lead.

While to the foolish or inexperienced they may initially appear related or even congruent, true leadership and titular authority are, in fact, orthogonal concepts. This false first impression, moreover, does not bear significant scrutiny in an environment that includes intelligent, self-motivated, and experienced subordinates, as may be typically found at the level of technical lead within an IT organization.

Allow me to draw your attention to a recent out-of-office-reply received in our department. Although names have been removed to protect the guilty, the reply was sent from the email account of the director who established our email signature and out-of-office message policy as well as our voicemail greeting policy. The creation of these policies was among the very first official acts that she executed once hired:

From: [REDACTED]
Sent: Tuesday, December 22, 2009 2:07 PM
To: [REDACTED]
Subject: Out of Office AutoReply: Time out process

Thank you for your request. I am out of the office today, but planning to return on Wednesday, December 23, 2009.
If you have a computer or telephone maintenance request, please contact our IS Service Desk at extension 4968. Thank you.

Brief, professional, to the point. So what’s the problem?

Allow me to quote the relevant sections from the policy:

Information Services Standard Email Signature Expectations

All IS Outlook email signatures will follow this format in black Arial 10 pt font:

Name
Job title
[organization REDACTED]
[address REDACTED]
St. Louis, MO [ZIP code REDACTED]
tel: 314-[REDACTED]
cell: 314-[REDACTED]
fax: 314-[REDACTED]
email: xxxxx@[REDACTED]
 Stay Green, keep this email paperless

Information Services Standard Out of Office Email Message Expectations

All IS Outlook Out of Office Assistant greeting will follow this format:

I will be out of the office until day, date. If your request is dealing with a computer or telephone issue, please contact the IS Help Desk at extension 4968. Thank you.

Leaving aside the pervasive bad grammar, passive voice, and overly-wordy language (and perhaps it’s my background in the legislative process), use of the verbs “shall”, “will”, “must”, “can”, “should”, and “may” each conveys specific and unambiguously different meanings. Further, after this policy was published, management made multiple follow-up calls, paid personal visits, and sent multiple follow-up email messages to both out-of-compliance individuals as well as the department as a whole. These communications emphasized the need for conformity and uniformity, and repeatedly pushed the message that precise and accurate adherence to the policy was expected. No “close” or “individualized” variations were to be permitted.

Not a problem, really, although the emphasis so early in her tenure on such a nit-picking triviality seemed to me like a misplaced use of her limited time.

However, on top of this “violation” of her own policy regarding out-of-office messages, know this:

Since October, all earned time off (vacation, holidays, etc., all in one big pool) for the department is on hold “indefinitely”.

In other words, this director has told everyone else that during Christmas week, the rules apply to them, but not to her.

As a result, this reply stands out as a lesson in how not to do it.

Mon, 21 Dec 2009

Quotes of the Day

How to upset and distract your co-workers

On the wall of my office cubicle I keep a medium-sized whiteboard. The whiteboard is designated in the upper left-hand corner as “QotD”, which stands for Quote of the Day. Most work days I refresh the quote with something topical or thematic, although there are days when it isn’t updated, or when I’m lazy and use an unattributable aphorism.

For example, thoughout the first two weeks of December, I simply appended an asterisk (*) to the “QotD” title, and then placed a common disclaimer or catchphrase at the bottom of the whiteboard, e.g., “batteries not included”, “some assembly required”, or “for external use only”.

That was an easy two weeks, as I didn’t have to find and document any of my quotes.

Most days, however, I do try for something more meaningful. Among the gems I’ve included on my Quote of the Day board are:

“If the automobile had followed the same development path as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year, killing everyone inside.”

— Robert X. Cringely

“If all economists were laid end to end, they would not reach a conclusion.”

— George Bernard Shaw

Occasionally, I include a bon mot from one of my co-workers:

“Once the data leaves the back of the server, it’s your problem.”

— Alvin Lamkey

“Just conform. It’s so much easier.”

— Martha Provaznik

As the former editor of such world-spanning publications as the C.A.S.H. Register, as well as having myself been quoted in the Los Angeles Times, the Sacramento Bee, and the San Jose Mercury News, I understand how important it is to get a quote right. I’m generally very conscientious about attributions and verifiability, even in a demi monde that consists solely of my office mates.

Today, however, I managed to offend the Computer Services Manager by quoting him. His comment was relatively innocuous; but apparently, he was dissastisfied with how it reflected upon him:

“I don’t have time for useless meetings.”

— [REDACTED]

I wasn’t aiming for [REDACTED] with this quote, nor was I aiming at anyone else with his comment. I simply thought it an elegant statement of how many people feel when asked to participate in yet another discussion, when the probability of conclusive decision is low. In fact, if anything the criticism implicit in [REDACTED]’s comment was aimed at me, since he made this statement when he declined a meeting invitation from me.

Not ten minutes after I had written it on the QotD whiteboard, however, [REDACTED] came and asked me whether he had said that. When I confirmed the event and context, he asked me to remove the attribution. I laughed, at which point [REDACTED] said, “Was that a giggle or a ‘yes’?”

“A giggle,” I said.

[REDACTED] then proceeded to erase the whiteboard, concluding his comments with:

“If you’re going to play those kinds of silly games with your co-workers, leave me out of it.”

— [REDACTED]

Given the reality of the office, the QotD whiteboard now simply says [REDACTED]. Nevertheless, I leave you with this choice quote:

“Never argue with a man who buys ink by the barrel.”

— Charles Bruce Brownson

Also relevant is the “Streisand Effect” entry on Wikipedia.

Happy Holidays!

I’m Mad as Hell and I’m Not Going to Take It Any More

Life in the workplace

In the classic movie Network, fired newsanchor Howard Beale exhorts his audience to fling open their windows and shout “I’m mad as Hell and I’m not going to take it any more!”

Some have commented recently on the lack of activity on my blog. There have been dry spells before, but eight months is certainly among the longest. One reason for my ongoing silence is the complete lack of anything positive to say about my workplace.

However, I have concluded that this restraint on my part is pointless.

In fact, I have resolved in the New Year to go ahead and publish my thoughts, fearless of retribution or negative consequence, as bottling up my anger and frustration has simply become too much. It is damaging both my psyche and my relationships with my family and friends.

This blog category will include details of some of the more outrageous workplace blunders, guided by three principles:

1. Is it true? — Are the facts documented or verifiable? Can my co-workers or other witnesses vouch for the accuracy of the anecdote?
2. Is it relevant? — Do other people elsewhere share similar experiences? Is the subject of interest to a wide audience? Will others find the story reflective of their own lives and careers?
3. Is it constructive? — Does the story help define a clear path to improvement? Will my advice, if followed, help build a foundation for a better workplace? Can readers use the account as a jumping-off point for additional advice?

Sadly, some of these tales will reflect poorly on their subjects. Unfortunately, this is due to the choices made by those subjects, and that is not something I will gloss over.

Please send me feedback and stories if the muse moves you.

Sat, 02 May 2009

Social Networks

The latest rage on Facebook apparently is apps that ask you to list five of your favorite things in a category. I don't think I can even name five wrestlers, so it's probably just as well I'm not on Facebook. Although my wife is, and she thinks it's wonderful. I did once, long ago, join Classmates.com. None of the people I would have considered renewing relationships with ever seem to have joined. Before I reached my current viewpoints on Internet-enabled social networking, I also joined Friendster.com; but it has been nearly a decade since then.

Facebook, though, for me, is just not a draw. Yes, I do have a blog, which I even occasionally update; but as you can see, it's on my personal website, where I have total control of the context and copyrights. But when it comes to interpersonal relationships, I like to keep my one-on-one communications confidential. The idea of a "wall" where I get drive-by comments from acquaintances, or having the entire subscriber base (or even just the people already on my friends list) know who is in my social network, or having people tag pictures with my name, with or without me actually being in them — even with the "privacy" controls Facebook provides —  just leaves me cold.

Further, having to maintain personas on multiple networks (Linked-in, Facebook, Classmates.com, MySpace, Friendster, Orkut, etc.) to maintain separation between professional and personal lives, as well as hit the "right" sites so that the "right" people see I'm a member of the same communities… well, it's burdensome. The MySpace people want to use MySpace; the Facebook people want to use Facebook. It's just too much work for me. The single option of Geocities in the '90s was easier.

Finally, I've had a website since 1996. My Google PageRank is excellent. If people I already know want to find me online, it's a piece of cake. If people I don't already know want to find someone with my skills (e.g., recruiters), my resume is at or near the top of Google's results in all the markets I care about.

That being said, for many, Facebook and its ilk are pretty darn good. They have reasonably featureful interfaces, a critical mass of users, and the backing of major corporations to ensure they stay available. And that's fine; it's just not for me.