Making Things Work - Part the Fourth - Do things the customer's way

One of the fastest ways to alienate a customer while trying to deliver an integration project is to fail to ask how they like things done.

Unless you are dealing with a completely new site, with virtually no existing infrastructure, there will be an accepted "right way" to do things such as labelling cables and managing backups which is widely understood by the customer's IT staff. They may not have any of their normal procedures and practices documented, but they will be deeply offended if you don't follow their standards. There is little as depressing as having a customer representative come by the racks of equipment that you are busily installing in their data centre, looking disapproving and making some helpful remark such as "the management cables should all be green". Particularly if you have already installed all the cables. If the customer has a cabling standard, they will expect you to comply with it, and failure to do so may hold up acceptance and sign off. And "we didn't know" will not fly as an excuse, so don't try it.

Involve your customer in the planning of the installation, and get their commitment on the details (in writing!). This will save arguments and rework, and rework is to be avoided at all costs: rework tends to introduce errors, and is of itself expensive. If they insist on something that you know is wrong, stupid or hazardous, make sure that you explain to them in writing why whatever they want will cause a problem. If they still insist, make sure the project manager has record of who said what and why, in the sure and certain knowledge that you will need this information when the project reaches the "allocation of blame" phase, and then do what they want. I have never seen a project stopped because a customer wanted something stupid done, though I've seen a few that should have been.

So what should you expect to need to know about the customer's preferences?

First, if you want a reference for good data centre design, try Rob Snevely's "Enterprise Data Center Design and Methodology". There are other books on the subject, but that one has always worked for me. However, whatever references you may have read, please temper their application with a big dose of common sense: no customer has a data centre that actually looks like the ones in the references.

Cabling standards. Do they use particular colours for particular purposes? How do they do label cables? How do they do cable management? In my experience, if the customer has a regular contractor who does their cabling, it can been quicker and cheaper to subcontract that party to do whatever is needed for the project.

What is the company policy for operating system installations? For firewalls and routers? For the use of encryption? If they have nothing (and that is still very common), and request that you apply "best practice", make absolutely certain that they are fully briefed about what your interpretation of best practice is before you start installing. I have wasted days, and in some extreme cases weeks, negotiating the minefield that is "security best practice" on customer sites. It's a subject everyone thinks they know about, usually because they read something in an inflight magazine or some other similiarly authoritative publication. When time permits, I'll do a blog entry on idiotic security practices I have seen. For now, try to avoid getting hung up on this particular reef.

Is there a standard administration account name that they use?

Unless the backup facilities are part of your deployment, how are they going to backup the systems you are installing now? What connectivity do you need to allow for in your build? I was once handed a "design" for a large, N-tier network installation. On close inspection, I realised that there was no way to connect the new equipment to the customer's existing infrastructure: the design had connection points for the internet, and for the customer's partners, but not for their own admin staff. It had a single "console" (actually a small desktop machine which had a screen), from which everything in the build was supposed to be controlled. I went to manager responsible for the job and said "there's no way to connect this lot to the customer's network". And he said "that feature wasn't listed in their Request For Proposal", and took the position that, since they hadn't asked for it, it wasn't our problem to provide the functionality.

I could see that this was going to be a problem, but reasoned argument made no headway at the time, so the infrastructure engineering team went ahead and installed everything according to the "design" (which had numerous other defects, which we had to fix as we went along). A few months later (it was a big and complex build), the equipment was on the floor of the customer's data centre, and they realised two things. The first was that the only way to administer the new equipment was to walk into the data centre operator's room, sit down at the "console" and work from there: it was unreachable from anywhere else. And the second things was that they couldn't print any reports from the new system, because it had no connectivity to their printers.

The technically correct solution in my view was to redesign the whole thing to integrate tidily and securely into the existing customer infrastructure, but no one (including the customer) was prepared to contemplate that option. They had a deadline to get the new systems into production, and the project was already running late. Instead, we had to design and build an ugly chunk of bridging network to connect the new systems to their network. This cost the customer more money (it was unquestionably a variation to the scope of works), complicated the build, extended the test phase and looked like what it was: a belated after thought. One phone call could have headed this off, if it had been made at the beginning of the project, if anyone had been prepared to tell the customer that they were making a mistake. Even if they chose not to address the problem at the time, at least we as the vendor would have retained a little credibility.

So if you can see problems in the design, try and make someone pay attention early on, while there is still time to fix things. Even if all you achieve is an email trail that documents recognition of a potential problem, that may be enough to make you look less foolish if the wheels fall off further into the build.

Double check what they are planning to do about backups: this may be the point where you discover that nobody quoted them the extra backup client licences that they will need for whatever product they use. Be particularly careful if you have to deal with applications that require special backup clients: I've seen jobs where the salesman only quoted operating system backup clients, and missed the licences needed for Oracle. In the worst case, the customer may need to upgrade a tape silo or SAN to get sufficient capacity for backups.

What are the customer's expectations of user acceptance testing (UAT)? On a large project with a lot of software, there is a tendency for the testing phase to focus on the applications and their functionality: "does this stuff do what we wanted?" testing. And I have seen projects where the entire test plan was written by the software development team: they were astounded when they found that the customer expected to see test cases for the hardware as well. Find out what your customer wants, and get ready to deliver it, because without a completed UAT, you will not get sign off. Be extremely wary of contracts that contain vague statements about "mutually agreed test plans". That phrase just means that the people who drafted the contract had no idea what testing could or should be done. Chances are the contract was worked out by business people, not IT staff. But you can bet that the UAT will have to be acceptable to the IT staff, and they may insist on all sorts of time consuming tests.

The whole issue of testing can blow your schedule right out of the water, and it can add unanticipated extra work to the build. Take a fairly common case: you build a set of infrastructure that is supposed to be firewalled off from the internet on one side, and connected to the customer's existing network on the other. When you come to test it, the customer insists that you conduct tests to prove that the infrastructure is secure before they will let you connect it to either their network or the internet. You essentially need to perform a penetration test, but in an isolated environment. A common requirement is that you facilitate access for a neutral third party to conduct the penetration test (and this is quite reasonable: the people who build secure systems should not be the same people who test them). I've seen projects that had to install additional equipment just provide adequate facilities for testing to take place in a manner that the customer was pleased to accept. The equipment had to be sourced, installed and configured. It all takes time.

My strongest recommendation on testing is that you start writing the test plan before you unpack a single piece of equipment. Write an outline, parcel out the work, make sure the project manager realises that this is a non-trivial task, get regular customer feedback. If nothing else, the work of writing test cases can be used to keep the team busy if some other part of the project stalls. And stall it will...

Making things work - Part the third - The Project Manager Is Your Friend

A good project manager is a jewel beyond price. I've worked on projects where, without the effort and expertise of the project manager, we would never have closed the job. Unfortunately a bad project manager can make the simple jobs next to impossible, and drive the engineers to distraction.

Good project managers keep management off your back, control scope creep, track open issues, sort out the logistics of equipment supply and delivery, handle the resourcing problems (when you absolutely have to have the company expert in "whatever" for 4 straight days if you are to complete the project, and he's already booked to another job), worry about time cards and P&L and other financial matters and prevent you from going crazy by helping you prioritise problems in a sensible way.

Bad project managers agree to everything the customer asks for, regardless of scope, have no idea how to keep an issues log, leave you to sort out your own logistics and resourcing and generally cause more problems than they solve. The character who insists that a team, all of whom are working in the same small room, have several "catch up meetings" every day, with him and the customer's project manager, and then wonder's why work is proceeding slowly. The guy who demands rapid reconfiguration of production equipment in the customer's data centre, to suit his project, but in violation of the customer's change control procedures. The genius whose project schedule doesn't include any time for testing. You do get them.

If you've got a good project manager, cherish them. If you have a inferior model, be prepared to do part of their work yourself, because it has to be done, like it or not. If your project manager is just a bit inexperienced, you can help them be better by explaining what you need from them and why. And consider the fact that a project manager who has only ever worked on software development projects will need your help to get their head around a project that includes a lot of hardware.

Of course, there's always the possibility that you are a relatively inexperienced engineer. Perhaps you've never worked on a big project, or with a project manager before. There was a first time for all of us. So let's step through some basics here:

Scope creep

Every project, large or small, is supposed to have a fixed scope. The scope is the body of work that is to be performed, for example "install Linux on four servers". The customer agrees to pay for that work. There is a contract (or work order, or whatever the local lingo requires). The customer pays $N and the engineers do Y tasks. But if the customer then produces a fifth machine, and asks to have that installed as well, that would be an increase in scope. That's a fairly obvious case, but the thing is that if you are already installing four machines (and you're probably doing them in parallel), doing a fifth doesn't look like much extra work. There's a tendency to agree to it.

Scope creep becomes much more hazardous in a complex environment. Adding one extra web server or proxy server to a load balanced multi-tier network can involve reconfiguring firewalls and load balancing switches, and testing all the changes. That one extra machine can cause hours of extra work, and if it triggers a problem (as under-analysed changes to designs so often do), the whole project may grind to a halt until a fix is found. My basic rule is that I, as the lead engineer, never under any circumstances agree to anything that causes scope creep. All such matters must be referred to the project manager. A good project manager will then say "what could go wrong if we do this?" and "how long will it take?", and make a decision about what to do next. Really little things should be agreed to, but they still need to be documented and tracked in the project documentation. Larger things may require a variation to the scope of work, and an additional charge. And as soon as you talk about extra money, someone on the customer's side will have to approve the additional cost. Sometimes this will cause the proposed change to be halted, because no one is prepared to pay for it.

Be very wary of requests for additional work coming from the customer's operations and support staff: the chances are very high that they do not have the authority to incur additional costs, and they probably don't have the right to vary an agreed scope of work. Sorting this stuff out is the project manager's responsibility, and an too-helpful engineer on the team can make the project manager's life difficult. If you stop what you are doing for an hour to help Larry-the-customer's-engineer get something working, and as a consequence then run late to finish your appointed task, the project manager will be better able to handle complaints about the job running late if he/she can say "as agreed, we stopped for an hour to help your staff fix a problem".

Consider also the financial terms of your project. If it is a fixed price job, and you accept additional tasks into the scope at no charge, you are working for free. If, on the other hand, you are on time-and-materials, and you perform extra work and charge for the time, you will increase the size of the customer's final bill. The first case shouldn't make you happy, and the second case may make them very unhappy, particularly if they are on a tight budget. Spurn scope creep: you are not here to make everything perfect in the customer's environment, you are here to deliver a project. Focus.

Issue logs and customer communications

The issues log should be kept by the project manager. They may call it something else, but its purpose is to record problems, issues and questions, and what was done about them. So when an issue develops, say "not enough power in the data centre to power up the machines", the project manager logs an issue: "Issue 1, 6 additionals 32 AMP power connections needed in the data centre", and the task of resolving the issue is assigned to someone. Then every time someone in management asks why the project has stalled, the project manager can say "we have a open issue that is preventing us moving forward, and it's been assigned to Joe Blogs". This will normally deflect the wrath of management onto Joe Blogs, who in this case will probably be the customer's data centre manager, and capable of taking care of himself (I've never met a female data centre manager).

The issues log also becomes a repository of the project's history. It can be a big help when, six months in, no one can remember why a certain thing was done in a certain way (Why did we set all those subnet masks to 27 bits?). The issue log may remind you of an old, forgotten problem, and what you did to get around it.

It is also vital that all email relating to the project is stored. Hard copy communication normally gets stored by default, but email can get overlooked. You MUST keep it: everything you send to the customer, and everything you get back. Like the issue log, it can remind you of how you got to a particular place. It can also be very useful when disputes arise. So:

Many moons ago, the marketing department of a certain telco decided that they would offer their customer's an application hosting service (this happened long enough ago that application hosting was something of a novelty). Accordingly, their staff did a design, and handed it to my business development manager so that we could price the work. We looked at it, and pointed out that the design called for a separate firewall, router and switch for each new customer, and that, given the equipment specified, they would only be able to fit two customer deployments per rack. In case you are not familiar with the economics of data centres, rack space is expensive, and you try to minimise the number of racks used. So we wrote the customer's representative an email, pointing out that the proposed solution would rapidly become extremely onerous to manage, and that the whole things could be done more cheaply by virtualising the network components. We were trying to help.

What we got back was a curt note stating that the design had been done by "our expert staff" and requesting that we mind our own business and get on with pricing and installing the first set of equipment. The initial deployment was enough for two customers, one rack of kit.

And the BDM said "what do we do now?" and I said "File that email for future reference, and we'll get on with the job", which we did. Job complete, signed off and accounts paid.

Weeks passed, and one day the BDM got a very grumpy email from someone in authority on the customer's side. We had installed a very bad design, they were going to have to buy a new firewall, router and switch for every new customer, the system would soon be unmanageable. What did we proposed to do about it?

In response we sent back a copy of our email, pointing out the deficiencies of the design, and of their response to same. Silence fell. I believe their may have been some blood letting on their side.

Save all correspondence. I used to have a project manager who would print all relevant emails to PDF, and publish them to a web page which belonged to the project. That worked well. If you get a project email that the project manager is not copied on, forward to them. But make sure they are storing the emails somewhere, for later.