Making things work - Part the first

In every project that I have ever worked on, no matter who I was working for, there was a phase between the customer signing the order and the new "system" being handed over to the customer, and that phase was usually called "system integration". Sometimes it was called "system build" or "install and implement" or something similar, but it always meant "put everything together and make it work the way we told the customer it would".

In the early days, the system integration phase would be short and simple: install a couple of computers and a printer and configure the print queues, or some such thing. But as the projects got bigger, the integration tasks became more numerous and varied. And that increased complexity never seems to get factored into anyone's estimated costs, and I suspect this is because no one but the people who actually do the integration can list all the possible sub tasks, and explain why some of them are both difficult and time consuming. One of the things that always worries me is the sight of a project manager who has just been handed their first big integration job. You just know that they are going to suffer torments for the entire duration of the project, and no warnings will convince them that there will be problems until the job gets properly under way. By about the second week they will be looking stressed and clutching at their Gantt charts for comfort, and comfort there will not be. For verily, systems integration can be extremely tough.

Now you may never have done this sort of work, but have a job like this in your future. Or perhaps you have done an integration job, and never want to suffer like that again. Or perhaps you're in sales, and you'd like to know why the contingency budget on all projects is usually not only fully consumed, but over run (and why all the infrastructure engineers hate your intestines). Let me try and explain how integration works from the point of view of the people who do it, and share what I've learned about things that can make it hurt less. I'm not claiming that this is the best way: it's just the best I've come up with so far. The job I'm going to imagine here is somewhere between $AU3 million and $AU5 million. It will include several racks of computer equipment, firewalls, load balancers, switches and routers, a lot of licenced software and some that is bespoke.

Almost every integration job that I have ever been assigned has featured implementing something that someone else designed. It is a sad fact in the IT industry that engineers who work in pre-sales are not compelled to work on the implementation of the things they design. It might make some of them more careful. I always start be reading whatever documentation exists: the design document that was given to the customer, any internal documentation that the sales people have for the job, anything we got from the customer, the bill of materials and (if it exists at this point), the project schedule.

It annoys me deeply that sales people and project managers create project schedules without asking the people who are going to do the work how long they think it will take to do, but this particular form of insanity seems common across the industry. People who couldn't perform the required tasks if their lives depended on it will guess how long the tasks that they don't understand will take to complete. Anyway, take note of where they think the job will end. The end date usually bears a close relation to the next end of financial quarter, the often reflects no urgency on the customer's part, but merely a requirement to get revenue recognised by a particular date. If you happen to be a customer embarking on one of these projects, pause and consider what your vendor's revenue recognition deadlines may be doing to your project quality.

However, back to the documentation. Most designs feature at least one diagram which represents the network in which the proposed solution will reside. For many of the jobs which I have done, the network itself has been part of the solution. Unfortunately, the diagram will usually be a marketing diagram: little pictures of computers joined together with lines. This type of diagram is extremely dangerous, because it conceals more information than it reveals. Consider a diagram such as this:



Three computers, how bad can it be you say? Well the Internet isn't really a cloud: typically, we see its physical entry point as a port on a router somewhere in the data centre. Two computers can't connect to the same router port, so either we need two router ports, or a switch between the computers and the router. Or there might be a whole lot of other infrastructure between those two machines and the internet - routers, switches, hubs, firewalls. The diagram doesn't tell us. Nor does the diagram really tell us how the two computers in the middle connect to the one at the bottom. Does the third machine have two network cards, or is there some other device between them? What is it? A switch or a router? And if there are switches, routers or extra networks cards needed, do they already exist or are they coming out of our project's budget?

Start by drawing a detailed network diagram that shows every device and every port, and work out what is required, what already exists and what has to be purchased. Compare what has to be purchased to the bill of materials. I've done several jobs where critical components have been missed from the bill of materials, and you need to get missing equipment ordered quickly, or the delay in supply will hold up your build.

The missing equipment issue can be complicated if your contract with the customer includes hardware maintenance. If you have to add extra components to make it work, you will also be adding the cost of the hardware maintenance contracts. This will eat into your contingency budget, assuming you are on a fixed price agreement, which these things almost always are. Furthermore, if you add extra components, you will increase the amount of heat the completed solution will generate, and you will require more power and probably more rack space. You see the problem?

Now while you are doing all this, your project manager is probably agitating for you to get the build under way. Don't. You are no where near ready to start building anything. First you need a proper racking design, and to do that properly you need a very detailed network diagram. You need a clear picture of what network cabling is going to have to go where. Once you understand what must physically connect to what, and you have the physical specs of all the equipment, you can draft the racking diagram. Good rack design will save you time in the build, and make your customer's life easier when they take control of your creation. Equipment manufacturers all have recommendations for how their gear should be racked (you did check that all the racking rails got ordered, didn't you?). As a rule of thumb, put the big heavy stuff near the bottom of the racks and try to minimise the amount of cable that has to go between racks.

While you are doing all this, there is other information that you are going to need soon. Find out who your contact is on the customer side, and ask them for their standard for naming equipment in their network. Also ask what their procedure is for allocating blocks of IP addresses, and get the details of their DNS, default routers, time servers and anything else of that type that you are going to need. Finally, ask if they have a standard for operating system installation. Do they minimise or harden the OS in any way? Do you need to do anything special to conform to their normal practices? It may take them a while to collate this information, so ask for it early. You still have a lot of work to do before you are ready to start installing anything.....