Photo by Massimo Botturi on Unsplash

Never Be The Last Person Who Touched The Firewall

PublicBlog Hits: 1021
This is somewhat of an in-joke amongst network engineers - whoever is the last person to have touched the network (and in particular the firewall) always, but always, gets the blame for any subsequent issues no matter whether they are the fault of the network or not

Ideally, when we make a change, we should always aim for minimum user impact, maximum uptime and the ability to quickly determine if some change has failed, together with the ability to easily switch back to a safe known state.

These are some of the key drivers to being able to make changes in a traceable, managable way

A change board

It's a bit of a formal title but basically this is a group of people with a business interest in having the change work out well. Do be careful though, in a big company it's very easy to have a large number of people with only a marginal interest in what's going on which can make it difficult to get any change approved!

Change control system

A process and possibly a computer system to help you track changes and manage their impact

Most importantly, this is a process whereby one should

  1. Get the engineers to document what they are going to change
  2. Explain what you are going to change, when you are going to change it and how long it will take to make the change to the users or their managers (should not be in technical detail unless the change board request it)
  3. Explain the business drivers for the change
  4. Take feedback on the proposed changes
  5. Explain what your rollback plans are and where your go/no-go points are
  6. Get sign-off from your change board that they agree it's OK to make the changes at the agreed time.
It's remarkable how reluctant technical teams are to answer some of these issues, particulary the question regarding rollback plans but you really need these questions addressed before getting the change board to agree the changes

Change Windows

Have standard times of day/week where users know you will be making changes

If you run the sort of company where this is possible, it would be a good idea to nominate an hour or so every week for possible changes as it will ease the process of getting the changes past the change board but please do not let it be a substitute for going through the change process above on the grounds that 'everybody knows we make changes then'

Rollback

Do you have a plan to get back to where you were before you started the changes?

This is absolutely essential for any but the most basic changes; if something does go awry during the change then everyone needs to know how to get back to where the network was at the start, and how to check that it has happened.

What this procedure looks like is very dependant on what the changes were, but at the very least a statement such as "We will reverse each of the steps we took in turn and when we have reversed them all we will test the system by means of " may be sufficient

Design changes so they can easily be rolled forward and rolled back

Are there any break-points where you can review what you have done so far?

If you are making a complicated set of changes you might consider breaking them up into smaller, self-contained groups of changes each one of which has a defined buisness impact. At the end of each group of changes you can test to ensure that everything is still working as desired. Maybe even consider spreading this across a number of change windows to minimise impact?

Monitoring

Never install anything you don't know how to monitor

Know what your network ought to look before and after the change which you are making (Present Mode of Operation vs Future Mode of Operation)

Ticketing system

Allow users to log issues so you can track impact of changes

You really need to have a method for users to log faults, perhaps by phone to a helpdesk or preferably directly into a trouble-ticketing application; even if you are a small company this will be of benefit as you will end up with a single place where all problems can be raised and recorded. This is how you can tell if your network changes are causing any pain, although you'll have to sort a certain amount of wheat from the inevitable chaff.

Sometimes this function can be integrated with the change management system or sometimes it can be on a separate system

We can help

At Layer3 Systems, we provide the attention to detail necessary to ensure your network does exactly what you need. We can liberate your business by managing your network devices so that you can confidently deliver what you do best. If you want to improve uptime, remove uncertainties, and increase the performance of your systems and networks, then we can provide what you need. Come and talk to us. We can help. Call 0203 805 7795 and talk directly to one of the Layer3 Systems team.
Print