The Problem

In an ideal world, changes to the control system are thought out well in advance, and their impact is fully analyzed and understood before any changes are made to the system. The engineer making the changes has an audit trail for each change request and documents any changes made, along with robust comments, so that other plant personnel are able to work on and maintain the system.

In reality, many changes are made on the fly due to pressure from production and commissioning. Changes are not consistently documented, and it is often unclear who requested the changes to be made and why they were made.

The Consequence

Up-to-date documentation sits at the heart of any control system. Without documentation, maintenance is very difficult, if not outright impossible. In extreme situations our clients have downloaded what they believe to be the latest version of their system configuration, only to find that undocumented changes had been made. The download process wiped out essential fixes and adversely compromised plant performance.

This problem is particularly insidious on safety systems, where cause-and-effects have to be considered ahead of any system changes.

Alarm settings are an often-forgotten aspect of control systems. These are accurately calculated and set at the time of commissioning, but over time drift sets in. Alarm limits are ‘tweaked’ to eliminate nuisance alarms, without fully understanding the consequence of the change, and without documenting the fact that changes have been made.

The Cause

Root cause #1: No documented process for managing change.

The Solution: It is highly recommend that a change management process is agreed and documented as part of the day-to-day maintenance of the system. Your policies and procedures should be subject to regular internal audits to ensure compliance. Training is essential to ensure all affected parties understand how to use the processes.

Root cause #2: Time pressures mean no time to update documents.

The Solution: Automated electronic change-management tools provide an efficient and automated audit trail including system hardware changes and logic changes in the controllers. Each change is automatically recorded in the tool, with facilities to record notes on the changes made. Changes can easily be compared between the current version and historical versions to see what has been changed. Changes can also be rolled back if required.

Root cause #3: System backups are out of date

The Solution: Any maintenance solution should include regular site audits, during which system backup checks are performed, and the integrity of the backup to restore systems is validated. This process verifies that the backups actually work as intended, with no software corruption. Using a version control tool also greatly facilitates management of backups.

Root cause #4: Lack of adequate access control to manage changes to the system configuration.

The Solution: Access-control and user policies should be clearly set out and should account for all users of the system and their various access levels. These policies should be audited periodically to ensure passwords are set at the right level and are changed per the policy. Access to physical devices, such as keys to server rooms and CPU lockouts, should be strictly controlled.