With cybersecurity threats increasing, the requirement to protect systems and address security flaws has become more prevalent. This blog describes some of the lessons learned during the commissioning process of complex security and audio-visual systems.

In 2020, IBM found the public sector lags behind other industries in terms of time to spot and contain data breaches. The global average across all sectors to find a breach is 177 days. Meanwhile, the average in the public sector is 231 days [1]. Additional trends to consider:

• The rate of cybercrime increased by 600% during the COVID-19 pandemic
• The financial sector suffered the largest financial losses in 2020
• The total cost of all cybercrime damages in 2021 is expected to amount to about $6 trillion worldwide[2]

As the rate of increase in cyber threats has risen, the need for networked device hardening presents opportunities and challenges to ensure these systems perform as designed and that sufficient measures are taken to ensure the detection of threats and the protection of critical systems.

History/Background

To strengthen the resilience of this infrastructure, the Cybersecurity Enhancement Act of 2014 (CEA) (S.1353), updated the role of the National Institute of Standards and Technology (NIST) to “facilitate and support the development of” cybersecurity risk frameworks[3].

NIST designed the framework to provide a nationally recognized approach to cyber risk management using best practices and proven processes. As more sectors and organizations implement the framework, its approach will serve as an accepted baseline for cybersecurity practices in critical infrastructure organizations[4].

The framework focuses on using business drivers to guide cybersecurity activities and considering cybersecurity risks as part of the organization’s risk management processes. The framework consists of three parts:

• The Framework Core
• The Implementation Tiers
• The Framework Profiles

The Framework Core is a set of cybersecurity activities, outcomes, and informative references that are common across sectors and critical infrastructure. Elements of the Framework Core provide detailed guidance for developing individual organizational profiles. Through the use of profiles, the framework will help an organization align and prioritize its cybersecurity activities with its business/mission requirements, risk tolerances, and resources. The Implementation Tiers provide a mechanism for organizations to view and understand the characteristics of their approach to managing cybersecurity risk, which will help in prioritizing and achieving cybersecurity objectives [3].

In developing this framework, a common platform was developed from which multiple organization types can address security threats and build security into their system, not as an afterthought but as an asset.

The framework’s five functions are developed in such a way that allows organizations to develop security programs that address the core aspects of computer security.

Depending on the client, type of organization (financial, educational, governmental, etc.), and risk tolerance, the framework’s core structure provides a means to achieve specific cybersecurity outcomes[3]:

• Identify – Develop an organizational understanding to manage cybersecurity risk to systems, people, assets, data, and capabilities
• Protect – Develop and implement appropriate safeguards to ensure the delivery of critical services
• Detect – Develop and implement appropriate activities to identify the occurrence of a cybersecurity event
• Respond – Develop and implement appropriate activities to act regarding a detected cybersecurity incident
• Recover – Develop and implement appropriate activities to maintain plans for resilience and to restore any capabilities or services that were impaired due to a cybersecurity incident

Security As an Asset

The tenets of commissioning align quite well with the security framework. The themes of beginning at the beginning and verifying and validating performance metrics, configurations, etc. are shared by both.

In many cases, various aspects of computer security are add-ons to existing computer systems. Computer systems is the generalized term for IT infrastructure. This encompasses the network, equipment on the network, access controls, operating systems, hardware, firmware, software, etc. What add-on security means, in terms of both security and commissioning, is that the planning and design efforts lack security considerations from the beginning.

The early stages of the commissioning process address the problem:

OPR

In commissioning terms, the OPR (Owners Project Requirements) provides metrics on the quality, performance, and capabilities of the owners’ IT infrastructure’s security objectives. As an add-on, this usually causes complications and problems that require patches or a complete redesign of portions of the infrastructure.

To protect these systems, a thorough understanding of how these systems operate, the conditions under which they operate, and a means to verify and maintain their proper operation over time is required.

The basis of design (BoD) conveys the assumptions made in developing a design solution that fulfills the intent and criteria in the OPR document. Narrative descriptions of equipment, systems, and assemblies are developed and included in the BoD, and the Cx Plan is expanded to include the details of the Construction Phase and Occupancy and Operations Phase activities. The Framework profiles provide the opportunity to vertically align design criteria with freely available recommendations on port access and available uses and mitigation solutions for a number of networked devices updated on a periodic basis.

Design

These design criteria are then used to build systems having known operating configurations successfully deployed in the field.  There are many advantages to such deployments, one being the improved capability to address zero-day threats/threats of unknown origin.

At a minimum, these framework functions should be applied to the following critical systems:

• Data Centers
• Pumps
• UPS
• Generators
• Computer Networks
• AV Equipment
• Fire Alarm Equipment
• Elevators
• Emergency Announcement Equipment
• Communication Systems

The Role of the Cybersecurity Engineer

It is important to have a Cybersecurity Engineer to represent the owner on matters of the OPR, BOD, and design. The Cybersecurity Engineer will be a valuable member of the Cx team. The engineer will aid in the design of Critical Systems plans and provide review and feedback on Cx specifications, test plans, pre-functional checks, start-ups, and functional performance tests.

The Cybersecurity Engineer will assist with developing the testing specifications and test requirements for acceptance. The Cybersecurity Engineer will also help develop the “in-house” resources to be used.  These resources are often kept separate or on isolated networks to prevent intrusion onto the main network. The test space will need to be monitored for threats constantly.

The Cybersecurity Engineer will review pre-functional checks, pre-functional tests, and functional performance tests for acceptance. He/she will make recommendations on which test can be performed under laboratory conditions and which test needs to be performed or repeated on the target network. The process of developing pre-functional checklists, pre-functional tests, and functional performance tests are all augmented similarly as outlined below.

Other Considerations

The type of owner or client needs to be considered when developing the commissioning plan as the amount of risk, security requirements, and investment varies based on the organization and/or industry of which the Owner is a part of. Regulated industries and governmental agencies are at the high end of the low tolerance / high investment scale. Social and civic organizations such as cultural institutions and schools often have a higher tolerance and lower investment in cybersecurity. Ideally, these considerations will be captured as part of the OPR.

In planning for security in the Cx process, it is important to consider scheduling early in the process.  Most governmental-type agencies and some larger corporations have a configuration control board that reviews and approves modifications to the computer network. These reviews and decisions can take weeks or even months and must be planned into the process. Denials and modifications to plans submitted to the board can cause severe delays and can be costly.  These problems can be mitigated by using equipment with well-known profiles and reliable systems architectures that have been known to be successful in their implementation.

Lead times are also another important consideration when planning the commissioning task. Often the equipment scanning is done by a third-party vendor. The vendor typically will receive the equipment from the contractor performing the work. Using their own internal setup, the equipment is scanned against its framework profile for vulnerabilities. A report is generated and if any failures appear in the report, they are addressed.  The equipment is rescanned until it shows no failures in the report. Once the equipment passes the test, it is then sent back to the contractor with a report that shows it passed the test. In the case of the first project, this process threw the schedule off by more than 7 months.

A Test Lab-Like Experience

Let’s consider the following scenario. Two design-build projects with two different General Contractors implementing AV and Security systems, in addition to critical infrastructure, are started about a year apart. For the first project, security planning was more of an afterthought. The second project utilized the lessons learned from the earlier project and implemented a better-planned approach complete with a sandbox. An RFI was submitted to request clarification on security hardening requirements, which prompted a scope of work and a list of targeted equipment.  Because the owner’s tenant has detailed requirements for the introduction of new equipment on their network, each piece of equipment was subjected to a thorough cybersecurity process, including imaging, hardening, and vulnerability scanning.  This is applied to Building Information Technology (BIT) data & voice, and Special Systems (Radio, Security, Fire Life Safety systems (Automatic Fire Suppression systems), Fire Alarm systems, HVAC controls, Building Automation Systems, etc. installed.  The tenant’s requested scope of work included:

• Initial baseline and post-baseline validation scanning of in-scope assets
• Vulnerability scans on test environment utilizing tenant-specified tools
• Generated report of vulnerability scanning results to include preliminary analytics and scanner-provided recommendations
• A risk analysis of the results leading to a prioritization of vulnerabilities to be addressed and detailed recommended action steps

The outcome is a plan with the targeted devices tested and retested in a controlled environment using profiles of known devices if available and scanning results for all devices with recommendations to address uncovered vulnerabilities.

In summarizing the lessons learned from Cybersecurity Hardening in the commissioning process for design-build projects, it is important to develop the security concepts early before the bridging documents are produced, identify which portions of the framework are applicable, and work with the Owner to identify the Owner’s security requirements and, if applicable, the tenant’s security requirements as well.  For these types of projects, a cybersecurity engineer needs to be part of the commissioning team.  This person would serve as the liaison between the owner, tenant, contractor, and subcontractors. The cybersecurity engineer would work with the Commissioning Authority to develop test scripts, work with the Owner and Tenants to develop test specifications and testbeds, and work with the Contractor to develop test plans and verify the network security configuration for the test bed. The early adoption of a hardening plan for networked systems to be commissioned reduces the time required to bring these systems online by providing a clear and unambiguous path to start-ups and functional performance testing.  Early adoption allows for the selection and approval of equipment with known security profiles as the Basis of Design. These ideas combined with other available Cybersecurity assessment/Risk Management tools increase the prospects of timely and successful delivery of secure systems.

[1] Security Intelligence, IBM “Roundup: Government Data Security Threats in 2021” Government, November 11, 20221

[2] “Top Cyber Security Statistics, Facts & Trends in 2022”, Cloudwards.net, Max Pitchkites (Writer), Last Updated: 22 Mar’22

[3] Framework for Improving Critical Infrastructure Cybersecurity, Version 1.1, National Institute of Standards and Technology, April 16, 2018

[4] Cyber Security Framework Implementation Guidance, May 2020, U. S. Department of Homeland Security, Cybersecurity and Infrastructure Security Agency

[5] NIST Special Publication 800-70 Revision 4, National Checklist Program for IT Products – Guidelines for Checklist Users and Developers