As pharmaceutical manufacturing increasingly moves to 24/7 operation, any interruptions have the potential to be extremely costly, both financially and in terms of reputational damage. All supporting services — which include pharma laboratories providing critical analytical data — must also operate round-the-clock and take steps to ensure continuous delivery.
Many different internal and external events can impact the ability of a business to operate continuously, from local power outages to global supply chain problems. However, in today’s highly connected industrial environment, there is particular concern with maintaining the continuity of digital systems and protecting valuable data. The growing incidence of cyberattacks poses a significant threat to the pharma industry, which in recent years has seen an increase in the number of companies dealing with the damaging consequences of such events.
Key in mitigating the risk of interruption is the development of a business continuity plan (BCP) within which laboratory data systems must be protected against the consequences of any adverse events. The choice of systems installed in a lab affects the ability to maintain uninterrupted operation and safeguard data — and having the right hardware and software helps deliver greater resilience. Modern chromatography data systems (CDS) are equipped with intelligent tools that provide the flexibility, scalability and robustness required to ensure the security of laboratory data and support business continuity.
The BCP process
Maintaining productivity in the face of unforeseen circumstances is challenging. Businesses are prone to interruptions that range from minor to catastrophic. Modern laboratories are highly infrastructure-dependent, and their operation is predicated on access to complex support services, including sophisticated IT systems.
Business continuity planning is crucial for mitigating the risks of interruptions. The BCP process helps companies develop a system for prevention of, and recovery from, potential threats to the company (Exhibit 1). The BCP ensures personnel and assets are protected and can function quickly in the event of an incident. It identifies all risks that can affect operations, from natural disasters to human-made events, such as cyberattacks.
The business continuity planning process begins with conducting a risk assessment for all potential threats and their impact on the laboratory and the business. The second stage is to design a recovery solution that gets the business back up and running quickly, and the third is to implement the solution. Next, it is critically important to fully test the solution, make sure it works in practice, and then accept it.
Finally, regular maintenance and review is essential to make sure that the plan keeps up with a changing operating environment and takes account of new and evolving threats. The outcome is a plan that defines how risks will affect operations; the safeguards and procedures intended to mitigate those risks; the testing procedures in place to make sure they work; and the review process that ensures the plan is up to date. Communicating the plan effectively across the business is vital and sometimes overlooked.
A key part of any BCP is a disaster recovery plan (DRP). This is aimed at IT infrastructure and how to get it back up and running if things go wrong. Unlike the BCP, which is proactive and designed to identify and mitigate risks across the whole business, the DRP is reactive. It is designed to restore operations and outlines the steps needed to restart, reconfigure and recover systems and networks, and it includes data backup and restore. In the laboratory, choosing data systems that are designed to comply with zero-loss data security and continuous operation contributes to the resilience of both laboratory operations and the business overall.
Picking the right CDS
Within the pharma laboratory, chromatography-based and mass spectrometry-based analytics are fundamental to both development and manufacturing operations. This makes the choice of a chromatography data system (CDS) critically important, with an emphasis on a resilient architecture that offers the flexibility, scalability and robustness needed for both data security and business continuity. The software should also provide protection from unplanned downtime and safeguard against cyberattacks.
So, what are the key CDS capabilities for business continuity? Examining the various risks and the questions they raise helps illustrate what makes for a good CDS:
Power outage
What if servers go offline? What happens to the data in a full power outage, and what happens when the power comes back on?
Depending on where the power outage occurs, the CDS can help in different ways. A full power outage in the lab will mean that no instruments or computers are working. However, what is important is that the data from any applications running at the time is secured. If power goes off elsewhere in the building, and the local instrument controller PCs lose connection to the main server, an innovative CDS will help access data locally and keep the lab running. When power is restored, the CDS should automatically reconnect the instruments to the network and upload data.
Network failure
Network failure is arguably the most common issue labs face. So, what happens to data that was acquiring, how can data be accessed, and is it possible to start new acquisitions?
Again, the primary concern is to secure the data. Whatever was running must be stored and not lost. In a regulated environment all the data must remain compliant with data integrity maintained at all times.
An advanced CDS will ensure that the network-based resources required to properly operate the software, such as license information and user management data, are automatically cached on local instrument controllers and computers, so that operation can continue. If the network is down, the local instrument controller PCs will continue the analysis without interrupting data acquisition. This enables access to the data for processing and reporting and allows initiation of new sequences, all in accordance with compliance and data integrity guidelines.
Ideally, in the event of network failure, the CDS will enable a specified recovery period to allow sufficient time for resolution of the issue or implementation of a longer-term solution. Furthermore, switching to network failure mode should happen automatically and immediately without any manual intervention. Upon restoration of the network, the CDS must have the capability to upload the interrupted data, complete with audit trails from within the software because any data export and backup may have data integrity implications.
Third-party service provider outage
A recent data industry survey found that third-party services are uncommon sources of outages since the security offered by cloud service providers includes measures to prevent system failure and ensure continued access to services (Exhibit 2). However, they can occur, and this raises the question of whether or not it is possible to have a mixed setup that allows onsite as well as third-party data storage.
The key goal is to keep the lab running — so a CDS that enables the use of local and regional as well as global data centers increases system performance and reliability. As with network failure, in the event of cloud provider outage the CDS needs to operate using local systems that enable user logon, license availability, data access and the ability to acquire new data, all within the framework of data compliance and integrity.
Human error
Human error is a significant factor in operational failures, so increasing the levels of automation and reducing the need for personnel interactions with data systems is highly desirable.
IT departments have requirements to apply controls to operating systems and ensure appropriate user restrictions are in place. The CDS too must have a user management system that can either be linked to wider systems or sit as a separate operation. A CDS with a separate login allows controlled data access and provides an extra layer of security. Being able to roll out and update systems centrally from a remote position is also important and ensures all such processes take place in a controlled manner.
Within the CDS, increased automation ensures users carry out tasks in a consistent fashion and follow standard operating procedures. This means having a streamlined way of creating a sequence, and as much automation as possible around data processing. Being able to put all calculations into a validated and integrated reporting engine is also important, ensuring that everything is backed up and avoiding the use of uncontrolled tools, such as Excel, in the laboratory.
Software or hardware failure
Can the CDS help to automate switchover to another IT resource, and does it help with the robustness of the system?
While load balancing (the sharing of workload between two or more servers) and failover (automated switching between servers if one fails) is often handled by IT teams, there are benefits to having a CDS that provides its own local systems. Built-in load balancing and failover, when configured, increase the performance, reliability and robustness of the data system.
If failure does occur, then it must be possible to rapidly restore the system. Consistent, automated backup ensures that the CDS can be restored quickly without data loss. A service-level agreement with the CDS provider is crucial to ensure both fast support from the manufacturer and access to their wider knowledge base and issue resolution.
Unauthorized access and cyberattack
Whether it is accidental or purposeful, unauthorized access to data and software risks severe damage to the system and may result in inoperable software, data corruption or even data loss. To prevent this, controlled access is critically important, with multiple security layers offering the necessary protection.
Equally important is the architecture of the software, which should ideally separate administration and data management. This concept allows functional separation to minimize risk, improve system reliability and facilitate security. It enables central administration to be achieved, which helps remove bottlenecks from usage, improve efficiency and maintain greater system control.
In modern innovative software, data storage is managed through a dedicated Data Vault service that connects the database and file storage. Access is routed through this service to provide a high level of robustness, security and scalability. Secure encryption of all communications — data in transit or storage — ensures the highest network security and data integrity. Since the management system provided by the CDS maintains access control, users can only perform actions designated by their logon role. Data cannot be intercepted or lost, productivity can be maintained and there is less vulnerability to cyberattack.
Ensuring streamlined operations
Pharmaceutical laboratories are under increasing pressure to ensure 24/7 operations, and as more data is digitized there is particular concern over maintaining the continuity of digital systems and protecting valuable data. Business continuity planning and implementation is essential to mitigate this and must take account of all possible threats to operations and the impact each is likely to have. Laboratory data systems play a significant role in data security and in supporting business continuity, and when making selections it pays to examine exactly what an advanced CDS has to offer.
A modern CDS is equipped with intelligent tools that provide the flexibility, scalability and robustness required to ensure the security of laboratory data and support business continuity. The choice of system can positively affect a lab’s ability to plan successfully for unforeseen events and maintain 24/7 laboratory operation. To perform effectively, the CDS must meet the needs of both the laboratory and IT teams. It must deliver a solution that minimizes and protects against system outages and malicious attacks. And it has to enable uninterrupted access to data and instruments. Ultimately, making the right choice of CDS is a critical step in building robust operations.
