Consigning SneakerNet to the Graveyard of Technology

“SneakerNet,” or the manual transfer of data using a disk or USB stick from one computer system to another, should be long dead, but this noncompliant transfer process still survives. In this edition of “Focus on Quality,” we discuss why SneakerNet should be consigned to the graveyard of technology.

Walk through any laboratory and you’ll see a variety of spectrometers with their own data systems to control the instrument, to acquire and process data, and then report the results. Look a little harder and you’ll see that most (if not all) of these systems are stand-alone, unconnected to a IT network, and their output is paper. A few may be interfaced to a laboratory information management system (LIMS) for data transfer, but these are in the minority. Instead of manually typing results into a LIMS or another computerized system from the paper printouts, the more enlightened are using “SneakerNet” to transfer data from one system to another.

In the last century, the term SneakerNet was coined to describe the practice of a person (presumably wearing a mandatory pair of sneakers) transferring data on a floppy disk from one computer system to another instead of transferring data electronically by interfacing the two systems together, or manually entering data into the other computer system. Originally, the process involved 5.25-inch disks (if you are old and can remember that far back), and then matured to use higher capacity 3.25-inch disks (your memory may still be a bit hazy). You would think that, in the third decade of the 21st century, this method of data transfer would be consigned to the graveyard of technology. But no, alas, it has matured further to encompass even higher capacity universal serial bus (USB) devices. Unfortunately, SneakerNet is alive and kicking. The purpose of this column can be summarized by quoting that eminent spectroscopist William Shakespeare:

I come to bury SneakerNet, not to praise it, The evil that men do lives after them.

As we shall see, the evil of using USB devices for transferring regulated good clinical, laboratory, and manufacturing practices (GxP) data are not held in the best regard by either regulatory authorities, company IT departments, cybersecurity guidance, or me. Also, we will consider what is the impact of using a USB stick to transfer regulated GxP data; do you need to archive the USB stick used to transfer the data? (Note to self: Consider buying shares in companies making USB sticks)

USB devices can be split into two main areas: external hard drives and USB sticks or thumb drives. We will only consider the use of USB sticks or thumb drives for transferring data by SneakerNet, and exclude the use of USB external drives for automatic data backup from this discussion.

Why Use USB Data Transfer?

Look at the top workflow of Figure 1; a SneakerNet data transfer by USB seems very innocuous and simple. Just think about it—plug a USB device into the instrument data system, drag and drop the required data file to the stick via Windows Explorer, remove the device, then wander over to a LIMS workstation, plug in the USB stick, and drag and drop the data to a LIMS directory for upload into the database. Easy peasy! Cheap, quick, none of that painful validation stuff, and you can even use your own stick with your latest holiday photos and malware to show colleagues.

FIGURE 1: SneakerNet and interfaced transfer mechanisms between an instrument data system and a LIMS.

What’s the problem with this approach? First, let’s see what the regulators say about this.

Dirty Deeds Done Dead Cheap

We’ll start our discussion with two companies that have earned the highest form of praise from the US Food and Drug Administration (FDA) by way of a warning letter. The first is Zhejiang Hisun Pharmaceutical Co., Ltd that had a warning letter in 2015:

We note that some records we requested during the inspection were not provided in a timely manner.

As we shall see, this is an understatement.

During the inspection, an analyst removed a USB thumb drive from a computer controlling an HPLC (high-performance liquid chromatography instrument). When asked to provide the drive, the analyst instead exited the room with the thumb drive. After approximately 15 minutes, management provided our investigator with what they asserted was the USB thumb drive in question. It is impossible to know whether management provided the same USB thumb drive that the analyst had removed (1).

This was followed by a warning that the company was delaying and limiting the inspection (2). Using the USB thumb drive was bad, but the analyst doing a runner with it was unconscionable.

Our second example, BBT Biotech, did not use a USB drive but a CD, however, it is not the medium but the means of data transfer that is important. Citation 4 is a “failure to exercise sufficient controls over computerized systems.....to provide adequate controls to prevent omission of data.”

..... Our investigator also noted that your firm copied raw data to a CD <redacted>, and then deleted the data from the <redacted> system to free space on the hard drive. Files copied to the CD were selected manually; the selection process was not supervised. Without audit trail capabilities or supervised file selection, there was no assurance that all raw data files were copied to the CD before they were permanently deleted from the system (3).

It seems that the FDA are not happy with USB sticks and manual transfer of data without checks. Let’s consider what advice is offered by data integrity regulatory guidance.

PIC/S Data Integrity Guidance

The draft Pharmaceutical Inspection Cooperation Scheme (PIC/S) data integrity guidance (4) has in Section 9.3 a specific section on restricting the use of USB devices:

For reasons of system security, USB ports should be default disabled on computer clients and servers hosting GxP critical data. If necessary, ports should only be opened for approved purposes and all USB devices should be properly scanned before use. The use of private USB devices (flash drives, cameras, smartphones, keyboards, etc.) on company computer clients and servers hosting GxP data, or the use of company USB devices on private computers, should not be allowed.

Rationale: This is especially important for Windows environments where system vulnerabilities are known that allow USB devices to trick the computer, by pretending to be another external device, e.g. keyboard, and can contain and start executable code.

Summarizing and interpreting this section:

• Disable USB ports on computers and servers by default
• USB devices must be only used for approved purposes
• Only use company supplied and approved USB devices
• Don’t use company USB devices on private computers
• Private USB devices are not permitted under any circumstances
• Before use the device must be scanned, typically on a separate isolated computer to prevent infection

What Are the Problems with USB Data Transfer?

What the warning letter citations and regulatory guidance cover is relatively high-level but we need to go into more detail to see the real problems. We’ll start with the transfer process that was mentioned in the BBT BioTech warning letter using manual transfer (3), then slide further down the SneakerNet slippery slope.

• Drag and Drop Transfer: This is a manual process that can be selective either intentionally or by mistake. If a transfer to a USB drive is to be used, as a minimum any transfer to and from the stick should be automatic and validated.
• Data File Format: This is where there could be serious issues as in many cases the file transferred is a comma separated value (CSV) or spreadsheet file format. The file is usually unprotected, with no audit trail to monitor if there have been any changes. The issues with spreadsheets have been discussed recently (5,6), but an unprotected text or CSV file is a critical data integrity vulnerability. Although files can be protected by a checksum or encryption, it adds complexity to a process that should not even be considered.
• File Transfer Location: Where does the file reside after transfer to the receiving computer? Is it in a directory that has restricted access or open to anybody? How long is the file left before it is imported into the LIMS or the second system.
• Second Person Review: What looked to be a simple transfer now is a can of worms when the lucky individual who is the second person reviewer comes on the scene. As the process is not validated, they need to check the data in the instrument data system, the file transferred and the data in the LIMS as a minimum. This will slow the overall business process.

What started as a simple and ideal solution for data transfer at the start of this column, SneakerNet now is turning into a regulatory nightmare, but the nightmare is not over yet.

Cyber Security and USB Devices

Users and analytical instrument suppliers who like USB devices won’t like this section at all. Here we discuss the IT and cybersecurity perspectives on using USB sticks, regardless if you are a regulated laboratory or not. Rather than quote verbatim all different requirements, this section summarizes the cybersecurity from the US Cybersecurity and Infrastructure Security Agency (7), and an unnamed major pharmaceutical company’s requirements for (not) using USB sticks. The rationale for this list is that hackers can use USB drives to infect other computers with malware that can detect when the USB drive is plugged into a computer. The malware then downloads malicious code onto the drive. When the USB drive is plugged into another computer, the malware infects that computer (7). The advice is:

• Avoid using USB whenever possible
• Use of USB devices should be the last resort
• Limit USB devices to approved and documented business needs (justification or as part of the user requirements).
• Do not allow any writing to USB devices if this is not justified and authorized by management (the process/ data owner).
• Configure active devices (workstations and servers) not to auto-run content from USB drives.
• Monitor the network to detect external device use.
• If allowed, ensure that there is an automatic anti-virus scan of the USB drive when connected.
• If USB sticks are permitted, any data written to them must be encrypted.
• Never, never, ever use an unknown USB device. Indeed, some guidance suggests only allowing the use of specific USB device identified to an individual serial number (providing this can be enforced by network software).

Many of these points are like those abstracted from the PIC/S guidance earlier. The requirements for USB devices is in addition to keeping all software current and patching as soon as practicable to avoid security breaches that malware can take advantage of.

User’s and Supplier’s Responsibilities

What is the root cause of the problem? In my view, it is in part the design of analytical instruments and the data systems that are designed to work in standalone mode with little or no consideration for networking. To protect the electronic records generated, there must be an option for the software to store data on a secure and resilient drive on a network. However, the other part of the problem are the users failing to specify better connectivity. Suppliers are market driven, and if the market does not require this, then suppliers come up with USB options for data transfer.

Therefore, it is the responsibility of both regulated users to demand better design of laboratory applications to avoid the need to use poor methods of data transfer and avoid buying solutions not offering this.

Exit, Pursued by a Bear

Now, we come to a question posed in the beginning of this column: Do we need to archive the USB sticks used to transfer regulated data files between systems? Here we need to consider the FDA data integrity guidance and question 12 (8): When does electronic data become a current good manufacturing practice (CGMP) record? I have copied the applicable text which is shown in italic and added my comments underneath each.

When generated to satisfy a CGMP requirement, all data become a CGMP record.

If you are working on a regulated analysis this is self-evident within the instrument. What about the file you are transferring on the USB stick for further analysis? Is this a GMP record? Of course, it is as shown in Figure 1.

You must document, or save, the data at the time of performance to create a record in compliance with CGMP requirements, including, but not limited to, §§ 211.100(b) and 211.160(a) (8).

The transfer file is a key component in the data chain from acquisition and interpretation by the instrument to further analysis and reporting. The question is can you delete the transfer file from the USB stick or must you retain it? The problem is that the transfer has not been validated and who authorizes the deletion of each GMP record?

FDA expects processes to be designed so that data required to be created and maintained cannot be modified without a record of the modification. ...... Similarly, it is not acceptable to store electronic records in a manner that allows for manipulation without creating a permanent record (8).

If the file is unprotected, how do you ensure that the data have not been modified after transfer and up to the point of import? This is the role of the second person review, and they will need to check the source and files in the originating and receiving systems to demonstrate there is no change. The extent of the checks required are shown in Figure 2, and at its simplest is a comparison of the source and transferred files but there are other potential data vulnerabilities where the file could be modified and the ideal review would be all stages from the instrument data system to the LIMS. If the file were protected, say by a checksum, has the process been defined and validated to reduce the amount of checking? Probably not. I would strongly suggest that SneakerNet is not a viable approach to data transfer under data integrity guidance and current interpretation of regulations. What should we do?

FIGURE 2: Problem with SneakerNet transfer and the extent of a second person review.

Alternative Interfacing Approaches

One alternative approach is to interface a standalone instrument data system to the network and have a script that automatically transfers a file to a secure directory on the LIMS server for the LIMS to scan and automatically import. In terms of data flow the direct network connection to a LIMS is better than USB SneakerNet. The script would be started by a user who would enter the file name to be transferred and the rest would be performed by the script. Naturally, this process must be defined and validated but would drastically reduce the amount of second person review required.

An ideal solution is to use a computerized system that is designed for regulatory compliance with controls to protect e-records and capture all appropriate data and metadata, including audit trail entries of data export to LIMS. The data system should be interfaced to the LIMS using an Application Programming Interface. This is programmed by a supplier and is validated by the laboratory, but is a far more secure interfacing approach that ensures data integrity as no file is created during data transfer between the applications.

Summary

We have discussed the regulatory, business and cybersecurity problems of SneakerNet data transfer typically using USB sticks in lieu of instrument interfacing. USB transfer should not be used. Alternatively, are you going to take the Clint Eastwood approach to risk management with SneakerNet: Am I feeling lucky?

Acknowledgments

I would like to thank John Waldron for helpful comments in writing this article.

References

(1) FDA Warning Letter: Zhejiang Hisun Pharmaceutical Co., Ltd (Warning Letter: 320-16-06) (Food and Drug Administration, Silver Spring, Maryland, 2015)

(2) FDA Guidance for Industry Circumstances that Constitute Delaying, Denying, Limiting, or Refusing a Drug Inspection (Food and Drug Administration: Rockville, Maryland, 2014).

(3) FDA Warning Letter BBT Biotech Gmbh (Warning Letter 320-16-12). (Food and Drug Administration, Silver Spring, Maryland, 2016)

(4) PIC/S PI-041-3 Good Practices for Data Management and Integrity in Regulated GMP / GDP Environments Draft (Pharmaceutical Inspection Convention/Pharmaceutical Inspection Cooperation Scheme, Geneva, Switzerland, 2018).

(5) R.D.McDowall, LCGC Europe 33(9), 468–476 (2020).

(6) R.D.McDowall, Spectroscopy 35(9), 27–31 (2020).

(7) Security Tip (ST08-001): Using Caution with USB Drives (2019) Available from: https://us-cert.cisa.gov/ncas/tips/ST08-001

(8) FDA Guidance for Industry Data Integrity and Compliance With Drug CGMP Questions and Answers (Food and Drug Administration, Silver Spring, Maryland, 2018)

R.D. McDowall is the director of R.D. McDowall Limited and the editor of the “Questions of Quality” column for LCGC Europe, Spectroscopy’s sister magazine. Direct correspondence to: SpectroscopyEdit@MMHGroup.com ●