Data corruption

Overview

There are two types of data corruption associated with computer systems: undetected and detected. Undetected data corruption, also known as silent data corruption, results in the most dangerous errors as there is no indication that the data is incorrect. Detected data corruption may be permanent with the loss of data, or may be temporary when some part of the system is able to detect and correct the error; there is no data corruption in the latter case.

Data corruption can occur at any level in a system, from the host to the storage medium. Modern systems attempt to detect corruption at many layers and then recover or correct the corruption; this is almost always successful but very rarely the information arriving in the systems memory is corrupted and can cause unpredictable results.

Data corruption during transmission has a variety of causes. Interruption of data transmission causes information loss. Environmental conditions can interfere with data transmission, especially when dealing with wireless transmission methods. Heavy clouds can block satellite transmissions. Wireless networks are susceptible to interference from devices such as microwave ovens.

Hardware and software failure are the two main causes for data loss. Background radiation, head crashes, and aging or wear of the storage device fall into the former category, while software failure typically occurs due to bugs in the code. Cosmic rays cause most soft errors in DRAM.

{{Anchor|SILENT}}Silent

Some errors go unnoticed, without being detected by the disk firmware or the host operating system; these errors are known as silent data corruption.{{cite web

There are many error sources beyond the disk storage subsystem itself. For instance, cables might be slightly loose, the power supply might be unreliable, external vibrations such as a loud sound, the network might introduce undetected corruption, cosmic radiation and many other causes of soft memory errors, etc. In 39,000 storage systems that were analyzed, firmware bugs accounted for 5–10% of storage failures. The error rates as observed by a CERN study on silent corruption are far higher than one in every 1016 bits. Amazon Web Services acknowledged that data corruption was the cause of a widespread outage of their Amazon S3 storage network in 2008. In 2021, faulty processor cores were identified as an additional cause in publications by Google and Facebook; cores were found to be faulty at a rate of several in thousands of cores.

One problem is that hard disk drive capacities have increased substantially, but their error rates remain unchanged. The data corruption rate has always been roughly constant in time, meaning that modern disks are not much safer than old disks. In old disks the probability of data corruption was very small because they stored tiny amounts of data. In modern disks the probability is much larger because they store much more data, whilst not being safer. That way, silent data corruption has not been a serious concern while storage devices remained relatively small and slow. In modern times and with the advent of larger drives and very fast RAID setups, users are capable of transferring 1016 bits in a reasonably short time, thus easily reaching the data corruption thresholds.{{cite web |access-date = 14 December 2020 |archive-url = https://web.archive.org/web/20131029210013/http://www.necam.com/docs/?id=54157ff5-5de8-4966-a99d-341cf2cb27d3 |archive-date = 29 October 2013

As an example, ZFS creator Jeff Bonwick stated that the fast database at Greenplum, which is a database software company specializing in large-scale data warehousing and analytics, faces silent corruption every 15 minutes.{{cite web |access-date = 14 December 2020 |url-status = live |archive-url = https://web.archive.org/web/20110716221142/http://queue.acm.org/detail.cfm?id=1317400 |archive-date = 16 July 2011

Silent data corruption may result in cascading failures, in which the system may run for a period of time with undetected initial error causing increasingly more problems until it is ultimately detected. For example, a failure affecting file system metadata can result in multiple files being partially damaged or made completely inaccessible as the file system is used in its corrupted state.

{{Anchor|END-TO-END-PROTECTION|INTEL-REPLAY}}Countermeasures

When data corruption behaves as a Poisson process, where each bit of data has an independently low probability of being changed, data corruption can generally be detected by the use of checksums, and can often be corrected by the use of error correcting codes (ECC).

If an uncorrectable data corruption is detected, procedures such as automatic retransmission or restoration from backups can be applied. Certain levels of RAID disk arrays have the ability to store and evaluate parity bits for data across a set of hard disks and can reconstruct corrupted data upon the failure of a single or multiple disks, depending on the level of RAID implemented. Some CPU architectures employ various transparent checks to detect and mitigate data corruption in CPU caches, CPU buffers and instruction pipelines; an example is Intel Instruction Replay technology, which is available on Intel Itanium processors.{{cite web |access-date = 2016-01-27 |url-status = live |archive-url = https://web.archive.org/web/20160202125833/http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/itanium-9500-reliability-mission-critical-applications-paper.pdf |archive-date = 2016-02-02

Many errors are detected and corrected by the hard disk drives using the ECC codes which are stored on disk for each sector. If the disk drive detects multiple read errors on a sector it may make a copy of the failing sector on another part of the disk, by remapping the failed sector of the disk to a spare sector without the involvement of the operating system (though this may be delayed until the next write to the sector). This "silent correction" can be monitored using S.M.A.R.T. and tools available for most operating systems to automatically check the disk drive for impending failures by watching for deteriorating SMART parameters.

Some file systems, such as Btrfs, HAMMER, ReFS, and ZFS, use internal data and metadata checksumming to detect silent data corruption. In addition, if a corruption is detected and the file system uses integrated RAID mechanisms that provide data redundancy, such file systems can also reconstruct corrupted data in a transparent way.{{cite web |access-date = 2014-01-02 |url-status = live |archive-url = https://web.archive.org/web/20140102193726/http://www.oracle.com/technetwork/articles/servers-storage-admin/advanced-btrfs-1734952.html |archive-date = 2014-01-02 | access-date = 2014-08-12

Data scrubbing is another method to reduce the likelihood of data corruption, as disk errors are caught and recovered from before multiple errors accumulate and overwhelm the number of parity bits. Instead of parity being checked on each read, the parity is checked during a regular scan of the disk, often done as a low priority background process. The "data scrubbing" operation activates a parity check. If a user simply runs a normal program that reads data from the disk, then the parity would not be checked unless parity-check-on-read was both supported and enabled on the disk subsystem.

If appropriate mechanisms are employed to detect and remedy data corruption, data integrity can be maintained. This is particularly important in commercial applications (e.g. banking), where an undetected error could either corrupt a database index or change data to drastically affect an account balance, and in the use of encrypted or compressed data, where a small error can make an extensive dataset unusable.

References

References

1. Scientific American. (2008-07-21). "Solar Storms: Fast Facts". [[Nature Publishing Group]].
2. Eric Lowe. (16 November 2005). "ZFS saves the day(-ta)!". Oracle.
3. bcantrill. (31 December 2008). "Shouting in the Datacenter". YouTube.
4. jforonda. (31 January 2007). "Faulty FC port meets ZFS". Blogger – Outside the Box.
5. "Are Disks the Dominant Contributor for Storage Failures? A Comprehensive Study of Storage Subsystem Failure Characteristics". USENIX.
6. Bernd Panzer-Steindel. (8 April 2007). "Draft 1.3". CERN.
7. "AWS Service Availability".
8. (2021). "Proceedings of the Workshop on Hot Topics in Operating Systems".
9. (27 May 2021). "HotOS 2021: Cores That Don't Count (Fun Hardware)".
10. David S. H. Rosenthal. (October 1, 2010). "Keeping Bits Safe: How Hard Can It Be?". ACM Queue.
11. "Silent corruptions".
12. (November 2012). "Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing". [[IEEE]].
13. "Read Error Severities and Error Management Logic".