Data remanence
Encyclopedia
Data remanence is the residual representation of data
that remains even after attempts have been made to remove or erase the data. This residue may result from data being left intact by a nominal file deletion
operation, by reformatting of storage media that does not remove data previously written to the media, or through physical properties of the storage medium
that allow previously written data to be recovered. Data remanence may make inadvertent disclosure of sensitive information
possible, should the storage media be released into an uncontrolled environment (e.g., thrown in the trash, or given or sold to a third party).
Various techniques have been developed to counter data remanence. These techniques are classified as clearing, purging/sanitizing or destruction. Specific methods include overwriting, degaussing, encryption, and physical destruction.
Effective application of countermeasures can be complicated by several factors, including media that are inaccessible, media that cannot effectively be erased, advanced storage systems that maintain histories of data throughout the data's life cycle, and persistence of data in memory that is typically considered volatile.
Several standards exist for the secure removal of data and the elimination of data remanence.
s, file manager
s, and other software provide a facility where a file
is not immediately deleted
when the user requests that action. Instead, the file is moved to a holding area
, to allow the user to easily revert a mistake. Similarly, many software products automatically create backup copies of files that are being edited, to allow the user to restore the original version, or to recover from a possible crash (autosave
feature).
Even when an explicit deleted file retention facility is not provided or when the user does not use it, operating systems do not actually remove the contents of a file when it is deleted. Instead, they simply remove the file's entry from the file system
directory
, because this requires less work and is therefore faster. The contents of the file—the actual data—remain on the storage medium
. The data will remain there until the operating system
reuses the space for new data. In some systems, enough filesystem metadata
are also left behind to enable easy undeletion
by commonly available utility software
. Even when undelete has become impossible, the data, until it has been overwritten, can be read by software that reads disk sector
s directly. Computer forensics
often employs such software.
Likewise, reformatting
, repartitioning
or reimaging
a system is not always guaranteed to write to every area of the disk, though all will cause the disk to appear empty or, in the case of reimaging, empty except for the files present in the image, to most software.
Finally, even when the storage medium is overwritten, physical properties of the medium may make it possible to recover the previous contents. In most cases however, this recovery is not possible by just reading from the storage device in the usual way, but requires using laboratory techniques such as disassembling the device and directly accessing/reading from its components.
The section on complications gives further explanations for causes of data remanence.
Clearing is typically an administrative protection against accidental disclosure within an organization. For example, before a hard drive is re-used within an organization, its contents may be cleared to prevent their accidental disclosure to the next user.
The simplest overwrite technique writes the same data everywhere—often just a pattern of all zeros. At a minimum, this will prevent the data from being retrieved simply by reading from the medium again using standard system functions.
In an attempt to counter more advanced data recovery techniques, specific overwrite patterns and multiple passes have often been prescribed. These may be generic patterns intended to eradicate any trace signatures, for example, the seven-pass pattern: 0xF6, 0x00, 0xFF, random, 0x00, 0xFF, random; sometimes erroneously attributed to the US standard DOD 5220.22-M.
One challenge with an overwrite is that some areas of the disk may be inaccessible, due to media degradation or other errors. Software overwrite may also be problematic in high-security environments which require stronger controls on data commingling than can be provided by the software in use. The use of advanced storage technologies may also make file-based overwrite ineffective (see the discussion below under Complications).
There are specialized machines and software that are capable of doing overwriting. The software can sometimes be a standalone Operating System specifically designed for data destruction. There are also machines specifically designed to wipe hard drives to the department of defense specifications DOD 5220.22-M as well.
investigated data recovery from nominally overwritten media in the mid-1990s. He suggested magnetic force microscopy may be able to recover such data, and developed specific patterns, for specific drive technologies, designed to counter such. These patterns have come to be known as the Gutmann method
.
Daniel Feenberg, an economist at the private National Bureau of Economic Research
, claims that the chances of overwritten data being recovered from a modern hard drive amount to "urban legend". He also points to the "18½ minute gap" Rose Mary Woods
created on a tape of Richard Nixon
discussing the Watergate break-in. Erased information in the gap has not been recovered, and Feenberg claims doing so would be an easy task compared to recovery of a modern high density digital signal.
As of November 2007, the United States Department of Defense
considers overwriting acceptable for clearing magnetic media within the same security area/zone, but not as a sanitization method. Only degaussing or physical destruction is acceptable for the latter.
On the other hand, according to the 2006 NIST Special Publication 800-88 (p. 7): "Studies have shown that most of today’s media can be effectively cleared by one overwrite" and "for ATA disk drives manufactured after 2001 (over 15 GB) the terms clearing and purging have converged." An analysis by Wright et al. of recovery techniques, including magnetic force microscopy, also concludes that a single wipe is all that is required for modern drives. They point out that the long time required for multiple wipes "has created a situation
where many organisations ignore the issue all together – resulting in data leaks and
loss. "
, degaussing may purge an entire media element quickly and effectively.
Degaussing often renders hard disk
s inoperable, as it erases low-level formatting that is only done at the factory during manufacturing. It is possible, however, to return the drive to a functional state by having it serviced at the manufacturer. Degaussed floppy disks can generally be reformatted and reused with standard consumer hardware.
In some high-security environments, one may be required to use a degausser that has been approved for the task. For example, in US
government and military jurisdictions, one may be required to use a degausser from the NSA
's "Evaluated Products List".
data before it is stored on the medium may mitigate concerns about data remanence. If the decryption key
is strong and carefully controlled (i.e., not itself subject to data remanence), it may effectively make any data on the medium unrecoverable. Even if the key is stored on the medium, it may prove easier or quicker to overwrite just the key, vs the entire disk.
Encryption may be done on a file-by-file
basis, or on the whole disk
. Cold boot attack
s are one of the few possible methods for subverting a full-disk encryption
method, as there is no possibility of storing the plain text key in an unencrypted section of the medium. See the section Complications: Data in RAM for further discussion.
Other side-channel attacks, like the use of hardware-based keyloggers or acquisition of a written note containing the decryption key, may offer a greater chance to success, but do not rely on weaknesses in the cryptographic method employed. As such, their relevance for this article is minor.
Specific destruction techniques include:
may develop new "bad sector
s" after data have been written, and tapes require inter-record gaps. Modern hard disk
s often feature automatic remapping of marginal sectors or tracks, which the OS
may not even be aware of. This problem is especially significant in solid state drives
(SSDs) that rely on relatively large relocated bad block tables. Attempts to counter data remanence by overwriting may not be successful in such situations, as data remnants may persist in such nominally inaccessible areas.
Journaling file system
s increase the integrity of data by recording write operations in multiple locations, and applying transaction
-like semantics. On such systems, data remnants may exist in locations "outside" the nominal file storage location.
Some file systems implement copy-on-write
or built-in revision control
, with the intent that writing to a file never overwrites data in-place.
Technologies such as RAID
and anti-fragmentation
techniques may result in file data being written to multiple locations, either by design (for fault tolerance
), or as data remnants.
Wear leveling
can also defeat data erasure, by relocating blocks between the time when they are originally written and the time when they are overwritten.
are not magnetic, they cannot be erased by degaussing. Write-once
optical media (CD-R
, DVD-R
, etc.) also cannot be purged by overwriting. Read/write optical media, such as CD-RW
and DVD-RW
, may be receptive to overwriting. Methods for successfully sanitizing optical discs include delaminating or abrading the metallic data layer, shredding, incinerating, destructive electrical arcing (as by exposure to microwave energy), and submersion in a polycarbonate solvent (e.g., acetone).
s (SSDs). Researchers discovered three problems with file storage on SSDs:
Flash-based solid-state drives differ from hard drives in two ways: first, in the way data is stored and second, the way the algorithms are used to manage and access that data. These differences can be exploited to recover previously erased data. SSDs maintain a layer of indirection between the logical addresses used by computer systems to access data and the internal addresses that identify physical storage. This layer of indirection enhances SSD performance and reliability by hiding idiosyncratic interfaces and managing flash memory's limited lifetime. But it can also produce copies of the data that are invisible to the user and that a sophisticated attacker could recover. For sanitizing entire disks, sanitize commands built into the SSD hardware have been found to be effective when implemented correctly, and software-only techniques for sanitizing entire disks have been found to work most, but not all, of the time. In testing, none of the software techniques were effective for sanitizing individual files. These included well-known algorithms such as the Gutmann method
, US DoD 5220.22-M
, RCMP TSSIT OPS-II, Schneier 7 Pass, and Mac OS X Secure Erase Trash.
(SRAM), which is typically considered volatile (i.e., contents are erased with loss of electrical power). In the study, data retention was sometimes observed even at room temperature.
Data remanence has also been observed in dynamic random access memory
(DRAM). Modern DRAM chips have a built-in self-refresh module, as they not only require a power supply to retain data, but must also be periodically refreshed to prevent their data contents from fading away from the capacitors in their integrated circuits. A study found data remanence in DRAM with data retention of seconds to minutes at room temperature and "a full week without refresh when cooled with liquid nitrogen." The study authors were able to use a cold boot attack
to recover cryptographic keys
for several popular full disk encryption
systems, including Microsoft Bitlocker
, Apple FileVault
, dm-crypt
for Linux, and TrueCrypt
. Despite some memory degradation, they were able to take advantage of redundancy in the way keys are stored after they have been expanded for efficient use, such as in key scheduling. The authors recommend that computers be powered down, rather than be left in a "sleep
" state, when not in physical control of the owner. In some cases, such as certain modes of the software program Bitlocker, the authors recommend that a boot password or a key on a removable USB device be used.
Canada
New Zealand
United States
There are dozens of other tools for various operating systems.
Data
The term data refers to qualitative or quantitative attributes of a variable or set of variables. Data are typically the results of measurements and can be the basis of graphs, images, or observations of a set of variables. Data are often viewed as the lowest level of abstraction from which...
that remains even after attempts have been made to remove or erase the data. This residue may result from data being left intact by a nominal file deletion
File deletion
File deletion is a way of removing a file from a computer's file system.The reasons for deleting files are#Freeing the disk space#Removing duplicate or unnecessary data to avoid confusion#Making sensitive information unavailable to others...
operation, by reformatting of storage media that does not remove data previously written to the media, or through physical properties of the storage medium
Data storage device
thumb|200px|right|A reel-to-reel tape recorder .The magnetic tape is a data storage medium. The recorder is data storage equipment using a portable medium to store the data....
that allow previously written data to be recovered. Data remanence may make inadvertent disclosure of sensitive information
Information sensitivity
Information sensitivity is the control of access to information or knowledge that might result in loss of an advantage or level of security if disclosed to others who might have low or unknown trustability or undesirable intentions....
possible, should the storage media be released into an uncontrolled environment (e.g., thrown in the trash, or given or sold to a third party).
Various techniques have been developed to counter data remanence. These techniques are classified as clearing, purging/sanitizing or destruction. Specific methods include overwriting, degaussing, encryption, and physical destruction.
Effective application of countermeasures can be complicated by several factors, including media that are inaccessible, media that cannot effectively be erased, advanced storage systems that maintain histories of data throughout the data's life cycle, and persistence of data in memory that is typically considered volatile.
Several standards exist for the secure removal of data and the elimination of data remanence.
Causes
Many operating systemOperating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
s, file manager
File manager
A file manager or file browser is a computer program that provides a user interface to work with file systems. The most common operations performed on files or groups of files are: create, open, edit, view, print, play, rename, move, copy, delete, search/find, and modify file attributes, properties...
s, and other software provide a facility where a file
Computer file
A computer file is a block of arbitrary information, or resource for storing information, which is available to a computer program and is usually based on some kind of durable storage. A file is durable in the sense that it remains available for programs to use after the current program has finished...
is not immediately deleted
File deletion
File deletion is a way of removing a file from a computer's file system.The reasons for deleting files are#Freeing the disk space#Removing duplicate or unnecessary data to avoid confusion#Making sensitive information unavailable to others...
when the user requests that action. Instead, the file is moved to a holding area
Recycle bin (computing)
In computing, the trash is temporary storage for files that have been deleted in a file manager by the user, but not yet permanently erased from the physical media...
, to allow the user to easily revert a mistake. Similarly, many software products automatically create backup copies of files that are being edited, to allow the user to restore the original version, or to recover from a possible crash (autosave
Autosave
Autosave is a function in many computer applications or programs which saves an opened document automatically, helping to reduce the risk or impact of data loss in case of a crash or freeze...
feature).
Even when an explicit deleted file retention facility is not provided or when the user does not use it, operating systems do not actually remove the contents of a file when it is deleted. Instead, they simply remove the file's entry from the file system
File system
A file system is a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device which contain it. A file system organizes data in an efficient manner and is tuned to the...
directory
Directory (file systems)
In computing, a folder, directory, catalog, or drawer, is a virtual container originally derived from an earlier Object-oriented programming concept by the same name within a digital file system, in which groups of computer files and other folders can be kept and organized.A typical file system may...
, because this requires less work and is therefore faster. The contents of the file—the actual data—remain on the storage medium
Data storage device
thumb|200px|right|A reel-to-reel tape recorder .The magnetic tape is a data storage medium. The recorder is data storage equipment using a portable medium to store the data....
. The data will remain there until the operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
reuses the space for new data. In some systems, enough filesystem metadata
Metadata
The term metadata is an ambiguous term which is used for two fundamentally different concepts . Although the expression "data about data" is often used, it does not apply to both in the same way. Structural metadata, the design and specification of data structures, cannot be about data, because at...
are also left behind to enable easy undeletion
Undeletion
Undeletion is a feature for restoring computer files which have been removed from a file system by file deletion. Deleted data can be recovered on many file systems, but not all file systems provide an undeletion feature. Recovering data without an undeletion facility is usually called data...
by commonly available utility software
Utility software
Utility software is system software designed to help analyze, configure, optimize or maintain a computer. A single piece of utility software is usually called a utility or tool....
. Even when undelete has become impossible, the data, until it has been overwritten, can be read by software that reads disk sector
Disk sector
In computer disk storage, a sector is a subdivision of a track on a magnetic disk or optical disc. Each sector stores a fixed amount of user data. Traditional formatting of these storage media provides space for 512 bytes or 2048 bytes of user-accessible data per sector...
s directly. Computer forensics
Computer forensics
Computer forensics is a branch of digital forensic science pertaining to legal evidence found in computers and digital storage media...
often employs such software.
Likewise, reformatting
Disk formatting
Disk formatting is the process of preparing a hard disk drive or flexible disk medium for data storage. In some cases, the formatting operation may also create one or more new file systems...
, repartitioning
Disk partitioning
Disk partitioning is the act of dividing a hard disk drive into multiple logical storage units referred to as partitions, to treat one physical disk drive as if it were multiple disks. Partitions are also termed "slices" for operating systems based on BSD, Solaris or GNU Hurd...
or reimaging
Disk image
A disk image is a single file or storage device containing the complete contents and structure representing a data storage medium or device, such as a hard drive, tape drive, floppy disk, CD/DVD/BD, or USB flash drive, although an image of an optical disc may be referred to as an optical disc image...
a system is not always guaranteed to write to every area of the disk, though all will cause the disk to appear empty or, in the case of reimaging, empty except for the files present in the image, to most software.
Finally, even when the storage medium is overwritten, physical properties of the medium may make it possible to recover the previous contents. In most cases however, this recovery is not possible by just reading from the storage device in the usual way, but requires using laboratory techniques such as disassembling the device and directly accessing/reading from its components.
The section on complications gives further explanations for causes of data remanence.
Countermeasures
There are three levels commonly recognized for eliminating remnant data:Clearing
Clearing is the removal of sensitive data from storage devices in such a way that there is assurance that the data may not be reconstructed using normal system functions or software file/data recovery utilities. The data may still be recoverable, but not without special laboratory techniques.Clearing is typically an administrative protection against accidental disclosure within an organization. For example, before a hard drive is re-used within an organization, its contents may be cleared to prevent their accidental disclosure to the next user.
Purging
Purging or sanitising is the removal of sensitive data from a system or storage device with the intent that the data can not be reconstructed by any known technique. Purging, proportional to the sensitivity of the data, is generally done before releasing media outside of control, such as before discarding old media, or moving media to a computer with different security requirements.Destruction
The storage medium is physically destroyed. Effectiveness of physical destruction varies. Depending on recording density of the medium, and/or the destruction technique, this may leave data recoverable by laboratory methods. Conversely, physical destruction using appropriate techniques is generally considered the most secure method available.Overwriting
A common method used to counter data remanence is to overwrite the storage medium with new data. This is often called wiping or shredding a file or disk. Because such methods can often be implemented in software alone, and may be able to selectively target only part of a medium, it is a popular, low-cost option for some applications. Overwriting is generally an acceptable method of clearing, as long as the media is writable and not damaged.The simplest overwrite technique writes the same data everywhere—often just a pattern of all zeros. At a minimum, this will prevent the data from being retrieved simply by reading from the medium again using standard system functions.
In an attempt to counter more advanced data recovery techniques, specific overwrite patterns and multiple passes have often been prescribed. These may be generic patterns intended to eradicate any trace signatures, for example, the seven-pass pattern: 0xF6, 0x00, 0xFF, random, 0x00, 0xFF, random; sometimes erroneously attributed to the US standard DOD 5220.22-M.
One challenge with an overwrite is that some areas of the disk may be inaccessible, due to media degradation or other errors. Software overwrite may also be problematic in high-security environments which require stronger controls on data commingling than can be provided by the software in use. The use of advanced storage technologies may also make file-based overwrite ineffective (see the discussion below under Complications).
There are specialized machines and software that are capable of doing overwriting. The software can sometimes be a standalone Operating System specifically designed for data destruction. There are also machines specifically designed to wipe hard drives to the department of defense specifications DOD 5220.22-M as well.
Feasibility of recovering overwritten data
Peter GutmannPeter Gutmann (computer scientist)
Peter Gutmann is a computer scientist in the Department of Computer Science at the University of Auckland, Auckland, New Zealand. He has a Ph.D. in computer science from the University of Auckland. His Ph.D. thesis and a book based on the thesis were about a cryptographic security architecture...
investigated data recovery from nominally overwritten media in the mid-1990s. He suggested magnetic force microscopy may be able to recover such data, and developed specific patterns, for specific drive technologies, designed to counter such. These patterns have come to be known as the Gutmann method
Gutmann method
The Gutmann method is an algorithm for securely erasing the contents of computer hard drives, such as files. Devised by Peter Gutmann and Colin Plumb, it does so by writing a series of 35 patterns over the region to be erased....
.
Daniel Feenberg, an economist at the private National Bureau of Economic Research
National Bureau of Economic Research
The National Bureau of Economic Research is an American private nonprofit research organization "committed to undertaking and disseminating unbiased economic research among public policymakers, business professionals, and the academic community." The NBER is well known for providing start and end...
, claims that the chances of overwritten data being recovered from a modern hard drive amount to "urban legend". He also points to the "18½ minute gap" Rose Mary Woods
Rose Mary Woods
Rose Mary Woods was Richard Nixon's secretary from his days in the Congress in 1951, through his Vice Presidency, Presidency, and until the end of his political career. Before H.R...
created on a tape of Richard Nixon
Richard Nixon
Richard Milhous Nixon was the 37th President of the United States, serving from 1969 to 1974. The only president to resign the office, Nixon had previously served as a US representative and senator from California and as the 36th Vice President of the United States from 1953 to 1961 under...
discussing the Watergate break-in. Erased information in the gap has not been recovered, and Feenberg claims doing so would be an easy task compared to recovery of a modern high density digital signal.
As of November 2007, the United States Department of Defense
United States Department of Defense
The United States Department of Defense is the U.S...
considers overwriting acceptable for clearing magnetic media within the same security area/zone, but not as a sanitization method. Only degaussing or physical destruction is acceptable for the latter.
On the other hand, according to the 2006 NIST Special Publication 800-88 (p. 7): "Studies have shown that most of today’s media can be effectively cleared by one overwrite" and "for ATA disk drives manufactured after 2001 (over 15 GB) the terms clearing and purging have converged." An analysis by Wright et al. of recovery techniques, including magnetic force microscopy, also concludes that a single wipe is all that is required for modern drives. They point out that the long time required for multiple wipes "has created a situation
where many organisations ignore the issue all together – resulting in data leaks and
loss. "
Degaussing
Degaussing is the removal or reduction of a magnetic field of a disk or drive, using a device called a degausser that has been designed for the media being erased. Applied to magnetic mediaMagnetic storage
Magnetic storage and magnetic recording are terms from engineering referring to the storage of data on a magnetized medium. Magnetic storage uses different patterns of magnetization in a magnetizable material to store data and is a form of non-volatile memory. The information is accessed using...
, degaussing may purge an entire media element quickly and effectively.
Degaussing often renders hard disk
Hard disk
A hard disk drive is a non-volatile, random access digital magnetic data storage device. It features rotating rigid platters on a motor-driven spindle within a protective enclosure. Data is magnetically read from and written to the platter by read/write heads that float on a film of air above the...
s inoperable, as it erases low-level formatting that is only done at the factory during manufacturing. It is possible, however, to return the drive to a functional state by having it serviced at the manufacturer. Degaussed floppy disks can generally be reformatted and reused with standard consumer hardware.
In some high-security environments, one may be required to use a degausser that has been approved for the task. For example, in US
United States
The United States of America is a federal constitutional republic comprising fifty states and a federal district...
government and military jurisdictions, one may be required to use a degausser from the NSA
National Security Agency
The National Security Agency/Central Security Service is a cryptologic intelligence agency of the United States Department of Defense responsible for the collection and analysis of foreign communications and foreign signals intelligence, as well as protecting U.S...
's "Evaluated Products List".
Encryption
EncryptingEncryption
In cryptography, encryption is the process of transforming information using an algorithm to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key. The result of the process is encrypted information...
data before it is stored on the medium may mitigate concerns about data remanence. If the decryption key
Key (cryptography)
In cryptography, a key is a piece of information that determines the functional output of a cryptographic algorithm or cipher. Without a key, the algorithm would produce no useful result. In encryption, a key specifies the particular transformation of plaintext into ciphertext, or vice versa...
is strong and carefully controlled (i.e., not itself subject to data remanence), it may effectively make any data on the medium unrecoverable. Even if the key is stored on the medium, it may prove easier or quicker to overwrite just the key, vs the entire disk.
Encryption may be done on a file-by-file
Filesystem-level encryption
Filesystem-level encryption, often called file or folder encryption, is a form of disk encryption where individual files or directories are encrypted by the file system itself...
basis, or on the whole disk
Disk encryption
Disk encryption is a special case of data at rest protection when the storage media is a sector-addressable device . This article presents cryptographic aspects of the problem...
. Cold boot attack
Cold boot attack
In cryptography, a cold boot attack is a type of side channel attack in which an attacker with physical access to a computer is able to retrieve encryption keys from a running operating system after using a cold reboot to restart the machine from a completely "off" state...
s are one of the few possible methods for subverting a full-disk encryption
Disk encryption
Disk encryption is a special case of data at rest protection when the storage media is a sector-addressable device . This article presents cryptographic aspects of the problem...
method, as there is no possibility of storing the plain text key in an unencrypted section of the medium. See the section Complications: Data in RAM for further discussion.
Other side-channel attacks, like the use of hardware-based keyloggers or acquisition of a written note containing the decryption key, may offer a greater chance to success, but do not rely on weaknesses in the cryptographic method employed. As such, their relevance for this article is minor.
Physical destruction
Thorough physical destruction of the entire data storage medium is generally considered the most certain way to counter data remanence. However, the process is generally time-consuming and cumbersome. Physical destruction may require extremely thorough methods, as even a small media fragment may contain large amounts of data.Specific destruction techniques include:
- Physically breaking the media apart, by grinding, shredding, etc.
- Incinerating
- Phase transitionPhase transitionA phase transition is the transformation of a thermodynamic system from one phase or state of matter to another.A phase of a thermodynamic system and the states of matter have uniform physical properties....
(i.e., liquefaction or vaporization of a solid disk) - Application of corrosiveCorrosiveA corrosive substance is one that will destroy or irreversibly damage another surface or substance with which it comes into contact. The main hazards to people include damage to the eyes, the skin, and the tissue under the skin; inhalation or ingestion of a corrosive substance can damage the...
chemicals, such as acidAcidAn acid is a substance which reacts with a base. Commonly, acids can be identified as tasting sour, reacting with metals such as calcium, and bases like sodium carbonate. Aqueous acids have a pH of less than 7, where an acid of lower pH is typically stronger, and turn blue litmus paper red...
s, to recording surfaces - For magnetic media, raising its temperature above the Curie pointCurie pointIn physics and materials science, the Curie temperature , or Curie point, is the temperature at which a ferromagnetic or a ferrimagnetic material becomes paramagnetic on heating; the effect is reversible. A magnet will lose its magnetism if heated above the Curie temperature...
- For many electric volatile and non-volatile storage mediums, application of extremely high voltage as compared to safe operational specifications
Inaccessible media areas
Storage media may have areas which become inaccessible by normal means. For example, magnetic disksMagnetic storage
Magnetic storage and magnetic recording are terms from engineering referring to the storage of data on a magnetized medium. Magnetic storage uses different patterns of magnetization in a magnetizable material to store data and is a form of non-volatile memory. The information is accessed using...
may develop new "bad sector
Bad Sector
Bad Sector is an ambient/noise project formed in 1992 in Tuscany, Italy by Massimo Magrini. While working at the Computer Art Lab of ISTI in Pisa , he developed original gesture interfaces that he uses in live performances: 'Aerial Painting Hand' , 'UV-Stick' Bad Sector is an ambient/noise...
s" after data have been written, and tapes require inter-record gaps. Modern hard disk
Hard disk
A hard disk drive is a non-volatile, random access digital magnetic data storage device. It features rotating rigid platters on a motor-driven spindle within a protective enclosure. Data is magnetically read from and written to the platter by read/write heads that float on a film of air above the...
s often feature automatic remapping of marginal sectors or tracks, which the OS
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
may not even be aware of. This problem is especially significant in solid state drives
Solid-state drive
A solid-state drive , sometimes called a solid-state disk or electronic disk, is a data storage device that uses solid-state memory to store persistent data with the intention of providing access in the same manner of a traditional block i/o hard disk drive...
(SSDs) that rely on relatively large relocated bad block tables. Attempts to counter data remanence by overwriting may not be successful in such situations, as data remnants may persist in such nominally inaccessible areas.
Advanced storage systems
Data storage systems with more sophisticated features may make overwrite ineffective, especially on a per-file basis.Journaling file system
Journaling file system
A journaling file system is a file system that keeps track of the changes that will be made in a journal before committing them to the main file system...
s increase the integrity of data by recording write operations in multiple locations, and applying transaction
Transaction processing
In computer science, transaction processing is information processing that is divided into individual, indivisible operations, called transactions. Each transaction must succeed or fail as a complete unit; it cannot remain in an intermediate state...
-like semantics. On such systems, data remnants may exist in locations "outside" the nominal file storage location.
Some file systems implement copy-on-write
Copy-on-write
Copy-on-write is an optimization strategy used in computer programming. The fundamental idea is that if multiple callers ask for resources which are initially indistinguishable, they can all be given pointers to the same resource...
or built-in revision control
Revision control
Revision control, also known as version control and source control , is the management of changes to documents, programs, and other information stored as computer files. It is most commonly used in software development, where a team of people may change the same files...
, with the intent that writing to a file never overwrites data in-place.
Technologies such as RAID
RAID
RAID is a storage technology that combines multiple disk drive components into a logical unit...
and anti-fragmentation
File system fragmentation
In computing, file system fragmentation, sometimes called file system aging, is the inability of a file system to lay out related data sequentially , an inherent phenomenon in storage-backed file systems that allow in-place modification of their contents. It is a special case of data fragmentation...
techniques may result in file data being written to multiple locations, either by design (for fault tolerance
Fault-tolerant design
In engineering, fault-tolerant design is a design that enables a system to continue operation, possibly at a reduced level , rather than failing completely, when some part of the system fails...
), or as data remnants.
Wear leveling
Wear leveling
Wear leveling is a technique for prolonging the service life of some kinds of erasable computer storage media, such as Flash memory used in solid-state drives and USB Flash drives...
can also defeat data erasure, by relocating blocks between the time when they are originally written and the time when they are overwritten.
Optical media
As optical mediaOptical disc
In computing and optical disc recording technologies, an optical disc is a flat, usually circular disc which encodes binary data in the form of pits and lands on a special material on one of its flat surfaces...
are not magnetic, they cannot be erased by degaussing. Write-once
Write Once Read Many
A Write Once Read Many or WORM drive is a data storage device where information, once written, cannot be modified. On ordinary data storage devices, the number of times data can be modified is not limited, except by the rated lifespan of the device, as modification involves physical changes that...
optical media (CD-R
CD-R
A CD-R is a variation of the Compact Disc invented by Philips and Sony. CD-R is a Write Once Read Many optical medium, though the whole disk does not have to be entirely written in the same session....
, DVD-R
DVD-R
DVD-R is a DVD recordable format. A DVD-R typically has a storage capacity of 4.71 GB. Pioneer has also developed an 8.5 GB dual layer version, DVD-R DL, which appeared on the market in 2005....
, etc.) also cannot be purged by overwriting. Read/write optical media, such as CD-RW
CD-RW
A CD-RW is a rewritable optical disc. It was introduced in 1997, and was known as "CD-Writable" during development. It was preceded by the CD-MO, which was never commercially released....
and DVD-RW
DVD-RW
A DVD-RW disc is a rewritable optical disc with equal storage capacity to a DVD-R, typically 4.7 GB. The format was developed by Pioneer in November 1999 and has been approved by the DVD Forum. The smaller Mini DVD-RW holds 1.46 GB, with a diameter of 8 cm.The primary advantage of DVD-RW over...
, may be receptive to overwriting. Methods for successfully sanitizing optical discs include delaminating or abrading the metallic data layer, shredding, incinerating, destructive electrical arcing (as by exposure to microwave energy), and submersion in a polycarbonate solvent (e.g., acetone).
Data on solid-state drives
Research from the Center for Magnetic Recording and Research, University of California, San Diego has uncovered problems inherent in erasing data stored on solid-state driveSolid-state drive
A solid-state drive , sometimes called a solid-state disk or electronic disk, is a data storage device that uses solid-state memory to store persistent data with the intention of providing access in the same manner of a traditional block i/o hard disk drive...
s (SSDs). Researchers discovered three problems with file storage on SSDs:
- First, built-in commands are effective, but manufacturers sometimes implement them incorrectly. Second, overwriting the entire visible address space of an SSD twice is usually, but not always, sufficient to sanitize the drive. Third, none of the existing hard drive-oriented techniques for individual file sanitization are effective on SSDs.
Flash-based solid-state drives differ from hard drives in two ways: first, in the way data is stored and second, the way the algorithms are used to manage and access that data. These differences can be exploited to recover previously erased data. SSDs maintain a layer of indirection between the logical addresses used by computer systems to access data and the internal addresses that identify physical storage. This layer of indirection enhances SSD performance and reliability by hiding idiosyncratic interfaces and managing flash memory's limited lifetime. But it can also produce copies of the data that are invisible to the user and that a sophisticated attacker could recover. For sanitizing entire disks, sanitize commands built into the SSD hardware have been found to be effective when implemented correctly, and software-only techniques for sanitizing entire disks have been found to work most, but not all, of the time. In testing, none of the software techniques were effective for sanitizing individual files. These included well-known algorithms such as the Gutmann method
Gutmann method
The Gutmann method is an algorithm for securely erasing the contents of computer hard drives, such as files. Devised by Peter Gutmann and Colin Plumb, it does so by writing a series of 35 patterns over the region to be erased....
, US DoD 5220.22-M
National Industrial Security Program
The National Industrial Security Program, or NISP, is the nominal authority for managing the needs of private industry to access classified information.The NISP was established in 1993 by Executive Order 12829...
, RCMP TSSIT OPS-II, Schneier 7 Pass, and Mac OS X Secure Erase Trash.
Data in RAM
Data remanence has been observed in static random access memoryStatic random access memory
Static random-access memory is a type of semiconductor memory where the word static indicates that, unlike dynamic RAM , it does not need to be periodically refreshed, as SRAM uses bistable latching circuitry to store each bit...
(SRAM), which is typically considered volatile (i.e., contents are erased with loss of electrical power). In the study, data retention was sometimes observed even at room temperature.
Data remanence has also been observed in dynamic random access memory
Dynamic random access memory
Dynamic random-access memory is a type of random-access memory that stores each bit of data in a separate capacitor within an integrated circuit. The capacitor can be either charged or discharged; these two states are taken to represent the two values of a bit, conventionally called 0 and 1...
(DRAM). Modern DRAM chips have a built-in self-refresh module, as they not only require a power supply to retain data, but must also be periodically refreshed to prevent their data contents from fading away from the capacitors in their integrated circuits. A study found data remanence in DRAM with data retention of seconds to minutes at room temperature and "a full week without refresh when cooled with liquid nitrogen." The study authors were able to use a cold boot attack
Cold boot attack
In cryptography, a cold boot attack is a type of side channel attack in which an attacker with physical access to a computer is able to retrieve encryption keys from a running operating system after using a cold reboot to restart the machine from a completely "off" state...
to recover cryptographic keys
Key (cryptography)
In cryptography, a key is a piece of information that determines the functional output of a cryptographic algorithm or cipher. Without a key, the algorithm would produce no useful result. In encryption, a key specifies the particular transformation of plaintext into ciphertext, or vice versa...
for several popular full disk encryption
Full disk encryption
Disk encryption uses disk encryption software or hardware to encrypt every bit of data that goes on a disk or disk volume. Disk encryption prevents unauthorized access to data storage. The term "full disk encryption" is often used to signify that everything on a disk is encrypted, including the...
systems, including Microsoft Bitlocker
BitLocker Drive Encryption
BitLocker Drive Encryption is a full disk encryption feature included with the Ultimate and Enterprise editions of Microsoft's Windows Vista and Windows 7 desktop operating systems, as well as the Windows Server 2008 and Windows Server 2008 R2 server platforms. It is designed to protect data by...
, Apple FileVault
FileVault
FileVault is a system which encrypts files on a Macintosh computer. It can be found in the Mac OS X v10.4 "Tiger" operating system and later....
, dm-crypt
Dm-crypt
dm-crypt is a transparent disk encryption subsystem in Linux kernel versions 2.6 and later and in DragonFly BSD. It is part of the device mapper infrastructure, and uses cryptographic routines from the kernel's Crypto API...
for Linux, and TrueCrypt
TrueCrypt
TrueCrypt is a software application used for on-the-fly encryption . It is free and open source. It can create a virtual encrypted disk within a file or encrypt a partition or the entire storage device .- Operating systems :TrueCrypt supports Microsoft Windows, Mac OS X, and...
. Despite some memory degradation, they were able to take advantage of redundancy in the way keys are stored after they have been expanded for efficient use, such as in key scheduling. The authors recommend that computers be powered down, rather than be left in a "sleep
Power management
Power management is a feature of some electrical appliances, especially copiers, computers and computer peripherals such as monitors and printers, that turns off the power or switches the system to a low-power state when inactive. In computing this is known as PC power management and is built...
" state, when not in physical control of the owner. In some cases, such as certain modes of the software program Bitlocker, the authors recommend that a boot password or a key on a removable USB device be used.
Standards
Australia- DSDDefence Signals DirectorateDefence Signals Directorate is an Australian government intelligence agency responsible for signals intelligence and information security .-Overview:According to its website, DSD has two principal functions:...
ISM 2010, Australian Government Information Security Manual, Nov 2010
Canada
- RCMPRoyal Canadian Mounted PoliceThe Royal Canadian Mounted Police , literally ‘Royal Gendarmerie of Canada’; colloquially known as The Mounties, and internally as ‘The Force’) is the national police force of Canada, and one of the most recognized of its kind in the world. It is unique in the world as a national, federal,...
B2-002, IT Media Overwrite and Secure Erase Products, May 2009 - Communications Security EstablishmentCommunications Security EstablishmentThe Communications Security Establishment Canada is the Canadian government's national cryptologic agency. Administered under the Department of National Defence , it is charged with the duty of keeping track of foreign signals intelligence , and protecting Canadian government electronic...
Clearing and Declassifying Electronic Data Storage Devices, July 2006
New Zealand
- GCSBGovernment Communications Security BureauThe Government Communications Security Bureau is an intelligence agency of the New Zealand government.The mission statement is given as:To contribute to the national security of New Zealand through:...
NZISM 2010, New Zealand Information Security Manual, Dec 2010
United States
- NISTNational Institute of Standards and TechnologyThe National Institute of Standards and Technology , known between 1901 and 1988 as the National Bureau of Standards , is a measurement standards laboratory, otherwise known as a National Metrological Institute , which is a non-regulatory agency of the United States Department of Commerce...
Special Publication 800-88, Guidelines for Media Sanitization, September 2006 - DoD 5220.22-MNational Industrial Security ProgramThe National Industrial Security Program, or NISP, is the nominal authority for managing the needs of private industry to access classified information.The NISP was established in 1993 by Executive Order 12829...
, National Industrial Security Program Operating Manual (NISPOM), February 2006- Current editions no longer contain any references to specific sanitization methods. Standards for sanitization are left up to the Cognizant Security Authority.
- Although the NISPOM text itself never described any specific methods for sanitization, past editions (1995 and 1997) did contain explicit sanitization methods within the Defense Security Service (DSS) Clearing and Sanitization Matrix inserted after Section 8-306. The DSS still provides this matrix and it continues to specify methods. As of the Nov 2007 edition of the matrix, overwriting is no longer acceptable for sanitization of magnetic media. Only degaussing (with an NSA approved degausser) or physical destruction is acceptable.
- ArmyUnited States ArmyThe United States Army is the main branch of the United States Armed Forces responsible for land-based military operations. It is the largest and oldest established branch of the U.S. military, and is one of seven U.S. uniformed services...
AR380-19, Information Systems Security, February 1998 - Air ForceUnited States Air ForceThe United States Air Force is the aerial warfare service branch of the United States Armed Forces and one of the American uniformed services. Initially part of the United States Army, the USAF was formed as a separate branch of the military on September 18, 1947 under the National Security Act of...
AFSSI 8580, Remanence Security, 17 November 2008 (formerly AFSSI 5020) - NavyUnited States NavyThe United States Navy is the naval warfare service branch of the United States Armed Forces and one of the seven uniformed services of the United States. The U.S. Navy is the largest in the world; its battle fleet tonnage is greater than that of the next 13 largest navies combined. The U.S...
NAVSO P5239-26, Remanence Security, September 1993
See also
- Computer forensicsComputer forensicsComputer forensics is a branch of digital forensic science pertaining to legal evidence found in computers and digital storage media...
- CryptographyCryptographyCryptography is the practice and study of techniques for secure communication in the presence of third parties...
- Data erasureData erasureData erasure is a software-based method of overwriting data that completely destroys all electronic data residing on a hard disk drive or other digital media. Permanent data erasure goes beyond basic file deletion commands, which only remove direct pointers to data disk sectors and make data...
- Data recoveryData recoveryData recovery is the process of salvaging data from damaged, failed, corrupted, or inaccessible secondary storage media when it cannot be accessed normally. Often the data are being salvaged from storage media such as internal or external hard disk drives, solid-state drives , USB flash drive,...
- Electronic wasteElectronic wasteElectronic waste, e-waste, e-scrap, or Waste Electrical and Electronic Equipment describes discarded electrical or electronic devices. There is a lack of consensus as to whether the term should apply to resale, reuse, and refurbishing industries, or only to product that cannot be used for its...
- EncryptionEncryptionIn cryptography, encryption is the process of transforming information using an algorithm to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key. The result of the process is encrypted information...
- File deletionFile deletionFile deletion is a way of removing a file from a computer's file system.The reasons for deleting files are#Freeing the disk space#Removing duplicate or unnecessary data to avoid confusion#Making sensitive information unavailable to others...
- Forensic identificationForensic identificationForensic identification is the application of forensic science, or "forensics", and technology to identify specific objects from the trace evidence they leave, often at a crime scene or the scene of an accident. Forensic means "for the courts"....
- Gutmann methodGutmann methodThe Gutmann method is an algorithm for securely erasing the contents of computer hard drives, such as files. Devised by Peter Gutmann and Colin Plumb, it does so by writing a series of 35 patterns over the region to be erased....
- National Association for Information Destruction (NAID)
- Paper shredderPaper shredderA paper shredder is a mechanical device used to cut paper into chad, typically either strips or fine particles. Government organizations, businesses, and private individuals use shredders to destroy private, confidential, or otherwise sensitive documents...
- Physical information securityPhysical Information SecurityPhysical information security is concerned with physically protecting data and means to access that data . Many individuals and companies place importance in protecting their information from a software and/or network perspective, but fewer devote resources to protecting data physically...
- PlaintextPlaintextIn cryptography, plaintext is information a sender wishes to transmit to a receiver. Cleartext is often used as a synonym. Before the computer era, plaintext most commonly meant message text in the language of the communicating parties....
(security discussion) - Recovering Overwritten Data on the ForensicsWiki.
- Sanitization (classified information)Sanitization (classified information)Sanitization is the process of removing sensitive information from a document or other medium, so that it may be distributed to a broader audience. When dealing with classified information, sanitization attempts to reduce the document's classification level, possibly yielding an unclassified...
- Secure USB drive
- Universal Data Copy
- ZeroisationZeroisationIn cryptography, zeroisation is the practice of erasing sensitive parameters from a cryptographic module to prevent their disclosure if the equipment is captured. This is generally accomplished by altering or deleting the contents to prevent recovery of the data...
Software
- BCWipeBCWipeBCWipe is a commercial data erasure utility for Windows and UNIX, developed by Jetico Inc.-Features:*Delete with wiping - permanently delete selected files through right-click context menu...
- Darik's Boot and NukeDarik's Boot and NukeDarik's Boot and Nuke is an open source project hosted on SourceForge. The program is designed to securely erase a hard disk until data is permanently removed and no longer recoverable, which is achieved by overwriting the data with random numbers generated by Mersenne twister or ISAAC...
- Data ShredderData ShredderData Shredder is a data destruction utility designed to securely erase a hard disk or digital storage device, completely removing the data and making it unrecoverable...
- HDDeraseHDDeraseHDDerase is a freeware utility that securely erases data on hard drives using the security erase unit command built into the firmware of ATA and SATA drives manufactured after 2001. HDDerase was developed by the Center for Magnetic Recording Research at the University of California San Diego...
- PGP Desktop
- Shred (Unix)Shred (Unix)shred is a Unix command that can be used to securely delete files and devices so that they can be recovered only with great difficulty with specialised hardware, if at all. It is a part of GNU Core Utilities.-Background:...
(part of the GNUGNUGNU is a Unix-like computer operating system developed by the GNU project, ultimately aiming to be a "complete Unix-compatible software system"...
Coreutils package) - ProtectStar Data Shredder
- ProtectStar iShredder
There are dozens of other tools for various operating systems.