Memory-mapped file
Encyclopedia
A memory-mapped file is a segment of virtual memory
which has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource. This resource is typically a file that is physically present on-disk, but can also be a device, shared memory object, or other resource that the operating system
can reference through a file descriptor
. Once present, this correlation between the file and the memory space permits applications to treat the mapped portion as if it were primary memory.
(file cache), meaning that no copies need to be created in user space.
Certain application level memory-mapped file operations also perform better than their physical file counterparts. Applications can access and update data in the file directly and in-place, as opposed to seeking from the start of the file or rewriting the entire edited contents to a temporary location. Since the memory-mapped file is handled internally in pages, linear file access (as seen, for example, in flat file data storage or configuration files) requires disk access only when a new page boundary is crossed, and can write larger sections of the file to disk in a single operation.
A possible benefit of memory-mapped files is a "lazy loading", thus using small amounts of RAM even for a very large file. Trying to load the entire contents of a file that is significantly larger than the amount of memory available can cause severe thrashing as the operating system reads from disk into memory and simultaneously pages from memory back to disk. Memory-mapping may not only bypass the page file completely, but the system only needs to load the smaller page-sized sections as data is being edited, similarly to demand paging
scheme used for programs.
The memory mapping process is handled by the virtual memory manager
, which is the same subsystem responsible for dealing with the page file. Memory mapped files are loaded into memory one entire page at a time. The page size is selected by the operating system for maximum performance. Since page file management is one of the most critical elements of a virtual memory system, loading page sized sections of a file into physical memory is typically a very highly optimized system function.
, but is not yet mapped into the process's virtual memory space. In some circumstances, memory mapped file I/O can be substantially slower than standard file I/O.
Another drawback of memory mapped files relates to a given architecture's address space
: a file larger than the addressable space can have only portions mapped at a time, complicating reading it. For example, a 32-bit
architecture such as Intel's IA-32
can only directly address 4 GiB
or smaller portions of files. This drawback is avoided in the case of devices addressing memory when an IOMMU
is present.
and Unix-like
systems.) When a process is started, the operating system uses a memory mapped file to bring the executable file, along with any loadable modules, into memory for execution. Most memory-mapping systems use a technique called demand paging
, where the file is loaded into physical memory in subsets (one page each), and only when that page is actually referenced. In the specific case of executable files, this permits the OS to selectively load only those portions of a process image that actually need to execute.
Another common use for memory-mapped files is to share memory between multiple processes. In modern protected mode
operating systems, processes are generally not permitted to access memory space that is allocated for use by another process. (A program's attempt to do so causes invalid page faults or segmentation violations.) There are a number of techniques available to safely share memory, and memory-mapped file I/O is one of the most popular. Two or more applications can simultaneously map a single physical file into memory and access this memory. For example, the Microsoft Windows operating system provides a mechanism for applications to memory-map a shared segment of the system's page file itself and share data via this section.
, which creates a mapping of a file given a file descriptor, starting location in the file, and a length, is part of the POSIX
specification, so the wide variety of POSIX-compliant systems, such as UNIX
, Linux
, Mac OS X
or OpenVMS
, support a common mechanism for memory mapping files. The Microsoft Windows operating systems also support a group of API functions for this purpose, such as CreateFileMapping.
The Boost C++ Libraries provide a portable implementation of memory-mapped files for Microsoft Windows and POSIX-compliant platforms.
The Java programming language
provides classes and methods to access memory mapped files, such as .
The D programming language
supports memory mapped files in its standard library (std.mmfile module).
Ruby
has a gem (library) called Mmap, which implements memory-mapped file objects.
Since version 1.6, Python has included a mmap module in its Standard Library. Details of the module vary according to whether the host platform is Windows
or Unix-like
.
For Perl
there are a several modules available for memory mapping files on the CPAN
, such as Sys::Mmap and File::Map.
In the Microsoft .NET runtime, P/Invoke can be used to use memory mapped files directly through the Windows API
. Managed access (P/Invoke not necessary) to memory mapped files was introduced in version 4 of the runtime (see Memory-Mapped Files). For previous versions, there are third-party libraries which provide managed API's.
PHP supported memory-mapping techniques in a number of native file access functions such as file_get_contents but has removed this in 5.3 (see revision log).
Virtual memory
In computing, virtual memory is a memory management technique developed for multitasking kernels. This technique virtualizes a computer architecture's various forms of computer data storage , allowing a program to be designed as though there is only one kind of memory, "virtual" memory, which...
which has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource. This resource is typically a file that is physically present on-disk, but can also be a device, shared memory object, or other resource that the operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
can reference through a file descriptor
File descriptor
In computer programming, a file descriptor is an abstract indicator for accessing a file. The term is generally used in POSIX operating systems...
. Once present, this correlation between the file and the memory space permits applications to treat the mapped portion as if it were primary memory.
Benefits
The primary benefit of memory mapping a file is increasing I/O performance, especially when used on large files. For small files, memory-mapped files can result in a waste of slack space as memory maps are always aligned to the page size, which is mostly 4 KiB. Therefore a 5 KiB file will allocate 8 KiB and thus 3 KiB are wasted. Accessing memory mapped files is faster than using direct read and write operations for two reasons. Firstly, a system call is orders of magnitude slower than a simple change to a program's local memory. Secondly, in most operating systems the memory region mapped actually is the kernel's page cachePage cache
In computing, page cache, sometimes ambiguously called disk cache, is a "transparent" buffer of disk-backed pages kept in main memory by the operating system for quicker access. Page cache is typically implemented in kernels with the paging memory management, and is completely transparent to...
(file cache), meaning that no copies need to be created in user space.
Certain application level memory-mapped file operations also perform better than their physical file counterparts. Applications can access and update data in the file directly and in-place, as opposed to seeking from the start of the file or rewriting the entire edited contents to a temporary location. Since the memory-mapped file is handled internally in pages, linear file access (as seen, for example, in flat file data storage or configuration files) requires disk access only when a new page boundary is crossed, and can write larger sections of the file to disk in a single operation.
A possible benefit of memory-mapped files is a "lazy loading", thus using small amounts of RAM even for a very large file. Trying to load the entire contents of a file that is significantly larger than the amount of memory available can cause severe thrashing as the operating system reads from disk into memory and simultaneously pages from memory back to disk. Memory-mapping may not only bypass the page file completely, but the system only needs to load the smaller page-sized sections as data is being edited, similarly to demand paging
Demand paging
In computer operating systems, demand paging is an application of virtual memory. In a system that uses demand paging, the operating system copies a disk page into physical memory only if an attempt is made to access it...
scheme used for programs.
The memory mapping process is handled by the virtual memory manager
Memory management
Memory management is the act of managing computer memory. The essential requirement of memory management is to provide ways to dynamically allocate portions of memory to programs at their request, and freeing it for reuse when no longer needed. This is critical to the computer system.Several...
, which is the same subsystem responsible for dealing with the page file. Memory mapped files are loaded into memory one entire page at a time. The page size is selected by the operating system for maximum performance. Since page file management is one of the most critical elements of a virtual memory system, loading page sized sections of a file into physical memory is typically a very highly optimized system function.
Drawbacks
The major reason to choose memory mapped file I/O is performance. Nevertheless there can be tradeoffs. The standard I/O approach is costly due to system call overhead and memory copying. The memory mapped approach has its cost in minor page faults - when a block of data is loaded in page cachePage cache
In computing, page cache, sometimes ambiguously called disk cache, is a "transparent" buffer of disk-backed pages kept in main memory by the operating system for quicker access. Page cache is typically implemented in kernels with the paging memory management, and is completely transparent to...
, but is not yet mapped into the process's virtual memory space. In some circumstances, memory mapped file I/O can be substantially slower than standard file I/O.
Another drawback of memory mapped files relates to a given architecture's address space
Memory address
A digital computer's memory, more specifically main memory, consists of many memory locations, each having a memory address, a number, analogous to a street address, at which computer programs store and retrieve, machine code or data. Most application programs do not directly read and write to...
: a file larger than the addressable space can have only portions mapped at a time, complicating reading it. For example, a 32-bit
32-bit
The range of integer values that can be stored in 32 bits is 0 through 4,294,967,295. Hence, a processor with 32-bit memory addresses can directly access 4 GB of byte-addressable memory....
architecture such as Intel's IA-32
IA-32
IA-32 , also known as x86-32, i386 or x86, is the CISC instruction-set architecture of Intel's most commercially successful microprocessors, and was first implemented in the Intel 80386 as a 32-bit extension of x86 architecture...
can only directly address 4 GiB
Gib
Gib may refer to:* A castrated male cat or ferret* Gibibit , measurement unit of digitally stored computer information* Gibraltar, British overseas territory* Drywall, construction material...
or smaller portions of files. This drawback is avoided in the case of devices addressing memory when an IOMMU
IOMMU
In computing, an input/output memory management unit is a memory management unit that connects a DMA-capable I/O bus to the main memory...
is present.
Common uses
Perhaps the most common use for a memory-mapped file is the process loader in most modern operating systems (including Microsoft WindowsMicrosoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
and Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
systems.) When a process is started, the operating system uses a memory mapped file to bring the executable file, along with any loadable modules, into memory for execution. Most memory-mapping systems use a technique called demand paging
Demand paging
In computer operating systems, demand paging is an application of virtual memory. In a system that uses demand paging, the operating system copies a disk page into physical memory only if an attempt is made to access it...
, where the file is loaded into physical memory in subsets (one page each), and only when that page is actually referenced. In the specific case of executable files, this permits the OS to selectively load only those portions of a process image that actually need to execute.
Another common use for memory-mapped files is to share memory between multiple processes. In modern protected mode
Protected mode
In computing, protected mode, also called protected virtual address mode, is an operational mode of x86-compatible central processing units...
operating systems, processes are generally not permitted to access memory space that is allocated for use by another process. (A program's attempt to do so causes invalid page faults or segmentation violations.) There are a number of techniques available to safely share memory, and memory-mapped file I/O is one of the most popular. Two or more applications can simultaneously map a single physical file into memory and access this memory. For example, the Microsoft Windows operating system provides a mechanism for applications to memory-map a shared segment of the system's page file itself and share data via this section.
Platform support
Most modern operating systems or runtime environments support some form of memory-mapped file access. The function mmapMmap
In computing, mmap is a POSIX-compliant Unix system call that maps files or devices into memory. It is a method of memory-mapped file I/O. It naturally implements demand paging, because initially file contents are not entirely read from disk and do not use physical RAM at all...
, which creates a mapping of a file given a file descriptor, starting location in the file, and a length, is part of the POSIX
POSIX
POSIX , an acronym for "Portable Operating System Interface", is a family of standards specified by the IEEE for maintaining compatibility between operating systems...
specification, so the wide variety of POSIX-compliant systems, such as UNIX
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
, Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
, Mac OS X
Mac OS X
Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems...
or OpenVMS
OpenVMS
OpenVMS , previously known as VAX-11/VMS, VAX/VMS or VMS, is a computer server operating system that runs on VAX, Alpha and Itanium-based families of computers. Contrary to what its name suggests, OpenVMS is not open source software; however, the source listings are available for purchase...
, support a common mechanism for memory mapping files. The Microsoft Windows operating systems also support a group of API functions for this purpose, such as CreateFileMapping.
The Boost C++ Libraries provide a portable implementation of memory-mapped files for Microsoft Windows and POSIX-compliant platforms.
The Java programming language
Java (programming language)
Java is a programming language originally developed by James Gosling at Sun Microsystems and released in 1995 as a core component of Sun Microsystems' Java platform. The language derives much of its syntax from C and C++ but has a simpler object model and fewer low-level facilities...
provides classes and methods to access memory mapped files, such as .
The D programming language
D (programming language)
The D programming language is an object-oriented, imperative, multi-paradigm, system programming language created by Walter Bright of Digital Mars. It originated as a re-engineering of C++, but even though it is mainly influenced by that language, it is not a variant of C++...
supports memory mapped files in its standard library (std.mmfile module).
Ruby
Ruby (programming language)
Ruby is a dynamic, reflective, general-purpose object-oriented programming language that combines syntax inspired by Perl with Smalltalk-like features. Ruby originated in Japan during the mid-1990s and was first developed and designed by Yukihiro "Matz" Matsumoto...
has a gem (library) called Mmap, which implements memory-mapped file objects.
Since version 1.6, Python has included a mmap module in its Standard Library. Details of the module vary according to whether the host platform is Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
or Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
.
For Perl
Perl
Perl is a high-level, general-purpose, interpreted, dynamic programming language. Perl was originally developed by Larry Wall in 1987 as a general-purpose Unix scripting language to make report processing easier. Since then, it has undergone many changes and revisions and become widely popular...
there are a several modules available for memory mapping files on the CPAN
CPAN
CPAN, the Comprehensive Perl Archive Network, is an archive of nearly 100,000 modules of software written in Perl, as well as documentation for it. It has a presence on the World Wide Web at and is mirrored worldwide at more than 200 locations...
, such as Sys::Mmap and File::Map.
In the Microsoft .NET runtime, P/Invoke can be used to use memory mapped files directly through the Windows API
Windows API
The Windows API, informally WinAPI, is Microsoft's core set of application programming interfaces available in the Microsoft Windows operating systems. It was formerly called the Win32 API; however, the name "Windows API" more accurately reflects its roots in 16-bit Windows and its support on...
. Managed access (P/Invoke not necessary) to memory mapped files was introduced in version 4 of the runtime (see Memory-Mapped Files). For previous versions, there are third-party libraries which provide managed API's.
PHP supported memory-mapping techniques in a number of native file access functions such as file_get_contents but has removed this in 5.3 (see revision log).