Wget
Encyclopedia
GNU Wget is a computer program
that retrieves content from web servers, and is part of the GNU Project
. Its name is derived from World Wide Web
and get. It supports downloading via HTTP, HTTPS
, and FTP
protocols.
Its features include recursive download, conversion of links for offline viewing of local HTML, support for proxies, and much more. It appeared in 1996, coinciding with the boom of popularity of the Web, causing its wide use among Unix
users and distribution with most major GNU/Linux based distributions. Written in portable
C
, Wget can be easily installed on any Unix-like system and has been ported to many environments, including Microsoft Windows
, Mac OS X
, OpenVMS
, MorphOS
and AmigaOS
.
It has been used as the basis for graphical programs such as GWget for the GNOME
Desktop and KGet
for the KDE
Desktop.
program named GetURL, written by James Burton in AREXX
.
Wget filled a gap in the web downloading software available in the mid-1990s. No single program was able to reliably download files via both HTTP and FTP
protocols. Existing programs either only supported FTP (such as NcFTP
and [ftp://gnjilux.srk.fer.hr/pub/unix/util/dl/ dl]) or were written in Perl
, which was not yet ubiquitous at the time. While Wget was inspired by features of some of the existing programs, it aimed to support both HTTP and FTP and to enable the users to build it using only the standard development tools found on every Unix
system.
At that time many Unix users struggled behind extremely slow university and dial-up
Internet
connections, leading to a growing need for a downloading agent that could deal with transient network failures without assistance from the human operator.
problem, Wget will automatically try to continue the download from where it left off, and repeat this until the whole file has been retrieved. It was one of the first clients to make use of the then-new
by extracting resources linked
from HTML
pages
and downloading them in sequence, repeating the process recursively
until all the pages have been downloaded or a maximum recursion depth specified by the user has been reached. The downloaded pages are saved in a directory structure resembling that on the remote server. This "recursive download" enables partial or complete mirroring of web sites via HTTP. Links in downloaded HTML pages can be adjusted to point to locally downloaded material for offline viewing. When performing this kind of automatic mirroring
of web sites, Wget supports the Robots Exclusion Standard
(unless the option
Recursive download works with FTP
as well, where Wget issues the
. Shell-like wildcards
are supported when the download of FTP URLs is requested.
When downloading recursively over either HTTP or FTP
, Wget can be instructed to inspect the timestamps of local and remote files, and download only the remote files newer than the corresponding local ones. This allows easy mirroring of HTTP and FTP
sites, but is considered inefficient and more error-prone when compared to programs designed for mirroring from the ground up, such as rsync
. On the other hand, Wget doesn't require special server-side software for this task.
, being able to log its progress to a separate file for later inspection. That way the user can start Wget and log off, leaving the program unattended. By contrast, most graphical
or text user interface
web browser
s require the user to remain logged in and to manually restart failed downloads, which can be a great hindrance when transferring a lot of data.
with minimal dependencies on third-party libraries, Wget requires little more than a C compiler and a BSD-like interface to TCP/IP
networking. Designed as a Unix
program invoked from the Unix shell
, the program has been ported to numerous Unix-like environments and systems, including Microsoft Windows
via Cygwin
, and Mac OS X. It is also available as a native Microsoft Windows
program as one of the GnuWin packages.
More complex usage includes automatic download of multiple URLs into a directory hierarchy.
Collect only specific links listed line by line in the local file "my_movies.txt". Use a random wait of 0 to 33 seconds between files, and use 512 kilobytes per second of bandwidth throttling
. When there is a failure, retry for up to 22 times with 48 seconds between each retry. Send no tracking user agent
or HTTP referer
to a restrictive site and ignore robot exclusions. Place all the captured files in the local "movies" directory and collect the access results to the local file "my_movies.log". Good for downloading specific sets of files without hogging the network:-
Instead of an empty referer
and user-agent
use a real one that does not cause an “ERROR: 403 Forbidden” message from a restrictive site. It is also possible to create a .wgetrc file that holds some default values.
To get around cookie tracked sessions:-
Mirror and convert CGI, ASP or PHP and others to HTML for offline browsing:-
s, also included with the program. Wget is currently maintained by Giuseppe Scrivano.
The copyright to Wget belongs to the Free Software Foundation
, whose policy is to require copyright assignments for all non-trivial contributions to GNU software.
, version 3 or later, with a special exception that allows distribution of binaries linked against the OpenSSL
library. The text of the exception follows:
It is expected that the exception clause will be removed once Wget is modified to also link with the GnuTLS
library.
Wget's documentation
, in the form of a Texinfo
reference manual, is distributed under the terms of the GNU Free Documentation License
, version 1.2 or later. The man page usually distributed on Unix-like systems is automatically generated from a subset of the Texinfo manual and falls under the terms of the same license.
generated by the diff
utility. Patches intended for inclusion in Wget are submitted to the mailing list where they are reviewed by the maintainers. Patches that pass the maintainers' scrutiny are installed in the sources. Instructions on patch creation as well as style guidelines are outlined on the project's wiki.
The source code can also be tracked via a remote version control repository that hosts revision history beginning with the 1.5.3 release. The repository is running Bazaar. Previously, Mercurial
was used. Before that, it had been hosted on Subversion, and prior to that via CVS
.
Releases are numbered as versions of the form of major.minor[.revision], such as Wget 1.11 or Wget 1.8.2. An increase of the major version number represents large and possibly incompatible changes in Wget's behavior or a radical redesign of the code base. An increase of the minor version number designates addition of new features and bug fixes. A new revision indicates a release that, compared to the previous revision, only contains bug fixes. Revision zero is omitted, meaning that for example Wget 1.11 is the same as 1.11.0. Wget does not use the odd-even release number convention popularized by the Linux.
frontend for of Wget. It is developed by David Sedeño Fernández and is part of the GNOME
project. GWget supports all of the main features that Wget does, as well as parallel downloads.
. It is the default download manager of Konqueror
, but can also be used with Mozilla Firefox
. KGet is part of the KDE Network package.
Computer program
A computer program is a sequence of instructions written to perform a specified task with a computer. A computer requires programs to function, typically executing the program's instructions in a central processor. The program has an executable form that the computer can use directly to execute...
that retrieves content from web servers, and is part of the GNU Project
GNU Project
The GNU Project is a free software, mass collaboration project, announced on September 27, 1983, by Richard Stallman at MIT. It initiated GNU operating system development in January, 1984...
. Its name is derived from World Wide Web
World Wide Web
The World Wide Web is a system of interlinked hypertext documents accessed via the Internet...
and get. It supports downloading via HTTP, HTTPS
Https
Hypertext Transfer Protocol Secure is a combination of the Hypertext Transfer Protocol with SSL/TLS protocol to provide encrypted communication and secure identification of a network web server...
, and FTP
File Transfer Protocol
File Transfer Protocol is a standard network protocol used to transfer files from one host to another host over a TCP-based network, such as the Internet. FTP is built on a client-server architecture and utilizes separate control and data connections between the client and server...
protocols.
Its features include recursive download, conversion of links for offline viewing of local HTML, support for proxies, and much more. It appeared in 1996, coinciding with the boom of popularity of the Web, causing its wide use among Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
users and distribution with most major GNU/Linux based distributions. Written in portable
Porting
In computer science, porting is the process of adapting software so that an executable program can be created for a computing environment that is different from the one for which it was originally designed...
C
C (programming language)
C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
, Wget can be easily installed on any Unix-like system and has been ported to many environments, including Microsoft Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
, Mac OS X
Mac OS X
Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems...
, OpenVMS
OpenVMS
OpenVMS , previously known as VAX-11/VMS, VAX/VMS or VMS, is a computer server operating system that runs on VAX, Alpha and Itanium-based families of computers. Contrary to what its name suggests, OpenVMS is not open source software; however, the source listings are available for purchase...
, MorphOS
MorphOS
MorphOS is an Amiga-compatible computer operating system. It is a mixed proprietary and open source OS produced for the Pegasos PowerPC processor based computer, PowerUP accelerator equipped Amiga computers, and a series of Freescale development boards that use the Genesi firmware, including the...
and AmigaOS
AmigaOS
AmigaOS is the default native operating system of the Amiga personal computer. It was developed first by Commodore International, and initially introduced in 1985 with the Amiga 1000...
.
It has been used as the basis for graphical programs such as GWget for the GNOME
GNOME
GNOME is a desktop environment and graphical user interface that runs on top of a computer operating system. It is composed entirely of free and open source software...
Desktop and KGet
KGet
KGet is a free download manager for KDE. By default it is the download manager used for Konqueror, but can also be used with Mozilla Firefox. KGet is part of the KDE Network package, and was featured by Tux Magazine and Free Software Magazine.-History:...
for the KDE
KDE
KDE is an international free software community producing an integrated set of cross-platform applications designed to run on Linux, FreeBSD, Microsoft Windows, Solaris and Mac OS X systems...
Desktop.
History
Wget is the descendant of an earlier program named Geturl by the same author, the development of which commenced in late 1995. The name was changed to Wget after the author became aware of an earlier AmigaAmiga
The Amiga is a family of personal computers that was sold by Commodore in the 1980s and 1990s. The first model was launched in 1985 as a high-end home computer and became popular for its graphical, audio and multi-tasking abilities...
program named GetURL, written by James Burton in AREXX
ARexx
ARexx is an implementation of the REXX language for the Amiga, written in 1987 by William S. Hawes, with a number of Amiga-specific features beyond standard REXX facilities. Like most REXX implementations, ARexx is an interpreted language...
.
Wget filled a gap in the web downloading software available in the mid-1990s. No single program was able to reliably download files via both HTTP and FTP
File Transfer Protocol
File Transfer Protocol is a standard network protocol used to transfer files from one host to another host over a TCP-based network, such as the Internet. FTP is built on a client-server architecture and utilizes separate control and data connections between the client and server...
protocols. Existing programs either only supported FTP (such as NcFTP
NcFTP
NcFTP is an FTP client program which debuted in 1990 as the first alternative FTP client. It was created as an alternative to the standard UNIX ftp program, and offers a number of additional features and greater ease of use....
and [ftp://gnjilux.srk.fer.hr/pub/unix/util/dl/ dl]) or were written in Perl
Perl
Perl is a high-level, general-purpose, interpreted, dynamic programming language. Perl was originally developed by Larry Wall in 1987 as a general-purpose Unix scripting language to make report processing easier. Since then, it has undergone many changes and revisions and become widely popular...
, which was not yet ubiquitous at the time. While Wget was inspired by features of some of the existing programs, it aimed to support both HTTP and FTP and to enable the users to build it using only the standard development tools found on every Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
system.
At that time many Unix users struggled behind extremely slow university and dial-up
Dial-up access
Dial-up Internet access is a form of Internet access that uses the facilities of the public switched telephone network to establish a dialled connection to an Internet service provider via telephone lines...
Internet
Internet
The Internet is a global system of interconnected computer networks that use the standard Internet protocol suite to serve billions of users worldwide...
connections, leading to a growing need for a downloading agent that could deal with transient network failures without assistance from the human operator.
Robustness
Wget has been designed for robustness over slow or unstable network connections. If a download does not complete due to a networkComputer network
A computer network, often simply referred to as a network, is a collection of hardware components and computers interconnected by communication channels that allow sharing of resources and information....
problem, Wget will automatically try to continue the download from where it left off, and repeat this until the whole file has been retrieved. It was one of the first clients to make use of the then-new
Range HTTP header to support this feature.Recursive download
Wget can optionally work like a web crawlerWeb crawler
A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or—especially in the FOAF community—Web scutters.This process is called Web...
by extracting resources linked
Hyperlink
In computing, a hyperlink is a reference to data that the reader can directly follow, or that is followed automatically. A hyperlink points to a whole document or to a specific element within a document. Hypertext is text with hyperlinks...
from HTML
HTML
HyperText Markup Language is the predominant markup language for web pages. HTML elements are the basic building-blocks of webpages....
pages
Web page
A web page or webpage is a document or information resource that is suitable for the World Wide Web and can be accessed through a web browser and displayed on a monitor or mobile device. This information is usually in HTML or XHTML format, and may provide navigation to other web pages via hypertext...
and downloading them in sequence, repeating the process recursively
Recursion
Recursion is the process of repeating items in a self-similar way. For instance, when the surfaces of two mirrors are exactly parallel with each other the nested images that occur are a form of infinite recursion. The term has a variety of meanings specific to a variety of disciplines ranging from...
until all the pages have been downloaded or a maximum recursion depth specified by the user has been reached. The downloaded pages are saved in a directory structure resembling that on the remote server. This "recursive download" enables partial or complete mirroring of web sites via HTTP. Links in downloaded HTML pages can be adjusted to point to locally downloaded material for offline viewing. When performing this kind of automatic mirroring
Mirror (computing)
In computing, a mirror is an exact copy of a data set. On the Internet, a mirror site is an exact copy of another Internet site.Mirror sites are most commonly used to provide multiple sources of the same information, and are of particular value as a way of providing reliable access to large downloads...
of web sites, Wget supports the Robots Exclusion Standard
Robots Exclusion Standard
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to...
(unless the option
-e robots=off is used).Recursive download works with FTP
File Transfer Protocol
File Transfer Protocol is a standard network protocol used to transfer files from one host to another host over a TCP-based network, such as the Internet. FTP is built on a client-server architecture and utilizes separate control and data connections between the client and server...
as well, where Wget issues the
LIST command to find which additional files to download, repeating this process for directories and files under the one specified in the top URLUniform Resource Locator
In computing, a uniform resource locator or universal resource locator is a specific character string that constitutes a reference to an Internet resource....
. Shell-like wildcards
Wildcard character
-Telecommunication:In telecommunications, a wildcard character is a character that may be substituted for any of a defined subset of all possible characters....
are supported when the download of FTP URLs is requested.
When downloading recursively over either HTTP or FTP
File Transfer Protocol
File Transfer Protocol is a standard network protocol used to transfer files from one host to another host over a TCP-based network, such as the Internet. FTP is built on a client-server architecture and utilizes separate control and data connections between the client and server...
, Wget can be instructed to inspect the timestamps of local and remote files, and download only the remote files newer than the corresponding local ones. This allows easy mirroring of HTTP and FTP
File Transfer Protocol
File Transfer Protocol is a standard network protocol used to transfer files from one host to another host over a TCP-based network, such as the Internet. FTP is built on a client-server architecture and utilizes separate control and data connections between the client and server...
sites, but is considered inefficient and more error-prone when compared to programs designed for mirroring from the ground up, such as rsync
Rsync
rsync is a software application and network protocol for Unix-like and Windows systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar...
. On the other hand, Wget doesn't require special server-side software for this task.
Non-interactiveness
Wget is non-interactive in the sense that, once started, it does not require user interaction and does not need to control a TTYTty (Unix)
tty is a Unix command that prints to standard output the name of the terminal connected to standard input. The name of the program comes from teletypewriter, abbreviated "TTY".When the program runs, it will output something like this:$ tty/dev/pts/4...
, being able to log its progress to a separate file for later inspection. That way the user can start Wget and log off, leaving the program unattended. By contrast, most graphical
Graphical user interface
In computing, a graphical user interface is a type of user interface that allows users to interact with electronic devices with images rather than text commands. GUIs can be used in computers, hand-held devices such as MP3 players, portable media players or gaming devices, household appliances and...
or text user interface
Text user interface
TUI short for: Text User Interface or Textual User Interface , is a retronym that was coined sometime after the invention of graphical user interfaces, to distinguish them from text-based user interfaces...
web browser
Web browser
A web browser is a software application for retrieving, presenting, and traversing information resources on the World Wide Web. An information resource is identified by a Uniform Resource Identifier and may be a web page, image, video, or other piece of content...
s require the user to remain logged in and to manually restart failed downloads, which can be a great hindrance when transferring a lot of data.
Portability
Written in a highly portable style of CC (programming language)
C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
with minimal dependencies on third-party libraries, Wget requires little more than a C compiler and a BSD-like interface to TCP/IP
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
networking. Designed as a Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
program invoked from the Unix shell
Unix shell
A Unix shell is a command-line interpreter or shell that provides a traditional user interface for the Unix operating system and for Unix-like systems...
, the program has been ported to numerous Unix-like environments and systems, including Microsoft Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
via Cygwin
Cygwin
Cygwin is a Unix-like environment and command-line interface for Microsoft Windows. Cygwin provides native integration of Windows-based applications, data, and other system resources with applications, software tools, and data of the Unix-like environment...
, and Mac OS X. It is also available as a native Microsoft Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
program as one of the GnuWin packages.
Other features
- Wget supports download through proxiesProxy serverIn computer networks, a proxy server is a server that acts as an intermediary for requests from clients seeking resources from other servers. A client connects to the proxy server, requesting some service, such as a file, connection, web page, or other resource available from a different server...
, which are widely deployed to provide web access inside company firewalls and to cache and quickly deliver frequently accessed content. - It makes use of persistent HTTP connections where available.
- IPv6IPv6Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4...
is supported on systems that include the appropriate interfaces. - SSL/TLSTransport Layer SecurityTransport Layer Security and its predecessor, Secure Sockets Layer , are cryptographic protocols that provide communication security over the Internet...
is supported for encrypted downloads using the OpenSSLOpenSSLOpenSSL is an open source implementation of the SSL and TLS protocols. The core library implements the basic cryptographic functions and provides various utility functions...
library. - Files larger than 2 GiBGibibyteThe gibibyte is a standards-based binary multiple of the byte, a unit of digital information storage. The gibibyte unit symbol is GiB....
are supportedLarge file supportLarge file support, often abbreviated to LFS, is the term frequently applied to the ability to create files larger than 2 GiB on 32-bit operating systems.- Rationale :...
on 32-bit systems that include the appropriate interfaces. - Download speed may be throttledBandwidth throttlingBandwidth throttling is a reactive measure employed in communication networks to regulate network traffic and minimize bandwidth congestion. Bandwidth throttling can occur at different locations on the network. On a local area network , a sysadmin may employ bandwidth throttling to help limit...
to avoid using up all of the available bandwidthBandwidth (computing)In computer networking and computer science, bandwidth, network bandwidth, data bandwidth, or digital bandwidth is a measure of available or consumed data communication resources expressed in bits/second or multiples of it .Note that in textbooks on wireless communications, modem data transmission,...
.
Basic usage
Typical usage of GNU Wget consists of invoking it from the command line, providing one or more URLs as arguments.
- Download the title page of example.com to a file
- named "index.html".
wget http://www.example.com/
- Download Wget's source code from the GNU ftp site.
wget ftp://ftp.gnu.org/pub/gnu/wget/wget-latest.tar.gz
More complex usage includes automatic download of multiple URLs into a directory hierarchy.
- Download *.gif from a website
- (globbing, like "wget http://www.server.com/dir/*.gif", only works with ftp)
wget -e robots=off -r -l1 --no-parent -A.gif ftp://www.example.com/dir/
- Download the title page of example.com, along with
- the images and style sheets needed to display the page, and convert the
- URLs inside it to refer to locally available content.
wget -p -k http://www.example.com/
- Download the entire contents of example.com
wget -r -l 0 http://www.example.com/
Advanced examples
Download a mirror of the errata for a book you just purchased, follow all local links recursively and make the files suitable for off-line viewing. Use a random wait of up to 5 seconds between each file download and log the access results to "myLog.log". When there is a failure, retry for up to 7 times with 14 seconds between each retry. (The command must be on one line.)
wget -t 7 -w 5 --waitretry=14 --random-wait -m -k -K -e robots=off
http://www.oreilly.com/catalog/upt3/errata/ -o ./myLog.log
Collect only specific links listed line by line in the local file "my_movies.txt". Use a random wait of 0 to 33 seconds between files, and use 512 kilobytes per second of bandwidth throttling
Bandwidth throttling
Bandwidth throttling is a reactive measure employed in communication networks to regulate network traffic and minimize bandwidth congestion. Bandwidth throttling can occur at different locations on the network. On a local area network , a sysadmin may employ bandwidth throttling to help limit...
. When there is a failure, retry for up to 22 times with 48 seconds between each retry. Send no tracking user agent
User agent
In computing, a user agent is a client application implementing a network protocol used in communications within a client–server distributed computing system...
or HTTP referer
HTTP referer
The referrer, or HTTP referrer — also known by the common misspelling referer that occurs as an HTTP header field — identifies, from the point of view of an Internet webpage or resource, the address of the webpage The referrer, or HTTP referrer — also known by the common misspelling...
to a restrictive site and ignore robot exclusions. Place all the captured files in the local "movies" directory and collect the access results to the local file "my_movies.log". Good for downloading specific sets of files without hogging the network:-
Instead of an empty referer
HTTP referer
The referrer, or HTTP referrer — also known by the common misspelling referer that occurs as an HTTP header field — identifies, from the point of view of an Internet webpage or resource, the address of the webpage The referrer, or HTTP referrer — also known by the common misspelling...
and user-agent
User agent
In computing, a user agent is a client application implementing a network protocol used in communications within a client–server distributed computing system...
use a real one that does not cause an “ERROR: 403 Forbidden” message from a restrictive site. It is also possible to create a .wgetrc file that holds some default values.
wget -t 22 --waitretry=48 --wait=33 --random-wait --referer="" --user-agent=""
--limit-rate=512k -e robots=off -o ./my_movies.log -P./movies -i ./my_movies.txt
To get around cookie tracked sessions:-
- Using wget to download content protected by referer and cookies.
- 1. get base url and save its cookies in file
- 2. get protected content using stored cookies
wget --cookies=on --keep-session-cookies --save-cookies=cookie.txt http://first_page
wget --referer=http://first_page --cookies=on --load-cookies=cookie.txt
--keep-session-cookies --save-cookies=cookie.txt http://second_page
Mirror and convert CGI, ASP or PHP and others to HTML for offline browsing:-
- Mirror website to a static copy for local browsing.
- This means all links will be changed to point to the local files.
- Note --html-extension will convert any CGI, ASP or PHP generated files to HTML (or anything else not .html).
wget --mirror -w 2 -p --html-extension --convert-links -Phttp://www.yourdomain.com
Authors and copyright
GNU Wget was written by Hrvoje Nikšić with contributions by many other people, including Dan Harkless, Ian Abbott, and Mauro Tortonesi. Significant contributions are credited in the AUTHORS file included in the distribution, and all remaining ones are documented in the changelogChangelog
A changelog is a log or record of changes made to a project, such as a website or software project, usually including such records as bug fixes, new features, etc...
s, also included with the program. Wget is currently maintained by Giuseppe Scrivano.
The copyright to Wget belongs to the Free Software Foundation
Free Software Foundation
The Free Software Foundation is a non-profit corporation founded by Richard Stallman on 4 October 1985 to support the free software movement, a copyleft-based movement which aims to promote the universal freedom to create, distribute and modify computer software...
, whose policy is to require copyright assignments for all non-trivial contributions to GNU software.
License
GNU Wget is distributed under the terms of the GNU General Public LicenseGNU General Public License
The GNU General Public License is the most widely used free software license, originally written by Richard Stallman for the GNU Project....
, version 3 or later, with a special exception that allows distribution of binaries linked against the OpenSSL
OpenSSL
OpenSSL is an open source implementation of the SSL and TLS protocols. The core library implements the basic cryptographic functions and provides various utility functions...
library. The text of the exception follows:
Additional permission under GNU GPL version 3 section 7
If you modify this program, or any covered work, by linking or
combining it with the OpenSSL project's OpenSSL library (or a
modified version of that library), containing parts covered by the
terms of the OpenSSL or SSLeay licenses, the Free Software Foundation
grants you additional permission to convey the resulting work.
Corresponding Source for a non-source form of such a combination
shall include the source code for the parts of OpenSSL used as well
as that of the covered work.
It is expected that the exception clause will be removed once Wget is modified to also link with the GnuTLS
GnuTLS
GnuTLS , the GNU Transport Layer Security Library, is a free software implementation of the SSL and TLS protocols. Its purpose is to offer an application programming interface for applications to enable secure communication protocols over their network transport layer.-Features:GnuTLS consists of...
library.
Wget's documentation
Documentation
Documentation is a term used in several different ways. Generally, documentation refers to the process of providing evidence.Modules of Documentation are Helpful...
, in the form of a Texinfo
Texinfo
Texinfo is a typesetting syntax used for generating documentation in both on-line and printed form with a single source file...
reference manual, is distributed under the terms of the GNU Free Documentation License
GNU Free Documentation License
The GNU Free Documentation License is a copyleft license for free documentation, designed by the Free Software Foundation for the GNU Project. It is similar to the GNU General Public License, giving readers the rights to copy, redistribute, and modify a work and requires all copies and...
, version 1.2 or later. The man page usually distributed on Unix-like systems is automatically generated from a subset of the Texinfo manual and falls under the terms of the same license.
Development
Wget is developed in an open fashion, most of the design decisions typically being discussed on the public mailing list followed by users and developers. Bug reports and patches are relayed to the same list.Source contribution
The preferred method of contributing to Wget's code and documentation is through source updates in the form of textual patchesPatch (Unix)
patch is a Unix program that updates text files according to instructions contained in a separate file, called a patch file. The patch file is a text file that consists of a list of differences and is produced by running the related diff program with the original and updated file as arguments...
generated by the diff
Diff
In computing, diff is a file comparison utility that outputs the differences between two files. It is typically used to show the changes between one version of a file and a former version of the same file. Diff displays the changes made per line for text files. Modern implementations also...
utility. Patches intended for inclusion in Wget are submitted to the mailing list where they are reviewed by the maintainers. Patches that pass the maintainers' scrutiny are installed in the sources. Instructions on patch creation as well as style guidelines are outlined on the project's wiki.
The source code can also be tracked via a remote version control repository that hosts revision history beginning with the 1.5.3 release. The repository is running Bazaar. Previously, Mercurial
Mercurial (software)
Mercurial is a cross-platform, distributed revision control tool for software developers. It is mainly implemented using the Python programming language, but includes a binary diff implementation written in C. It is supported on Windows and Unix-like systems, such as FreeBSD, Mac OS X and Linux...
was used. Before that, it had been hosted on Subversion, and prior to that via CVS
Concurrent Versions System
The Concurrent Versions System , also known as the Concurrent Versioning System, is a client-server free software revision control system in the field of software development. Version control system software keeps track of all work and all changes in a set of files, and allows several developers ...
.
Release
When a sufficient number of features or bug fixes accumulate during development, Wget is released to the general public via the GNU FTP site and its mirrors. Being entirely run by volunteers, there is no external pressure to issue a release nor are there enforceable release deadlines.Releases are numbered as versions of the form of major.minor[.revision], such as Wget 1.11 or Wget 1.8.2. An increase of the major version number represents large and possibly incompatible changes in Wget's behavior or a radical redesign of the code base. An increase of the minor version number designates addition of new features and bug fixes. A new revision indicates a release that, compared to the previous revision, only contains bug fixes. Revision zero is omitted, meaning that for example Wget 1.11 is the same as 1.11.0. Wget does not use the odd-even release number convention popularized by the Linux.
Notable releases
The following releases represent notable milestones in Wget's development. Features listed next to each release are edited for brevity and do not constitute comprehensive information about the release, which is available in the NEWS file distributed with Wget.- Geturl 1.0, released January 1996, was the first publicly available release. The first English-language announcement can be traced to this Usenet news posting, which probably refers to Geturl 1.3.4 released in June.
- Wget 1.4.0, released November 1996, was the first version to use the name Wget. It was also the first release distributed under the terms of the GNU GPL, Geturl having been distributed under an ad-hoc no-warrantyWarrantyIn business and legal transactions, a warranty is an assurance by one party to the other party that specific facts or conditions are true or will happen; the other party is permitted to rely on that assurance and seek some type of remedy if it is not true or followed.In real estate transactions, a...
license. - Wget 1.4.3, released February 1997, was the first version released as part of the GNU projectGNUGNU is a Unix-like computer operating system developed by the GNU project, ultimately aiming to be a "complete Unix-compatible software system"...
with the copyright assigned to the FSFFree Software FoundationThe Free Software Foundation is a non-profit corporation founded by Richard Stallman on 4 October 1985 to support the free software movement, a copyleft-based movement which aims to promote the universal freedom to create, distribute and modify computer software...
. - Wget 1.5.3, released September 1998, was a milestone in the program's popularity. This version was bundled with many GNU/Linux based distributions, which exposed the program to a much wider audience.
- Wget 1.6, released December 1999, incorporated many bug fixes for the (by then stale) 1.5.3 release, largely thanks to the effort of Dan Harkless.
- Wget 1.7, released June 2001, introduced SSLTransport Layer SecurityTransport Layer Security and its predecessor, Secure Sockets Layer , are cryptographic protocols that provide communication security over the Internet...
support, cookiesHTTP cookieA cookie, also known as an HTTP cookie, web cookie, or browser cookie, is used for an origin website to send state information to a user's browser and for the browser to return the state information to the origin site...
, and persistent connections. - Wget 1.8, released December 2001, added bandwidth throttlingBandwidth throttlingBandwidth throttling is a reactive measure employed in communication networks to regulate network traffic and minimize bandwidth congestion. Bandwidth throttling can occur at different locations on the network. On a local area network , a sysadmin may employ bandwidth throttling to help limit...
, new progress indicators, and the breadth-first traversal of the hyperlink graphGraph (mathematics)In mathematics, a graph is an abstract representation of a set of objects where some pairs of the objects are connected by links. The interconnected objects are represented by mathematical abstractions called vertices, and the links that connect some pairs of vertices are called edges...
. - Wget 1.9, released October 2003, included experimental IPv6IPv6Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4...
support, and ability to POST data to HTTP servers. - Wget 1.10, released June 2005, introduced large file supportLarge file supportLarge file support, often abbreviated to LFS, is the term frequently applied to the ability to create files larger than 2 GiB on 32-bit operating systems.- Rationale :...
, IPv6IPv6Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4...
support on dual-family systems, NTLMNTLMIn a Windows network, NTLM is a suite of Microsoft security protocols that provides authentication, integrity, and confidentiality to users....
authorization, and SSLTransport Layer SecurityTransport Layer Security and its predecessor, Secure Sockets Layer , are cryptographic protocols that provide communication security over the Internet...
improvements. The maintainership was picked up by Mauro Tortonesi. - Wget 1.11, released January 2008, moved to version 3 of the GNU General Public LicenseGNU General Public LicenseThe GNU General Public License is the most widely used free software license, originally written by Richard Stallman for the GNU Project....
, and added preliminary support for theContent-Dispositionheader, which is often used by CGICommon Gateway InterfaceThe Common Gateway Interface is a standard method for web servers software to delegate the generation of web pages to executable files...
scripts to indicate the name of a file for downloading. Security-related improvements were also made to the HTTP authentication code. Micah Cowan took over maintainership of the project. - Wget 1.12, released September 2009, added support for parsing URLs from CSSCascading Style SheetsCascading Style Sheets is a style sheet language used to describe the presentation semantics of a document written in a markup language...
content on the web, and for handling Internationalized Resource IdentifierInternationalized Resource IdentifierOn the Internet, the Internationalized Resource Identifier is a generalization of the Uniform Resource Identifier . While URIs are limited to a subset of the ASCII character set, IRIs may contain characters from the Universal Character Set , including Chinese or Japanese kanji, Korean, Cyrillic...
s. - Wget 1.13, released August 2011, support HTTP/1.1, fix some portability issues, use GnuTLSGnuTLSGnuTLS , the GNU Transport Layer Security Library, is a free software implementation of the SSL and TLS protocols. Its purpose is to offer an application programming interface for applications to enable secure communication protocols over their network transport layer.-Features:GnuTLS consists of...
library by default for secure connections.
Related works
GWget
GWget is a free graphicalGraphical user interface
In computing, a graphical user interface is a type of user interface that allows users to interact with electronic devices with images rather than text commands. GUIs can be used in computers, hand-held devices such as MP3 players, portable media players or gaming devices, household appliances and...
frontend for of Wget. It is developed by David Sedeño Fernández and is part of the GNOME
GNOME
GNOME is a desktop environment and graphical user interface that runs on top of a computer operating system. It is composed entirely of free and open source software...
project. GWget supports all of the main features that Wget does, as well as parallel downloads.
KGet
KGet is yet another free frontend for KDEKDE
KDE is an international free software community producing an integrated set of cross-platform applications designed to run on Linux, FreeBSD, Microsoft Windows, Solaris and Mac OS X systems...
. It is the default download manager of Konqueror
Konqueror
Not to be confused with the Conqueror web browser.Konqueror is a web browser and file manager that provides file-viewer functionality for file systems such as local files, files on a remote ftp server and files in a disk image. It is a core part of the KDE desktop environment...
, but can also be used with Mozilla Firefox
Mozilla Firefox
Mozilla Firefox is a free and open source web browser descended from the Mozilla Application Suite and managed by Mozilla Corporation. , Firefox is the second most widely used browser, with approximately 25% of worldwide usage share of web browsers...
. KGet is part of the KDE Network package.
See also
- The GNU ProjectGNUGNU is a Unix-like computer operating system developed by the GNU project, ultimately aiming to be a "complete Unix-compatible software system"...
- cURLCURLcURL is a computer software project providing a library and command-line tool for transferring data using various protocols. The cURL project produces two products, libcurl and cURL...
- NcFTPNcFTPNcFTP is an FTP client program which debuted in 1990 as the first alternative FTP client. It was created as an alternative to the standard UNIX ftp program, and offers a number of additional features and greater ease of use....
- Web crawlerWeb crawlerA Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or—especially in the FOAF community—Web scutters.This process is called Web...
- HTTrackHTTrackHTTrack is a free and open source Web crawler and offline browser, developed by Xavier Roche and licensed under the GNU General Public License....
- lftpLftplftp is a command-line file transfer program for UNIX and Unix-like systems. It was written by Alexander Lukyanov, and is made available under the GNU General Public License....