|
|
|
|
Freenet
|
| |
|
| |
Freenet is a decentralized, censorship-resistant distributed data store originally designed by Ian Clarke. Freenet aims to provide freedom of speech through a peer-to-peer network with strong protection of anonymity. Freenet works by pooling the contributed bandwidth and storage space of member computers to allow users to anonymously publish or retrieve various kinds of information. It can be thought of as a large storage device which uses key based routing similar to a distributed hash table to locate peers' data.
Discussion
Ask a question about 'Freenet' Start a new discussion about 'Freenet' Answer questions from other users |
Encyclopedia
Freenet is a decentralized, censorship-resistant distributed data store originally designed by Ian Clarke. Freenet aims to provide freedom of speech through a peer-to-peer network with strong protection of anonymity. Freenet works by pooling the contributed bandwidth and storage space of member computers to allow users to anonymously publish or retrieve various kinds of information. It can be thought of as a large storage device which uses key based routing similar to a distributed hash table to locate peers' data. When a file is stored in Freenet, a key which can be used to retrieve the file is generated. The storage space is distributed among all connected nodes on Freenet.
Freenet has been under continuous development since 2000; a version 1.0 has not yet been released but current builds are functionally usable. The project has already seen a ground-up rewrite for version 0.7, however. Released under the GNU General Public License, Freenet is free software.
Content
Freenet's founders argue that only with true anonymity comes true freedom of speech, and that what they view as the beneficial uses of Freenet outweigh its negative uses. Their view is that free speech, in itself, is not in contradiction with any other consideration — the information is not the crime. Freenet attempts to remove the possibility of any group imposing their beliefs or values on any data. Although many states censor communications to different extents, they all share one commonality in that a body must decide what information to censor and what information to allow. What may be acceptable to one group of people may be considered offensive or even dangerous to another. In essence, the purpose of Freenet is that nobody is allowed to decide what is acceptable. Tolerance for each others' values is encouraged and failing that, the user is asked to turn a blind eye to content which opposes his or her views.
One analysis of Freenet files conducted in the year 2000 (before Freenet had proper support for web pages and chat) claimed that the top 3 types of files contained in Freenet were text (37%), audio (21%), and images (14%). 59% of all the text files were drug-related, 71% of all audio files were rock music, and 89% of all images were pornographic. The article qualifies itself with the proviso: "the design of Freenet makes accurate analysis of its content difficult" Due to the nature of Freenet, a typical user may unknowingly host this sort of information, which may hypothetically make them subject to severe civil and criminal penalties. Freenet attempts to prevent this through "plausible deniability", preventing the user from knowing what is on his or her own node and making it difficult to determine if a piece of information is in any given node without causing the distribution of that piece of information throughout the network to change in the process. No court cases have tested any of this to date.
Reports of Freenet's use in authoritarian nations is difficult to track due to the very nature of Freenet's goals. One group, Freenet-China, has translated the Freenet software to Chinese and is distributing it within China on CD and floppy disk.
Technical design
The Freenet file sharing network stores documents and allows them to be retrieved later by an associated key, as is now possible with protocols such as HTTP. The network is designed to be highly survivable, with all internal processes completely anonymized and decentralized across the network. The system has no central servers and is not subject to the control of any one individual or organization. Even the designers of Freenet do not have any control over the overall system. The stored information is encrypted and replicated across participating computers around the world, which are anonymized and intended to be many and continuously-changing. It is theoretically difficult for anyone to find out which participants are hosting a given file, since the contents of each file are encrypted, and may be broken into pieces that are distributed over many different computers. Due to the nature of encryption it is, by mathematical proof, highly impractical for a participant to view files hosted on his or her own system.
Distributed storage and caching of data
Unlike other p2p networks, Freenet not only transmits data between nodes but actually stores them, working as a huge distributed cache.
To achieve this, each node allocates some amount of disk space for data store and cache, typically 10 GB.
Information flow in Freenet is different from networks like eMule or BitTorrent:
- A user wishing to share a file or update a freesite "inserts" the file or HTML page "to the network"
- After "insertion" is finished, the publisher is free to shut down his node, since the file is stored in the network. It will remain available for other users whether the original publishing node is online or not.
Two advantages of this design are high reliability and anonymity. Information remains available even if the publisher node goes offline, and is anonymously spread over many hosting nodes as encrypted blocks, not entire files. Freenet is also not affected by the typical BitTorrent problem, a lack of "seeds", or full copies of a file or torrent.
A trade-off is that a node operator has no control over what kind of content is stored on his node.
Network
The network consists of a number of nodes that pass messages among themselves. Typically, a host computer on the network runs the software that acts as a node, and it connects to other hosts running that same software to form a large distributed network of peer nodes. Some nodes are end user nodes, from which documents are requested and presented to human users. Other nodes serve only to route data. All nodes communicate with each other identically — there are no dedicated "clients" or "servers". It is not possible for a node to rate another node except by its capacity to insert and fetch data associated with a key. This is unlike most other P2P networks where node administrators can employ a ratio system, where users have to share a certain amount of content before they can download.
Freenet may also be considered a small world network.
The Freenet protocol is intended to be used on a network of complex topology, such as the Internet (Internet Protocol). Each node knows only about some number of other nodes that it can reach directly (its conceptual "neighbors"), but any node can be a neighbor to any other; no hierarchy or other structure is intended. Each message is routed through the network by passing from neighbor to neighbor until it reaches its destination. As each node passes a message to a neighbor, it does not know or care whether the neighbor will forward the message to another node, or is the final destination or original source of the message. This is intended to protect the anonymity of users and publishers.
Each node maintains a data store containing documents associated with keys, and a routing table associating nodes with records of their performance in retrieving different keys.
Protocol
The Freenet protocol uses a key based routing protocol, similar to distributed hash tables. The routing algorithm changed significantly in version 0.7. Prior to 0.7, Freenet used a heuristic routing algorithm where each node had no fixed location, and routing was based on which node had served a key closest to the key being fetched (in ~ 0.3) or which we estimate would serve it faster (in 0.5); either way, new connections were (sometimes) added to downstream nodes (e.g. the node that answered the request) when requests succeeded, and old nodes were thrown out in least recently used order (or something close to it). Oskar Sandberg's research (during the development of 0.7) shows that this "path folding" is critical, and that a very simple routing algorithm will suffice provided there is path folding.
The disadvantage of this is that it is very easy for an attacker to find Freenet nodes, and connect to them, because every node is continually attempting to find new connections. In version 0.7, Freenet supports both 'Opennet' (similar to the old algorithms, but simpler), and 'Darknet' (all node connections are setup manually, so only your friends know your node's IP address). Darknet is less convenient, but much more secure against a distant attacker.
This change required major changes in the routing algorithm. Every node has a location, which is a number between 0 and 1. When a key is requested, first the node checks the local data store. If it's not found, the key's hash is turned into another number in the same range, and the request is routed to the node whose location is closest to the key. This goes on until some number of hops is exceeded, there are no more nodes to search, or the data is found. If the data is found, it is cached on each node along the path. So there is no one source node for a key, and attempting to find where it is currently stored will result in it being cached more widely. Essentially the same process is used to insert a document into the network: the data is routed according to the key until it runs out of hops, and if no existing document is found with the same key, it is stored on each node. If older data is found, the older data is propagated and returned to the originator, and the insert "collides".
But this only works if the locations are clustered in the right way. Freenet assumes that the Darknet (a subset of the global social network) is a small-world network, and nodes constantly attempt to swap locations (using the Metropolis–Hastings algorithm) in order to minimize their distance to their neighbors. If the network actually is a small-world network, Freenet should find data reasonably quickly; it's to be hoped on the order of hops. However, it does not guarantee that data will be found at all.
Eventually, either the document is found or the hop limit is exceeded. The terminal node sends a reply that makes its way back to the originator along the route specified by the intermediate nodes' records of pending requests. The intermediate nodes may choose to cache the document along the way. Besides saving bandwidth, this also makes documents harder to censor as there is no one "source node."
Effect
Initially, the locations are distributed randomly (whether on Opennet or Darknet). This means that routing of requests is essentially random. But since different nodes have different randomness, they will disagree about where to send a request, given a key. So the data in a newly-started Freenet will be distributed somewhat randomly.
As location swapping (on Darknet) and path folding (on Opennet) progress, nodes which are close to one another will increasingly have close locations, and nodes which are far away will have distant locations. Data with similar keys will be stored on the same node.
The result is that the network will self-organize into a distributed, clustered structure where nodes tend to hold data items that are close together in key space. There will probably be multiple such clusters throughout the network, any given document being replicated numerous times, depending on how much it is used. This is a kind of "spontaneous symmetry breaking", in which an initially symmetric state (all nodes being the same, with random initial keys for each other) leads to a highly asymmetric situation, with nodes coming to specialize in data that has closely related keys.
There are forces which tend to cause clustering (shared closeness data spreads throughout the network), and forces that tend to break up clusters (local caching of commonly used data). These forces will be different depending on how often data is used, so that seldom-used data will tend to be on just a few nodes which specialize in providing that data, and frequently used items will be spread widely throughout the network. This automatic mirroring counteracts the times when web traffic becomes overloaded, and due to a mature network's intelligent routing, a network of size n should only require log(n) time to retrieve a document on average.
Keys
Keys are hashes: there is no notion of semantic closeness when speaking of key closeness. Therefore there will be no correlation between key closeness and similar popularity of data as there might be if keys did exhibit some semantic meaning, thus avoiding bottlenecks caused by popular subjects.
There are two main varieties of keys in use on Freenet, the (CHK) and the (SSK).
A CHK is a SHA-256 hash of a document (after encryption, which itself depends on the hash of the plaintext) and thus a node can check that the document returned is correct by hashing it and checking the digest against the key. This key contains the meat of the data on Freenet. It carries all the binary data building blocks for the content to be delivered to the client for reassembly and decryption. The CHK is unique by nature and provides tamperproof content. A hostile node altering the data under a CHK will immediately be detected by the next node or the client. CHKs also reduce the redundancy of data since the same data will have the same CHK.
SSKs are based on public-key cryptography. Currently Freenet uses the DSA algorithm. Documents inserted under SSKs are signed by the inserter, and this signature can be verified by every node to ensure that the data is not tampered with. SSKs can be used to establish a verifiable pseudonymous identity on Freenet, and allow for documents to be updated securely by the person who inserted them. A subtype of the SSK is the Keyword Signed Key, or KSK, in which the key pair is generated in a standard way from a simple human-readable string. Inserting a document using a KSK allows the document to be retrieved and decrypted if and only if the requester knows the human-readable string; this allows for more convenient (but less secure) URIs for users to refer to.
Scalability
A network is said to be scalable if its performance does not deteriorate even if the network is very large. The scalability of Freenet is being evaluated, but similar architectures have been shown to scale logarithmically. It is believed based on this work that Freenet should find data in around O(log^2 n) hops on Opennet or on a Darknet which is a small-world network. However, this is only true on a mature network, and the level of churn typical on Freenet, due to nodes not running continually, and due to people trying it out and then leaving, may prevent this.
Darknet versus Opennet: goals and controversy
Freenet versions prior to 0.7 were based on opennet topology. Freenet 0.7 introduced support for a scalable darknet, where users only connect directly to other users they know and trust. The purpose of this change is to protect users who may be placed at risk simply by using the software, irrespective of what they are using it for. In the new model, users will choose to whom they connect, and only those users will know that they are running the software. Previous darknets, such as WASTE, have been limited to relatively small disconnected networks. The core innovation in Freenet 0.7 is to allow a globally scalable darknet, capable (at least in theory) of supporting millions of users. This is made possible by the fact that human relationships tend to form small-world networks, a property that can be exploited to find short paths between any two people. The work is based on a speech given at DEF CON 13 by Ian Clarke and Swedish mathematician Oskar Sandberg.
For much of the development process of Freenet 0.7, there was no Opennet mode, so that users would have to find Darknet connections. This was partly because it simply wasn't implemented, but partly because of developers' hopes that a true F2F network would emerge.
However, this did not work out, because in practice most users didn't know anyone else using Freenet, so had to use an IRC channel or a Frost board to find total strangers to connect to. This didn't work well because it produced a very poor network topology, resulting in bad performance, and significantly increased the effort needed to get a new node working. Even after getting some noderefs, some of the people connected to would inevitably leave Freenet, so it was necessary to come back for more.
The implementation of Opennet in Freenet 0.7, in late 2007, therefore greatly improved both performance and usability. However, in hostile environments where Opennet may be attacked, it will be necessary to use pure Darknet, so the official advice is to start off with Opennet, and add Friend-to-Friend connections as quickly as possible, eventually turning off opennet mode and becoming invisible.
The Freenet wiki contains some in-depth discussion of possible attacks on opennet.
Current development
Freenet 0.7, released on the 8th of May 2008, is a major re-write incorporating a number of fundamental changes. The most fundamental change is support for darknet operation, described above. Other modifications include switching from TCP to UDP, which allows UDP hole punching along with faster transmission of messages between peers in the network.
It is planned that future versions will support "passive requests" (a kind of server push), broadcast streams, and anonymous "channels" to a particular node allowing for dynamic content. Applications of this range from Internet Relay Chat to RSS-feeds. Future versions will also feature increased security via amongst other things onion routing. Freenet 0.7 continues to only support publishing and fetching data.
A large number of Freenet users continue to use Freenet 0.5 for various reasons, a common one being distrust of the darknet concept. User produced builds of Freenet are being released within the 0.5 community, the latest being Freenet 0519. While there has been talk of creating a separate fork based on the 0.5 source, no real steps have been taken.
Freenet is fundamentally different from other peer-to-peer networks; it is still somewhat more difficult to use and significantly slower. However, after continued use and activity on the network, nodes become faster and more efficient at fetching data.
Currently, Freenet cannot be used to create or distribute dynamic content, such as content that utilizes databases and scripting. According to the Freenet Project group, such tradeoffs are expected since Freenet's primary goals are neither ease-of-use nor performance. Although there are plans to support this in the future, it is considered a low priority. Unlike other peer-to-peer networks, Freenet is primarily intended to combat censorship and allow people to communicate freely and with near-total anonymity.
There is a primitive (not fully distributed) keyword search function in development, but due to the small size of the network several freesites serve as directories listing published freesites. Upon the creation of a new freesite, the author can add a listing to the directory allowing others to discover the freesite. The directory owners also periodically spider or automatically attempt to retrieve the freesites they list. One of the most famous directories is the Freedom Engine.
Related tools and Freenet applications
Unlike many other P2P applications, Freenet does not have a single application which provides all functionality. Instead, Freenet has a modular structure: the core application focuses on connecting to the network, and acts as a proxy, providing an open application interface named FCP for other programs to use. Those additional applications use the API to implement services like message boards, file sharing, or online chat. Several applications are distributed with Freenet and installed by the installation wizard together with the Freenet core.
Distributed forums in Freenet
Message boards (forums) are particularly popular among Freenet users: slow speed is not a problem, content is generated by users themselves so there's no lack of content, and the wish to stay anonymous is natural among forum users.
Frost is a well-maintained and popular message board system for Freenet. It uses a web of trust concept as well as public and private keys to prevent excess spam. It is very popular in the area of file sharing and is the most widely used Freenet messaging application. Frost is written in Java. Frost is not currently bundled with Freenet and can be downloaded from on Sourceforge. However, because of sustained denial of service attacks since late 2007 (flooding board queues with bogus messages, which are invisible to users but make it almost impossible to find or post real messages), most Frost boards (at least most of the ones included by default) are now inactive.
The Freenet Message Board (FMB) is a message board system written in Java, and the original author does not maintain it. However, the community continues to provide several branches which are generally up to date.
Freenet Messaging System (FMS) is a new application brought to solve limitations and issues with current messaging protocol of Frost (in particular the denial of service attacks mentioned above). It uses a true web of trust with outbox polling (which may or may not scale, although there have been some improvements to Freenet itself to improve this). It can be downloaded from freesites in Freenet proper. FMS is much less user friendly than Frost, in that you need FMS, a news reader, and a web browser, to use it. Eventually Frost will use a similar mechanism, but this is currently waiting for a Java port of FMS to be completed.
Anonymous wikis
There are two projects that implement anonymous wiki in Freenet:
and . Both projects are under development now.
FreemulET is a file sharing application in Freenet, with look and feel similar to a widespread eMule software. FreemulET provides on-demand file sharing and can reinsert only blocks that are actually missing. FreemulET is not bundled but can be downloaded from the FreemulET freesite.
Thaw is a file sharing application included with Freenet, which is primarily a download manager and tool for browsing and creating file indexes. File indexes can link to one another and thus form a kind of web of download channels.
Freesite tools
Freenet 0.7 includes a simple tool to upload freesites called jSite, and a blogging engine (capable of publishing either to Freenet or to a web site) based on a fork of Thingamablog.
Freenet Utility for Queued Inserts and Downloads (FUQID) is a Microsoft Windows tool typically used to retrieve large splitfiles and to insert non-Freesite content such as binaries, audio, and archives. It is written in Borland Delphi, and it is maintained sporadically.
Command line tools are also available. Older tools include the Freesite Insertion Wizard (FIW), which hasn't been ported to 0.7, and the unmaintained Fishtools.
Development libraries
FCPLib (Freenet Client Protocol Library) aims to be a cross-platform but natively compiled set of C-based functions for storing and retrieving information to and from Freenet. There are routines for storing documents to Freenet from the local disk, and other routines for moving data in memory to and from Freenet. FCPLib is now routinely compiled on the following platforms: Microsoft Windows NT/2K/XP, Debian GNU/Linux, BSD, Solaris, and Mac OS X. The FCPTools are command-line driven programs for inserting and retrieving files with Freenet. They are linked against FCPLib and serve as examples of library use.
The Freenet Tools perform roughly the same tasks as FCPTools, however it does not include a client library for use in other projects. The Freenet Tools are written in ANSI C, and runs on Unix-like OSs.
Publicity
According to CiteSeer, Ian Clarke's "Freenet: A Distributed Anonymous Information Storage and Retrieval System" was the most cited computer science paper of 2000. Freenet has also had significant publicity in the mainstream press, including articles in the New York Times, and coverage on CNN, 60 Minutes II, the BBC, and elsewhere. The mainstream press coverage has been primarily concerned with Freenet's impact on copyright enforcement, rather than Freenet's core goal of freedom of communication.
See also
External links
-
-
- An explanation of the Freenet architecture and implementation (as of December 30, 2005) given by Ian Clarke and Oskar Sandberg
-
-
|
| |
|
|
