Letter frequencies
Encyclopedia
The frequency of letters in text has often been studied for use in cryptography
Cryptography
Cryptography is the practice and study of techniques for secure communication in the presence of third parties...

, and frequency analysis
Frequency analysis
In cryptanalysis, frequency analysis is the study of the frequency of letters or groups of letters in a ciphertext. The method is used as an aid to breaking classical ciphers....

 in particular. No exact letter frequency distribution underlies a given language, since all writers write slightly differently. Linotype machine
Linotype machine
The Linotype typesetting machine is a "line casting" machine used in printing. The name of the machine comes from the fact that it produces an entire line of metal type at once, hence a line-o'-type, a significant improvement over manual typesetting....

s sorted the letters' frequencies as etaoin shrdlu
ETAOIN SHRDLU
ETAOIN SHRDLU is a nonsense phrase that sometimes appeared in print in the days of "hot type" publishing because of a custom of Linotype machine operators. It appeared frequently enough that it became part of the lore of newspapers...

 cmfwyp vbgkqj xz
based on the experience and custom of manual compositors. Likewise, Modern International Morse code
Morse code
Morse code is a method of transmitting textual information as a series of on-off tones, lights, or clicks that can be directly understood by a skilled listener or observer without special equipment...

 encodes the most frequent letters with the shortest symbols; arranging the Morse alphabet into groups of letters that require equal amounts of time to transmit, and then sorting these groups in increasing order, yields e it san hurdm wgvlfbk opjxcz yq. Similar ideas are used in modern data-compression techniques such as Huffman coding
Huffman coding
In computer science and information theory, Huffman coding is an entropy encoding algorithm used for lossless data compression. The term refers to the use of a variable-length code table for encoding a source symbol where the variable-length code table has been derived in a particular way based on...

.

More recent analyses show that letter frequencies, like word frequencies, tend to vary, both by writer and by subject. One cannot write an essay about x-rays without using frequent Xs, and the essay will have an especially strange letter frequency if the essay is about the frequent use of x-rays to treat zebra
Zebra
Zebras are several species of African equids united by their distinctive black and white stripes. Their stripes come in different patterns unique to each individual. They are generally social animals that live in small harems to large herds...

s in Qatar
Qatar
Qatar , also known as the State of Qatar or locally Dawlat Qaṭar, is a sovereign Arab state, located in the Middle East, occupying the small Qatar Peninsula on the northeasterly coast of the much larger Arabian Peninsula. Its sole land border is with Saudi Arabia to the south, with the rest of its...

. Different authors have habits which can be reflected in their use of letters. Hemingway
Ernest Hemingway
Ernest Miller Hemingway was an American author and journalist. His economic and understated style had a strong influence on 20th-century fiction, while his life of adventure and his public image influenced later generations. Hemingway produced most of his work between the mid-1920s and the...

's writing style, for example, is visibly different from Faulkner
William Faulkner
William Cuthbert Faulkner was an American writer from Oxford, Mississippi. Faulkner worked in a variety of media; he wrote novels, short stories, a play, poetry, essays and screenplays during his career...

's. Letter, bigram
Bigram
Bigrams or digrams are groups of two written letters, two syllables, or two words, and are very commonly used as the basis for simple statistical analysis of text. They are used in one of the most successful language models for speech recognition...

, trigram
Trigram
Trigrams are a special case of the N-gram, where N is 3. They are often used in natural language processing for doing statistical analysis of texts.-Frequency:The 16 most common trigrams in English are:-Examples:...

, word frequencies, word length, and sentence length can be calculated for specific authors, and used to prove or disprove authorship of texts, even for authors whose styles aren't so divergent.

Accurate average letter frequencies can only be gleaned by analyzing a large amount of representative text. With the availability of modern computing and collections of large text corpora
Corpus linguistics
Corpus linguistics is the study of language as expressed in samples or "real world" text. This method represents a digestive approach to deriving a set of abstract rules by which a natural language is governed or else relates to another language. Originally done by hand, corpora are now largely...

, such calculations are easily made. This Deafandblind link details examples from a variety of sources, (press reporting, religious text, scientific text and general fiction) and there are differences especially for general fiction with the position of 'h' and 'i'. The example differs from the linotype 'etaoin shrdlu' to come out as 'etaoHn Isrdlu'. There is an unproven statement that conversation is similar in frequency to general fiction.

Herbert S. Zim
Herbert Zim
Herbert Spencer Zim was a naturalist, author, editor and educator best known as the founder and editor in chief of the Golden Guides series of nature books.-Biography:...

, in his classic introductory cryptography text
"Codes and Secret Writing", gives the English letter frequency
sequence as "ETAON RISHD LFCMU GYPWB VKXJQ Z", the most common letter
pairs as "TH HE AN RE ER IN ON AT ND ST ES EN OF TE ED OR TI HI
AS TO", and the most common doubled letters as "LL EE SS OO TT
FF RR NN PP CC".

The 'top twelve' letters comprise about 80% of the total usage.
The 'top eight" letters comprise about 65% of the total usage.
A spy using the VIC cipher
VIC cipher
The VIC cipher was a pencil and paper cipher used by the Soviet spy Reino Häyhänen, codenamed "VICTOR".It was arguably the most complex hand-operated cipher ever seen, when it was first discovered...

 or some other cipher based on a straddling checkerboard typically uses a mnemonic such as "a sin to err" (dropping the second "r") to remember the top 8 characters.

The use of letter frequencies and frequency analysis
Frequency analysis
In cryptanalysis, frequency analysis is the study of the frequency of letters or groups of letters in a ciphertext. The method is used as an aid to breaking classical ciphers....

 plays a fundamental role in cryptograms and several word puzzle games, including Hangman
Hangman (game)
Hangman is a paper and pencil guessing game for two or more players. One player thinks of a word and the other tries to guess it by suggesting letters.-Overview:...

, Scrabble
Scrabble
Scrabble is a word game in which two to four players score points by forming words from individual lettered tiles on a game board marked with a 15-by-15 grid. The words are formed across and down in crossword fashion and must appear in a standard dictionary. Official reference works provide a list...

, Bananagrams
Bananagrams
Bananagrams is a word game invented by Abraham Nathanson of Narragansett, Rhode Island, wherein lettered tiles are used to spell words.Nathanson conceived and developed the idea for the game with the help of his family...

, and the television game show Wheel of Fortune
Wheel of Fortune (U.S. game show)
Wheel of Fortune is an American television game show created by Merv Griffin, which premiered in 1975. Contestants compete to solve word puzzles, similar to those used in Hangman, to win cash and prizes determined by spinning a large wheel. The title refers to the show's giant carnival wheel that...

.

Letter frequencies had a strong effect on the design of some keyboard layout
Keyboard layout
A keyboard layout is any specific mechanical, visual, or functional arrangement of the keys, legends, or key–meaning associations of a computer, typewriter, or other typographic keyboard....

s.
The most-frequent letters are on the bottom row of the Blickensderfer typewriter
Blickensderfer typewriter
The Blickensderfer Typewriter was designed by George C Blickensderfer in 1893. It was originally intended to compete with Remington desk typewriters, but ended up being known for its portability. Blickensderfer's typewriter contained only 250 parts compared to the 2,500 parts of a standard...

.
The most-frequent letters are on the home row of the Dvorak Simplified Keyboard
Dvorak Simplified Keyboard
The Dvorak Simplified Keyboard is a keyboard layout patented in 1936 by Dr. August Dvorak and his brother-in-law, Dr. William Dealey. Over the years several slight variations were designed by the team led by Dvorak or by ANSI...

.

Relative frequencies of letters in the English language

The letter frequencies for English are listed below. However, this table differs slightly from others, such as Cornell University Math Explorer's Project, which produced this table after measuring 40,000 words.
Letter Frequency
a 8.167%
b 1.492%
c 2.782%
d 4.253%
e 12.702%
f 2.228%
g 2.015%
h 6.094%
i 6.966%
j 0.153%
k 0.772%
l 4.025%
m 2.406%
n 6.749%
o 7.507%
p 1.929%
q 0.095%
r 5.987%
s 6.327%
t 9.056%
u 2.758%
v 0.978%
w 2.360%
x 0.150%
y 1.974%
z 0.074%

In English, the space is slightly more frequent than the top letter (7% more frequent than, or 107% as frequent as, e), and the non-alphabetic characters (digits, punctuation, etc.) occupy the fourth position, between t and a.

Relative frequencies of the first letters of a word in the English language

First Letter of a word frequencies:
Letter Frequency
a 11.602%
b 4.702%
c 3.511%
d 2.670%
e 2.000%
f 3.779%
g 1.950%
h 7.232%
i 6.286%
j 0.631%
k 0.690%
l 2.705%
m 4.374%
n 2.365%
o 6.264%
p 2.545%
q 0.173%
r 1.653%
s 7.755%
t 16.671%
u 1.487%
v 0.619%
w 6.661%
x 0.005%
y 1.620%
z 0.050%

Relative frequencies of letters in other languages

Letter French
French language
French is a Romance language spoken as a first language in France, the Romandy region in Switzerland, Wallonia and Brussels in Belgium, Monaco, the regions of Quebec and Acadia in Canada, and by various communities elsewhere. Second-language speakers of French are distributed throughout many parts...

 
German
German language
German is a West Germanic language, related to and classified alongside English and Dutch. With an estimated 90 – 98 million native speakers, German is one of the world's major languages and is the most widely-spoken first language in the European Union....

 
Spanish
Spanish language
Spanish , also known as Castilian , is a Romance language in the Ibero-Romance group that evolved from several languages and dialects in central-northern Iberia around the 9th century and gradually spread with the expansion of the Kingdom of Castile into central and southern Iberia during the...

 
Portuguese
Portuguese language
Portuguese is a Romance language that arose in the medieval Kingdom of Galicia, nowadays Galicia and Northern Portugal. The southern part of the Kingdom of Galicia became independent as the County of Portugal in 1095...

 
Esperanto  Italian
Italian language
Italian is a Romance language spoken mainly in Europe: Italy, Switzerland, San Marino, Vatican City, by minorities in Malta, Monaco, Croatia, Slovenia, France, Libya, Eritrea, and Somalia, and by immigrant communities in the Americas and Australia...

Turkish
Turkish language
Turkish is a language spoken as a native language by over 83 million people worldwide, making it the most commonly spoken of the Turkic languages. Its speakers are located predominantly in Turkey and Northern Cyprus with smaller groups in Iraq, Greece, Bulgaria, the Republic of Macedonia, Kosovo,...

Swedish
Swedish language
Swedish is a North Germanic language, spoken by approximately 10 million people, predominantly in Sweden and parts of Finland, especially along its coast and on the Åland islands. It is largely mutually intelligible with Norwegian and Danish...

Polish
Polish language
Polish is a language of the Lechitic subgroup of West Slavic languages, used throughout Poland and by Polish minorities in other countries...

Dutch
Dutch language
Dutch is a West Germanic language and the native language of the majority of the population of the Netherlands, Belgium, and Suriname, the three member states of the Dutch Language Union. Most speakers live in the European Union, where it is a first language for about 23 million and a second...

 
a 7.636% 6.51% 12.53% 14.63% 12.12% 11.74% 11.68% 9.3% 8.0% 7.49%
b 0.901% 1.89% 1.42% 1.04% 0.98% 0.92% 2.95% 1.3% 1.3% 1.58%
c 3.260% 3.06% 4.68% 3.88% 0.78% 4.5% 0.97% 1.3% 3.8% 1.24%
d 3.669% 5.08% 5.86% 4.99% 3.04% 3.73% 4.87% 4.5% 3.0% 5.93%
e 14.715% 17.40% 13.68% 12.57% 8.99% 11.79% 9.01% 9.9% 6.9% 18.91%
f 1.066% 1.66% 0.69% 1.02% 1.03% 0.95% 0.44% 2.0% 0.1% 0.81%
g 0.866% 3.01% 1.01% 1.30% 1.17% 1.64% 1.34% 3.3% 1.0% 3.40%
h 0.737% 4.76% 0.70% 1.28% 0.38% 1.54% 1.14% 2.1% 1.0% 2.38%
i 7.529% 7.55% 6.25% 6.18% 10.01% 11.28% 8.27%* 5.1% 7.0% 6.50%
j 0.545% 0.27% 0.44% 0.40% 3.50% 0.00% 0.01% 0.7% 1.9% 1.46%
k 0.049% 1.21% 0.01% 0.02% 4.16% 0.00% 4.71% 3.2% 2.7% 2.25%
l 5.456% 3.44% 4.97% 2.78% 6.14% 6.51% 5.75% 5.2% 3.1% 3.57%
m 2.968% 2.53% 3.15% 4.74% 2.99% 2.51% 3.74% 3.5% 2.4% 2.21%
n 7.095% 9.78% 6.71% 5.05% 7.96% 6.88% 7.23% 8.8% 4.7% 10.03%
o 5.378% 2.51% 8.68% 10.73% 8.78% 9.83% 2.45% 4.1% 7.1% 6.06%
p 3.021% 0.79% 2.51% 2.52% 2.74% 3.05% 0.79% 1.7% 2.4% 1.57%
q 1.362% 0.02% 0.88% 1.20% 0.00% 0.51% 0 0.007%
0.009%
r 6.553% 7.00% 6.87% 6.53% 5.91% 6.37% 6.95% 8.3% 3.5% 6.41%
s 7.948% 7.27% 7.98% 7.81% 6.09% 4.98% 2.95% 6.3% 3.8% 3.73%
t 7.244% 6.15% 4.63% 4.74% 5.27% 5.62% 3.09% 8.7% 2.4% 6.79%
u 6.311% 4.35% 3.93% 4.63% 3.18% 3.01% 3.43% 1.8% 1.8% 1.99%
v 1.628% 0.67% 0.90% 1.67% 1.90% 2.10% 0.98% 2.4%
2.85%
w 0.114% 1.89% 0.02% 0.01% 0.00% 0.00% 0 0.03% 3.6% 1.52%
x 0.387% 0.03% 0.22% 0.21% 0.00% 0.00% 0 0.1%
0.04%
y 0.308% 0.04% 0.90% 0.01% 0.00% 0.00% 3.37% 0.6% 3.2% 0.035%
z 0.136% 1.13% 0.52% 0.47% 0.50% 0.49% 1.50% 0.02% 5.1% 1.39%
à 0.486% 0 0 see a 0 see a 0 0.0% 0 see a
å 0 0 0 0 0 0 0 1.6% 0
ä 0
0 0 0 0 0 2.1% 0 see a
ą 0
0 0 0 0 0 0 see a
œ 0.018% 0 0 0 0 0 0 0 0
ç 0.085% 0 0 see c 0 0 1.26% 0 0
ĉ 0 0 0 0 0.66% 0 0 0 0
ć 0
0 0 0 0 0 0 see c
è 0.271% 0 0 0 0 see e 0 0.0% 0 see e
é 1.904% 0 0 see e 0 see e 0 0.0% 0 see e
ê 0.225% 0 0 see e 0 0 0 0 0
ë 0.001% 0 0 0 0 0 0 0 0 see e
ę 0
0 0 0 0 0 0 see e
ĝ 0 0 0 0 0.69% 0 0 0 0
ğ 0 0 0 0 0 0 1.13% 0 0
ĥ 0 0 0 0 0.02% 0 0 0 0
î 0.045% 0 0 0 0 0 0 0 0
ì 0 0 0 0 0 see i 0 0 0 see i
ï 0.005% 0 0 0 0 0 0 0 0 see i
ı 0 0 0 0 0 0 5.20%* 0 0
ĵ 0 0 0 0 0.12% 0 0 0 0
ł 0
0 0 0 0 0 0 see l
ñ 0 0 0.31% 0 0 0 0 0 0
ń 0
0 0 0 0 0 0 see n
ò 0 0 0 0 0 see o 0 0 0 see o
ö 0
0 0 0 0 0.87% 1.5% 0 see o
ó 0
0 see o 0 0 0 0 see o see o
ŝ 0 0 0 0 0.38% 0 0 0 0
ş 0 0 0 0 0 0 1.94% 0 0
ś 0
0 0 0 0 0 0 see s
ß 0 0.31% 0 0 0 0 0 0 0
ù 0.058% 0 0 0 0 see u 0 0 0 see u
ŭ 0 0 0 0 0.52% 0 0 0 0
ü 0
0 0 0 0 1.99% 0 0 see u
ź 0
0 0 0 0 0 0 see z
ż 0
0 0 0 0 0 0 0.7%


*See Turkish dotted and dotless I
Turkish dotted and dotless I
The Turkish alphabet, which is a variant of the Latin alphabet, includes two distinct versions of the letter I, one dotted and the other dotless. The difference between the two versions is modelled after the letters Ö and Ü, which were taken from German. These two letters represent front-vowel...



The figure below illustrates the frequency distributions of the 26 most common Latin letters across some languages.
Based on these tables, the 'etaoin shrdlu'-equivalent results for each language is as follows:
  • French: 'esait nrulo'; (Indo-European: Romance; traditionally, 'esartinulop' is used, in part for its ease of pronunciation)
  • Spanish: 'eaosr nidlc'; (Indo-European: Romance)
  • Portuguese: 'aeosr indmt' (Indo-European: Romance)
  • Italian: 'eaion lrtsc'; (Indo-European: Romance)
  • Esperanto: 'aieon lsrtk' (artificial language – influenced by Indo-European languages, Romance, Germanic mostly)
  • German: 'enisr atdhu'; (Indo-European: Germanic)
  • Swedish: 'eantr slido'; (Indo-European: Germanic)
  • Turkish: 'aeinr ldkmu'; (Turkic: a non Indo-European language)
  • Dutch: 'enati rodsl'; (Indo-European: Germanic)
  • Polish: 'aoiez nscwr'; (Indo-European: Slavic)


All these languages use a basically similar 25+ character alphabet.

See also

  • Corpus linguistics
    Corpus linguistics
    Corpus linguistics is the study of language as expressed in samples or "real world" text. This method represents a digestive approach to deriving a set of abstract rules by which a natural language is governed or else relates to another language. Originally done by hand, corpora are now largely...

  • ETAOIN SHRDLU
    ETAOIN SHRDLU
    ETAOIN SHRDLU is a nonsense phrase that sometimes appeared in print in the days of "hot type" publishing because of a custom of Linotype machine operators. It appeared frequently enough that it became part of the lore of newspapers...

  • RSTLNE (Wheel of Fortune)
  • Frequency analysis (cryptanalysis)
  • Linotype machine
    Linotype machine
    The Linotype typesetting machine is a "line casting" machine used in printing. The name of the machine comes from the fact that it produces an entire line of metal type at once, hence a line-o'-type, a significant improvement over manual typesetting....

  • Most common words in English
    Most common words in English
    The list below of most common words in English cannot be definitive. It is based on an analysis of the Oxford English Corpus of over a billion words, and represents one study done by Oxford Online, associated with the Oxford English Dictionary...

  • Scrabble
    Scrabble
    Scrabble is a word game in which two to four players score points by forming words from individual lettered tiles on a game board marked with a 15-by-15 grid. The words are formed across and down in crossword fashion and must appear in a standard dictionary. Official reference works provide a list...


External links

The source of this article is wikipedia, the free encyclopedia.  The text of this article is licensed under the GFDL.
 
x
OK