Loebner prize
Encyclopedia
The Loebner Prize is an annual competition in artificial intelligence
that awards prizes to the chatterbot
considered by the judges to be the most human-like. The format of the competition is that of a standard Turing test
. In each round, a human judge simultaneously holds textual conversations with a computer program and a human being via computer. Based upon the responses, the judge must decide which is which.
The contest began in 1990 by Hugh Loebner
in conjunction with the Cambridge Center for Behavioral Studies
, Massachusetts
, United States
. It has since been associated with Flinders University
, Dartmouth College
, the Science Museum
in London
, and most recently the University of Reading. In 2004 and 2005, it was held in Loebner's apartment in New York City
.
Within the field of artificial intelligence, the Loebner Prize is somewhat controversial; the most prominent critic, Marvin Minsky
, has called it a publicity stunt that does not help the field along.
In addition, the time limit of 5 minutes and the use of untrained and unsophisticated judges has resulted in some wins that may be due to trickery rather than to plausible intelligence, as one can judge from transcripts of winning conversations (linked below).
In addition, there are two one-time-only prizes that have never been awarded. $25,000 is offered for the first chatterbot that judges cannot distinguish from a real human and which can convince judges that the human is the computer program. $100,000 is the reward for the first chatterbot that judges cannot distinguish from a real human in a Turing test that includes deciphering and understanding text, visual, and auditory input. Once this is achieved, the annual competition will end.
For the three entries in 2007, Robert Medeksza, Noah Duncan and Rollo Carpenter, some basic "screening questions" were used by the sponsor to evaluate the state of the technology. These included simple questions about the time, what round of the contest it is, etc.; general knowledge ("What is a hammer for?"); comparisons ("Which is faster, a train or a plane?"); and questions demonstrating memory for preceding parts of the same conversation. "All nouns, adjectives and verbs will come from a dictionary suitable for children or adolescents under the age of 12." Entries did not need to respond "intelligently" to the questions to be accepted.
For the first time in 2008 the sponsor allowed introduction of a preliminary phase to the contest opening up the competition to previously disallowed web-based entries judged by a variety of invited interrogators. The available rules do not state how interrogators are selected or instructed. Interrogators (who judge the systems) have limited time: 5 minutes per entity in the 2003 competition, 20+ per pair in 2004–2007 competitions, and 5 minutes to conduct simultaneous conversations with a human and the program since 2008.
(CEO of Televirtual) and Huma Shah. On August 30, the four finalists were announced:
The contest was held on 17 September in the VR theatre, Torrington Place campus of University College London
. The judges included the University of Reading's cybernetics
professor, Kevin Warwick
, a professor of artificial intelligence, John Barnden (specialist in metaphor research at the University of Birmingham
), a barrister, Victoria Butler-Cole and a journalist, Graham Duncan-Rowe. The latter's experience of the event can be found in an article in Technology Review
. The winner was 'Joan', based on Jabberwacky
, both created by Rollo Carpenter.
, Psychology Assistant Professor Clayton Curtis and English lecturer Scott Hutchins.
No bot passed the Turing Test, but the judges ranked the three contestants as follows:
The winner received $2,250 and the annual medal. The runners-up received $250 each.
. After testing by over one hundred judges during the preliminary phase, in June and July 2008, six finalists were selected from thirteen original entrants - artificial conversational entity (ACE). Five of those invited competed in the finals:
In the finals, each of the judges was given five minutes to conduct simultaneous, split-screen conversations with two hidden entities. Elbot of Artificial Solutions won the 2008 Loebner Prize bronze award, for most human-like artificial conversational entity, through fooling three of the twelve judges who interrogated it (in the human-parallel comparisons) into believing it was human. This is coming very close to the 30% traditionally required to consider that a program has actually passed the Turing test. Eugene Goostman and Ultra Hal both deceived one judge each that it was the human.
Will Pavia, a journalist for The Times
, has written about his experience; a Loebner finals' judge, he was deceived by Elbot and Eugene. Kevin Warwick
and Huma Shah have reported on the parallel-paired Turing tests here. Various misunderstandings of the 2008 Loebner Prize are blogged here.
Entrants were David Levy, Rollo Carpenter, and Mohan Embar, who finished in that order.
More complete results of the contest, and links to transcripts are available at Loebner Prize 2009.
The writer Brian Christian
participated in the 2009 Loebner Prize Competition as a human confederate, and described his experiences at the competition in his book The Most Human Human.
Artificial intelligence
Artificial intelligence is the intelligence of machines and the branch of computer science that aims to create it. AI textbooks define the field as "the study and design of intelligent agents" where an intelligent agent is a system that perceives its environment and takes actions that maximize its...
that awards prizes to the chatterbot
Chatterbot
A chatter robot, chatterbot, chatbot, or chat bot is a computer program designed to simulate an intelligent conversation with one or more human users via auditory or textual methods, primarily for engaging in small talk. The primary aim of such simulation has been to fool the user into thinking...
considered by the judges to be the most human-like. The format of the competition is that of a standard Turing test
Turing test
The Turing test is a test of a machine's ability to exhibit intelligent behaviour. In Turing's original illustrative example, a human judge engages in a natural language conversation with a human and a machine designed to generate performance indistinguishable from that of a human being. All...
. In each round, a human judge simultaneously holds textual conversations with a computer program and a human being via computer. Based upon the responses, the judge must decide which is which.
The contest began in 1990 by Hugh Loebner
Hugh Loebner
Hugh Loebner is notable as the sponsor of the Loebner Prize, an embodiment of the Turing test. He is an American inventor, holding six United States Patents. He is also an outspoken social activist for the decriminalization of prostitution.- Loebner prize:Loebner established the Loebner Prize in...
in conjunction with the Cambridge Center for Behavioral Studies
Cambridge Center for Behavioral Studies
The Cambridge Center for Behavioral Studies was founded in 1981 by American psychologist Robert Epstein as an advanced studies institute devoted to promoting the scientific study of behavior and its humane applications in human affairs...
, Massachusetts
Massachusetts
The Commonwealth of Massachusetts is a state in the New England region of the northeastern United States of America. It is bordered by Rhode Island and Connecticut to the south, New York to the west, and Vermont and New Hampshire to the north; at its east lies the Atlantic Ocean. As of the 2010...
, United States
United States
The United States of America is a federal constitutional republic comprising fifty states and a federal district...
. It has since been associated with Flinders University
Flinders University
Flinders University, , is a public university in Adelaide, South Australia. Founded in 1966, it was named in honour of navigator Matthew Flinders, who explored and surveyed the South Australian coastline in the early 19th century.The university has established a reputation as a leading research...
, Dartmouth College
Dartmouth College
Dartmouth College is a private, Ivy League university in Hanover, New Hampshire, United States. The institution comprises a liberal arts college, Dartmouth Medical School, Thayer School of Engineering, and the Tuck School of Business, as well as 19 graduate programs in the arts and sciences...
, the Science Museum
Science Museum (London)
The Science Museum is one of the three major museums on Exhibition Road, South Kensington, London in the Royal Borough of Kensington and Chelsea. It is part of the National Museum of Science and Industry. The museum is a major London tourist attraction....
in London
London
London is the capital city of :England and the :United Kingdom, the largest metropolitan area in the United Kingdom, and the largest urban zone in the European Union by most measures. Located on the River Thames, London has been a major settlement for two millennia, its history going back to its...
, and most recently the University of Reading. In 2004 and 2005, it was held in Loebner's apartment in New York City
New York City
New York is the most populous city in the United States and the center of the New York Metropolitan Area, one of the most populous metropolitan areas in the world. New York exerts a significant impact upon global commerce, finance, media, art, fashion, research, technology, education, and...
.
Within the field of artificial intelligence, the Loebner Prize is somewhat controversial; the most prominent critic, Marvin Minsky
Marvin Minsky
Marvin Lee Minsky is an American cognitive scientist in the field of artificial intelligence , co-founder of Massachusetts Institute of Technology's AI laboratory, and author of several texts on AI and philosophy.-Biography:...
, has called it a publicity stunt that does not help the field along.
In addition, the time limit of 5 minutes and the use of untrained and unsophisticated judges has resulted in some wins that may be due to trickery rather than to plausible intelligence, as one can judge from transcripts of winning conversations (linked below).
Prizes
Originally, $2,000 was awarded for the most human-seeming chatterbot in the competition. The prize was $3,000 in 2005 and $2,250 in 2006. In 2008, $3,000 was awarded.In addition, there are two one-time-only prizes that have never been awarded. $25,000 is offered for the first chatterbot that judges cannot distinguish from a real human and which can convince judges that the human is the computer program. $100,000 is the reward for the first chatterbot that judges cannot distinguish from a real human in a Turing test that includes deciphering and understanding text, visual, and auditory input. Once this is achieved, the annual competition will end.
Competition rules and restrictions
The rules have varied over the years and early competitions featured restricted conversation Turing tests but since 1995 the discussion has been unrestricted.For the three entries in 2007, Robert Medeksza, Noah Duncan and Rollo Carpenter, some basic "screening questions" were used by the sponsor to evaluate the state of the technology. These included simple questions about the time, what round of the contest it is, etc.; general knowledge ("What is a hammer for?"); comparisons ("Which is faster, a train or a plane?"); and questions demonstrating memory for preceding parts of the same conversation. "All nouns, adjectives and verbs will come from a dictionary suitable for children or adolescents under the age of 12." Entries did not need to respond "intelligently" to the questions to be accepted.
For the first time in 2008 the sponsor allowed introduction of a preliminary phase to the contest opening up the competition to previously disallowed web-based entries judged by a variety of invited interrogators. The available rules do not state how interrogators are selected or instructed. Interrogators (who judge the systems) have limited time: 5 minutes per entity in the 2003 competition, 20+ per pair in 2004–2007 competitions, and 5 minutes to conduct simultaneous conversations with a human and the program since 2008.
2006
In 2006, the contest was organised by Tim ChildTim Child
Tim Child is a British television producer. He is most famous for creating and producing the cult children's television program Knightmare, a fantasy role-playing adventure game which pioneered virtual studio or blue-screen production....
(CEO of Televirtual) and Huma Shah. On August 30, the four finalists were announced:
- Rollo CarpenterRollo CarpenterRollo Carpenter is the British-born creator of Jabberwacky and Cleverbot, learning Artificial Intelligence software. Carpenter has worked as CTO of a business software startup in Silicon Valley, but returned to the UK to work at Icogno....
- Richard Churchill and Marie-Claire Jenkins
- Noah Duncan
- Robert Medeksza
The contest was held on 17 September in the VR theatre, Torrington Place campus of University College London
University College London
University College London is a public research university located in London, United Kingdom and the oldest and largest constituent college of the federal University of London...
. The judges included the University of Reading's cybernetics
Cybernetics
Cybernetics is the interdisciplinary study of the structure of regulatory systems. Cybernetics is closely related to information theory, control theory and systems theory, at least in its first-order form...
professor, Kevin Warwick
Kevin Warwick
Kevin Warwick is a British scientist and professor of cybernetics at the University of Reading, Reading, Berkshire, United Kingdom...
, a professor of artificial intelligence, John Barnden (specialist in metaphor research at the University of Birmingham
University of Birmingham
The University of Birmingham is a British Redbrick university located in the city of Birmingham, England. It received its royal charter in 1900 as a successor to Birmingham Medical School and Mason Science College . Birmingham was the first Redbrick university to gain a charter and thus...
), a barrister, Victoria Butler-Cole and a journalist, Graham Duncan-Rowe. The latter's experience of the event can be found in an article in Technology Review
Technology Review
Technology Review is a magazine published by the Massachusetts Institute of Technology. It was founded in 1899 as "The Technology Review", and was re-launched without the "The" in its name on April 23, 1998 under then publisher R. Bruce Journey...
. The winner was 'Joan', based on Jabberwacky
Jabberwacky
Jabberwacky is a chatterbot created by British programmer Rollo Carpenter. Its stated aim is to "simulate natural human chat in an interesting, entertaining and humorous manner"...
, both created by Rollo Carpenter.
2007
The 2007 competition was held on 21 October in New York City. The judges were: Computer Science Professor Russ Abbott, Philosophy Professor Hartry FieldHartry Field
Hartry H. Field is a philosopher, the Silver Professor of Philosophy at New York University. He previously taught at the University of Southern California and The Graduate Center of the City University of New York. He earned his Ph.D...
, Psychology Assistant Professor Clayton Curtis and English lecturer Scott Hutchins.
No bot passed the Turing Test, but the judges ranked the three contestants as follows:
- 1st: Robert Medeksza from Zabaware, creator of Ultra Hal AssistantUltra Hal AssistantUltra Hal Assistant is a chatterbot intended to function as a personal assistant. It was developed by Zabaware, Inc.Ultra Hal won the 2007 Loebner Prize for "most human" chatterbot....
- 2nd: Noah Duncan, a private entry, creator of Cletus
- 3rd: Rollo Carpenter from Icogno, creator of Jabberwacky
The winner received $2,250 and the annual medal. The runners-up received $250 each.
2008
The 2008 competition was organised by Professor Kevin Warwick, coordinated by Huma Shah and held on 12 October at the University of Reading, UKUnited Kingdom
The United Kingdom of Great Britain and Northern IrelandIn the United Kingdom and Dependencies, other languages have been officially recognised as legitimate autochthonous languages under the European Charter for Regional or Minority Languages...
. After testing by over one hundred judges during the preliminary phase, in June and July 2008, six finalists were selected from thirteen original entrants - artificial conversational entity (ACE). Five of those invited competed in the finals:
- Brother Jerome, Peter Cole and Benji Adams
- ElbotElbotElbot is a chatterbot created by Fred Roberts.At the 18th Loebner Prize for artificial intelligence, held on the weekend of 11–12 October 2008, Elbot convinced three of the 12 human interrogators he was indistinguishable from human, beating the other contestants and taking the Bronze Prize.If Elbot...
, Fred Roberts / Artificial Solutions - Eugene GoostmanEugene GoostmanEugene Goostman is a computer programme, an Artificial Conversational Entity . It was runner up in the 2001 , 2005 and 2008 Loebner Prize for Artificial Intelligence....
, Vladimir Veselov, Eugene Demchenko and Sergey Ulasen - JabberwackyJabberwackyJabberwacky is a chatterbot created by British programmer Rollo Carpenter. Its stated aim is to "simulate natural human chat in an interesting, entertaining and humorous manner"...
, Rollo Carpenter - Ultra Hal, Robert Medeksza
In the finals, each of the judges was given five minutes to conduct simultaneous, split-screen conversations with two hidden entities. Elbot of Artificial Solutions won the 2008 Loebner Prize bronze award, for most human-like artificial conversational entity, through fooling three of the twelve judges who interrogated it (in the human-parallel comparisons) into believing it was human. This is coming very close to the 30% traditionally required to consider that a program has actually passed the Turing test. Eugene Goostman and Ultra Hal both deceived one judge each that it was the human.
Will Pavia, a journalist for The Times
The Times
The Times is a British daily national newspaper, first published in London in 1785 under the title The Daily Universal Register . The Times and its sister paper The Sunday Times are published by Times Newspapers Limited, a subsidiary since 1981 of News International...
, has written about his experience; a Loebner finals' judge, he was deceived by Elbot and Eugene. Kevin Warwick
Kevin Warwick
Kevin Warwick is a British scientist and professor of cybernetics at the University of Reading, Reading, Berkshire, United Kingdom...
and Huma Shah have reported on the parallel-paired Turing tests here. Various misunderstandings of the 2008 Loebner Prize are blogged here.
2009
The 2009 Loebner Prize Competition was held 6 September 2009 at the Brighton Centre, Brighton UK in conjunction with Interspeech 2009 conference. The prize amount for 2009 was USD 3000.Entrants were David Levy, Rollo Carpenter, and Mohan Embar, who finished in that order.
More complete results of the contest, and links to transcripts are available at Loebner Prize 2009.
The writer Brian Christian
Brian Christian
Brian Christian is an American author and poet, best known for his book The Most Human Human. He competed as a "confederate" in the 2009 Loebner Prize competition, attempting to seem "more human" than the humans taking the test, and succeeded...
participated in the 2009 Loebner Prize Competition as a human confederate, and described his experiences at the competition in his book The Most Human Human.
2010
The 2010 Loebner Prize Competition was held on October 23 at California State University, Los Angeles. The 2010 competition was the 20th running of the contest.2011
The 2011 Loebner Prize Competition is to be held on the 19th of October in Exeter Univerity, Devon, United Kingdom. This year there is an addition of a panel of junior judges, namedly the young "nerds" Jean-Paul Astal-Stain, William Dunne and Kiril Jerdev.Winners
Official list of winners.| Year | Winner | Program |
|---|---|---|
| 1991 | Joseph Weintraub | PC Therapist |
| 1992 | Joseph Weintraub | PC Therapist |
| 1993 | Joseph Weintraub | PC Therapist |
| 1994 | Thomas Whalen | TIPS |
| 1995 | Joseph Weintraub | PC Therapist |
| 1996 | Jason Hutchens | HeX |
| 1997 | David Levy David Levy (chess player) David Neil Laurence Levy , is a Scottish International Master of chess, a businessman noted for his involvement with computer chess and artificial intelligence, and the founder of the Computer Olympiads and the Mind Sports Olympiads. He has written more than 40 books on chess and computers.- Life... |
Converse |
| 1998 | Robby Garner Robby Garner Robby Garner is a natural language programmer and software developer. He won the 1998 and 1999 Loebner Prize Contests with the program called Albert One. He is listed in the 2001 Guinness Book of World Records as having written the "most human" computer program.-Life:A native of Cedartown,... |
Albert One Albert one Albert One is an AI chatterbot bot created by Robby Garner and designed to mimic the way humans make conversations using a multi-faceted approach in natural language programming.-History:... |
| 1999 | Robby Garner Robby Garner Robby Garner is a natural language programmer and software developer. He won the 1998 and 1999 Loebner Prize Contests with the program called Albert One. He is listed in the 2001 Guinness Book of World Records as having written the "most human" computer program.-Life:A native of Cedartown,... |
Albert One Albert one Albert One is an AI chatterbot bot created by Robby Garner and designed to mimic the way humans make conversations using a multi-faceted approach in natural language programming.-History:... |
| 2000 | Richard Wallace Richard Wallace (scientist) Richard Wallace is the author of AIML and Botmaster of ALICE . Dr. Wallace's work has appeared in the New York Times, WIRED, CNN, ZDTV and in numerous foreign language publications across Asia, Latin America and Europe.Richard Wallace was born in Portland, Maine in 1960. He earned his Ph.D... |
Artificial Linguistic Internet Computer Entity Artificial Linguistic Internet Computer Entity A.L.I.C.E. , also referred to as Alicebot, or simply Alice, is a natural language processing chatterbot—a program that engages in a conversation with a human by applying some heuristical pattern matching rules to the human's input, and in its online form it also relies on a hidden third person... (A.L.I.C.E.) |
| 2001 | Richard Wallace Richard Wallace (scientist) Richard Wallace is the author of AIML and Botmaster of ALICE . Dr. Wallace's work has appeared in the New York Times, WIRED, CNN, ZDTV and in numerous foreign language publications across Asia, Latin America and Europe.Richard Wallace was born in Portland, Maine in 1960. He earned his Ph.D... |
Artificial Linguistic Internet Computer Entity Artificial Linguistic Internet Computer Entity A.L.I.C.E. , also referred to as Alicebot, or simply Alice, is a natural language processing chatterbot—a program that engages in a conversation with a human by applying some heuristical pattern matching rules to the human's input, and in its online form it also relies on a hidden third person... (A.L.I.C.E.) |
| 2002 | Kevin Copple | Ella |
| 2003 | Juergen Pirner Juergen Pirner Juergen Pirner is the German creator of , a chatterbot that won the 2003 Loebner prize.Pirner created Jabberwock modelling the Jabberwocky from Lewis Carroll's poem of the same name. Initially, Jabberwock would just give rude or fantasy-related answers; but over the years, Pirner has programmed... |
Jabberwock |
| 2004 | Richard Wallace Richard Wallace (scientist) Richard Wallace is the author of AIML and Botmaster of ALICE . Dr. Wallace's work has appeared in the New York Times, WIRED, CNN, ZDTV and in numerous foreign language publications across Asia, Latin America and Europe.Richard Wallace was born in Portland, Maine in 1960. He earned his Ph.D... |
Artificial Linguistic Internet Computer Entity Artificial Linguistic Internet Computer Entity A.L.I.C.E. , also referred to as Alicebot, or simply Alice, is a natural language processing chatterbot—a program that engages in a conversation with a human by applying some heuristical pattern matching rules to the human's input, and in its online form it also relies on a hidden third person... (A.L.I.C.E.) |
| 2005 | Rollo Carpenter Rollo Carpenter Rollo Carpenter is the British-born creator of Jabberwacky and Cleverbot, learning Artificial Intelligence software. Carpenter has worked as CTO of a business software startup in Silicon Valley, but returned to the UK to work at Icogno.... |
George |
| 2006 | Rollo Carpenter Rollo Carpenter Rollo Carpenter is the British-born creator of Jabberwacky and Cleverbot, learning Artificial Intelligence software. Carpenter has worked as CTO of a business software startup in Silicon Valley, but returned to the UK to work at Icogno.... |
Joan |
| 2007 | Robert Medeksza | Ultra Hal Ultra Hal Assistant Ultra Hal Assistant is a chatterbot intended to function as a personal assistant. It was developed by Zabaware, Inc.Ultra Hal won the 2007 Loebner Prize for "most human" chatterbot.... |
| 2008 | Fred Roberts | Elbot Elbot Elbot is a chatterbot created by Fred Roberts.At the 18th Loebner Prize for artificial intelligence, held on the weekend of 11–12 October 2008, Elbot convinced three of the 12 human interrogators he was indistinguishable from human, beating the other contestants and taking the Bronze Prize.If Elbot... |
| 2009 | David Levy David Levy (chess player) David Neil Laurence Levy , is a Scottish International Master of chess, a businessman noted for his involvement with computer chess and artificial intelligence, and the founder of the Computer Olympiads and the Mind Sports Olympiads. He has written more than 40 books on chess and computers.- Life... |
Do-Much-More |
| 2010 | Bruce Wilcox Bruce Wilcox -MTS/LISP and Computer Go:Wilcox wrote the MTS/LISP interpreter back in the early 70's, in order to be able to write a Go program for Dr. Walter Reitman... |
Suzette |
| 2011 | Bruce Wilcox Bruce Wilcox -MTS/LISP and Computer Go:Wilcox wrote the MTS/LISP interpreter back in the early 70's, in order to be able to write a Go program for Dr. Walter Reitman... |
Rosette |
External links
- The Loebner Prize website. Includes transcripts of some competitions.
- Cocktail-Party Conversation -- With a Computer, The New York Times, 10 January 1993. Conversation with the 1992 winner; topic: men and women.
- Wired correspondent reports from the fourth Loebner contest, Wired, April 1995.
- Fiske-Harrison, AlexanderAlexander Fiske-HarrisonAlexander Rupert Fiske-Harrison is an English writer and actor. He is best known for writing and acting in The Pendulum in London's West End and for his research into bullfighting for his book Into The Arena, which has led The Times to describe him as "the bullfighter-philosopher." Into The Arena...
, A.L.I.C.E.'s springs - Do computers really converse?, The Times Literary Supplement, June 9, 2000. (from the 10th Loebner Prize in 2000 at Dartmouth College, USA)