|
|
|
|
Magic (cryptography)
|
| |
|
| |
Magic was an Allied cryptanalysis project during World War II. Magic's efforts were directed at breaking Japan's diplomatic crypotgraphic codes, allowing Allied policy-makers to read Japan's diplomatic messages.
Japanese used a cypher machine to encode their cable traffic. The machine was used by the Japanese Foreign Office, and was called PURPLE by U.S.
Discussion
Ask a question about 'Magic (cryptography)' Start a new discussion about 'Magic (cryptography)' Answer questions from other users |
Encyclopedia
Magic was an Allied cryptanalysis project during World War II. Magic's efforts were directed at breaking Japan's diplomatic crypotgraphic codes, allowing Allied policy-makers to read Japan's diplomatic messages.
Information carried in PURPLE
The Japanese used a cypher machine to encode their cable traffic. The machine was used by the Japanese Foreign Office, and was called PURPLE by U.S. cryptographers. A message was typed into an machine, which encoded and sent it to another unit. The receiving unit could decipher the code only if provided with correct code settings. American cryptographers were able to build a machine that could decode these messages. The PURPLE machine itself was first used by Japan in 1940. United States and British cryptographers had broken some of its codes well before the attack on Pearl Harbor. However the PURPLE codes were only used by the Foreign Office to carry diplomatic traffic to its embassies. The Japanese Navy used a completely different form of code, known as JN-25.
The U.S. found no hint of the attack on Pearl Harbor in PURPLE at the time, nor could they have as the Japanese were very careful to not discuss the planned attack in Foreign Office communications. In fact, no detailed information about the planned attack was even available to the Japanese Foreign Office; it was regarded by the military, particularly the more nationalistic military, as insufficiently 'reliable'. United States access to private Japanese diplomatic communications (even the most secret ones) was less useful than it might otherwise have been because policy in Japan in the pre-War period was controlled largely by military groups (e.g., in China and Manchuria), not by the Foreign Office. The Foreign Office itself deliberately kept from its embassies and consulates much of the information it did have, so the ability to read PURPLE messages was less than definitive regarding Japanese tactical or strategic military intentions.
United States cryptographers had decrypted and translated the 14-part Japanese diplomatic message breaking off ongoing negotiations with the United States at 1 p.m. Washington time on 7 December 1941, before the Japanese Embassy in Washington could do so. Due to difficulties at the Embassy the note was delivered late. When the two Japanese diplomats delivered the note to Secretary of State Cordell Hull, he had to pretend to be reading it for the first time, even though he already knew about the attack on Pearl Harbor.
Throughout the war the Allies routinely read both German and Japanese codes. The Japanese Ambassador to Germany, General Hiroshi Oshima, often sent priceless German military information to Tokyo. This information was routinely intercepted and read by Roosevelt, Churchill, and Eisenhower. According to Lowman, "The Japanese considered the PURPLE system absolutely unbreakable....Most went to their graves refusing to believe the code had been broken by analytic means....They believed someone had betrayed their system."
Handling of Magic prior to Pearl Harbor attacks
Even so, the diplomatic information discovered was of even more limited value to the U.S. because of its dissemination within the U.S. Government. "Magic" was distributed in such a way many policy makers who should have had access to it to do their jobs knew nothing of it, and those to whom it actually was distributed (at least before Pearl Harbor) saw each message only briefly, as the courier stood by to take it back, and in isolation from all others (no copies or notes were permitted). Before Pearl Harbor, in any case, they saw only those decrypts thought "important enough" by the distributing Army or Navy officers. Nonetheless, being able to read PURPLE messages gave the Allies a great advantage in the War; for instance, the Japanese ambassador to Germany produced long reports for Tokyo which were encrypted with the PURPLE machine. They included reports on personal discussions with Adolf Hitler and a report on a tour of the invasion defenses in Northern France (including the D-Day invasion beaches).
Dewey and Marshall
During the 1944 election, Thomas Dewey threatened to make Pearl Harbor a campaign issue, until General Marshall sent him a personal letter which said, in part:
- To explain the critical nature of this set-up, which would be wiped out in an instant if the least suspicion were aroused regarding it, the Battle of Coral Sea was based on deciphered messages and therefore our few ships were in the right place at the right time. Further, we were able to concentrate our limited forces to meet their naval advance on Midway when otherwise we almost certainly would have been some 3000 miles out of place. We had full information on the strength of their forces.
Dewey promised not to raise the issue, and kept his word.
Post-war debates
The break into the PURPLE system, and into Japanese messages generally, was the subject of acrimonious hearings in Congress post-World War II in connection with an attempt to decide who, if anyone, had allowed the disaster at Pearl Harbor to happen and who therefore should be blamed. During those hearings the Japanese learned, for the first time, the PURPLE cypher system had been broken. They had been continuing to use it, even after the War, with the encouragement of the American Occupation Government. Much confusion over who in Washington or Hawaii knew what and when, especially as "we were decrypting their messages," has led some to conclude "someone in Washington" knew about the Pearl Harbor attack before it happened, and, since Pearl Harbor was not expecting to be attacked, the "failure to warn Hawaii one was coming must have been deliberate, since it could hardly have been mere oversight". However, PURPLE was a diplomatic, not a military code; thus, only inferences could be drawn from PURPLE as to specific Japanese military actions.
History
When PURPLE was broken by the U.S. Army's Signals Intelligence Service (SIS), several problems arose for the Americans: who would get the decrypts, which decrypts, how often, under what circumstances, and crucially (given interservice rivalries) who would do the delivering. Both the U.S. Navy and Army were insistent they alone handle all decrypted traffic delivery, especially to highly placed policy makers in the U.S. Eventually, after much toing and froing, a compromise was reached: the Army would be responsible for the decrypts on one day, and the Navy the next.
The distribution list eventually included some—but not all—military intelligence leaders in Washington and elsewhere, some—but again not all—civilian policy leaders in Washington. The eventual routine for distribution included the following steps:
- the duty officer (Army or Navy, depending on the day) would decide which decrypts were significant or interesting enough to distribute
- they would be collected, locked into a briefcase, and turned over to a relatively junior officer (not always cleared to actually see the decrypts!) who would 'make the rounds' to the appropriate offices.
- no copies of any decrypts were left with anyone on the list. The recipient would be allowed to read the translated decrypt, in the presence of the distributing officer, and was required to return it immediately upon finishing. Before the beginning of the second week in December 1941, that was the last time anyone on the list saw that particular decrypt.
Decryption process
There were several prior steps needed before any decrypt was ready for distribution:
- Interception. The Japanese Foreign Office used both wireless transmission and cables to communicate with its off shore units. Wireless transmission was intercepted (if possible) and any of several listening stations (Hawaii, Guam, Bainbridge Island in Washington, etc.) and the raw cypher groups forwarded to Washington. Eventually, there were decryption stations (including a copy of the Army's PURPLE machine) in the Philippines as well. Cable traffic was (for many years before late 1941) collected at cable company offices by a military officer who made copies and started them to Washington. Cable traffic in Hawaii was not intercepted due to legal concerns until David Sarnoff of RCA agreed to allow it during a visit to Hawaii the first week of December 1941. At one point, intercepts were being mailed to (Army or Navy) Intelligence from the field.
- Deciphering. The raw intercept was deciphered by either the Army or the Navy (depending on the day). Deciphering was usually successful as the cypher had been broken.
- Translation. Having obtained the plain text, in Latin letters, it was translated. Because the Navy had more Japanese-speaking officers, much of the burden of translation fell onto the Navy. And because Japanese is a difficult language, with meaning highly dependent upon context, effective translation required not only fluent Japanese, but considerable knowledge of the context within which the message was sent.
- Evaluation. The translated decrypt had to be evaluated for its intelligence content. For example, is the ostensible content of the message meaningful? If it is, for instance, part of a power contest within the Foreign Office or some other part of the Japanese government, its meaning and implications would be quite different from a simple informational or instructional message to an Embassy. Or, might it be another message in a series whose meaning, taken together, is more than the meaning of any individual message. Thus, the fourteenth message to an Embassy instructing that Embassy to instruct Japanese merchant ships calling at that country to return to home waters before, say, the end of November would be more significant than a single such message meant for a single ship or port. Only after having evaluated a translated decrypt for its intelligence value could anyone decide whether it deserved to be distributed.
In the period before the attack on Pearl Harbor, the material was handled awkwardly and inefficiently, and was distributed even more awkwardly. Nevertheless, the extraordinary experience of reading a foreign government's most closely held communications, sometimes even before the intended recipient, was astonishing. It was so astonishing, someone (possibly President Roosevelt) called it magic. The name stuck.
Magic and United States Executive Order 9066
One aspect of Magic remains controversial to this day - the amount of involvement the intercepts played in the issuing of United States Executive Order 9066 on February 19, 1942, and subsequent Executive Order 9102 on March 18, 1942, which lead to the creation of the Wartime Relocation Authority (WRA). This is often confused with the issue of internment, which was actually handled by the Justice Department's Immigration and Naturalization Service (INS) and affected all citizens of countries at war with the United States in any location.
Internment of 'enemy aliens' by the U.S. government began two months prior to Executive Order 9066 on December 8, 1941, immediately after the attack at Pearl Harbor and included Germans, Italians, and Hungarians, not just the Japanese living on the U.S. West Coast. [see Stephen Fox's "The Unknown Internment: An Oral History of the Relocation of Italian Americans during World War II"]
David Lowman in his book MAGIC: the Untold Story reports that the primary justification for the Japanese-American relocations and internments was to protect against espionage and sabotage, because Magic could not be mentioned during the war. Those defending the relocation and evacuation programs, most notably author Michelle Malkin, point to Magic intercepts as being justification for EO 9066. He argues that those intercepts discuss the development of a spy ring among Japanese Americans by the Japanese consulates, showing that the Japanese American community was an espionage risk.
The following is the actual text of several Magic intercepts translated into English before and during the war and declassified and made public in 1978 by the U.S. government (The Magic Background of Pearl Harbor:, Government Printing Office, 8 volumes)
Magic intercepts Tokyo to Washington
Magic intercept Tokyo to Washington #44 - Jan 30, 1941
Intercept dated January 30, 1941 and noted as translated 2-7-41
Numbered #44
FROM: Tokyo (Matsuoka)
TO: Washington (Koshi)
(In two parts--complete).
(Foreign Office secret).
(1) Establish an intelligence organ in the Embassy which will maintain liaison with private and semi-official intelligence organs (see my message to Washington #591 and #732 from New York to Tokyo, both of last year's series).
With regard to this, we are holding discussions with the various circles involved at the present time.
(2) The focal point of our investigations shall be the determination of the total strength of the U.S. Our investigations shall be divided into three general classifications: political, economic, and military, and definite course of action shall be mapped out.
(3) Make a survey of all persons or organizations which either openly or secretly oppose participation in the war.
(4) Make investigations of all antisemitism, communism, movements of Negroes, and labor movements.
(5) Utilization of U.S. citizens of foreign extraction (other than Japanese), aliens (other than Japanese), communists, Negroes, labor union members, and anti-Semites, in carrying out the investigations described in the preceding paragraph would undoubtedly bear the best results.
These men, moreover, should have access to governmental establishments, (laboratories?), governmental organizations of various characters, factories, and transportation facilities.
(6) Utilization of our "Second Generations" and our resident nationals. (In view of the fact that if there is any slip in this phase, our people in the U.S. will be subjected to considerable persecution, and the utmost caution must be exercised).
(7) In the event of U.S. participation in the war, our intelligence set-up will be moved to Mexico, making that country the nerve center of our intelligence net. Therefore, will you bear this in mind and in anticipation of such an eventuality, set up facilities for a U.S.-Mexico international intelligence route. This net which will cover Brazil, Argentina, Chile, and Peru will also be centered in Mexico.
(8) We shall cooperate with the German and Italian intelligence organs in the U.S. This phase has been discussed with the Germans and Italians in Tokyo, and it has been approved.
Please get the details from Secretary Terasaki upon his assuming his duties there.
Please send copies to those offices which were on the distribution list of No. 43.
Magic intercepts from Japanese U.S. consulates to Tokyo
Throughout the rest of 1941, some of the messages between Tokyo and its embassies and consulates continued to be intercepted.
In response to the ordered shift from propaganda efforts to espionage collection, the Japanese consulates throughout the western hemisphere reported their information normally through the use of diplomatic channels, but when time-sensitive through the use of MAGIC encoded messages. This provided vital clues to their progress directly to the U.S. President and his top advisers.
Intercepts in May 1941 from the consulates in Los Angeles and Seattle report that the Japanese were having success in obtaining information and cooperation from "second generation" Japanese Americans and others.
Magic intercept LA to Tokyo #067 - May 9, 1941
Intercept dated May 9, 1941 and translated 5-19-41
Numbered #067
FROM: Los Angeles (Nakauchi)
TO: Tokyo (Gaimudaijin)
(In 2 parts--complete).
Strictly Secret.
Re your message # 180 to Washington.
We are doing everything in our power to establish outside contacts in connection with our efforts to gather intelligence material. In this regard, we have decided to make use of white persons and Negroes, through Japanese persons whom we can't trust completely. (It not only would be very difficult to hire U.S. (military?) experts for this work at present time, but the expenses would be exceedingly high.) We shall, furthermore, maintain close connections with the Japanese Association, the Chamber of Commerce, and the newspapers.
With regard to airplane manufacturing plants and other military establishments in other parts, we plan to establish very close relations with various organizations and in strict secrecy have them keep these military establishments under close surveillance. Through such means, we hope to be able to obtain accurate and detailed intelligence reports. We have already established contacts with absolutely reliable Japanese in the San Pedro and San Diego area, who will keep a close watch on all shipments of airplanes and other war materials, and report the amounts and destinations of such shipments. The same steps have been taken with regards to traffic across the U.S.-Mexico border.
We shall maintain connection with our second generations who are at present in the (U.S.) Army, to keep us informed of various developments in the Army. We also have connections with our second generations working in airplane plants for intelligence purposes.
With regard to the Navy, we are cooperating with our Naval Attache's office, and are submitting reports as accurately and speedily as possible.
We are having Nakazawa investigate and summarize information gathered through first hand and newspaper reports, with regard to military movements, labor disputes, communistic activities and other similar matters. With regard to anti-Jewish movements, we are having investigations made by both prominent Americans and Japanese who are connected with the movie industry which is centered in this area. We have already established connections with very influential Negroes to keep us informed with regard to the Negro movement.
Magic intercept Seattle to Tokyo #45 - May 11, 1941
Intercept dated May 11, 1941 and translated 6-9-41
Numbered # 45
FROM: Seattle (Sato)
TO: Tokyo
(3 parts--complete)
Re your # 180 to Washington
1. Political Contacts
We are collecting intelligences revolving around political questions, and also the questions of American participation in the war which has to do with the whole country and this local area.
2. Economic Contacts
We are using foreign company employees, as well as employees in our own companies here, for the collection of intelligence having to do with economics along the lines of the construction of ships, the number of airplanes produced and their various types, the production of copper, zinc and aluminum, the yield of tin for cans, and lumber. We are now exerting our best efforts toward the acquisition of such intelligences through competent Americans. From an American, whom we contacted recently, we have received a private report on machinists of German origin who are Communists and members of the labor organizations in the Bremerton Naval Yard and Boeing airplane factory. Second generation Japanese ----- ----- ----- [three words missing].
3. Military Contacts
We are securing intelligences concerning the concentration of warships within the Bremerton Naval Yard, information with regard to mercantile shipping and airplane manufacturing, movements of military forces, as well as that which concerns troop maneuvers.
With this as a basis, men are sent out into the field who will contact Lt. Comdr. OKADA, and such intelligences will be wired to you in accordance with past practice. KANEKO is in charge of this. Recently we have on two occasions made investigations on the spot of various military establishments and concentration points in various areas. For the future we have made arrangements to collect intelligences from second generation Japanese draftees on matters dealing with the troops, as well as troop speech and behavior. ----- ---- -----. [three words missing]
4. Contacts with Labor Unions
The local labor unions A.F. of L. and C.I.O. have considerable influence. The (Socialist?) Party maintains an office here (its political sphere of influence extends over twelve zones.) The C.I.O., especially, has been very active here. We have had a first generation Japanese, who is a member of the labor movement and a committee chairman, contact the organizer, and we have received a report, though it is but a resume, on the use of American members of the (Socialist ?) Party. ------ OKAMARU is in charge of this.
5. In order to contact Americans of foreign extraction and foreigners, in addition to third parties, for the collection of intelligences with regard to anti-participation organizations and the anti-Jewish movement, we are making use of a second generation Japanese lawyer.
This intelligence ---- ----- -----.
Magic intel restricted to just a handful of Roosevelt's cabinet
These intercepts plus other reports from the FBI and the Office of Naval Intelligence counter-espionage efforts, the infamous TACHIBANA espionage case during the summer of 1941, FBI organized crime efforts against Japanese Yakuza throughout the 1930s all along the west coast (the TOKOYO and TOYO CLUBs) were all available only to the most senior leadership of the Roosevelt cabinet. Of note - even J.Edgar Hoover, Director of the FBI, was not privy to the existence of Magic intelligence.
Opposing viewpoint
Those who consider that Executive Order 9066 regarding Japanese American internment was not based on Magic intercepts, argue that:
- the commanding officer on the West coast, Lt. Gen. J. L. DeWitt, was not on the Magic intercept list,
- his superior, Secretary of War Henry Stimson, was on the intercept list, and
- Stimson requested justification for the relocation program from DeWitt.
- If Magic intercepts provided justification, why ask DeWitt for further justification?
One theory is that Stimson wanted DeWitt to provide justifications that could be made public, because the Magic intercepts could not be made public.
The issue has been inflamed recently due to the release of Malkin's recent book, In Defense of Internment, in which the Magic intercepts play a major role in the defense of her thesis.
Other Japanese ciphers
PURPLE was an enticing, but quite tactically limited, window into Japanese planning and policy because of the peculiar nature of Japanese policy making prior to the War (see above). Early on, a better tactical window was the Japanese Fleet Code (an encoded cypher actually), called JN-25 by U.S. Navy cryptanalysts. Breaking into the version in use in the months after December 7, 1941 provided enough information to lead to U.S. naval victories in the battles of the Coral Sea and Midway, stopping the initial Japanese advances to the south and eliminating the bulk of Japanese naval.air power. Later, broken JN-25 traffic also provided the schedule and routing of the plane Admiral Yamamoto Isoroku would be flying in during an inspection tour in the southwest Pacific, giving U.S. Army Air Corps pilots a chance to ambush the officer who had conceived the Pearl Harbor attack. And still later, access to Japanese Army messages from decrypts of Army communications traffic assisted in planning the island hopping campaign to the Philippines and beyond.
Public notice had actually been served Japanese cryptography was dangerously inadequate by the Chicago Tribune, which published a series of stories just after Midway in 1942 claiming, correctly, victory was due in large part to U.S. breaks into Japanese crypto systems (in this case, the JN-25 cypher, though which system(s) had been broken was not mentioned in the newspaper stories). However, neither the Japanese nor anyone who might have told them seem to have noticed either the Tribune coverage, or the stories based on the Tribune account published in other U.S. papers. Nor did they notice announcements made on the floor of the United States Congress to the same effect. There were no changes in Japanese cryptography connected with those newspaper accounts or Congressional disclosures.
Other claimed breaks into PURPLE
The book Sword and Shield, by Christopher Andrew, based on the Mitrokhin Archive smuggled out of Russia in the early '90s by a KGB archivist, says that the Soviets independently broke into Japanese PURPLE traffic (as well as the Red predecessor machine), and that decrypted PURPLE messages contributed to the decision by Stalin to move troops from Far Eastern Asia to the area around Moscow for the counterattack against Germany in December 1941 as the messages convinced the Soviet government that there would not be a Japanese attack.
Fictional treatment
Neal Stephenson's novel Cryptonomicon includes a fictionalized version of Magic, with the Japanese cryptosystem being named "Indigo" rather than "PURPLE".
The W.E.B. Griffin series The Corps is a fictionalized account of United States Navy and Marine Corps intelligence operations in the Pacific Theater during World War II. Many of the main characters in the novels, both fictional and historical, have access to and use intelligence from Magic.
See also
- Ultra
- Japanese naval codes
Further reading
|
| |
|
|
