I recently learned the Japanese Foreign Ministry adopted Bolton's Telegraph Code (1871) in 1880. It was about the time when major cities in Japan came to be connected by telegraphy.
I added this note to "Nonsecret Code: An Overview of Early Telegraph Codes" and "日本の電信暗号".
According to John McVey's website, a copy of the codebook is in the British Library.
21/07/2024
Three Unknown Chinese Codebooks (ca. 1905)
I came across a reference to three Chinese codebooks used around 1905: 「密電秘鑰」（Midian Miyao）、「中国密電簡明表」（Zhongguo Mi dian Jianming Biao）、「行軍電報」（Xingjun Dianbao）. Their format seems to be similar to that of known codebooks. I added a note on this in "中国の暗号：1871-1945" and its abridged version in English: "Chinese Cryptography: 1871-1945".
19/07/2024
Ciphertext-only Attack on Classical Ciphers by Using AI
Even Chat GPT-4 could not tell me about papers about codebreaking with AI, but once I came across one paper (see the previous post), googling with its title returns a number of relevant papers. (That is, my question did not include relevant key words.)
My interest is in a ciphertext-only attack on homophonic ciphers or syllabic ciphers, which are the most common ciphers in the early modern period. Although I have not found a work on neural network approach on these, two (also cited by Closa (2023) mentioned in the previous post) are interesting in dealing with ciphertext-only attack on the Caesar (shift) cipher, the Vigenere cipher, and (monoalphabetic) substitution cipher.
Focardi et al. (2018) claims to be the first to provide a ciphertext-only attack on substitution ciphers based on neural networks (Abstract). It assumes a weakness of the cipher is given and the neural network exploits it.
(i) In the case of the Caesar cipher (a.k.a. the shift cipher), the key is a single number (e.g., up to 26). The frequencies of symbols generally reflects the frequency of letters (e.g. of English-language text). Thus, a neural network is trained to predict a key from the frequencies. The trained neural network can "recognize" the key without actually trying shifts to see whether readable text is obtained.
(ii) For the Vigenere cipher, a key length (m) is assumed and the Caesar classifier is applied to subtexts composed of symbols taken from the ciphertext at distance m. This is like a conventional method by using brute force but employs the Caesar classifier from (i).
(iii) For general (monoalphabetic) substitution ciphers, an overall framework is similar to conventional hill climbing. Starting from a random key, a "goodness" value is computed. After changing the key a little bit, the "goodness" value is computed again. If it has improved, the change is kept; otherwise the change is discarded. By iteration, the key is improved step by step. Again, the authors' method does not need to actually try intermediate keys to see whether a plausible plaintext is obtained. Instead, a neural network is trained with 3-grams of both plaintexts and ciphertexts. Although they "have no guratantees that this will provide a good classifier which is able to tell 'how' similar is a text to a plaintext and, consequently, how good is a key, but experimental results have confirmed that the method is effective." It will be interesting to see whether this works for homophonic ciphers or syllable ciphers.
Ahmadzadeh et al. (2021) seems to achieve what I thought impossible: training with plaintext/ciphertext pairs (Table 4) allows deciphering a ciphertext with an unknown key. The decryption function is learned "regardless of the cipher complexity or key length" (IV D). While experiments were done with Caesar, Vigenere, and (monoalphabetic) substitution, the authors consider their approach "has the potential to crack modern ciphers" (IV H).
To learn the decryption function from the plaintext/ciphertext pairs, they used an attention-based LSTM encoder-decoder model (Fig. 3). To quickly recall the terminology, a class of neural networks that can have "memory" is a recurrent neural network (RNN). A class of RNN that solves its problem (vanishing gradients in deep networks: "A small gradient value does not contribute very much to learning") is LSTM. A problem with LSTM ("a lengthy input sequence causes LSTM to forget important information along the sequence") is solved by an attention mechanism, which allows "dynamically highlighting important features of the input data" (III A).
As a prerequisite, they assume the ciphertext has "punctuation" and can be readily parsed into words (Fig. 4). It will be interesting to see whether their work can be generalized to ciphertext without punctuation.
References:
Riccardo Focardi and Flaminia L. Luccio (2018), "Neural Cryptanalysis of Classical Ciphers", Italian Conference on Theoretical Computer Science (ICTCS 2018), Urbino, Italy, September 18-20, 2018 (CEUR-WS.org).
Ezat Ahmadzadeh, Hyunil Kim, Ongee Jeong, and Inkyu Moon (2021), "A Novel Dynamic Attack on Classical Ciphers Using an Attention-Based LSTM Encoder-Decoder Model", IEEE Access, 2021, vol.9, pp.60960-60970, DOI: 10.1109,ACCESS.2021.3074268 (IEEE Xplore)
My interest is in a ciphertext-only attack on homophonic ciphers or syllabic ciphers, which are the most common ciphers in the early modern period. Although I have not found a work on neural network approach on these, two (also cited by Closa (2023) mentioned in the previous post) are interesting in dealing with ciphertext-only attack on the Caesar (shift) cipher, the Vigenere cipher, and (monoalphabetic) substitution cipher.
Focardi et al. (2018) claims to be the first to provide a ciphertext-only attack on substitution ciphers based on neural networks (Abstract). It assumes a weakness of the cipher is given and the neural network exploits it.
(i) In the case of the Caesar cipher (a.k.a. the shift cipher), the key is a single number (e.g., up to 26). The frequencies of symbols generally reflects the frequency of letters (e.g. of English-language text). Thus, a neural network is trained to predict a key from the frequencies. The trained neural network can "recognize" the key without actually trying shifts to see whether readable text is obtained.
(ii) For the Vigenere cipher, a key length (m) is assumed and the Caesar classifier is applied to subtexts composed of symbols taken from the ciphertext at distance m. This is like a conventional method by using brute force but employs the Caesar classifier from (i).
(iii) For general (monoalphabetic) substitution ciphers, an overall framework is similar to conventional hill climbing. Starting from a random key, a "goodness" value is computed. After changing the key a little bit, the "goodness" value is computed again. If it has improved, the change is kept; otherwise the change is discarded. By iteration, the key is improved step by step. Again, the authors' method does not need to actually try intermediate keys to see whether a plausible plaintext is obtained. Instead, a neural network is trained with 3-grams of both plaintexts and ciphertexts. Although they "have no guratantees that this will provide a good classifier which is able to tell 'how' similar is a text to a plaintext and, consequently, how good is a key, but experimental results have confirmed that the method is effective." It will be interesting to see whether this works for homophonic ciphers or syllable ciphers.
Ahmadzadeh et al. (2021) seems to achieve what I thought impossible: training with plaintext/ciphertext pairs (Table 4) allows deciphering a ciphertext with an unknown key. The decryption function is learned "regardless of the cipher complexity or key length" (IV D). While experiments were done with Caesar, Vigenere, and (monoalphabetic) substitution, the authors consider their approach "has the potential to crack modern ciphers" (IV H).
To learn the decryption function from the plaintext/ciphertext pairs, they used an attention-based LSTM encoder-decoder model (Fig. 3). To quickly recall the terminology, a class of neural networks that can have "memory" is a recurrent neural network (RNN). A class of RNN that solves its problem (vanishing gradients in deep networks: "A small gradient value does not contribute very much to learning") is LSTM. A problem with LSTM ("a lengthy input sequence causes LSTM to forget important information along the sequence") is solved by an attention mechanism, which allows "dynamically highlighting important features of the input data" (III A).
As a prerequisite, they assume the ciphertext has "punctuation" and can be readily parsed into words (Fig. 4). It will be interesting to see whether their work can be generalized to ciphertext without punctuation.
References:
Riccardo Focardi and Flaminia L. Luccio (2018), "Neural Cryptanalysis of Classical Ciphers", Italian Conference on Theoretical Computer Science (ICTCS 2018), Urbino, Italy, September 18-20, 2018 (CEUR-WS.org).
Ezat Ahmadzadeh, Hyunil Kim, Ongee Jeong, and Inkyu Moon (2021), "A Novel Dynamic Attack on Classical Ciphers Using an Attention-Based LSTM Encoder-Decoder Model", IEEE Access, 2021, vol.9, pp.60960-60970, DOI: 10.1109,ACCESS.2021.3074268 (IEEE Xplore)
13/07/2024
Codebreaking with AI
In 2016, the public was made aware of how AI (artificial intelligence) dramatically improved the quality of machine translation (Google's release in Japanese). It naturally raises a question: is codebreaking possible with AI?
For a long time, I couldn't find a positive answer to this question. Although my search was half-hearted, even Copilot with Chat-GPT-4 did not give a relevant answer to my question: "Are there papers about codebreaking by using AI?" Then, I noticed a poster abstract in HistoCrypt 2024: Oriol Closa, "Polyalphabetic cipher decryption function learning with LSTM networks", which seems to be based on her master thesis, Closa Oriol (2023), "LSTM-attack on polyalphabetic cyphers with known plaintext: Case study on the Hagelin C-38 and Siemens and Halske T52" (KTH). It teaches that "the application of Machine Learning to extract key information from intercepts is not a well researched area yet." (Abstract) and there are even "many authoritative opinions within the field" against utility of machine learning in classical cryptography (p.57).
Machine "learns" by finding a best parameter set for a model, which is like a very complex filter that receives an input and produces an output. In an example of machine translation, the input is a sequence of words in English and the output is a sequence of words in Japanese. In order to train a machine in this example, bilingual corpus of corresponding texts in the two languages is fed to the computer, whereby the computer learns an English-Japanese translation model. Given a new text in English, the trained computer can apply the model (filter) on the input to produce a text in Japanese as an output.
By analogy, given a corpus of ciphertext/plaintext pairs, AI may learn to decipher a new ciphertext into a plaintext. However, it should be limited to the case where the new ciphertext is based on the same cipher key used for training -- that's what I thought. But the thesis taught me machine learning can do more than that. (The key is included in the training data (p.39).)
Remember that the output of a trained model need not be similar to the input. For example, when the input is a text in English, the output may be some classification or labelling of the text rather than a text in another language. In Oriol (2023), in my understanding, the input is ciphertext, a crib (known plaintext, presumably corresponding to the ciphertext), and a null key (a placeholder in the input vector), and the output is plaintext (which should match the crib and is included only for analysis) and the extracted key (p.34). Thus, this seems to receive a maching ciphertext/plaintext pair to produce its key ("extract the external key given a combination of plaintext and ciphertext without the use of the internal setting" p.51).
The thesis deals with four ciphers: Vigenere, Playfair, Hagelin C-38, and Siemens and Halske T52 with LSTM networks (a kind of neural network). Main differences among them are the decipher function reflecting the cipher scheme (I guess this means the cipher algorithm is known and only the key is to be found out), the crib length (e.g., 15, 25), and the size of the hidden layer (e.g., 256, 2048) (p.33, 40). The author says LSTM networks can extract key information given a crib (in my understanding, this is a matching plaintext/ciphertext pair) for Vigenere, Hagelin C-38, and Siemens and Halske T52, but not Playfair.
(16 July 2024) Ajeet Singh, Kaushik Bhargav Sivangi, and Appala Naidu Tentu (2024), "Machine Learning and Cryptanalysis: An In-Depth Exploration of Current Practices and Future Potential", Journal of Computing Theories and Applications (JCTA, DOI: https://doi.org/10.62411/jcta.9851), Vol. 1 No. 3 (2024) also says "the integration of machine learning, and specifically deep learning, into cryptanalysis has been relatively unexplored."
For a long time, I couldn't find a positive answer to this question. Although my search was half-hearted, even Copilot with Chat-GPT-4 did not give a relevant answer to my question: "Are there papers about codebreaking by using AI?" Then, I noticed a poster abstract in HistoCrypt 2024: Oriol Closa, "Polyalphabetic cipher decryption function learning with LSTM networks", which seems to be based on her master thesis, Closa Oriol (2023), "LSTM-attack on polyalphabetic cyphers with known plaintext: Case study on the Hagelin C-38 and Siemens and Halske T52" (KTH). It teaches that "the application of Machine Learning to extract key information from intercepts is not a well researched area yet." (Abstract) and there are even "many authoritative opinions within the field" against utility of machine learning in classical cryptography (p.57).
Machine "learns" by finding a best parameter set for a model, which is like a very complex filter that receives an input and produces an output. In an example of machine translation, the input is a sequence of words in English and the output is a sequence of words in Japanese. In order to train a machine in this example, bilingual corpus of corresponding texts in the two languages is fed to the computer, whereby the computer learns an English-Japanese translation model. Given a new text in English, the trained computer can apply the model (filter) on the input to produce a text in Japanese as an output.
By analogy, given a corpus of ciphertext/plaintext pairs, AI may learn to decipher a new ciphertext into a plaintext. However, it should be limited to the case where the new ciphertext is based on the same cipher key used for training -- that's what I thought. But the thesis taught me machine learning can do more than that. (The key is included in the training data (p.39).)
Remember that the output of a trained model need not be similar to the input. For example, when the input is a text in English, the output may be some classification or labelling of the text rather than a text in another language. In Oriol (2023), in my understanding, the input is ciphertext, a crib (known plaintext, presumably corresponding to the ciphertext), and a null key (a placeholder in the input vector), and the output is plaintext (which should match the crib and is included only for analysis) and the extracted key (p.34). Thus, this seems to receive a maching ciphertext/plaintext pair to produce its key ("extract the external key given a combination of plaintext and ciphertext without the use of the internal setting" p.51).
The thesis deals with four ciphers: Vigenere, Playfair, Hagelin C-38, and Siemens and Halske T52 with LSTM networks (a kind of neural network). Main differences among them are the decipher function reflecting the cipher scheme (I guess this means the cipher algorithm is known and only the key is to be found out), the crib length (e.g., 15, 25), and the size of the hidden layer (e.g., 256, 2048) (p.33, 40). The author says LSTM networks can extract key information given a crib (in my understanding, this is a matching plaintext/ciphertext pair) for Vigenere, Hagelin C-38, and Siemens and Halske T52, but not Playfair.
(16 July 2024) Ajeet Singh, Kaushik Bhargav Sivangi, and Appala Naidu Tentu (2024), "Machine Learning and Cryptanalysis: An In-Depth Exploration of Current Practices and Future Potential", Journal of Computing Theories and Applications (JCTA, DOI: https://doi.org/10.62411/jcta.9851), Vol. 1 No. 3 (2024) also says "the integration of machine learning, and specifically deep learning, into cryptanalysis has been relatively unexplored."
12/07/2024
George Lasry's Paper on Syllabic Cipher Released
Last year, I reported George Lasry's solution of syllabic ciphers. Syllabic ciphers are ciphers with many symbols for syllables as well as those for single letters.
Details of his algorithm are now published in George Lasry, "Deciphering Historical Syllabic Ciphers" (HistoCrypt 2024).
One might be tempted to say extension of a homophonic solver to a syllabic solver only involves enlarging the search space for, say, 25 letters of the alphabet by adding variables for 25*25=625 syllables. However, since last year, I've been wondering how to design a scoring function for intermediate decipherments. In typical homophonic solvers, 4-grams or 5-grams are used in computing a score. For example, if an intermediate decipherment abounds in plausible 5-grams (e.g., "ISION", "EMENT", "ETLES" for the French language), it receives a high score. But when symbols represent syllables such as "SI", "ON", "EM" "EN", just random assignment of these syllables might result in a relatively high score.
The answer to my year-long question turned out to be very simple. The scoring function is based on 4-grams, composed of not only single letters but also syllables. Thus, a 4-gram may be "E-S-T-A" or "CO-N-TRO-L" (p.177).
While it is easy to say this, it is quite another to implement it, because this involves re-constructing the whole language model. A language model represents frequency characteristics of all possible 4-grams (or 5-grams etc.) in a large corpus of text in the language in question. For a language model for conventional homophonic solvers, creating a language model amounted to simply counting letter groups. For a syllabic solver, it is necessary to first break text in the corpus into syllables, but there can be many ways to split a word. For example, the word ESTABLISHED may be broken into E-S-TA-B-LI-S-HE-D (using only CV syllables), ES-TA-B-LI-S-H-ED (using CV and VC), E-STA-BLI-S-HE-D (allowing also CCV), or the like (p.174). Naturally, success of the algorithm depends on the choice of the set of syllables used and the decomposition scheme, which required "extensive trial-and-error to fine tune" (p.176, 179).
And of course, the more than ten-fold increase in the number of variables (which more than exponentially expands the search space) means "extensive computing power" is required. George used a 64-core Windows 10 Pro PC with 256 Gbytes of RAM memory (p.176), whereas one commercial PC I see on the web has 4-core and up to 3.40 GB of memory.
His groundbreaking new scheme has proven its merits by breaking ten ciphertexts (including five with known keys).
Details of his algorithm are now published in George Lasry, "Deciphering Historical Syllabic Ciphers" (HistoCrypt 2024).
One might be tempted to say extension of a homophonic solver to a syllabic solver only involves enlarging the search space for, say, 25 letters of the alphabet by adding variables for 25*25=625 syllables. However, since last year, I've been wondering how to design a scoring function for intermediate decipherments. In typical homophonic solvers, 4-grams or 5-grams are used in computing a score. For example, if an intermediate decipherment abounds in plausible 5-grams (e.g., "ISION", "EMENT", "ETLES" for the French language), it receives a high score. But when symbols represent syllables such as "SI", "ON", "EM" "EN", just random assignment of these syllables might result in a relatively high score.
The answer to my year-long question turned out to be very simple. The scoring function is based on 4-grams, composed of not only single letters but also syllables. Thus, a 4-gram may be "E-S-T-A" or "CO-N-TRO-L" (p.177).
While it is easy to say this, it is quite another to implement it, because this involves re-constructing the whole language model. A language model represents frequency characteristics of all possible 4-grams (or 5-grams etc.) in a large corpus of text in the language in question. For a language model for conventional homophonic solvers, creating a language model amounted to simply counting letter groups. For a syllabic solver, it is necessary to first break text in the corpus into syllables, but there can be many ways to split a word. For example, the word ESTABLISHED may be broken into E-S-TA-B-LI-S-HE-D (using only CV syllables), ES-TA-B-LI-S-H-ED (using CV and VC), E-STA-BLI-S-HE-D (allowing also CCV), or the like (p.174). Naturally, success of the algorithm depends on the choice of the set of syllables used and the decomposition scheme, which required "extensive trial-and-error to fine tune" (p.176, 179).
And of course, the more than ten-fold increase in the number of variables (which more than exponentially expands the search space) means "extensive computing power" is required. George used a 64-core Windows 10 Pro PC with 256 Gbytes of RAM memory (p.176), whereas one commercial PC I see on the web has 4-core and up to 3.40 GB of memory.
His groundbreaking new scheme has proven its merits by breaking ten ciphertexts (including five with known keys).
04/07/2024
Reconstructed Ciphers related to Mary, Queen of Scots, preserved in Scottish Catholic Archives (SCA)
Fourteen ciphers used in the correspondence of James Beaton, Archbishop of Glasgow, and reconstructed by Bishop Kyle in the nineteenth century are preserved in the Scottish Catholic Archives (SCA). I now added mention of these in "Ciphers of Mary, Queen of Scots".
03/07/2024
Early French Figure Ciphers
An early specimen of a French figure cipher (digit cipher, numerical cipher) is discussed in Camille Desenclos and George Lasry, "An early French digit cipher: deciphering a letter from the King of France to the Duke of Nevers (1592)" (HistoCrypt 2024).
It is used in a letter from Henry IV to Duke of Nevers, 12 September 1592 (BnF fr.3620, f.70-71)(p.47). The cipher employs variable-length figures written continuously, but the authors could parse the ciphertext into cipher symbols by assuming three-digit figures always start with "1" (p.48; see also my articles from 2017-2019 and 2018-2019). After the key was reconstructed, the authors found the original cipher table among Nevers' collection of keys in BnF fr.3995, fol.140 (p.49).
Many cipher letters between Henry IV and the Duke of Nevers have been known but they used conventional symbol ciphers in BnF fr.3995, fol.67 rather than numerical ciphers (p.49-50, 46; see also my article). The authors point out that the 1592 letter in question is countersigned by Martin Ruzé de Beaulieu, while all the other letters in cipher from the king to the duke from 1591 to 1594 are counstersigned by Louis Potier de Gesvres (p.49-50). The 1592-cipher was also used in letters in 1591 to Henry IV (two from Duke of Biron, one from sieur de Guitry, baron de Salagnac, and marquis de Pisani) (p.51).
The papers is also valuable in citing many examples of exclusively digit ciphers in the 1580s/1590s (p.50; see also other figure ciphers mentioned in my articles: Henry III etc., Buzenval).
(5 July 2024) I now remembered another French ciphertext in variable-length figures (1, 2, or 3 digits) written continuously (ca.1620?) is discussed in here.
It is used in a letter from Henry IV to Duke of Nevers, 12 September 1592 (BnF fr.3620, f.70-71)(p.47). The cipher employs variable-length figures written continuously, but the authors could parse the ciphertext into cipher symbols by assuming three-digit figures always start with "1" (p.48; see also my articles from 2017-2019 and 2018-2019). After the key was reconstructed, the authors found the original cipher table among Nevers' collection of keys in BnF fr.3995, fol.140 (p.49).
Many cipher letters between Henry IV and the Duke of Nevers have been known but they used conventional symbol ciphers in BnF fr.3995, fol.67 rather than numerical ciphers (p.49-50, 46; see also my article). The authors point out that the 1592 letter in question is countersigned by Martin Ruzé de Beaulieu, while all the other letters in cipher from the king to the duke from 1591 to 1594 are counstersigned by Louis Potier de Gesvres (p.49-50). The 1592-cipher was also used in letters in 1591 to Henry IV (two from Duke of Biron, one from sieur de Guitry, baron de Salagnac, and marquis de Pisani) (p.51).
The papers is also valuable in citing many examples of exclusively digit ciphers in the 1580s/1590s (p.50; see also other figure ciphers mentioned in my articles: Henry III etc., Buzenval).
(5 July 2024) I now remembered another French ciphertext in variable-length figures (1, 2, or 3 digits) written continuously (ca.1620?) is discussed in here.
01/07/2024
Cross-Cipher Errors - A New Modality of Communication Analysis
Errors in ciphertext may be caused by another cipher concurrently in use. The phenomenon, called cross-cipher errors, is discussed in our new coauthored paper:
Norbert Biermann, Satoshi Tomokiyo, and George Lasry, "What Encryption Errors Can Reveal: Cross-Cipher Errors in Mary Queen of Scots' Letters" (HistoCrypt2024).
When a ciphertext includes a significant amount of systematic errors, it may be worthwhile to look for a cipher causing the errors. It can reveal that correspondence in that other cipher was going on at the time, even if no letters are extant.
23/06/2024
Some Updates on Correspondence between Philip II and Vargas Mexia
Back in 2020, I pointed out there are undeciphered letters from Philip II to Juan de Vargas Mexia, his ambassador in France, and identified four ciphers used therein (see "Finding the Keys to Philip II's Cipher Letters to Juan de Vargas Mexia").
While re-reading Leader (1880), Mary Queen of Scots in captivity some weeks ago, I noticed Vargas Mexia's letters are printed in Teulet (1862). Now I could have the time to check it and confirmed the letters in BnF es. 132 are not printed in Teulet. (My comparison was based on the dates and correspondents. Although I found two letters with the matching dates and correspondents, the contents do not seem to match.)
I also found further undeciphered letters from Vargas Mexia are catalogued in BL Add MS 28421.
I added these under the section "(Additional Note, 23 June 2024)."
I also found further undeciphered letters from Vargas Mexia are catalogued in BL Add MS 28421.
I added these under the section "(Additional Note, 23 June 2024)."
09/05/2024
Early Japanese Syllabary Table in Milanese Archives
Vigenère's Traicté des chiffre printed a Japanese syllabary in the addenda.
A similar printed Japanese syllabary table is found in Milanese archives. Although it is filed with ciphers from the second half of the fifteenth century, I think it is contemporary with Vigenere. (If it were from the period of the Sforzas, it would be a sensational discovery, because standard history teaches that Japan had no direct contact with the western world until the 1540s.) I now added a section "Kana Syllabary in Milanese Archives" in "Vigenere's Introduction of Japanese Characters in Europe".
25/04/2024
Korean Telegraphic Code
Hangul is an artificial alphabet created in the fifteenth century as a script for Korean. It is interesting because graphic units for consonants and vowels are combined horizontally or vertically to form syllable symbols. Because of this system, I think substitution cipher is not possible in Hangul script.
Once characters are encoded into digits or roman letters, encryption methods including substitution and transposition are applicable. Today's computer can of course handle Hangul characters. But in the early years of telegraphy, telegrams in Korea had to be in Chinese characters or Latin letters.
So, a telegraph codebook for Chinese characters were used in Korea.
I have seen a Korean version (『漢電』) of a Chinese telegraph codebook.
Even after WWII, it appears a telegraph codebook with similar content was used, in view of an edition adapted for use by those who could not read Chinese characters (Korean Telegraphic Code Book, with characters arranged by sounds in English alphabetic order according to the McCune-Reischauer system of transliteration).
These are already covered in 電碼――中国の文字コード, dealing with Chinese telegraph codes, in which I now made small corrections.
(By the way, I wrote the above ten days ago, but I couldn't upload it because my smartphone failed and I couldn't pass the two-factor authentication for logging into the blog.)
Once characters are encoded into digits or roman letters, encryption methods including substitution and transposition are applicable. Today's computer can of course handle Hangul characters. But in the early years of telegraphy, telegrams in Korea had to be in Chinese characters or Latin letters.
So, a telegraph codebook for Chinese characters were used in Korea.
I have seen a Korean version (『漢電』) of a Chinese telegraph codebook.
Even after WWII, it appears a telegraph codebook with similar content was used, in view of an edition adapted for use by those who could not read Chinese characters (Korean Telegraphic Code Book, with characters arranged by sounds in English alphabetic order according to the McCune-Reischauer system of transliteration).
These are already covered in 電碼――中国の文字コード, dealing with Chinese telegraph codes, in which I now made small corrections.
(By the way, I wrote the above ten days ago, but I couldn't upload it because my smartphone failed and I couldn't pass the two-factor authentication for logging into the blog.)
31/03/2024
William Blencowe's "Safest and Most Expeditious" Cipher
I uploaded a new article "William Blencowe's "Safest and Most Expeditious" Cipher". Blencowe was a grandson of the celebrated mathematician and codebreaker, John Wallis.
The example ciphertext consists of an interesting mix of single letters and bigrams:
The example ciphertext consists of an interesting mix of single letters and bigrams:
a.c.e.o.t.ds,th.al,th.an,th,es,an,my,be,m.n.t.o.nd,jo.H.sh,y,in.y.After all, however, this is essentially a columnar transposition cipher with some additional twists. Using bigrams may mislead the codebreaker at first, but once transposition is suspected, they would help codebreaking rather than prevent it.
A.w.n,th,e.de.in.pi.sa.I.A.g.su.r.to,ye,ho.e,th.c.wth,de.re,en.t.at,
r.il,ti.A.n.nb.te.sts,nd,ri.rm.ot,re,d,n.m.m.r.ar,nd,ar,ty,us.i.m.c
in,a.e,d,h,ro.to,y.ig.&,of,to,ou,a.b.d.g.f.p,no,n.is,d,a.p,k.
ye,b.th.th.of,u.m.n.hl.th.t.S.t.n.e.ye,xt,u.w.A.ho,en.o.w.Ja.I
th,l.F.pe,n.e,h,hi.rs,ar,ty,la.ki.a.r.in,e.fo.to.m,w.tw.th.nd,ng,Sr,
no.le,at,ey,th.gh,ar.ed,an.e.w.co.h.n,h,a.n.e.E.ey,pa.ou.d,an,is,as.
r.p.m,g.f.c,a.n.do,ll,b.m,d.g.b.d.m,p.q.b,s.r,d.c,a.n
28/03/2024
Can a Lost Encoding Format be Recovered by Analysis?
It's now a decade ago that I read in a newspaper article that data obtained on Mars by Viking spacecraft could not be read 25 years after the landing because the format was lost (The Asahi Shimbun, 20 January 2014).
The case is also mentioned in a report on long-term data preservation by a Japanese think tank, CRDS (CRDS-FY2012-WR-07).
The source seems to be a news release of the University of Southern California (Spaceflight Now):
I got interested in this news because I was wondering whether "codebreaking" is possible for media data encoded on, say, DVD without knowing the format. (The compromise of the encryption system (Wikipedia) of DVD assumes knowledge of the format, and is thus another matter.) Considering the sheer number of pages of format documentation, I think it is near impossible. But of course, relying on secrecy of the scheme is not a good idea for cryptographic security. Security should rest on the key being kept secret (Kerchoff's principle).
By the way, the Viking data recovered in the 1990s was used to claim finding evidence of an organism on Mars (Miller's site, Levin's site). But the result is not established (Wikipedia).
The source seems to be a news release of the University of Southern California (Spaceflight Now):
"The data were on magnetic tapes, and written in a format so old that the programmers who knew it had died," Miller said.After all, the issue is not about loss of a data encoding scheme but physical format of the magnetic tapes.
Eventually, NASA was able to recover the data from printouts, luckily preserved by Levin and Straat - and so, Miller was able to pore over the numbers.
I got interested in this news because I was wondering whether "codebreaking" is possible for media data encoded on, say, DVD without knowing the format. (The compromise of the encryption system (Wikipedia) of DVD assumes knowledge of the format, and is thus another matter.) Considering the sheer number of pages of format documentation, I think it is near impossible. But of course, relying on secrecy of the scheme is not a good idea for cryptographic security. Security should rest on the key being kept secret (Kerchoff's principle).
By the way, the Viking data recovered in the 1990s was used to claim finding evidence of an organism on Mars (Miller's site, Levin's site). But the result is not established (Wikipedia).
27/03/2024
Two More English Ciphers from the 1650s
I've been going through my notebook file these days, and uploaded some materials I left unfinished before. Now, I added two ciphers (Richard Browne, John Bramhall) in "King Charles II's Ciphers during Exile".
26/03/2024
J.F.W. Herschel's Cipher Puzzle
I've been interested in ciphers that allow multiple readings (a Venetian example).
So, I took a note when I read about a ciphertext that yields two readings, posted by tonybaloney at:
http://www.aerobushentertainment.com/crypto/index.php?topic=36.60
(now it seems the link has changed).
The ciphertext (A) and the two readings (B,C) are as follows:
(A)Xabnsly ngpwpdetlews tbbbtzl aobl stheingdnxmccvv
(B)Several philosophers observe that chloroplatinate
(C)Sing, Celestial Muse, the destroying
(A)hclzepsf xo qskxybbbbui
(B)solution on silverplate
(C)wrath of Achilles,
(A)Egtubatjkh fba lwipizix eqjbnasv nfvj yjcin
(B)reproduces the luminous spectrum with great
(C)Peleus’ son, what myriad woes it
(A)cjzvekzxy gf nbyr gzrefcwxianst
(B)vividness in blue fluorescences,
(C)heaped on the Grecians,
(A)Jxkivu v xcnukwcxpv ifnnszp't tpdvm
(B)whilst a coppersalt insolated might
(C)Many a valiant hero’s soul
(A)lqaauuqrauaqqvso up mfijtxyz.
(B)photographically be coloured.
(C)dismissing to Hades.
Now, I find this ciphertext was posted by Klaus Schmeh on his blog Cipherbrain back in 2016: Wer knackt die Verschlüsselung des Astronomen John Herschel?. The article shows the source: The Photographic News, 5 January 1866 and identifies the author of the ciphertext as J.F.W. Herschel (1792-1871) (Wikisource). The blog readers found out additional information.
The weekly magazine (Google) carried the puzzle on 5 January 1866 (p.5-6), the solution (B) provided by a reader the next week (p.23), and Herschel's correct solution (C) the next week thereafter (p.35). After all, the reading (B) was wrong, and this cipher is not about double reading.
The readers of Cipherbrain worked out the general principle of this cipher: schorsch pointed out the first two words can be explained by word-by-word Caesar cipher, Norbert demonstrated this works for every word, and Thomas found the shift, counted in reverse direction, is the number of letters in the word plus 1, 2, 3, 4, ....
In the following, (a) is the ciphertext, (b) is the number of letters in the (deciphered!) word plus 1, 2, 3, 4, ..., (c) is the result of reverse-shifting, and (d) is the plaintext word.
(a)Xabnsly ngpwpdetlews tbbbtzl aobl stheingdnxmccvv
(b)4+1=5 9+2=11 4+3=7 3+4=7 10+5=15
(c)Svwingt cvelestiatlh muuumse thue desptyroyixnngg
(d)Sing, Celestial Muse, the destroying
(a)hclzepsf xo qskxybbbbui
(b)5+6=11 2+7=9 8+8=16
(c)wraotehu of acuhilllles
(d)wrath of Achilles,
(a)Egtubatjkh fba lwipizix eqjbnasv nfvj yjcin
(b)6+9=15 3+10=13 4+11=15 6+12=18 4+13=17 2+14=16
(c)Prefmleuvs son whtatkti myrjviad woes itmsx
(d)Peleus' son, what myriad woes it
(a)cjzvekzxy gf nbyr gzrefcwxianst
(b)6+15=21 2+16=18 3+17=20 8+18=26=0
(c)hoeajpecd on thex gzrefcwxianst
(d)heaped on the Grecians,
(a)Jxkivu v xcnukwcxpv ifnnszp't tpdvm
(b)4+19=23 1+20=21 7+21=28=2 5+22=27=1 4+23=27=1
(c)Manlyx a valsiuavnt hemmryo's socul
(d)Many a valiant hero's soul
(a)lqaauuqrauaqqvso up mfijtxyz.
(b)10+24=34=8 2+25=27=1 5+26=31=5
(c)dissmmijsmsiinkg to hadeost
(d)dismissing to Hades.
What remains unknown is (i) how the decipherer can know the number of letters in the deciphered word and (ii) how the decipherer can identify the letters to be discarded. The latter may be left to the decipherer's insight, but (i) is indispensable to allow proper deciphering.
Herschel presented this cipher as a challenge to believers of "an axiom that there is no cipher which cannot be read." I guess that in trying to make the puzzle more difficult, he forgot that the scheme need to be (difficult but) invertible.
So, I took a note when I read about a ciphertext that yields two readings, posted by tonybaloney at:
http://www.aerobushentertainment.com/crypto/index.php?topic=36.60
(now it seems the link has changed).
The ciphertext (A) and the two readings (B,C) are as follows:
(A)Xabnsly ngpwpdetlews tbbbtzl aobl stheingdnxmccvv
(B)Several philosophers observe that chloroplatinate
(C)Sing, Celestial Muse, the destroying
(A)hclzepsf xo qskxybbbbui
(B)solution on silverplate
(C)wrath of Achilles,
(A)Egtubatjkh fba lwipizix eqjbnasv nfvj yjcin
(B)reproduces the luminous spectrum with great
(C)Peleus’ son, what myriad woes it
(A)cjzvekzxy gf nbyr gzrefcwxianst
(B)vividness in blue fluorescences,
(C)heaped on the Grecians,
(A)Jxkivu v xcnukwcxpv ifnnszp't tpdvm
(B)whilst a coppersalt insolated might
(C)Many a valiant hero’s soul
(A)lqaauuqrauaqqvso up mfijtxyz.
(B)photographically be coloured.
(C)dismissing to Hades.
Now, I find this ciphertext was posted by Klaus Schmeh on his blog Cipherbrain back in 2016: Wer knackt die Verschlüsselung des Astronomen John Herschel?. The article shows the source: The Photographic News, 5 January 1866 and identifies the author of the ciphertext as J.F.W. Herschel (1792-1871) (Wikisource). The blog readers found out additional information.
The weekly magazine (Google) carried the puzzle on 5 January 1866 (p.5-6), the solution (B) provided by a reader the next week (p.23), and Herschel's correct solution (C) the next week thereafter (p.35). After all, the reading (B) was wrong, and this cipher is not about double reading.
The readers of Cipherbrain worked out the general principle of this cipher: schorsch pointed out the first two words can be explained by word-by-word Caesar cipher, Norbert demonstrated this works for every word, and Thomas found the shift, counted in reverse direction, is the number of letters in the word plus 1, 2, 3, 4, ....
In the following, (a) is the ciphertext, (b) is the number of letters in the (deciphered!) word plus 1, 2, 3, 4, ..., (c) is the result of reverse-shifting, and (d) is the plaintext word.
(a)Xabnsly ngpwpdetlews tbbbtzl aobl stheingdnxmccvv
(b)4+1=5 9+2=11 4+3=7 3+4=7 10+5=15
(c)Svwingt cvelestiatlh muuumse thue desptyroyixnngg
(d)Sing, Celestial Muse, the destroying
(a)hclzepsf xo qskxybbbbui
(b)5+6=11 2+7=9 8+8=16
(c)wraotehu of acuhilllles
(d)wrath of Achilles,
(a)Egtubatjkh fba lwipizix eqjbnasv nfvj yjcin
(b)6+9=15 3+10=13 4+11=15 6+12=18 4+13=17 2+14=16
(c)Prefmleuvs son whtatkti myrjviad woes itmsx
(d)Peleus' son, what myriad woes it
(a)cjzvekzxy gf nbyr gzrefcwxianst
(b)6+15=21 2+16=18 3+17=20 8+18=26=0
(c)hoeajpecd on thex gzrefcwxianst
(d)heaped on the Grecians,
(a)Jxkivu v xcnukwcxpv ifnnszp't tpdvm
(b)4+19=23 1+20=21 7+21=28=2 5+22=27=1 4+23=27=1
(c)Manlyx a valsiuavnt hemmryo's socul
(d)Many a valiant hero's soul
(a)lqaauuqrauaqqvso up mfijtxyz.
(b)10+24=34=8 2+25=27=1 5+26=31=5
(c)dissmmijsmsiinkg to hadeost
(d)dismissing to Hades.
What remains unknown is (i) how the decipherer can know the number of letters in the deciphered word and (ii) how the decipherer can identify the letters to be discarded. The latter may be left to the decipherer's insight, but (i) is indispensable to allow proper deciphering.
Herschel presented this cipher as a challenge to believers of "an axiom that there is no cipher which cannot be read." I guess that in trying to make the puzzle more difficult, he forgot that the scheme need to be (difficult but) invertible.
25/03/2024
A Cipher between Emperor Charles V and Young Prince Philip (1545)
A letter with a ciphered paragraph from Charles V to Prince Philip (1545) is presented on Spanish Ministry of Culture's website (pdf) and PARES' facebook page. The cipher turned out to be the same as the one broken by George Lasry and Carlos Köpte independently in 2023. So, I promoted the cipher as "Charles V-Prince Philip Cipher (1545)" in "Ciphers during the Reign of Emperor Charles V".
24/03/2024
Variable-length Figure Cipher Used by Gilbert Gifford?
Speaking of variable-length figure cipher mentioned yesterday, I found in my notebook file a possibly relevant record:
"The words in italics are in cipher, only partly deciphered. The cipher for the most part consists of figures which run on without a break, and are thus capable of various solutions, according as they are taken as single or double numbers; and this sometimes prevents the deciphering of one passage by the aid of another."
From: 'Appendix: January 1588', Calendar of State Papers Foreign, Elizabeth, Volume 21, Part 1: 1586-1588 (1927), pp. 661-671. URL: http://www.british-history.ac.uk/report.aspx?compid=74826&strquery=cipher Date accessed: 06 May 2013.
This note belongs to a letter from Stafford to Walsingham from January 1588, but may refer to an enclosed letter from Gilbert Gifford (under the pseudonym of Francis Hartley) to Thomas Phelippes (M. Wilsdon).
Inspection of the original manuscript is desired.
"The words in italics are in cipher, only partly deciphered. The cipher for the most part consists of figures which run on without a break, and are thus capable of various solutions, according as they are taken as single or double numbers; and this sometimes prevents the deciphering of one passage by the aid of another."
From: 'Appendix: January 1588', Calendar of State Papers Foreign, Elizabeth, Volume 21, Part 1: 1586-1588 (1927), pp. 661-671. URL: http://www.british-history.ac.uk/report.aspx?compid=74826&strquery=cipher Date accessed: 06 May 2013.
This note belongs to a letter from Stafford to Walsingham from January 1588, but may refer to an enclosed letter from Gilbert Gifford (under the pseudonym of Francis Hartley) to Thomas Phelippes (M. Wilsdon).
Inspection of the original manuscript is desired.
23/03/2024
Variable-length Figure Cipher used by Duke of Lorraine (ca.1620?)
I succeeded in reconstructing a cipher used in a letter to the Duke of Lorraine (ca.1620?) and uploaded it in a new article, "Variable-Length Figure Cipher of Duke of Lorraine (ca.1620?)". I abandoned this before, but I took it up again and this time, two occurrences of the word "temperament" provided a first clue. Unexpectedly, the cipher employed variable-length symbols. That is, code symbols consist of one to three symbols. Since the figures are written without a break, there was some difficulty in identifying code symbols corresponding to plaintext segments.
This kind of variable-length symbols written continuously without a break are often seen in Vatican ciphers. It is yet to be studied how this system came to be used by the Duke of Lorraine's correspondence.
This kind of variable-length symbols written continuously without a break are often seen in Vatican ciphers. It is yet to be studied how this system came to be used by the Duke of Lorraine's correspondence.
19/03/2024
Codebreaking of Sir Robert Southwell during the Popish Plot
The Popish Plot may refer to "a period of extraordinary political tension that took hold in England in 1678" (Fictitious treasons: 'The Popish Plot') instigated by revelation of a fictitious plot known by this name.
My notebook included a reference to Sir Robert Southwell's deciphering during this period since 2009, but I have not known whether this involved codebreaking or used a key obtained in a non-cryptographical way. The other day, I found Southwell himself wrote this was "without a key", which makes it interesting enough to be included in an additional section in "Ciphers of Coleman's Correspondence Discovered in the Popish Plot".
It's a pity it is not known specifically what cipher was broken by Southwell.
My notebook included a reference to Sir Robert Southwell's deciphering during this period since 2009, but I have not known whether this involved codebreaking or used a key obtained in a non-cryptographical way. The other day, I found Southwell himself wrote this was "without a key", which makes it interesting enough to be included in an additional section in "Ciphers of Coleman's Correspondence Discovered in the Popish Plot".
It's a pity it is not known specifically what cipher was broken by Southwell.
17/03/2024
Duke of Ormond's Ciphers during the 1660s
I added a section "Marquis of Ormond's Correspondence" in "English Ciphers during the Restoration Period". It covers some reconstructed ciphers used by the Duke of Ormond, the Earl of Anglesey, the Earl of Arran, and the Earl of Longford.
The Ormond-Anglesey Cipher used in 1663-1664 appears to be based on a printed template of DECODE R433. This shows the template was used as early as 1663.
The Ormond-Anglesey Cipher used in 1663-1664 appears to be based on a printed template of DECODE R433. This shows the template was used as early as 1663.
Subscribe to:
Posts (Atom)