Can artificial intelligence discover connections between Kannada, Tamil, Telugu and Malayalam that are difficult to notice because they use different scripts? A fascinating research study suggests that it can.

Introduction: What if our languages are more connected than we think?

India is home to an extraordinary variety of languages. Walk through Karnataka, Tamil Nadu, Andhra Pradesh, Telangana or Kerala, and you will hear languages that sound different, follow different writing systems and reflect distinct cultural histories.

Kannada, Tamil, Telugu and Malayalam are four major Dravidian languages spoken by millions of people. Although they have their own identities, they also share historical and linguistic connections.

But there is a challenge: how can a computer recognise similarities between two languages when their words are written using completely different alphabets?

This is where artificial intelligence (AI) and natural language processing (NLP) become useful.

A research paper titled “Bilingual Lexicon Induction across Orthographically-distinct Under-Resourced Dravidian Languages,” published in the proceedings of the 7th VarDial Workshop in 2020, explored how computers can identify possible word translations between Kannada, Tamil, Telugu and Malayalam.

The study demonstrates an important idea: by converting these languages into a common writing system and using a smarter way to compare words, AI can become better at discovering relationships between them.

1. Why do Indian languages need AI research?

Imagine asking a computer to find the meaning of a Kannada word in Malayalam.

A human who knows both languages might recognise the relationship immediately. However, a computer does not automatically understand the historical development of words or the relationship between different scripts.

For example, Kannada uses the Kannada script, Telugu uses the Telugu script, Tamil uses the Tamil script, and Malayalam uses the Malayalam script.

Even when words share a common origin or have similar meanings, differences in their written forms can make automated comparison difficult.

This problem becomes even more challenging when researchers have limited digital text, dictionaries and translation data available.

Many widely used AI systems benefit from enormous collections of text. English and several other major world languages have extensive digital resources. Smaller languages and less digitally represented languages often have fewer resources for training and testing AI systems.

The research paper addresses this challenge by investigating methods that can help computers learn relationships between Dravidian languages even when large parallel translation datasets are unavailable.

2. What exactly did the researchers investigate?

The researchers focused on four languages:

• Kannada

• Tamil

• Telugu

• Malayalam

Their goal was to develop better bilingual lexicons.

A bilingual lexicon is essentially a dictionary connecting words in one language with their translations in another language.

For example, a Kannada–Tamil lexicon might connect a Kannada word with its Tamil equivalent. If computers can generate these connections accurately, the resulting dictionaries could help improve translation systems and other language technologies.

The researchers investigated six language pairs:

1. Tamil and Telugu
2. Tamil and Kannada
3. Tamil and Malayalam
4. Telugu and Kannada
5. Telugu and Malayalam
6. Kannada and Malayalam

Instead of depending entirely on manually prepared dictionaries, the researchers used computational methods to discover possible word correspondences.

3. The first breakthrough: Giving different languages a common script

One of the study's most interesting ideas is transliteration.

Transliteration means representing a word from one writing system using another writing system. It aims to represent the word's written or spoken form, rather than translate its meaning.

For example, Kannada words can be written in Latin letters to represent their pronunciation. The same can be done for Tamil, Telugu and Malayalam.

This creates a common format in which a computer can compare words more easily.

Think of it this way: imagine four people writing the same type of information in four different formats. Comparing their records is difficult. If all four use a common format, identifying similarities becomes easier.

The researchers converted the language corpora into the Latin script before training their word representations.

Importantly, transliteration does not mean that the languages become the same. Kannada remains Kannada, Tamil remains Tamil, and so on. Only the representation used by the computer changes.

The experiment found that transliteration generally improved bilingual lexicon induction compared with methods that worked directly with the original scripts.

This is particularly valuable for languages whose scripts differ even when their vocabularies contain historically related words.

4. How can AI recognise words that have similar meanings?

The paper uses a technique called bilingual lexicon induction, or BLI.

Despite its technical name, the idea is straightforward: teach a computer to identify words in two languages that are likely to be translations of each other.

The system learns from the contexts in which words appear in large collections of text.

For instance, words associated with farming might frequently appear near terms related to crops, soil, irrigation or harvesting. A computer can represent these contextual patterns mathematically.

These numerical representations are called word embeddings.

Think of word embeddings as coordinates on a huge mathematical map. Words used in similar contexts may occupy nearby positions on that map.

However, independently trained representations for two languages do not automatically line up. The computer needs a way to connect the two maps.

The researchers used an iterative learning process to improve this alignment. They also investigated the use of a seed dictionary, which is a small collection of known word correspondences that gives the system a starting point.

For their experiments, they constructed seed dictionaries using IndoWordNet, a multilingual lexical resource that links words and concepts across Indian languages.

The computer could then use the initial correspondences to discover additional likely translations.

5. Why did the researchers look beyond ordinary spelling comparisons?

Another important contribution of the paper concerns how computers compare written words.

A simple approach is to count the character changes required to turn one word into another. This is known as Levenshtein distance.

For example, a computer can compare two spellings by counting the characters that need to be inserted, removed or replaced.

Although useful, this method has limitations. It may treat two words as similar because they share a few characters, even when those characters do not reflect a meaningful historical relationship.

The researchers therefore investigated another technique called the Longest Common Subsequence, or LCS.

What is the Longest Common Subsequence?

The longest common subsequence identifies the longest sequence of characters that two words share in the same order. The matching characters do not have to appear consecutively.

Consider these two made-up strings:

• Word A: AABCDH

• Word B: AABHEDE

Their longest common subsequence is AAB, which has three characters.

The computer can use this information to estimate how much written structure two words share.

Why is this useful?

Languages often develop related words with differences in spelling, pronunciation or word formation. A comparison method that considers character order and shared sequences can provide useful evidence when searching for possible cognates.

A cognate is a word related to another word through a shared historical origin. Cognates are not simply words that happen to look alike, and spelling similarity alone cannot prove that two words share an origin.

The study found that combining transliteration with the LCS-based method improved the researchers' word-matching results compared with their baseline approaches.

6. What did the experiment actually find?

The researchers trained word embeddings using Wikipedia text available for the four languages. They used data downloaded in July 2019.

The dataset sizes differed considerably:

A token is a unit of text processed by a computer, often a word or word-like unit.

These figures show that the researchers were working with digital text collections that were relatively small compared with the enormous datasets available for some high-resource languages.

The team compared multiple approaches, including systems working with native scripts, transliterated text, numerical seed dictionaries and dictionaries built from IndoWordNet.

Their strongest reported result for the proposed transliteration and LCS approach with a seed dictionary was an 11.00% translation accuracy for Tamil–Malayalam in the reported test setting.

The same approach reported 10.12% for Tamil–Telugu, 6.36% for Kannada–Tamil, 9.74% for Telugu–Kannada, 6.04% for Malayalam–Telugu and 5.36% for Kannada–Malayalam, using the paper's language-pair ordering.

These figures need to be interpreted carefully. They are experimental results from a specific word-translation task, not measures of how well people understand one another's languages or how accurately a complete translation system translates sentences.

The researchers also observed that methods using transliterated text generally outperformed their corresponding native-script baselines.

Why do these results matter?

The most important finding is not that AI can already translate every Dravidian word correctly. It cannot.

Rather, the study demonstrates that using information about how words are written, together with known relationships between languages, can substantially improve automated dictionary construction in low-resource settings.

The paper also shows why AI techniques designed for languages with enormous datasets cannot always be transferred directly to languages with fewer digital resources.

7. Does this mean Kannada, Tamil, Telugu and Malayalam are the same language?

No.

These languages belong to the Dravidian language family, but each has its own grammar, vocabulary, literature, pronunciation patterns and historical development.

Tamil and Malayalam are particularly closely related, while Kannada and Telugu also have notable historical and linguistic connections. Their scripts and vocabulary have developed through complex histories involving language change, contact and borrowing.

AI can help researchers identify patterns across languages, but it does not erase the differences between them.

There is also an important distinction between three kinds of similarity:

• Historical relationship: Words may descend from a common ancestral word.

• Borrowing: One language may adopt a word from another language.

• Accidental resemblance: Two words may look alike without sharing a historical origin.

A computer can generate candidates for investigation, but linguistic evidence is needed to establish why words are related.

The study's focus is on discovering likely translations, not proving the historical origin of every word it matches.

8. What could this mean for the future of Indian languages?

The research has potential implications beyond the four languages studied.

Better translation tools

Automatically generated dictionaries can help expand the vocabulary available to machine translation systems, particularly for words missing from existing translation resources.

Support for less digitally represented languages

The paper discusses a wider Dravidian language family of around 26 languages. Many have far fewer written and digital resources than the four major literary languages examined in the study.

Methods that use relationships between languages could help researchers build resources for languages that have limited digital documentation.

Better language technology for everyday users

Over time, improvements in language resources may contribute to tools such as search engines, digital dictionaries, educational applications, speech technologies and multilingual information services.

These are potential applications rather than outcomes directly demonstrated by this particular experiment. They depend on further research, more data and careful evaluation.

Preserving linguistic diversity

Digital language resources can help make linguistic knowledge more accessible to future generations. Research into multilingual AI may support efforts to document and use languages that otherwise receive less attention from mainstream technology.

However, technology alone cannot preserve a language. Communities, speakers, educators, writers and institutions remain central to keeping languages alive.

9. What are the limitations of this research?

The findings are promising, but several limitations matter.

First, the training data were limited. The researchers used Wikipedia collections that were much smaller than those available for some high-resource languages.

Second, translation accuracy remained low. The best reported result was 11.00% for the Tamil–Malayalam test setting. The method therefore represents an improvement in a difficult research task, not a ready-made solution for reliable translation.

Third, similar spelling does not guarantee the correct meaning. Words may resemble one another because of borrowing, common ancestry or coincidence. Context and linguistic knowledge remain important.

Fourth, the experiment covered only four Dravidian languages. Its results cannot automatically be generalised to every Indian language, including languages from the Indo-Aryan, Austroasiatic or Tibeto-Burman families.

Finally, a system that identifies individual word translations is not the same as a system that understands idioms, grammar, cultural references or entire conversations.

Further research is needed to build more reliable and comprehensive language technologies.

10. The bigger picture: AI as a tool for discovering linguistic connections

For a long time, building digital language resources has depended on the availability of dictionaries, parallel texts and large collections of written material.

Artificial intelligence offers additional ways to investigate relationships between languages, even when those resources are limited.

The study by Bharathi Raja Chakravarthi and colleagues illustrates how combining three ideas can help:

1. Converting related languages into a common writing system.
2. Using known word correspondences to guide learning.
3. Comparing word forms through the Longest Common Subsequence.

Together, these techniques improved bilingual lexicon induction for the four Dravidian languages examined in the paper.

The work is a reminder that language technology is not only about building systems for English or other globally dominant languages. It can also help researchers investigate the linguistic connections, diversity and digital needs of India's many languages.

For speakers of Kannada, Tamil, Telugu and Malayalam, this research offers an interesting example of how computational methods can help explore connections across distinct linguistic traditions without treating those traditions as identical.

Research paper reference

Chakravarthi, B. R., Rajasekaran, N., Arcan, M., McGuinness, K., O’Connor, N. E., & McCrae, J. P. (2020). Bilingual Lexicon Induction across Orthographically-distinct Under-Resourced Dravidian Languages. Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2020), 57–69.

Research paper: https://aclanthology.org/2020.vardial-1.6/

Conference proceedings: https://aclanthology.org/2020.vardial-1.0/

This article is a simplified, public-facing explanation of the cited research paper. The experimental results and technical claims above refer to the methods and findings reported by the authors.