Can a Computer Discover Hidden Similarities Between Kannada, Tamil, Telugu and Other World Languages?
Have you ever wondered why Kannada, Tamil, Telugu and Malayalam share certain similarities? Or why Arabic and Hebrew sometimes resemble each other in their words and grammatical structures, while Chinese and Japanese share many written characters?
Languages may look completely different on the surface, but could there be hidden patterns that computers can help us discover?
A research paper titled “Exploring Language Similarities with Dimensionality Reduction Techniques” investigates this question using data science and natural language processing. Its approach demonstrates how computers can represent languages as numerical data and display patterns of similarity on a two-dimensional graph.
The idea is fascinating: instead of relying only on what we can see or hear, researchers can use computational methods to explore patterns across different languages.
1. Why do we need computers to compare languages?
Imagine comparing thousands of sentences written in Kannada, Tamil, Telugu, Hindi, Sanskrit, Chinese, Arabic and English.
A human researcher would need considerable time to examine their vocabulary, sentence structures, grammar and writing systems. Comparing large collections of text manually would be difficult.
Computers can help by processing large amounts of text and identifying recurring patterns.
For example, researchers can investigate:
Whether languages use similar words or word patterns.
Whether their written forms share recurring features.
Whether words appear in similar contexts.
Whether different languages exhibit similarities in sentence structure or meaning.
However, these are different types of similarity. Two languages may use similar scripts without having closely related grammar, and two languages may be historically related even if their modern writing systems look very different.
This is where computational language analysis becomes useful.
2. How can a computer understand language?
Computers do not automatically understand languages in the same way humans do. Researchers first convert written text into numerical representations that mathematical models can process.
Think of this as converting a book into a large collection of numbers. Those numbers capture selected features of the text.
The research paper discusses several techniques for doing this.
A. Bag of Words: Counting the words
One of the simplest approaches is called the Bag of Words model.
Imagine two short English sentences:
“Agriculture supports farmers.”
“Farmers support agriculture.”
Both sentences contain the same three words, even though their word order differs.
A Bag of Words model records which words occur and how frequently they appear. It can therefore identify similarities in vocabulary, but it does not properly capture word order or sentence meaning.
Researchers can use this approach as an initial way to compare text collections.
B. TF-IDF: Identifying important words
Another technique is called TF-IDF, short for Term Frequency–Inverse Document Frequency.
In simple terms, TF-IDF gives greater importance to words that help distinguish one document from others, rather than treating every word as equally informative.
For example, a collection of agricultural documents may frequently contain the word “crop.” Because that word appears everywhere, it may be less useful for distinguishing individual documents than a more specific term.
TF-IDF helps computers represent these differences numerically.
In the paper, the author uses both Bag of Words and TF-IDF representations to explore language similarities.
C. Word2Vec: Learning from context
Word2Vec takes a different approach. It learns numerical representations of words by examining how they appear in surrounding contexts.
For example, if a particular word repeatedly appears near words associated with farming, irrigation and cultivation, the model can learn patterns from those contexts.
This does not mean the computer understands every cultural or historical meaning behind a word. It means that the model learns statistical relationships from the text it processes.
The research paper explores Word2Vec as a way to investigate similarities beyond simple word counting.
3. What is dimensionality reduction?
This is the central idea behind the paper.
A computer may represent a word, sentence or document using hundreds or thousands of numerical dimensions. Such representations are useful for mathematical analysis but are difficult for humans to visualize directly.
Dimensionality reduction attempts to represent this complex data in fewer dimensions while preserving selected patterns.
Imagine describing a city using thousands of measurements. A map cannot display all those measurements individually, but it can provide a simplified view that helps us understand spatial relationships.
Similarly, researchers can transform complex numerical representations of language into a two-dimensional graph.
The paper uses a technique called UMAP, which stands for Uniform Manifold Approximation and Projection.
In a UMAP plot, data points that are similar according to the chosen representation and model settings may appear close together. Groups of points can form clusters, helping researchers identify patterns worth investigating.
An important limitation is that the graph is a simplified representation. Its clusters are not definitive proof of historical relationships between languages.
4. What did the research paper observe about world languages?
The author explored different groups of languages using numerical text representations and visualizations.
English and Latin
The paper reports that English and Latin appeared close together in one visualization.
This is an interesting observation, but it needs careful interpretation.
English is a Germanic language, not a Romance language descended from Latin. However, English has borrowed a substantial amount of vocabulary from Latin, both directly and through French and other historical influences.
A model that compares word distributions may therefore detect similarities between English and Latin even though they belong to different branches of the Indo-European language family.
This illustrates an important point: similarity in a computer model does not necessarily mean common descent from one language.
Chinese and Japanese
The paper also reports a cluster involving Chinese and Japanese text.
One possible explanation is their historical relationship through Chinese characters. Chinese characters have been adopted and adapted in Japanese writing, where they are known as kanji. In Chinese, the corresponding characters are called hanzi.
However, Chinese and Japanese are not simply two versions of the same language. Their grammar, pronunciation and linguistic histories differ substantially.
Their written characters can create visible similarities in text-based analysis even when the underlying languages differ.
Arabic and Hebrew
The paper reports substantial overlap between Arabic and Hebrew in some visualizations.
This is consistent with an important linguistic fact: Arabic and Hebrew are both Semitic languages within the Afro-Asiatic language family.
They share historical linguistic relationships and have similarities in aspects of vocabulary and word formation, although they are distinct languages.
The computer-generated clusters are an interesting way to explore these patterns, but the historical relationship is established through linguistic evidence, not merely through the graph.
5. What does the paper suggest about Indian languages?
One particularly interesting section examines Indo-Aryan and Dravidian languages.
India contains several major language families, each with its own history and diversity.
The paper compares languages including Kannada, Tamil, Telugu, Malayalam, Hindi, Nepali, Marathi, Sanskrit and Persian (Farsi).
Kannada, Tamil and Telugu
Kannada, Tamil and Telugu belong to the Dravidian language family. Malayalam also belongs to this family.
These languages have distinct histories, literary traditions and grammatical characteristics. They are related, but they are not identical.
The paper reports that Kannada and Telugu overlapped in one Bag of Words visualization. A Word2Vec visualization also showed substantial overlap involving Kannada, Telugu and Tamil.
These observations suggest that the chosen text representations detected similarities in the datasets. They do not establish how closely related the languages are or prove that one language originated from another.
To interpret the results properly, researchers would need to examine the source texts, the amount of data used, the preprocessing methods and the numerical features that influenced the clusters.
Why might Malayalam appear closer to Sanskrit in a visualization?
The paper reports that Malayalam appeared near Indo-Aryan languages in one TF-IDF visualization.
There is historical context worth considering here. Malayalam is a Dravidian language, but Sanskrit has had a substantial influence on its vocabulary and literary traditions. Malayalam writing also developed through historical interactions among regional writing traditions.
A text-based model may detect these influences because of the vocabulary and written material included in its dataset.
Nevertheless, Malayalam is not classified as an Indo-Aryan language simply because it shares vocabulary or other features with Sanskrit.
This is an example of why computational observations must be interpreted alongside established linguistic research.
6. Can a computer discover similarities between unrelated languages?
This is one of the more interesting questions raised by the research.
The paper explores whether language representations can reveal patterns even when languages do not share obvious vocabulary or writing systems.
For example, two languages may express similar ideas using entirely different words. A sufficiently suitable model, trained on appropriate data, might detect some relationships through context or grammatical structure.
But there is a major challenge: the model's output depends on the data it receives.
The paper notes that the Bible was used as the text corpus for its Word2Vec comparison of classical languages. When the same translated material is used across languages, the model may learn similarities arising from the shared content rather than from a direct historical relationship between the languages.
This distinction is crucial.
If two books describe the same events, their words and contexts may be similar because they communicate the same content. That does not automatically mean the languages themselves are closely related.
Future research could investigate this question using larger, balanced and diverse collections of text, including everyday conversations, literature and other forms of writing.
7. How could this research help artificial intelligence in India?
India is home to many languages, dialects and regional varieties. While major languages have received considerable attention in language technology, many smaller or less-resourced languages still need better digital tools.
The approach discussed in the paper could contribute to several applications.
Improving translation systems: Researchers could investigate whether techniques developed for a well-resourced language can be adapted to a related language, provided the necessary linguistic similarities are established.
Supporting smaller languages: Computational analysis could help identify useful patterns in languages for which large training datasets are unavailable.
Studying historical texts: Researchers may use computational tools to compare ancient manuscripts, identify recurring patterns and generate new questions for historical linguists.
Exploring dialects: Similar methods could help compare regional varieties of a language, provided the data are representative and the models are evaluated carefully.
Building language-learning tools: Researchers could investigate shared vocabulary, recurring grammatical patterns and other features that may help learners understand related languages.
These are potential applications rather than outcomes demonstrated by the paper itself. Developing reliable systems would require additional experiments and evaluation.
8. What are the limitations of this research?
The paper provides an exploratory approach to visualizing language similarities. Its findings should not be interpreted as a final classification of the world's languages.
Several factors affect the results.
The choice of text matters. Religious texts, news articles, literature and everyday conversations have different vocabulary and writing styles.
Writing systems can influence similarity. Languages that use similar characters may appear closer in some representations even if their grammar and language-family relationships differ.
Vocabulary does not tell the whole story. Languages can borrow words from one another without sharing a recent common ancestor.
The visualization loses information. Reducing hundreds or thousands of dimensions to two dimensions necessarily simplifies the original data.
Clusters require linguistic interpretation. A graph can help researchers discover patterns, but historical relationships must be investigated using comparative linguistics and other relevant evidence.
The paper is therefore best understood as an exploration of how data science can help researchers formulate questions about languages, rather than as proof of new language-family relationships.
9. The future of language research
Artificial intelligence is creating new opportunities to study language at a scale that would be difficult to manage manually.
By combining computational analysis with historical records, linguistic knowledge and carefully selected datasets, researchers can investigate patterns across languages spoken by millions of people.
For India, this work could be particularly valuable in improving digital access to regional languages and developing language technologies for communities that are underrepresented in existing datasets.
The approach described in the research paper offers one way to visualize complex relationships and generate hypotheses for further investigation.
The next step is to test those hypotheses against reliable linguistic evidence and determine which similarities reflect vocabulary, writing systems, shared text, grammatical structure or genuine historical relationships.
Research paper reference
Veeraraghavan, S. Exploring Language Similarities with Dimensionality Reduction Techniques. Final-year undergraduate research paper, VIT Vellore.
The supplied copy does not specify a publication year, journal, DOI or public repository URL. These details should be added only if they can be independently verified.
Additional references cited in the research paper
- McInnes, L., Healy, J., & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426. https://arxiv.org/abs/1802.03426
- van der Maaten, L., Postma, E., & van den Herik, H. (2009). Dimensionality Reduction: A Comparative Review. Journal of Machine Learning Research, 10, 66–71.
- Jiang, J. J., & Conrath, D. W. (1997). Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy. Proceedings of the International Conference Research on Computational Linguistics (ROCLING X).
