Interview with Sukanya Mandal: Synthesizing multi-modal knowledge graphs for smart city intelligence

In their paper LLMasMMKG: LLM Assisted Synthetic Multi-Modal Knowledge Graph Creation For Smart City Cognitive Digital Twins, which was published in the AAAI Fall Symposium series, Sukanya Mandal and Noel O’Connor introduced an approach that leverages large language models to automate the construction of synthetic multi-modal knowledge graphs specifically designed for a smart city cognitive digital twin. Here, Sukanya tells us more about cognitive digital twins, the framework they employed, and some key results.

Could you start by introducing the idea of smart city cognitive digital twins and why this is an interesting area for study?

Cities grow increasingly complex and interconnected, demanding sophisticated tools for management. A cognitive digital twin (CDT) serves as an AI-enabled virtual replica that models the dynamic interplay of physical and social systems, enabling simulations, predictions, and optimized operations. Unlike traditional digital twins that merely mirror physical structures, CDTs reason over integrated insights from traffic sensors, healthcare systems, energy grids, and social media to anticipate issues like correlating traffic congestion with increased hospital visits.

Traditional structured data approaches struggle with the semantic relationships defining urban ecosystems. Knowledge graphs (KGs) address this by representing highly connected systems with explainability essential for decision-makers.

Could you talk about the approach that you introduce in your paper?

The paper presents LLMasMMKG, a modular four-stage framework that leverages large language models (LLMs) to automate synthetic multi-modal knowledge graph (MMKG) construction for smart city CDTs. Data from heterogeneous sources across smart homes, healthcare, transportation, and energy are gathered and preprocessed. Multimodal representation learning via Sentence-BERT embeddings unifies textual and sensor data into a shared semantic space. LLM-guided knowledge extraction employs fine-tuned BERT for entity recognition and GPT-4 for relationships. The process culminates in a resource description framework (RDF)-formatted knowledge graph population with a hierarchical domain ontology.

A key innovation uses LLM-driven synthetic data generation to overcome sparsity, privacy risks, and biases in real urban datasets. This reusable, scalable orchestrator pipeline adapts readily to new modalities.

What makes multi-modal knowledge graphs a good tool for the problem of creating cognitive digital twins of smart cities?

Smart cities generate inherently multi-modal data: text from social media, time-series from sensors, geospatial information. While relational databases store structured data efficiently, they falter on complex interdependencies. MMKGs excel through interconnected triples (e.g., Sensor123 locatedAt IntersectionA), enabling nuanced cross-modal queries.

They ensure interoperability via RDF and shared ontologies, support inference (e.g., broken traffic light disrupts flow), and provide transparent explainability. Crucially, MMKGs link modalities, like a smart home temperature reading to grid energy patterns or a traffic complaint to sensor data, forming the unified “world model” a KG based CDT needs to reason across city systems.

How did you go about implementing the method?

Implementation spans four smart city domains using synthetic data. GPT-4-turbo generated diverse text (device descriptions, social media posts, patient notes, traffic reports) via domain-specific prompts with tuned parameters. Deterministic sensor data via Python (pandas, numpy), simulates realistic patterns: sine waves for temperature/heart rate, interpolation for GPS, daylight cycles for solar energy (~10,000 points/domain over one year).

Sentence-BERT (all-mpnet-base-v2) embeds text for semantic similarity. Fine-tuned BERT handles entity recognition; GPT-4 extracts relationships (controls, affects, locatedAt). The RDF knowledge graph incorporates a custom hierarchical ontology. Codebase: GitHub.

What were some of the key results that you obtained?

The framework fuses heterogeneous sources into a unified, semantically rich MMKG, demonstrated across domains. Automated extraction – BERT for entities like “EcoTemp thermostat” or “Wi-Fi”, GPT-4 for relations like “John experiences fatigue” or “John has history of hypertension” – yields interconnected triples with cross-modal links via cosine similarity.

This proof-of-concept delivers a scalable pipeline for holistic smart city views, reducing manual KG effort while enabling reasoning. The modular design confirms reusability for broader cognitive digital twin frameworks.

Could you say a bit about future plans for this work?

Near-term priorities include entity linking across sources, comparative LLM evaluations, human-in-the-loop verification, and expansion to images, video, geospatial, and economic data. Advanced reasoning via graph neural networks and rule engines will support what-if analyses. Rigorous quality benchmarks, both quantitative (coherence, utility) and qualitative target real-synthetic data hybrids, addressing biases and scalability.

It gets even more interesting when these three threads evolve LLMasMMKG: a reusable library for generative-semantic-synthetic KG engineering – for the broader knowledge-based neurosymbolic AI community, where progress is stalled because of the lack of desired KG structure; privacy-by-synthesis and synthetic data as a privacy-enhancing technology (PET) approach resolving data scarcity-privacy paradoxes leading to KG-driven simulation and benchmark datasets; and integration as knowledge bootstrap as well as the world model for CogTwin (IJCAI 2025), the hybrid cognitive architecture, positioning it as the epistemic foundation: nursery (development phase) and wind tunnel (stress testing), for autonomous urban digital twins, aiding planners in sustainable decision-making.

More on these developments – soon. Stay tuned!

Find out more

• Read the full paper: LLMasMMKG: LLM-assisted synthetic multi-modal knowledge graph creation for smart city cognitive digital twins. Proceedings of the AAAI Symposium Series. Vol. 4. No. 1. 2024.
• Our 2024 interview with Sukanya as part of the doctoral consortium participant series. Developing a cognitive digital twin framework for smart cities.

About Sukanya

Sukanya Mandal is a Data and AI Strategist and Execution Specialist, and a researcher at Dublin City University, Ireland. Her research develops cognitive digital twin frameworks for smart cities, combining knowledge graphs, cognitive architectures, and privacy-preserving federated learning. With a decade of experience in AI and data science, her work appears at AAAI 2024, IJCAI 2025, IEEE WF-IoT 2024, and IEEE SWC 2023.

AIhub is dedicated to free high-quality information about AI.

AIhub is dedicated to free high-quality information about AI.