DIDYMOS-XR: Digital Twins That Keep Up with Reality

00:00:09: Welcome to Inside Immersion, the podcast of the Immersive Reality Lab at Hamm-Lippstadt University of Applied Sciences.

00:00:17: I'm your host, Jan-Niklas Voigt-Antons, Professor for Computer Science with a focus on immersive media.

00:00:24: In this series, we take you behind the scenes of immersive technologies like virtual and augmented reality, how they are researched, how they are developed, and how they might shape the way we live, work, and interact with each other in the future.

00:00:38: Today, we're diving into the future of urban and industrial navigation—powered by advanced 3D scene reconstruction, artificial intelligence, and extended reality.

00:00:50: The project we're exploring is called DIDYMOS-XR, and it's all about bringing digital twins to life.

00:00:57: My guest is Francesco Vona, a research associate in our lab and one of the key brains behind DIDYMOS-XR.

00:01:06: Francesco, welcome!

00:01:08: Thanks, Jan.

00:01:09: It's great to be here.

00:01:11: I'm really happy to join you today and look forward to the conversation.

00:01:15: Before we get too nerdy, tell our listeners a bit about yourself.

00:01:19: You're not just the "XR guy" in the hallway, right?

00:01:22: Yeah... I

00:01:24: guess not.

00:01:26: So I am a postdoctoral researcher at HSHL's Immersive Reality Lab.

00:01:31: I'm originally from Italy, and about three years ago I moved to Lippstadt.

00:01:36: Here we are. I work with immersive technologies, but I also have many interests.

00:01:43: I'm into comics, both manga and classic superhero comics... I love rap and techno music... I collect LEGO and yes, Pokémon cards too!

00:01:54: I contain multitudes.

00:01:56: My background spans computer engineering, user experience research and applied projects in healthcare and smart city contexts.

00:02:05: What really drives my work is the connection between technology and people.

00:02:09: I care about building advanced XR systems that don't just work, but actually make life better, more accessible and more intuitive.

00:02:22: Facts about the project.

00:02:24: The project name is DIDYMOS-XR – long title: Digital, Dynamic and Responsible Twins for XR.

00:02:32: Funder is European Union in the Horizon Europe program.

00:02:38: Funding is around six million euros, the duration of three years.

00:02:43: It started in twenty twenty-three and will wrap up end of twenty twenty-five.

00:02:49: The consortium consists of fourteen partners, including universities, industry, SMEs from different countries.

00:03:00: What is it about?

00:03:01: DIDYMOS-XR is making digital twins for XR that are always in sync with the real world.

00:03:09: Ethically designed and usable to things like urban navigation or industry automation.

00:03:15: We want XR apps that aren't just cool demos, but actually do something meaningful and are safe for users and their data.

00:03:25: And for anyone who missed it: "Didymos" means "twins" in Greek, right?

00:03:31: Exactly!

00:03:32: Twins, digital and physical, always connected.

00:03:36: Okay, let's start with DIDYMOS and why now... Okay, so let's get basic for a second.

00:03:44: What exactly is DIDYMOS-XR,

00:03:47: if you had to explain it to, say, your grandma?

00:03:50: To my grandma? I'd say: imagine the city as a mirror world inside of computer. A simple copy of the real city. When something changes outside, like roadworks or a closed street, it changes there too.

00:04:08: And with your phone or special glasses, you can look into that mirror and the city helps you!

00:04:14: It tells where to go, warns about problems... or shows you the easiest way to meet someone.

00:04:21: The important part is that it's safe, respects people's privacy and shows what is happening right now, not old information.

00:04:31: Why is this so urgent?

00:04:33: What's changed that made this project necessary now?

00:04:38: That's a great question!

00:04:39: The short answer is that the technology is everywhere now,

00:04:43: but it hasn't quite caught up with reality.

00:04:46: Digital twins are being used more and more for cities, mobility, planning, maintenance, but most of them still pretty static.

00:04:57: They get outdated quickly, or they need a lot manual effort to stay useful.

00:05:02: At the same time, we're collecting much more data than before, which raises some very real question: who controls it?

00:05:10: Who can see it?

00:05:12: And how people's privacy is protected.

00:05:15: So the urgency comes from bringing these two things together, making digital twins truly up-to-date and useful, while also making sure they are ethical, safe and trustworthy in everyday use.

00:05:29: Let's get concrete.

00:05:32: What are some things DIDYMOS-XR can actually do?

00:05:35: Paint us a picture.

00:05:38: For example, for city planning, urban planners can rearrange buildings virtually and immediately measure the impact of those changes.

00:05:50: Simulate different traffic scenarios or explore historical weather data to support long-term planning decisions.

00:05:57: Citizens and visitors also experience places before going there or receiving enhanced information directly in front important landmarks.

00:06:07: For city maintenance, for example, teams can use AR to identify and locate damages in streets or public infrastructure, prioritize interventions.

00:06:19: And coordinate repairs more efficiently based on real-time data.

00:06:24: In industrial environment, the digital twin can be continuously updated and directly interacted with through AR devices, allowing workers to inspect system, monitor processes and make informed decisions by linking the virtual model with physical environment.

00:06:43: Have you tested this already?

00:06:46: Yes!

00:06:46: We have already completed a first round of testing.

00:06:49: The main goal was collect early feedback to help us improve applications.

00:06:54: What's especially valuable is that participants highlighted issues and use cases we hadn't fully considered before.

00:07:04: And that's exactly why user feedback is so important at this stage.

00:07:08: Based on those insights, we have redefined the prototypes... and are now preparing for our next round of testing to see how updated versions perform in real-world conditions.

00:07:23: But we can definitely dive deeper into validation a bit later.

00:07:28: Now, let's geek out a bit.

00:07:30: How do you actually make these twins "dynamic"?

00:07:35: Where does the data come from?

00:07:37: Yeah, this is where it gets fun.

00:07:40: There's a whole stack behind it.

00:07:42: We start with data sources: fixed cameras, drones, IoT sensors, sometimes even smartphones, basically anything that can give us visual, spatial or contextual information about what's happening in the real world.

00:07:59: All of this data is brought together through AI-based data fusion, which means combining different streams in real time to keep the digital twin constantly updated.

00:08:11: The system is modular, so you run a lightweight setup for single site like factory or neighborhood, or scale it up into an entire city.

00:08:21: An important step comes right after the data arrives: we label it, understand what it represents and assess its quality.

00:08:31: Not all data is good data.

00:08:33: Some of them can be noisy, outdated or unreliable, so it needs to be filtered before it affects the twin.

00:08:40: Once we are confident in this data, we move from real world to digital one.

00:08:45: That's where we extract geometry, create or update meshes and textures... and align everything spatially, so the virtual model truly reflects reality.

00:08:56: This pipeline is what keeps the digital twin accurate, up to date... and usable in real-world XR applications.

00:09:05: And this all happens automatically?

00:09:09: Or do you have people in a room pressing buttons?

00:09:13: There is no control room with someone constantly pressing buttons.

00:09:17: The system is designed to operate automatically for routine updates: data streams are processed, synchronized and integrated into the digital twin in real time.

00:09:28: That said, we intentionally keep a human in loop for oversight. Experts monitor system integrity, validate critical updates and intervene if anomalies are detected.

00:09:40: This hybrid approach ensures scalability and efficiency while maintaining safety, reliability and accountability.

00:09:49: The goal is for the digital twin to remain continuously aligned with physical environment: dynamic, responsive and trustworthy, without requiring manual micromanagement.

00:10:01: What about data privacy?

00:10:04: Are we just tracking everyone?

00:10:06: Absolutely not.

00:10:08: Privacy is a foundational design principle, not an afterthought.

00:10:13: We implement strict safeguards such as data minimization, anonymization and pseudonymization techniques, explicit and informed user consent, full compliance with GDPR related regulatory frameworks.

00:10:30: The system is designed to support users and environments, not to monitor individuals.

00:10:36: People should feel empowered and protected while interacting with digital twin and not surveilled.

00:10:43: Let's talk people,

00:10:45: not just tech.

00:10:47: How do you make sure what you're building actually helps users?

00:10:52: For us technology is never the starting point – people are.

00:10:56: We combine rigorous technical validation with structured user-centered studies. That means bringing participants into controlled lab settings and testing in real-world environments.

00:11:10: We assign real tasks, observe how users interact with the XR system and collect multiple types of data.

00:11:18: Subjective feedback, behavioral measures... and sometimes even physiological signals to understand stress or cognitive load.

00:11:28: But beyond metrics, we pay attention to subtle signals – hesitation before clicking a button, confusion in navigation… or moments where users genuinely smile and say, "Oh, that's cool."

00:11:42: Those moments are incredibly informative.

00:11:44: And yes, we often hear things like: "This menu is too complex."

00:11:49: "I love the navigation, but I prefer voice guidance," or "This overlay blocks too much of my view."

00:11:56: So,

00:11:56: you're saying the users are your toughest critics?

00:12:00: Absolutely!

00:12:01: And honestly, that's a good thing.

00:12:03: When you build in a lab, everything feels intuitive because you design it. But once you put the system into the hands of real users, especially professionals working under pressure, you quickly discover blind spots.

00:12:18: Field tests are humbling.

00:12:19: They reveal context-specific challenges that could never simulate perfectly.

00:12:25: Lighting conditions, distractions, stress, different levels of technical literacy.

00:12:31: That's where the real learning happens.

00:12:34: Have you changed anything based on this feedback?

00:12:37: Many times!

00:12:38: Iteration is part of the DNA of XR design.

00:12:42: For example, during testing we realized that subtle visual alerts weren't always noticed, especially when users were cognitively busy.

00:12:53: So we introduced, for example, multimodal feedback, including tactile cues for critical notifications.

00:13:00: That small addition significantly improved the responsiveness.

00:13:05: In another case, users reported feeling overwhelmed by dense AR overlays.

00:13:10: We reduced visual clutter, prioritized essential information and redesigned layout using progressive disclosure, showing only what's necessary at right moment.

00:13:22: It turns out that in XR, "more features" doesn't mean "better experience."

00:13:27: Often, clarity, timing and simplicity make the biggest difference.

00:13:32: Let's get philosophical for a second.

00:13:35: What's the real impact you hope DIDYMOS-XR has, beyond the tech demos?

00:13:40: For me, the bigger vision goes far beyond impressive prototypes or polished demonstrations.

00:13:47: With DIDYMOS-XR, I hope we can demonstrate that advanced XRs and digital twins technologies are not only powerful or just technically sophisticated.

00:13:59: It's easy to build something that works, but it is much harder to build a thing for people!

00:14:05: The goal isn't just smart environments... but it's responsible environments.

00:14:11: Digital twins should empower citizens and professionals alike, offering clearer situational awareness, better decision support and meaningful control over how their data are used.

00:14:25: Users shouldn't feel observed, they should feel supported.

00:14:29: They shouldn't feel replaced, they should feel augmented.

00:14:33: If technology expands human capability while preserving autonomy, dignity and fairness, then we've done something worthwhile.

00:14:43: That's the standard I believe we should hold ourselves to.

00:14:47: Do you ever worry about the "Black Mirror" side of this? That it could go wrong?

00:14:53: Honestly? Yes, and I think anyone working in immersive technologies should.

00:14:59: Any powerful technology carries risks: misuse, over-surveillance, biased algorithms or systems that unintentionally exclude certain groups.

00:15:13: Pretending those risk don't exist would be irresponsible.

00:15:17: That's exactly why ethics cannot be an afterthought.

00:15:22: In DIDYMOS-XR, we integrate ethical reflection from the start, through an ethics advisory board, privacy-by-design architecture, strict data governance and continuous risk assessment.

00:15:36: But beyond formal structures, it's also a mindset.

00:15:41: Our responsibility isn't just to build something impressive.

00:15:45: It is ask difficult questions:

00:15:47: Who benefit from this system?

00:15:50: Who might be disadvantaged?

00:15:53: What happens if the data is misused?

00:15:55: Or how do we measure transparency and user control?

00:15:59: Innovation without reflection, it's dangerous.

00:16:02: But innovation with safeguards, inclusivity and accountability can genuinely improve how people live and work.

00:16:11: This is a huge project.

00:16:12: What's it like working with such a diverse team?

00:16:17: It's honestly energizing.

00:16:19: We have experts from computer science, robotics, social sciences, urban planning and industry all working together.

00:16:28: At times, it can feel like a modern Tower of Babel.

00:16:32: Different terminologies, different priorities... different ways of thinking.

00:16:37: But that's exactly where the values lies.

00:16:40: When you explain your idea to someone outside your discipline, you suddenly realize what we have been taking for granted.

00:16:47: Those conversations expose blind spots you didn't even know yet.

00:16:53: Interdisciplinarity forces humility, and that's incredibly healthy for innovation.

00:17:00: You start seeing the same technology from technical, social, economic and ethical perspective all at once.

00:17:07: Final words for the students or young researchers out there?

00:17:11: Stay curious, get comfortable being uncomfortable, step outside your discipline, talk with people who think differently about you.

00:17:20: Most importantly, don't just ask, "Can we build it?"

00:17:25: Ask should we build and who is really for.

00:17:29: In XR and AI especially, technical brilliance isn't enough.

00:17:33: Critical thinking, ethical awareness and empathy are just as important as algorithms.

00:17:40: Francesco, thank you for this deep dive.

00:17:42: I hope our listeners got a sense of how XR is evolving—beyond entertainment, into something that can drive real change.

00:17:51: Thank you, Jan, and thanks to everyone listening.

00:17:55: If you are curious, check out the show notes for project links and more information, and let's keep this conversation going!

00:18:02: As always: stay curious, stay critical,

00:18:05: and join us next time on Inside Immersion.