Paper accepted at IoT 2025!

Our paper has been accepted at the 15th International Conference on the Internet of Things (IoT) 2025!

A Context-Aware Benchmarking Approach for Scenario-Based Evaluation of Smart Devices

Nikolaos Papadakis (Télécom SudParis, France), Athanasios Tsirikas (University of Patras, Greece), Fotios Fouskas (AEGIS IT RESEARCH, Greece), Krish Chatterjie (Télécom SudParis, France), Christian Badolato (University of Maryland, Baltimore County, USA),Gregory Blanc (Télécom SudParis, France), Roberto Yus (University of Maryland, Baltimore County, USA), Kostas Magoutis (FORTH, Greece), Georgios Bouloukakis (Télécom SudParis, France; University of Patras, Greece)

Understanding how smart devices behave under real-world conditions remains challenging, as existing benchmarking tools often isolate metrics like energy consumption or protocol compliance without considering realistic interactions or coordinated use. This paper presents a scenario-driven benchmarking framework tailored for smart devices which integrates control, orchestration, instrumentation, and monitoring into a unified, reproducible workflow. Our framework enables structured specification of interaction scenarios, precise scheduling of device actions, and synchronized measurement of energy usage and network activity. We validate our approach through a smart home testbed comprised of off-the-shelf speakers, lights, and fans across scenarios that reflect typical user interactions and adversarial network conditions. Results reveal behavioral differences not evident from device specifications alone, highlighting the importance of context-rich, scenario-based evaluation for understanding smart device performance in practice.