Hyperparameter Optimization and Evaluation Metrics for Unsupervised Anomaly-Based Cyberattack Detection in Imbalanced Smart Home Datasets

In the rapidly progressing field of smart home technology, safeguarding the security of interconnected devices and systems is critical. Unsupervised anomaly detection is a recognized solution; however, its effectiveness heavily depends on the hyperparameter optimization process. In this work, we hypothesize that metrics tailored for imbalanced data, such as the Matthews correlation coefficient (MCC), can guide the development of more robust and generalizable unsupervised anomaly detection models. To test this hypothesis, we systematically evaluate the impact of optimizing seven distinct metrics (e.g., MCC, BACC, AUC-PR) across four widely used smart home datasets (N-BaIoT, IoTID20, Bot-IoT, and ToN-IoT). Our results show that MCC optimized models achieve an average normalized MCC of 0.801, outperforming those optimized for accuracy (0.781) and AUC-ROC (0.733), and reducing the cumulative distance to ideal performance by 66% compared to conventional metric optimization. These findings underscore the importance of metric selection in unsupervised settings, offering a foundation for building more reliable and generalizable security models for smart home environments. To our knowledge, this is the first work to systematically assess the effects of optimizing distinct evaluation metrics across multiple smart home datasets.