Privacy-Preserving Zero-Day Malware Detection in Encrypted Traffic Using Temporal-Structural Transformers

Michelle Raya de Luis *

Department of Computer Science, University of Colorado Boulder, Boulder, Colorado, USA.

Zohaib Ali

ITR Laboratories Canada Inc., Montreal, Canada.

Najlaa Jannah

Umm Al-Qura University, Makkah, Saudi Arabia.

Muhammad Arslan

Department of Computer Science Institute of Southern Punjab, Multan, Pakistan.

*Author to whom correspondence should be addressed.


Abstract

End-to-end encryption protocols such as TLS 1.3 and QUIC strengthen user privacy but reduce the visibility available to conventional payload-based network-security inspection. This study presents a privacy-preserving Temporal-Structural Transformer (TST) framework for zero-day malware detection in encrypted network traffic without decrypting packet payloads. The framework combines transformer-based temporal modelling of packet sequences with graph attention mechanisms for capturing structural relationships among flows. Observable metadata, including packet size, direction, inter-arrival time, flow statistics, and available handshake attributes, are used to represent encrypted communication behavior. A contrastive-learning objective is incorporated with classification loss to improve generalization to previously unseen malware families. The framework is evaluated using multiple benchmark datasets containing TLS 1.2, TLS 1.3, and QUIC traffic, with zero-day testing, ablation analysis, robustness testing, cross-validation, and real-time performance assessment. The proposed model achieves an F1-score of 0.93 in the overall evaluation and 0.934 in the zero-day setting, with a zero-day ROC-AUC of 0.95. Ablation results indicate contributions from the temporal, graph, and contrastive-learning components, while robustness tests show limited degradation under the evaluated perturbations. Reported inference latency is below 10 ms per flow, with throughput above 10,000 flows per second. These findings support metadata-based temporal and structural modelling as a privacy-preserving approach to zero-day malware detection in encrypted traffic.

Keywords: Zero-day malware detection, encrypted traffic analysis, privacy-preserving cybersecurity, Temporal-Structural Transformer, graph attention network, contrastive learning, TLS 1.3, QUIC, network intrusion detection, metadata-based detection


How to Cite

Luis, Michelle Raya de, Zohaib Ali, Najlaa Jannah, and Muhammad Arslan. 2026. “Privacy-Preserving Zero-Day Malware Detection in Encrypted Traffic Using Temporal-Structural Transformers”. Journal of Engineering Research and Reports 28 (9):1-12. https://doi.org/10.9734/jerr/2026/v28i91993.

Downloads

Download data is not yet available.