FedEC: Enhancing model federated averaging via two-sided method
DOI:
https://doi.org/10.54939/1859-1043.j.mst.CSCE8.2024.65-75Keywords:
Federated learning; Non-IID data; Elastic aggregation; Contrastive learning.Abstract
Federated learning is a key method for addressing data privacy and security in distributed AI training. However, non-IID data among local clients often causes issues like client drift, leading to slow and unstable model convergence. Current studies typically use one-sided methods that optimize either the client or server-side with the conventional aggregation method FedAvg. In contrast, we introduce FedEC, a two-sided strategy that combines distinct methods: an elastic aggregation algorithm for optimizing the global model on the server and contrastive learning techniques on the client side to reduce divergence between local and global models. This complementary approach fosters mutual reinforcement between client and server, allowing FedEC to better tackle non-IID data challenges. Results from experiments conducted on some benchmark datasets with different settings show that FedEC offers more efficient training and outperforms previous one-sided algorithms, underscoring the effectiveness of this two-sided approach in federated learning.
References
[1]. Yang, Qiang, et al., “Federated machine learning: Concept and applications”, ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10(2), pp. 1-19, (2019). DOI: https://doi.org/10.1145/3298981
[2]. P. e. a. Kairouz, “Advances and open problems in federated learning”, Foundations and Trends® in Machine Learning, vol. 14(1–2), pp. 1-210, (2021).
[3]. McMahan, Brendan, et al., “Communication-efficient learning of deep networks from decentralized data”. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), (2017).
[4]. T. Li, A. K. Sahu, A. Talwalkar e V. Smith, “Federated learning: Challenges, methods, and future directions”. IEEE Signal Processing Magazine, vol. 37(3), pp. 50-60, (2020). DOI: https://doi.org/10.1109/MSP.2020.2975749
[5]. Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin e V. Chandra, “Federated learning with non-iid data”, arXiv preprint arXiv:1806.00582, (2018).
[6]. Sattler, F., Müller, K. R., & Samek e W., “Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints”. IEEE Transactions on Neural Networks and Learning Systems, vol. 32(8), pp. 3170-3722, (2019). DOI: https://doi.org/10.1109/TNNLS.2020.3015958
[7]. L. Tian et al., “Federated optimization in heterogeneous networks”, Proceedings of Machine Learning and Systems, vol. 2, pp. 429-450, (2020).
[8]. H. Y. M. S. Y. P. D. &. K. Y. Wang, “Federated learning with matched averaging”, in International Conference on Learning Representations, (2020).
[9]. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich e A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning”, in In International Conference on Machine Learning (pp. 5132-5143). PMLR, (2020).
[10]. Q. Li, B. He e D. Song, “Model-contrastive federated learning”, in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, (2021).
[11]. Chopra, Sumit, Raia Hadsell, and Yann LeCun, “Learning a similarity metric discriminatively, with application to face verification”, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), (2005).
[12]. D. Chen, J. Hu, V.J. Tan, X. Wei e E. Wu, “Elastic aggregation for federated optimization”, in Proceedings of the IEEE/CVF Conference on Computer Vision, 2023. DOI: https://doi.org/10.1109/CVPR52729.2023.01173
[13]. M. Kayed, A. Anter e H. Mohamed, “Classification of garments from fashion mnist dataset using cnn lenet-5 architecture” in International conference on innovative trends in communication and computer engineering (ITCE), (2020). DOI: https://doi.org/10.1109/ITCE48509.2020.9047776
[14]. A. Baldominos, Y. Saez e P. Isasi, “A survey of handwritten character recognition with mnist and emnist”, Applied Sciences, vol. 9, p. 3169, (2019). DOI: https://doi.org/10.3390/app9153169
[15]. X. Zhang, “The alexnet, lenet-5 and vgg net applied to cifar-10”, in International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), (2021). DOI: https://doi.org/10.1109/ICBASE53849.2021.00083
[16]. A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images”, Technical Report TR, (2009).
[17]. T.-M. H. Hsu, H. Qi e M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification”, arXiv preprint arXiv:1909.06335, (2019).
