Di Wu

Xue Liu

Reinforcement learning from human feedback (RLHF) with proximal policy optimization (PPO) is widely used but often yields less diverse outpu… (voir plus)ts than supervised fine-tuning, suggesting an effect in which the policy’s support contracts during on-policy optimization. We formalize this “policy contraction” with the Support Retention Ratio (SRR)—the share of SFT completions that retain non-negligible probability under the RL policy—and additionally track token-entropy, Kullback–Leibler (KL) divergence to the reference, and repetition. We propose Contraction-Aware PPO (CaPPO), a minimum-norm multi-gradient update that co-optimizes reward, entropy, and KL, paired with a controller that steers exploration toward a target token entropy. On HH-RLHF, Summarize-from-Feedback, and UltraFeedback with Qwen2-7B, Qwen2.5-14B, Mistral-7B-Instruct, and Llama-3-8B-Instruct, CaPPO increases win rate by 2 to 4 points over PPO and improves diversity, gaining 0.2 to 0.3 higher SRR. The gains persist under decoding sweeps and are robust to reward scaling and critic variance. Treating reward, diversity, and stability as first-class objectives, CaPPO mitigates contraction without sacrificing alignment performance.

2025-12-31

International Conference on Learning Representations (Accept (Poster))

openreview.net

A Unified Solution to Diverse Heterogeneities in One-Shot Federated Learning

Jun Bai

Yiliao Song

Atul Sajjanhar

Yong Xiang

Wei Zhou

Xiaohui Tao

Yan Li

Yue Li

One-Shot Federated Learning (OSFL) restricts communication between the server and clients to a single round, significantly reducing communic… (voir plus)ation costs and minimizing privacy leakage risks compared to traditional Federated Learning (FL), which requires multiple rounds of communication. However, existing OSFL frameworks remain vulnerable to distributional heterogeneity, as they primarily focus on model heterogeneity while neglecting data heterogeneity. To bridge this gap, we propose FedHydra, a unified, data-free, OSFL framework designed to effectively address both model and data heterogeneity. Unlike existing OSFL approaches, FedHydra introduces a novel two-stage learning mechanism. Specifically, it incorporates model stratification and heterogeneity-aware stratified aggregation to mitigate the challenges posed by both model and data heterogeneity. By this design, the data and model heterogeneity issues are simultaneously monitored from different aspects during learning. Consequently, FedHydra can effectively mitigate both issues by minimizing their inherent conflicts. We compared FedHydra with five SOTA baselines on four benchmark datasets. Experimental results show that our method outperforms the previous OSFL methods in both homogeneous and heterogeneous settings. The code is available at https://github.com/Jun-B0518/FedHydra.

2025-08-02

Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (publié)

Self-Refined Generative Foundation Models for Wireless Traffic Prediction

Chengming Hu

Hao Zhou

Xi Chen

Jun Yan

Xue Liu

With a broad range of emerging applications in 6G networks, wireless traffic prediction has become a critical component of network managemen… (voir plus)t. However, the dynamically shifting distribution of wireless traffic in non-stationary 6G networks presents significant challenges to achieving accurate and stable predictions. Motivated by recent advancements in Generative AI (GenAI)-enabled 6G networks, this paper proposes a novel self-refined Large Language Model (LLM) for wireless traffic prediction, namely TrafficLLM, through in-context learning without parameter fine-tuning or model training. The proposed TrafficLLM harnesses the powerful few-shot learning abilities of LLMs to enhance the scalability of traffic prediction in dynamically changing wireless environments. Specifically, our proposed TrafficLLM embraces an LLM to iteratively refine its predictions through a three-step process: traffic prediction, feedback generation, and prediction refinement. Initially, the proposed TrafficLLM conducts traffic predictions using task-specific demonstration prompts. Recognizing that LLMs may generate incorrect predictions on the first attempt, this paper designs feedback demonstration prompts to provide multifaceted and valuable feedback related to these initial predictions. The validation scheme is further incorporated to systematically enhance the accuracy of mathematical calculations during the feedback generation process. Following this comprehensive feedback, our proposed TrafficLLM introduces refinement demonstration prompts, enabling the same LLM to further refine its predictions and thereby enhance prediction performance. Evaluations on two realistic datasets demonstrate that the proposed TrafficLLM outperforms LLM-based in-context learning methods, achieving performance improvements of 23.17% and 17.09%, respectively.

2024-12-31

IEEE Transactions on Vehicular Technology (publié)

Generative AI as a Service in 6G Edge-Cloud: Generation Task Offloading by In-context Learning

Hao Zhou

Xue Liu

Zhu Han

Jianzhong (Charlie) Zhang

Generative artificial intelligence (GAI) is a promising technique towards 6G networks, and generative foundation models such as large langua… (voir plus)ge models (LLMs) have attracted considerable interest from academia and telecom industry. This work considers a novel edge-cloud deployment of foundation models in 6G networks. Specifically, it aims to minimize the service delay of foundation models by radio resource allocation and task offloading, i.e., offloading diverse content generation tasks to proper LLMs at the network edge or cloud. In particular, we first introduce the communication system model, i.e., allocating radio resources and calculating link capacity to support generated content transmission, and then we present the LLM inference model to calculate the delay of content generation. After that, we propose a novel in-context learning method to optimize the task offloading decisions. It utilizes LLM's inference capabilities, and avoids the difficulty of dedicated model training or fine-tuning as in conventional machine learning algorithms. Finally, the simulations demonstrate that the proposed edge-cloud deployment and in-context learning task offloading method can achieve satisfactory generation service quality without dedicated model training or fine-tuning.

2024-12-22

IEEE Wireless Communications Letters (publié)

Accelerating Digital Twin Calibration with Warm-Start Bayesian Optimization

Abhisek Konar

Amal Feriani

Seowoo Jang

Xue Liu

Digital twins are expected to play an important role in the widespread adaptation of AI-based networking solutions in the real world. The ca… (voir plus)libration of these virtual replicas is critical to ensure a trustworthy replication of the real environment. This work focuses on the input parameter calibration of radio access network (RAN) simulators using real network performance metrics as supervision signals. Usually, the RAN digital twin is considered a black-box function and each calibration problem is viewed as a standalone search problem. RAN simulators are slow and non-differentiable, often posing as the bottleneck in the execution time for these search problems. In this work, we aim to accelerate the search process by reducing the number of interactions with the simulator by leveraging RAN interactions from previous problems. We present a sequential Bayesian optimization framework that uses information from the past to warm-start the calibration process. Assuming that the network performance exhibits gradual and periodic changes, the stored information can be reused in future calibrations. We test our method across multiple physical sites over one week and show that using the proposed framework, we can obtain better calibration with a smaller number of interactions with the simulator during the search phase.

2024-06-08

ICC 2024 - IEEE International Conference on Communications (publié)

Adaptive Dynamic Programming for Energy-Efficient Base Station Cell Switching

Junliang Luo

Yi Tian Xu

M. Jenkin

Xue Liu

Energy saving in wireless networks is growing in importance due to increasing demand for evolving new-gen cellular networks, environmental a… (voir plus)nd regulatory concerns, and potential energy crises arising from geopolitical tensions. In this work, we propose an approximate dynamic programming (ADP)-based method coupled with online optimization to switch on/off the cells of base stations to reduce network power consumption while maintaining adequate Quality of Service (QoS) metrics. We use a multilayer perceptron (MLP) given each state-action pair to predict the power consumption to approximate the value function in ADP for selecting the action with optimal expected power saved. To save the largest possible power consumption without deteriorating QoS, we include another MLP to predict QoS and a long short-term memory (LSTM) for predicting handovers, incorporated into an online optimization algorithm producing an adaptive QoS threshold for filtering cell switching actions based on the overall QoS history. The performance of the method is evaluated using a practical network simulator with various real-world scenarios with dynamic traffic patterns.

2024-06-08

2024 IEEE International Conference on Communications Workshops (ICC Workshops) (publié)

Anomaly Detection for Scalable Task Grouping in Reinforcement Learning-based RAN Optimization

Jimmy Li

Igor Kozlov

Xue Liu

The use of learning-based methods for optimizing cellular radio access networks (RAN) has received increasing attention in recent years. Thi… (voir plus)s coincides with a rapid increase in the number of cell sites worldwide, driven largely by dramatic growth in cellular network traffic. Training and maintaining learned models that work well across a large number of cell sites has thus become a pertinent problem. This paper proposes a scalable framework for constructing a reinforcement learning policy bank that can perform RAN optimization across a large number of cell sites with varying traffic patterns. Central to our framework is a novel application of anomaly detection techniques to assess the compatibility between sites (tasks) and the policy bank. This allows our framework to intelligently identify when a policy can be reused for a task, and when a new policy needs to be trained and added to the policy bank. Our results show that our approach to compatibility assessment leads to an efficient use of computational resources, by allowing us to construct a performant policy bank without exhaustively training on all tasks, which makes it applicable under real-world constraints.

2024-06-08

2024 IEEE International Conference on Communications Workshops (ICC Workshops) (publié)

Optimizing Energy Saving for Wireless Networks Via Offline Decision Transformer

Yi Tian Xu

M. Jenkin

Seowoo Jang

Xue Liu

With the global aim of reducing carbon emissions, energy saving for communication systems has gained tremendous attention. Efficient energy-… (voir plus)saving solutions are not only required to accommodate the fast growth in communication demand but solutions are also challenged by the complex nature of the load dynamics. Recent reinforcement learning (RL)-based methods have shown promising performance for network optimization problems, such as base station energy saving. However, a major limitation of these methods is the requirement of online exploration of potential solutions using a high-fidelity simulator or the need to perform exploration in a real-world environment. We circumvent this issue by proposing an offline reinforcement learning energy saving (ORES) framework that allows us to learn an efficient control policy using previously collected data. We first deploy a behavior energy-saving policy on base stations and generate a set of interaction experiences. Then, using a robust deep offline reinforcement learning algorithm, we learn an energy-saving control policy based on the collected experiences. Results from experiments conducted on a diverse collection of communication scenarios with different behavior policies showcase the effectiveness of the proposed energy-saving algorithms.

2024-06-08

ICC 2024 - IEEE International Conference on Communications (publié)

Probabilistic Mobility Load Balancing for Multi-band 5G and Beyond Networks

Saria Al Lahham

Ekram Hossain

Xue Liu

2024-06-08

2024 IEEE International Conference on Communications Workshops (ICC Workshops) (publié)

Hallucination Detection and Hallucination Mitigation: An Investigation

Junliang Luo

Tianyu Li

M. Jenkin

Steve Liu

2024-01-15

ArXiv (prépublication)

AdaTeacher: Adaptive Multi-Teacher Weighting for Communication Load Forecasting

Chengming Hu

Ju Wang

Yan Xin

Charlie Zhang

Xue Liu

To deal with notorious delays in communication systems, it is crucial to forecast key system characteristics, such as the communication load… (voir plus). Most existing studies aggregate data from multiple edge nodes for improving the forecasting accuracy. However, the bandwidth cost of such data aggregation could be unacceptably high from the perspective of system operators. To achieve both the high forecasting accuracy and bandwidth efficiency, this paper proposes an Adaptive Multi-Teacher Weighting in Teacher-Student Learning approach, namely AdaTeacher, for communication load forecasting of multiple edge nodes. Each edge node trains a local model on its own data. A target node collects multiple models from its neighbor nodes and treats these models as teachers. Then, the target node trains a student model from teachers via Teacher-Student (T-S) learning. Unlike most existing T-S learning approaches that treat teachers evenly, resulting in a limited performance, AdaTeacher introduces a bilevel optimization algorithm to dynamically learn an importance weight for each teacher toward a more effective and accurate T-S learning process. Compared to the state-of-the-art methods, Ada Teacher not only reduces the bandwidth cost by 53.85%, but also improves the load forecasting accuracy by 21.56% and 24.24% on two real-world datasets.

2023-12-03

Global Communications Conference (publié)