Reinforcement Learning for Feature Compatibility Optimization in Large Language Models

An Adaptive Large Language Model Learning Framework Using Distributed Parameter Sharing with Automated Feedback

Software applications have transitioned from a static request/response model to an interactive, unbounded user experience driven by natural language intent and high-fidelity responses. This level of fluid application engagement requires an increased step function in system performance and data precision to ensure users obtain the most relevant and accurate results to their natural language requests and/or recommendations. As new information is added to a natural language application to improve the user experience such as a new product attributes or updated information for a specific topic, guardrails must be established to ensure the new and existing data is correlated accurately, increases the performance response gradient, and can scale as new data is introduced. For example, applications and APIs that use large language models (LLM) have been trained on a specific dataset sample, but as new data is introduced those models need to be updated and learn the correct relationships between existing and updated content. This process, which may require manual intervention, attenuates the system’s ability to scale, can introduce unnecessary performance entropy, and produce non-deterministic results for the end users.

In this article, we explore a new LLM and reinforcement learning system called the Adaptive Compatibility Learning with Dynamic Parameter Optimization (ACO). The intent of the ACO system is to independently learn n-dimensional relationships between distinct datasets using a distributed network of deterministic and stochastic generative AI reward models across the artificial learning pipeline. An intrinsic improvement in software application and API performance is realized by federating the trained ACO system into multiple reinforcement policy and value functions that correlate reward parameters and automatically optimizes policy weights through a global value function orchestrator. ACO ensures that the environment learns independently by routing the orchestrators output gradients to the supervised fine-tuned (SFT) base model and thus influencing the probability distribution of taking possible actions during information relationship learning and modeling.

Training Pipeline

The controllable input to ACO begins with the trained or SFT base model applied on a new knowledge base or dataset. Autoencoders with denoising objective functions are used for the base model to allow for bidirectional processing during training and enabling improved contextual understanding. The ACO SFT model functions as the query generator that defines the relational guardrails for datasets to intersect and endemically model the conditional probability of the intersection of more than two distinct dataset events while the probability distribution of all the preceding events is considered.

The objective of the ACO pipeline is to train the reward model used to optimize ACO’s probability distribution of taking various possible actions that lead to the most accurate permutation of dataset states. Rather than using manual feedback mechanisms to train ACO’s reward model, we introduce a query rewriter and conditional ranker that function as independent distilled LLMs that are quantized to optimize the accuracy and relevancy of the conditional dataset parameters. Those data parameters then function as inputs to the ACO reward model training.

ACO training pipeline. Image by author.

First, the SFT base model is trained on a new dataset and produces a query that attempts to intersect related and unrelated content. For example, a ACO request to explain how two similar products are related and a request to explain how two completely different products are related. The output query from the SFT based model is then federated as input to the query rewriter which is a distilled LLM that disambiguates and refines the query parameters into subquery partitions. This will decorate the original SFT output request with additional context and attributes so that the input to the ranker has high fidelity semantic relationships between the subquery states. Furthermore, the ACO rewriter output will be self-contained, preserve semantic intent across datasets or knowledge base, and improve retrieval performance for search or external context decoration.

ACO training pipeline. Image by author.

The ACO rewriter is designed as a conditional language model where wt′ is the t-th token in the rewritten query and T is the length of the rewritten query:

The rewriting objective during the training is to minimize the standard cross-entropy loss:

With the rewriter loss attenuated and query decorated, the next step in the ACO algorithm is the ranker model that consumes the output of the rewriter to produce the conditional rules that rank the subqueries by relevance, validity, and information gain. The ranker is also a distilled autoregressive LLM that maximizes the log-likelihood under the forward autoregressive factorization with an objective function to rank the subqueries as inputs to train the reward model’s scalar outputs. This contrasts with typical reinforcement learning approaches that have a human feedback mechanism to rank parameters or statement relevance used to model a reward scalar, train the value function, and optimize an SFT base model policy. Given our rewriter query q and candidate set D = {d1, d2,…dn}, the ACO ranking model outputs a permutation such that:

where score is a relevance measure learned by the ACO ranking model.

Shared Reward

Overall ACO rank training involves optimizing the model parameters through gradient descent with the objective of minimizing the ranking loss across training batches. The ACO SFT base model, rewriter, and ranker agent constructs are all components in the ACO environment that have access to observation information, the action space, and a shared global reward that is mutually exclusive from the reward model being trained.

ACO shared learning and reward. Image by author.

The share global reward is a novel approach in ACO that enables the constructs to collaborate in a unified learning environment and maximize the shared reward outcome to achieve optimal precision, recall, and weighted F1 results. In this shared heuristic, the global reward optimizes the value function by updating the value of a state using the sum of the current reward, the maximum discounted value of any of the next states, and using the global ACO environment parameters for additional context learning. In addition to a shared reward among the agents, penalty objects are assigned to each agent to improve convergence velocity and attenuate training variance. For example, the penalty object for the ranker reward function prevents the agent from ranking queries that are contextually similar within 2–3 standard deviations.

With a shared reward and parameter mechanisms among agents defined, the ACO reward model is then trained from the conditional ranker output and uses the reward signals to subsequently train the value function, which we will refer to as the ACO critic. Reward training learns a scalar reward function that approximates the conditional preferences from the ranked model over model outputs such as prompt-response pairs, so that ACO can later optimize actions against this reward when the agent is in a given state and maximizes the state value. Although training the reward model does not use human feedback to define preferred parameters and instead relies on a trained ACO agent network with shared rewards, the model does output an unbounded scalar reward similar to the human feedback approach such that:

and an objective function to train, for example whenever the ranking agent prefers A over B:

This encourages the ACO reward model to assign higher scores to preferred responses. A scalar reward head as a single linear layer is added on top of the final hidden state while KL (Kullback-Leibler) regularization is applied to prevent the reward model from drifting too far from the base language model. With a SFT base model and trained reward model that has learned correct and incorrect cross-dataset correlations, the next step is to use the reward signals from the trained reward model to then train the value function that predicts the expected cumulative reward based on the signals. The trained value function will be used to influence the SFT’s policy which takes the current state as input and returns a probability distribution over the available actions.

The dependency between the environment’s value function and policy enables us to structure such agents as actor and critic constructs, with the value function as the critic and SFT as the actor. The value function needs to output the value estimates for the input state observation as received from the environment, such that it could be used as a baseline for the SFT policy estimator (actor). As we will soon see, the actor and critic can be federated into multiple objects that can improve system performance while integrating with a global centralized network parameter server to store and retrieve the parameters for both the actors and the critics.

In the ACO architecture, the critic will learn the state-action value function, which will estimate the expected return from taking an action in each state. This type of learning function for our critic can directly evaluate the quality of specific actions, compute the policy gradient for the actor more effectively, and provide action-specific feedback to the actor. Furthermore, we use the Markov Decision Process (MDP) as part of critic’s value function training to define the state transition probability function, such as the probabilities of transitioning from the current state to any of the next possible states by taking an action. During ACO training, the value function uses transitions sampled from the MDP environment, where each sample consists of the current state, action taken, reward received, and the resulting next state. Using the bellman optimality equation for our critic’s Q-learning, which is derived from MDP principles, ACO bootstraps from future state-action values:

The optimality explicitly uses the MDP’s reward function and transition dynamics, while the max operation reflects the actor’s ability to choose optimal actions in future states.

The Markov property ensures that the Q-function can make accurate predictions based solely on the current state-action pair, without needing historical information and gradually converging to optimal action-values that capture both immediate rewards and future consequences of actions.

To improve our critic training performance and stabilize value function updates we leverage prioritized experience replay (PER) by replaying more important experiences more often and those that are expected to have higher learning value. PER uses temporal difference (TD) error as the criteria to prioritize specific experience instances for replay in subsequent iterations of the training. We first assign a priority to each transition in the replay buffer:

Next, we sample transitions proportionally to their priorities so that experiences with higher TD errors are more likely to be sampled for training which focuses learning on “surprising” or poorly predicted state-action values.

Distributed Parameter Optimization

With the ACO SFT base model and trained rewriter, ranker, reward, and critic models — we then need to optimize the actor policy and subsequent actions when the agent is in a given state to maximize that state value. The globally distributed fabric requires a system that is performant and low latency because the agent/critic may be partitioned without locality guardrails to the requesting user. For example, a user in EU may request information about a product local to NA and other non-EU geographies. A sequential actor/critic approach will result in delayed dataset compatibility responses based on a non-edge distributed environment in addition to potential instability and convergence related discrepancies for the approximator.

ACO attempts to overcome these limitations by avoiding sequential single actor/critic learning and instead creates multiple partitioned asynchronous actor/critics that are instantiated and trained in parallel across various instances of the environment and across multiple GPU cores.

ACO distributed parameter optimization. Image by author.

The critics value estimates that the actor uses to compute whether an action did better or worse than expected is managed by a global network parameter server that stores the value estimates for the input state observation as received from the environment as well as the adjusted policy that maximizes expected reward for each actor/critic instance. New ACO actor and critic instances that are created regardless of geo distribution retrieves the latest value and policy estimates from the centralized server and updates their parameters while training with their individual instances of the environment independently. For instances that are currently training and/or inferencing, their parameters are merged into the centralized server after n fixed epochs, update their new parameters from the global source, and continue interacting with their environment instance.

This distributed parameter network enables our global actor and critic agents to learn simultaneously and influence each other through dynamic interactions while executing their policies at individual states independently during deployment. ACO training performance and stability for each actor/critic agent is achieved by having them operate on different instances of the same environment class, thus ensuring uncorrelated global parameter updates and attenuating the need for provisioning additional memory for agent experience replay at every object instance. With a federated network established, the final stage of the ACO algorithm is to enable the SFT (actor) model to find a policy that results in the highest possible expected return and leads to a maximum total return across its trajectories.

A new prompt is requested through the actor agent which then leverages its endemic policy to generate an output response. The actor agent selects actions based on the current state for each sequence of the output generation which is then consumed by the reward model that produces a scalar reward value based on state, action, or trajectory. This scalar value is used by the critic’s value function to estimate the expected return (value) of states or actions, thus assisting the actor by providing a learning signal such as an advantage estimate or TD error as described earlier. The ACO actor learns by adjusting its parameters to choose actions that the critic estimates will lead to higher long-term rewards, which in ACO represents correlating related datasets with elevated accuracy while omitting unrelated tokens that are outside the query relevancy guardrails.

Actor policy updates. Image by author.

Overall, our goal is to maximize the learning award which is equivalent to finding the set of weights that optimizes the total rewards and value inferred from the ACO reward and critic models. This is done by minimizing the losses as an outcome of the training and computing the gradient of the expectancy of reward function and then moving in the direction of the gradient until the local maximum is achieved, so as to maximize the reward expectancy.

ACO’s value-based approach allows our critic to learn the value function from which the actor’s policy is derived, using the estimates generated by an optimal value function. Although the value function is stochastic by generating estimates for selecting different actions in a particular state, the implied ACO policy is mostly deterministic because the policy in the case of ACO’s value estimation suggest a single action. This action could be either the best action suggested by the estimation function or any random action. The actor’s goal is to maximize expected cumulative reward, which depends on how states evolve and is based on the expectation over both policy and transition dynamics:

Therefore, the ACO actor must implicitly learn to act optimally given the transition dynamics of the environment. Our transition function will determine the next state of the environment given the current state and selected action. The transition function is a component of MDP described earlier and describes how the environment changes in response to actions.

Laplace Smoothing

To ensure that the MDP and transition function can effectively manage ACO’s unexplored state-action pairs, prevent zero probabilities, and improve policy learning in sparse environments we introduce Laplace smoothing to our transition function estimation. This novel approach prevents ACO from potentially leading to overly confident or incorrect decisions if a particular transition has never been observed and a probability of zero was assigned. By ensuring non-zero transition probabilities, ACO is less likely to get stuck in local optima or make decisions based on incomplete information. In ACO’s discrete states/actions environment, we can empirically estimate the transition function:

where the number of times action in state led to next state divided by total number of times action was taken in state. Applying Laplace smoothing, we add a small constant α>0 to each count:

The smoothing parameter α can be adjusted based on the confidence in the measurement — a larger α leads to more uniform probabilities (more uncertainty), while a smaller α preserves more of the original proportions (more confidence in observed data).

With smoothing applied we can then allow ACO try new actions to discover potentially better rewards or use known good actions to maximize the immediate reward. The more deterministic the transition and MDP is in ACO, the less it needs to be explored. To balance the exploration and exploitation surface area, ACO adds an epsilon factor to represent the probability with which the agents decide to explore. The epsilon value is based on how stochastic or deterministic our MDP outcome is — if it is more stochastic then our agents need to be explored and hence wanting a larger value for the epsilon whereas more deterministic the less it needs to be explored and a smaller value for the epsilon. We also introduce annealing epsilon to gradually reduce exploration over time. When ACO encounters discrete actions, we could wrap the actor’s action selection in epsilon-greedy, then anneal epsilon (ϵ) over time to rely more on the actor’s policy:

In continuous action spaces, ACO can anneal the scale of exploration noise and decrease σt over time:

Independent Learning

Now that the critics state or action value has been computed and measurement insight obtained on how much better or worse the actor’s action was than expected via advantage or TD error, the next step is to adjust the actor’s policy to make optimal actions more likely in the future. The ACO actor updates its policy using the gradient of the log-probability of the action scaled by the critic’s estimate:

This means: if δ is positive, increase the probability of taking that action; if δ is negative, decrease it. To help ACO maintain stability during policy optimization, a KL penalty is applied to maintain stability during policy optimization. The penalty artifact works by measuring and controlling the difference between the new policy and the old policy during updates, preventing the ACO policy from changing too drastically in a single step. This penalty term is added to the optimization objective to maintain similarity between new and old policies and support smoother policy updates.

With the actor’s policy updated, we continue through the query generation, reward, and value function estimate iteration as ACO continues to maximize the actor’s cumulative rewards earned through various action-state transitions. ACO also force multiplies the self-learning gradient and increases prediction accuracy with each iteration because the critics advantage function not only optimizes the actor’s policy but also the original SFT base model which generates the queries for reward model training.

ACO end-to-end independent learning. Image by author.

This novel approach allows ACO to self-learn by bifurcating policy optimizations to both the actor and SFT base model while achieving mutual exclusivity so both reward model training and actor policy optimization can compute in parallel while also sharing optimized parameters at various training and inference epochs. The ACO SFT model serves as the baseline for the actor model therefore can use stochastic gradient ascent to update its policy parameters:

Where α is the learning rate and the log function is the log-probability under the current policy, thus shifting the SFT policy to favor actions with positive advantage as was done with the prior actor model. Those actions that lead to the various possible states during ACO query generation improves not just the accuracy of the reward model but also increases the accuracy and performance of the ACO component system including the rewriter model, ranker model, shared pipeline reward, and the critic value functions that predict ACO’s expected cumulative reward from the reward signals. This is especially advantageous as new structured or unstructured datasets are introduced to ACO, and guardrails must be developed to ensure accurate correlations and joining with the existing sample space. For example, as new product attributes are introduced the ACO SFT base model and actor model parameters have already been optimized from prior product information, therefore ACO’s performance demands to train the components with correlated guardrails is intrinsically reduced. A product dataset that has already been included in the SFT query generator will require ACO to only learn the incremental product attributes that are included with the dataset instead of training against the entire dataset surface area.

Conclusion

The ACO system represents an improvement in how large language models and reinforcement learning systems handle dataset relationships and parameter optimization. Through its architecture incorporating distributed actor-critic networks and shared rewards, ACO enables independent learning of n-dimensional relationships between distinct datasets while maintaining efficiency and performance.The key components we introduced in this article include:

• A distributed network of deterministic and stochastic generative AI reward models

• A shared global reward mechanism that enables collaborative learning among system components

• Laplace smoothing for effective management of unexplored state-action pairs

• Independent learning capability through bifurcated policy optimizations

• A centralized network parameter service for efficient distribution and updating of model parameters

  
  

The system’s ability to self-learn and optimize through parallel computing of reward model training and actor policy optimization makes it particularly effective when incorporating new structured or unstructured datasets into generative AI applications and APIs. It reduces the performance demands for training with correlated guardrails, as the system only needs to learn incremental changes rather than retraining on entire datasets. ACO attenuates the need for manual human feedback through its query rewriter and conditional ranker components, providing the foundation for natural language systems to adapt to new information while raising the bar on performance and accuracy.