Dual-Discriminator PPO for Game Recommendation

Multimodal Reinforcement Learning for Game Recommendation · COMSNETS 2026

Ayush Deshmukh*, Siddharth Kelkar*, Anand Kumar M
* Equal contribution

TL;DR I built a recommender that treats a list of games as a sequence of decisions. A PPO policy picks games one at a time, guided by two discriminators, and sees each game through its cover art, title and description. On 88,310 Steam users it reaches 2.3× higher precision than conventional baselines and 58% better cold-start performance.

2.3×higher precision than baselines

58%better cold-start performance

1.8×faster convergence with the pretrained discriminator

88,310Steam users evaluated

Why RL for recommendations?

Most recommenders score each game on its own and show you the top ten. But a list is a sequence: what you suggest third should depend on the first two, and a good list keeps someone playing next month, not just clicking today. That trade-off between engagement now and satisfaction later is exactly what RL is designed for. So I made the whole list of ten one episode: the state is the player plus everything recommended so far, and each action is the next game.

Illustration of the pipeline: a Steam game's art, title and description are encoded by CLIP and BART, a PPO generator picks games one at a time for a player, and two discriminators send reward back. — **The whole loop.** Every game becomes a fused CLIP + BART embedding. The generator picks the next game given the player and the picks so far, and two discriminators judge the list and send reward back to it.

What I built

Multimodal game embeddings. For every Steam game I embed the header image and the title with CLIP. The "About the game" text is usually long and noisy, so BART summarises it first and CLIP embeds the summary. The normalised embeddings are fused, so the model sees what a game looks like and what it's about, not just its ID.
A similarity-based PPO generator. Two transformers encode the player's history and the games picked so far, and gated cross-attention connects them. The policy then chooses the next game by similarity in embedding space. That turns an impossible action space (tens of thousands of games) into something PPO can actually learn.
Two discriminators instead of hand-written rewards. One is pretrained and frozen: it scores whether a list matches what the player likes, and it keeps training stable. The other is a GAN-style discriminator trained alongside the policy that learns to tell generated lists from real play histories. Together they reward lists that are both relevant and coherent.

Results

Evaluated on 88,310 Steam users:

2.3× higher precision than conventional baselines at modelling how preferences change over time
58% better cold-start performance: for players with little history, the model fills the gaps by synthesising a plausible history
The pretrained discriminator makes training converge 1.8× faster without making the recommendations less diverse

Abstract

Recommendation systems struggle to balance immediate engagement with long-term user satisfaction, especially in dynamic domains like gaming. We propose a Reinforcement Learning (RL) framework enhanced with Generative Adversarial Networks (GANs) to generate personalized game sequences that optimize both short-term relevance and sustained engagement. The proposed approach introduces three key components: (1) a similarity-based Proximal Policy Optimization algorithm leveraging normalized multimodal embeddings (text, image, metadata) for action selection; (2) dual discriminators - a fixed, pretrained model and a trainable GAN-style discriminator - that collaboratively guide policy learning; and (3) a CLIP+BART-driven multimodal fusion module to capture rich cross-modal game semantics. Evaluated on a dataset of 88,310 Steam users, our method outperforms conventional baselines, achieving 2.3× higher precision in modeling evolving preferences and 58% better cold-start performance through sparse-history synthesis. The pretrained discriminator improves the convergence speed by 1.8× while preserving sequence diversity. By aligning GAN-based feedback with RL objectives, our model adapts to changing user contexts while maintaining coherence in the progression of the game. This work advances adaptive recommendation by unifying multimodal understanding with dual-feedback optimization, offering robust performance in fast-evolving digital environments.

BibTeX

@inproceedings{deshmukh2026multimodal,
  title     = {Multimodal Reinforcement Learning for Game Recommendation using
               Dual-Discriminator {PPO} Framework},
  author    = {Deshmukh, Ayush and Kelkar, Siddharth and M, Anand Kumar},
  booktitle = {2026 18th International Conference on COMmunication Systems
               \& NETworks (COMSNETS)},
  year      = {2026},
  doi       = {10.1109/COMSNETS67989.2026.11418281}
}