MSMF-Q: Multi-Scale Mean Field Q-Learning for Scalable Multi-Agent Coordination
This research investigates MSMF-Q, a Multi-Scale Mean Field Q-Learning approach for scalable coordination in large multi-agent systems. The method addresses a limitation of conventional mean-field reinforcement learning, which can compress a heterogeneous population into a single global average, and purely local approaches, which may overlook broader population behavior. MSMF-Q combines an attentive near-field representation of influential neighboring agents with a far-field summary of global action distributions. Initial proof-of-concept experiments evaluate the method across changing population sizes and heterogeneity levels. The research remains in progress, with broader environments, stronger statistical validation, and long-horizon evaluation planned.
The question
Multi-agent reinforcement learning becomes increasingly difficult as the number of interacting agents grows. Explicitly modeling every pairwise interaction can become computationally expensive, while compressing an entire population into a single average can remove information that matters for local decision-making.
Mean-field reinforcement learning addresses the scalability problem by approximating the behavior of many surrounding agents through an aggregate population statistic. This can substantially simplify the learning problem, but it assumes that the population-level average contains enough information for an individual agent to act effectively.
That assumption can become problematic in heterogeneous environments. An agent may be strongly affected by a small number of nearby or strategically important agents even when their behavior differs substantially from the global population average. Conversely, methods that focus primarily on local neighbors may capture these interactions while missing broader system-wide trends.
The central research question is:
Can scalable multi-agent learning represent local interactions and global population behavior at different scales, and does combining those representations improve coordination as agent populations and heterogeneity change?
MSMF-Q explores this question through a dual-scale representation.
The near-field component focuses on locally relevant agents and uses attention to assign different importance to different neighbors. The far-field component summarizes the wider population through a global mean-field representation. These signals are combined before estimating action values.
The current scope is a proof-of-concept investigation of whether this representation provides useful information beyond global mean-field, local mean-field, attention-only, or simpler dual-context alternatives.
Important open questions include long-horizon credit assignment, convergence behavior, communication constraints, adversarial agents, continuous action spaces, and performance in established multi-agent reinforcement learning benchmarks.
The approach
MSMF-Q, or Multi-Scale Mean Field Q-Learning, is designed around the idea that interactions in a large multi-agent environment can occur at different spatial or behavioral scales.
The model uses two complementary representations.
Near-field representation
The near-field component represents locally relevant agents rather than treating every neighbor as equally important. Multi-head attention is used to weight neighboring interactions dynamically so that the representation can emphasize agents whose state or behavior is more relevant to the current decision.
This component is intended to preserve local heterogeneity that may disappear when all interactions are averaged together.
Far-field representation
The far-field component summarizes the wider population through a mean-field statistic. Instead of explicitly encoding every distant agent, it represents broader population behavior through an aggregate action distribution.
This provides a scalable approximation of macro-level system conditions.
The local and global representations are then combined with the individual agent's state when estimating action values.
Work completed
A controlled proof-of-concept benchmark has been implemented to test the core representation.
The model was trained using 100-agent scenarios and evaluated across populations ranging from 50 to 500 agents. The evaluation included three levels of population heterogeneity and three random seeds.
The comparison included:
- Global mean-field Q-learning;
- Local mean-field Q-learning;
- Local attention;
- a simpler dual-mean representation; and
- the full MSMF-Q representation.
The primary metrics included action accuracy and regret relative to the target coordination policy.
These experiments were designed to answer a narrow question: whether combining attentive near-field information with global far-field context can improve decision quality in a controlled heterogeneous coordination problem.
Planned evaluation
The current benchmark is intentionally limited and does not establish general MARL performance.
Planned work includes:
- increasing the number of experimental seeds;
- reporting confidence intervals and statistical comparisons;
- evaluating longer-horizon sequential environments;
- studying training stability and convergence;
- testing sensitivity to neighborhood size and attention configuration;
- measuring compute and memory requirements as population size increases;
- evaluating communication failures and changing population behavior; and
- testing the method in stronger multi-agent benchmarks such as traffic, market, PettingZoo-style, or SMAC-style environments.
Graph-based multi-agent approaches are also relevant comparison points for future evaluation.
Limitations
The existing experiment is a controlled one-step coordination benchmark. It does not yet demonstrate general convergence, long-horizon credit assignment, robustness to adversarial agents, continuous control, or production-scale distributed execution. These limitations define the next stage of the research rather than established capabilities of MSMF-Q.
What it explores
The current results come from a controlled proof-of-concept experiment rather than a comprehensive multi-agent reinforcement learning benchmark.
Across the tested settings, MSMF-Q achieved a mean regret of 0.00125 and an overall action accuracy of 96.82%. Within this experimental setup, the combined near-field and far-field representation performed best overall among the evaluated variants.
However, one of the most important observations is that the simpler Dual-Mean approach remained close to MSMF-Q.
That result is important because it prevents the current experiment from supporting an overly broad conclusion that attention is always necessary. Instead, it suggests a more useful research question: under what levels of heterogeneity and interaction structure does attentive local modeling provide enough additional information to justify its complexity?
What the research explores
The broader contribution being investigated is whether large multi-agent systems should be represented at more than one interaction scale.
A single global mean field is computationally attractive but can hide locally important behavior. A local-only representation can preserve neighborhood structure but may lose information about population-wide trends.
MSMF-Q explores whether an agent can benefit from seeing both:
- near-field context, representing the most relevant local interactions; and
- far-field context, representing broader population behavior.
The current evidence supports continued investigation of this dual-scale representation, but it does not yet establish superiority across general MARL environments.
The next stage will focus on determining when the additional attention mechanism materially improves coordination, how performance changes with increasing population size and heterogeneity, and whether the approach remains useful in sequential environments with delayed rewards, changing neighborhoods, and communication constraints.
The intended contribution is therefore not simply a higher benchmark score, but a clearer understanding of how local and global interaction structure should be represented when multi-agent learning must scale to large heterogeneous populations.