IOSG: From Compute to Intelligence, a Reinforcement Learning-Driven Decentralized AI Investment Map

Verifiability

ZK and Proof-of-Learning provide means to verify whether nodes are genuinely performing inference, addressing integrity issues in open networks. In deterministic tasks such as code and mathematical reasoning, validators only need to check the answers to confirm the workload, significantly enhancing the trustworthiness of decentralized RL systems.

Incentive Layer, Feedback Production Mechanism Based on Tokenomics

The token mechanism of Web3 directly rewards contributors to RLHF/RLAIF preference feedback, making preference data generation transparent, settlement-ready, and permissionless. Staking/Slashing further constrain the feedback quality, forming a more efficient and aligned feedback market than traditional crowdsourcing.

Multi-Agent Reinforcement Learning (MARL) Potential

Blockchain is fundamentally an open, transparent, and constantly evolving multi-agent environment, where accounts, contracts, and agents continuously adjust their strategies under incentives, making it naturally poised to create a large-scale MARL experimental field. Although still in its early stages, its characteristics of transparent state, verifiable execution, and programmable incentives provide a foundational advantage for the future development of MARL.

Classical Web3 + Reinforcement Learning Project Analysis

Based on the above theoretical framework, we will provide a brief analysis of the most representative projects in the current ecosystem:

Prime Intellect: Asynchronous Reinforcement Learning Paradigm prime-rl

Prime Intellect is dedicated to building a global open computing power market, reducing training barriers, driving collaborative decentralized training, and developing a complete open-source superintelligent technology stack. Its system includes: Prime Compute (Unified Cloud/Distributed Computing Environment), INTELLECT Model Family (10B–100B+), Open Reinforcement Learning Environment Center (Environments Hub), and Large-Scale Synthetic Data Engine (SYNTHETIC-1/2).

The core infrastructure component of Prime Intellect, the prime-rl framework, is specifically designed for asynchronous distributed environments and highly related to reinforcement learning. Other components include the OpenDiLoCo Communication Protocol overcoming bandwidth bottlenecks and the TopLoc Verification Mechanism ensuring computational integrity.

Overview of Prime Intellect Core Infrastructure Components

Technical Pillar: prime-rl Asynchronous Reinforcement Learning Framework

prime-rl is the core training engine of Prime Intellect, designed for large-scale asynchronous decentralized environments, achieving high-throughput inference and stable updates through complete decoupling of Actor-Learner. Rollout Workers and Trainers are no longer synchronously blocked, and nodes can join or exit at any time, just needing to continuously pull the latest policy and upload generated data:

· Executor (Rollout Workers): Responsible for model inference and data generation. Prime Intellect innovatively integrates the vLLM inference engine at the Executor side. The vLLM's PagedAttention technology and continuous batching capability allow the Executor to generate inference trajectories at extremely high throughput.

· Learner (Trainer): Responsible for policy optimization. The Learner asynchronously pulls data from a shared experience replay buffer for gradient updates, without waiting for all Executors to complete the current batch.

· Orchestrator: Responsible for scheduling model weights and data flow.

Key Innovations of prime-rl

· Full Asynchrony: prime-rl abandons the traditional PPO synchronous paradigm, does not wait for slow nodes, and does not require batch alignment, enabling any number and performance of GPUs to join at any time, laying the foundation for decentralized RL viability.

· Deep Integration of FSDP2 and MoE: Through FSDP2 parameter slicing and MoE sparse activation, prime-rl efficiently trains hundred-billion-scale models in a distributed environment, with Executors running only active experts, significantly reducing GPU memory and inference costs.

· GRPO+ (Group Relative Policy Optimization): GRPO eliminates the Critic network, significantly reducing computation and memory overhead, naturally adapting to an asynchronous environment. prime-rl's GRPO+ further ensures reliable convergence under high-latency conditions through stabilization mechanisms.

INTELLECT Model Family: A Symbol of Decentralized RL Technology Maturity

INTELLECT-1 (10B, October 2024) first proved that OpenDiLoCo could efficiently train in a heterogeneous network across three continents (communication ratio <2%, compute utilization 98%), breaking the physical boundaries of cross-continental training;

INTELLECT-2 (32B, April 2025) serves as the first Permissionless RL model, validating the stable convergence capabilities of prime-rl and GRPO+ in a multi-step delayed, asynchronous environment, achieving decentralized RL with global open compute participation;

INTELLECT-3 (106B MoE, November 2025) adopts a sparse architecture activating only 12B parameters, trained on 512×H200 to achieve flagship inference performance (AIME 90.8%, GPQA 74.4%, MMLU-Pro 81.9%, etc.), with overall performance approaching or even surpassing significantly larger scale centralized proprietary models.

In addition, Prime Intellect has built several supporting infrastructures: OpenDiLoCo reduces cross-region training communication by orders of magnitude through time-sparsity communication and quantized weight differentials, maintaining 98% utilization for INTELLECT-1 across a three-continent network; TopLoc + Verifiers form a decentralized trusted execution layer to activate fingerprint and sandbox verification ensuring the authenticity of inference and reward data; the SYNTHETIC data engine produces large-scale high-quality inference chains and efficiently runs the 671B model on consumer-grade GPU clusters through pipelined parallelism. These components provide a crucial engineering foundation for data generation, validation, and inference throughput of decentralized RL. The INTELLECT series demonstrates that this technology stack can produce state-of-the-art world-class models, marking the transition of decentralized training systems from the conceptual stage to the practical stage.

Gensyn: RL Swarm and SAPO Reinforcement Learning Core Stack

The goal of Gensyn is to aggregate global idle compute power into an open, trustless, and infinitely scalable AI training infrastructure. Its core includes a cross-device standardized execution layer, peer-to-peer coordination network with a trustless task verification system, and automatic task and reward allocation through smart contracts. Built around the characteristics of reinforcement learning, Gensyn introduces core mechanisms such as RL Swarm, SAPO, and SkipPipe, decoupling the three stages of generation, evaluation, and updating, utilizing a globally heterogeneous GPU "swarm" to achieve collective evolution. Its ultimate delivery is not just raw compute power but Verifiable Intelligence.

Reinforcement Learning Application of the Gensyn Stack

RL Swarm: Decentralized Collaborative Reinforcement Learning Engine

RL Swarm showcases a novel collaborative pattern. It is no longer a simple task distribution but a decentralized "generate-assess-update" loop that mimics human societal learning, analogous to the collaborative learning process, in an infinite loop:

· Solvers: Responsible for local model inference and Rollout generation, ensuring node agnostic interoperability. Gensyn integrates a high-throughput inference engine locally (such as CodeZero) capable of outputting complete trajectories rather than just answers.

· Proposers: Dynamically generate tasks (math questions, code problems, etc.), supporting task diversity and Curriculum Learning-like adaptive difficulty.

· Evaluators: Use a frozen "referee model" or rules to evaluate local Rollout, generating local reward signals. The evaluation process is auditable, reducing malicious behavior.

All three together form a P2P RL organizational structure, enabling large-scale collaborative learning without centralized scheduling.

SAPO: Policy Optimization Algorithm for Decentralized Refactoring

SAPO (Swarm Sampling Policy Optimization) centers around "sharing Rollout and filtering out gradient-free samples instead of sharing gradients." Through large-scale decentralized Rollout sampling, SAPO treats received Rollouts as locally generated, enabling stable convergence in a decentralized, significantly delayed node environment. Compared to PPO, which relies on a Critic network and has high computational costs, or GRPO, which is based on intra-group advantage estimation, SAPO allows even consumer-grade GPUs to effectively participate in large-scale RL optimization with very low bandwidth.

Through RL Swarm and SAPO, Gensyn demonstrates that reinforcement learning (especially the RLVR phase) inherently fits a decentralized architecture—because it relies more on large-scale, diverse exploration (Rollout) rather than high-frequency parameter synchronization. Combined with PoL and Verde's validation system, Gensyn offers an alternative path for training trillion-parameter models that no longer depends on a single tech giant: a self-evolving superintelligent network composed of millions of heterogeneous GPUs worldwide.

Nous Research: Verifiable Reinforcement Learning Environment Atropos

Nous Research is building a set of decentralized, self-evolving cognitive infrastructure. Its core components — Hermes, Atropos, DisTrO, Psyche, and World Sim are organized into a continuous closed-loop intelligent evolutionary system. Unlike the traditional "pre-training–fine-tuning–inference" linear process, Nous employs DPO, GRPO, rejection sampling, and other reinforcement learning technologies to unify data generation, validation, learning, and inference into a continuous feedback loop, creating a continuously self-improving closed-loop AI ecosystem.

Nous Research Component Overview

Model Layer: Hermes and the Evolution of Inference Capabilities

The Hermes series is Nous Research's primary model interface to users, and its evolution clearly demonstrates the industry's path from traditional SFT/DPO alignment to reasoning reinforcement learning (Reasoning RL) migration:

· Hermes 1–3: Instructional alignment and early agent capabilities: Hermes 1–3 rely on low-cost DPO to achieve robust instruction alignment and, in Hermes 3, leverage synthetic data and the first introduction of the Atropos validation mechanism.

· Hermes 4 / DeepHermes: Embedding System-2 slow thinking into weights through a thought chain, enhancing mathematical and code performance with Test-Time Scaling, and building high-purity inference data relying on "rejection sampling + Atropos validation."

· DeepHermes further adopts GRPO instead of the hard-to-distribute PPO, enabling inference RL to run on the decentralized GPU network Psyche, laying the engineering foundation for scalable open-source inference RL.

Atropos: Verifiable Reward-Driven Reinforcement Learning Environment

Atropos is the true hub of the Nous RL system. It encapsulates prompts, tool invocations, code execution, and multi-round interactions into a standardized RL environment that can directly verify the correctness of outputs, thereby providing deterministic reward signals, replacing expensive and non-scalable human annotations. More importantly, in the decentralized training network Psyche, Atropos acts as an "arbiter" to validate whether nodes genuinely improve strategies, supporting auditable Proof-of-Learning, fundamentally addressing the reward trustworthiness issue in distributed RL.

DisTrO and Psyche: Decentralized Reinforcement Learning Optimizer Layer

Traditional RLF (RLHF/RLAIF) training relies on centralized high-bandwidth clusters, which is a core barrier to open-source replication. DisTrO reduces the communication cost of RL by several orders of magnitude through momentum decoupling and gradient compression, enabling training to run on Internet bandwidth; Psyche then deploys this training mechanism on the chain network, allowing nodes to perform local inference, validation, reward evaluation, and weight updates, forming a complete RL loop.

In the Nous framework, Atropos validates the chain of thought; DisTrO compresses training communication; Psyche runs the RL loop; World Sim provides a complex environment; Forge collects real-world inference; Hermes records all learning in the weights. Reinforcement learning is not just a training phase, but a core protocol in the Nous architecture that connects data, environment, model, and infrastructure, allowing Hermes to become a living system that can continuously self-improve on an open-source computational network.

Gradient Network: Reinforcement Learning Architecture Echo

The core vision of the Gradient Network is to reconstruct the AI computing paradigm through the "Open Intelligence Stack." Gradient's tech stack consists of a set of independently evolving, yet heterogeneously collaborative core protocols. The system includes from bottom-layer communication to upper-layer intelligent collaboration: Parallax (distributed inference), Echo (decentralized RL training), Lattica (P2P network), SEDM / Massgen / Symphony / CUAHarm (memory, collaboration, security), VeriLLM (trustworthy verification), Mirage (high-fidelity simulation), collectively forming a continuously evolving decentralized intelligent infrastructure.

Echo—Reinforcement Learning Training Architecture

Echo is Gradient's reinforcement learning framework, whose core design philosophy is to decouple the training, inference, and data (reward) paths in reinforcement learning, allowing Rollout generation, policy optimization, and reward evaluation to independently scale and schedule in a heterogeneous environment. It operates collaboratively in a heterogeneous network composed of inference-side and training-side nodes, maintaining training stability in a wide-area heterogeneous environment through a lightweight synchronization mechanism, effectively alleviating the SPMD failure and GPU utilization bottleneck caused by the mixed inference and training in traditional DeepSpeed RLHF/VERL.

Echo adopts the "Inference-Training Dual Swarm Architecture" to maximize computing power utilization, with each swarm running independently without blocking each other:

· Maximize Sampling Throughput: Inference Swarm consists of consumer-grade GPUs and edge devices, leveraging Parallax to build a high-throughput sampler using pipeline-parallelism, focusing on trajectory generation;

· Maximize Gradient Computing Power: Training Swarm comprises consumer-grade GPU networks that can run in centralized clusters or multiple locations globally, responsible for gradient updates, parameter synchronization, and LoRA fine-tuning, focusing on the learning process.

To maintain consistency between policy and data, Echo provides two lightweight synchronization protocols: Sequential and Asynchronous, achieving bidirectional consistency management of policy weights and trajectories:

· Sequential Pull Mode｜Precision First: The training side forces the inference node to refresh the model version before pulling new trajectories to ensure trajectory freshness, suitable for tasks highly sensitive to outdated policies;

· Asynchronous Push-Pull Mode｜Efficiency First: The inference side consistently generates version-tagged trajectories, while the training side consumes them at its own pace, with the coordinator monitoring version deviations and triggering weight refreshes to maximize device utilization.

At its core, Echo is built on Parallax (heterogeneous inference in low-bandwidth environments) and lightweight distributed training components (e.g., VERL), relying on LoRA to reduce cross-node synchronization costs, enabling reinforcement learning to operate stably across a global heterogeneous network.

Grail: Reinforcement Learning in the Bittensor Ecosystem

Through its unique Yuma consensus mechanism, Bittensor has built a vast, sparse, non-stationary reward function network.

Within the Bittensor ecosystem, Covenant AI has established a vertical integrated pipeline from pre-training to RL post-training through SN3 Templar, SN39 Basilica, and SN81 Grail. Here, SN3 Templar is responsible for pre-training base models, SN39 Basilica provides a distributed computing power marketplace, and SN81 Grail serves as the "Verifiable Inference Layer" for RL post-training, carrying out the core processes of RLHF/RLAIF alignment optimization from base models to aligned policies.

The GRAIL goal is to cryptographically prove the authenticity of each reinforcement learning rollout and bind it to the model's identity, ensuring RLHF can be securely executed in a trustless environment. The protocol establishes a trusted chain through a three-layer mechanism:

1. Deterministic Challenge Generation: Utilizing a drand random beacon and block hash to generate an unpredictable yet reproducible challenge task (e.g., SAT, GSM8K), eliminating pre-computation cheating;

2. PRF-index sampling and sketch commitments enable validators to cost-effectively sample token-level logprob and inference chain, confirming that the rollout is indeed generated by the declared model;

3. Model Identity Binding: Binding the inference process with the model's weight fingerprint and the structural signature of token distribution to ensure that replacing the model or replaying results will be immediately detected. Thus, it provides a foundation of authenticity for inference trajectories (rollouts) in RL.

Building on this mechanism, the Grail subnet achieves a GRPO-style verifiable post-training process: miners generate multiple inference paths for the same question, validators evaluate based on correctness, inference chain quality, and SAT satisfaction score, and normalize the results on-chain as TAO weights. Public experiments have shown that this framework has increased the MATH accuracy of Qwen2.5-1.5B from 12.7% to 47.6%, proving its ability to prevent cheating and significantly enhance model capabilities. In Covenant AI's training stack, Grail serves as the cornerstone of decentralized RLVR/RLAIF trust and execution, which has not yet officially launched on the mainnet.

Fraction AI: Competition-based Reinforcement Learning RLFC

The architecture of Fraction AI is explicitly built around Competition-based Reinforcement Learning (RLFC) and gamified data labeling, replacing the static rewards and manual annotations of traditional RLHF with an open, dynamic competitive environment. Agents compete in different Spaces, where their relative rankings and AI judge scores collectively form real-time rewards, transforming the alignment process into a continuous online multi-agent gaming system.

Core Difference between Traditional RLHF and Fraction AI's RLFC:

Core Value of RLFC lies in rewards no longer coming from a single model, but from ever-evolving opponents and evaluators, avoiding model exploitation and preventing the ecosystem from getting stuck in a local optimum through policy diversity. The structure of Spaces determines the nature of the game (zero-sum or non-zero-sum), driving the emergence of complex behaviors in adversarial and cooperative scenarios.

In terms of system architecture, Fraction AI breaks down the training process into four key components:

· Agents: Lightweight policy units based on open-source LLM, extended with differential weights through QLoRA, enabling low-cost updates;

· Spaces: Isolated task domain environments where agents pay to participate and receive rewards based on win-loss outcomes;

· AI Judges: Instant reward layer built with RLAIF, providing scalable, decentralized evaluation;

· Proof-of-Learning: Binding policy updates to specific competitive results, ensuring a verifiable and cheat-resistant training process.

The essence of Fraction AI is to build a human-machine collaborative evolutionary engine. Users act as the "meta-optimizer" of the policy layer, guiding exploration directions through Prompt Engineering and hyperparameter configuration, while agents autonomously generate vast amounts of high-quality preference pair data in micro-level competitions. This pattern allows data labeling to achieve a commercial closed loop through "Trustless Fine-tuning."

Comparison of Reinforcement Learning Web3 Project Architectures

Summary and Outlook: Path and Opportunities of Reinforcement Learning × Web3

Based on the deconstruction analysis of the above cutting-edge projects, we observe that: despite the various teams' entry points (algorithm, engineering, or market), when reinforcement learning (RL) is combined with Web3, the underlying architectural logic converges into a highly consistent "Decoupling-Verification-Incentivization" paradigm. This is not only a technical coincidence but also the inevitable result of decentralized networks adapting to the unique properties of reinforcement learning.

Reinforcement Learning General Architecture Features: Addressing Core Physical Constraints and Trust Issues

1. Rollout-Learning Physics Separation (Decoupling of Rollouts & Learning)——Default Compute Topology

Communication-sparse, parallelizable Rollouts outsourced to global consumer-grade GPUs, high-bandwidth parameter updates concentrated at a few training nodes, from Prime Intellect's asynchronous Actor-Learner to Gradient Echo's dual-cluster architecture.

2. Verification-Driven Trust Layer (Verification-Driven Trust)——Infrastructureized

In a permissionless network, computational integrity must be enforced through mathematics and mechanism design, representations include Gensyn's PoL, Prime Intellect's TOPLOC, and Grail's cryptographic verification.

3. Tokenized Incentive Loop (Tokenized Incentive Loop)——Market Self-Adjustment

Compute supply, data generation, validation ordering, and reward distribution form a closed loop, driving participation through rewards, suppressing cheating through Slash, enabling the network to remain stable and continuously evolve in an open environment.

Differentiated Technical Paths: Different "Breakthrough Points" Under a Consistent Architecture

Despite architectural convergence, each project has chosen a different technological moat based on its own genes:

· Algorithm Breakthrough Faction (Nous Research): Seeks to fundamentally address the bandwidth bottleneck of distributed training from the mathematical foundation. Its DisTrO optimizer aims to compress gradient communication by thousands of times, with the goal of enabling large-scale model training even on home broadband, a "dimensionality reduction strike" against physical limitations.

· System Engineering Faction (Prime Intellect, Gensyn, Gradient): Focuses on building the next-generation "AI runtime system." Prime Intellect's ShardCast and Gradient's Parallax are both designed to squeeze out the highest heterogeneous cluster efficiency through extreme engineering means under existing network conditions.

· Market Game Theorists (Bittensor, Fraction AI): Focus on designing Reward Functions. By designing sophisticated scoring mechanisms, they guide miners to spontaneously seek the optimal strategy to accelerate the emergence of intelligence.

Advantages, Challenges, and Endgame Outlook

In the paradigm of combining Reinforcement Learning with Web3, the system-level advantages are first reflected in the cost structure and governance structure rewrite.

· Cost Reshaping: Post-training RL has an infinite demand for rollouts, and Web3 can mobilize global long-tail computing power at extremely low cost, a cost advantage that centralized cloud providers find hard to match.

· Sovereign Alignment: Breaking the big-tech monopoly on AI value alignment, the community can vote through tokens to determine what constitutes a "good answer" for models, achieving democratized AI governance.

At the same time, this system also faces two major structural constraints.

· Bandwidth Wall: Despite innovations like DisTrO, physical latency still limits full-scale training of super large models (70B+), with current Web3 AI mostly limited to fine-tuning and inference.

· Reward Hacking: In a highly incentivized network, miners are highly prone to "overfitting" reward rules (gaming the system) rather than enhancing real intelligence. Designing cheat-resistant robust reward functions is an eternal game.

· Malicious Byzantine Worker Attacks: Actively manipulating and poisoning training signals to disrupt model convergence. The key is not just designing cheat-resistant reward functions but also building mechanisms with adversarial robustness.

The integration of Reinforcement Learning and Web3 fundamentally involves rewriting the mechanism of "how intelligence is produced, aligned, and value is distributed." Its evolutionary path can be summarized into three complementary directions:

1. Decentralized Inference Network: From compute mining rigs to policy networks, globally outsource parallel and verifiable rollouts to long-tail GPUs, with short-term focus on verifiable inference markets and medium-term evolution into reinforcement learning subnets clustered by task;

2. Preference and Reward Assetization: From Annotation Labor to Data Equity. Achieving the assetization of preference and reward, transforming high-quality feedback under the Reward Model into a governable, distributable data asset, upgrading from "annotation labor" to "data equity."

3. 'Small and Beautiful' Evolution in Verticals: Nurturing small yet powerful specialized RL Agents in vertical scenarios where results are verifiable and returns are quantifiable, such as DeFi strategy execution, code generation, enabling strategy improvement to be directly linked to value capture and potentially outperforming general-purpose closed-source models.

Overall, the true opportunity of Reinforcement Learning × Web3 lies not in replicating a decentralized version of OpenAI, but in rewriting the "intelligent production relations": making training execution an open compute market, making rewards and preferences governable on-chain assets, shifting the value brought by intelligence away from platform centralization and towards reallocation among trainers, aligners, and users.

Original Article Link