PyTorch
ScholarEnv

ScholarEnv — OpenEnv for Research Paper's

GitHub Repo HuggingFace Space

T5 Zero-Shot Transfer · +27%

0.1397 → 0.1771 reward on prompt-injection detection. Zero T5 training examples — pure cross-task generalisation.
View on GitHub

Citation Verification (T4) · +33%

0.3604 → 0.4807 reward across 115 graded completions. Veri-R1 design with live CrossRef + Semantic Scholar lookups.
View on GitHub

Formatting Compliance (T1) · +63%

0.1709 → 0.2787 reward across 91 completions. 3-stage Progressive Reward Shaping (arXiv:2512.07478).
View on GitHub

Smoke Run Peak · 58×

0.008 frozen baseline → 0.745 peak reward on T3 claim audit, 25 GRPO steps. F-beta(0.5) grader, no annotation.
View on GitHub

Smoke Run · T3 Claim Audit

Peak reward · T3 claim audit · 416 episodes 0.905
HuggingFace
PyTorch
Meta
Scaler

ScholarEnv 6.7 — an open RL environment
for the HuggingFace Spaces community

Infinite-data, verifiable-reward RL training for scientific paper auditing.
GRPO trains a Qwen2.5-1.5B with Unsloth LoRA on five tasks: format, consistency,
claim verification, citation checking, and adversarial-injection detection.

Open on HuggingFace

Reward Functions

F-beta Score Precision & recall on flagged claims.
× 0.60
Specificity Exact table_id and value matching.
× 0.15
Reasoning Quality Grounded chain-of-thought scoring.
× 0.25
Sum of weights Pre-weighted total reward in [0, 1].
1.00

Algorithms & Techniques

HuggingFace
GRPO trl 0.22.2 · DeepSeek-R1
Group Relative Policy Optimisation — removes the critic from PPO using sampled-group advantage.
PyTorch
LoRA + Unsloth PyTorch · 60% less VRAM
Low-rank adaptation with Unsloth-fused kernels and Flash Attention 2 on PyTorch.
Scaler
RLVR Scaler School of Tech
Programmatic ground-truth rewards — no learned reward model required.
Meta
Chain-of-Thought Meta · Grounded reasoning
Reasoning-trace reward (×0.25) for logical coherence and factual grounding.
Meta
OpenEnv Meta × PyTorch protocol
Standard for verifiable, infinite-diverse RL environments — five paper domains.
HuggingFace
Firecrawl + arXiv Real-paper evaluation
OOD validation — trained model audits real arXiv papers fetched via Firecrawl.

Tasks T1 — T5

Five verifiable audit tasks. Four are trained jointly via GRPO; the fifth is held out for zero-shot generalisation evaluation.
T1
Format Compliance 0.1709 → 0.2787 · +63% · n=91
T2
Internal Consistency 0.0187 → 0.0176 · v8 schema fix
T3
Claim Auditing Smoke peak 0.745 · 58× over 0.008 baseline
Primary
T4
Citation Verification 0.3604 → 0.4807 · +33% · n=115
T5
Injection Detection 0.1397 → 0.1771 · +27% · zero-shot · n=19
Zero-shot

Key Papers & References 16 implemented

# Paper Where implemented Tag
01 RLVE — Adaptive Verifiable Environments arXiv:2511.07317 server/paper_generator.py Core env
02 PRS — Progressive Reward Shaping arXiv:2512.07478 graders/formatting_grader.py T1 reward
03 DAPO — Structured JSON Training arXiv:2503.14476 GRPOConfig(loss_type="dapo") GRPO loss
04 PBRS — Potential-Based Reward Shaping Ng, Harada & Russell, ICML 1999 server/reward_shaper.py Navigation
05 AdaRFT — Adaptive Curriculum arXiv:2504.05520 server/curriculum.py Curriculum
06 Agent-RLVR — Partial Credit arXiv:2506.11425 graders/audit_grader.py T3 grader
07 UniDoc-RL — Hierarchical Actions arXiv:2604.14967 models.py (action space) v6 new
08 ProRL Agent — Rollout-as-Service arXiv:2603.18815 server/app.py (session pool) v6 new
09 RAGEN-2 — SNR Filtering arXiv:2604.06268 train.py snr_filter_batch() v6 new
10 Experience Replay arXiv:2604.08706 train.py ExperienceReplayBuffer v6 new
11 Abstain-R1 — Calibrated Abstention arXiv:2604.17073 train.py CITATION_ABSTAIN_REWARD v6 new
12 Veri-R1 — Claim Verification RL arXiv:2510.01932 Task 3 design rationale T3 basis
13 CiteAudit — Citation Hallucination arXiv:2602.23452 Task 4 design rationale T4 basis
14 GDPO — Decoupled Reward Normalisation ★ arXiv:2601.05242 GRPOConfig reward_aggregation NEW v6
15 Dr. GRPO — Remove Std Bias ★ arXiv:2503.20783 GRPOConfig scale_rewards="batch" NEW v6
16 AgentReview — Peer Review Bias arXiv:2406.12708 Domain motivation Motivation
383 graded completions 5 tasks · 5 domains Free T4
View full bibliography

Training Configuration

Step 200 / 200
T3 smoke peak · 25 steps 0.745 peak
OpenEnv compliant · 5 tasks 383 graded completions
Base model Qwen2.5-1.5B-Instruct · 4-bit
Hub
PEFT method Unsloth LoRA · r=16 · α=32 · q/k/v/o
GitHub
RL algorithm GRPO · DAPO loss · scale_rewards=batch
Docs
Hyperparameters lr 5e-6 · num_generations 4 · effective bs 4
200 steps
Dataset 200 rows · 4 tasks × 50 papers · T5 held out
5 domains
Hardware Colab T4 · 14 GB VRAM · free tier
Free T4
train.py Push to Hub