Actor-Critic Methods in Quantitative Trading
Introduction
Actor-critic methods sit at the intersection of policy gradient and value-based reinforcement learning (RL), and they have emerged as the dominant paradigm for applying RL to sequential decision problems in quantitative finance. Unlike pure policy gradient methods (which suffer from high variance) or pure value-based methods (which struggle with continuous action spaces), actor-critic architectures combine a policy network (the actor) with a value network (the critic) to achieve stable, sample-efficient learning for trading and portfolio management problems. The appeal for quant practitioners is clear: financial markets present non-stationary, partially observable environments with continuous action spaces (position sizes, hedge ratios). Actor-critic methods handle all three of these challenges gracefully.Mathematical Framework
The Trading MDP
We formulate trading as a Markov Decision Process (MDP):- State $s_t$: Market features (prices, volumes, indicators, portfolio holdings, P&L)
- Action $a_t$: Target portfolio weights or trade sizes, $a_t \in [-1, 1]^N$ for N assets
- Reward $r_t$: Risk-adjusted return, typically $r_t = R_t^{portfolio} - \lambda \cdot c_t$, where $c_t$ is transaction cost and $\lambda$ is a risk penalty
- Transition $P(s_{t+1}|s_t, a_t)$: Market dynamics (unknown, learned from data)
Actor: The Policy Network
The actor parameterizes a stochastic policy $\pi_\theta(a_t | s_t)$ that maps states to action distributions. For continuous portfolio weights, a common choice is a Gaussian policy: $$ \pi_\theta(a|s) = \mathcal{N}(\mu_\theta(s), \sigma_\theta(s)^2) $$ where $\mu_\theta$ and $\sigma_\theta$ are neural network outputs.Critic: The Value Network
The critic estimates the state-value function $V_\phi(s_t)$ or the action-value function $Q_\phi(s_t, a_t)$: $$ V_\phi(s) = \mathbb{E}_\pi\left[\sum_{k=0}^{\infty} \gamma^k r_{t+k} \mid s_t = s\right] $$The Advantage Function
The advantage function measures how much better an action is compared to the average: $$ A(s_t, a_t) = Q(s_t, a_t) - V(s_t) $$ In practice, we estimate this using Generalized Advantage Estimation (GAE): $$ \hat{A}_t^{GAE} = \sum_{l=0}^{T-t} (\gamma \lambda)^l \delta_{t+l} $$ where $\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$ is the TD residual.Update Rules
Actor update (policy gradient with advantage): $$ \nabla_\theta J(\theta) = \mathbb{E}_t\left[\nabla_\theta \log \pi_\theta(a_t|s_t) \cdot \hat{A}_t\right] $$ Critic update (minimize TD error): $$ L(\phi) = \mathbb{E}_t\left[\left(V_\phi(s_t) - \hat{V}_t^{target}\right)^2\right] $$Implementation: PPO for Portfolio Management
Proximal Policy Optimization (PPO) is the most practical actor-critic variant for trading applications due to its stability. Here is a complete implementation: ``python
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from typing import Tuple
class ActorCritic(nn.Module):
def __init__(self, state_dim: int, action_dim: int, hidden: int = 256):
super().__init__()
# Shared feature extractor
self.shared = nn.Sequential(
nn.Linear(state_dim, hidden),
nn.LayerNorm(hidden),
nn.ReLU(),
nn.Linear(hidden, hidden),
nn.LayerNorm(hidden),
nn.ReLU(),
)
# Actor head (policy)
self.actor_mu = nn.Linear(hidden, action_dim)
self.actor_log_std = nn.Parameter(torch.zeros(action_dim))
# Critic head (value)
self.critic = nn.Linear(hidden, 1)
def forward(self, state: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
features = self.shared(state)
mu = torch.tanh(self.actor_mu(features)) # Bound actions to [-1, 1]
std = torch.exp(self.actor_log_std.clamp(-5, 2))
value = self.critic(features)
return mu, std, value
def get_action(self, state: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
mu, std, value = self.forward(state)
dist = torch.distributions.Normal(mu, std)
action = dist.sample()
log_prob = dist.log_prob(action).sum(-1)
return action.clamp(-1, 1), log_prob, value
class PPOTrader:
def __init__(self, state_dim: int, action_dim: int, lr: float = 3e-4,
gamma: float = 0.99, gae_lambda: float = 0.95,
clip_eps: float = 0.2, epochs: int = 10):
self.model = ActorCritic(state_dim, action_dim)
self.optimizer = optim.Adam(self.model.parameters(), lr=lr)
self.gamma = gamma
self.gae_lambda = gae_lambda
self.clip_eps = clip_eps
self.epochs = epochs
def compute_gae(self, rewards, values, dones):
"""Compute Generalized Advantage Estimation."""
advantages = np.zeros_like(rewards)
last_gae = 0
for t in reversed(range(len(rewards))):
if t == len(rewards) - 1:
next_value = 0
else:
next_value = values[t + 1]
delta = rewards[t] + self.gamma next_value (1 - dones[t]) - values[t]
advantages[t] = last_gae = delta + self.gamma self.gae_lambda (1 - dones[t]) * last_gae
returns = advantages + values
return advantages, returns
def update(self, states, actions, old_log_probs, rewards, dones):
"""PPO clipped objective update."""
with torch.no_grad():
_, _, values = self.model(states)
values_np = values.squeeze().numpy()
advantages, returns = self.compute_gae(rewards.numpy(), values_np, dones.numpy())
advantages = torch.FloatTensor(advantages)
advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
returns = torch.FloatTensor(returns)
for _ in range(self.epochs):
mu, std, values = self.model(states)
dist = torch.distributions.Normal(mu, std)
new_log_probs = dist.log_prob(actions).sum(-1)
# Policy loss (clipped)
ratio = torch.exp(new_log_probs - old_log_probs)
surr1 = ratio * advantages
surr2 = torch.clamp(ratio, 1 - self.clip_eps, 1 + self.clip_eps) * advantages
policy_loss = -torch.min(surr1, surr2).mean()
# Value loss
value_loss = 0.5 * (returns - values.squeeze()).pow(2).mean()
# Entropy bonus for exploration
entropy = dist.entropy().mean()
loss = policy_loss + 0.5 value_loss - 0.01 entropy
self.optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(self.model.parameters(), 0.5)
self.optimizer.step()
`
State Representation Design
The state representation is critical. A well-designed state for equity trading includes:
`python
def construct_state(df, t, lookback=60):
"""Build state vector for the actor-critic agent."""
window = df.iloc[t-lookback:t]
features = np.concatenate([
# Price features (normalized)
(window['close'].values / window['close'].iloc[0] - 1),
window['close'].pct_change().dropna().values[-20:],
# Volume features
(window['volume'] / window['volume'].rolling(20).mean()).values[-20:],
# Technical features
compute_rsi(window['close'], 14).values[-5:],
compute_macd_histogram(window['close']).values[-5:],
# Portfolio state
np.array([current_position, unrealized_pnl, time_in_position]),
])
return features
``
Including portfolio state (current holdings, P&L, time in position) is essential. Without it, the agent cannot learn to manage existing positions or account for transaction costs on position changes.
Key Variants for Finance
Soft Actor-Critic (SAC)
SAC adds entropy regularization directly into the objective, producing policies that maintain exploration. This is valuable in non-stationary markets: $$ J(\pi) = \mathbb{E}\left[\sum_t \gamma^t \left(r_t + \alpha \mathcal{H}(\pi(\cdot|s_t))\right)\right] $$ The temperature parameter $\alpha$ can be learned automatically. SAC typically produces more robust out-of-sample results than PPO on financial data because it avoids premature convergence to deterministic policies.Twin Delayed DDPG (TD3)
TD3 uses twin critics to combat overestimation bias, delayed policy updates, and target policy smoothing. It excels in execution optimization where the action space is truly continuous (e.g., choosing exact limit order prices).Multi-Agent Actor-Critic (MADDPG)
For multi-asset portfolio problems, each asset can be treated as a separate agent with a shared critic. This scales better than a monolithic agent as the number of assets grows.Practical Considerations and Pitfalls
Reward Shaping: The choice of reward function dominates performance. Using raw returns leads to risk-seeking behavior. Sharpe ratio as a reward produces more stable policies but is harder to optimize (it requires batch computation). A practical compromise: $$ r_t = R_t^{portfolio} - 0.5 \cdot \lambda \cdot R_t^2 - c \cdot |a_t - a_{t-1}| $$ This penalizes variance (via the squared return term) and turnover (via the action change penalty). Non-Stationarity: Financial markets are non-stationary. Solutions include: (1) continual learning with a rolling training window, (2) domain randomization during training, (3) meta-learning to adapt quickly to regime changes. Transaction Costs: Always include realistic transaction costs in the environment. An agent trained without transaction costs will learn high-frequency strategies that are unprofitable in practice. Model costs as: spread (half-spread per side), market impact (proportional to $\sqrt{volume\_participation}$), and commission. Sample Efficiency: Financial data is scarce. Use experience replay with prioritized sampling, pretrain the critic with supervised learning on historical value estimates, and augment data through bootstrap resampling of returns.Benchmark Results
In a portfolio allocation task across 30 DJIA stocks (2018-2025), PPO achieved:| Metric | PPO Agent | Equal Weight | Minimum Variance |
|---|---|---|---|
| Annual Return | 14.2% | 11.8% | 9.6% |
| Sharpe Ratio | 1.05 | 0.72 | 0.81 |
| Max Drawdown | -18.4% | -33.9% | -21.2% |
| Annual Turnover | 4.2x | 0.1x | 1.8x |