Deep Q-Networks (DQN)

Deep Q-Networks (DQN) combine Reinforcement Learning with Deep Neural Networks to solve problems where the number of possible states is too large for traditional Q-learning tables.

DQN was popularized by DeepMind in 2015, where it learned to play Atari games directly from pixels and achieved human-level performance on many titles.

1?? The Problem DQN Solves

In traditional Q-learning, we store values in a Q-table:

Q(s,a)Q(s,a)Q(s,a)

But this works only when:

• State space is small

• States are discrete

? In real-world problems:

• Images (millions of pixel combinations)

• Robotics (continuous states)

• Complex environments

A Q-table becomes impossible.

2?? The Core Idea of DQN

Instead of storing a Q-table, DQN uses a neural network to approximate the Q-function:

Q(s,a;θ)Q(s,a; \theta)Q(s,a;θ)

Where:

• sss = state

• aaa = action

• θ\thetaθ = neural network weights

The network predicts Q-values for all possible actions given a state.

3?? How DQN Works (Step-by-Step)

Step 1: Observe State

Agent receives state sss

Step 2: Choose Action (ε-greedy)

• With probability ε → explore (random action)

• Otherwise → exploit (choose action with highest Q-value)

Step 3: Take Action

Environment returns:

• Reward rrr

• Next state s′s's′

Step 4: Store Experience

Store tuple:

(s,a,r,s′,done)(s, a, r, s', done)(s,a,r,s′,done)

Step 5: Train the Network

Update network to minimize:

Loss=(Target−Predicted)2Loss = (Target - Predicted)^2Loss=(Target−Predicted)2

Where:

Target=r+γmax?a′Q(s′,a′)Target = r + \gamma \max_{a'} Q(s', a')Target=r+γa′max?Q(s′,a′)

4?? Two Key Innovations in DQN

DQN introduced two major stability improvements:

???? 1. Experience Replay

Instead of learning from sequential data (which is correlated):

• Store experiences in a replay buffer

• Randomly sample mini-batches

• Train on those samples

? Breaks correlation
? Improves stability
? Increases data efficiency

???? 2. Target Network

DQN uses:

• Online network → being trained

• Target network → updated periodically

The target network:

• Stabilizes training

• Prevents oscillating Q-values

5?? DQN Architecture (Conceptually)

Input → Hidden Layers → Output Layer

• Input: State (e.g., image pixels)

• Hidden: Deep neural network (CNN for images)

• Output: Q-values for each action

Example:
If 4 actions → output layer has 4 neurons
Each neuron = Q-value of one action

6?? Why DQN Was Breakthrough

Before DQN:

• RL + Deep Learning didn’t work well together

• Training was unstable

DQN showed:

Neural networks can successfully approximate value functions in high-dimensional spaces.

7?? Where DQN Is Used

• ???? Game playing (Atari)

• ???? Robotics control

• ???? Autonomous navigation

• ???? Resource optimization

• ? Smart grid optimization

(Relevant for energy load balancing and outage decision optimization in utility AI systems.)

8?? Limitations of DQN

• Only works well for discrete action spaces

• Can overestimate Q-values

• Sample inefficient

• Struggles with continuous control

9?? Advanced Variants

• Double DQN

• Dueling DQN

• Rainbow DQN

• Prioritized Experience Replay

???? One-Line Summary

Deep Q-Networks replace the Q-table with a deep neural network to handle complex, high-dimensional state spaces in reinforcement learning.