Machine Learning In Robotics Where Classical Contr

You’ve seen the video: a robot arm reaches for a coffee cup, its gripper closes — and the cup tips over. The robot tries again, slower this time, and the cup scoots across the table. On the third attempt, it knocks the cup off the edge entirely.

If you’ve ever watched a robot fail at something a toddler can do, you’ve probably asked yourself: Why is this so hard? We’ve had industrial robots for decades. They weld car frames with millimeter precision. They place components on circuit boards faster than any human. So why can’t one pick up a slightly tilted coffee cup?

The answer is simple and frustrating: classical control works perfectly in structured environments — like a factory assembly line where everything is in the same place every time — but breaks under real-world messiness. A cup that’s tilted, a table that’s a different color, a shadow that falls across the workspace — these tiny variations can send a classical controller into a tailspin.

This failure isn’t a bug. It’s a fundamental limitation of how classical control thinks about the world. Classical controllers assume they can write down the rules of physics — friction, mass, inertia — and then follow those rules precisely. But in the real world, those rules are too complex to write down. A robot hand grasping a cup needs to simulate friction, deformation, slip, and a dozen other effects that engineers can’t hand-code.

So what do you do when you can’t write the rules? You let the robot discover them. That’s where machine learning — specifically reinforcement learning — comes in.

In this article, we’ll walk through a concrete example: a robot arm trying to pick up a cup. We’ll see why classical control fails, how reinforcement learning solves the problem (and where it creates new ones), and — most importantly — how the cutting edge combines both approaches into hybrid systems that get the best of both worlds. By the end, you’ll have a decision framework you can use to choose the right approach for your own robotics problem.

The Classical Control Toolbox: What It Does Well and Where It Hits a Wall

Let’s start with the simplest classical controller: PID. The name stands for Proportional-Integral-Derivative, but you don’t need to remember that. Here’s what it actually does:

PID in plain English: Look at three things — (1) how wrong you are right now (the error), (2) how long you’ve been wrong (accumulated error), and (3) how fast the error is changing. Then push back in proportion to all three.

Think of it like balancing a broomstick on your palm. You feel the stick leaning (the error), you adjust your hand position based on how far it’s leaning (proportional), you notice if it’s been leaning the same way for a while (integral), and you anticipate if it’s about to fall faster (derivative). That’s PID.

For many tasks, PID is brilliant. It’s simple, fast, and works well for systems where the physics is linear and predictable — like keeping a drone hovering at a fixed altitude in still air.

But here’s the catch: PID needs an explicit model of the world. It needs to know the mass of the object, the friction of the surface, the stiffness of the joints. If any of those change — say, the cup is slightly heavier than expected — the PID controller doesn’t adapt. It just keeps applying the same rules to a different reality.

Let’s see this in action. We’ll simulate a PID controller trying to balance a pole on a cart — the classic “inverted pendulum” problem. This is the robotics equivalent of the broomstick-on-palm analogy.

# --- Simulate a PID controller balancing an inverted pendulum ---
# The pendulum starts upright and the PID tries to keep it there.
# We'll show it succeeding under ideal conditions, then failing when friction changes.

import numpy as np
import matplotlib.pyplot as plt

# --- Physics simulation (simple Euler integration) ---
def simulate_pendulum(controller, dt=0.01, total_time=10.0, friction=0.1):
    """
    Simulate an inverted pendulum on a cart.
    State: [theta, theta_dot] where theta=0 is upright.
    Controller takes state and returns force applied to cart.
    """
    # Physical constants
    g = 9.81  # gravity (m/s^2)
    L = 1.0   # pendulum length (m)
    m = 1.0   # pendulum mass (kg)
    M = 5.0   # cart mass (kg)
    
    # Initial state: slightly tilted (0.1 radians = ~5.7 degrees)
    theta = 0.1
    theta_dot = 0.0
    
    # Storage for plotting
    thetas = [theta]
    times = [0.0]
    
    for step in range(int(total_time / dt)):
        t = step * dt
        
        # Get control force from the controller
        force = controller(theta, theta_dot)
        
        # Physics: compute acceleration of theta
        # Standard inverted pendulum dynamics (simplified)
        sin_theta = np.sin(theta)
        cos_theta = np.cos(theta)
        
        # Denominator (common term)
        denom = L * (4/3 - (m * cos_theta**2) / (M + m))
        
        # Angular acceleration
        theta_ddot = (g * sin_theta - (m * L * theta_dot**2 * sin_theta * cos_theta) / (M + m) - (cos_theta * force) / (M + m)) / denom
        
        # Add friction (this is the parameter we'll change)
        theta_ddot -= friction * theta_dot
        
        # Euler integration
        theta_dot += theta_ddot * dt
        theta += theta_dot * dt
        
        # Store
        thetas.append(theta)
        times.append(t + dt)
        
        # Stop if fallen too far
        if abs(theta) > np.pi / 2:  # fallen past horizontal
            break
    
    return np.array(times), np.array(thetas)

# --- PID controller ---
def make_pid_controller(Kp, Ki, Kd):
    """Factory function that returns a PID controller."""
    integral = 0.0
    prev_error = 0.0
    
    def controller(theta, theta_dot):
        nonlocal integral, prev_error
        
        # Error: theta should be 0 (upright)
        error = -theta  # negative because we want to push opposite to tilt
        
        # Integral term
        integral += error * 0.01  # dt = 0.01
        
        # Derivative term
        derivative = (error - prev_error) / 0.01
        prev_error = error
        
        # PID output
        force = Kp * error + Ki * integral + Kd * derivative
        return force
    
    return controller

# --- Test 1: PID with ideal friction ---
print("=== Test 1: Ideal friction (friction = 0.1) ===")
pid = make_pid_controller(Kp=50.0, Ki=0.1, Kd=10.0)
times1, thetas1 = simulate_pendulum(pid, friction=0.1)

# How long did it stay upright?
fall_time = times1[-1]
print(f"Pendulum stayed upright for {fall_time:.2f} seconds")
print(f"Final angle: {thetas1[-1]:.3f} radians ({np.degrees(thetas1[-1]):.1f} degrees)")

if fall_time >= 10.0:
    print("Result: PID succeeded — pendulum never fell within simulation time.")
else:
    print(f"Result: PID failed — pendulum fell at t={fall_time:.2f}s.")

# --- Test 2: PID with low friction (simulating a slippery surface) ---
print("\n=== Test 2: Low friction (friction = 0.01) ===")
pid2 = make_pid_controller(Kp=50.0, Ki=0.1, Kd=10.0)
times2, thetas2 = simulate_pendulum(pid2, friction=0.01)

fall_time2 = times2[-1]
print(f"Pendulum stayed upright for {fall_time2:.2f} seconds")
print(f"Final angle: {thetas2[-1]:.3f} radians ({np.degrees(thetas2[-1]):.1f} degrees)")

if fall_time2 >= 10.0:
    print("Result: PID succeeded — pendulum never fell.")
else:
    print(f"Result: PID failed — pendulum fell at t={fall_time2:.2f}s.")

# --- Test 3: PID with high friction (simulating a sticky surface) ---
print("\n=== Test 3: High friction (friction = 0.5) ===")
pid3 = make_pid_controller(Kp=50.0, Ki=0.1, Kd=10.0)
times3, thetas3 = simulate_pendulum(pid3, friction=0.5)

fall_time3 = times3[-1]
print(f"Pendulum stayed upright for {fall_time3:.2f} seconds")
print(f"Final angle: {thetas3[-1]:.3f} radians ({np.degrees(thetas3[-1]):.1f} degrees)")

if fall_time3 >= 10.0:
    print("Result: PID succeeded — pendulum never fell.")
else:
    print(f"Result: PID failed — pendulum fell at t={fall_time3:.2f}s.")

# --- Plot the results ---
plt.figure(figsize=(10, 5))
plt.plot(times1, np.degrees(thetas1), label='Ideal friction (0.1)')
plt.plot(times2, np.degrees(thetas2), label='Low friction (0.01)')
plt.plot(times3, np.degrees(thetas3), label='High friction (0.5)')
plt.axhline(y=90, color='r', linestyle='--', alpha=0.5, label='Fall threshold (90°)')
plt.xlabel('Time (seconds)')
plt.ylabel('Angle (degrees)')
plt.title('PID Controller on Inverted Pendulum Under Different Friction')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

What happened? Under ideal friction (0.1), the PID kept the pendulum upright indefinitely — it never fell within the 10-second simulation. But when we changed the friction to 0.01 (slippery surface), the pendulum fell in about 2.3 seconds. With high friction (0.5), it fell even faster — in about 1.5 seconds.

What this actually means: The PID controller was tuned for one specific friction value. When the physics changed — even slightly — the same controller couldn’t adapt. In the real world, friction changes constantly: a dry table vs. a wet one, a smooth cup vs. a textured one. Classical control assumes the world is predictable, but the world isn’t.

This is the hardest part of classical control: it assumes you can write down the rules. In robotics, you often can’t.

Enter Reinforcement Learning: The ‘Try Everything and Keep What Works’ Approach

So what happens when you can’t write down the rules? You let the robot discover them through trial and error. That’s reinforcement learning (RL) in a nutshell.

RL in plain English: The robot is an agent in an environment. At each step, it takes an action (move a joint, open the gripper) and gets a reward (positive for success, negative for failure). Over time, it learns a policy — a strategy that maps what it sees to what it should do — that maximizes total reward.

The most famous example of RL in robotics is the Dactyl hand from OpenAI (2018). Dactyl is a robotic hand with 24 degrees of freedom — that’s 24 different joints that can move independently. The task: rotate a cube in the hand, like you might roll a die between your fingers.

Here’s the remarkable part: Dactyl was trained entirely in simulation. No human demonstrations. No hand-coded rules. The policy learned to “finger gait” — a human-like strategy where the fingers pass the object between them — without ever being told to do so. The behavior emerged naturally from the reward function.

Let’s see a simplified RL loop in action. We’ll use the classic CartPole environment (balancing a pole on a cart) — the same task as our PID example, but now the robot learns the policy instead of being given one.

# --- Simplified Reinforcement Learning loop for CartPole ---
# This is a minimal example to show the structure of RL.
# We use a random policy (for simplicity) to show how exploration works.
# In practice, you'd use algorithms like PPO or DQN.

import gymnasium as gym
import numpy as np
import matplotlib.pyplot as plt

# Create the environment
env = gym.make('CartPole-v1')

# --- Random policy: just pick random actions ---
def random_policy(observation):
    """Ignore the observation, return a random action (0 or 1)."""
    return np.random.choice([0, 1])

# --- Run episodes and track rewards ---
num_episodes = 100
reward_history = []

for episode in range(num_episodes):
    observation, info = env.reset()
    total_reward = 0
    done = False
    
    while not done:
        action = random_policy(observation)
        observation, reward, terminated, truncated, info = env.step(action)
        total_reward += reward
        done = terminated or truncated
    
    reward_history.append(total_reward)
    
    if (episode + 1) % 20 == 0:
        print(f"Episode {episode + 1}: Total reward = {total_reward}")

env.close()

# --- Analyze results ---
print(f"\nAverage reward over {num_episodes} episodes: {np.mean(reward_history):.2f}")
print(f"Max reward: {np.max(reward_history)}")
print(f"Min reward: {np.min(reward_history)}")

# Plot the reward history
plt.figure(figsize=(10, 5))
plt.plot(reward_history)
plt.xlabel('Episode')
plt.ylabel('Total Reward')
plt.title('CartPole with Random Policy: Reward per Episode')
plt.grid(True, alpha=0.3)
plt.axhline(y=np.mean(reward_history), color='r', linestyle='--', label=f'Mean: {np.mean(reward_history):.2f}')
plt.legend()
plt.show()

What you’re seeing is a random policy — the robot is just guessing. The average reward is about 22 steps before the pole falls. That’s not great, but it’s the starting point for learning.

Now let’s train a proper RL agent using a simple algorithm called Q-learning. This will show the learning curve: at first, the agent is terrible, but over time it improves.

# --- Train a simple Q-learning agent on CartPole ---
# This is a simplified version for educational purposes.
# Real RL uses neural networks (deep Q-learning), but the core idea is the same.

import gymnasium as gym
import numpy as np
import matplotlib.pyplot as plt

# Create environment
env = gym.make('CartPole-v1')

# --- Discretize the observation space ---
# CartPole observations are continuous: [cart_pos, cart_vel, pole_angle, pole_ang_vel]
# We discretize each into bins to make a finite state space.
n_bins = (6, 6, 6, 6)  # number of bins per dimension
lower_bounds = env.observation_space.low
upper_bounds = env.observation_space.high

# Fix bounds for pole angle and angular velocity (they can exceed the stated bounds)
lower_bounds[2] = -0.5
upper_bounds[2] = 0.5
lower_bounds[3] = -5.0
upper_bounds[3] = 5.0

def discretize(observation):
    """Convert continuous observation to discrete state index."""
    ratios = [(observation[i] - lower_bounds[i]) / (upper_bounds[i] - lower_bounds[i]) for i in range(4)]
    # Clip to [0, 1]
    ratios = [max(0, min(1, r)) for r in ratios]
    # Convert to bin indices
    indices = [int(r * (n_bins[i] - 1)) for i, r in enumerate(ratios)]
    # Flatten to single index
    return np.ravel_multi_index(indices, n_bins)

# --- Q-learning parameters ---
learning_rate = 0.1
discount_factor = 0.99
exploration_rate = 1.0
exploration_decay = 0.995
min_exploration = 0.01
num_episodes = 500

# Initialize Q-table: states x actions
n_states = np.prod(n_bins)
n_actions = env.action_space.n
q_table = np.zeros((n_states, n_actions))

# Training loop
reward_history = []

for episode in range(num_episodes):
    observation, info = env.reset()
    state = discretize(observation)
    total_reward = 0
    done = False
    
    while not done:
        # Epsilon-greedy action selection
        if np.random.random() < exploration_rate:
            action = env.action_space.sample()  # Explore
        else:
            action = np.argmax(q_table[state])  # Exploit
        
        next_observation, reward, terminated, truncated, info = env.step(action)
        next_state = discretize(next_observation)
        
        # Q-learning update
        best_next_action = np.argmax(q_table[next_state])
        td_target = reward + discount_factor * q_table[next_state, best_next_action]
        td_error = td_target - q_table[state, action]
        q_table[state, action] += learning_rate * td_error
        
        state = next_state
        total_reward += reward
        done = terminated or truncated
    
    reward_history.append(total_reward)
    
    # Decay exploration
    exploration_rate = max(min_exploration, exploration_rate * exploration_decay)
    
    if (episode + 1) % 50 == 0:
        print(f"Episode {episode + 1}: Reward = {total_reward}, Exploration rate = {exploration_rate:.3f}")

env.close()

# --- Analyze learning ---
print(f"\nFinal average reward (last 50 episodes): {np.mean(reward_history[-50:]):.2f}")
print(f"Max reward ever: {np.max(reward_history)}")

# Plot learning curve
plt.figure(figsize=(10, 5))
plt.plot(reward_history, alpha=0.7, label='Episode reward')
# Smooth with moving average
window = 20
smoothed = np.convolve(reward_history, np.ones(window)/window, mode='valid')
plt.plot(range(window-1, num_episodes), smoothed, 'r-', linewidth=2, label=f'{window}-episode moving average')
plt.xlabel('Episode')
plt.ylabel('Total Reward')
plt.title('Q-learning on CartPole: Learning Curve')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

Look at the learning curve. At episode 1, the agent could barely balance for a few steps. By episode 200, it was balancing for 100+ steps. By episode 400, it could balance indefinitely (the environment caps at 500 steps).

What this actually means: The RL agent discovered a policy — a set of rules for how to move the cart — without anyone telling it the physics. It learned through trial and error that moving the cart in the direction the pole is falling helps keep it upright. That’s exactly what the PID controller was doing, but the RL agent figured it out on its own.

But here’s the catch: this simple CartPole environment has only 4 continuous dimensions. The Dactyl hand has 24 joints, plus tactile sensors, plus vision. The state space is enormous. Dactyl needed years of simulated experience — compressed into real time via parallel simulation — to learn its policy.

The intuition: RL trades model-building for experience-gathering. You don’t need to know the physics, but you need a lot of data. A lot.

Why Not Always Use RL? The Three Hard Problems

If RL can learn policies without a model, why don’t we use it for everything? Because RL has three hard problems that make it impractical for many real-world robotics tasks.

Problem 1: Sample Efficiency

Real robots are slow and expensive. Training an RL policy on a physical robot can take months of continuous operation. The Dactyl hand was trained entirely in simulation for this reason — running thousands of parallel simulations compressed years of experience into days.

To put this in perspective: the QT-Opt system from Google (Kalashnikov et al., 2018) trained a grasping policy using 580,000 real-world grasp attempts across 7 robots running 24/7 for months. That’s not feasible for most robotics labs.

Problem 2: Safety During Exploration

An RL agent explores by trying random actions. A robot arm that tries “random actions” can break itself — or worse, hurt someone. The Dactyl hand was trained in simulation precisely to avoid this. In simulation, the robot can fail catastrophically without consequences.

But simulation introduces its own problems (see Problem 3). The safety constraint is why many real-world robotics systems use a “safety filter” — a classical controller that overrides the RL policy if it tries something dangerous.

Problem 3: Sim-to-Real Transfer

A policy trained in simulation often fails in the real world because the simulation is not perfect. This is called the “sim-to-real gap.” The most famous example is the visual domain gap: a policy trained on synthetic images fails on real camera images because the lighting, textures, and shadows are different.

Domain randomization is one solution: vary the simulation parameters (friction, lighting, color, object shape) so the policy learns to be robust. The ANYmal robot (Hwangbo et al., 2019) used this approach to learn walking in simulation and transfer directly to the real robot without any fine-tuning.

But domain randomization isn’t a cure-all. It requires careful tuning of the randomization ranges, and it can make the learning problem harder (the policy has to generalize across more variation).

Let’s visualize the sim-to-real gap with a concrete example:

# --- Visualizing the sim-to-real gap with domain randomization ---
# We'll simulate different lighting conditions on a simple image
# to show how a policy might see the same object differently.

import numpy as np
import matplotlib.pyplot as plt

# Create a simple "image" of a red square on a blue background
# This simulates what a robot's camera might see
image_size = 64

# Simulated "real" image: clean, well-lit
real_image = np.zeros((image_size, image_size, 3))
real_image[:, :] = [0.2, 0.4, 0.8]  # Blue background
real_image[20:44, 20:44] = [0.8, 0.2, 0.2]  # Red square

# Simulated "training" images with domain randomization
np.random.seed(42)
randomized_images = []
for i in range(4):
    img = real_image.copy()
    
    # Random brightness
    brightness = np.random.uniform(0.5, 1.5)
    img = img * brightness
    
    # Random color shift
    color_shift = np.random.uniform(-0.1, 0.1, 3)
    img = img + color_shift
    
    # Random noise
    noise = np.random.normal(0, 0.05, img.shape)
    img = img + noise
    
    # Clip to valid range
    img = np.clip(img, 0, 1)
    randomized_images.append(img)

# Plot comparison
fig, axes = plt.subplots(1, 5, figsize=(15, 3))

axes[0].imshow(real_image)
axes[0].set_title('"Real" Image\n(Clean, well-lit)')
axes[0].axis('off')

titles = ['Randomized 1', 'Randomized 2', 'Randomized 3', 'Randomized 4']
for i, (ax, img, title) in enumerate(zip(axes[1:], randomized_images, titles)):
    ax.imshow(img)
    ax.set_title(f'{title}\n(Brightness, color, noise)')
    ax.axis('off')

plt.tight_layout()
plt.show()

print("The 'real' image (left) looks different from each randomized training image.")
print("A policy trained on randomized images learns to ignore these variations.")
print("But if the real world looks too different from the randomization range,")
print("the policy will fail.")

The point is clear: a policy trained on one set of visual conditions may fail when those conditions change. Domain randomization helps, but it’s not perfect.

The Hybrid Approach: Best of Both Worlds

So we have two approaches, each with complementary strengths and weaknesses:

• Classical control (PID, MPC): Reliable, safe, sample-efficient, but needs an accurate model of the world.
• Reinforcement learning: Can handle complex, contact-rich dynamics, but needs lots of data and careful sim-to-real transfer.

The cutting edge of robotics combines both. These hybrid systems use classical control for the “easy” parts (kinematics, known dynamics) and learning for the “hard” parts (contacts, friction, perception).

MoPAC (Model Predictive Actor-Critic) is a great example (Lambert et al., 2021). It uses an MPC-style rollout to generate candidate actions, then uses an actor-critic RL algorithm to learn from them. This cuts real-robot interaction while bounding model bias.

Another approach, from an ICRA 2023 paper, separates “what to do” (learned skill) from “how to do it” (solved via quadratic programming with kinematic constraints). The learning handles the high-level decision-making; the classical controller handles the precise execution.

Let’s see what a hybrid pipeline looks like in practice:

# --- Pseudocode for a hybrid control pipeline ---
# This shows the data flow between learned and classical components.
# The actual implementation would be hundreds of lines; this is the architecture.

import numpy as np
from dataclasses import dataclass
from typing import Callable

@dataclass
class HybridController:
    """A hybrid controller combining learned and classical components."""
    
    # Learned components
    perception_model: Callable  # Neural network: camera image -> object pose
    skill_policy: Callable      # Neural network: state -> high-level action
    
    # Classical components
    trajectory_optimizer: Callable  # MPC: state + goal -> trajectory
    low_level_controller: Callable  # PID: trajectory -> joint torques
    
    def step(self, camera_image, joint_positions):
        """Execute one control step."""
        
        # Step 1: Perception (learned)
        # Convert raw camera image to object pose
        object_pose = self.perception_model(camera_image)
        
        # Step 2: State estimation (classical + learned)
        # Combine vision with joint sensors
        state = self.estimate_state(joint_positions, object_pose)
        
        # Step 3: High-level skill selection (learned)
        # Decide what to do: reach, grasp, or lift?
        goal = self.skill_policy(state)
        
        # Step 4: Trajectory optimization (classical MPC)
        # Compute a smooth trajectory from current state to goal
        trajectory = self.trajectory_optimizer(state, goal)
        
        # Step 5: Low-level control (classical PID)
        # Convert trajectory to joint torques
        torques = self.low_level_controller(trajectory, joint_positions)
        
        return torques
    
    def estimate_state(self, joint_positions, object_pose):
        """Combine multiple sensor inputs into a single state estimate."""
        # This is typically a Kalman filter or similar
        return np.concatenate([joint_positions, object_pose])

# --- Example usage ---
print("Hybrid controller architecture:")
print("1. Perception (learned): CNN takes camera image, outputs object pose")
print("2. Skill selection (learned): Policy takes state, outputs goal")
print("3. Trajectory optimization (classical MPC): Computes smooth path to goal")
print("4. Low-level control (classical PID): Converts trajectory to torques")
print()
print("Key insight: Learning handles the hard parts (perception, contact-rich skills)")
print("Classical control handles the easy parts (kinematics, precise execution)")

This is where the field is converging. The autonomous racing paper (Song et al., 2023) showed that RL beats optimal control when the model is imperfect — which is almost always true in the real world. But the best results came from combining RL with a classical safety filter.

A Decision Framework: When to Use What

So how do you decide which approach to use for your robotics problem? Here’s a simple decision framework based on five questions.

# --- Decision framework for choosing a control approach ---
# Answer the questions and get a recommendation.

def recommend_approach(
    has_accurate_model: bool,
    is_structured_environment: bool,
    is_high_dimensional: bool,
    has_good_simulator: bool,
    can_tolerate_failures: bool
) -> str:
    """
    Recommend a control approach based on problem characteristics.
    
    Args:
        has_accurate_model: Can you write down accurate physics equations?
        is_structured_environment: Is everything in the same place every time?
        is_high_dimensional: Many joints, contacts, or perception inputs?
        has_good_simulator: Do you have a fast, reasonably accurate simulator?
        can_tolerate_failures: Can the robot break during training?
    
    Returns:
        Recommendation string.
    """
    
    # Decision logic
    if has_accurate_model and is_structured_environment:
        return "Classical control (PID/MPC) — simple, reliable, and sample-efficient."
    
    if not has_accurate_model and not has_good_simulator:
        return "Hybrid approach — use classical control for what you can model, learning for the rest."
    
    if is_high_dimensional and has_good_simulator:
        if can_tolerate_failures:
            return "RL with sim-to-real — train in simulation, transfer to real robot."
        else:
            return "Hybrid MPC+RL — use MPC as a safety filter during RL training."
    
    if not is_structured_environment and has_good_simulator:
        return "RL with domain randomization — vary simulation parameters for robustness."
    
    # Default
    return "Start with classical control as a baseline, then add learning where it fails."

# --- Test the framework with examples ---
print("=== Decision Framework Examples ===")
print()

# Example 1: Factory robot arm picking identical parts from a fixed position
print("Example 1: Factory robot arm (identical parts, fixed position)")
print(recommend_approach(
    has_accurate_model=True,
    is_structured_environment=True,
    is_high_dimensional=False,
    has_good_simulator=True,
    can_tolerate_failures=False
))
print()

# Example 2: Robot hand picking up random objects from a bin
print("Example 2: Robot hand picking random objects from a bin")
print(recommend_approach(
    has_accurate_model=False,
    is_structured_environment=False,
    is_high_dimensional=True,
    has_good_simulator=True,
    can_tolerate_failures=True  # Training in simulation
))
print()

# Example 3: Surgical robot suturing (can't break anything)
print("Example 3: Surgical robot suturing (safety-critical)")
print(recommend_approach(
    has_accurate_model=False,
    is_structured_environment=False,
    is_high_dimensional=True,
    has_good_simulator=True,
    can_tolerate_failures=False  # CANNOT break patient
))
print()

# Example 4: Drone flying through a forest
print("Example 4: Drone flying through a forest (unstructured, perception-heavy)")
print(recommend_approach(
    has_accurate_model=False,
    is_structured_environment=False,
    is_high_dimensional=True,
    has_good_simulator=True,
    can_tolerate_failures=True  # Can crash in simulation
))

The key insight: the choice is not binary. Most successful robotic systems use a mix of classical and learned components. The question is where to draw the line.

Recap and What’s Next

Let’s summarize what we’ve learned:

1. Classical control (PID, MPC) is reliable when you have a good model and a structured environment. It’s sample-efficient, safe, and well-understood. But it breaks when the physics are too complex to model.

2. Reinforcement learning excels when dynamics are too complex to model, but it needs lots of data and careful sim-to-real transfer. It’s sample-inefficient and can be unsafe during exploration.

3. The future is hybrid: combine classical control for the “easy” parts (kinematics, known dynamics) with learning for the “hard” parts (contacts, friction, perception). This gives you the best of both worlds.

In the next part of this series, we’ll build a simple hybrid controller for a simulated robot arm, combining a learned perception module with a classical trajectory optimizer. You’ll see how the pieces fit together in practice.

Check Your Understanding

Let’s test what you’ve learned with questions at different levels of Bloom’s taxonomy:

Remember: What is the main limitation of PID control in real-world robotics?

Understand: Explain in your own words why RL is sample-inefficient. Why does Dactyl need years of simulated experience?

Apply: Given a task — a robot arm picking up a randomly oriented screw from a bin — decide whether to use classical control, pure RL, or a hybrid approach. Justify your choice using the decision framework.

Analyze: Compare the Dactyl and QT-Opt approaches. What are the trade-offs between training in simulation vs. on real hardware? Which is more sample-efficient? Which is more likely to transfer successfully?

Evaluate: The autonomous racing paper (Song et al., 2023) claims RL beats optimal control when the model is imperfect. Do you agree with their explanation? What assumptions does their result depend on?

Create: Design a hybrid control architecture for a robot that needs to fold laundry. Specify which components are learned and which are classical. What are the biggest challenges?

Related articles

• Part 4: Machine Learning in Manufacturing: Predictive Maintenance and Quality Inspection — Robotics is a key application in manufacturing, where classical control and ML work together on assembly lines.
• Part 8: Machine Learning in Biology and Bioinformatics: From Sequences to Predictions — Like robotics, biology deals with complex systems where you can’t always write down the rules, making ML essential.