The Gradient Descent Algorithm is an optimization technique used in machine learning and deep learning to minimize the loss (or cost) function and find the best parameters (weights) for a model.

🔧 How It Works:

1. Initialize parameters (e.g., weights) randomly.
2. Compute the loss using the current parameters.
3. Calculate the gradient (partial derivatives) of the loss with respect to each parameter.
4. Update parameters by moving in the opposite direction of the gradient (because we want to minimize the loss).

1. θ\theta: parameter
2. α\alpha: learning rate (step size)
3. ∇J(θ): gradient of the cost function
4. Repeat the process until convergence (i.e., when the change in loss is very small).

---

📦 Types of Gradient Descent:

Type	Description
Batch Gradient Descent	Uses the whole dataset to compute the gradient.
Stochastic Gradient Descent (SGD)	Uses one data point at a time—faster, but more noisy.
Mini-Batch Gradient Descent	Uses a small batch of data—balances speed and accuracy.

---

📈 Why It’s Important:

• Used in linear regression, logistic regression, neural networks, and many other models.
• Helps models learn from data by iteratively improving accuracy.

Visual Intuition:

Imagine you’re standing on a hill (the loss surface), and your goal is to reach the bottom (minimum loss).
You look at the slope and take steps downhill (opposite to the gradient). The size of your step is the learning rate.

🐍 Python Code Example: Gradient Descent for Linear Regression

import numpy as np
import matplotlib.pyplot as plt

Example data

X = np.array([1, 2, 3, 4, 5])
y = np.array([3, 4, 2, 4, 5])

Parameters initialization

m = 0 # slope
b = 0 # intercept
alpha = 0.01 # learning rate
epochs = 1000
n = len(X)

Gradient Descent

for _ in range(epochs):
y_pred = m * X + b
error = y_pred – y

# Compute gradients
dm = (2/n) * np.dot(error, X)
db = (2/n) * np.sum(error)

# Update parameters
m -= alpha * dm
b -= alpha * db

print(f”Final slope (m): {m:.2f}, intercept (b): {b:.2f}”)

Plotting

plt.scatter(X, y, color=’blue’, label=’Data points’)
plt.plot(X, m * X + b, color=’red’, label=’Fitted line’)
plt.legend()
plt.title(“Linear Regression using Gradient Descent”)
plt.xlabel(“X”)
plt.ylabel(“y”)
plt.show()