Day 1 of 100 days of building

04 Jul, 2026

Neural Network Phase

Project

Building a Neural Network from Scratch

Today I worked through Andrew Trask's "A Neural Network in 11 Lines of Python".

Goal: Understand the intuition behind forward propagation and backpropagation rather than simply copying the implementation.

The Code

import numpy as np

X = np.array([
    [0,0,1],
    [0,1,1],
    [1,0,1],
    [1,1,1]
])

y = np.array([[0,1,1,0]]).T

w1 = 2*np.random.random((3,4)) - 1
w2 = 2*np.random.random((4,1)) - 1

for j in range(6000):

    l1 = 1/(1+np.exp(-(np.dot(X,w1))))
    l2 = 1/(1+np.exp(-(np.dot(l1,w2))))

    l2_delta = (y-l2)*(l2*(1-l2))
    l1_delta = l2_delta.dot(w2.T)*(l1*(1-l1))

    w2 += l1.T.dot(l2_delta)
    w1 += X.T.dot(l1_delta)

Variables

Variable	Meaning
`X`	Input data
`y`	Ground truth labels
`w1`	Weights between Input Layer and Hidden Layer
`w2`	Weights between Hidden Layer and Output Layer
`l1`	Hidden layer activations
`l2`	Output prediction

Network Architecture

Input (X)
     │
     ▼
Linear Layer (X @ W1)
     │
     ▼
Sigmoid
     │
     ▼
Hidden Activations (L1)
     │
     ▼
Linear Layer (L1 @ W2)
     │
     ▼
Sigmoid
     │
     ▼
Prediction (L2)

There are two linear layers in this network.

First Linear Layer

X @ W1

Second Linear Layer

L1 @ W2

Every np.dot() operation is a linear transformation.

Tensor Shapes

X          (4,3)

W1         (3,4)

↓

L1         (4,4)

W2         (4,1)

↓

L2         (4,1)

Y          (4,1)

L2_delta   (4,1)

L1_delta   (4,4)

Understanding tensor shapes is one of the most important debugging skills in deep learning.

What does a neuron compute?

Every neuron first computes

z = Wx + b

where

W = weights
x = inputs
b = bias

Then it applies an activation function

a = sigmoid(z)

where a becomes the neuron's output.

Why random weights?

If every neuron started with identical weights,

every neuron would learn exactly the same feature.

Random initialization breaks this symmetry and allows different neurons to learn different patterns.

Why do we need an activation function?

Without an activation function,

multiple linear layers are still equivalent to one single linear layer.

Activation functions introduce non-linearity, allowing the network to learn complex patterns.

This tutorial uses Sigmoid because its derivative is easy to derive by hand.

Modern neural networks usually use

Hidden Layers → ReLU (or GELU/SiLU)
Binary Classification Output → Sigmoid
Multi-class Classification Output → Softmax

Loss Function

This tutorial uses Mean Squared Error (MSE)

L = 1/2 (y - l2)^2

Training tries to minimize this loss.

Forward Pass

Input

↓

Linear (W1)

↓

Sigmoid

↓

Hidden Layer

↓

Linear (W2)

↓

Sigmoid

↓

Prediction

Backward Pass (Backpropagation)

Prediction

↓

Compute Output Gradient (l2_delta)

↓

Propagate Error to Hidden Layer (l1_delta)

↓

Update W2

↓

Update W1

Backpropagation simply means

propagating the error backwards through the network.

Why can't we simply add the error to every weight?

Every weight contributes differently to the prediction.

Some weights have a huge influence.

Others barely affect the output.

The gradient tells us

"How much should this particular weight change to reduce the loss?"

Understanding `l2_delta`

l2_delta = (y-l2) * (l2*(1-l2))

Break it into two parts.

1. Prediction Error

(y-l2)

This measures how wrong the prediction is.

2. Sigmoid Derivative

Since

a = sigmoid(z)

its derivative is

σ'(z) = σ(z)(1-σ(z))

Since l2 already stores

σ(z)

we can compute

l2*(1-l2)

Therefore

l2_delta

=

Prediction Error

×

Sigmoid Derivative

This is the output gradient.

Understanding `l1_delta`

l1_delta = l2_delta.dot(w2.T) * (l1*(1-l1))

This one is more difficult.

The hidden layer does not know the true labels.

Instead,

the error must first travel backwards from the output layer.

Step 1

Take the output gradient

l2_delta

Step 2

Send the error backwards through the hidden→output weights

l2_delta.dot(w2.T)

This distributes the output error among all hidden neurons according to how much each hidden neuron influenced the output.

A stronger connection means

larger influence
larger responsibility
larger propagated error

Step 3

The hidden neurons also use sigmoid,

so multiply by its derivative

l1*(1-l1)

Therefore

l1_delta

=

Output Gradient

×

Connection Strength (W2)

×

Hidden Sigmoid Derivative

l1_delta tells each hidden neuron

"How much should you change?"

Why do we use `W2` instead of `W1`?

The output neuron only receives information from the hidden layer.

Therefore,

to determine how much each hidden neuron contributed to the final prediction,

the error must first travel backwards through W2.

Only after computing the hidden layer gradient do we update W1.

Output Error

↓

W2

↓

Hidden Layer Gradient

↓

Update W1

Why transpose (`W2.T`)?

This is purely because of matrix dimensions.

L2_delta

(4×1)

×

W2.T

(1×4)

=

(4×4)

Without the transpose,

matrix multiplication would be impossible because the dimensions would not align.

Why use `dot()`?

If there were only one hidden neuron,

we could simply write

hidden_error = output_error * weight

But there are 4 hidden neurons.

Each hidden neuron receives a different amount of the output error.

Matrix multiplication (dot) efficiently distributes the error to every hidden neuron simultaneously.

Why is matrix multiplication important?

Instead of computing

one neuron at a time
one training example at a time

matrix multiplication computes

every neuron
for every training example

in one operation.

This is one of the reasons neural networks run efficiently on GPUs.

Things I Learned Today

A neuron computes a weighted sum followed by an activation function.
Neural networks learn by updating weights, not neurons.
Forward propagation makes predictions.
Backpropagation propagates gradients backwards through the network.
l2_delta is the output gradient.
l1_delta is the hidden layer gradient.
Error reaches the hidden layer through W2, not W1.
Matrix multiplication makes neural networks computationally efficient.

Questions for Tomorrow

Derive l2_delta using the chain rule.
Understand why the derivative of sigmoid is σ(x)(1−σ(x)).
Reimplement the network from memory.
Replace the hidden sigmoid with ReLU and compare the results.