Machine Learning

k-Nearest Neighbors (k-NN): learning from your neighbors

k-Nearest Neighbors is one of the most intuitive algorithms in Machine Learning, and precisely for this reason it is often underestimated. There is no explicit training phase, no complex optimization, no hidden parameters learned in advance. And yet, k-NN captures a powerful idea: similar problems tend to have similar solutions. Instead of learning a global model of the world, k-NN reasons locally, using the past as a reference point for the present.

Learning by Comparison

At its heart, k-NN works by comparison. When the model is asked to make a prediction for a new data point, it looks at the existing data and finds the k most similar examples, called neighbors. For classification, it assigns the class that is most common among those neighbors. For regression, it averages their values. Learning, in this case, does not happen during training, but at prediction time.

This local nature is what makes k-NN conceptually different from models like linear or polynomial regression. Instead of fitting a function that explains the entire dataset, k-NN defers all decisions until it sees a new input. The dataset itself becomes the model. This makes the algorithm incredibly flexible, but also introduces important trade-offs.

The Importance of Distance

One of the most interesting aspects of k-NN is how strongly it depends on the notion of distance. The algorithm assumes that distance in feature space corresponds to similarity in meaning. If two points are close, their outcomes should be similar. This assumption sounds reasonable, but it hides complexity. The choice of distance metric and the scaling of features can completely change the model's behavior. A feature measured in large numerical ranges can dominate distance calculations and drown out more meaningful signals if not handled carefully.

The value of k also plays a crucial role. A small k makes the model sensitive to local variations and noise. Predictions become sharp and reactive, but also unstable. A larger k smooths predictions by averaging over more neighbors, making the model more robust but less sensitive to local structure. Choosing k is therefore not a technical detail; it is a statement about how much trust you place in local evidence versus global consistency.

Trade-offs and Practicality

Because k-NN stores the entire dataset, it has no abstraction layer separating raw data from decisions. This can be a strength. The model can adapt instantly to new data by simply adding it to the dataset. But it also means that prediction time can be expensive, especially for large datasets, since distances must be computed repeatedly. In practice, this makes k-NN more suitable for smaller datasets, prototyping, or problems where interpretability and flexibility matter more than raw speed.

Let's now look at a concrete example to see how k-NN behaves in practice.

A Practical Example

Imagine you are building a simple system to classify houses as either "cheap" or "expensive" based on two features: size and distance from the city center. You don't want to assume a linear boundary. You simply want to say: show me similar houses, and I'll decide based on them.

Here is a minimal but realistic Python example.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

First, let's create a small dataset.

# Features: [size (sqm), distance from city center (km)]
X = np.array([
    [50, 8],
    [60, 7],
    [70, 6],
    [80, 5],
    [90, 4],
    [100, 3],
    [110, 2],
    [120, 1]
])

# Labels: 0 = cheap, 1 = expensive
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])

Before applying k-NN, we scale the features. This step is essential because distance-based models are highly sensitive to feature magnitude.

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Now we split the data and create the k-NN model.

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.25, random_state=42
)

model = KNeighborsClassifier(n_neighbors=3)
model.fit(X_train, y_train)

Even though we call fit, the model is simply storing the data. The real work happens during prediction.

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)

print("Accuracy:", accuracy)

Let's make a prediction for a new house: 85 square meters, 4.5 km from the city center.

new_house = np.array([[85, 4.5]])
new_house_scaled = scaler.transform(new_house)

prediction = model.predict(new_house_scaled)
print("Predicted class:", prediction[0])

Understanding the Results

The prediction is based entirely on nearby examples. You can inspect which neighbors influenced the decision, making k-NN surprisingly interpretable. Instead of explaining coefficients or learned weights, you explain decisions by pointing to similar cases in the past.

This example shows why k-Nearest Neighbors is such an important algorithm to understand early on. It embodies a pure form of data-driven reasoning. There are no hidden abstractions, only comparisons. This makes its strengths and weaknesses very clear. It performs well when similar cases truly lead to similar outcomes, and poorly when distance does not capture meaning.

k-NN teaches a valuable lesson: not all learning requires building a global model. Sometimes, remembering the past and reasoning locally is enough. Understanding when this approach works, and when it breaks down, builds intuition that carries over to far more complex Machine Learning systems.