Revisiting Logistic Regression

When picking a baseline for CTR prediction, the candidates are many. Gradient Boosting, Neural Networks, and Logistic Regression. Among them, LR is still often selected as the baseline. There are reasons for that.

LR Properties

Lightweight. The model is a single dot product. Training and inference both scale linearly with the number of features.

Interpretable. Every coefficient directly indicates “how much this feature contributes to the outcome.”

Probability output. It outputs values between 0 and 1. In ads, you multiply those directly against a bid.

Model Structure

The most direct way to understand Logistic Regression is to start from linear regression.

Linear regression outputs a weighted sum of the inputs.

The problem is that $z$ ranges over all real numbers. To produce a probability like CTR, the output must lie between 0 and 1. Linear regression does not guarantee that.

The sigmoid function solves this.

Sigmoid smoothly compresses the entire real line into $(0, 1)$. No matter how large the input, it approaches 1; no matter how small, it approaches 0. Pass the output of linear regression through sigmoid, and you get a probability.

This simple composition is all there is to Logistic Regression: a linear model with a probability layer on top.

One thing worth noting. The probability output is nonlinear, but the decision boundary, the surface that separates the two sides at probability 0.5, remains linear. The hyperplane $w \cdot x + b = 0$ is itself the boundary. LR is “a linear classifier with probabilities bolted on.”

log-loss

Once the model structure is fixed, training becomes “finding good $w$ and $b$.” We need a criterion for “good.”

Linear regression uses MSE (Mean Squared Error). LR does not. The reason lies in the output shape.

LR’s output is a probability. There is a more suitable choice of loss for probabilistic models: log-loss (a.k.a. cross-entropy).

When the label is 1, the loss shrinks as $\log \hat{y}$ grows; when the label is 0, the loss shrinks as $\log(1 - \hat{y})$ grows. The closer the predicted probability gets to the truth, the closer the loss gets to zero.

Log-loss is convex for LR. No local minima. The optimization converges to the global optimum. This property is the mathematical reason LR trains quickly on large-scale data.

Why These Properties

The three characteristics from the overview, lightweight, interpretable, probability output, all follow from the structure above.

Lightweight

A trained LR model is ultimately a weight vector $w$ and a bias $b$. Inference is one dot product and one sigmoid. Whether you have a million features or ten million, the computation scales linearly with the feature count. Compared to the many multiplications and nonlinearities in tree ensembles or neural networks, LR requires far less computation.

Interpretable

A coefficient $w_i$ means “when feature $i$ increases by one unit, the log-odds shift by $w_i$.” The sign indicates direction; the magnitude indicates influence. When you want to know “which feature contributes positively to clicks” in the ad domain, LR answers with a single table of coefficients. This satisfies the accountability requirements on the operations side.

Probability Output

Many classifiers output only a ranking score. LR outputs a calibrated probability. Ad expected-value math requires multiplying that number directly: predicted CTR × bid = expected revenue. A score that is not a probability cannot be used directly in the bidding formula.

CTR Prediction

Looking at CTR prediction as a problem reveals why LR fits.

Sparse. Most features are one-hot-encoded categoricals. Out of millions of dimensions, only a handful are 1; the rest are 0.

High-dimensional. The cross-product of ad, user, and context spreads across millions to hundreds of millions.

Large-scale. Training data accumulates in large daily volumes.

LR aligns with all three. The dot product of a sparse vector only needs to touch the non-zero entries, so the computation scales with the actual count of populated features, not the raw dimensionality. Training is easy to distribute via the SGD (Stochastic Gradient Descent) family. Inference fits inside the tight latency budget of real-time bidding.

When bringing up a CTR model for the first time, these characteristics become decisive. You need to establish a baseline quickly, covering the training pipeline, serving, and monitoring, and validate the entire lifecycle first. A more complex model delays that validation.

Limits and What Comes Next

Having seen why LR serves as the baseline, we should also see why it is eventually replaced.

The biggest limit is the absence of nonlinear interactions. Products of features, conditional effects, complex combinations. LR cannot discover those on its own. A human has to define them in advance through feature engineering. As feature combinations grow, the engineering cost increases and operations become constrained by feature-design reviews.

So when do you move on? When data and operational headroom reach a point “feature engineering can no longer absorb.” Gradient Boosting Decision Trees learn interactions on their own. Neural networks go further, converting high-cardinality categoricals into continuous vectors through embeddings. Both directions address exactly LR’s limits.

That said, LR remains a reasonable starting point. Without a baseline, if you start with a complex model, you cannot distinguish the model’s contribution from the pipeline’s. The numbers LR provides become the reference line for every comparison that follows.

The Statistical Meaning of z — the logit

The body said a coefficient “moves the log-odds.” Pinning down what that log-odds is changes how you read LR’s score.

The linear combination $z = w \cdot x + b$ looks like an intermediate value on the way into the sigmoid. Statistically, though, it means something on its own. Invert the sigmoid and it shows. With the predicted probability $p = \sigma(z)$,

$p/(1-p)$ is the odds — the probability of success divided by the probability of failure. Take its log and you get the logit, which is exactly $z$. The raw score LR produces is a direct estimate of the log-odds of a click.

Seen this way, the coefficients read cleanly. A unit increase in a feature moves $z$ by $w_i$, which means the log-odds shifts by $w_i$. It is nonlinear in probability space but linear in log-odds space. The log-odds space is precisely where LR earns the label “linear model.”

Inside fit — the Convergence Loop and the solver

A single model.fit(X, y) runs an iterative loop. It computes a prediction from the current weights (the forward pass), measures the gap between prediction and label with log-loss, and the solver computes the gradient of that loss to update the weights. It repeats until the loss stops decreasing (tol) or it hits a fixed count (max_iter), and stops once it converges.

How the weights get updated is the solver, and each solver looks at the data differently.

lbfgs looks at the gradient over the entire dataset at every step. It picks a direction from the full landscape laid out at once, with no randomness. The same data and the same initialization produce the same weights every time.

saga and sag update by sweeping over the data sample by sample — a stochastic approach. Looking at only part of the data per step makes them strong at scale, but shuffling the sample order introduces randomness. Without a fixed random_state, the weights vary slightly from run to run.

liblinear, despite the name, is not a stochastic solver. It is a deterministic solver based on coordinate descent, and on the randomness axis it sits closer to lbfgs.

Training on Aggregated Logs with sample_weight

In domains like CTR prediction, where tens of millions of logs pile up daily, feeding the trainer row by row is wasteful. The same feature combination shows up across tens of thousands of duplicate impressions.

LR’s linear structure and the additivity of log-loss let you compress this. Collapse the logs that share a feature combination into a single row, put that combination’s click-through rate (CTR) in $y$, and put the impression count in sample_weight.

model.fit(X, y, sample_weight=impressions)

Training on this aggregated data is, in theory, identical to training on the fully expanded logs. The weights $W$ match, and so does the log-loss. The reason is in the loss itself. For a group with $k$ clicks out of $n$ impressions, the expanded loss is $-[\,k \log \sigma(z) + (n-k)\log(1-\sigma(z))\,]$, which equals the weighted loss given $y = k/n$ and weight $= n$. The gradients match, so the optimum matches.

This equivalence, though, is tied to the solver. With a full-batch, deterministic solver like lbfgs, the math holds exactly. With a stochastic solver like saga, the sampling unit — individual logs versus aggregated groups — changes the per-minibatch gradient scale, so small convergence differences can appear. When exact equivalence matters, a batch solver is the safe choice.

Serving Latency and the 2-Stage Pipeline

The reason to leave LR — its lack of nonlinear interactions — was covered above. Yet in practice LR doesn’t disappear. It stays, in a different seat.

The reason is latency. In an environment that has to finish a request within 5ms (0.005s), like real-time bidding, nothing keeps up with LR’s “one dot product plus one sigmoid.” GBDT (Gradient Boosting Decision Tree, LightGBM being the common one) has to traverse hundreds of trees, and an NN (Neural Network) has several layers of matrix multiplication. They buy a bit of accuracy by spending more latency.

So modern large-scale recommendation and ad systems don’t finish with one model — they split into stages. First an NN narrows hundreds of millions of candidates down to a few thousand (Retrieval), then GBDT or LR ranks those few thousand precisely (Ranking). The first stage sweeps wide, the second works narrow and precise. Each stage takes a different point on the latency-accuracy trade-off.

In the final ranking stage, where the latency budget is tightest, the simplest model — LR — still remains a candidate.

Closing

Choosing the old model had its reasons.

Those reasons are in the structure. The composition of a linear model and sigmoid, the convexity of log-loss, the efficiency in sparse, high-dimensional spaces. Together, these three keep LR as the baseline for CTR prediction.

Even when the time comes to move to the next model, the numbers LR provided remain as the baseline.