Constant feature ignored by Spark LinearRegression?

Is this expected behavior?

TL;DR: Yes, this behaviour is expected. PySpark’s LinearRegression requires all input features, including constants intended as intercepts, to be assembled explicitly into vectors using VectorAssembler. Providing scalar columns directly without correct vectorisation can cause coefficients to be zero due to collinearity or rank deficiency.

Details

Relation to an offset in linear regression

This situation is similar to an offset term in linear regression in that the algebra is essentially the same, but the interpretation, estimation, and conceptual meaning are fundamentally different. Algebraically, both intercepts and offsets appear similarly ($Y = C + X\beta + C_1 + \varepsilon$, where $X$ is the model matrix so that $X\beta$ is the linear predictor). Conceptually, an intercept ($C$in this case) is estimated from the data, whereas an offset ($C_1$) is fixed and known, explicitly NOT estimated.

Mathematical Details

The linear regression model with intercept can be formally defined as:

$$ Y = \beta_0 + \beta_1 X + \varepsilon $$ where:

• $ \beta_0$is the intercept
• $\beta_1$ is the fixed effects parmeters
• $X$ is the model matrix
• $\varepsilon$ are the errors
• If fitIntercept=True, PySpark internally estimates the intercept term ($\beta_0$).
• If fitIntercept=False, the intercept must be explicitly included as a constant feature vector to estimate $\beta_0$ directly.

In cases of incorrect feature assembly, collinearity or rank deficiency can lead to zero-valued coefficients.

PySpark’s ML regression models require feature vectors rather than scalar inputs. Directly providing scalar columns results in them being converted into single-dimensional vectors. However, if a constant column intended as an explicit intercept is manually added without proper vector assembly, PySpark assigns it a coefficient of zero. This occurs because the constant feature becomes perfectly collinear with PySpark's internal intercept term, causing rank deficiency — not merely because the feature is constant.

Therefore, the observed behaviour reflects PySpark's intended handling of feature collinearity, rather than ignoring features outright. The key to avoiding this issue lies in correctly assembling features and choosing an appropriate intercept strategy.

A Better Approach

In PySpark there are two clear approaches to handling intercepts. First, allow PySpark to handle the intercept internally by setting fitIntercept=True. No explicit constant feature is necessary. Alternatively, manually add a constant column, assemble it correctly with covariates using VectorAssembler, and explicitly disable the internal intercept estimation (fitIntercept=False). This allows PySpark to treat your intercept as a standard feature. I prefer the first option.

For detailed guidance, consult the Apache Spark MLlib Documentation on VectorAssembler. Additional context on feature collinearity issues can be found in standard machine learning regression modelling references, such as Hastie, Tibshirani, and Friedman's The Elements of Statistical Learning - see references below.

Example of the use of VectorAssembler

from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression
from pyspark.sql.functions import lit

spark = SparkSession.builder.appName("LinearRegressionExample").getOrCreate()

# Sample data from the OP: single constant feature
data = [
    (1, 1.0, 0.01),
    (2, 1.0, 0.02),
    (3, 1.0, 0.03),
    (4, 1.0, 0.04),
    (5, 1.0, 0.03),
    (6, 1.0, 0.022)
]
columns = ["ID", "X", "Y"]
df = spark.createDataFrame(data, columns)

# Create a constant 'intercept' column
df = df.withColumn("Intercept", lit(1.0))

# Assemble features into a vector (including constant)
assembler = VectorAssembler(inputCols=["X", "Intercept"], outputCol="features")
df_vectorised = assembler.transform(df)

# Fit the linear regression model with fitIntercept=False
lr = LinearRegression(
    featuresCol="features",
    labelCol="Y",
    fitIntercept=False,
    solver="normal"
)

model = lr.fit(df_vectorised)
print("Coefficients:", model.coefficients)
print("Intercept:", model.intercept)

Summing Up

The observed PySpark behaviour is expected. Correctly vectorising features and clearly deciding on how to handle intercepts prevents confusion and helps to ensure accurate coefficient estimation.

References

Apache Software Foundation. (2024). Apache Spark MLlib: Feature extraction and transformation. Retrieved March 25, 2025, from https://spark.apache.org/docs/latest/ml-features.html#vectorassembler

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

McCullagh, P., & Nelder, J. A. (1989). Generalized linear models (2nd ed.). Chapman and Hall/CRC.