Overfitting

Overfitting

What is over-adjustment?

Over-fitting is a modeling error that occurs when a function is too close to a limited set of data points. Over-fitting the model generally takes the form of a model that is too complex to explain the specifics of the data under study.

In reality, the data often studied contain some degree of error or random noise. Thus, attempting to make the model conform too closely to slightly inaccurate data can infect the model with substantial errors and reduce its predictive power.

Key points to remember

  • Over-fitting is a modeling error that occurs when a function is too close to a limited set of data points.
  • Finance professionals should always be aware of the dangers of over-adapting a model based on limited data.

Understanding the over-adjustment

For example, a common problem is to use computer algorithms to search large databases of historical market data in order to find patterns. With enough study, it is often possible to develop elaborate theorems that seem to predict things like stock market returns with near accuracy.

However, when applied to data outside the sample, these theorems may turn out to be simply the over-fitting of a model to what was in reality only chance occurrences. In all cases, it is important to test a model against data external to the sample used to develop it.

Finance professionals should always be aware of the dangers of over-adapting a model based on limited data.

Leave a Comment

Your email address will not be published. Required fields are marked *