Smoothing Spline - stop messing around and go smart on your time-series method
Updated: Feb 7, 2022
It is really possible: intra-day patterns better and easier
Each forecaster recognizes the following problem: you make an intra-day pattern by taking the (weighted) average of historical patterns, but to get a nice smooth pattern you have to go further back in time than you would like ... the result is a non-representative pattern because you go back too many weeks or a pattern that has characteristics of "overfitting", random peaks and troughs that are not averaged out because you use too few days. The forecaster then accepts this diabolical dilemma, makes a non-optimal pattern, and proceeds to the order of the day.
What if we stop muddling around and look for a smart method that makes nice smooth patterns with little data? Such a method exists and makes use of the fact that a smooth pattern does not make large leaps. In the accompanying graph, you can see what such a "smoothing spline" looks like. In black, you see 5 historical patterns, in red their average with significant ups and downs. The smoothing spline, however, is a nice smooth curve that predicts the underlying customer behavior much better. Even with one day of data, the spline is quite smooth and accurately predicts future patterns.
The moral of the story? Sometimes we are so used to using a certain method that we cannot imagine that something smarter exists: we do not know what we do not know. Mathematics and AI, in this case statistics, often contain methods that are perfectly applicable to WFM. The result: a better forecast with less effort. This nicely illustrates the existence of scientific methods that can make the work of the WFMer more fun and easier. I intend to be writing further blogs in the near future and will share and discuss more of these methods.