By Doug Schneider
If you own a smart phone, you probably have a weather app installed on it. But do you know where the weather information in the app is coming from? Do you know how its forecasts are being produced? The answers to these questions can often make a big difference in the accuracy of your app's forecast.
While watching radar and forecasting for the 24 Hours of Le Mans last weekend, I came across some forecasts that were posted to Twitter that seemed very inaccurate compared to what I was seeing. Here's the scenario that I was watching Saturday night around 11 pm Le Mans time:
If you own a smart phone, you probably have a weather app installed on it. But do you know where the weather information in the app is coming from? Do you know how its forecasts are being produced? The answers to these questions can often make a big difference in the accuracy of your app's forecast.
While watching radar and forecasting for the 24 Hours of Le Mans last weekend, I came across some forecasts that were posted to Twitter that seemed very inaccurate compared to what I was seeing. Here's the scenario that I was watching Saturday night around 11 pm Le Mans time:
Rain in central France will be dissipating as it moves N. Should not be an issue until near sunrise. @radiolemans pic.twitter.com/uTSvvOVbmu
— Doug Schneider (@Race4caster) June 13, 2015
I expected that the rain would be decreasing in coverage and intensity as it moved north because of my analysis of observed weather data across northern France. The projections of some models that indicated the rain would dissipate added confidence to what I expected. My forecast that I posted just before the race began had a 20% chance of rain Sunday morning, with a 40% chance in the afternoon before the end of the race.
I saw some tweets that talked about rain arriving at the circuit around midnight, just a little over one hour from the time of the above radar image. I'm not sure exactly where those projections were coming from, but I suspect it came from a weather app forecast. A simple look at the radar would have made it clear that the rain was well over an hour away. Some weather apps have an algorithm that takes current radar data and project the movement of rain, but this cannot account for the rain changing speed or direction, dissipating, or new rain areas developing. Perhaps some weather app forecast used model output that projected rain to develop over Le Mans in the next hour, but from my own analysis of the environment, that seemed highly unlikely.
Then I saw a tweet that mentioned a 79% chance of rain at 4 am, and an 82% rain chance at 7 am. I was amazed at the "skill" of this forecast that could predict a 3% increase in rain chances over three hours, followed by a drop of 10% in the next three hours, followed by another rise by 9% in the following three hours! The source for this forecast was World Weather Online.
I saw some tweets that talked about rain arriving at the circuit around midnight, just a little over one hour from the time of the above radar image. I'm not sure exactly where those projections were coming from, but I suspect it came from a weather app forecast. A simple look at the radar would have made it clear that the rain was well over an hour away. Some weather apps have an algorithm that takes current radar data and project the movement of rain, but this cannot account for the rain changing speed or direction, dissipating, or new rain areas developing. Perhaps some weather app forecast used model output that projected rain to develop over Le Mans in the next hour, but from my own analysis of the environment, that seemed highly unlikely.
Then I saw a tweet that mentioned a 79% chance of rain at 4 am, and an 82% rain chance at 7 am. I was amazed at the "skill" of this forecast that could predict a 3% increase in rain chances over three hours, followed by a drop of 10% in the next three hours, followed by another rise by 9% in the following three hours! The source for this forecast was World Weather Online.
Source: http://blog.memphisweather.net/
If a forecast calls for an 80% chance of rain within the next six hours, that implies a high level of confidence that it's going to happen. And in this case, it didn't. There were some light sprinkles at the circuit in the morning after sunrise, but there was no measurable rain recorded until after 2 pm, during the final hour of the race. The measurable rain arrived just when the rain chance in this forecast was falling from 83% to 55%! I don't mean to pick on WWO (they may very well be accurate most of the time, outside of this particular example), but I think this example demonstrates the problems with many weather apps.
Another example was a story in April about how the Miami Marlins, who play in a baseball stadium with a retractable roof, had a rain delay during a home game because club executives relied on weather apps for their information rather than trained meteorologists.
Not all weather apps are the same. Some produce forecasts that are manually produced by a meteorologist. These are the forecasts that will most often be the most accurate and of the highest quality. Some produce a "human in the loop" forecast where a meteorologist monitors automated data from a models but modifies is based on meteorological reasoning. Some produce a "human over the loop" forecast that is simply automated data output from a model that is quality controlled by a person (probably not a meteorologist) who may edit them only if there are obvious errors. Many apps take data straight from the weather models and display them on your phone as a forecast with no human intervention at all - these forecasts are strictly automated. The quality and accuracy of the forecast is strongly dependent on how much human intervention there is in creating the forecast - the most accurate forecasts will usually be the ones that are produced by a meteorologist, while the worst accuracy and quality will most often come from the automated forecast apps.
But meteorologists rely on forecast models all the time to create their forecasts, so why are automated forecasts bad (read more about the forecast process here)? Sometimes, they're not. Certain weather patterns are more accurately predicted by models than others. The problem is that automated forecast apps are captive to one particular forecast model and will change every time the model runs and produces new output (up to 4 times a day). If the computer model has a good handle on the forecast, they can be very accurate. If the model is wrong ("garbage in"), the output is wrong ("garbage out"). The farther into the future you look, the more likely you are to see wildly-varying forecasts that are just plain wrong.
Most of the time, the most accurate forecast is one that accounts for the output of multiple weather models, and assimilates data from numerous sources. Trained meteorologists view output from several weather models, and improve on that output by using their personal experience, knowledge of local terrain influences, knowledge of favored regional weather patterns, knowledge of model strengths and weaknesses, analysis of recent model trends and biases, and analysis of the latest observed data.
Another example was a story in April about how the Miami Marlins, who play in a baseball stadium with a retractable roof, had a rain delay during a home game because club executives relied on weather apps for their information rather than trained meteorologists.
Not all weather apps are the same. Some produce forecasts that are manually produced by a meteorologist. These are the forecasts that will most often be the most accurate and of the highest quality. Some produce a "human in the loop" forecast where a meteorologist monitors automated data from a models but modifies is based on meteorological reasoning. Some produce a "human over the loop" forecast that is simply automated data output from a model that is quality controlled by a person (probably not a meteorologist) who may edit them only if there are obvious errors. Many apps take data straight from the weather models and display them on your phone as a forecast with no human intervention at all - these forecasts are strictly automated. The quality and accuracy of the forecast is strongly dependent on how much human intervention there is in creating the forecast - the most accurate forecasts will usually be the ones that are produced by a meteorologist, while the worst accuracy and quality will most often come from the automated forecast apps.
But meteorologists rely on forecast models all the time to create their forecasts, so why are automated forecasts bad (read more about the forecast process here)? Sometimes, they're not. Certain weather patterns are more accurately predicted by models than others. The problem is that automated forecast apps are captive to one particular forecast model and will change every time the model runs and produces new output (up to 4 times a day). If the computer model has a good handle on the forecast, they can be very accurate. If the model is wrong ("garbage in"), the output is wrong ("garbage out"). The farther into the future you look, the more likely you are to see wildly-varying forecasts that are just plain wrong.
Most of the time, the most accurate forecast is one that accounts for the output of multiple weather models, and assimilates data from numerous sources. Trained meteorologists view output from several weather models, and improve on that output by using their personal experience, knowledge of local terrain influences, knowledge of favored regional weather patterns, knowledge of model strengths and weaknesses, analysis of recent model trends and biases, and analysis of the latest observed data.
A forecast 15 days away from WorldWeatherOnline.com. Is it realistic to expect this level of detail to be accurate 15 days in advance?
Another problem with weather apps is how they present a false expectation of accuracy. Some apps give a forecast over two weeks in advance, with Day 15 of the forecast having nearly the same level of detail as Day 1. While the science of meteorology has advanced tremendously in the past few decades, it is beyond the state of the science to predict the temperature at a certain hour of the day two weeks in advance. The public demands pinpoint, personalized forecasts days or weeks in advance, but sometimes a meteorologist simply has to say "I don't know". Even getting the details of a storm hitting your location in the next hour can be difficult to near impossible due to the variability and complexity of the atmosphere. The detailed forecasts of some weather apps are dangerous because they create absurdly high expectations that ultimately erode overall trust in forecasts because people see “I don’t know” as some sort of proof that all forecasts are guesses or just plain wrong. The graphical presentation of the weather apps leaves no room for forecast uncertainty to be expressed, giving the false impression that the late periods of the forecast have equal confidence to the early periods of the forecast.
Anyone can post a forecast from their automated weather app, passing it off as an accurate forecast, and spread it far and wide through social media. Be wary of forecasts that are passed on second-hand through social media. Verify that the source of the forecast is from a trusted source and produced by an experienced meteorologist. I know that forecasts from the National Weather Service are entirely produced by meteorologists with local expertise and experience.
At Racecast Weather, we use our experience combined with analysis of numerous data sources to produce our forecasts. In addition to our forecasts graphics, we put a lot of effort into producing posts on our website to help you better understand the weather pattern that is causing the weather, and we try to let you know when we have lower or higher confidence in our forecasts. We try to tailor our forecasts to the needs of fans, teams, and race officials, which is not possible from a weather app. You can see for yourself how accurate we are with our verification posts. We appreciate your feedback on our forecasts, both good and bad. To learn more about Scott and I, visit our About page.
Anyone can post a forecast from their automated weather app, passing it off as an accurate forecast, and spread it far and wide through social media. Be wary of forecasts that are passed on second-hand through social media. Verify that the source of the forecast is from a trusted source and produced by an experienced meteorologist. I know that forecasts from the National Weather Service are entirely produced by meteorologists with local expertise and experience.
At Racecast Weather, we use our experience combined with analysis of numerous data sources to produce our forecasts. In addition to our forecasts graphics, we put a lot of effort into producing posts on our website to help you better understand the weather pattern that is causing the weather, and we try to let you know when we have lower or higher confidence in our forecasts. We try to tailor our forecasts to the needs of fans, teams, and race officials, which is not possible from a weather app. You can see for yourself how accurate we are with our verification posts. We appreciate your feedback on our forecasts, both good and bad. To learn more about Scott and I, visit our About page.
