|
I am looking at streaming data (i.e. online model), and looking for a specific discrete event. I want to stochastically model the time until this even happens, or if easier, say, model the probability that it happens withing the next 30 seconds. What is a simple, practical way to tackle this problem? What kind of technique can I use, and how can I train the model and backtest it? Note that the training is happening offline on historical data, and then the model is applied online, on live, streaming data. |
|
You probably want to use Poisson Process for this problem. I suspect you will run into further complications with needing to specify how much variance or Overdispersion you expect in the arrival times and Non-homogenous Poisson procceses where the arrival rate changes over time. |
I'm not too familiar with handling time series events, but if no one here has an answer for you, you might check out Cross Validated we well and ask them about time series analysis.
If you can model the events as a poisson process then the interarrival times (times between each event) will have an exponential distribution. I can't provide much more information (hence a comment not an answer), but perhaps this could help get you started in the right direction.