|
Can one treat solution as an unknown binary random variable and estimate it giving a probability? y = X*b + k Where y and k are nx1 column vectors, b is px1 and X is nxp. y is a noisy time-series signal modeled by linear combinations of a basis set X described by b. k is additive Gaussian noise. Given y and X, what is b? Here's the twist: From the way I found X, b is either 0 or 1, while all other variables are continuous data. What I'd like to be able to do, is find b to be a value between 0 or 1 which corresponds to a probability the corresponding column of X is present in the reconstruction. The closest thing I can find is logistic regression. But that's not what this is.. correct? I'm new to these methods, and have come across this problem in my research. If anyone has any keywords I can look into, or references to point out, that would be very appreciated! P.S. If anyone can give a Bayesian interpretation.. Bonus points! The people this is for love that stuff. Thanks in advance! |
|
This is just linear regression. The probabilistic model here is y ~ N(b^Tx, beta^-1) where beta is the precision (inverse variance) of the additive Gaussian noise. You can fit the model using maximimum likelihood by minimizing beta sum_i (y-b^T x_i)^2 + log sqrt(beta/2pi) where the summation is over examples in the training set. Usually in linear regression you can solve this in two steps; first fixing beta and solving for b, then solving for beta. Unfortunately, when you want b to be binary it is harder you can't do the minimization with linear algebra due to the constraint that b is binary. Really it has become a discrete optimization problem. You might want to read about discrete optimization algorithms like genetic algorithms or Bayesian optimization. A simple thing you could do is start with all of b set to 0. Then repeatedly pick a random bit in b to flip, measure whether flipping that bit made the training set objective improve, and if it does, then keep the bit flip. |
I don't think logistic regression is appropriate here, as logistic regression assumes binary outcomes, i.e., y is binary, while the regression coefficients b are still real numbers R^p.
Peter rabbit your explanation doesn't really make sense to me. what i think you are asking for is to develop a model y=b(x)x+k where b(x) is a nonlinear function of x from R^p to {0,1}^p
however i don't see that b(x) can then be a probability, since you have multiple combinations of features + noise that give you the same y
you might want to look up "sparseness"