Has anyone here had any experience with Hierarchical Temporal Memory? wiki: http://en.wikipedia.org/wiki/Hierarchical_temporal_memory
I'm thinking of using it to build a set of rules from a large dataset as it appears to be more suitable than the neural net approach I am taking at the moment. Has anyone used this for their research work? If so would you recommend it?
It seems to be an interesting idea but currently I don't really know too much about it, so I'm looking for any papers or learning resources. Any suggestions would be appreciated!
asked Oct 17 '10 at 03:37
I've read the papers and Hawkins book, it seems like an interesting and intuitive idea, but I think it ends there. No one has (afaik) picked up on this research and everyone I've talked to in the ML community is pretty dismissive of the work in general. You do have to be careful because it's not clear that this work gets the level of exposure because of Hawkins fame (from unrelated fields) or because it is a useful technology.
answered Oct 17 '10 at 04:38
Take a look at Hinton's Deep Belief Networks. Numenta seems to converge to them. Hawkins recently recognized that representing a pattern by a single number is limiting. In the future, Numenta is going to use sparse distributed representations.
You can experiment with Deep Belief Networks by following the Deep Learning Tutorials.
Yes! There are people who have experience with HTMs!
There is a small community of HTMs hackers and researchers, as can be seen for example on Numenta's forums
More visible are a few commercial applications and various "Proofs of Concepts" (beyond these produced by Numenta). For example:
It's not surprising that many of the early successes of HTMs are in the area of vision. Also, it is fair to note that several of the effects/features supported by HTMs networks in these applications can be achieved by other means. Never the less these applications provide validation of the concepts put forth by Jeff Hawkins. As indicated in Ivo Danihelka's answer, Numenta's approach to HTMs and Hinton's research on Deep Belief Networks seem to converge in several regards; this observation can too be interpreted as validation.
I hope this answer provides a bit of balance, as some of the other answers and comments seemed to be, IMHO, a bit quick in dismissing the HTM approach. Certainly Hawkins' personality and fame, along with the biomimetic nature of the concepts he champions so eloquently (see for ex. this video presentation) all contribute to some WOW factor.
answered Dec 23 '10 at 04:33
I've had the same experience as anonymous_4235: i.e. people tend to be fairly dismissive of the technology. Despite that, I find the ideas intriguing and I like that Numenta is exploring a different approach to the goal many of us share.
I'll point you to: http://www.numenta.com/htm-overview/education.php
They just released a PDF detailing their new algorithms that use the sparse distributed representations. They provide a lot of psuedo-code so you can probably do what I tried to do and implement the whole thing yourself (it's not trivial because they are a bit vague on many parameter settings), but they also claim that they are going to release some source code early next year.
answered Dec 18 '10 at 15:30
Actually, there's a company that was founded on this idea and is actively building the core technology and applications. It's called Numenta. http://www.numenta.com/
answered Oct 17 '10 at 20:24
im actually implementing it now... to tell you the truth, you get the basic idea off Hawkins but this realm of computing is pretty wild west, you have to come up with your own ideas, cause Hawkins doesnt even know the whole thing, but if you want to take HTM as a starting point, im interested in interactive video, thats my reason behind studying this.
Heres the gist in mine so far.
Theres two different kinds of synapses, proximal and distal. the proximal synapses go in from the sensory data into the layer of nodes and back again. getting them to go in and out means you have to store a list of shapes that it compares the input to and it picks the closest shape to modify (or teach) there can be 16 - 256 or more of them per node. I dont know how important the sparse representation is, but at the moment, its in my version. My reason for that is, Im guessing it saves more space (more free proximal dendrites, connected to each node to the previous layer) to store more video.
you need more than one shape per node, because otherwise you have an overwriting problem, you need more "space" inside the node to store more feedback outputs, because youve pooled all the data to one node, thats how you restore it.
then there is the distals, these i think are done in a similar way, you pick the closest temporal sequence to increment or decrement synapses of, and that way you dont overwrite temporality either.
if you have both of these things, you should be able to pump video into your network, and then play it back out of it. ;)
My best advice is, unfortunately all us scientists are all on our own when it comes to this "brain" stuff, but whatever you do, dont be afraid to come up with your own ideas, its the secret of learning.
One other thing, if the eye is small enough - it runs realtime really quickly, its shocking.