diff options
| -rw-r--r-- | microposts/neural-turing-machine.md | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/microposts/neural-turing-machine.md b/microposts/neural-turing-machine.md new file mode 100644 index 0000000..96f422e --- /dev/null +++ b/microposts/neural-turing-machine.md @@ -0,0 +1,8 @@ +--- +date: 2018-05-09 +--- +> One way RNNs are currently being used is to connect neural networks more closely to traditional ways of thinking about algorithms, ways of thinking based on concepts such as Turing machines and (conventional) programming languages. [A 2014 paper](https://arxiv.org/abs/1410.4615) developed an RNN which could take as input a character-by-character description of a (very, very simple!) Python program, and use that description to predict the output. Informally, the network is learning to "understand" certain Python programs. [A second paper, also from 2014](https://arxiv.org/abs/1410.5401), used RNNs as a starting point to develop what they called a neural Turing machine (NTM). This is a universal computer whose entire structure can be trained using gradient descent. They trained their NTM to infer algorithms for several simple problems, such as sorting and copying. +> +> As it stands, these are extremely simple toy models. Learning to execute the Python program `print(398345+42598)` doesn't make a network into a full-fledged Python interpreter! It's not clear how much further it will be possible to push the ideas. Still, the results are intriguing. Historically, neural networks have done well at pattern recognition problems where conventional algorithmic approaches have trouble. Vice versa, conventional algorithmic approaches are good at solving problems that neural nets aren't so good at. No-one today implements a web server or a database program using a neural network! It'd be great to develop unified models that integrate the strengths of both neural networks and more traditional approaches to algorithms. RNNs and ideas inspired by RNNs may help us do that. + +Michael Nielsen, [Neural networks and deep learning](http://neuralnetworksanddeeplearning.com/chap6.html#other_approaches_to_deep_neural_nets) |