From 3419d521d3301fc66ec0eb61fb000a16fcfdb4b6 Mon Sep 17 00:00:00 2001 From: Yuchen Pei Date: Wed, 9 May 2018 15:49:39 +0200 Subject: added an mpost --- microposts/neural-turing-machine.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 microposts/neural-turing-machine.md (limited to 'microposts') diff --git a/microposts/neural-turing-machine.md b/microposts/neural-turing-machine.md new file mode 100644 index 0000000..96f422e --- /dev/null +++ b/microposts/neural-turing-machine.md @@ -0,0 +1,8 @@ +--- +date: 2018-05-09 +--- +> One way RNNs are currently being used is to connect neural networks more closely to traditional ways of thinking about algorithms, ways of thinking based on concepts such as Turing machines and (conventional) programming languages. [A 2014 paper](https://arxiv.org/abs/1410.4615) developed an RNN which could take as input a character-by-character description of a (very, very simple!) Python program, and use that description to predict the output. Informally, the network is learning to "understand" certain Python programs. [A second paper, also from 2014](https://arxiv.org/abs/1410.5401), used RNNs as a starting point to develop what they called a neural Turing machine (NTM). This is a universal computer whose entire structure can be trained using gradient descent. They trained their NTM to infer algorithms for several simple problems, such as sorting and copying. +> +> As it stands, these are extremely simple toy models. Learning to execute the Python program `print(398345+42598)` doesn't make a network into a full-fledged Python interpreter! It's not clear how much further it will be possible to push the ideas. Still, the results are intriguing. Historically, neural networks have done well at pattern recognition problems where conventional algorithmic approaches have trouble. Vice versa, conventional algorithmic approaches are good at solving problems that neural nets aren't so good at. No-one today implements a web server or a database program using a neural network! It'd be great to develop unified models that integrate the strengths of both neural networks and more traditional approaches to algorithms. RNNs and ideas inspired by RNNs may help us do that. + +Michael Nielsen, [Neural networks and deep learning](http://neuralnetworksanddeeplearning.com/chap6.html#other_approaches_to_deep_neural_nets) -- cgit v1.2.3