blob: b4212c2c37fa555c231e3d4597770591fda5dda6 (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
#+title: neural-turing-machine
#+date: <2018-05-09>
#+begin_quote
One way RNNs are currently being used is to connect neural networks
more closely to traditional ways of thinking about algorithms, ways of
thinking based on concepts such as Turing machines and (conventional)
programming languages. [[https://arxiv.org/abs/1410.4615][A 2014
paper]] developed an RNN which could take as input a
character-by-character description of a (very, very simple!) Python
program, and use that description to predict the output. Informally,
the network is learning to "understand" certain Python programs.
[[https://arxiv.org/abs/1410.5401][A second paper, also from 2014]],
used RNNs as a starting point to develop what they called a neural
Turing machine (NTM). This is a universal computer whose entire
structure can be trained using gradient descent. They trained their
NTM to infer algorithms for several simple problems, such as sorting
and copying.
As it stands, these are extremely simple toy models. Learning to
execute the Python program =print(398345+42598)= doesn't make a
network into a full-fledged Python interpreter! It's not clear how
much further it will be possible to push the ideas. Still, the results
are intriguing. Historically, neural networks have done well at
pattern recognition problems where conventional algorithmic approaches
have trouble. Vice versa, conventional algorithmic approaches are good
at solving problems that neural nets aren't so good at. No-one today
implements a web server or a database program using a neural network!
It'd be great to develop unified models that integrate the strengths
of both neural networks and more traditional approaches to algorithms.
RNNs and ideas inspired by RNNs may help us do that.
#+end_quote
Michael Nielsen,
[[http://neuralnetworksanddeeplearning.com/chap6.html#other_approaches_to_deep_neural_nets][Neural
networks and deep learning]]
|
