aboutsummaryrefslogtreecommitdiff
path: root/site/microblog.html
diff options
context:
space:
mode:
authorYuchen Pei <me@ypei.me>2018-05-11 17:10:58 +0200
committerYuchen Pei <me@ypei.me>2018-05-11 17:10:58 +0200
commitdb786e35abb644d83f78c21e8c4f10e1d6568a5e (patch)
treeddbc2df9a7072c86830122a89b7ac683a494c7e5 /site/microblog.html
parent08a326a3a2fc423ce498b0d80edecb7e596f1f28 (diff)
changed layout of blog.html
- cut post length on blog.html to synopsis, defaulted to one paragraph long - edited engine accordingly
Diffstat (limited to 'site/microblog.html')
-rw-r--r--site/microblog.html4
1 files changed, 2 insertions, 2 deletions
diff --git a/site/microblog.html b/site/microblog.html
index 8d3ba5a..33e017f 100644
--- a/site/microblog.html
+++ b/site/microblog.html
@@ -21,8 +21,8 @@
<div class="bodyitem">
<span id=rnn-fsm><p><a href="#rnn-fsm">2018-05-11</a></p></span>
<h3 id="some-notes-on-rnn-fsm-fa-tm-and-utm">Some notes on RNN, FSM / FA, TM and UTM</h3>
-<p>Related to a previous micropost.</p>
-<p><a href="http://www.cs.toronto.edu/~rgrosse/csc321/lec9.pdf">The slides from Toronto</a> is a nice introduction to RNN (recurrent neural network) from a computational point of view. It states that RNN can simulate any FSM (finite state machine, a.k.a. finite automata abbr. FA) with a toy example computing the parity of a binary string.</p>
+<p>Related to <a href="#neural-turing-machine">a previous micropost</a>.</p>
+<p><a href="http://www.cs.toronto.edu/~rgrosse/csc321/lec9.pdf">These slides from Toronto</a> is a nice introduction to RNN (recurrent neural network) from a computational point of view. It states that RNN can simulate any FSM (finite state machine, a.k.a. finite automata abbr. FA) with a toy example computing the parity of a binary string.</p>
<p><a href="http://www.deeplearningbook.org/contents/rnn.html">Goodfellow et. al.’s book</a> (see page 372 and 374) goes one step further, stating that RNN with a hidden-to-hidden layer can simulate Turing machines, and not only that, but also the <em>universal</em> Turing machine abbr. UTM (the book referenced <a href="https://www.sciencedirect.com/science/article/pii/S0022000085710136">Siegelmann-Sontag</a>), a property not shared by the weaker network where the hidden-to-hidden layer is replaced by an output-to-hidden layer (page 376).</p>
<p>By the way, the RNN with a hidden-to-hidden layer has the same architecture as the so-called linear dynamical system mentioned in <a href="https://www.coursera.org/learn/neural-networks/lecture/Fpa7y/modeling-sequences-a-brief-overview">Hinton’s video</a>.</p>
<p>From what I have learned, the universality of RNN and feedforward networks are therefore due to different arguments, the former coming from Turing machines and the latter from an analytical view of approximation by step functions.</p>