Home on Sid Mani's Blog
https://sidmani.com/
Recent content in Home on Sid Mani's BlogHugo -- gohugo.ioen-usSun, 11 Apr 2021 12:38:05 -0700Connecting the halting problem and induction
https://sidmani.com/posts/halting-problem/
Sun, 11 Apr 2021 12:38:05 -0700https://sidmani.com/posts/halting-problem/Suppose we have a program \(\mathbf{H}(p, x)\) that solves the halting problem for any program \(p\) evaluated on input \(x\), returning \(\mathrm{True}\) if the program halts and \(\mathrm{False}\) otherwise. Consider statements of the form \(\forall n \in \mathbb{N}: A(n)\), where \(A\) is some proposition that can be easily evaluated for any particular natural number. Let \(p(A)\) be the following program:
i = 1 loop forever: if not A(i): return i += 1 Then the evaluation of \(\neg\mathbf{H}(p, A)\) constitutes a proof of the truth or falsehood of the statement \(A\) for all natural numbers, if the evaluation of \(A(n)\) is guaranteed to halt.Continuous-time computation
https://sidmani.com/brain/continuous-time/
Fri, 09 Apr 2021 10:45:38 -0700https://sidmani.com/brain/continuous-time/The brain operates asynchronously over continuous time. Since our artificial computers use global clocks and discrete time, a natural question is whether the continuity of time matters with respect to the results or efficiency of the computation.
Fortunately, it is easy to see that we can ignore most of the implications of continuous-time computation. Dynamical systems can be approximated arbitrarily well in discrete time by decreasing the duration of the timestep.The binding problem
https://sidmani.com/brain/binding/
Mon, 29 Mar 2021 15:01:40 -0700https://sidmani.com/brain/binding/The binding problem in cognitive science refers to the lack of a known mechanism for decomposing and associating information in fixed biological neural networks.1 Several limitations of artificial neural networks are also directly related to binding.2
Important questions: How are visual scenes broken down into parts? How are attributes (color, size, etc.) associated with objects? How do we compose objects hierarchically, like in language and in visual scenes? How are numbers, associations, and more generally memories, stored?Backpropagation, hierarchy, and adversarial examples
https://sidmani.com/posts/backprop/
Sun, 28 Mar 2021 19:26:00 -0700https://sidmani.com/posts/backprop/One could easily believe that backpropagation ranks among the three most-computed algorithms in history.1 Backpropagation (backprop) exploits the chain rule from multivariable calculus to enable optimization of a parameterized function, usually a neural network.2 In order to support calculus, the function must be differentiable everywhere, and therefore continuous. We’ve already taken a look at the dangers of continuity. In this post, we’ll examine how the mechanics of backprop could harm representation learning.Not hardcoded
https://sidmani.com/brain/hardcoded/
Sat, 27 Mar 2021 21:29:19 -0700https://sidmani.com/brain/hardcoded/What if the brain is just a tangled, specialized mess?
The brain contains around \(10^{11}\) neurons and \(10^{14}\) synapses.1 If the exact connectivity of these elements is critical to the brain’s function, then there may not be underlying principles We’d be totally unable to identify units of functionality, or create effective abstractions “Hardcoding” is a programming term to describe handling cases individually, without abstraction or generality Fortunately, this cannot be true.Not quantum
https://sidmani.com/brain/quantum/
Sat, 27 Mar 2021 21:28:39 -0700https://sidmani.com/brain/quantum/Quantum computers are Turing-equivalent, so even if the brain was a quantum computer, all of its algorithms would still be computable on classical computers.
But quantum computers are much faster for certain problems; Quantum supremacy Shor's algorithm for factoring integers is exponentially faster than best classical algorithms So if the brain exploits a quantum algorithm with no efficient classical alternative, standard computers will be unable to run AGI efficiently The brain is almost certainly not a quantum computer.Reservoir computing
https://sidmani.com/brain/reservoir/
Sat, 27 Mar 2021 10:31:43 -0700https://sidmani.com/brain/reservoir/Description Reservoir computing exploits the properties of random dynamical systems to perform computation. Two components of a reservoir computer: A source of nonlinear dynamics (the “reservoir”) For example, a random recurrent neural network, or a bucket of water with a camera observing the surface1 Provides complex interactions that can be mapped to computations Reservoir parameters are fixed, so the reservoir is not involved in training A linear readout system Can measure values from the reservoir Imagine that there are “probes” that the readout system uses to measure certain aspects of the system Measurement happens at the same fixed “place” (could be a specific variable, or some combination of variables) at regular intervals Usually a linear classifier Parameters are trained per problem Multiple readouts can be used for parallel computation Since the reservoir is fixed, fewer trainable parameters needed for the same model complexity The key idea is that by feeding data into the reservoir, computations happen “naturally”, and just need to be decoded linearly Two kinds of reservoir computing relevant to the brain: Echo state networks2 (ESNs) are a random recurrent neural network + linear classifier Liquid state machines3 (LSMs) are a random spiking neural network + linear classifier Discussion Surprisingly, computation is possible even if the computer is mostly random Easy way to get parallelism is to use “branches” (multiple readout layers in this case) This is true even for regular neural networks The reservoir turns a learning problem into a search problem: find the desired nonlinear dynamics in the reservoir so a random reservoir has to be much larger than a learned reservoir to reliably contain those dynamics*Be skeptical!RNNs are not Turing-equivalent!
https://sidmani.com/brain/rnn-computation/
Fri, 26 Mar 2021 15:58:16 -0700https://sidmani.com/brain/rnn-computation/A lot of papers mention that recurrent neural networks (RNNs) are Turing-complete: that they can simulate a Turing machine. This belief is extremely pervasive; a paper1 from DeepMind states, without qualification:
Moreover, it is known that RNNs are Turing-Complete (Siegelmann and Sontag, 1995), and therefore have the capacity to simulate arbitrary procedures…
The referenced paper, titled On the Computational Power of Neural Nets,2 is responsible for a slew of misunderstandings.Difficulties with continuous approximation
https://sidmani.com/posts/difficulties-continuous-approximation/
Sat, 20 Mar 2021 20:32:59 -0700https://sidmani.com/posts/difficulties-continuous-approximation/Deep learning with artificial neural networks has several serious limitations that, in my opinion, render it completely incapable of modeling the kind of intelligence that humans possess. In the last few years, deep learning has seen several huge successes: GPT-3, AlphaGo, AlphaFold 2. These achievements were obtained through clever architectures based mostly on artificial neural networks and trained with massive computational scale.
The scaling laws diagram from the OpenAI team has frequently been presented as empirical proof of the power of large models:A glossary of Japanese self-study
https://sidmani.com/posts/glossary-of-japanese/
Thu, 11 Jul 2019 11:27:46 -0800https://sidmani.com/posts/glossary-of-japanese/Japanese is arguably the most difficult language for English speakers to learn,1 yet the relatively recent popularization of Japanese media has resulted in a disproportionate number of learners on the Internet. Consider the number of subscribers to the various language-learning subreddits,2 which puts Japanese at more than two times as popular as the next-largest language, French. The explosive growth of the Japanese-learning community has produced a large number of free online resources, albeit of varying quality, and beginners are likely to be completely overwhelmed.Brain is Turing-equivalent?
https://sidmani.com/brain/turing/
Mon, 01 Jan 0001 00:00:00 +0000https://sidmani.com/brain/turing/Definitions A Turing-equivalent system can compute the same set of functions that an idealized Turing machine can.
All of our computers (variations on the von Neumann architecture) are Turing-equivalent for functions (programs) that use finite memory and terminate in finite time (halt). Lots of subtleties around the idea of finite memory No real computer can truly be Turing-equivalent, since the Turing machine is assumed to be able to allocate an unbounded amount of memory So a computer with finite memory is actually a finite-state machine (a deterministic finite automaton).