Original Blurb from 2012
Nancy Fulda did a great job of explaining local minima last time, didn’t she? I thought so, anyway. But maybe it only made sense to me because I spent two and a half years of a postdoc dealing with similar problems. I was building models of a biological neural network. Not the kind of artificial neural network she worked with, where the individual neurons are simple and identical, and all the complexity comes out of the way they connect to one another during the learning process. I was trying to reproduce the spiking patterns of real neurons by varying the properties of individual neurons connected into a network with a wiring diagram based on experiments, injecting colored dyes into real neurons. My bosses had been fairly successful at tuning very small networks of 2 or 3 neurons by hand this way, but with a network of 20 neurons, each of which had dozens of electrical parameters and multiple connections, that was not going to work. With such a huge parameter space, there was no way to guarantee that any solution I stumbled onto by hand was the best solution, the solution with the smallest error, or whether it was only a local minimum in the error function. We needed to search the parameter space in a more systematic and automatic fashion, and we needed to do it fast.
I am not a good mathematician, and barely a competent computer programmer, so it took me about a year to convince them of this, and another year to adapt a search algorithm into something that kind of worked, slowly. I found an expert who promised his methods would be much faster and more accurate, but I had no idea how they worked. I’ve never studied linear algebra, with the matrices and stuff, beyond reading a lot of image processing papers when I was studying vision, so even his explanations made no sense to me. I could see the results, but as to how they got there, it was magic. He could have drawn a circle on the ground in pig blood and salt, and it would have been just as good, as far as I could tell. I could at least explain to people how my crappy slow solution worked.
That was a humbling experience, but an empowering one, too. I gave up the idea that I knew what I was doing, or that I ever would know, or even that I needed to know exactly what I was doing to make progress. It’s a lesson that has come in extremely handy as a parent, as we’ll hear in today’s episode. As long as I can keep the cost of my mistakes low, and I don’t have to make the same ones over and over, I’m happy to keep making new ones.
Anyway, what did you all think [of the episode] ?
References:
Nancy Fulda's machine learning papers
Nancy Fulda's "Nothing this Fun Could be Good for You"
How Huizinga relates to video games
A play/game called Homo Ludens
Updates from 2022
I’ve had good practical experience of the trade-off in speed and democracy this week. On Monday my students chose a real star system in which to place their alien creations. After learning about main sequence stars, and some of their more exotic cousins (above and below in the linked graph), they spent some considerable amount of time poking around the Open Exoplanet Catalogue, orienting themselves and generating lists of candidates. After much argument and multiple rounds of voting, they settled on Kepler 62, a K-class orange dwarf star about 1200 light-years away, which is far too dim to be seen with the naked eye. That’s why we needed a giant floating space telescope to observe the tiny changes in brightness that happen whenever one of its five planets passes in front of it.
Today, having duly demonstrated democracy and found it wanting, I instead became an efficient dictator, assigning people in pairs and giving each pair full sub-dictator’s power to invent all the unknown details of their chosen planet. Because I found a paper about a hypothetical moon around Kepler 62 f (the furthest out), I gave that maybe-moon a team, too, for a total of six celestial bodies. Boom! Done. Getting agreement from two people is far easier and faster than getting agreement from twelve, and the students seemed to recognize that benefit.
VSI Episode 45