>> In the business world, machine learning often goes by the annoying moniker of
“artificial intelligence.” That science-fiction buzzword evokes visions of
godlike sentient computers, when in fact, the product is much closer to a
statistical regression. Machine learning is about using algorithms to predict
things — whether a web-security image contains a cat, what a Google user wants
to search for, or whether a self-driving car should brake to avoid a crash. No
one yet knows how to give a single computer system the mental flexibility to
reason and learn like a human being.
While the last senetence is true, the rest of the opening paragraph of this
article commits the typical sin of promulgating a reductionist definition of
"machine learning", probably because of a lack of historical perspective into
the origins and the use of the term.
Historically, then, the broadest definition of "machine learning" is the one
given by Tom Mitchell, according to whom a machine learning system is one that
can improve its behaviour over time. This covers machine learning algorithms
that are really nothing like statistical regression- decision tree learners,
or the wide array of algorithms and systems that learn logic-based
representations that predominated in the early years of the discipline (and
are still going, if I may, with Inductive Logic Programming).
In short, neither prediction, nor statistical regression are necessary or
sufficient characteristics of machine learning. Indeed, there is nothing
stopping a system that would learn to reason from examples from being categorised as a
"machine learning system".
Nothing- except the incomplete understanding of experts from other
disciplines, perhaps, like the author of this piece.
I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error. Regression is also model fittining (polynomials) by minimizing mean squared error. Not a huge difference...
>> I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error.
That's only true if you define machine learning as "model fitting", however there are many algorithms and techniques that have nothing to do with fitting a curve to data points and are very commonly considered to be machine learning.
E.g. decision tree learners, nearest-neighbour algorithms, Bayesian inference algorithms, Expectation-maximisation (and friends), clustering algorithms- none of these really um fit under "curve fitting", although of course you can broaden the scope of what is meant by "model fitting" to include potentially anything that looks kinda like it from a few steps away.
Then of course there is the very broad family of logic-based machine learning systems, pioneered by Ryszard Michalski and others in the '70s and Inductive Logic Programming, that learn logic theories from relational data and do not perform optimisation or model fitting (they tend to be greedy).
Basically, it's mostly the gradient-descent based algorithms that do curve fitting (linear regression, perceptrons, SVMs and, er, that; oh, and gradient boosting). These approaches are currently dominant but that's all.
There is much more to machine learning than curve fitting.
All of those require a training step, and that is the same error minimization as curve fitting.
Basically they are curve fitting in a high enough dimension.
Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it.
That said, of course there is much more to it, than training. (And as we saw with OpenAI's dota bots and DeepMind's AlphaGo Zero, with good feature engineering, adversarial learning setup, beating human performance is only a matter of processing power.)
>> All of those require a training step, and that is the same error minimization as curve fitting.
>> Basically they are curve fitting in a high enough dimension.
>> Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it.
I don't understand most of what you are saying and the bits I understand, I don't see how they're true. Could you please explain in a little more detail? For example, how is, say, recursive partitioning, as in decision tree learners, or storing data points for later comparison, as in KNN, "curve fitting in a high enough dimension"? What is the function being fit and where are its parameters?
Full disclosure: I study ILP algorithms for my PhD and I have implemented a couple myself. There is nothing like optimisation going on in there, just (inverse) resolution and some ordering of the search space. Most have nothing like an "error metric", greedy or not. And I have never heard anyone describe logic-based learners, including ILP algorithms, as curve fitting before.
If something requires training, it constructs a model, the model can be represented as a probability distribution. I think of that as the curve.
ILP (let's say simplex algorithm or other hill climbers) don't require training, but they nevertheless try to optimize the utility (or goal) function (which implicitly defines a metric on the space), which is a hypersurface, which is a curve in an even higher dimension.
Since the cardinality of the set of solutions is not one, and usually infinite, picking one solution over the other is optimization. (And - if I'm not mistaken - unless we know that the search space is convex, we don't know if there are better solutions or not.)
I don't think anything that doesn't build a model is machine learning. (Decision trees build a model, Bayes classifiers build a model, etc. And I'd say all are curve fitting.)
I'm not doing a PhD at all, so I'm not claiming that I'm an authority on each correct terminus technicus for AI/ML/optimization math/CompSci, but to me they fall under the model/curve fitting umbrella.
The Simplex algorithm is a linear programming algorithm. ILP is Inductive
Logic Progamming, not linear programming. ILP algorithms do not perform
linear programming, do not have a utility function and do not use
optimisation. What they do is search a space of hypotheses for a hypothesis
that is consistent with some background theory and that explains some set of
examples. Wikipedia has an introduction:
You will not be able to understand the operation of ILP algorithms if you try
to look at them from the point of view of function optimisation, or, indeed
"curve fitting". You need a different set of tools, that of logic programming
and automated theorem proving- because the process is essentially that of
logical deduction. Please let me know if you delve into the subject and have
any questions (see my profile for my email address). I'm always happy to
support attempts to understand my research subject.
The same goes for other machine learning algorithms: if you use the wrong
abstractions to understand them, you'll get the wrong idea about them.
For instance- KNN doesn't "require training" as per your condition for machine
learning. It just memorises all its examples. Is KNN not machine learning? It
is- because it gets better the more examples it has. If you stick to thinking
of everything as "curve fitting", either you'll misunderstand what KNN does,
or you'll start thinking that KNN is not machine learning.
Decision tree nodes are all unique - their probability is always 1/n, where n
is the number of all nodes in the tree. In other words, their distribution is
uniform. If you try to understand decision trees as "a model represented as a probability distribution" you'll
find out that every decision tree learner builds the same model. Obviously that's not true, because each decision tree
recognises different classes.
And so on.
When we talk of "curve fitting" what we mean is that we have the function of
a line (not necessarily straight) like αx + βy + γz + ... where the variables are the attributes of a
set of examples and the constants are unknown and we wish to learn them. These
are the "parameters" of the function that are optimised by curve fitting
algorithms. We say that a curve is being "fit" because we assume there is some
function that we don't know that describes a line whose points are the entire
class of entities from where our examples are drawn. And since we don't know
what this function is, we instead try to approximate it with a function that
"steps" on the few points that we have- or in other words, "fits" those
points.
In decision trees, KNN, Bayesian inference, etc, etc, there is no line
function and therefore no curve that we are trying to fit. You won't find a
line in higher dimensions either. Whereas there is probably some curve that
can fit the data points given as training examples to the algorithm, the
algorithm, lacking a line function, cannot represent that curve. And while you can sometimes
visualise the end result as a curve, this is only a convenience and, more
importantly, it is not always possible. For example- decision trees again
(because they're easy): look around for visualisations of decision trees. You
will find tree diagrams aplenty, but not a single line curve. Because you
can't really fit a line on a bunch of decision tree nodes. It's a graph.
Where's the line? Where are the parameters?
You've obviously made an effort to organise what you know about machine
learning, the better to understand it. But you're missing lots of details and
the result is actually hindering your undersanding. In a sense, by looking at
everything from the point of view of "curve fitting" you're missing the forest
for the um, trees.
My advice is: if you really want to understand machine learning (or at least
to sound a bit knowledgeable about it) keep an open mind and don't rush to
conclusions. There is much more to machine learning than what you can get from
reading a few blogs, a couple of wikipedia pages and a few popular articles in
the tech press. Even experts don't yet have a high-level theory of what
machine learning is and how to treat different machine learning algorithms in a high-level, unifrom manner, as
some well defined class of objects with common characteristics. That should give you a hint about how close your all-encompasing, "it's all curve fitting" definition is to explaining anything.
Interestingly I thought ILP is integer linear programming. (The first hit for ILP algorithm.)
And then I read DeepMind's delta-ILP post you linked and was even more confused as that talked about gradient descent, and then looked at your Metagol example, and ...
So, decision trees are a great example, as they were the first thing I've encountered ~15 years ago about machine learning, and I think it's a first class ML citizen. But they are in their pure form absolutely vulnerable to noise.
And as soon as noise enters the picture it becomes optimization. And I think that's the model fitting that should be the primary concept. (I used curve-slash-model, but I should have used just model.) [And even if there's no "fitting", like with kNN, you still have data, and either you use a constant classification set of points, or you'll see bias, but then the question becomes what set to use.]
And you're right, that treating everything as X is folly, after all, it gives you no information, decreases no entropy. But I'm looking at it from the perspective of what's intelligence. And it seems to be always on this spectrum of how to optimize for something. (Making hard things easy, by learning rules/algorithms.) After all that's why are we really interested in ML, no?
> But you're missing lots of details and the result is actually hindering your undersanding. In a sense, by looking at everything from the point of view of "curve fitting" you're missing the forest for the um, trees.
If you were so kind to elaborate on this, I'm very interested. And in general, anything that comes to your mind about my ramblings! Thanks for your detailed comments!
OK, different comment because this one might be construed as a bit personal. I mean, I really hope it's not, but I've heard one too many people recently claim they know about machine learning who only mean they've heard of it in the last five or six years (2012 is the cut-off point) and only know the algorithms that are typically taught in machine learning courses at degree and Master's levels (perceptrons, linear regression, decision trees, naive Bayes, logistic regression, KNN, SVMs, K-means, maybe some deep learning bits) and who wouldn't know what the hell CART is or who Arthur Samuel, or, indeed, Warren McCulloch and Walter Pitts, were.
So I have to ask: when you say "a fair amount"- what exactly do you mean?
When the likes of Google, Uber, and Tesla hype “machine learning” as the solution to all problems, then the historical reality of the academics who named it doesn’t really matter anymore. If the press and the financiers believe it’s a silver bullet then the only way to fight that misconception is to use their own terms.
I would add that the OP is motivated, in part, by Corporate America's current understanding of "AI" as little more than "glorified regression" and "glorified classification." There's a tremendous amount of hype, but really very little understanding outside of a few academic centers and Silicon Valley companies.
This is exactly the point of the article. If you can let go of your attachment to the academic meaning, the hype long ago was a done deal, and so the responsibility of the business press is to call BS on what the business world says about ML and AI.
AI is a lot dumber than people expect... which is the scary thing. If you look at Leela Zero's training, you'll see a lot of huge flaws pointing to massive failures of reason and areas where it is frankly incapable of making basic generalizations. Yet variants of this same technology beat the best Go players, synthesize realistic voices, give sentence labels to images, drive cars, translate language, etc. Makes you wonder that, if such an unintelligent bit of math can fake so much, whether intelligence is really that mysterious or whether it just seems that way because we don't know how to write it.
I don't think we are that close to creating artificial intelligence for a very simple reason.
To solve a problem you must first be able to define the problem to be solved. So, define intelligence. Take your time, this comment will wait for you. Done? Good. Now define it in a way that everyone else agrees with and that doesn't fall to pieces under scrutiny.
Philosophers have been trying this for thousands of years and are still no closer to an answer. As far as I can tell we don't have any test for intelligence that is good enough to establish, for absolute certainty, that another human being is 'intelligent' let alone anything else.
Define the problem and the requirements to consider the problem solved and only then do we have any hope of coming up with a solution.
Well, this seems to be the definition Deepmind uses: https://arxiv.org/abs/0712.3329 . And there has been work to establish a quantitative measure of the extent to which an agents behavior is intelligent or mechanical: https://arxiv.org/abs/1805.12387 .
Shane Legg, one of the authors of the paper, is also the co-founder of Deepmind and it is cited in several of their papers. There are some old videos of him discussing it here: https://www.youtube.com/playlist?list=PL985E81376A4D061E .
I agree with you but for a different reason. In some other sense you don't need to define intelligence. You can assume humans are intellugent and collect data as to how behave and cross validate your machine's bahevior with that. This should include talking, walking, feeling, problem solving, convincing etc... except in a simulated world.
Intelligence is a criteria for which many assorted algorithms may be judged.
The criteria is making an improbable outcome - more probable.
Any process, biological, mechanical or electronic - that results in narrowing the search space is exhibiting intelligence.
Consciousness is not intelligence - it's a feature of biological life that is frequently conflated with intelligence because we have both. No amount of computational power or data creates an emergent consciousness.
From a practical point of view you can look at reproducing the mental skills of humans. Some things computers have got quite good at - go, chess and face recognition for instance. Others like common sense and figuring what people are up to, not so much. It's not so much a single thing as a set of abilities.
Take AlphaGo, because I know Go quite well. A computer is making better decisions than a human, using a pretty reasonable approximation of the same mechanism as a human but scaled up. It needs relatively more resources (gameplay time, expert attention, etc) to a achieve a more focused result than a human expert.
At the moment, it is very easy to suspect that it is only the economics of training a neural net (hardware, specialist attention, data gathering an processing) that is holding us back from AGI.
Human brains can evolve by chance. Nothing fundamental is stopping us from creating a synthetic one. Natural Intelligence isn't that smart either, realistically.
We're getting really good at climbing trees. We've got guys who can climb even the tallest trees, no problem. Any day now, we're gonna finally climb to the moon.
The tree climbers may well build a rocket and land on the moon anyhow.
I think it’s a good idea not to add unwarranted mystery to the subject. We have good reason to believe our computers are underpowered for the task. This is a challenge, and while I think it’s going very fast, our collective sense of proportion of the timeline seems optimistic. In my view, if it happens this century, humans will have achieved an impressive feat.
If we continue with geometric increases in flops/watt, maybe the rosy predictions will be closer than expected.
I suppose what I’m implying is that nervous systems are doing computation, and not something mysterious that we don’t understand. The tree climbers knew they needed to climb higher and realized they’ll need another tool not a different understanding of spatial geometry.
Beneath the marketing, ML is a souped-up version of "draw a line through these points". Great for image recognition etc. But the line/curve/manifold won't just suddenly "wake up" once we add enough points.
There's something deeper missing (maybe something related to recursion or self-reflection?) and we don't even know what it is, much less how to add it. Yes maybe we'll make a breakthrough and build a rocket tomorrow but it won't be because of any insight we gained from tree-climbing.
It's like the Singularitists who passionately believe we're somehow going to transfer our consciousness into a machine, despite the fact that thousands of years of philosophy and science from countless angles of study have yet to actually solve the question of what consciousness even is.
Or, it's solved, and people stubbornly refuse to believe, that it's not much. A software that runs in your brain that makes you think that you are conscious, gives you the feeling of agency when your predictions and the world match up.
The hard problem is how to wire up adaptivity and the specialized faculties.
>> The tree climbers may well build a rocket and land on the moon anyhow.
Sticking with the OP's analogy, the number of people who think it is impossible to get to the mooon in any way is much, much smaller than the number of people who, misunderstanding how tree-climbing works, believe that is the way, that climbing to the moon is just a manner of climbing trees better than we presently do and that if everyone keeps climbing trees someone will reach the moon any time now.
The problem of course, is that there are no trees that reach all the way to the moon. We can keep climbing all we want and we'll never reach as high as the moon. To get to the moon, we need a radically different approach.
A rocket, perhaps- certainly! But, how long have people been climbing trees for, before the first rocket was made?
And where are the people proposing rocket designs, in modern AI research?
> And where are the people proposing rocket designs, in modern AI research?
We like people like Geoffrey Hinton as their work produces results - he is the first to admit you can't get there from here.
I think it's the people who are concerned with fundamentals, like the people at the Sante Fe Institute - complex adaptive systems.
In the beginning we believed in programming AI from the axioms of mathematics - not many people believe that now. A bit like configuring an animal from atoms. Difficult, pointless.
The first story is that 'fundamentals' might mean some evolutionary framework being required to plug in results from other areas like computer vision and machine learning, my guess would be something from genetic algorithms.
The other story is there exists a 'God Algorithm' that all sapient beings possess - but I find that difficult to believe.
Another story is that we simply have to start from the start - literally everything we know is nearly worthless because consciousness is too unlikely with computer hardware - that the entire history of computing is a dead end. Great for human technology but just not the route to a synthetic version of biological beings. I suspect we're more biased against this possibility than we should be.
I hope that it is the first story - my rationale is that biological intelligence and consciousness is not as far away as we think it is. If we look to evolution we see these vast periods of time in which nature works on different paths and it's only in the last few moments that an animal begins building cities and civilizations. There are also lots of the neurological version of transition fossils if you look around - lots of animals exist that exhibit human level cognitive abilities and some of them outright outrank us. I'm sure you've seen those birds assessing situations and solving puzzles, heard of monkeys coming up with original innovations like salted potatoes - a lot of animals outperform human children.
My thought is that when/if we break Moravec's Paradox and develop a robot that can walk, really walk, then we're some way towards developing machines that perform comparably to biological brains.
The catch is that people who think a walking robot is trivial are so wrong - it is spectacularly difficult. Boston Dynamics is impressive until you understand that a three or four year old is the Albert Einstein of getting about on arms and legs. Show me a robot that can mimic a human walking down the street and I'll stand up at the other side of this internet connection and start clapping. Most people start to change the definition of walking because that is easier than admitting we can't do something that looks easy. I didn't say give a machine locomotive power by changing the shape of the machine or its environment. I said walk - I meant walk.
> We have good reason to believe our computers are underpowered for the task.
What are those reasons?
It appeared to me as if we had amassed a terrific reservoir of computational ability - but we wasted nearly all because it was expedient to waste resources when they were ample.
If I gave you a computer with 1 billion times the processing ability of current supercomputers, can you convince me that you'd be capable of replicating the functionality of a biological brain?
Do we have a terrific reservoir of computational ability though? The number are important because they seem to be about a million times too inefficient.
A human brain might take 36.8e15 computations per second [1] and does all this using about ~20 watts! That's pretty impressive at around 1.84e+15 ops / watt vs our current 6e+9 ops / watt [2] from silicon. We have to make about a million times more efficient processors to match the brains efficiency. My 4lb laptop takes what feels like forever to do a limited deep learning task given how inefficient it is.
If you gave just me the sort of computer needed, that alone would very likely not be sufficient. I just don't think we should draw too strong of a negative conclusions before we all have the necessary equipment to match the work of the brain.
But if myself and fellow computer users did have access to efficient powerful computers, we'd probably want to write algorithms that allowed us to make many attempts at the solution per second. I'd imagine writing a simulator to project many agents into where they are competing for resources and many of these simulations taking place concurrently. Something of a multiverse of simulation and optimization.
> Do we have a terrific reservoir of computational ability though?
We do (and I wrote a long post with a list of bullet points proving so before I just deleted it), but I don't think we should waste time debating whether this is true or not true. We're on the same side after all, your objective is my objective.
> But if myself and fellow computer users did have access to efficient powerful computers, we'd probably want to write algorithms that allowed us to make many attempts at the solution per second. I'd imagine writing a simulator to project many agents into where they are competing for resources and many of these simulations taking place concurrently.
I recommend you look at David Krateneur's ideas, he has a video called "The Stupid Ways That We Have Thought About Intelligence". https://www.youtube.com/watch?v=pi7h6nmkvAM
> Something of a multiverse of simulation and optimization.
My 100% unfounded, total wild guess is that a computer with 1 billion times the processing ability of current supercomputers could replicate the functionality of a biological honey bee brain.
From what I can tell though, I agree that there are alot of very subtle, but fundamental mistakes and omissions in both the design and combinations of current generation intelligence algorithms that are impeding a viable path toward general intelligence. Of course, if all we care about is dramatically increasing the efficiency of solving problems humans care about, this totally isn't a problem, because we apparently don't need general intelligence to do that.
I thought I'd scribble on the back of the envelope.
Henry Markram, a neuroscience person with a desire to emulate the human brain believes it takes 1 exaflop. It's a start.
Existing supercomputers reach 50 petaflops. 50 billion petaflops is 50 million exaflops.
Is this 50 million brain EM moments?
It's possible it was in the ballpark of a human brain emulation although I strongly suspect a fair comparison isn't easily comprehensible.
> I agree that there are alot of very subtle, but fundamental mistakes and omissions in both the design and combinations of current generation intelligence algorithms that are impeding a viable path toward general intelligence
I think that is true of specialized intelligence, and that zero progress of any sort has been made on AGI. I don't mind being wrong if you know of primitive examples that qualify.
It is unclear how to begin. Maybe rehearsing something like real biological evolution.
> , if all we care about is dramatically increasing the efficiency of solving problems humans care about, this totally isn't a problem, because we apparently don't need general intelligence to do that.
Daniel Dennett makes this point philosophically in a video I watched yesterday. Other people have made the observation that we're already surrounded by AGIs in the form human society based collective intelligences we know well.
My hunch with regard to neural nets is that we’ve got something that looks like biology, and has a certain level of efficacy, but is missing a lot of the finer details that we’re unaware of in biology. Right now we’re just brute forcing problems with as much computing power and data as possible, biology obviously does better.
True, i think the real feasible way to grow a AI is- to actually repeat the process that grows us. As in- grow a single neuron creature in a challenging environment, and by acceleration, environment acceleration and evolutionary algorithm, herd the smarts toward sentience.
Is it really "making better decisions than a human using a pretty reasonable approximation of the same mechanism as a human" though? To a layman, it looks rather more like an example of a high-powered calculator estimating Nash equilibria by crunching vast amounts of data. More human-like than Deep Blue, certainly, because it's not able to evaluate all possible game states so there's a bit of pattern recognition going on to cut down its search space, but it's succeeding in Go because with Go the higher level of abstractions humans work at is a coping mechanism for not having enough calculating power to simulate outcomes, not the only way of simulating possible outcomes.
The issue that the real world is mostly not composed of optimisation problems with possible moves and win conditions as neatly and completely described in code and as replayable as Go is probably a thornier one for building AGI (as opposed to useful tools) through learning processes than the adequacy of the hardware...
> At the moment, it is very easy to suspect that it is only the economics of training a neural net (hardware, specialist attention, data gathering an processing) that is holding us back from AGI.
This is too early to call. NN acquires knowledge in a fundamentally different way than human.
For example, to have a reasonable understanding of a certain language, like producing a sample of grammatically correct utterance, it definitely won't require a human to read the amount of text of Wikipedia, let alone multiple times.
You're right but to be fair an individual human sits at the very top of billions of years of evolution. We don't have to have to read Wikipedia personally because most of the work was done by evolution, through our ancestors.
We're really just hosts for our genetic code. It's the star of the show, not us.
>> At the moment, it is very easy to suspect that it is only the economics of
training a neural net (hardware, specialist attention, data gathering an
processing) that is holding us back from AGI.
Neural nets have fundamental limitations, particularly their inability to
generalise outside a narrow band around their training set [1].
For instance- you will not find anything in the neural network bibliography
about learning such a fundamental concept as counting, in the sense of finding
the number n+1 that follows from a given number n, for arbitrary n. A neural
net can certainly be trained on that task for numbers in a range [k,n] - but,
given numbers outside this range its performance would "fall off a cliff".
That's because neural networks can't learn general rules such as "x > y → y =
x + k" etc.
Add to this the fact that neural nets, while very good at specific tasks, must
be trained anew for each new task- and you see that there is a big problem in
getting anything approaching "general intelligence" (which might involve any
number of wildly varying tasks) just with neural nets. For one thing- even if
you could train a neural net model for something like a million tasks, there
would remain the question of somehow stitching them all together in a coherent
whole capable of performing the right task at the right time or combining
decisions from multiple models.
Finally, note that nobody is even trying to train a neural net to learn
"general intelligence" end-to-end from examples; first of all, because we have
no idea what constitutes general intelligence and therefore how to collect its
examples.
So it's not a matter of scale. More of a matter that there isn't anyone living
today that has any idea how to get to AGI with any existing technique-
including neural nets.
As an aside, in order to be able to speculate about how close we are to AGI, with neural networks or any other technique, it is important to first unerstand the state of the art in AI.
As usual, I'll recommend the classic AI textbook, "AI - A modern approach" by Russel and Norvig:
Are you really under attack for recommending the standard textbook?
This website's community has got peculiar more and more over the last 3-5 years and not in a positive way. There were the assholes of yesteryear but even they seemed like more sincere assholes than present day assholes.
I'm guessing my comment was construed as being patronising. I can't imagine anyone would downvote a recommendation to read Russel & Norvig. It's a good book.
I feel like Jeff Hawkins and his research group know how to get to AGI. There neural model has demonstrated generalization outside of training set that you say doesn't exist.
Could you please point at the time in that video where this superior generalisation ability is discussed? It's a long video and I'm not sure what I'm looking for.
Better still, could you link to a paper where the relevant results are presented?
Not OP, but I think they are referring to the basic premise of evolution. We still don't have a concrete theory on what causes mutation to occur, only one about which mutations end up surviving and sustaining across generations.
As such the evolution of human intelligence is largely an enormous chain of favorable mutations having been coupled and survived over billions of years. And each mutation that did occur was owing to nothing more than chance.
Since, as you say, we don't have a concrete theory on what causes mutations to occur, how can you then go onto say the reason they occur is nothing more than chance?
> Nothing fundamental is stopping us from creating a synthetic one.
Ethical considerations are preventing us or at the very least limiting the speed of development. But mostly our lack of knowledge of the brain was what stopped us. Now that we are advancing on all fronts, I think we are going to be forced to deal with the ethics of neural/brain creation/destruction/experimentation.
If scientists are able to keep disembodied brains "alive", then it will open up avenues for experimentation and ethical issues.
There are even ethical issues of hybrid organ growth ( starting from the embryonic level ) where human organs are grown in pigs. The question is whether human cells can travel to pig's brains and cause pigs to be "more conscious".
Glad we are finally seeing some gradual pushback on the ridiculous hype of the past few years. What is currently called “AI” can be useful in some very narrow scenarios but even then it’s dangerous when people put too much trust in it (same as any system—computerized, mechanical, or bureaucratic). Blindly trusting the algorithms that brought you the Facebook and Twitter timelines to make decisions about real things in human lives would be disastrous, and we already do far too much of it.
Even for very low-paid labour labelled as "low skilled", there are tons of intricate judgements call. I've read sociology articles on this matter, for instance a woman working in a factory at arranging cakes in boxes. She explained all of the small evaluations, complex decisions she had to make at the quick pace of the factory chain, like how to dispose different cakes to maximize box fill, of get sure that all cakes are properly seated and won't move in transport, etc. She deemed herself "unskilled", however at the same time concluded that it took her many months to master the skill of arranging cakes in boxes.
Ditto for mailmen: now they're using a program call "GeoPost" to "optimize routes"; actually the program is unable to account for myriads of things that an actual, human postman must take into account, for instance the postman is supposed to deliver mail on the even side of the street first, than on the odd side on his return trip an hour later, according to the software. That works well for wide avenues, but in small residential areas people will get out of their house and call the postman if he delivers mail for the opposite-side neighbour but not them. Silly human, with their feelings :)
In fact the craze for AI reminds me so much (once again) of good ol' grandpa Marx. As competition rages and given the tendency of profit to fall, companies try to get rid of living labour and replace it with "dead labour" (machine labour). But once again it meets the wall of value, which only comes from living labour. This all adds up perfectly.
Sigh. Here they go alternating from AI is a "super genius threat" to "it isn't that smart".
Maybe an objective, honest and non-clickbait "Current State of AI" would mean they wouldn't have to swing from one extreme to the other every few months.
However you define "smart", currently AI is mostly domain-specific "smart". Though there have been advances in general AI (non-domain specific AI), but we are nowhere close an autonomous generalized AI.
The state of AI is such that it can be used to detect faces from a crowd to falsely arrest you, deny you a home loan based on nothing, cut you off from social media based on a misunderstanding, and still can't transfer knowledge between domains.
>> In the business world, machine learning often goes by the annoying moniker of “artificial intelligence.” That science-fiction buzzword evokes visions of godlike sentient computers, when in fact, the product is much closer to a statistical regression. Machine learning is about using algorithms to predict things — whether a web-security image contains a cat, what a Google user wants to search for, or whether a self-driving car should brake to avoid a crash. No one yet knows how to give a single computer system the mental flexibility to reason and learn like a human being.
While the last senetence is true, the rest of the opening paragraph of this article commits the typical sin of promulgating a reductionist definition of "machine learning", probably because of a lack of historical perspective into the origins and the use of the term.
Historically, then, the broadest definition of "machine learning" is the one given by Tom Mitchell, according to whom a machine learning system is one that can improve its behaviour over time. This covers machine learning algorithms that are really nothing like statistical regression- decision tree learners, or the wide array of algorithms and systems that learn logic-based representations that predominated in the early years of the discipline (and are still going, if I may, with Inductive Logic Programming).
In short, neither prediction, nor statistical regression are necessary or sufficient characteristics of machine learning. Indeed, there is nothing stopping a system that would learn to reason from examples from being categorised as a "machine learning system".
Nothing- except the incomplete understanding of experts from other disciplines, perhaps, like the author of this piece.
I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error. Regression is also model fittining (polynomials) by minimizing mean squared error. Not a huge difference...
>> I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error.
That's only true if you define machine learning as "model fitting", however there are many algorithms and techniques that have nothing to do with fitting a curve to data points and are very commonly considered to be machine learning.
E.g. decision tree learners, nearest-neighbour algorithms, Bayesian inference algorithms, Expectation-maximisation (and friends), clustering algorithms- none of these really um fit under "curve fitting", although of course you can broaden the scope of what is meant by "model fitting" to include potentially anything that looks kinda like it from a few steps away.
Then of course there is the very broad family of logic-based machine learning systems, pioneered by Ryszard Michalski and others in the '70s and Inductive Logic Programming, that learn logic theories from relational data and do not perform optimisation or model fitting (they tend to be greedy).
Basically, it's mostly the gradient-descent based algorithms that do curve fitting (linear regression, perceptrons, SVMs and, er, that; oh, and gradient boosting). These approaches are currently dominant but that's all.
There is much more to machine learning than curve fitting.
All of those require a training step, and that is the same error minimization as curve fitting.
Basically they are curve fitting in a high enough dimension.
Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it.
That said, of course there is much more to it, than training. (And as we saw with OpenAI's dota bots and DeepMind's AlphaGo Zero, with good feature engineering, adversarial learning setup, beating human performance is only a matter of processing power.)
>> All of those require a training step, and that is the same error minimization as curve fitting.
>> Basically they are curve fitting in a high enough dimension.
>> Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it.
I don't understand most of what you are saying and the bits I understand, I don't see how they're true. Could you please explain in a little more detail? For example, how is, say, recursive partitioning, as in decision tree learners, or storing data points for later comparison, as in KNN, "curve fitting in a high enough dimension"? What is the function being fit and where are its parameters?
Full disclosure: I study ILP algorithms for my PhD and I have implemented a couple myself. There is nothing like optimisation going on in there, just (inverse) resolution and some ordering of the search space. Most have nothing like an "error metric", greedy or not. And I have never heard anyone describe logic-based learners, including ILP algorithms, as curve fitting before.
If something requires training, it constructs a model, the model can be represented as a probability distribution. I think of that as the curve.
ILP (let's say simplex algorithm or other hill climbers) don't require training, but they nevertheless try to optimize the utility (or goal) function (which implicitly defines a metric on the space), which is a hypersurface, which is a curve in an even higher dimension.
Since the cardinality of the set of solutions is not one, and usually infinite, picking one solution over the other is optimization. (And - if I'm not mistaken - unless we know that the search space is convex, we don't know if there are better solutions or not.)
I don't think anything that doesn't build a model is machine learning. (Decision trees build a model, Bayes classifiers build a model, etc. And I'd say all are curve fitting.)
I'm not doing a PhD at all, so I'm not claiming that I'm an authority on each correct terminus technicus for AI/ML/optimization math/CompSci, but to me they fall under the model/curve fitting umbrella.
The Simplex algorithm is a linear programming algorithm. ILP is Inductive Logic Progamming, not linear programming. ILP algorithms do not perform linear programming, do not have a utility function and do not use optimisation. What they do is search a space of hypotheses for a hypothesis that is consistent with some background theory and that explains some set of examples. Wikipedia has an introduction:
https://en.wikipedia.org/wiki/Inductive_logic_programming
You will not be able to understand the operation of ILP algorithms if you try to look at them from the point of view of function optimisation, or, indeed "curve fitting". You need a different set of tools, that of logic programming and automated theorem proving- because the process is essentially that of logical deduction. Please let me know if you delve into the subject and have any questions (see my profile for my email address). I'm always happy to support attempts to understand my research subject.
The same goes for other machine learning algorithms: if you use the wrong abstractions to understand them, you'll get the wrong idea about them.
For instance- KNN doesn't "require training" as per your condition for machine learning. It just memorises all its examples. Is KNN not machine learning? It is- because it gets better the more examples it has. If you stick to thinking of everything as "curve fitting", either you'll misunderstand what KNN does, or you'll start thinking that KNN is not machine learning.
Decision tree nodes are all unique - their probability is always 1/n, where n is the number of all nodes in the tree. In other words, their distribution is uniform. If you try to understand decision trees as "a model represented as a probability distribution" you'll find out that every decision tree learner builds the same model. Obviously that's not true, because each decision tree recognises different classes.
And so on.
When we talk of "curve fitting" what we mean is that we have the function of a line (not necessarily straight) like αx + βy + γz + ... where the variables are the attributes of a set of examples and the constants are unknown and we wish to learn them. These are the "parameters" of the function that are optimised by curve fitting algorithms. We say that a curve is being "fit" because we assume there is some function that we don't know that describes a line whose points are the entire class of entities from where our examples are drawn. And since we don't know what this function is, we instead try to approximate it with a function that "steps" on the few points that we have- or in other words, "fits" those points.
In decision trees, KNN, Bayesian inference, etc, etc, there is no line function and therefore no curve that we are trying to fit. You won't find a line in higher dimensions either. Whereas there is probably some curve that can fit the data points given as training examples to the algorithm, the algorithm, lacking a line function, cannot represent that curve. And while you can sometimes visualise the end result as a curve, this is only a convenience and, more importantly, it is not always possible. For example- decision trees again (because they're easy): look around for visualisations of decision trees. You will find tree diagrams aplenty, but not a single line curve. Because you can't really fit a line on a bunch of decision tree nodes. It's a graph. Where's the line? Where are the parameters?
You've obviously made an effort to organise what you know about machine learning, the better to understand it. But you're missing lots of details and the result is actually hindering your undersanding. In a sense, by looking at everything from the point of view of "curve fitting" you're missing the forest for the um, trees.
My advice is: if you really want to understand machine learning (or at least to sound a bit knowledgeable about it) keep an open mind and don't rush to conclusions. There is much more to machine learning than what you can get from reading a few blogs, a couple of wikipedia pages and a few popular articles in the tech press. Even experts don't yet have a high-level theory of what machine learning is and how to treat different machine learning algorithms in a high-level, unifrom manner, as some well defined class of objects with common characteristics. That should give you a hint about how close your all-encompasing, "it's all curve fitting" definition is to explaining anything.
Interestingly I thought ILP is integer linear programming. (The first hit for ILP algorithm.)
And then I read DeepMind's delta-ILP post you linked and was even more confused as that talked about gradient descent, and then looked at your Metagol example, and ...
So, decision trees are a great example, as they were the first thing I've encountered ~15 years ago about machine learning, and I think it's a first class ML citizen. But they are in their pure form absolutely vulnerable to noise.
And as soon as noise enters the picture it becomes optimization. And I think that's the model fitting that should be the primary concept. (I used curve-slash-model, but I should have used just model.) [And even if there's no "fitting", like with kNN, you still have data, and either you use a constant classification set of points, or you'll see bias, but then the question becomes what set to use.]
And you're right, that treating everything as X is folly, after all, it gives you no information, decreases no entropy. But I'm looking at it from the perspective of what's intelligence. And it seems to be always on this spectrum of how to optimize for something. (Making hard things easy, by learning rules/algorithms.) After all that's why are we really interested in ML, no?
> But you're missing lots of details and the result is actually hindering your undersanding. In a sense, by looking at everything from the point of view of "curve fitting" you're missing the forest for the um, trees.
If you were so kind to elaborate on this, I'm very interested. And in general, anything that comes to your mind about my ramblings! Thanks for your detailed comments!
>> I've studied machine learning a fair amount,
OK, different comment because this one might be construed as a bit personal. I mean, I really hope it's not, but I've heard one too many people recently claim they know about machine learning who only mean they've heard of it in the last five or six years (2012 is the cut-off point) and only know the algorithms that are typically taught in machine learning courses at degree and Master's levels (perceptrons, linear regression, decision trees, naive Bayes, logistic regression, KNN, SVMs, K-means, maybe some deep learning bits) and who wouldn't know what the hell CART is or who Arthur Samuel, or, indeed, Warren McCulloch and Walter Pitts, were.
So I have to ask: when you say "a fair amount"- what exactly do you mean?
But they have invented gradient descent! /s
When the likes of Google, Uber, and Tesla hype “machine learning” as the solution to all problems, then the historical reality of the academics who named it doesn’t really matter anymore. If the press and the financiers believe it’s a silver bullet then the only way to fight that misconception is to use their own terms.
Exactly.
I would add that the OP is motivated, in part, by Corporate America's current understanding of "AI" as little more than "glorified regression" and "glorified classification." There's a tremendous amount of hype, but really very little understanding outside of a few academic centers and Silicon Valley companies.
This is exactly the point of the article. If you can let go of your attachment to the academic meaning, the hype long ago was a done deal, and so the responsibility of the business press is to call BS on what the business world says about ML and AI.
AI is a lot dumber than people expect... which is the scary thing. If you look at Leela Zero's training, you'll see a lot of huge flaws pointing to massive failures of reason and areas where it is frankly incapable of making basic generalizations. Yet variants of this same technology beat the best Go players, synthesize realistic voices, give sentence labels to images, drive cars, translate language, etc. Makes you wonder that, if such an unintelligent bit of math can fake so much, whether intelligence is really that mysterious or whether it just seems that way because we don't know how to write it.
I don't think we are that close to creating artificial intelligence for a very simple reason.
To solve a problem you must first be able to define the problem to be solved. So, define intelligence. Take your time, this comment will wait for you. Done? Good. Now define it in a way that everyone else agrees with and that doesn't fall to pieces under scrutiny.
Philosophers have been trying this for thousands of years and are still no closer to an answer. As far as I can tell we don't have any test for intelligence that is good enough to establish, for absolute certainty, that another human being is 'intelligent' let alone anything else.
Define the problem and the requirements to consider the problem solved and only then do we have any hope of coming up with a solution.
Well, this seems to be the definition Deepmind uses: https://arxiv.org/abs/0712.3329 . And there has been work to establish a quantitative measure of the extent to which an agents behavior is intelligent or mechanical: https://arxiv.org/abs/1805.12387 .
Thanks for sharing such interesting papers!
May I ask what makes you think that the first paper has Deepmind's definition of intelligence? Was it said so by Deepmind at some point?
Shane Legg, one of the authors of the paper, is also the co-founder of Deepmind and it is cited in several of their papers. There are some old videos of him discussing it here: https://www.youtube.com/playlist?list=PL985E81376A4D061E .
I agree with you but for a different reason. In some other sense you don't need to define intelligence. You can assume humans are intellugent and collect data as to how behave and cross validate your machine's bahevior with that. This should include talking, walking, feeling, problem solving, convincing etc... except in a simulated world.
My thoughts are:
Intelligence is not an algorithm.
Intelligence is a criteria for which many assorted algorithms may be judged.
The criteria is making an improbable outcome - more probable.
Any process, biological, mechanical or electronic - that results in narrowing the search space is exhibiting intelligence.
Consciousness is not intelligence - it's a feature of biological life that is frequently conflated with intelligence because we have both. No amount of computational power or data creates an emergent consciousness.
From a practical point of view you can look at reproducing the mental skills of humans. Some things computers have got quite good at - go, chess and face recognition for instance. Others like common sense and figuring what people are up to, not so much. It's not so much a single thing as a set of abilities.
Take AlphaGo, because I know Go quite well. A computer is making better decisions than a human, using a pretty reasonable approximation of the same mechanism as a human but scaled up. It needs relatively more resources (gameplay time, expert attention, etc) to a achieve a more focused result than a human expert.
At the moment, it is very easy to suspect that it is only the economics of training a neural net (hardware, specialist attention, data gathering an processing) that is holding us back from AGI.
Human brains can evolve by chance. Nothing fundamental is stopping us from creating a synthetic one. Natural Intelligence isn't that smart either, realistically.
We're getting really good at climbing trees. We've got guys who can climb even the tallest trees, no problem. Any day now, we're gonna finally climb to the moon.
The tree climbers may well build a rocket and land on the moon anyhow.
I think it’s a good idea not to add unwarranted mystery to the subject. We have good reason to believe our computers are underpowered for the task. This is a challenge, and while I think it’s going very fast, our collective sense of proportion of the timeline seems optimistic. In my view, if it happens this century, humans will have achieved an impressive feat.
If we continue with geometric increases in flops/watt, maybe the rosy predictions will be closer than expected.
I suppose what I’m implying is that nervous systems are doing computation, and not something mysterious that we don’t understand. The tree climbers knew they needed to climb higher and realized they’ll need another tool not a different understanding of spatial geometry.
Beneath the marketing, ML is a souped-up version of "draw a line through these points". Great for image recognition etc. But the line/curve/manifold won't just suddenly "wake up" once we add enough points.
There's something deeper missing (maybe something related to recursion or self-reflection?) and we don't even know what it is, much less how to add it. Yes maybe we'll make a breakthrough and build a rocket tomorrow but it won't be because of any insight we gained from tree-climbing.
It's like the Singularitists who passionately believe we're somehow going to transfer our consciousness into a machine, despite the fact that thousands of years of philosophy and science from countless angles of study have yet to actually solve the question of what consciousness even is.
Or, it's solved, and people stubbornly refuse to believe, that it's not much. A software that runs in your brain that makes you think that you are conscious, gives you the feeling of agency when your predictions and the world match up.
The hard problem is how to wire up adaptivity and the specialized faculties.
>> The tree climbers may well build a rocket and land on the moon anyhow.
Sticking with the OP's analogy, the number of people who think it is impossible to get to the mooon in any way is much, much smaller than the number of people who, misunderstanding how tree-climbing works, believe that is the way, that climbing to the moon is just a manner of climbing trees better than we presently do and that if everyone keeps climbing trees someone will reach the moon any time now.
The problem of course, is that there are no trees that reach all the way to the moon. We can keep climbing all we want and we'll never reach as high as the moon. To get to the moon, we need a radically different approach.
A rocket, perhaps- certainly! But, how long have people been climbing trees for, before the first rocket was made?
And where are the people proposing rocket designs, in modern AI research?
> And where are the people proposing rocket designs, in modern AI research?
We like people like Geoffrey Hinton as their work produces results - he is the first to admit you can't get there from here.
I think it's the people who are concerned with fundamentals, like the people at the Sante Fe Institute - complex adaptive systems.
In the beginning we believed in programming AI from the axioms of mathematics - not many people believe that now. A bit like configuring an animal from atoms. Difficult, pointless.
The first story is that 'fundamentals' might mean some evolutionary framework being required to plug in results from other areas like computer vision and machine learning, my guess would be something from genetic algorithms.
The other story is there exists a 'God Algorithm' that all sapient beings possess - but I find that difficult to believe.
Another story is that we simply have to start from the start - literally everything we know is nearly worthless because consciousness is too unlikely with computer hardware - that the entire history of computing is a dead end. Great for human technology but just not the route to a synthetic version of biological beings. I suspect we're more biased against this possibility than we should be.
I hope that it is the first story - my rationale is that biological intelligence and consciousness is not as far away as we think it is. If we look to evolution we see these vast periods of time in which nature works on different paths and it's only in the last few moments that an animal begins building cities and civilizations. There are also lots of the neurological version of transition fossils if you look around - lots of animals exist that exhibit human level cognitive abilities and some of them outright outrank us. I'm sure you've seen those birds assessing situations and solving puzzles, heard of monkeys coming up with original innovations like salted potatoes - a lot of animals outperform human children.
My thought is that when/if we break Moravec's Paradox and develop a robot that can walk, really walk, then we're some way towards developing machines that perform comparably to biological brains.
The catch is that people who think a walking robot is trivial are so wrong - it is spectacularly difficult. Boston Dynamics is impressive until you understand that a three or four year old is the Albert Einstein of getting about on arms and legs. Show me a robot that can mimic a human walking down the street and I'll stand up at the other side of this internet connection and start clapping. Most people start to change the definition of walking because that is easier than admitting we can't do something that looks easy. I didn't say give a machine locomotive power by changing the shape of the machine or its environment. I said walk - I meant walk.
1950 - https://www.youtube.com/watch?v=Fj0Mr3mLujE
A toy, but not bad for £16.00
66 years, a team of experts and millions of dollars.
2016 - https://www.youtube.com/watch?v=rVlhMGQgDkY
We could be waiting until 2082 or longer.
> We have good reason to believe our computers are underpowered for the task.
What are those reasons?
It appeared to me as if we had amassed a terrific reservoir of computational ability - but we wasted nearly all because it was expedient to waste resources when they were ample.
If I gave you a computer with 1 billion times the processing ability of current supercomputers, can you convince me that you'd be capable of replicating the functionality of a biological brain?
Do we have a terrific reservoir of computational ability though? The number are important because they seem to be about a million times too inefficient.
A human brain might take 36.8e15 computations per second [1] and does all this using about ~20 watts! That's pretty impressive at around 1.84e+15 ops / watt vs our current 6e+9 ops / watt [2] from silicon. We have to make about a million times more efficient processors to match the brains efficiency. My 4lb laptop takes what feels like forever to do a limited deep learning task given how inefficient it is.
If you gave just me the sort of computer needed, that alone would very likely not be sufficient. I just don't think we should draw too strong of a negative conclusions before we all have the necessary equipment to match the work of the brain.
But if myself and fellow computer users did have access to efficient powerful computers, we'd probably want to write algorithms that allowed us to make many attempts at the solution per second. I'd imagine writing a simulator to project many agents into where they are competing for resources and many of these simulations taking place concurrently. Something of a multiverse of simulation and optimization.
[1] https://en.wikipedia.org/wiki/Computer_performance_by_orders... [2] https://en.wikipedia.org/wiki/Performance_per_watt
> Do we have a terrific reservoir of computational ability though?
We do (and I wrote a long post with a list of bullet points proving so before I just deleted it), but I don't think we should waste time debating whether this is true or not true. We're on the same side after all, your objective is my objective.
> But if myself and fellow computer users did have access to efficient powerful computers, we'd probably want to write algorithms that allowed us to make many attempts at the solution per second. I'd imagine writing a simulator to project many agents into where they are competing for resources and many of these simulations taking place concurrently.
I recommend you look at David Krateneur's ideas, he has a video called "The Stupid Ways That We Have Thought About Intelligence". https://www.youtube.com/watch?v=pi7h6nmkvAM
> Something of a multiverse of simulation and optimization.
An artificial imagination really.
I really recommend watching that video.
My 100% unfounded, total wild guess is that a computer with 1 billion times the processing ability of current supercomputers could replicate the functionality of a biological honey bee brain.
From what I can tell though, I agree that there are alot of very subtle, but fundamental mistakes and omissions in both the design and combinations of current generation intelligence algorithms that are impeding a viable path toward general intelligence. Of course, if all we care about is dramatically increasing the efficiency of solving problems humans care about, this totally isn't a problem, because we apparently don't need general intelligence to do that.
I thought I'd scribble on the back of the envelope.
Henry Markram, a neuroscience person with a desire to emulate the human brain believes it takes 1 exaflop. It's a start.
Existing supercomputers reach 50 petaflops. 50 billion petaflops is 50 million exaflops.
Is this 50 million brain EM moments?
It's possible it was in the ballpark of a human brain emulation although I strongly suspect a fair comparison isn't easily comprehensible.
> I agree that there are alot of very subtle, but fundamental mistakes and omissions in both the design and combinations of current generation intelligence algorithms that are impeding a viable path toward general intelligence
I think that is true of specialized intelligence, and that zero progress of any sort has been made on AGI. I don't mind being wrong if you know of primitive examples that qualify.
It is unclear how to begin. Maybe rehearsing something like real biological evolution.
> , if all we care about is dramatically increasing the efficiency of solving problems humans care about, this totally isn't a problem, because we apparently don't need general intelligence to do that.
Daniel Dennett makes this point philosophically in a video I watched yesterday. Other people have made the observation that we're already surrounded by AGIs in the form human society based collective intelligences we know well.
I love this comment.
My hunch with regard to neural nets is that we’ve got something that looks like biology, and has a certain level of efficacy, but is missing a lot of the finer details that we’re unaware of in biology. Right now we’re just brute forcing problems with as much computing power and data as possible, biology obviously does better.
it’s missing a lot of the finer details that we are aware of, too.
a big one is timing — neurons in the real brain can delay firing depending on what signals they receive.
there is research in training artificial neural networks with timing but it is not widely used for AI.
True, i think the real feasible way to grow a AI is- to actually repeat the process that grows us. As in- grow a single neuron creature in a challenging environment, and by acceleration, environment acceleration and evolutionary algorithm, herd the smarts toward sentience.
I think it is more akin to “any day now we shall have the equations of general relativity”. But I like your comment.
Is it really "making better decisions than a human using a pretty reasonable approximation of the same mechanism as a human" though? To a layman, it looks rather more like an example of a high-powered calculator estimating Nash equilibria by crunching vast amounts of data. More human-like than Deep Blue, certainly, because it's not able to evaluate all possible game states so there's a bit of pattern recognition going on to cut down its search space, but it's succeeding in Go because with Go the higher level of abstractions humans work at is a coping mechanism for not having enough calculating power to simulate outcomes, not the only way of simulating possible outcomes.
The issue that the real world is mostly not composed of optimisation problems with possible moves and win conditions as neatly and completely described in code and as replayable as Go is probably a thornier one for building AGI (as opposed to useful tools) through learning processes than the adequacy of the hardware...
> At the moment, it is very easy to suspect that it is only the economics of training a neural net (hardware, specialist attention, data gathering an processing) that is holding us back from AGI.
This is too early to call. NN acquires knowledge in a fundamentally different way than human.
For example, to have a reasonable understanding of a certain language, like producing a sample of grammatically correct utterance, it definitely won't require a human to read the amount of text of Wikipedia, let alone multiple times.
You're right but to be fair an individual human sits at the very top of billions of years of evolution. We don't have to have to read Wikipedia personally because most of the work was done by evolution, through our ancestors.
We're really just hosts for our genetic code. It's the star of the show, not us.
>> At the moment, it is very easy to suspect that it is only the economics of training a neural net (hardware, specialist attention, data gathering an processing) that is holding us back from AGI.
Neural nets have fundamental limitations, particularly their inability to generalise outside a narrow band around their training set [1].
For instance- you will not find anything in the neural network bibliography about learning such a fundamental concept as counting, in the sense of finding the number n+1 that follows from a given number n, for arbitrary n. A neural net can certainly be trained on that task for numbers in a range [k,n] - but, given numbers outside this range its performance would "fall off a cliff". That's because neural networks can't learn general rules such as "x > y → y = x + k" etc.
Add to this the fact that neural nets, while very good at specific tasks, must be trained anew for each new task- and you see that there is a big problem in getting anything approaching "general intelligence" (which might involve any number of wildly varying tasks) just with neural nets. For one thing- even if you could train a neural net model for something like a million tasks, there would remain the question of somehow stitching them all together in a coherent whole capable of performing the right task at the right time or combining decisions from multiple models.
Finally, note that nobody is even trying to train a neural net to learn "general intelligence" end-to-end from examples; first of all, because we have no idea what constitutes general intelligence and therefore how to collect its examples.
So it's not a matter of scale. More of a matter that there isn't anyone living today that has any idea how to get to AGI with any existing technique- including neural nets.
____________________
[1] https://blog.keras.io/the-limitations-of-deep-learning.html
As an aside, in order to be able to speculate about how close we are to AGI, with neural networks or any other technique, it is important to first unerstand the state of the art in AI.
As usual, I'll recommend the classic AI textbook, "AI - A modern approach" by Russel and Norvig:
http://aima.cs.berkeley.edu/
Are you really under attack for recommending the standard textbook?
This website's community has got peculiar more and more over the last 3-5 years and not in a positive way. There were the assholes of yesteryear but even they seemed like more sincere assholes than present day assholes.
I'm guessing my comment was construed as being patronising. I can't imagine anyone would downvote a recommendation to read Russel & Norvig. It's a good book.
I feel like Jeff Hawkins and his research group know how to get to AGI. There neural model has demonstrated generalization outside of training set that you say doesn't exist.
https://youtu.be/dyD0kNZnaAk
A list of some of my favorite talks given by Jeff Hawkins
https://www.youtube.com/playlist?list=PLGtryE0TPZPGReCdQ_F-n...
Here is a synopsis of my complete model with out my complete model of human psychology.
https://groups.google.com/forum/#!topic/artificial-general-i...
Could you please point at the time in that video where this superior generalisation ability is discussed? It's a long video and I'm not sure what I'm looking for.
Better still, could you link to a paper where the relevant results are presented?
https://youtu.be/6ufPpZDmPKA?list=PLGtryE0TPZPGReCdQ_F-n0aWz...
Sparse Distributed Representations, The data model of the brain and the key to how the brain generalizes.
https://youtu.be/izO2_mCvFaw?list=LL3W7v7dYI4PwfYukN4hJCSw&t...
All the working examples he shares in this talk are exciting, but this one stands out.
Yes, I will find the demo this for you.
really now?
> there isn't anyone living today that has any idea how to get to AGI
Not true.
Well then- who has?
> Human brains can evolve by chance.
What do you mean by this?
Not OP, but I think they are referring to the basic premise of evolution. We still don't have a concrete theory on what causes mutation to occur, only one about which mutations end up surviving and sustaining across generations.
As such the evolution of human intelligence is largely an enormous chain of favorable mutations having been coupled and survived over billions of years. And each mutation that did occur was owing to nothing more than chance.
Since, as you say, we don't have a concrete theory on what causes mutations to occur, how can you then go onto say the reason they occur is nothing more than chance?
for a good introduction to order and complexity (including intelligence) arising from nothing, read "the origins of order" by stuart kauffman.
> Nothing fundamental is stopping us from creating a synthetic one.
Ethical considerations are preventing us or at the very least limiting the speed of development. But mostly our lack of knowledge of the brain was what stopped us. Now that we are advancing on all fronts, I think we are going to be forced to deal with the ethics of neural/brain creation/destruction/experimentation.
If scientists are able to keep disembodied brains "alive", then it will open up avenues for experimentation and ethical issues.
https://news.ycombinator.com/item?id=16924764
There are even ethical issues of hybrid organ growth ( starting from the embryonic level ) where human organs are grown in pigs. The question is whether human cells can travel to pig's brains and cause pigs to be "more conscious".
https://www.newsweek.com/it-ethical-grow-human-organs-pigs-4...
Glad we are finally seeing some gradual pushback on the ridiculous hype of the past few years. What is currently called “AI” can be useful in some very narrow scenarios but even then it’s dangerous when people put too much trust in it (same as any system—computerized, mechanical, or bureaucratic). Blindly trusting the algorithms that brought you the Facebook and Twitter timelines to make decisions about real things in human lives would be disastrous, and we already do far too much of it.
For an informed opinion see https://news.ycombinator.com/item?id=17766473
It doesn't have to be intelligent to be usefully applied broadly and upend our economies.
Even for very low-paid labour labelled as "low skilled", there are tons of intricate judgements call. I've read sociology articles on this matter, for instance a woman working in a factory at arranging cakes in boxes. She explained all of the small evaluations, complex decisions she had to make at the quick pace of the factory chain, like how to dispose different cakes to maximize box fill, of get sure that all cakes are properly seated and won't move in transport, etc. She deemed herself "unskilled", however at the same time concluded that it took her many months to master the skill of arranging cakes in boxes.
Ditto for mailmen: now they're using a program call "GeoPost" to "optimize routes"; actually the program is unable to account for myriads of things that an actual, human postman must take into account, for instance the postman is supposed to deliver mail on the even side of the street first, than on the odd side on his return trip an hour later, according to the software. That works well for wide avenues, but in small residential areas people will get out of their house and call the postman if he delivers mail for the opposite-side neighbour but not them. Silly human, with their feelings :)
In fact the craze for AI reminds me so much (once again) of good ol' grandpa Marx. As competition rages and given the tendency of profit to fall, companies try to get rid of living labour and replace it with "dead labour" (machine labour). But once again it meets the wall of value, which only comes from living labour. This all adds up perfectly.
Sigh. Here they go alternating from AI is a "super genius threat" to "it isn't that smart".
Maybe an objective, honest and non-clickbait "Current State of AI" would mean they wouldn't have to swing from one extreme to the other every few months.
However you define "smart", currently AI is mostly domain-specific "smart". Though there have been advances in general AI (non-domain specific AI), but we are nowhere close an autonomous generalized AI.
The state of AI is such that it can be used to detect faces from a crowd to falsely arrest you, deny you a home loan based on nothing, cut you off from social media based on a misunderstanding, and still can't transfer knowledge between domains.
...and still can't avoid running over pedestrians because they were pushing bikes which confused it.
> Here they go alternating from AI is a "super genius threat" to "it isn't that smart".
It’s possible to be both you know. I can think of at least one example of a moron who is also a veritable genius at being threatening.