randcraw 7 years ago

Recent advances in AI still have some big limitations, especially that all the benchmarks at which it has excelled are 1) synthetic and 2) end points.

First, real world telemetry is very often less clean than the synthetic data presently used to train face or word recognizers. In the wild, bad lighting, poor focus, motion artifacts, occlusions, odd angles, floppy hats and unfamiliar expressions will reduce accuracy enormously when used in the real world. This often isn't a game stopper when the usage is lightweight (single picture matches, willingness to re-scan the object, patience from the user, etc). But in "live fire" situations like driving a car, the cost of misidentifying road signage, pedestrian behavior, or failing to perceive imminent risk can be very dire. In fact, good performance in synthetic situs like a lab are a LONG way from practical utility outside the lab.

Second, because these benchmarks are end points, where the only objective is to put a label on each input and add them up to gain a high total score, there is little impact after a label fails, when it misclassifies an object or misunderstands a word, and has to deal with the consequence.

Beyond mere benchmarks, when a situated AI is obliged to process a sequence of data and reactions to each appropriately (driving past street signs like "BRIDGE OUT" or "BEWARE OF DOWNED POWER LINES" or "DEAF CHILD") and the AI fails to understand some or all of an unfamiliar observation, unless it's also able to ask itself questions like "What ELSE might that sign mean?" or "Maybe I should slow down here because I may have misread, or because flooding is common hereabouts", the consequences of a 5% error rate on synthetic benchmarks looks a lot less favorable.

IMHO, AI is still a long way from doing more than advising humans. In a great many practical use cases, I suspect AI is a nowhere near robust and reliable enough to take the reigns from the status quo.

  • hacker_9 7 years ago

    I agree, and there has already been one fatality from when a Tesla system failed to classify a lorry that was in the way. What we have is really a great advancement of mimicking the sensory organs (eyes, ears), but we still haven't figured out the thinking component.

    'Seeing' is more than just classifying pixels, it's also about understanding the visual relationships between objects and being able to act on that data. How this fits into our current view of neural networks I've no idea, and worry that self driving engineers just fill this part in with lots of 'if .. else if ...' statements, which of course really isn't the same as actual thinking.

    • greglindahl 7 years ago

      A fatality caused by a driver not remaining in control of the car. If the Tesla system was Level 3, it would be Tesla's fault. It was Level 2.

amelius 7 years ago

I'm looking at this table of applications, but one application is mysteriously missing: Search. What kind of ML is Google using in its engine?

  • jorgemf 7 years ago

    On one hand you have all the information it extracts from non text sources: books, images, audio, video, non English languages, etc. They use machine learning from them. On the other her hand they also use ML in the search engine to understand better your query (that is why it shows results that sometime doesn't have the keywords you typed), to rank the results based on what you would like more, etc

    So the question is where Google doesn't use ML. If there is data there is a way to apply ML and create value.

mark_l_watson 7 years ago

Great article. I also believe that for the near term (next ten years [1]) the most significant use of AI will be systems that cooperate with human expert users.

I also agree that leaders who are creative in determining novel use of ML and who support rapid experimentation, might have more impact than technology leads.

[1] I believe that further out than ten years, general AI systems with broad knowledge and less need for labeled training data will become more valuable than AI/human teams.

  • radarsat1 7 years ago

    > the most significant use of AI will be systems that cooperate with human expert users.

    except instead of calling it AI, we'll call it "photoshop filters" :P