Last month, experts from the AI world came together for the Artificial Intelligence Conference at San Francisco to discuss insights, opportunities, challenges, and trends related to the rapidly expanding field of AI. The conference included hands-on trainings, tutorials, startup showcase (which was won by Clobotics), keynotes, sessions, expo, and social events.
Here is the report on Key Takeaways from AI Conference SF, Day 1: Domain Specific Architectures, Emerging China, AI Risks. Below is the report on Day 2.
Across the keynotes and sessions (on 9/6 and 9/7), the following points appeared in multiple talks and do provide a sense of current prevailing trends:
Domain Specific Architectures – the next big thing
Despite recent advances the CPU/GPU architectures are falling behind to keep up with the immense computation appetite of training tasks for AI projects. DSAs provide a great opportunity for innovation as hardware and software are designed from the scratch focused on a very specific goal.
Emerging China – evolving from copying ideas to true innovation
For decades China lagged far behind the west in the field of technology innovation. However, in the field of AI China has made formidable progress in the last few years including several unicorn startups and initiatives from major technology leaders (such as Alibaba and Baidu). In the next 5 years, it is seeming increasingly possible that China can outpace US in the field of AI.
Addressing Risks in AI – Security, Privacy, and Ethics
As more and more AI solutions make their way to mainstream adoption, the inherent risks become increasingly important and harder to ignore. The concerns here include not just data breaches and privacy violations but also AI systems being hacked to execute malicious intent leading to fatal incidents.
The key takeaways from Day 1 (Thursday, September 6th, 2018), can be found **here.**
Here are the key takeaways from Day 2 (Friday, September 7, 2018):
Dawn Song, Professor, UC Berkeley and founder CEO, Oasis Labs delivered a highly informative keynote on AI and Security: Lessons, Challenges, and Future Directions. As we are seeing an exponential growth in the capabilities and deployment of deep learning systems, there has also been an increasing trend in the scale and sophistication of malicious attacks on deep learning systems. In recent years, we had one of the biggest DDoS (Distributed Denial of Service) attacks in history (Mirai Botnet, which had over 1 Tbps combined attack traffic) and some of the biggest data breaches (eg. Yahoo - 3 billion accounts impacted). The attacks have also become a lot more sophisticated recently such as hackers causing power outage (impacting 250,000 customers in Ukraine) and attacking the SWIFT banking system stealing millions of dollars.
As AI becomes more and more capable, the consequence of misuse by attacker is becoming more and more severe. Hackers can target the integrity of AI solutions leading to incorrect results or even worse, the targeted outcome designed by hacker.
Through an example of self-driving cars reading from pictures of road side signs, she showed how attackers can fool the learning system of self-driving cars through simple acts of putting some well-designed stickers on the roadside signs which can lead to fatal errors – such as the manipulated stop sign being mis-classified as “speed limit sign 45 km/h”. Such manipulations of road side signs cannot be detected by humans.
Attackers can even modify physical objects in real world creating adversarial examples that remain effective under different viewing distances, angels, and other conditions.
Such examples fall under the realm of Adversarial Machine Learning which studies ML in the presence of adversaries. It also includes situations where Evasion Attacks where attackers fool the learning system by delaying the inference time beyond a threshold to escape malware detection or fraud detection. Other cases include Poisoning attacks, where attacker poisons training dataset to fool learning system to learn wrong model (eg. Microsoft’s Tay twitter chatbot). Such attacks are very difficult to detect.
She explained how the current ML frameworks are insufficient for protecting privacy and provided recommendations on solving the problem.
One of her recent work led to the finding that neural networks remember training data and attackers can extract secrets (in Training Data) from (querying) Learned Models. She showed how an attacker can extract SSNs and credit card numbers from a language model trained on Enron email dataset.
She introduced the notion of Differential Privacy, in which a neural network is trained in a way preventing memorization and thus, blocking attackers from extracting secrets. She described the Optio system designed and developed by her team to enable privacy-preserving shared analytics.
To address the problem of untrusted architecture, she recommended the idea of Secure Enclave which can authenticate device, authenticate software, and guarantee the confidentiality of execution. Keystone, an open-source secure enclave built by Berkeley & MIT, remedies these issues, and can be publicly analyzed and verified. Hardware manufactures are addressing this problem through a variety of trusted hardware solutions.
Dawn’s startup Oasis Labs is working on privacy-first cloud computing solutions based on blockchain for the democratization of AI. One of the applications is Kara, a privacy-preserving tokenized data market for medical data, which incentivizes doctors and patients to share data to improve medical research without compromising privacy.
Huma Abidi, Engineering Director, Intel AI Products Group gave a keynote on Accelerating AI on Xeon through SW optimization. She informed that recent advances at Intel in hardware and software optimization for AI have led to 200x increase in performance for training and 250x for inference. Explaining the Intel AI portfolio, she explained why one size does not fit all. MKL-DNN (Math Kernel Library for Deep Neural Networks) is an open-source library that highly optimizes the kernel implementations of popular DNNs. nGraph is a framework-neutral compiler that acts as an abstraction layer between software frameworks and underlying hardware architectures. Intel powered solution won the Best Inference Result at the Stanford Dawnbench AI competition.
David Patterson, Turing award winner and professor, UC Berkeley delivered a very encouraging keynote on A New Golden Age for Computer Architecture. Moore’s law is dead. Compared to Moore’s law the current processor technology is lagging by about 15x. Similarly, Dennard Scaling law is also outdated (particularly since 2006). Today, the key bottleneck for advancing hardware design is managing power.
In terms of architecture, a fundamental change has been the limitations and inefficiencies in exploiting instruction level parallelism, which ended the uniprocessor era in 2004. Furthermore, Amdahl’s law and its implications ended the “easy” multicore era.
From software-centric perspective, the key opportunities for performance improvement are to manage ease-of-programming vs computational efficiency. The modern scripting languages are interpreted, dynamically-typed, and encourage reuse. They are efficient for programmers but not for execution. From hardware-centric perspective the key opportunities are domain-specific architectures, which do just a few tasks but do them extremely well. More importantly, a combination of domain-specific languages and architectures can provide a significant boost in computational performance. The size of this opportunity can be seen in the chart below – it can make a typical Python code run up to 63,000 times faster!!!
Domain Specific Architectures are very aptly suited to drive the next wave of improvements because of their more effective parallelism for a specific domain (SIMD vs MIMD, VLIW vs Speculative), more effective use of memory bandwidth (user controlled vs. caches), elimination of unneeded accuracy (32-64 bit integers to 8-16 bit integers), and a very closed coupled software focused on performance gains.
Deep learning is causing a machine learning revolution. For one to keep up with research updates in the field of machine learning, one needs to read 50 papers on arxiv a day as of now, while that numbers keeps growing exponentially. The compute hungry ML Training tasks have been growing by 10x per year, far above the Moore’s Law.
The version 1 of Google TPU (Tensor Processing Unit) launched in 2015 had a peak capacity of 92 TeraFLOPs per second. TPU v3 (liquid cooled) launched in May 2018 has computation capacity of 100+ PetaFLOPs per second.
He concluded his talk with the assertion that we are living in the Golden Age for Computer Architecture and we will witness a lot of exciting innovation in next few years.
Related: