深度學習領域取得了指數級的發展,像BERT、GPT-3、ResNet等ML模型的足跡也在不斷擴大。雖然它們工作得很好,但在生產中訓練和部署這些大型(且不斷增長的)模型是昂貴的。你可能想在智能手機上部署你的面部濾鏡模型,讓你的用戶在他們的自拍上添加一個小狗濾鏡。但它可能太大或太慢,或者您可能想提高基于云的垃圾郵件檢測模型的質量,但又不想花錢購買更大的云VM來承載更精確但更大的模型。如果您的模型沒有足夠的標記數據,或者不能手動調優您的模型,該怎么辦? 所有這些都是令人生畏的!
如果您可以使您的模型更高效: 使用更少的資源(模型大小、延遲、訓練時間、數據、人工參與),并提供更好的質量(準確性、精確度、召回等),會怎么樣呢?這聽起來太棒了! 但如何?
這本書將通過在谷歌研究,Facebook人工智能研究(FAIR),和其他著名的人工智能實驗室使用算法和技術的研究人員和工程師訓練和部署他們的模型,設備從大型服務器端機器到微型微控制器。在這本書中,我們提出了一個基本的平衡,以及實踐知識,以充分賦能你繼續前進,并優化你的模型訓練和部署工作流,這樣你的模型表現和以前一樣好或更好,與一小部分資源。我們還將深入介紹流行的模型、基礎設施和硬件,以及具有挑戰性的項目,以測試您的技能。
目錄內容:
Part I: 高效深度學習導論 Introduction to Efficient Deep Learning 導論 Introduction Introduction to Deep Learning Efficient Deep Learning Mental Model of Efficient Deep Learning Summary
Part II: Effciency Techniques 壓縮技術導論 Introduction to Compression Techniques An Overview of Compression Quantization Exercises: Compressing images from the Mars Rover Project: Quantizing a Deep Learning Model Summary 學習技術導論 Introduction to Learning Techniques
Project: Increasing the accuracy of an speech identification model with Distillation. Project: Increasing the accuracy of an image classification model with Data Augmentation.
Project: Increasing the accuracy of a text classification model with Data Augmentation. Learning Techniques and Efficiency Data Augmentation Distillation Summary
高效架構 Efficient Architectures Project: Project: Snapchat-Like Filters for Pets Project: News Classification Using RNN and Attention Models Project: Using pre-trained embeddings to improve accuracy of a NLP task. Embeddings for Smaller and Faster Models Learn Long-Term Dependencies Using Attention Efficient On-Device Convolutions Summary
高級壓縮技術 Advanced Compression Techniques Exercise: Using clustering to compress a 1-D tensor. Exercise: Mars Rover beckons again! Can we do better with clustering? Exercise: Simulating clustering on a dummy dense fully-connected layer Project: Using Clustering to compress a deep learning model Exercise: Sparsity improves compression Project: Lightweight model for pet filters application Model Compression Using Sparsity Weight Sharing using Clustering Summary
高級學習技術 Advanced Learning Techniques Contrastive Learning Unsupervised Pre-Training Project: Learning to classify with 10% labels. Curriculum Learning 自動化 Automation Project: Layer-wise Sparsity to achieve a pareto optimal model. Project: Searching over model architectures for boosting model accuracy. Project: Multi-objective tuning to get a smaller and more accurate model. Hyper-Parameter Tuning AutoML Compression Search
Part 3 - Infrastructure
軟件基礎 Software Infrastructure PyTorch Ecosystem iOS Ecosystem Cloud Ecosystems 硬件基礎 Hardware infrastructure GPUs Jetson TPU M1 / A4/5? Microcontrollers
Part 3 - Applied Deep Dives Deep-Dives: Tensorflow Platforms Project: Training BERT efficiently with TPUs. Project: Face recognition on the web with TensorFlow.JS. Project: Speech detection on a microcontroller with TFMicro. Project: Benchmarking a tiny on-device model with TFLite.
Mobile Microcontrollers Web Google Tensor Processing Unit (TPU) Summary Deep-Dives: Efficient Models Project: Efficient speech detection models. Project: Comparing efficient mobile models on Mobile. Project: Training efficient BERT models.
BERT MobileNet EfficientNet architectures Speech Detection
Along with the massive growth of the Internet from the 1990s until now, various innovative technologies have been created to bring users breathtaking experiences with more virtual interactions in cyberspace. Many virtual environments with thousands of services and applications, from social networks to virtual gaming worlds, have been developed with immersive experience and digital transformation, but most are incoherent instead of being integrated into a platform. In this context, metaverse, a term formed by combining meta and universe, has been introduced as a shared virtual world that is fueled by many emerging technologies, such as fifth-generation networks and beyond, virtual reality, and artificial intelligence (AI). Among such technologies, AI has shown the great importance of processing big data to enhance immersive experience and enable human-like intelligence of virtual agents. In this survey, we make a beneficial effort to explore the role of AI in the foundation and development of the metaverse. We first deliver a preliminary of AI, including machine learning algorithms and deep learning architectures, and its role in the metaverse. We then convey a comprehensive investigation of AI-based methods concerning six technical aspects that have potentials for the metaverse: natural language processing, machine vision, blockchain, networking, digital twin, and neural interface, and being potential for the metaverse. Subsequently, several AI-aided applications, such as healthcare, manufacturing, smart cities, and gaming, are studied to be deployed in the virtual worlds. Finally, we conclude the key contribution of this survey and open some future research directions in AI for the metaverse.
Data processing and analytics are fundamental and pervasive. Algorithms play a vital role in data processing and analytics where many algorithm designs have incorporated heuristics and general rules from human knowledge and experience to improve their effectiveness. Recently, reinforcement learning, deep reinforcement learning (DRL) in particular, is increasingly explored and exploited in many areas because it can learn better strategies in complicated environments it is interacting with than statically designed algorithms. Motivated by this trend, we provide a comprehensive review of recent works focusing on utilizing DRL to improve data processing and analytics. First, we present an introduction to key concepts, theories, and methods in DRL. Next, we discuss DRL deployment on database systems, facilitating data processing and analytics in various aspects, including data organization, scheduling, tuning, and indexing. Then, we survey the application of DRL in data processing and analytics, ranging from data preparation, natural language processing to healthcare, fintech, etc. Finally, we discuss important open challenges and future research directions of using DRL in data processing and analytics.
In the past decade, we have witnessed the rise of deep learning to dominate the field of artificial intelligence. Advances in artificial neural networks alongside corresponding advances in hardware accelerators with large memory capacity, together with the availability of large datasets enabled researchers and practitioners alike to train and deploy sophisticated neural network models that achieve state-of-the-art performance on tasks across several fields spanning computer vision, natural language processing, and reinforcement learning. However, as these neural networks become bigger, more complex, and more widely used, fundamental problems with current deep learning models become more apparent. State-of-the-art deep learning models are known to suffer from issues that range from poor robustness, inability to adapt to novel task settings, to requiring rigid and inflexible configuration assumptions. Ideas from collective intelligence, in particular concepts from complex systems such as self-organization, emergent behavior, swarm optimization, and cellular systems tend to produce solutions that are robust, adaptable, and have less rigid assumptions about the environment configuration. It is therefore natural to see these ideas incorporated into newer deep learning methods. In this review, we will provide a historical context of neural network research's involvement with complex systems, and highlight several active areas in modern deep learning research that incorporate the principles of collective intelligence to advance its current capabilities. To facilitate a bi-directional flow of ideas, we also discuss work that utilize modern deep learning models to help advance complex systems research. We hope this review can serve as a bridge between complex systems and deep learning communities to facilitate the cross pollination of ideas and foster new collaborations across disciplines.
This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.
數據科學和人工智能是令人著迷的計算領域。微軟在這些新技術上下了很大的賭注,但我們也知道,數據科學家都是訓練有素的專業人士,并不是每個軟件開發人員都能創建和維護復雜的數據模型,執行線性代數或購買昂貴的GPU設備來運行這些模型。這正是我們創造認知服務的原因。這套服務提供了預訓練模型,您可以使用開箱即用的模型來執行視覺、語音、知識、搜索和語言方面的操作。在本次會議上,微軟的云開發者倡導者Laurent Bugnion將向您展示如何使用認知服務增強應用程序的高級功能,如何使用自己的數據細化訓練過的模型,以及如何將認知服務與其他Azure服務集成以實現任務自動化。
由Marc Peter Deisenroth,A Aldo Faisal和Cheng Soon Ong撰寫的《機器學習數學基礎》“Mathematics for Machine Learning” 最新版417頁pdf版本已經放出,作者表示撰寫這本書旨在激勵人們學習數學概念。這本書并不打算涵蓋前沿的機器學習技術,因為已經有很多書這樣做了。相反,作者的目標是通過該書提供閱讀其他書籍所需的數學基礎。這本書分為兩部分:數學基礎知識和使用數學基礎知識進行機器學習算法示例。值得初學者收藏和學習!
This paper surveys the machine learning literature and presents machine learning as optimization models. Such models can benefit from the advancement of numerical optimization techniques which have already played a distinctive role in several machine learning settings. Particularly, mathematical optimization models are presented for commonly used machine learning approaches for regression, classification, clustering, and deep neural networks as well new emerging applications in machine teaching and empirical model learning. The strengths and the shortcomings of these models are discussed and potential research directions are highlighted.
This tutorial is based on the lecture notes for the courses "Machine Learning: Basic Principles" and "Artificial Intelligence", which I have (co-)taught since 2015 at Aalto University. The aim is to provide an accessible introduction to some of the main concepts and methods within machine learning. Many of the current systems which are considered as (artificially) intelligent are based on combinations of few basic machine learning methods. After formalizing the main building blocks of a machine learning problem, some popular algorithmic design patterns formachine learning methods are discussed in some detail.