Uber AI Labs Senior Research Scientist Talks TensorFlow 2.0 [Interview]

There’s no doubt that TensorFlow is one of the most popular machine learning libraries right now. However, newbie developers who want to experiment with TensorFlow often face difficulties in learning TensorFlow; the framework has a not unjustified reputation for having a steep learning curve that can make it hard for developers to get to grips with quickly.

Although TensorFlow 2.0, released back in September, is reportedly
more accessible than its predecessor, what does this really mean for
developers? And how should they approach it if they’re cautious about the
framework’s well-documented steep learning curve?

To find out, Packt Publishing spoke Ankit Jain, who is a Senior Research Scientist at Uber AI Labs and one of the authors of TensorFlow Machine Learning Projects.

In the interview, Ankit gave his perspective on TensorFlow 2.0. He considers ‘eager execution by default’ a major paradigm shift and is excited about interoperability between TensorFlow 2.0 and other machine learning frameworks.

He also shared insight into the limitations of AI algorithms (generalization, AI ethics, labelled data to name a few) - an important topic in the context of bias and discrimination.

Why TensorFlow 2 upgrade is paradigm shifting in more ways than one

TensorFlow 2 was released last month. What are some of your top features in TensorFlow 2.0? How do you think it has upgraded the machine learning ecosystem?

TF 2.0 is a major upgrade from its predecessor in many ways. It addressed many of the shortcomings of TF 1.x and with this release the difference between PyTorch and TF has narrowed.

One of the biggest paradigm shifts in TF 2.0 is eager execution by default. This means you don’t have to pre-define a static computation graph, create sessions, deal with unintuitive interface or have painful experience in debugging your deep learning model code. However, you lose on some performance in run time when you switch to complete eager mode. For that purpose, they have introduced tf.function decorator which can help you translate your Python functions to TensorFlow graphs. This way you can retain both code readability and ease of debugging while getting the performance of TensorFlow graphs.

Another major update is that many confusing redundancies have been consolidated and many functions are now integrated with Keras API. This will help to standardize the communication of data/models among various components of TensorFlow ecosystem. TF 2.0 also comes with backward compatibility to TF 1.X with an easy optional way to convert your TF 1.X code into TF 2.0.

TF 1.X suffered from lack of standardization in how we load/save trained machine learning models. TF 2.0 fixed this by defining a single API SavedModels. As SavedModels is integrated with TensorFlow ecosystem, it becomes much easier to deploy models using TensorFlow Lite, TensorFlow.js to other devices/applications. 

With the onset of TensorFlow 2, TensorFlow and Keras are integrated into one module (tf.keras). TF 2.0 now delivers Keras as the central high-level API used to build and train models. What is the future/benefits of TensorFlow + Keras?

Keras has been a very popular high level API for faster prototyping and production and even for research. As the field of AI/ML is in nascent stages, ease of development can have a huge impact for people getting started in machine learning.

Previously, a developer new to machine learning started from Keras while an experienced researcher used only TensorFlow 1.x due to its flexibility to build custom models. With Keras integrated as high level API for TF 2.0, we can expect both beginners and experts working on same framework which can lead to better collaboration and better exchange of ideas in the community.

Additionally, a single high level easy to use API reduces confusion and streamlines consistency across use cases of production and research.

Overall, I think it’s a great step in the right direction by Google which will enable more developers to hop on Tensorflow ecosystem.

TensorFlow, NLP and structured learning

Recently, Transformers 2.0, a popular OS NLP library, was released that provides interoperability between TensorFlow 2.0 and PyTorch. What’s your take on it?

One of the areas where deep learning has made an immense impact is Natural Language Processing (NLP). Research in NLP is moving very fast and it is hard to keep up with all the papers and code releases by various research groups around the world.

Hugging Face, the company behind the library “Transformers” has really eased the usage of state of the art (SOTA) models and process of building new models by simplifying the preprocessing and model building pipeline through an easy to use Keras like interface.

“Transformers 2.0” is the recent release from the company and the most
important feature is the interoperability between PyTorch and TF 2.0. TF 2.0 is more production ready while PyTorch is more oriented towards research. With this upgrade, you can pretty much move from one framework to another for training, validation and deployment of the model.

Interoperability between frameworks is very important for the AI community as it enables development velocity. Moreover, as none of the frameworks can be perfect at everything, it makes the framework developers focus more on their strengths and make those features seamless. This will create greater efficiency going forward.

Overall, I think this is a great development and I expect other libraries in domains like Computer Vision, Graph Learning etc. to follow suit. This will enable a lot more application of state-of-the-art models to production.

Google recently launched Neural Structured Learning (NSL), an open-source Tensorflow based framework for training neural networks with graphs and structured data. What are some of the potential applications of NSL? What do you think can be some Machine Learning Projects based around NSL?

Neural structured learning is a concept of learning using neural network parameters with structured signals as well as more typical features. Many real-world datasets contain some structured information like Knowledge graphs or molecular graphs in biology. Incorporating these signals can lead to a more accurate and robust model. From an implementation perspective, it boils down to adding a regularizer to the loss function such that representation of neighboring nodes in the graph are similar.

Any application where the amount of labelled data is limited but has structural information like a Knowledge Graph that can be exploited is a good candidate for these types of models. A possible example could be fraud detection in online systems.

Fraud data generally has sparse labels and fraudsters create multiple accounts which are connected to each other through some information like devices etc. This structured information can be utilized to learn a better representation of fraud accounts.

Ankit Jain on his book TensorFlow Machine Learning Projects

Tell us the motivation behind writing your bookTensorFlow Machine Learning Projects. Why is TensorFlow ideal for building machine learning projects? What are some of your favorite machine learning projects from this book?

When I started learning TensorFlow, I stumbled upon many tutorials (including the official ones) which explained various concepts on how TensorFlow works. While that was helpful in that it allowed me to understand the basics, most of my learning came from building projects with TensorFlow. That is when I realized the need for a resource that teaches using a ‘learn by doing’ approach.

This book is unique in the way that it teaches machine learning theory, TensorFlow utilities and programming concepts all while developing a project which you can have fun building and is also of practical use.

My favorite chapter from the book is “Generating Uncertainty in Traffic Signs Classifier using Bayesian Neural Networks”. With the development of self-driving cars, traffic signs detection is a major problem that needs to be solved. This chapter explains an advanced AI concept of Bayesian Neural Networks and shows step by step how to use those to detect traffic signs using Tensorflow. Some of the readers of the book have started to use this concept in their practical applications already.

Machine Learning challenges and advice to those developing TensorFlow models

What are the biggest challenges today in the field of Machine Learning and AI? What do you see as the greatest technology disruptors in the next 5 years?

While AI and machine learning has seen huge success in recent years, there are few limitations of AI algorithms as we see today. Some of the major ones are:

• Labeled Data: Most of the success of AI has come from supervised learning. Many of the recent supervised deep learning algorithms require huge quantities of labelled data which is expensive to obtain. For example, obtaining huge amounts of clinical trial data for healthcare prediction is very challenging. The good news is that there is some research around building good ML models using sparse data labels.

• Explainability: Deep learning models are essentially a “black box” where you don’t know what factor(s) led to the prediction. For some applications like money lending, disease diagnosis, fraud detection etc. the explanations of predictions become very important. Currently, we see some nascent work in this direction with LIME and SHAP libraries.

• Generalization: In the current state of AI, we build one model for each application. We still don’t have good generality of models from one task to another. Generalization, if solved, can lead us to truly Artificial General Intelligence (AGI). Thankfully approaches like transfer learning and meta learning are trying to solve this challenge.

• Bias, Fairness and Ethics: An output of machine learning model is heavily based on the input training data. Many a time, training data can have biases towards particular ethnicities, classes, religions etc. We need more solutions in this direction to build trust in AI algorithms.

Overall, I feel, AI is becoming mainstream and in the next 5 years we will see many traditional industries adopt AI to solve critical business problems and achieve more automation. At the same time, tooling for AI will keep on improving which will also helps in its adoption.

What advice do you have for those developing machine learning projects on TensorFlow?

Building projects with new techniques and technologies is a hard process. It requires patience, dealing with failures and hard work. For that reason, it is very important to pick up a project that you are passionate about. This way, you will continue building even if you are stuck somewhere. The selection of the right projects is by far the most important criterion in the project-based learning method.

You can find out more about Ankit Jain’s book on the Packt store here. Packt Publishing is a technology publisher that provides resources on a diverse range of emerging technologies. A key player in a competitive space that includes O’Reilly, Pluralsight, Manning, and DataCamp, Packt’s mission is to help software developers keep their skills up to date with practical and accessible content.