What Is TensorFlow 2?

TensorFlow is an open-source distributed numerical computation framework released by Google. It’s mainly intended to alleviate the painful details of implementing a neural network (for example, computing derivatives of the loss function with respect to the weights of the neural network). TensorFlow takes this a step further by providing efficient implementations of such numerical computations using Compute Unified Device Architecture (CUDA), which is a parallel computational platform introduced by NVIDIA. The application programming interface (API) of TensorFlow shows that TensorFlow provides thousands of operations that make our lives easier.

TensorFlow was not developed overnight. This is a result of the persistence of talented, good-hearted developers and scientists who wanted to make a difference by bringing deep learning to a wider audience. If we’re interested, we can take a look at the TensorFlow code. Currently, TensorFlow has around 3,000 contributors, and it sits on top of more than 115,000 commits, evolving to be better and better every day.

Getting started with TensorFlow 2

Now let’s learn about a few essential components in the TensorFlow framework by working through a code example. Let’s write an example to perform the following computation, which is very common for neural networks:

$$h= sigmoid(W \cdot x+b)$$

This computation encompasses what happens in a single layer of a fully connected neural network. Here $W$ and $x$ are matrices, and $b$ is a vector. Then, “$\cdot$” denotes the dot product. The $sigmoid$ function is a nonlinear transformation given by the following equation:

$$sigmoid(x) = \dfrac{1}{1 + e^{-x}}$$

Implementing the single layer

We’ll discuss how to do this computation through TensorFlow step by step.

First, we’ll need to import TensorFlow and NumPy. NumPy is another scientific computation framework that provides various mathematical and other operations to manipulate data. Importing them is essential before we run any type of TensorFlow or NumPy-related operation in Python:

import tensorflow as tf
import numpy as np 

First, we’ll write a function that can take the inputs x, W, and b and perform this computation for us:

def layer(x, W, b):
  # Building the graph
  h = tf.nn.sigmoid(tf.matmul(x, W) + b) # Operation to perform 
  return h

Put simply, a Python decorator is just another function. A Python decorator provides a clean way to call another function whenever we call the decorated function. In other words, every time the layer() function is called, tf.function() is called. This can be used for various purposes, such as:

• Logging the content and operations in a function
• Validating the inputs and outputs of another function

When the layer() function is passing through tf.function(), TensorFlow will trace the content (in other words, the operations and data) in the function and build a computational graph automatically.

The computational graph (also known as the dataflow graph) builds a directed acyclic graph (DAG) that shows what kind of inputs are required and what sort of computations need to be done in the program.

In our example, the layer() function produces h by using inputs x, W, and b, and some transformations or operations, such as + and tf.matmul():

If we look at an analogy for a DAG, if we think of the output as a cake, then the graph would be the recipe to make that cake using ingredients (that is, inputs).

The feature that builds this computational graph automatically in TensorFlow is known as AutoGraph. AutoGraph is not just looking at the operations in the passed function; it also scrutinizes the flow of operations. This means that we can have if statements or for/while loops in our function, and AutoGraph will take care of those when building the graph. We’ll see more on AutoGraph in the next section.

Note: In TensorFlow 1.x, we needed to implement the computational graph explicitly. This meant we could not write typical Python code using if-else statements or for loops, but had to explicitly control the flow of operations using special bespoke TensorFlow operations, such as tf.cond() and tf.control_dependencies(). This is because, unlike TensorFlow 2.x, TensorFlow 1.x didn’t immediately execute operations when we called them. Rather, after they were defined, they needed to be executed explicitly using the context of a TensorFlow Session. For example, when we run the following in TensorFlow 1,

$$h = tf.nn.sigmoid(tf.matmul(x,W) + b)$$

h will not have any value until h is executed in the context of a Session. Therefore, h could not be treated like any other Python variable. Don’t worry if it isn’t clear how the Session works. It will be discussed in the coming sections.

Next, we can use this function right away as follows:

x = np.array([[0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]], dtype=np.float32)

Here, x is a simple NumPy array:

init_w = tf.initializers.RandomUniform(minval = -0.1, maxval = 0.1) (shape = [10, 5])
W = tf.Variable(init_w, dtype=tf.float32, name='W')

init_b = tf.initializers.RandomUniform()(shape = [5])
b = tf.Variable(init_b, dtype=tf.float32, name='b')

W and b are TensorFlow variables defined using the tf.Variable object. W and b hold tensors. A tensor is essentially an $n$-dimensional array. For example, a one-dimensional vector or a two-dimensional matrix are called tensors. A tf.Variable is a mutable structure, which means the values in the tensor stored in that variable can change over time. For example, variables are used to store neural network weights, which change during the model optimization.

Also, note that for W and b, we provide some important arguments, such as the following:

init_w = tf.initializers.RandomUniform(minval = -0.1, maxval = 0.1)(shape = [10, 5])
init_b = tf.initializers.RandomUniform()(shape = [5])

These are called variable initializers and are the tensors that will be assigned to the W and b variables initially. A variable must have an initial value provided. Here, tf.initializers.RandomUniform means that we uniformly sample values between minval (-0.1) and maxval (0.1) to assign values to the tensors. There are many different initializers provided in TensorFlow. It’s also very important to define the shape of our initializer when we are defining the initializer itself. The shape property defines the size of each dimension of the output tensor. For example, if shape is [10, 5], this means that it will be a two-dimensional structure and will have 10 elements on axis 0 (rows) and 5 elements on axis 1 (columns):

 h = layer(x, W, b)

Finally, h is called a TensorFlow tensor in general. A TensorFlow tensor is an immutable structure. Once a value is assigned to a TensorFlow tensor, it cannot be changed.

As we can see, the term “tensor” is used in two ways:

• To refer to an n-dimensional array
• To refer to an immutable data structure in TensorFlow

For both, the underlying concept is the same because they hold an n-dimensional data structure, only differing in the context they are used. The term will be used interchangeably to refer to these structures in our discussion.

Finally, we can immediately see the value of h using the following:

print(f"h = {h.numpy()}")

This will give:

h = [[0.7027744 0.687556  0.635395  0.6193934 0.6113584]]

The numpy() function retrieves the NumPy array from the TensorFlow tensor object.

Final code

The full code is as below.

For future reference, let’s call our example the sigmoid example.

As we can already see, defining a TensorFlow computational graph and executing that is very “Pythonic.” This is because TensorFlow executes its operations eagerly or immediately after the layer() function is called. This is a special mode in TensorFlow known as eager execution mode. This was an optional mode for TensorFlow 1 but has been made the default in TensorFlow 2.