Generative AI with Python and TensorFlow 2/

...

/

Creating a DBN with the Keras Model API

Creating a DBN with the Keras Model API

Learn how to implement deep belief networks (DBN) using TensorFlow 2.0.

We'll cover the following...

The creation of a single-layer RBM to generate images is the building block required to create a full-fledged DBN. Usually, for a model in TensorFlow 2, we only need to extend tf.keras.Model and define an initialization (where the layers are defined) and a call function (for the forward pass). For our DBN model, we also need a few more custom functions to define its behavior.

Implementing the RBM

Let’s continue by combining multiple RBMs in layers to create a more powerful model—the deep belief network (DBN).

RBM initialization

First, in the initialization, we need to pass a list of dictionaries that contain the parameters for our RBM layers (number_hidden_units, number_visible_units, and learning_rate, cd_steps):

Python 3.10.4
class DBN(tf.keras.Model):
    def __init__(self, rbm_params=None, name='deep_belief_network', num_epochs=100,
                tolerance=1e-3, batch_size=32, shuffle_buffer=1024, **kwargs):
                
        super().__init__(name=name, **kwargs)
        self._rbm_params = rbm_params
        self._rbm_layers = list()
        self._dense_layers = list()
        
        for num, rbm_param in enumerate(rbm_params):
            self._rbm_layers.append(RBM(**rbm_param))
            self._rbm_layers[-1].build([rbm_param["number_visible_units"]])
            
            if num < len(rbm_params)-1:
                self._dense_layers.append(
                    tf.keras.layers.Dense(rbm_param["number_hidden_units"], 
                    activation=tf.nn.sigmoid))
            else:
                self._dense_layers.append(
                    tf.keras.layers.Dense(rbm_param["number_hidden_units"], 
                    activation=tf.nn.softmax))
            self._dense_layers[-1].build([rbm_param["number_visible_units"]])
        
        self._num_epochs = num_epochs
        self._tolerance = tolerance
        self._batch_size = batch_size
        self._shuffle_buffer = shuffle_buffer

At the same time, we also initialize a set of sigmoidal dense layers with a softmax at the end, which we can use for fine-tuning through backpropagation once we’ve trained the model using the generative procedures outlined earlier.

To train the DBN, we begin a new code block to start the generative learning process for the stack of RBMs:

Python 3.10.4
# pretraining:
inputs_layers = []
for num in range(len(self._rbm_layers)):
    if num == 0:
        inputs_layers.append(inputs)
        self._rbm_layers[num] = self.train_rbm(self._rbm_layers[num], inputs)
    else: 
        # Pass all data through the previous layer 
        inputs_layers.append(inputs_layers[num-1].map(
            self._rbm_layers[num-1].forward))
        self._rbm_layers[num] = self.train_rbm(
            self._rbm_layers[num], inputs_layers[num])

It initializes an empty list inputs_layers ...