Speech emotion recognition using ANN

Key takeaways:

• Speech emotion recognition (SER) utilizes AI to identify emotional states from voice characteristics, with applications in therapy, education, and customer service, leveraging techniques like Artificial Neural Networks (ANN).

• The process of emotion recognition involves several key steps: importing necessary libraries, collecting a labeled speech dataset (such as the RAVDESS dataset), preprocessing audio files to extract relevant features, and mapping extracted emotions to integer labels.

• Data visualization plays a crucial role, as the final step includes generating count plots to visually represent the distribution of different emotions in the dataset, enhancing understanding and analysis of the recognized emotions.

Speech emotion recognition (SER) is a field of artificial intelligence (AI) that focuses on identifying a speaker’s emotional state based on their voice characteristics. It can analyze emotional states in therapy sessions, identify struggling students in education, and gauge customer satisfaction in calls.

The process of employing Artificial Neural Networks (ANN) for speech emotion recognition entails teaching a neural network to identify different emotions based on characteristics taken from speech signals. This is a general overview of how we could go about completing this task.

Step-by-step guide

Here’s the step-by-step process of recognizing speech emotions in ANN:

Importing libraries

Let’s start this task by importing the required libraries or modules.

Libraries

We’ll use matplotlib, librosa, pandas, numpy, and keras library to recognize speech emotions in ANN.

Dataset collection

Gather a dataset of speech recordings labeled with the corresponding emotions. There are several publicly available datasets for speech emotion recognition. We’ll use ravdess dataset for this task.

Dataset

 The dataset that we will be using in this task can be downloaded from herehttps://www.kaggle.com/datasets/uwrfkaggler/ravdess-emotional-speech-audio.

• Line 1: Ravdess = "Audioss" specifies the directory path where the audio files are located.

• Lines 2–3: file_emotion = [ ], file_path = [ ] initialize lists to store emotions and file paths.

Preprocess files in the directory

Preprocess the speech recordings to extract features that are relevant for emotion recognition.

• Line 1: The script iterates through each item (actor_dir) in ravdess_directory_list.

• Line 2: For each actor_dir, list all files (actor_path) in the corresponding subdirectory using os.listdir(actor_path).

• Line 3: Construct the full file path.

• Line 5: Skip processing files that are not .wav files or hidden/system files.

Extract emotions from file names

Extract the emotions from file names by splitting the file names.

• Line 1: For each file, split the file name by periods (.) to remove the file extension and then split the resulting string by hyphens (-).

• Lines 2-5: Check if there are at least three parts in the split result. If so, extract the emotion label (emotion) from the third part (index 2) and append it to the file_emotion list.

• Lines 6-7: If the file name does not follow the expected format, print an error message.

Create DataFrame

Create a DataFrame from the emotions list and file path.

• Line 1: Rename the column to 'Emotions' using emotion_df = pd.DataFrame(file_emotion, columns=['Emotions'])

• Line 2: Rename the column 'Path' using path_df = pd.DataFrame(file_path, columns=['Path']).

• Line 3: After processing all files, create a DataFrame (Ravdess_df) by concatenating emotion_df and path_df along the columns axis using pd.concat([emotion_df, path_df], axis=1).

Map emotions

Now, map integer emotions to actual emotions.

• Lines 1–2: Map integer emotion labels to actual emotions using a predefined mapping dictionary (emotion_map) and Ravdess_df['Emotions'].map(emotion_map).

Display DataFrame

Finally, it prints the first few rows of the DataFrame.

• Line 1: Print the first 5 rows of Ravdess_df DataFrame.

Visualize and plot data

Generate a count plot using seaborn to visualize the distribution of different emotions present in the DataFrame Ravdess_df.

• Line 1: Set the title of the plot.

• Line 2: Count plot for visualizing the data in the Emotions column of a DataFrame Ravdess_df.

• Line 3: Set the label "Count" for the y-axis of the plot and adjust its font size to 12. 

• Line 4: Set the label "Emotions" for the x-axis of the plot and adjust its font size to 12. 

• Line 5: Customize the appearance of the plot’s lines on the axes. The arguments specify which lines on the axis to remove (top=True, right=True) and which to keep (left=False, bottom=False). In this case, it removes the top and right lines of the axis, potentially creating a cleaner visual presentation.

• Line 6: Show the plot.

Try it yourself

Click the “Run” button, then click the link provided below it to open the Jupyter Notebook.