An In-Depth Look at Audio Classification Using CNNs and Transformers

Introduction

Audio classification is a fascinating area of machine learning that involves categorizing audio signals into predefined classes. In this blog, we will delve into the specifics of an audio classification project, exploring the architectures, methodologies, and results obtained from experimenting with Convolutional Neural Networks (CNNs) and Transformers.

Dataset

The project utilized the ESC-50 dataset, a compilation of environmental audio clips categorized into 50 different classes. Specifically, the ESC-10 subset was used, narrowing the dataset to 10 categories for more focused experimentation.

Architecture 1: Convolutional Neural Networks (CNNs)

Initial Setup

The initial model setup for audio classification relied heavily on CNNs. These networks use convolutional layers to extract features from the audio signals progressively, increasing the output channel size from 16 to 64. Each convolutional layer is followed by a max-pooling layer to reduce the spatial dimensions and highlight the most critical features.

Original Model

The original model focused solely on feature extraction without incorporating dropout, early stopping, or other regularization techniques. This led to a basic yet effective structure for understanding audio data's complex patterns.

Enhanced Model

To combat overfitting and improve generalization, several enhancements were made:

• Dropout: Introduced to randomly deactivate neurons during training, thereby preventing over-reliance on specific paths.
• Early Stopping: Implemented to halt training when validation performance plateaued, ensuring the model does not overfit to the training data.
• Regularization: Additional techniques were employed to further stabilize the training process and enhance generalization.

Results

The use of k-fold cross-validation, with fold 1 reserved for validation, provided a comprehensive evaluation of the model's performance. Key observations from hyperparameter tuning include:

• Reduced Overfitting: The enhanced model exhibited lower test losses and higher test accuracies, F1 scores, and ROC AUC values across all folds compared to the original model.

The following table summarizes the performance across different folds:

Confusion Matrix and ROC Curve

The confusion matrix and ROC curve for the best performing fold (Fold 2) highlight the classifier's ability to distinguish between most classes effectively. However, there are instances of misclassification, suggesting the need for further refinement in the model.

Architecture 2: Transformers

Transformers, known for their success in natural language processing, were adapted for audio classification in this project. The core of this architecture involves:

• Convolutional Layers: Used initially to extract basic audio features such as tones and rhythms.
• Transformer Blocks: Employed to process these features using attention mechanisms, enabling the model to focus on different parts of the audio sequence dynamically.
• Multi-Head Attention: Utilized to attend to various representation subspaces simultaneously, enhancing the model's interpretive capabilities.
• Positional Encodings: Incorporated to retain the sequential order of audio data, allowing the model to adapt positional information effectively.

Performance Metrics

The transformer model was evaluated with different numbers of attention heads (1, 2, and 4). Key observations include:

• Two Heads Model: This configuration outperformed others in terms of test accuracy and F1 score, suggesting an optimal balance between feature learning and generalization.
• Four Heads Model: Despite higher train accuracy, this model exhibited signs of overfitting, with less effective feature integration for classification.

The table below outlines the performance metrics for different configurations:

Enhanced Model with Transformers

The enhanced model employed additional techniques such as gradient clipping and the AdamW optimizer, coupled with a learning rate scheduler. This configuration significantly improved the model's stability and generalization capabilities.

• Gradient Clipping: Applied to prevent exploding gradients, ensuring stable training.
• AdamW Optimizer: Recognized for its weight decay regularization, enhancing the model's performance on validation data.

The enhanced model demonstrated superior performance across several metrics:

Trainable Parameters

• SoundClassifier: Approximately 16.4 million trainable parameters.
• AudioClassifierWithTransformer: About 8.9 million trainable parameters.

Conclusion

This project illustrates the potential of both CNNs and Transformers in audio classification tasks. While CNNs provide a solid foundation for feature extraction, Transformers offer advanced capabilities through attention mechanisms, enhancing the model's ability to interpret complex audio signals. By incorporating regularization techniques and advanced optimizers, the enhanced models achieved significant improvements in generalization and stability, highlighting the importance of these strategies in machine learning.

The results underscore the effectiveness of using a combination of traditional convolutional methods and modern transformer architectures to tackle the challenges of audio classification, paving the way for further innovations in this exciting field.

---