GitHub Repo

💡 Please note: The chat messages shown are strictly for testing purposes and do not reflect my personal views or manner of speaking. As a French person, I value respectful dialogue. 😊

⚠️ Be careful that following this project may generate some costs when training and deploying a model on AWS SageMaker. See the Cleanup and Costs section at the end for more info.

Project Description

This project implements a content moderation system for a chat application. It leverages a BERT-based model for toxic comment classification (detecting categories such as toxic, obscene, insult, etc.) and uses OpenAI's gpt-4o for generating conversational responses. The model is trained and deployed on Amazon SageMaker, while the chat interface is served using a Flask web app deployed on PythonAnywhere. The purpose of this project was to gain valuable knowledge in Cloud solutions.

Environment Setup

Clone the repository, create a virtual environment and install dependencies as follows. There are multiple requirements files in the repo, use requirements-project.txt, which should cover everything you need.

https://github.com/alexdjulin/chat-moderator-sagemaker.git
cd chat-moderator-sagemaker
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements-project.txt

This project has been developed on python 3.12. However, SageMaker uses python 3.8, and pythonanywhere python 3.10, that's why some modules are listed under older versions for compatibility reason in the different requirements files.

Dataset Download and Review

I trained the model on the Jigsaw Toxic Comment Classification Challenge dataset, hosted on Kaggle. It consists of user-generated comments from Wikipedia discussions. Each comment is labeled for six types of toxicity:

• toxic

• severe_toxic

• obscene

• threat

• insult

• identity_hate

The dataset is structured as a CSV file where each row contains a comment and corresponding binary labels (1 for presence, 0 for absence of a toxicity type). It is meant to train models to classify whether a comment belongs to one or more of these categories. This dataset is widely used for natural language processing tasks, particularly for content moderation and text classification, helping to build models that detect harmful language in online discussions, which is exactly what we need.

Download the Dataset

You can download the dataset from Kaggle directly or use the download_dataset.py script, that will download and unzip it in a data folder. You need Kaggle credentials for this, see Documentation.

python download_dataset.py

On success, the script should print the shape, column names and comments distribution.

Dataset downloaded successfully. Shape: (159571, 8)

Column names: ['id', 'comment_text', 'toxic', 'severe_toxic', 'obscene', 'threat',
'insult', 'identity_hate']

Sample toxic comments distribution:
toxic            15294
severe_toxic      1595
obscene           8449
threat             478
insult            7877
identity_hate     1405
dtype: int64

Review the Dataset

In the notebooks folder, go through the dataset_exploration notebook. It explores the content of the dataset, do a bit of data cleaning and feature engineering and generates two CSV files that can be used for training and testing: processsed_train.csv and processed_val.csv.

As you can see, the dataset is highly imbalanced, but I did not spend too much time fixing it, as this project was focused on cloud training and deployment.

Create Model

The model folder contains 3 python scripts and 2 requirements files.

moderator.py

This module defines a BERT-based content moderation model for detecting toxic comments. It classifies text into six toxicity categories using a pre-trained BERT encoder and a classification head. The ContentModerator class provides a predict() method to return toxicity probabilities for given text. The forward() method processes tokenized input through BERT, extracts the CLS token representation, applies a linear classification layer, and outputs probability scores using a sigmoid activation function.

train.py

This script trains a BERT-based content moderation model on our dataset. It tokenizes text, prepares a PyTorch Dataset and DataLoader, and fine-tunes the ContentModerator model using Binary Cross-Entropy Loss (BCELoss). The model is trained on GPU (if available) and saves its weights to a .pth file. The train_model() function handles data loading, training, and logging, with TensorBoard support for tracking loss trends.

inference.py

This script handles inference requests for the ContentModerator model in AWS SageMaker. It defines functions for loading the trained model (model_fn), parsing incoming requests (input_fn), running inference (predict_fn), and formatting responses (output_fn). The model is automatically loaded from /opt/ml/model/, and predictions are returned as JSON-formatted toxicity probabilities.

requirements-train.txt, requirements-infer.txt

They are packed into the model.tar.gz archive and sent to the S3 bucket respectively during the training and endpoint deployment steps. They list the dependencies specifically required for these steps. They are renamed automatically when used.

Training on SageMaker

💡 While it would be easy to train a model locally or on Colab, the purpose of this project was to learn how to train and deploy a model using AWS SageMaker.

AWS Setup

To train and deploy a job on SageMaker, you will need to do the following:

• Create an AWS account (you can use the free tier to start)

• Create an S3 bucket in your region. Mine is called chat-moderator and is stored in eu-central-1.

• In your S3 bucket, create 3 folders: data, models and output. Upload processed_train.csv and processed_val.csv into the data folder.

• In IAM, create a User with AdministratorAccess. You will need its Access Key and password to run inference on the model later (from PythonAnywhre)

• In IAM, create a new Role with AmazonS3FullAccess, AmazonSageMakerFullAccess and CloudWatchFullAccess permissions. Copy it's ARN, you will need it for training and inference.

• In Service Quotas, SageMaker AWS service, make sure you have at least one ml.g5.xlarge GPU for training and endpoint usage for your region. Request one if not.

• Finally install the AWS CLI and call aws configure to enter your credentials.

For more information, feel free to check Andreas Trolle's in-depth tutorial on how to Train and Deploy a Multimodal AI Model.

Create Training Job

Edit start_sagemaker_training.py with your bucket name and role ARN, then run it. It will go through the following steps and send a training job to SageMaker using the PyTorch estimator:

• Package train.py, moderator.py and requirements-train.txt into an archive

• Upload it to a new folder in your S3 bucket

• Create the Training Job in SageMaker

• Train and stop automatically once completed or if it runs into an error

I trained my model for 3 epochs, which took about 2.5 hours.

python start_sagemaker_training.py

The job should be visible in the SageMaker Training Jobs section. Click on it to access its logs in CloudWatch and make sure it is training as expected.

⚠️ I was unable to start a training job simply by using the model.fit() method as shown in the documentation. Therefore I had to implement the steps manually using the boto3 AWS SDK module.

Download trained model

Once the training is done, you can access your model in your S3 bucket, in the output folder. Download and uncompress the model.tar.gz file. It should contain a model.pth model (the trained weights) and a tensorboard folder. Put them both in a model_output folder inside your local model folder.

If you don't want to go through the training steps, you can download my model and tensorboard logs.

You can preview the tensorboard metrics at this address using the following command pointing to your tensorboard folder.

tensorboard --logdir=./model/model_output/tensorboard

We can see the loss curve improving slightly over the 3 epochs, although the improvement is very minimal.

Creating Flask App

Now it's time to create our application.

Frontend

For the frontend, I asked Claude 3.5 to create a simple chat interface for me. It generated the following 3 files:

• index.html in the templates folder

• css/styles.css and js/main.js in the static folder.

I asked for some modification to get a reactive chat, that would print all messages and then moderate them based on the model's predictions.

Backend

For the backend, I used Flask to run inference on the model. I hooked it up with a gpt-4o client to generate AI responses and offer a chat experience for testing purposes. I created an app_local.py file to run inference on my local model or on a sagemaker endpoint based on input arguments.

The flask endpoint goes through the following steps:

• Configure an OpenAI client using gpt-4o with a conversation context, so the AI assistant keeps track of the discussion

• Load the local model stored in model_output and set it in eval model

• Alternatively run inference on a SageMaker endpoint, to test the deployment done in the next steps

• Run predictions on the model and classify the user's messages

• Send the results to the frontend to display the chat messages with or without moderation, depending on the model's predictions

You can run the Flask chat app like this and access it at http://127.0.0.1:5000.

python app_local.py -m local  # to infer on local model (default)
python app_local.py -m sagemaker  # to infer on sagemaker endpoint

As you can see, the second user message has been flagged as insulting and toxic and therefore moderated. For debugging purposes I am just crossing it out and adding any prediction results over the given threshold (0.5) in red below it.

We can use this local app to run tests on our model and re-train it if necessary to achieve better results.

Deploying Model on SageMaker

Once we have a model we are happy about, we can deploy it as a SageMaker endpoint so we can infer on it from a local or deployed app.

python deploy_sagemaker_endpoint.py

Running the deployment script will go through the following steps:

• Wrap up the model into a model.tar.gz archive, along the required scripts to run inference on it inside a code folder:

model.tar.gz  
├── model.pth  
└── code  
    ├── inference.py  
    ├── moderator.py  
    └── requirements.txt

• Upload the archive to the models folder of my S3 bucket

• Delete any existing SageMaker model or endpoint configuration

• Create a SageMaker model

• Create a SageMaker endpoint configuration

• Deploy the endpoint

⚠️ Same issue as for the training step, I was unable to use the model.deploy() method as indicated in the Documentation to automate this step and I had to go through these steps manually using the boto3 module.

The endpoint should appear in SageMaker Inference Endpoints and be InService after a short time.

You can use the following script to test the endpoint and infer on the model. You should get the prediction results corresponding to your input text. I set the threshold to 0.5 so it is pretty strict, for testing purposes. If any category has a prediction above this value, the message will be moderated.

python infer_on_sagemaker_endpoint.py

Alternatively you can test it in the local Flask app too by passing -m sagemaker as argument. You should get the same results as when infering on the local model.

python app_local.py -m sagemaker

Deploying Flask App on PythonAnywhere

This is our final step. Our model is now available in the AWS cloud, but our Flask app is still running locally. We need a simple online service where we can upload and run our app.

For simplicity and to avoid additional costs, I used pythonanywhere which offers a free tier with 512 MiB quota. It's not much but enough to deploy our frontend and backend scripts, as well as create a small environment to infer on the sagemaker endpoint.

Here are the steps I went through for this last step:

• Create a pythonanywhere free account

• Create a new Web App using the manual setup (not Flask)

• In the Files tab, upload the app_deployed.py and requirements-app.txt files, as well as the static and templates folders.

• Uploaded an .env file too, where you specified your OPENAI_API_KEY, as well as the 3 AWS environment variables needed to run inference. See .env-template.

  

• In the Consoles tab, open a bash console, create a virtual environment and install the dependencies from requirements-app.txt. See Environment Setup section on how to do this.

• In the Web tab, edit the WSGI configuration file as follows to point to your Flask app:

import sys

# Add your project folder to the Python path
project_home = '/home/alexdjulin/chat_moderator'
if project_home not in sys.path:
    sys.path.append(project_home)

# Import Flask app
from app_deployed import app as application

• In Virtualenv, specify the path to the virtual environment you just created

• Force HTTPS and add a password to access the app if needed

Reload the app, which should now be accessible at your account address, like mine:
https://alexdjulin.pythonanywhere.com/ (password protected).

You should see the chat interface and be able to interact with the AI Assistant. If infering on the SageMaker endpoint does not work, an [uncensored] tag will be added next to the user's messages.

If you encounter any issue, check the Error log in the Web tab.

Cleanup and costs

Be careful of the resulting costs when using cloud services!

SageMaker training fees apply during the training step only, so you won't be charged once the job has completed or failed. Overall my latest model trained for 2.5 hours (3 epochs) and cost me $5.

Make sure you delete your SageMaker endpoint when you don't need it anymore, as deployment costs about $1.5 per hour of uptime.

S3 bucket storage is very cheap, but feel free to delete your bucket once you are done working on the project and you don’t need it anymore. It’s easy to create a new one when required.

IAM users and roles as well as GPU quota requests are free.

Hosting your app on pythonanywhere is free when using a Beginner account. Your are limited to 512 MiB storage and your app is running for 3 months. You will need to reload it after this time to keep it available.

Conclusion

This project demonstrates how to build a scalable and efficient chat moderation system by integrating a BERT-based model trained on a public dataset for toxicity detection.

By leveraging Amazon SageMaker, the computationally intensive training and inference tasks are offloaded to AWS GPUs, ensuring optimized performance. Meanwhile, deploying the Flask app on PythonAnywhere provides a lightweight and accessible web interface, making the system easy to interact with.

This architecture effectively scales while maintaining a clear separation between content moderation and conversational AI.

A more streamlined approach could involve fully relying on an LLM to both generate responses and moderate user input simultaneously. This would likely improve classification accuracy and require minimal or even zero-shot learning. However, the current approach offers a local model inference option, which can be advantageous for privacy-focused applications where external API calls might not be desirable.

• 🌟 Model Overview: Release context and promises

• 🧠 Architecture: Key design choices for performance

• 📊 Benchmarks: Evaluation results, comparisons with peers

• 🔧 Fine-Tuning: Generalization across tasks and datasets

• 🚦 Guardrails: Strategies for ensuring responsible generation

• 💡 Key Use-Cases: Implementation scenarios

• 🏁 Conclusion: Key insights and discussion points

What makes Mistral 7B so efficient, for such a small model? Buckle up and let's take a deep dive into its mechanics...

Arxiv papers:

Mistral 7B | Attention Is All You Need | Longformer | GQA

This is an in-depth presentation of my final project at Ironhack AI Engineering Bootcamp, November 2024.

Meet Alice, your personal AI Librarian! Alice isn’t just any chatbot — she’s a next-level conversational assistant, leveraging the capabilities of an LLM and a flexible RAG, designed to:

• 📚 Answer all your questions about books and literature.

• 🔍 Search a vector database for precise, relevant information.

• 🌐 Fetch detailed book insights straight from Wikipedia.

• 🎥 Pull transcripts of book reviews from YouTube.

This architecture allows it to provide accurate and up-to-date information on various topics. The integration of a personality element, language models, and tools enables it to act as a virtual librarian with a rich user experience.

Alice was voted best bootcamp project and I presented her at the 2024 HackShow. Let me know what you think of the presentation and share insightful use-cases where you think Alice would shine! 🤖

A huge thank you to Ironhack for giving me the opportunity to showcase my work, and once again to our amazing teachers Carlos Fenollosa and Isabella Bicalho Frazeto for their guidance and support throughout this journey.

Github repo

This is a demo of my movie-advisor project: I created a chatbot driven by a langchain agent and RAG tools.

The agent is handling all conversations between me and the LLM. Based on my queries, it will search for preferences and update my watch lists stored on the postgreSQL database Xata, or fetch missing information from the movie database TMDB.

I am using my other project ai_chatbot as submodule to initialise the agent and handle voice conversations using Google STT and Edge TTS engines.

Tools: Langchain | Xata | TMDB

Don't miss my other AI-related articles on my dev blog!

Music by Denys Kyshchuk from Pixabay.

Project Presentation

GitHub repo

The goal of this project was to implement a zero- or few-shot intent classifier that can be used to provide inferencing services via an HTTP interface. For this, I used the ATIS Dataset to fine-tuned the latest version of OpenAI GPT3, in order to improve intent predictions.

Project Research

See the project_research.ipynb notebook on Github for my review of the ATIS dataset, as well as in-depth research on testing and fine-tuning GPT3 to perform intent classification.

The required modules to run the notebook code are listed at the beginning. Create a virtual environment if necessary and run the Modules Installation cell.

This notebook is available on Kaggle too.

Run the Project

See the Readme on Github to clone the project, install the required modules and run the Flask server. A Docker file to run the project in a container is available too.

Feel free to play around with the instructions in the prompt and intents files to tweak the results to better match your use case.

You will need to fine-tune your own GPT3 model on the ATIS dataset, to get better results, as I cannot share my version. See the project research notebook for that.

Conclusion

While intent classification has significantly advanced, challenges remain. Ambiguity in user messages, evolving language use, and domain-specific jargon can complicate accurate intent detection.

The future of intent classification in AI chatbots lies in continuous learning and adaptation. As models become more sophisticated, they will better understand and predict user intentions, making chatbots even more integral to digital communication.

In conclusion, intent classification is a cornerstone of modern AI chatbots, enabling them to understand and respond to user needs effectively. As technology advances, its role will only become more significant, driving innovation and enhancing the way we interact with machines.

Github Project

I created and presented this project to complete my 120-hour Deep Learning bootcamp in April 2024. I then spent some more time improving it until I get convincing results.

Project presentation

Unlike other countries, Germany does not have a strict law regarding bicycle helmets. To this day, it is not compulsory to wear one, but only a recommended safety measure. Unfortunately, many people don't take this measure seriously, which can lead to severe injuries if an accident arises. Especially in a big city like Berlin, cyclist casualties hit the news on a daily bases.

My goal with this project was to raise cyclists awareness, using a friendly and direct feedback method. You may have seen those interactive street signs for cars: If you drive under the speed limit, you get a nice green smiley. But if you drive above it, you get an angry red one instead. Simple but effective. I surprised myself slowing down unconsciously, just for the pleasure to see a smiling face. Who doesn't need one on a Monday morning?

The goal of this project was therefore to train a Deep Learning model to detect on a video source if a cyclist is wearing a helmet or not. Displayed on a traffic light for instance, cyclists would get a direct feedback, in form of a green happy or red angry smiley.

Apart from raising awareness, the model could also be used to count cyclists and collect statistical data.

You Only Look Once

The main reason why I chose this project was to finally get hold of a YOLO model and play around with it.

If you don't know about YOLO, check this link which goes in-depts about the internal structure and use-cases of the 8th version.

To sup-up, YOLO is an incredible Deep Learning model used to classify, detect, segment or track multiple elements of an image or video in real-time. Widely used everywhere in modern use-cases like healthcare, surveillance, self-driving cars or face detection, you can't set a foot in Computer Vision without hearing about it.

I started browsing through the different versions that Ultralytics offers and opted for YOLOv8n, which is the most lightweight version, hoping it would perform well on my old computer. Lightweight but still quite powerful, if you look at the numbers: 225 layers, 3.2M params for an input shape of 640 pixels.

You can chose between different versions, pre-trained on famous datasets:

• Coco (80 classes)

• Open Image v7 (600 classes)

• ImageNet (1000 classes)

Test a pre-trained model

You will find all my tests and method implementatoin in the BikeHelmetDetection_Testing notebook.

I started testing YOLOv8n pre-trained on Open Image v7, as it can already detect a person (man, woman), a bicycle and a bicycle helmet.

I loaded the model and ran a prediction on a test image. Looking good!

When I tried it on a video file though, the results were not as good as I expected. While the model can classify a person or bicycle prettey accurately, it often fails at detecting the helmet, especially if it's not fully visible or only covers a small part of the picture.

On the other side, I tested the model version pre-trained on the COCO dataset and it seems to do a better job detecting a person and a bicycle. The issue is that there is no helmet class in the COCO dataset...

What if we fine-tune it and teach it how to recognize a helmet? Sounds like fun!

Fine-Tune a YOLO model

My training steps are available in a Kaggle and Colab notebooks.

Training YOLOv8n is quite straightforward. The API is cristal clear and you can do that with 3 lines of code. But for those who like to get their hands dirty (like me), you can still access and play around with countless hyperparameters and callbacks.

But regardless how complex your model is, you still need a valuable dataset to train it if you want to get the best out of it.

For this project I used the Helmet Detection dataset on Kaggle. It's not exactly what I need, as it's classifying all kind of driving helmets, also motorbike ones. But let's still give it a try.

• License: CC0: Public Domain

• Content: 764 images and annotations

• 2 classes (with/without helmet)

• Images: PNGs, various sizes

• Annotations: XMLs, PASCAL VOC format

Convert input labels

The dataset provides PNG images and XML labels using the Pascal VOC format.

Example:

<annotation>
    <folder>images</folder>
    <filename>BikesHelmets1.png</filename>
    <size>
        <width>400</width>
        <height>300</height>
        <depth>3</depth>
    </size>
    <segmented>0</segmented>
    <object>
        <name>With Helmet</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <bndbox>
            <xmin>161</xmin>
            <ymin>0</ymin>
            <xmax>252</xmax>
            <ymax>82</ymax>
        </bndbox>
    </object>
</annotation>

That's nice and easy to read, but Yolo requires a different label format as input. For each image, a txt file should list the classes and the bounding boxes top left and bottom right points, normalized btw 0 and 1.

{class} {bbox x1} {bbox y1} {bbox x2} {bbox y2}

# Example:
1 0.5162 0.1366 0.2275 0.2733

So I first generated these text labels from the xml files.

Train / Validation Split

I used the common 80/20 % ratio when dividing my PNG images and text labels into train and validation sets.

Generate the training config file

To initiate training, yolo requires a yaml file containing the paths to the train and validation sets, as well as a list of output classes. I generated it before training.

path: dataset_dir  # root dir
train: train/images  # train dir
val: val/images  # val dir

# Classes
names:
  0: without helmet
  1: with helmet

Train the model

You need to chose a pre-trained model to start from, as you will get better results than training it from scratch.

I picked the version pre-trained on the COCO dataset.

As said, you can play around with hyperparameters and callbacks. I experimented a bit and eventually used the following settings:

• Start from pre-trained Coco model

• No augmentation

• 150 epochs

• Automatic Batch Size (16)

• EarlyStopping with a patience of 50

• Automatic Optimizer (AdamW)

• Automatic Learning Rate (0.0017)

• Dropout to reduce overfitting: 0.25

I trained my model on Google Colab using a Tesla V100. It took 15 min for 150 epochs and cost me about 90 cts.

Evaluate the model

Yolo outputs loads of graphs and metrics during the training, to help you visualize the results and assess your model's performance.

Let's have a look at the curves first:

The train/box loss and trail/cls loss shows us the model's accuracy when generating the bounding boxes (left) and classifying features as with or without helmet (right). Both decrease as expected over time. We can see that our model learns the classification part pretty quickly (drop over the first 20 epochs), but it takes longer to learn how to generate accurate bounding boxes (linear drop).

The validation loss curves (testing the model on images it has never seen during training) give us similar drops for the classification. However, the bounding box loss seems to stagnate around 1.4 and does not decrease much, I'm not sure why.

As for the other metrics like precision and recall, the curves go up and stabilize quickly.

The two bottom curves show the mean average precision at different IoU thresholds. The Intersection over Union (IoU) threshold acts as a gatekeeper, classifying predicted bounding boxes as true positives if they pass the threshold and false positives if they fall below it. By adjusting the threshold, we can control the trade-off between precision and recall.

On the left curve, the threshold is set at 50%. Higher is generally better, leading to more accurate detections. On the right curve, the mAP ranges between 50 and 95% and gives a comprehensive view of the model's performance across different levels of detection difficulty.

The Confusion Matrix is also a powerful graph to assess how accurate our model is at classifying heads with or without helmets, and quantifies false positives and false negatives. Our model seems to be doing a good job, predicting with accuracy if a person is wearing one or not (top left and middle squares). We have just a few false positives. It detects sometimes heads on empty backgrounds (right columnn), although this should not be too bothering for our use-case.

Note that our dataset seems to be uneven, with more features of people wearing a helmet than without (193 images Vs. 92 images here). This is confirmed on the labels graphs below. It is usually recommended to train a model on a dataset with an even distribution of images per classes.

Test the model

Now it's time to see what our model can do. Yolo will save the last and best version of it, which is not necessarely the same one. In case of EarlyStopping, you will get the best version over the amount of epochs defined as patience.

You will find my best version in the models folder.

Let's load it and predict on a few test pictures. I gathered all my tests in the BikeHelmetDetection_Testing notebook. A cell is running and comparing 3 predictions on the same image (COCO, OpenImageV7 and my fine-tuned model). Here are examples of how my model performs:

Our model is doing a good job on images. But what about predicting on a video source, which is what we need in the end? Let's import OpenCV and give it a try on the webcam. Grab your helmet!

All good. The confidence is not as high as I would have expected, but let's move on and see if this is enough for our use-case.

Detecting a cyclist wearing a helmet

We are now hitting the main challenge of this project:
How to detect a cyclist wearing a helmet?

As discussed in the YOLO section above, Pre-trained versions of YOLOv8 can recognize up to 1000 classes out of the box (ImageNet). But you will not find a 'Cyclist with helmet' class, rather separate classes like 'person', 'bike' or 'helmet'. We should therefore come up with an algorithm that not only will detect these classes but will be able to tell if this person is actually sitting on a bike and if this helmet is actually on the cyclist's head. And that's not as easy as it sound!

Are you making fun of me !?

In the process of fine-tuning YOLOv8n to classify heads with or without helmets, we altered the output layer of our model and we lost all its pre-trained proficiency. We picked the version pre-trained on COCO, which originally could classify persons and bikes. But now it can only classify heads with or without helmets, which is not enough for our use case. We don't want a random person walking in the street to get a red frowning smiley for not wearing a helmet!

Therefore, the most flexible solution would be to combine predictions from different models on the same input image. For instance:

• YOLOv8n pre-trained on COCO --> Track Person and Bike

• My fine-tuned version --> Track Helmet

  

We would end up with 2 or 3 bounding boxes and we can use simple maths to detect if a bounding box overlaps with another.

• Bike box inside a Person box --> This is a cyclist

• Helmet box inside a Person box --> This person wears a helmet

We can now combine both deductions and conclude if a cyclist is wearing a helmet or not.

Of course, this method will only work if a cyclist is clearly visible in the frame and detached from other background elements. If three of them are waiting closse to each others at a traffic light, bounding boxes may interfere and falsify the results. But our use-case aims at tracking and sending feedback to cyclists one at a time, so we should be fine.

Building our final solution

We now have all the cards we need in our hand. In the BikeHelmetDetection_Testing notebook, I defined some helper methods that helped me parse and process the prediction results:

• parse_result: extracts all info we need from a YOLO prediction results, like classes, confidence values and bounding box coordinates.

• print_predictions: prints the parsed results.

• draw_result_on_frame: draws the predicted bounding box and class label on an input frame.

• get_bbox_center: returns the center of a bounding box.

• bbox_inside_bbox: checks if a bounding box overlaps with another one. For this, I simply check if the center of the child box is inside the parent box.

• combine_bboxes: combines predicted bounding boxes (person, bike, helmet) into one (cyclist with or without helmet).

• track_cyclist_with_helmet: this is our main method processing all the predictions, splitting them in classes (person, bike, helmet) and checking if what we see is a person on a bike and if this person is wearing a helmet or not.

Thanks to these helpers, we can now build our OpenCV loop that will run through the video input, process each frame and draw the result on it before displaying it.

As said, we need a way to assign our predicted classes to different models. I'm doing this with a simple assignment dictionnary, where I list each model and the classes I expect it to predict. This will undoubtedly impact our solution's performance, as we run predictions on multiple models at once, for each single frame. To help with that, I am running predictions on different threads and I combine them at the end, before sending them to track_cyclist_with_helmet, to process and draw the result.

Debug flag on: The bounding boxes of the different classes are displayed in white, the combined box in red or green.

Et voilà! Our cyclists now get a green or red bounding box around them and we can display the resulting smiley as feedback.

To finish, I moved everything to project_demo.py and added some more settings and features, like confidence thresholds to detect the different classes, an FPS counter and debugging elements to visualise predictions.

Conclusion and Improvements

This was intended as a short-term project. I spent two days on it to complete my module and another 4 days afterwards to come up with a clean solution. While I achieved my main objective, performance remains the main issue of this project. The tracking examples displayed on this page are played in real-time, but my prediction model is running at about 7 FPS on my local computer. This is not fast enough to detect cyclists accurately and give them a real-time feedback.

To move this project towards a final solution, we should first research what device could be used to detect cyclists in the street (Raspberry Pi?), what performance it offers and how we could optimize our prediction model to run much faster.

This project has been very instructive and was a perfect conclusion to my 4-week Deep Learning bootcamp (I got the best grade!). I hope I will get the chance to work on it further in the near future and eventually see it put to good use in the streets, hopefully raising cyclists awareness and reducing casualties.

BE SAFE OUT THERE, WEAR A HELMET !

Project Demo

Project Repo

This project is available on my github:

RockPaperScissorsCNN

Datasets

I used the following 3 datasets to train the CNN from scratch:

DRGFREEMAN - Edited to remove the green screen, see remove_greenscreen notebook
SANI KAMAL - Also available in tensorflow_datasets, see download_dataset notebook
ALEXDJULIN - I created this one myself

Model

The latest version of my model is available here:
rps_v01_56ep_0.9641acc_0.1089loss.h5

Additional Ressources

This model is inspired by the following tutorials and repos:
Python for Computer Vision with OpenCV and Deep Learning, by Jose Portilla
rps-cv repo from drgfreeman
Rock-Paper-Scissors Image Classification Using CNN from Farah Amalia

GitHub repository

This is the output of my AI avatar chat project, using Google speech recognition, OpenAI chat completion and Elevenlabs voice cloning. The lipsync is done using NVIDIA audio2face and the avatar is a metahuman rendered in the Unreal Engine.

The code repo is covering the AI-chat interface only (yellow blocks), not the lipsync generation (in green) or the avatar creation and rendering (in red), which are done in 3rd-party software.

Project description

The goal of this simple project was to learn how to interact with the OpenAI API and get useful information from ChatGPT in a format I can easily manipulate (JSON dictionaries of artists and songs). On the other side, it uses the Spotipy module to interact with Spotify and create playlists. Linked together, these two modules can help you create creative playlists from songs suggested by ChatGPT according to a given prompt (Example: "Peaceful songs to listen to when it's raining"). It's a great way to discover new tunes!

This project is built upon a tutorial by Colt Steele: Mastering OpenAI Python APIs that I highly recommend, as it covers the basics of how ChatGPT works and how to interact with it. On top of the automatic playlist creation mode covered by the tutorial, I created an interactive mode, which lets you decide which songs should be added and gives you the possibility to blacklist artists and songs. I completed the project with additional methods to play and print playlists.

Prerequisites

To use this tool, you need both an OpenAI and a Spotify Developer account.

OpenAI

It's the company that created and offers access to different ChatGPT models. Go to the OpenAI website and create an account. It comes with some free credit to start and test the API. See documentation to generate your own API-Key.

Spotify

Create a Spotify account if you don't already have one, it's free. After that, log in to Spotify for Developers and create a project. In the project settings, you can generate the client-ID and -Secret needed to connect and send requests to Spotify. See the documentation for more info.

Once you have your credentials to access both APIs, you can add them to playlist_generator.py. I am using the keyring module to store and retrieve mine, but you can use an .env file, store them in environment variables or just use them directly in the code. In this last case, be careful not to share your code with anyone!

# Set openai key
openai.api_key = keyring.get_password('openai_key', 'main_key')
# Set spotify client id and secret
client_id=keyring.get_password('spotify', 'client_ID')
client_secret=keyring.get_password('spotify', 'client_secret')

Spotify Desktop

If you want to be able to play songs during the playlist creation (in the interactive mode for instance), you will need Spotify Desktop installed on your machine. If you want to skip this part, comment all calls to the method play_song_in_spotify in playlist_generator.py, as it will not be able to ask your Spotify player to play the given songs.

Installation

Clone project and install required libraries (in a virtual environment for instance)

git clone https://github.com/alexdjulin/spotify-playlist-generator.git
python -m venv venv
venv\Scripts\activate # on windows
source venv/bin/activate # on macOS/linux
pip install -r requirements.txt

How to use it

The most convenient way to start the playlist generator is to call main.py.

python main.py

It will ask for the 4 required input variables:

• Playlist Prompt (-p):
  A short description sent to ChatGPT to generate the playlist. Example: Best songs for a road trip with your best friend.

• Playlist Length (-l):
  How many songs should be in the playlist. Default value is 10.

• Playlist Name (-n):
  The playlist name on Spotify. Optional, leave blank to use the prompt as a name. Example: Road Trip Songs.

• Interactive Mode (-i):
  Y to activate or N to deactivate the interactive mode. When active, each song will be played in Spotify and the user can choose to Add the song, blacklist it or blacklist the artist.

Playlist Prompt: Best songs for a road trip with your best friend
Playlist/Batch Length: 10
Playlist Name (leave blank to use prompt): Road Trip Songs
Interactive Mode? [y]es or [n]o: n
Creating playlist. Please wait...
>> end of playlist creation

Additionally, you can call playlist_generator.py and pass these arguments in the command line directly. Don't forget the quotes around the strings. The interactive mode flag is TRUE if you pass -i, FALSE if you don't pass it:

# interactive flag off
python playlist_generator.py -p "Best songs for a road trip with your best friend" -l 10 -n "Road Trip Songs"
# interactive flag on
python playlist_generator.py -p "Best songs for a road trip with your best friend" -l 10 -n "Road Trip Songs" -i

Automatic Playlist Creation

When the interactive mode is off, the playlist is created and filled automatically. It is great to create quick playlists, without worrying too much about the contents... Providing that you trust ChatGPT's suggestions, which can be sometimes surprising!

Output when the interactive mode is off:

----------------------------------------------------------------------------------------------------
------------------------------------ SPOTIFY PLAYLIST GENERATOR ------------------------------------
----------------------------------------------------------------------------------------------------
Prompt: Best songs for a road trip with your best friend
Length: 10
Name: Road Trip Songs
Interactive: False
----------------------------------------------------------------------------------------------------
1. Journey - Don't Stop Believin' | Escape (Bonus Track Version)
2. Smash Mouth - All Star | Astro Lounge
3. Bruce Springsteen - Born to Run | Born To Run
4. Guns N' Roses - Sweet Child O' Mine | Appetite For Destruction
5. John Denver - Take Me Home, Country Roads - Original Version | Poems, Prayers and Promises
6. ABBA - Dancing Queen | Arrival
7. Bon Jovi - Livin' On A Prayer | Slippery When Wet
8. Queen - Bohemian Rhapsody | Bohemian Rhapsody (The Original Soundtrack)
9. Lynyrd Skynyrd - Sweet Home Alabama | Second Helping (Expanded Edition)
10. Whitney Houston - I Wanna Dance with Somebody (Who Loves Me) | Whitney
----------------------------------------------------------------------------------------------------

Spotify should open automatically, the songs are added to the playlist and the first one from the list should play when done.

Interactive Playlist Creation

If the interactive flag is on, we get a first batch of suggestions from ChatGPT (using the input playlist length as batch length) and we play the songs one by one in Spotify. For each song, we have the choice to:

• Add the current song to the Playlist

• Blacklist this song (don't suggest it anymore)

• Blacklist this artist (don't suggest any songs from this artist anymore)

• Quit generating the playlist and keep it as it is

Gary Jules - Mad World   (1/5)
[1] Add to Playlist
[2] Not this song
[3] Not this artist
[q] Quit playlist generation
Your choice:

When the choice is done, we jump to the other song. When the batch is done, we get the following choice:

• Ask ChatGPT for another batch of songs

• Quit generating the playlist and keep it as it is

Do you want another batch of songs?
[1] Yes, give me more songs!
[2] No, I'm down with this playlist
Your choice:

When asking ChatGPT for a new batch, we provide it with the list of songs we already have, as well as the lists of blacklisted songs and artists, to take into account for the next suggestions.

Ouput when the interactive playlist is done:

----------------------------------------------------------------------------------------------------
------------------------------------ SPOTIFY PLAYLIST GENERATOR ------------------------------------
----------------------------------------------------------------------------------------------------
Prompt: Lonely songs to play when it's raining
Length: 10
Name: Rainy Mood
Interactive: True
Artists Blacklist: {'The Carpenters', 'Willie Nelson'}
Songs Blacklist: {'Adore', "I Can't Make You Love Me"}
----------------------------------------------------------------------------------------------------
1. System Of A Down - Lonely Day | Hypnotize
2. a-ha - Crying in the Rain | East of the Sun, West of the Moon
3. Brook Benton - Rainy Night in Georgia | Brook Benton Today
4. Green Day - Boulevard of Broken Dreams | Boulevard of Broken Dreams
5. The Doors - Riders on the Storm | L.A. Woman
6. R.E.M. - Everybody Hurts | Automatic For The People
----------------------------------------------------------------------------------------------------

See GitHub readme for additional notes and contributions.

Conclusion

This has been a fun little project and a good first experience playing around with the OpenAI API. The Spotipy library on the other side is very intuitive and you can quickly get around with it and interact with your Spotify account. Obviously, some prompts like "80's hits" will return you more accurate results than creative ones like "Sad songs to listen to when you just broke up and your fridge is empty". But I think switching the model to GPT4 and future models will probably improve on this.

Click on the image to display the interactive one.

Link to the Github project

I'm a huge fan of maps, probably since I played around with my dad's roadmaps as a kid (he hated it, as he could never fold them back properly!). I cannot spend a day without fiddling with Google Maps. Although my trail running hobby and my GPS-watch addiction play a big part in that!

I have a running blog where I share reviews of events I took part in 🏃 But over the years, I kind of lost track of them. How many were there? Where did I run? How was the route? I need some kind of map to visualize them. What if I could make it so simple, that I would only need to update a Google spreadsheet with race information and the map on my blog would update accordingly? Let's see what we can do...

FOLIUM. That's our code word for today. Folium is a Python module that is going to provide exactly what we need: Manipulate data in Python, then visualize it in a Leaflet map.

Let's start with a simple example: Creating a map from one race event logged in this spreadsheet.

See tutorial.py on my git repo.

Download the spreadsheet

Let's first download the Google spreadsheet as a CSV file, it will be easier to work with the data. I shared the document so anyone with the link can access it. The following command will download the spreadsheet.

import os
sheet_url = 'https://docs.google.com/spreadsheets/d/1WghWJbxdCeKpbi3-H_6FmBAv_jILD_1_woRhuKGJ190/export?exportFormat=csv'
current_folder = os.path.dirname(os.path.abspath(__file__))
csv_filepath = os.path.join(current_folder, 'run_events.csv')
command = f'curl -L {sheet_url} -o {csv_filepath}'
os.system(command)

You should now have a run_events.csv file next to your script.

Load CSV data

Who doesn't love Pandas 🐼? Load the contents of the CSV file into a Python dictionary.

import pandas as pd
data = pd.read_csv(csv_filepath).to_dict(orient='records')[0]
print(data)

# Output:
# {'Date': '28.09.2014', 'Race': '41. Berlin Marathon', 'Latitude': 52.51625499, 'Longitude': 13.37757535, 'Time': '4:17:35', 'Link': 'https://www.bmw-berlin-marathon.com/'}

Create the HTML map

Now we can start having fun! You will find Folium tutorials everywhere and the documentation itself is pretty straightforward. Let's generate a simple map and populate it with our data.

import folium

# create map object and center it on our event
import folium
run_map = folium.Map(location=[data['Latitude'], data['Longitude']], tiles=None, zoom_start=12)

# add Openstreetmap layer
folium.TileLayer('openstreetmap', name='OpenStreet Map').add_to(run_map)

# save and open map
run_map.save('run_map.html')
import webbrowser
webbrowser.open('run_map.html')

Run the code, the map should be generated, saved and opened in your default web browser. As you can see, it's empty and just centered on the race location (Berlin). Click images to see HTML maps.

Populate the map

Let's add a marker point for our race to the map. We want it to be part of a 'Marathons' feature group, display the race name as a tooltip, assign a color to it and display a short legend in the corner.

# add feature group for Marathons
fg_marathons = folium.FeatureGroup(name='Marathons').add_to(run_map)

# create marker and add it to marathon feature group
folium_marker = folium.Marker(location=[data['Latitude'], data['Longitude']], tooltip=data['Race'], icon=folium.Icon(color='red'))
folium_marker.add_to(fg_marathons)

# add legend in top right corner
run_map.add_child(folium.LayerControl(position='topright', collapsed=False, autoZIndex=True))

Our marker now shows up at the given coordinates. It displays the name of the race if we hover over it and we can display or hide it from the corner legend.

Add pop-up windows

It would be nice to make that marker clickable and offer more information about the race. Let's create an HTML iframe that will pop up when the user clicks on the marker. You will need some basic HTML knowledge for that, but nothing fancy at this point.

# create an iframe pop-up for the marker
popup_html = f"<b>Date:</b> {data['Date']}<br/>"
popup_html += f"<b>Race:</b> {data['Race']}<br/>"
popup_html += f"<b>Time:</b> {data['Time']}<br/>"
popup_html += '<b><a href="{}" target="_blank">Event Page</a></b>'.format(data['Link'])
popup_iframe = folium.IFrame(width=200, height=110, html=popup_html)

# modify the marker object to display the pop-up
folium_marker = folium.Marker(location=[data['Latitude'], data['Longitude']], tooltip=data['Race'], popup=folium.Popup(popup_iframe), icon=folium.Icon(color='red'))
folium_marker.add_to(fg_marathons)

There it is. We even created a link on the URL, opening the event page in another tab. And since it's an HTML iframe, we can now basically display anything we want in it (pictures, videos, links, CSS styling, and so on).

Add GPX trace

The cherry on top, it would be amazing to display the route, like you usually see on the event website.

This was easier than I thought. GPX files recorded by GPS watches or phones are XML documents easy to read and parse. I used the gpxpy library for that, which is doing exactly what we need: read the GPX file and extract points/segments from it to display on our map. You can find GPX files everywhere, on Strava or Komoot for instance.

Here is how I opened and extracted segments from my own GPX file recorded during the race. I am using a step value when slicing all the points, to smooth out the curve (loading 1 every 10 coordinate points).

# parse gpx file
import gpxpy
gpx_file = 'berlin_marathon_2014.gpx'
gpx = gpxpy.parse(open(gpx_file))
track = gpx.tracks[0]
segment = track.segments[0]

# load coordinate points
points = []
for track in gpx.tracks:
    for segment in track.segments:
        step = 10
        for point in segment.points[::step]:
            points.append(tuple([point.latitude, point.longitude]))

# add segments to the map
folium_gpx = folium.PolyLine(points, color='red', weight=5, opacity=0.85).add_to(run_map)

# add the gpx trace to our marathon group
folium_gpx.add_to(fg_marathons)

Wonderful, the trace is showing up as expected on the map. Notice that, as we added it to the Marathons feature group, it inherits the red color and is affected by the legend checkbox too. You can now play around with the segment's weight, opacity, color and step value to fine-tune it.

That's it, you have all you need to build a kick-ass map. Just add multiple lines to the spreadsheet and modify your data frame to load all the data into lists.

Improve the map

If you browse through my run_map.py script, you will notice that it is a bit more advanced. Here are some improvements I added to my map and to the project itself, to make it more appealing and fit my needs:

• Add additional tile layers (ArcGIS)

• Group my events by type (halfs, marathons, ultras) and assign to each one a different color, also affecting the pop-up title and the G{X trace

• Customize the pop-up window with a picture of the race, a nice font, links, etc

• Move all paths and map settings to a JSON file, so there are no hard-coded values in the code and it is easier to change a setting (GPX trace weight or opacity for instance)

• Put all race events info into the Google spreadsheet

• Add an FTP upload method at the end to send the HTML, JPG and GPX files onto the online storage my blog is using

You can see the final result in the Events tab of my running blog.

Finally, you may notice that my script does not only update the map but also a table of events information displayed below it, as well as a little event-o-meter gadget in the sidebar. Both are generated when I run the script and are updated according to the updated Google spreadsheet information.

Conclusion

I therefore succeeded in my holy quest to put all my running events on a pretty nice-looking map. All I need to do now, after completing a new event, is to fill in the information in the Google doc and provide a jpg thumbnail and the GPX trace of my run, then run the script to generate a new map and update the table and gadget. This last step could be automated of course, if we had our script running in the cloud and checking any updates done on the spreadsheet.

Have fun playing with folium and don't forget to share your maps! Take care and see you on the trail 🏔️🏃‍♂️

It's that time of the year, AGAIN! It's been a fast one, but I do feel like I learned a lot, on my holy quest to become a rock-star programmer. That say, I would like to finish the year on a fun simple project. So when my mom asked me to draw our Secret Santas this year, I had it.

If you are not familiar with the concept, it's a nice way to celebrate xmas amongst a large group of friends or family members, withoug having to spend a salary on gifs. Every member of the group is assigned one recipient, and as his/her Secret Santa (since these random assignments are kept secret), you need to buy a gift for this person only. This way everyone will get a nice present instead of dozens of cheap ones.

In the past years, I used a basic random name generator for that, but with the downside that I knew all the assignments and who my Secret Santa was. So this year, let's be clever and use a python script to do the draw and contact Santas per email.

Get list of Santas from a CSV file

I leaned the Pandas library this year, which is great for extracting information from a CSV file. It's a user-friendly way to input data, easier than writing in the script directly. Here is our CSV file, with our six friends and the information we need to contact them.

The Black List

Black list, quid est? I like to spice things up and I thought it would be handy to add a black list feature, i.e. a list of names that one Santa should not get for any reason. In our case for instance:

• Ross and Rachel should not get each other, cause they had a big fight and they are 'on a break'
• Monica and Chandler should not get each other, as they are married and they wanna give each others more expensive presents
• Joey is broke, he bought a cheap present but it's for a guy so he cannot be one of the girl's Secret Santa
• Phoebe likes everyone so her black list is empty

Create the SecretSanta class

I learned programming using C++ and I have the CLASS word inprinted in caps inside my brain. So why not organise our script a bit by adding a SecretSanta class? Here are the class members and methods we need for the draw:

class SecretSanta:

    def __init__(self, name, email, black_list = None):
        """ initialize class variables """
        self.name = name
        self.email = email 
        self.recipient = None  # will be allocated later
        self.black_list = list()
        self.black_list.append(self.name)  # adding own name
        if black_list:
            self.black_list += black_list.split('|')

    def __repr__(self):
        """ override print method (optional) """
        # return string to print

    def contact_secret_santa(self):
        """ contact recipient """
        # use SendGrid example here

Load settings from a json file

Again, I find it more friendly to use external files to load variables instead of editing the code directly. So I usually add a settings.json file to all my projects. Here are the variables that we need to define there:

• Path to our CSV file containing Secret Santas information
• Path to the email template (.txt or .html)
• Max attempts that should be done while trying to assign recipients to secret santas. Due to the black lists, an assignment may not be possible and therefore this variable will break the while loop and raise an error.
• Email of your SendGrid account
• Personal SendGrid API key (private, don't share it with ANYONE)

{
    "csv_file": "data/secret_santas_list.csv",
    "email_file": "data/email.html",
    "attempts_limit": 100,
    "sg_sender_email": "youremail@domain.com",
    "sg_api_key": "your_api_key"
}

The json library helps you import and store these parameters in a SETTINGS dictionnary, that you can use further in your script.

import json
with open("data/settings.json", 'r') as json_file:    
    SETTINGS = json.load(json_file)
print(SETTINGS['csv_file'])
# output: 'data/secret_santas_list.csv'

Main - Create Santas

We now have everything we need to write our main procedure.

To extract the CSV infos into a Pandas dataframe and create our SecretSanta instances is a child game.

from pandas import read_csv
df = read_csv(csv_file_path).fillna('')
secret_santas = []
for i in range(len(df)):
    new_name = str(df['Name'][i])
    new_email = str(df['Email'][i])
    new_black_list = str(df['Black List'][i])
    new_santa = SecretSanta(name=new_name, email=new_email, black_list=new_black_list)
    secret_santas.append(new_santa)

Main - Draw

Finally we can draw our Secret Santas!

Since we introduced the black-list option, we face the issue that a draw might not be possible due to too many constraints (for instance, some poor guy who would be on everyone's black list!). I'm sure we could come up with a clever algorithm that would analyse all black-lists first and determine if a draw is possible or not. But let's keep it simple and take the easy road of trying, until we succeed or decide it's not gonna happen!

We will therefore wrap everything inside a while loop, that will break if the draw is successful (everyone has been assigned as Secret Santa) or after a maximum numner of attempts (defined in our settings file).

Here are the main steps I am following, check the GitHub project for the (well-commented!) code:

1. Initiate the while loop with our two conditions (success or reaching max attempts)
2. Shuffle our list of names and delete any previous assignment
3. Go through our list of SecretSanta instances and look for a recipient, which is not on the black-list. If we find one, we assign it and we jump to the next SecretSanta. If not, the draw fails and we start a new one (the while loop restart, we are back at step 2)
4. If we assign a recipient to all Secret Santas, the draw is successful and the while loop will break. If it never happens, the loop will break once reaching the max attemps.
5. Process the result: A succesful draw will trigger contacting the Secret Santas, while a failed one will raise an error and ask the user to review the input parameters.

E-mailing Santas

This is our last step. We are going to use Twilio SendGrid for this, which provides a cloud-based service that assists businesses with email delivery. This is a very easy step thanks to their crystal-clear tutorials. Here are the steps you need to follow to use SendGrid services.

Create a SendGrid account
You need to register for an account first. The free offer allows you to send up to 100 emails a day, which is good enough for our little project.

Create a Single Sender Verification
Your recipients will receive emails from a specific address. Terefore this e-mail needs to be existing and verified as your own.

Create a personal API key
This is an important step. The API key is personal and should never be shared. It will serve as a badass password (seriously, just look at it!) that will need to be passed everytime e-mails are sent from your account. For simplicity, you can add it to your environment variables.

Install the SendGrid python library
Finally you can install the python library. The documentation gives you some simple examples on how to send your first emails. I did not need to dig much further, those are great. This project uses the "Without Mail Helper Class" example, that I implemented in my contact_secret_santa() class method.

Personalising the email content

This is an optional but nice little step. I found it more joyful and christmassy to write the email contents as from the hand of Santa himself, asking people to help him spread the joy. Sendgrid let you use a text or html format for the contents of the email. So why not draft an html file with some pictures, colors, a handwriting font, etc? You can also put tags like [NAME] and [RECIPIENT] that will be replaced with your class attributes to address the person directly. Put some CSS to make it look more personal and believable. I don't know much about web dev but feel free to check my html template.

Once all these steps are done and if your draw was successful, you can now call the contact_secret_santa() method of all your instances. They will be notified within a few minutes and receive an email based on your template. Hey, you got one too!

Conclusion

I hope this little project taught you something new, whether you are playing with CSV and json files, trying to send emails or simply improving your python skills. It was a fun last projet to wrap up this year, which has been for me rich in learning new stuff and hopefully a few more steps towards my goals.

I wish you all a wonderful Christmas. Have a great Secret Santa sharing time with the ones you love (and the ones from your black list too!). Take care and read you next year :)

Friends pictures © Warner Bros. Entertainment Inc
Cover picture from Rachel Middleton

About MetaHumans

I recently started playing around with the mind-blowing MetaHuman (MH) feature from Unreal Engine. Through the Quixel Bridge editor, you can create and customize a highly realistic avatar in a few clicks, and send it to your Unreal project. Everything is fine-tuned to get the best of Unreal shaders and will leave you speechless 😍 I invite you to take a peek at it, plenty of resources out there.

About Live Link Face

As a mocap director, my interest quickly shifted over the live-retarget capabilities, i.e. how to transfer an actor's facial performance onto a metahuman, in real-time. Epic offers an app for that, which takes advantage of the iPhone's TrueDepth camera: Live Link Face (LLF). It also takes a few click to link your phone to your Unreal project and start driving your MetaHuman's face. Again, it's working 'Epic-ly' fast and well!

But there is the snag... The app was clearly designed to work in real-time with Unreal, where you can use the Sequencer to record facial data along body mocap, blueprints functions and so on. However, if used as a stand-alone app, it's another story.

LLF saves for each take 4 files:

• A CSV file containing the animation

• A video file of the performance

• A json file containing some technical information

• A jpg thumbnail

  

No maya scene, no unreal asset, not even a simple fbx file that we could use to import and clean the motion in a 3rd-party-software like MotionBuilder. Just a CSV file filled with random blendshape names and millions of values. Like that zucchini-peeler you got from aunt Edna for xmas, it's a nice gesture but what the heck am I gonna do with that!? 🤔

A quick look into the documentation confirmed the idea:

This data file is not currently used by Unreal Engine or by the Live Link Face app. However, the raw data in this file may be useful for developers who want to build additional tools around the facial capture.

So, I guess we are on our own now!

Let's dig in

It's not the most convenient format, but if you take a good look at this CSV file, it's all there: For each timecode frame (first column), you get a value of all the shapes defining the facial expression at that frame. These shapes come from Apple's ARKit Framework, which uses the iphone's TrueDepth camera to detect facial expressions and generate blendshape coefficients. All we have to do is extract those time/value pairs and store then in some handy container. This is where the Pandas library 🐼 and its dataframes are gonna do wonders!

Now what should I do with all this?

I need to recopy the animation values onto a target model that Unreal can import and read. Let's have a look at the MetaHuman project, that you can find on the UE Marketplace.

Facial and body motions are handled separately. A blueprint is applied to a facial version of the skeleton, which contains only the root, spine, neck, head and numerous facial joints, as Unreal relies on joint-based motions.

I can get this skeleton out of Unreal by creating an animation pose and exporting it as FBX. Now we can open it in motionbuilder and... HOLY MOLY GUACAMOLE, what's all this!? 😵 I'm not talking about the Hellraiser facial rig but about the root joint properties. It contains a huge amount of custom properties!

I run a quick line of code and count about 1100 entries:

• 254 properties starting with CTRL_expressions_, which are clearly blendshapes, like CTRL_expressions_jawOpen

• 757 properties starting with head_, which seem to be additional correctives (fixing for instance a collision between two blendshapes activated at the same time).

• 35 properties starting with cartilage_, eyeLeft_, eyeRight_ or teeth_, which are probably more correctives?

• Last but not least: 53 properties, whose name are identical to the ARKit shapes in the CSV files from LLF, like EyeBlinkLeft or JawOpen.

We can therefore assume that these last 53 properties are correspond to the ARKit blendshapes from the app and should receive our key values during retargeting.

How is Unreal computing animations?

Before jumping into our favourite code editor, let's see how Unreal is reading and applying an animation onto the MetaHuman. I open the Face_AnimBP blueprint from the project, which is the one handling live-retargeting facial animations from the LLF app onto the rig.

BINGO, there it is! Inside a Live Link Face box, I find what seems to be a mapping block. It's doing all the connections. For each ARKit blendshape used by LLF, it is mapping the corresponding CTRL expressions and head correctives on the metahuman, by specifying an influence value between 0 (no influence) and 1 (full influence). You can see below how BrowOuterUpLeft triggers the 3 expressions browLateralL, browRaiseInL and browRaiseOuterL with different weights, while having no effect on any other expression.

To confirm, I create random keys on the different properties in MotionBuilder and export the takes as FBX files to Unreal. It's working like a charm!

• Keys on the ARKit custom properties are read and retarget, but only if the animation goes through the mapping block first, so the animation blueprint is mandatory.

• However, if I put keys on the CTRL shapes and head correctives, I can apply my animation directy onto the MH, without the need of an animation blueprint.

As you can see, we have two solutions taking shape here: The easy road where we simply transfer animation values from the CSV file to the ARKit blendshape properties, or the long road where we convert them into MH expression and corrective values. Trying both?

Retargeting on ARKit blendshapes

As said, this is the easy stroll in the park. Our CSV file and our root bone are sharing the same ARKit blendshape information. All we need to do is transfer the keys. Here are the main steps I am following:

1. Use Pandas to read the CSV file and extract all data. I create for this a BlendShape class, which receives the ARKit shape name, as well as animation keys, as pairs of timecode/values.

2. For all BlendShape objects created, I search on the skeleton root for the corresponding property name and create keys for all timecode/value pairs.

3. The scene framerate is very important here and should match the app's one, or keys won't be created at the correct frames.

Once done, adjust your timespan to frame the animation and have a look at the ARKit properties of the root. They should be animated.

Export your animation as FBX to Unreal and test it in the Face_AnimBP blueprint. Don't forget it needs to go through that mapping block so Unreal can convert ARKit blendshape values to MH property values, which will be then turned into joint information. Cool beans!

Retargeting on MetaHuman properties

Now let's get our hands dirty and explore that second solution! We want to be able to use a retarget file directly onto our MH character without going through all this mapping knick-knacks. But for that we need to find a way to recreate this mapping node in MotionBuilder. As a reminder, it is linking each ARKit shape to the corresponding MH ctrl and head properties with weighted values ranging from 0 to 1. So we first need to extract these values and weights from Unreal 🥵

After looking around for a while, I noticed that Unreal nodes can be exported as T3D ascii files, which contain all information on what the node is doing. I tried and I must admit that it wasn't love at first sight. Fortunately, it's only 13 lines long!

But let's not judge a book by its cover. Again, if you take a closer look, everything we need is here. Time to see if my regex ninja training has paid off. After playing around with three patterns, I manage to extract from the T3D file:

• The ARKit blendshapes names

• The MH properties names

• The mapping weights for each property I add to my BlendShape class (characterising an ARKit shape), the target MH properties and the mapping weights. Done! I now have all the information I need for retargeting.

Back to MotionBuilder, I can run my magic loops again. This time we have to be a bit more careful though:

1. Read the T3D file and create BlendShape instances for all ARKit shapes.

2. Use Pandas to read the CSV file and extract the animation keys for each ARKit shape. I fill up these timecode/value pairs into my BlendShape class instances.

3. MotionBuilder is not very fast at finding root properties, maybe due to the high amount. To avoid searching for the same ones multiple times, I loop through the MH properties first and then through the timecode. For each frame, I check which ARKit blendshapes triggers my property and get the animation value weighted by my mapping information. For instance, if the ARKit value is 0.8 and the mapping value for this property is 0.5, then my animation key will be 0.8x0.5=0.4. If multiple ARKit shapes are triggering it, I take the max value corresponding to the highest influence.

4. Again, be careful to retarget at the same framerate you used when recording takes.

I see keys on my root's properties! Let's export it as FBX to Unreal and assign it to my Face component. WUNDERBAR! Unreal is reading the animation and transfering it to the facial joints without having to go through that mapping node anymore.

Wrapping up

To finalise the script and make it more user-friendly, I added the following features:

• A batching option, in case you point to a folder containing multiple CSV files

• The possibility to offset the starting timecode, to synchronise the animation file with other sources

• A simple PySyde UI for the user to enter the required paths files

• Some logs and exception handlings to make sure that our batch is performing properly

We are done, good job everyone! 😉

Conclusion

It only took patience and observation, as well as some ancient Regex voodoo to transfer the animation values recorded by the Live Link Face app as CSV files onto our brand new MetaHuman in Unreal. While not perfect, the result is good enough to move the animation to a cleaning stage.

This is therefore a quick way to record facial animations from your actors, even if they cannot perform in real-time. The ability to process them at a later stage using an animation software like MotionBuilder and a fairly simple python script like this one makes of Live Link Face a cheap and reliable facial motion-capture solution for MetaHumans characters.

I hope this will help you write your own solution. Feel free to message me if you need any help or buy me a croissant and cappuccino next time you're in town 🥐☕ Cheers!

I'm not gonna lie to you... I LOVE REGEX!

As a kid, I grew up playing adventure games full of puzzles and riddles. Looking for the solution was a personal quest, a treasure hunt. Finding it was so exciting, but not as much as jumping on another riddle!

When I discovered regular expressions on my journey to become a (good) python programmer, I felt that same excitement. I was blown away by the countless possibilities these were offering. Deciphering one was like suddenly being able to read hieroglyph, writing one was like discovering I could speak a foreign language. Although I know they should be used with caution and in special cases only, I keep pushing myself to use them everywhere I can.

That's why, when Codewars challenged me to write a function to convert roman numerals from/to Arabic numbers, I could not resist writing a regex to help me solve that problem.

Enough chit-chat, let's get our hands dirty.

My first task was to validate if the user input was a valid roman numeral. To sum up, roman numerals consists of the following symbols:Source: Wikipedia

It seems that the thousands unit [M] does not extend past [MMM], which means that the biggest roman number would be [MMMCMXCIX], or 3999. I'm not sure if numbers could go higher than that and why the limit, anyway for the sake of this problem I limited myself with numbers between 1 and 3999.

Now the trick is that the symbols placement is very important. If you don't put them in the right order, the resulting number would be invalid and unreadable. As listed in the table up there, numerals should start with thousands [M] 1000, followed by hundreds [D/C] 500/100, then dozens [L/X] 50/10, and finally units [V/1] 5/1.

BUT, that's not it! Numerals can only repeat 3 times like [CCC] 300 before switching to a combo of two numerals like [CD] 400. So you can still have a [C] 100 before an [M] 1000, like in [CM] 900 for instance.

Bit confusing, isn't it?

Alright, let's recap our conditions:

• Roman numbers are ranging from [I] 1 to [MMMCMXCIX] 3999
• Numerals should follow a precise order: [M] 1000 / [D] 500 / [C] 100 / [L] 50 / [X] 10 / [V] 5 / [I] 1
• A numeral cannot repeat more than 3 times, it then uses a pair
• The following pairs are allowed: [CM] 900 / [CD] 400 / [XC] 90 / [XL] 40 / [IX] 9 / [IV] 4

Do you start to see our REGEX showing up? :)

Let's translate this into code.

For this we are going to use a tag I find really helpful when writing regex is the verbose one (re.VERBOSE or re.X) It allows you to spread your pattern on multiple lines and be more readable. Let's try it!

import re

def is_roman_number(num):

    pattern = re.compile(r"""    
                                ^M{0,3}
                                (CM|CD|D?C{0,3})?
                                (XC|XL|L?X{0,3})?
                                (IX|IV|V?I{0,3})?$
            """, re.VERBOSE)

    if re.match(pattern, num):
        return True

    return False

Wow, that looks amazing already! Let's take a closer look at these 4 lines:

• ^M{0,3} = Between 0 and 3 [M] at the beginning [^] of the string
• (CM|CD|D?C{0,3})? = One pair [CM] or one pair [CD] or [D], followed by up to 3 [C]. Each element is optional [?], as well as the whole block [()?]
• (XC|XL|L?X{0,3})? = One pair [XC] or one pair [XL] or [L], followed by up to 3 [X]. Each element is optional [?], as well as the whole block [()?]
• (IX|IV|V?I{0,3})?$ = One pair [IX] or one pair [IV] or [V], followed by up to 3 [I]. Each element is optional [?], as well as the whole block [()?], which should be at the end of the string [$]

Let's test our code

I'm using a simple fstring calling my function and comparing the string against our pattern to validate the numeral or not:

num_valid = 'MMDCCLXXIII'
num_invalid = 'CCCMMVIIVV'

print(f"{num_valid} is {'not' if not is_roman_number(num_valid) else ''}a roman number")
print(f"{num_invalid} is {'not ' if not is_roman_number(num_invalid) else ''}a roman number")

# Output:
# MMDCCLXXIII is a roman number
# CCCMMVIIVV is not a roman number

That wasn't so bad after all! Now look at this and tell me it's not the most beautiful thing you've seen in your life:

^M{0,3}(CM|CD|D?C{0,3})?(XC|XL|L?X{0,3})?(IX|IV|V?I{0,3})?$'

That's all folks! Let me know if you are interested by the second part of the challenge: converting roman numerals from/to Arabic numbers and I will share my solution.

Stay safe out there and read you soon :)