How to Develop AI based App with React Native? Facial Landmark Detection Example

Businesses not only want to build a mobile or web application; they also want to integrate the richness of artificial intelligence. When it comes to application development, React Native is one of the most popular app development technologies. The reason behind its popularity is that React Native offers a cross-platform and cost-effective solution for businesses. Furthermore, development time is significantly reduced, allowing startups to launch their applications more quickly.

While exploring the dynamics associated with React Native, artificial intelligence stays in the trending zone. Businesses want to leverage artificial intelligence in their React Native applications by hiring a React Native development agency and achieving their business goals in terms of technological advancements. If you want to explore examples of AI in applications, Snapchat is a very easy reference. The application uses facial landmark detection to change the size and shape of the face, letting people have fun with diverse camera filters.

What about you? What can you achieve by combining artificial intelligence and React Native? In this article, we will explore AI machine learning models with React Native. We will also look at a use case explaining how to build a facial landmark detection app using React Native and TensorFlow. So, without further ado, let us get started with the popular available AI and ML models.

AI Machine Learning Models for React Native Application

Although there are ample machine learning models that you can explore using the React Native application, here are a few of the popular ones.

Vision models

The machine learning models falling into the vision category are used to analyze features in images and videos. Whether for mobile apps or web apps, vision models apply intelligent algorithms to let businesses achieve real-time experiences.

Examples of vision models

• Image classification
• Object detection
• Semantic Segmentation

Body-based ML models

Artificial intelligence technologies can determine points and poses by recognizing the face, hands, and other body elements. These models can be used for predicting the boy’s weight, fat level, and more, giving a competitive edge to businesses aligning with the use of body-based ML models.

Examples of body-based ML models

• Face detection
• Face Landmark detection
• Pose detection
• Body segmentation
• Hand pose detection
• Portrait depth estimation

Text & Audio based ML models

Text-based ML models are used to analyze and classify the textual form of data. Natural language processing uses this type of model extensively to achieve the desired activities. Tools like Chat GPT embed intelligent text-based ML models to produce the results. Tools like Siri are examples of audio-based ML models.

Examples of Text & Audio based ML models

• Natural Language question answering
• Text toxicity detection
• Universal sentence encoder
• Speech command recognition

The above models are dominating the artificial intelligence industry. Applications are exploring the opportunities and key areas to implement these advanced ML models and provide the best value to their users. Let us guide you through a practical demonstration of face landmark detection using React Native to further strengthen your knowledge. We advise you to hire dedicated mobile app developers to build AI-rich examples due to their complexities.

AI-based App with React Native- Building Face Landmark detection app

In this section of the article, we will give you a brief demonstration of how to build a face landmark detection app- an AI-based app with React Native.

What is facial landmark detection?

Facial landmark detection is a rich AI-based machine-learning technique that detects different points on the surface of a human face (like eyes, nose, ears, and more). With machine learning capabilities, an application successfully generates a 3D model of a human’s face, detecting nearly 500 points on the human face surface. Popular applications like Snapchat use facial landmark detection techniques to apply different filters to human faces.

How will we achieve it?

In order to achieve this, we will use React Native and TensorFlow. React Native will help us easily build the front end on the web browser; however, you can port the application to Android or iOS platforms. So, we will be:

• Creating webcam steam inside the front-end react application
• Loading FaceMesh from tensorflow.js and making detection from a stream
• Drawing key points and triangles on the javascript canvas

Steps to Build an AI-Based Mobile App with React Native

After completing all the steps, you will be able to see the drawing points making a 3D diagram of your face and detecting all your facial motions effectively. The complete code for landmark face detection can be found on the TensorFlow Landmark Face Detection GitHub page.

1. Create the React Native application on your system to build the assets.
2. Install the dependencies (Tensorflow, fashmesh from Tensorflow, react-webcam) and import them to your application.
3. Setup Webcam and canvas
4. Define references
5. Load facemesh (Loading the TensorFlow function in your application)
6. Build detect function (get video properties, set video width, set canvas width, make detection, get canvas context for drawing)
7. Draw utilities
8. Setup the triangle path (from the browser’s console)
9. Setup point drawing
10. Add drawMesh to detect the function

Code sample for building AI AI-based mobile app with React Native

// Face Mesh from Tensor Flow

import React, { useRef, useEffect } from "react";

import "./App.css";

import * as tf from "@tensorflow/tfjs";

// OLD MODEL

//import * as facemesh from "@tensorflow-models/facemesh";

// NEW MODEL

import * as facemesh from "@tensorflow-models/face-landmarks-detection";

import Webcam from "react-webcam";

import { drawMesh } from "./utilities";

function App() {

const webcamRef = useRef(null);

const canvasRef = useRef(null);

// Load posenet

const runFacemesh = async () => {

// OLD MODEL

// const net = await facemesh.load({

// inputResolution: { width: 640, height: 480 },

// scale: 0.8,

// });

// NEW MODEL

const net = await facemesh.load(facemesh.SupportedPackages.mediapipeFacemesh);

setInterval(() => {

detect(net);

}, 10);

};

const detect = async (net) => {

if (

typeof webcamRef.current !== "undefined" &&

webcamRef.current !== null &&

webcamRef.current.video.readyState === 4

) {

// Get Video Properties

const video = webcamRef.current.video;

const videoWidth = webcamRef.current.video.videoWidth;

const videoHeight = webcamRef.current.video.videoHeight;

// Set video width

webcamRef.current.video.width = videoWidth;

webcamRef.current.video.height = videoHeight;

// Set canvas width

canvasRef.current.width = videoWidth;

canvasRef.current.height = videoHeight;

// Make Detections

// OLD MODEL

// const face = await net.estimateFaces(video);

// NEW MODEL

const face = await net.estimateFaces({input:video});

console.log(face);

// Get canvas context

const ctx = canvasRef.current.getContext("2d");

requestAnimationFrame(()=>{drawMesh(face, ctx)});

}

};

useEffect(()=>{runFacemesh()}, []);

return (

<div className="App">

<header className="App-header">

<Webcam

ref={webcamRef}

style={{

position: "absolute",

marginLeft: "auto",

marginRight: "auto",

left: 0,

right: 0,

textAlign: "center",

zindex: 9,

width: 640,

height: 480,

}}

/>

<canvas

ref={canvasRef}

style={{

position: "absolute",

marginLeft: "auto",

marginRight: "auto",

left: 0,

right: 0,

textAlign: "center",

zindex: 9,

width: 640,

height: 480,

}}

/>

</header>

</div>

);

}

export default App;

Final words

Artificial intelligence brings a pool of opportunities and exploration for mankind. And, as the technology evolves, we can explore a lot of dimensions associated with it. Machine learning models are dominating the space of AI, and technologies like Tensorflow are demonstrating the ease of embedding advanced models into applications. If you are planning to build a sustainable business, you must explore and embed trends like AI in your mobile or web app.