HarmonyOS Next Smart Security: Face Comparison and Heterogeneous Computing

This blog post delves into the practical application of face comparison and heterogeneous computing in developing a smart security system using Huawei's HarmonyOS Next (API 12). We'll explore the system's architecture, implementation details, performance optimization strategies, and address challenges encountered during development.

I. Smart Security System: Requirements and Architecture

(1) Functional Requirements Analysis

1. Real-time Face Detection and Recognition: The system must quickly and accurately detect faces in video streams, comparing them against a preset database to identify individuals. This is crucial for applications like airport security, where real-time monitoring and identification of suspicious individuals are paramount.
2. Efficient Multi-Camera Data Processing: The system must efficiently handle data from multiple cameras simultaneously, addressing parallel processing, resource allocation, and data transmission optimization. A large shopping mall security system, with dozens of cameras, exemplifies this need for scalable data processing.

(2) HarmonyOS Next-Based Architecture Design

1. Hardware Selection: Choosing devices supporting heterogeneous computing, such as smart cameras with CPUs and NPUs, is critical. The CPU manages system logic, data preprocessing, and non-computation-intensive tasks (e.g., video decoding), while the NPU accelerates computation-intensive tasks (e.g., face comparison) leveraging its parallel processing capabilities.
2. Software Hierarchical Design:
   • Data Collection Layer: Collects video data from multiple cameras using a reliable protocol like RTSP, handling potential network failures.
   • Processing Layer: The core layer, employing the Core Vision Kit for face feature extraction and comparison. The HiAI Foundation Kit enables heterogeneous computing, assigning tasks to the CPU or NPU based on computational needs. Image preprocessing might run on the CPU, while feature extraction and comparison are offloaded to the NPU.
   • Storage Layer: Stores the face database, system configurations, logs, etc., using a database system like MySQL or SQLite, with data encryption and backup measures in place.
3. Face Comparison and Heterogeneous Computing Collaboration: The CPU preprocesses video data, the Core Vision Kit extracts features (accelerated by the NPU via the HiAI Foundation Kit), and comparison occurs on the NPU. Heterogeneous computing dynamically allocates resources based on system load and task priorities, ensuring timely and accurate face comparison.

(3) Handling Complex Scenarios

1. Multiple Face Recognition: The system detects multiple faces, extracts features for each, and performs parallel comparisons (leveraging multi-core CPU/NPU capabilities). Ambiguous results might require secondary verification or manual review.
2. Occlusion Handling: Partial face occlusion is addressed using local feature-based recognition (focusing on unoccluded areas) and potentially integrating multi-modal information (e.g., posture, gait).

II. Core Function Implementation and Technology Integration

(1) Face Comparison Function: Implementation and Optimization

Implementation Process Using Core Vision Kit

import { FaceDetector, FaceFeatureExtractor } from '@kit.CoreVisionKit';

// Initialize the face detection model
let faceDetector = new FaceDetector();
await faceDetector.init();

// Initialize the face feature extraction model
let faceFeatureExtractor = new FaceFeatureExtractor();
await faceFeatureExtractor.init();

// Read the video frame (assuming the video frame data has been obtained)
let frame = getVideoFrame();

// Detect the face
let faceResults = await faceDetector.detect(frame);
if (faceResults.length > 0) {
    // Extract the face features
    let faceFeature = await faceFeatureExtractor.extract(frame, faceResults[0].boundingBox);
    // Here, the extracted features can be stored or compared with the features in the database, etc.
}

Performance Optimization Techniques

Optimizations include techniques such as image pyramid processing for face detection (detecting faces at multiple resolutions) and image normalization for feature extraction stability.

import { ImagePyramid, FaceDetector } from '@kit.CoreVisionKit';

// Create an image pyramid
let imagePyramid = new ImagePyramid(frame, { minSize: 30, maxSize: 300, scaleFactor: 1.2 });

// Traverse each level of the image pyramid for face detection
for (let i = 0; i < imagePyramid.getNumLevels(); i++) {
    let levelFrame = imagePyramid.getLevel(i);
    let faceResults = await faceDetector.detect(levelFrame);
    if (faceResults.length > 0) {
        // Process the detected face results, such as converting the coordinates to the original image coordinates, etc.
        for (let face of faceResults) {
            face.boundingBox = imagePyramid.convertToOriginalCoordinates(face.boundingBox, i);
        }
        break; // If a face is detected at a certain level, the loop can be ended early
    }
}

(2) Heterogeneous Computing in the Security System

Task Allocation with HiAI Foundation Kit

import { HiaiEngine } from '@kit.HiAIFoundationKit';

// Define the matrix multiplication calculation function (here it is assumed that the matrix multiplication algorithm has been implemented)
function matrixMultiplication(a: number[][], b: number[][]) {
    // The calculation logic is omitted, and the matrix multiplication algorithm needs to be implemented in practice
    return result;
}

// Create a CPU engine instance
let cpuEngine = new HiaiEngine('CPU');
// Create an NPU engine instance
let npuEngine = new HiaiEngine('NPU');

// Define two matrices
let matrixA = [[1, 2, 3], [4, 5, 6]];
let matrixB = [[7, 8], [9, 10], [11, 12]];

// Execute the matrix multiplication task on the CPU
let cpuResultPromise = cpuEngine.executeTask(() => {
    return matrixMultiplication(matrixA, matrixB);
});

// Execute the matrix multiplication task on the NPU (assuming the NPU supports matrix multiplication acceleration operations, this is just an example)
let npuResultPromise = npuEngine.executeTask(() => {
    return matrixMultiplication(matrixA, matrixB);
});

// Wait for the CPU task to complete and get the result
cpuResultPromise.then((result) => {
    console.log('CPU calculation result:', result);
});

// Wait for the NPU task to complete and get the result
npuResultPromise.then((result) => {
    console.log('NPU calculation result:', result);
});

Resource Scheduling Strategies

For multi-task scenarios, resource scheduling strategies prioritize tasks based on urgency and resource availability. Task queues manage the order of execution, preventing resource conflicts.

III. Performance Evaluation and System Optimization

(1) Performance Evaluation Indicators and Methods

1. Face Comparison Accuracy: Evaluated using accuracy and recall rate, using a test set with diverse conditions (lighting, pose, occlusion).
2. Response Speed: Measured as the time from video frame capture to result output, under varying system loads.

(2) Performance Bottleneck Analysis and Optimization Strategies

1. Data Transmission Bottlenecks: Addressing bandwidth limitations through higher-bandwidth network devices, optimized network configuration, efficient transmission protocols, and video compression.
2. Computing Resource Allocation: Monitoring CPU and NPU utilization to identify and rectify imbalances. Dynamic task allocation and task scheduling algorithms (e.g., First Come First Served, Shortest Job First) improve efficiency.

(3) Optimized System Application Effects and Lessons Learned

1. Application Effects: A real-world example demonstrates the system's capabilities in a smart community setting, achieving high accuracy and fast response times.
2. Lessons Learned: Proper utilization of HarmonyOS Next features, data quality considerations, and careful hardware/network selection are highlighted.