OpenIAI-SNIO: A Systematic AR-Based Assembly Guidance System for Small-Scale, High-Density Industrial Components

1 Introduction

Digital array modules, electrical connectors, and other smallscale, high-density industrial components (SHIC) can achieve complex functions within a limited space and are widely used in high-end electromechanical devices such as aircraft, satellites, rockets, and ships [Zhao et al., 2024; Zhou et al., 2024]. SHIC assembly requires connecting small parts to small and dense assembly positions according to strict correspondence [Hartisch and Haninger, 2024]. The operations are complicated, time-consuming, prone to errors, and may even require magnifying glasses, which has become a bottleneck in the product assembly process.

Due to the virtual-real fusion characteristics of augmented reality (AR) [Geng et al., 2024; Geng et al., 2022], which can significantly improve human operational capabilities, the use of augmented reality technology for fine assembly assistance and guidance has become one of the most promising research directions [Wang et al., 2024]. For SHIC, the key to its AR-based assembly guidance is to accurately identify, locate, and sequence the assembly targets and effectively guide the entire assembly process in the case of occlusion [Eswaran and Bahubalendruni, 2022].In recent years, some researchers have combined prior knowledge of SHIC design to compensate for the fuzziness and uncertainty of computer vision methods. The research focuses mainly on the following three aspects:

∗Corresponding author.

1. SHIC prior knowledge and its application. Visual cognition of complex product is very difficult and often requires some prior knowledge for better cognitive outcomes [Gulivindala et al., 2020]. The prior knowledge of SHIC mainly includes deep learning image samples, structural features of SHIC, and the layout, position, and sequence of SHIC assembly targets. Almost all the researchers [Zhao et al., 2022a; Wu and Li, 2020; Zhao et al., 2022b] constructed the data set and annotated images manually and used prior knowledge by manual construction or hard code writing. This method is time-consuming and difficult to dynamically adapt to the new type of SHIC.
2. SHIC visual cognition. The core of AR guidance for SHIC assembly is to accurately identify, locate, and sequence the high-density and small assembly targets under occlusion situations. This is a tough challenge for the current visual cognition of AR [Wang et al., 2024]. Some methods [Quan et al., 2023; Pan et al., 2020] have shown success in identifying and positioning simpler SHIC components, but are not sufficient for more complex SHICs, where higher precision and target sequencing are required. Additionally, while some methods [Zhao et al., 2022a] can repair false detections and extract geometric attributes of components, they do not sequence targets or address occlusion. Other sequencing methods [Li et al., 2021] rely on clustering targets based on certain features, but these methods are also limited by occlusion.
3. Operation guidance of SHIC assembly. Various visual guidance techniques have been explored to improve guidance effectiveness, such as highlighting mismatched components [Li et al., 2021] or using wearable devices to de-
4. tect assembly errors [Kucukoglu et al., 2018]

these methods are primarily result-oriented, lacking in continuous guidance for assembly operations. Some commercial AR systems [Upskill, 2017; Dassault Syst`emes, 2024], project assembly instructions on AR glasses or large screens but do not offer intelligent recognition, positioning, or sequencing. Furthermore, occlusion causes visual recognition algorithms to fail and prevents the completion of the assembly process [Eswaran et al., 2023], which remains unresolved in current research, limiting the practical application of AR

guidance for SHIC assembly.

In conclusion, the existing AR-guided technologies for SHIC face three significant challenges:

1. Almost all existing research relies on manually generating prior knowledge, which seriously affects the degree of automation and usability.
2. Existing research cannot simultaneously complete and sequence SHIC assembly targets using prior knowledge for better visual cognition.
3. There is still no effective method for continuously guiding the whole assembly process with different strategies in the case of occlusion.

To address these challenges, this research develops a systematic AR-based assembly guidance system OpenIAI-SNIO (Open Industrial Augmented Intelligence - Smart Navigator of Industrial Operations) for SHIC based on computer vision, deep learning, rule-based reasoning, 3D tracking, virtual reality matching, and other technologies. In this work, we make the following contributions:

1. Systematic technical framework of AR-based assembly guidance for SHIC. This framework combines prior knowledge reasoning and computer vision cognition in a simple and interpretable way. The theoretical innovation and effectiveness of this technical framework have been verified [Geng et al., 2025]. It provides a practicable technology architecture for the intelligent assistance of small-scale assembly and expands the research of AR technology to the fine operation field.
2. OpenIAI-SNIO system. This system combines a universal AR-based assistance software and a portable integrated desktop AR hardware. Our system integrates automatic generation of prior knowledge and visual recognition of physical assembly targets, and has achieved demonstration applications and commercial promotion in aviation field.

2 Overview

2.1 Technical Framework

The proposed technical framework, shown in Figure 1. The first part of the framework is the offline automatic generation of prior knowledge using computer vision, deep learning and rule reasoning. The prior knowledge comes from the SHIC 3D model, which includes three aspects: the deep learning recognition model used to recognize the virtual and physical assembly targets of SHIC; the design information of assembly target (including assembly position and sequence) automatically obtained from the SHIC 3D model using the recognition model; and the sparse view model for real-time tracking of physical SHIC.

The second part involves online visual recognition of physical assembly targets using 3D tracking, deep learning, image processing and point set registration, which has three main steps.

1. The sparse viewpoint model is used to track the physical SHIC, and the real-time pose of the SHIC is obtained, and then the ROI of the SHIC is obtained.
2. By applying the deep learning recognition model to ROI, the positions of some assembly targets are obtained, and then the accuracy of positions is optimized by combining image processing.
3. The positions of some recognized assembly targets are matched with prior knowledge, and all unrecognized assembly targets are completed and sequenced by reasoning.

The third part is online AR guidance for all assembly targets based on their positions and sequence. Different guidance strategies are automatically adopted to cover the entire assembly process depending on the occlusion severity.

2.2 Software-Hardware System

This research develops a universal AR-based assistance system OpenIAI-SNIO using OpenCV, Unity, and Visual Studio, shown in Figure 2.

The hardware layer forms the hardware foundation of the entire system. The system's visual module consists of a binocular vision camera and a depth camera, with resolutions of 4032×3040 and 1280×800 respectively. The system's calculation module consists of Intel(R) Core(TM) i7-12700 CPU @ 2.40GHz and NVIDIA GeForce RTX 3050. The system's also has a touch screen.

The data layer encompasses all the data necessary for the system's operation, including data base, algorithm base, and knowledge base. The data base includes AR instruction, 3D model, process specification, etc. The algorithm base includes various algorithms such as deep learning, 3D tracking, and industrial vision. The knowledge base includes sparse viewpoint model, deep learning recognition model, prior assembly target information, etc.

The offline function layer involves preliminary preparations for using this system for assembly operations, including prior knowledge generation, 3D model management, and AR instruction design.

The online function layer involves various links in the implementation of intelligent assembly operations, including

(a) Position

-

based Precise Guidance

(b) Region based Heuristic Guidance

-

Figure 3: Visual change of guidance modes in OpenIAI-SNIO.

AR instruction loading, AR assembly guidance, and assembly execution recording.

The application layer involves the main processes of the system's operation and the participants involved. The calculation module automatically extracts prior information from the 3D model, and designers generate instruction information through AR instruction design. Then the prior information and the instruction information are combined and utilized by workers for on-site assembly operations. Figure 3 shows the application modes of this system at different stages.

3 Application Experiment

The complex electrical connectors [U.S. Department of Defense, 2023; U.S. Department of Defense, 2021] is a representative object of SHIC, characterized by extremely high density, small scale, and complicated assembly. This research conducted multiple experiments on the connector insertion process using OpenIAI-SNIO to test the system. This research used the repetitions and the cognition duration of insertion to evaluate the study's effectiveness. The repetitions refer to the times of repetitive insertion operation after the assembly personnel found the insertion error, reflecting the change in robustness. The cognition duration of insertion refers to the time spent reviewing process documents and observing assembly targets, reflecting the change in efficiency. The test results in Table 1 shows that robustness and efficiency of the SHIC assembly operation have been improved by 92.95% and 87.06%, respectively. Figure 4 shows that OpenIAI-SNIO can effectively reduce the workload of workers. OpenIAI-SNIO can meet the industrial requirements of performance, availability and effectiveness.

4 Conclusion and FutureWorks

This research develops a systematic AR-based assembly guidance system OpenIAI-SNIO that can meet the performance, availability, and effectiveness requirements of the industry. OpenIAI-SNIO has been successfully deployed and applied to a certain company in the aviation field. The followup research of this research involves fully using prior knowledge to carry out real-time error correction and quality inspection in the assembly process and then forming a more systematic SHIC AR assistance operation system.

Acknowledgments

This research was supported by the National Natural Science Foundation of China under Grant No. 52175484 and the Stable Support Project of the Ministry of Industry and Information Technology of China under Grant No. WD-2024-7-3.