SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval

Longtao Jiang, Min Wang, Zecheng Li, Yao Fang, Wengang Zhou, Houqiang Li

abstract

• propose a novel representation framework called Semantically Enhanced Dual-Stream Encoder (SEDS), which aggregates Pose and RGB modalities to represent local and global information of sign language videos.
• The aggregated clip-level video features are then fed into the CLIP vision encoder for interaction, and matched with the word-level text features embedded using the CLIP text encoder.

In recent years, a series of document pre training models represented by Microsoft LayoutLM have also achieved significant results in document understanding tasks.The Quark Vertical Innovation Team is responsible for solving the problem of document multimodal understanding in library type documents and web-based documents. The task of understanding web-based documents includes layout classification, page experience, semantic segmentation, node extraction, etc; The task of understanding library type documents includes layout classification, layout experience, domain categories, title extraction, etc. The two are very similar in terms of multimodal model structure and downstream analysis and understanding tasks. Quark explored modal document pre training techniques suitable for web-based data based on its own search engine’s massive web page data, and achieved significant improvements in downstream tasks of analyzing and understanding multiple web pages.

In view of the current shortage of human resources under the COVID-19 epidemic, the high risk of medical personnel, and the demand for the automation of the medical distribution service system. A medical intelligent delivery robot system based on embedded technology has been designed. The staff can provide the door number room that the robot should deliver at the initial pharmacy location. After automatic detection of drug cargo loading, the robot will complete a series of delivery tasks including identifying the door number of the delivery ward, path planning, parking and unloading. The system uses the STM32 high-performance ARM chip as the control core and is equipped with the K210 embedded platform with KPU neural network operator for house number recognition. It uses OPENMV4Plus open-source hardware for machine vision tracking and can accurately complete its own delivery tasks in complex industrial environments.