KAWASAKI, JAPAN: Fujitsu, a provider of information technology and services, announces the development of a new technology which generates synthesized 3D panoramic screens and provides them to remote supporting operators; enabling them to fully understand work site conditions and provide precise, omnidirectional instructions to field workers.

Fujitsu Laboratories has developed 3D image synthesis technology that uses images captured by a worker-side camera to generate a 3D panoramic image to convey the entire scene of a field site in a way that is easy to grasp. It has also developed user interface and work instruction AR presentation technology that, by using camera images and sensors to estimate a worker's position in real time, enables precise instructions to be conveyed directly to the field site from a remote location.

Challenges faced by the technology

While there are technologies that capture still images of the field site with a smart device's built-in camera, transmit them to the remote support operator at a remote location, and enable the field worker to then receive instructions. The problem is that the narrow field of view of most cameras makes it difficult for the remote support operator to get a good grasp of overall conditions in the field. On the other hand, when transmitting live video, blurred images caused by the worker's irregular movements can leave remote support operators feeling dizzy.

Also, as these technologies cannot directly convey the worker's position and orientation within the workenvironment to the remote support operator, the two of them need to interrupt the task at hand to discuss such issues as the place being addressed in the instructions and the direction in which the worker should be looking. This then requires a discussion which takes time and reduces efficiency. The technology features:

Online 3D image synthesis technology for field site images

This technology extracts a number of characteristic points in a scene from multiple images shot at the worksite to tracks the camera's position and orientation in real time, and continuously estimates the worker's position and orientation. Using the estimate results, clean images that are not blurred are selected and placed at the appropriate spot and orientation in the 3D scene, and then transmitted to the remote support operator. On the remote support operator side, camera parameters such as focal length and image center are used to lay out the images in three-dimensional space in real time, with image sizes adjusted to take into account the distance between the camera and the characteristic points. By sequentially placing each selected image in three-dimensional space, it immediately and accurately displays a 3D image with a strong sense of presence that captures the entire scene of a work site over a wide scope.

 Worker activity monitoring and AR presentation technology for work guidance

In order to faithfully reproduce the three-dimensional spatial structure of the actual worksite using a generated 3D image, this technology incorporates information from multiple sensors to make a highly precise estimate of the worker's position and orientation, which is used to depict a virtual scene that places the worker as a 3D model in a synthesized panoramic image. This allows the remote support operator to view the scene from any desired perspective and to monitor the worker's actions. In addition, the remote support operator can give guidance in the form of pointers or notes placed in the 3D image. This guidance is immediately conveyed to the worker's smart device and is presented as AR information in real time, corrected for the worker's position and orientation. Furthermore, if that AR information is outside of the worker's visual scope, the worker can be directed to that area, which improves the effectiveness of remote support.

This technology allows the worker to receive precise guidance from the remote support operator and enables less-experienced workers to expand their scope of operations. In addition skilled workers who may find it difficult to work in the field can effectively utilize their skills, enabling improved flexibility in leveraging the talents of employees.