On October 24, Vivo held this year's video strategy conference. According to the habit of the previous year, this is basically the preview of the video part of the Vivo X90 series and the self developed ISP of Vivo V2. In addition, vivo claims that it has more than 1000 photographic R&D teams and 1200+image technology patents.
Summary of provincial flow:
(X90 series) will adopt a new generation of CMOS, and its photosensitivity is 77% higher than GNV (1/1.3 inch level), almost named Sony IMX989 (1 inch+)
The telephoto will have an optical supersection algorithm to recover 35% of the definition information, and the resolution power will be improved by 64% by more than 5 times
Zeiss equivalent 50 mm 2 times portrait, the main body segmentation claimed that the IOU reached 96.15%, and the front micro slit double soft light (should be S series?)
However, the next generation of vivo self-developed ISP (the final name is unknown, and it is tentatively called vivo V2), instead of AI-ISP architecture, customizes 10bit MAC circuits. The 10bit computing and reasoning delay is reduced by 96% at most compared with the traditional NPU, and the energy efficiency ratio is improved by 200% at most.
The on-chip memory is upgraded to DDR Less on-chip near memory, with a peak throughput of 1.3 trillion bits per second.
The DLA (i.e. Deep Learning Accelerator) of the AI computing unit has shared SRAM, with 16.3 TOPS/W per watt (the Mariana X of TSMC's 6nm process is 11.6 TOPS/W, with a peak value of 18 TOPS).
The image processing unit has improved the effects of algorithms such as AI-NR noise reduction, HDR image tone fusion, MEMC frame insertion, and claimed that the dynamic range of night scene video has been increased by 4EV at most, and the noise reduction effect has been increased by 20%.
In terms of image algorithm and effect, compared with X80, the accuracy of white balance is increased by 8%, and the intelligent white plus black subtraction (exposure) is increased by 14%. ISO up to 102400, improving the night view by 100%. Compared with IMX866, the signal-to-noise ratio is increased by 20%, and the color restoration ability is increased by 15%. In terms of video, BT2020 color space is supported, with 10bit LOG+3D LUTs.
Computational Photography and Algorithm
Vivo divides its image technology into seven parts: optical perception system, color restoration engine, ultra clear image quality engine, computing acceleration engine, environmental understanding technology, super moving human image system, and sky night view system.
[Optical perception system]
Optical perception system includes optical module and coating technology. For example, VCS biomimetic spectrum technology can improve the sensor QE spectrum, improve the signal-to-noise ratio and color restoration ability, make the original information received by the sensor constantly close to the human eye, and create a good imaging foundation.
[Zhencai Restoration Engine]
The main supporting technologies are intelligent AWB and intelligent white plus black subtraction, which can improve the accuracy of white balance and the exposure accuracy of large areas of "white" and "black" scenes, and optimize the color performance and tone perception of vivo to the extreme.
[Ultra clear image quality engine]
The typical technology of the ultra clear image quality engine is the optical hyper division algorithm. Through modeling and analyzing the defects of the lens, about 35% of the lost information was recovered. The optical super division algorithm is the starting point for achieving ultra clear image quality. The engine also has Denoise, Demosaic, Debrur and other modules, which are coupled to train, forming a systematic AI image quality scheme - ultra clear image quality engine. It can realize multi frame image quality reconstruction to achieve better image quality and wider dynamic range.
[Computational Force Acceleration Engine]
Computational force acceleration engine is a set of software hardware combination acceleration scheme. Vivo has designed and developed CV heterogeneous acceleration engine and vDNN deep learning acceleration engine based on self-developed chip and platform SOC. It is an acceleration engine for traditional CV algorithm and AI algorithm respectively, which can support heterogeneous acceleration of multi-core combination and achieve multi information collaboration and high-speed processing.
Environmental Understanding
Environment understanding will recognize the scene, such as color temperature and brightness detection technology, motion detection technology, information extraction based on the environment, assist 3A and other modules to make the best shooting judgment (3A technology, namely, auto focus (AF), auto exposure (AE) and automatic white balance (AWB)).
[Super moving image system]
Three core technical modules of the super moving portrait system: portrait understanding, portrait beautification, and portrait atmosphere.
The first is the understanding of portraits - to understand the information of portraits like a photographer. Vivo's "Portrait Understanding Technology" will, like professional graphic artists, perform partition detection, extract the semantic focus and contour details of the face, and establish the key face coordinates of up to 103 feature points. The point accuracy error is less than 2 pixels. At the same time, the key points of skin, gestures and limbs in the picture will be detected, and the accuracy of the subject segmentation IOU has reached 96.15%, providing fine processing capability for static portrait remodeling and dynamic portrait capturing.
The second step is to beautify the portrait - beautify the micron level portrait details. With millions of materials of different ages, genders and scenes, after repeated machine learning and growth, Vivo launched a new and upgraded micron level skin rejuvenation technology, which makes the skin look more natural and healthy, and also better retains the three-dimensional sense of the five senses.
The third step is the portrait atmosphere - to create a unique atmosphere. The "portrait atmosphere fusion" technology, based on the semantic understanding of the portrait, integrates people and the scene at the levels of tone, color, brightness, and virtualization to achieve the harmonious and natural effect of people and the scene.
[Sky Night View System]
AI upgrade was carried out, and three new models were trained based on millions of night pictures: super photosensitive model, full segmentation semantic model and color enhancement tone model. Hypersensitivity model: the photosensitivity is increased by 100% at most, and ISO can support up to 102400; The combination of full segmentation semantic model and color enhancement tone model realizes adaptive tone and color adjustment.
Follow our Weibo @ Love Computer
Follow our WeChat official account: playphone
Of course, we also pay attention to our Bilibili account: love computer
