Behind the collective self-developed ISP chip of Xiaomi OV, the truth is not simple

Some companies have been ISPs for more than ten years, but they have not entered the first-class level in the industry.

In the past ten years, Chinese mobile phone manufacturers have used SoC chips from Qualcomm and MediaTek to promote the popularity of smartphones, and they have also ranked among the forefront of the global mobile phone industry. However, with the intensification of competition in the mobile phone market and consumers’ higher pursuit of mobile phone experience, general-purpose mobile phone SoCs have become a bottleneck for mobile phone giants to enhance their competitiveness.

As a result, Xiaomi, OPPO, and VIVO have embarked on the road of self-developed chips one after another. Their ultimate goal is to develop self-developed SoC chips, but now they focus on the research and development of ISP (Image Signal Processor, image signal processor). Why do mobile phone giants choose to start with self-developed ISP chips? Consumer demand, the technical accumulation of mobile phone manufacturers, the technical characteristics of ISPs, and the future trends of the mobile phone market are all key reasons, which are also the questions that this article will answer in detail.

As for whether the mobile phone giants can successfully develop their own chips? A number of industry veterans have told Lei that this lies in the determination of mobile phone manufacturers.

The mobile phone giant “trapped” in the chip has embarked on the road of self-research

In December 2013, China started the 4G era. Three years later, in 2016, domestic mobile phones began to rise in an all-round way. In this year, the market shares of Apple and Samsung in China began to decline. The market shares of Huawei, OPPO and VIVO increased rapidly, and Xiaomi also quickly occupied the market with its high cost performance.

According to the data of the two major market research institutions Counterpoint and canalys in the second quarter of 2021, Xiaomi, OPPO, and VIVO ranked among the world’s top five mobile phone manufacturers, ranking second, third and fifth respectively, with a combined market share of more than 35%. Lei Jun, the founder, chairman and CEO of Xiaomi, has recently released a rhetorical statement, “In five to 10 years, Xiaomi will become the world’s number one mobile phone company.”

The top five mobile phone manufacturers in the world have one thing in common. The major improvements in the functions of each generation of mobile phones are closely related to the characteristics of the latest SoCs of Qualcomm and MediaTek. This also makes them face a common challenge. In the sluggish mobile phone market, the difference in performance and experience is constrained by chips.

Self-developed chips are a proven feasible path. Apple, Huawei and Samsung have achieved differentiation through self-developed chips and successfully occupied the high-end mobile phone market. In fact, Xiaomi, who wanted to hit the world’s first, embarked on the road of self-development of chips as early as 2014, and released the first mobile phone SoC Surging S1 in 2017, but the experience was not satisfactory.

An industry insider told Lei “A very critical reason for the failure of Pengbai S1 is that it is too eager to achieve success.”

The development of SoC was frustrated, and Xiaomi developed an ISP chip Surging C1 in 2019. This chip was unveiled at the same time as the Xiaomi folding screen mobile phone MIX FOLD released in March 2021, which can achieve better 3A (autofocus, auto white balance) , automatic exposure) processing, self-developed ISP + self-developed algorithm has also become a major selling point of this flagship phone.

Here, MIX FOLD is already on sale, and there OPPO and VIVO are also moving fast on the road of self-developed chips.

In September 2019, Hu Boshan, executive vice president of VIVO, revealed that he began to consider deeply participating in the design of SoC at the beginning of 2018, and released the SoC chip Exynos 980 jointly developed with Samsung at the end of that year. Although it has been deeply involved in SoC research and development, self-development is VIVO’s goal. Jiemian News reported in July that VIVO’s first self-developed chip codenamed “Yueying” will be launched soon, which may also be an ISP chip, and will be launched on the X70 series of mobile phones.

OPPO announced the progress of self-developed chips relatively late. In February 2020, the special assistant to the OPPO CEO published an internal article proposing three major plans for chips, software development, and cloud services. The chip project is called the “Mariana Plan”. A source revealed that the first chip developed by OPPO is also an ISP chip, which will be launched on the Find X4 series of mobile phones that will be launched in early 2022.

This means that the three mobile phone giants “trapped” in third-party mobile phone SoCs will enter a new round of chip competition next year. But why the ISP chip?

Four reasons for self-developed ISP

Since the growth rate of the global mobile phone market began to decline in 2017, the competition among mobile phone manufacturers has become increasingly fierce. The 2018 mobile phone industry marketing white paper released by Toutiao pointed out that in 2018, the competition points of mobile phone function marketing tend to be concentrated. There are 8 AI features, 5 screen types, 4 unlocking methods and 4 color matching. AI photography and photography are mostly.

In the following three years, the mobile phone industry’s attention to cameras has only increased. The 2020 Tencent Mobile Industry Insights White Paper pointed out that consumers’ attention to cameras, uses and batteries has increased significantly, and the growth rate of cameras’ attention is second only to the operating system. .

“The imaging system is actually a very long chain, including front-end sensors, lenses, motors, ISP chips, the core component of image processing, and back-end calibration and tuning processes.” Chai Weihua, senior product manager of Arm Technology, told Lei Feng Net said.

“The same lens and sensor will have different effects after passing through different ISPs.” Zhang Xing, head of ISP and system architect of Aixin Technology, said at the same time, “The value of ISP is to improve image quality. But ISP is a non-standardized chip IP. , there are objective performance indicators, such as the number of cameras, resolution frame rate, bandwidth, power consumption, delay, etc. There are also image evaluation indicators that are difficult to fully objective, including brightness indicators (dynamic range, contrast, exposure accuracy and speed). etc.), color-related indicators (saturation, color accuracy, pleasing to the eye, etc.), sharpness, etc. From experience, there are about 40-60 kinds of subjective and objective tests for images.” Love Core Technology It is an AI vision chip startup company, and AI ISP is one of its self-developed technologies.

Although it is difficult to have a unified standard for good picture quality, there are also some objective indicators of brightness, color and clarity, such as a standard color model to measure color difference. “After the objective indicators reach a certain level, customers usually adjust the subjective indicators according to different needs. One of the characteristics of ISP is that there are many adjustable parameters, and engineers can adjust the designed ISP hardware parameters to adapt to different environments. One A good general-purpose ISP should provide sufficient flexibility and scalability.” Chai Weihua pointed out.

Image function is the focus of consumers, and ISP is the key to achieve differentiation, and the shipment of mobile phones is huge. If the self-developed chip is successful, it can eventually be transformed into a cost advantage of the whole machine. This is why mobile phone giants choose to develop their own ISP. The initial motivation is also the first reason.

“Huawei can use its self-developed NPU+ISP to achieve unique AI camera functions. In contrast, mobile phone manufacturers using Qualcomm and MediaTek SoCs will have limited space to play.” said Yu Guang, investment director of Yaotu Capital, ” Due to the long period of hardware iteration, the needs of mobile phone manufacturers cannot be quickly met, especially the differentiated needs, so self-developed ISP chips to achieve close integration of hardware and software is a good choice.”

Zhang Xing further explained that once the ISP design is completed, the parameters that can be adjusted are roughly determined, but these parameters are limited. If you want to provide adjustable parameters for debugging all styles, the chip area will be too large to bear. There needs to be a balance in between.

Mobile phone manufacturers who want to achieve differentiation have years of accumulation in the field of imaging systems, and they have the confidence to develop their own ISPs. This is the second reason why mobile phone manufacturers choose to develop their own ISPs.

The feasibility of developing ISP chips is the key for mobile phone manufacturers to firmly enter the road of self-developed chips, and it is also the third reason. Chen Feng, founder and CEO of Yizhi Electronics, said: “Because ISP chips are more dedicated, the ecological constraints of developing ISPs are much smaller. There are three main challenges in developing ISPs: because image algorithms have new methods every year, the technology is constantly changing. Iteration; in addition, the effect of the image needs to meet the objective indicators of the image test, but also need to work hard on the adjustability of the subjective effect; finally, it needs to adapt to different scenarios, and the ISP must be able to adaptively adjust the parameters in each environment. .”

The widest moat of chip companies is ecology. Since the ecological restrictions of ISP chips are relatively small, and mobile phone manufacturers have rich experience in algorithms and understanding of scene requirements, and each has its own characteristics, especially in scene adaptation. In terms of mobile phone manufacturers, they have rich experience.

The development of ISP by mobile phone manufacturers has the long-term value of adapting to the future competition of video photography + AI functions. This is the fourth reason.

“Mobile phone manufacturers have already enjoyed the benefits of AI algorithms in taking pictures, including functions such as super night scenes and AI beauty. From the perspective of the trend, in the next few years, the most popular features of AI photography will be pushed from pictures to videos. An efficient AI ISP chip is needed, as well as a better combination of AI algorithms and traditional ISPs.” Zhang Xing believes.

Chai Weihua also believes: “With the development of the demand for short videos, mobile phones are also paying more and more attention to the performance of video shooting. In addition to the requirements of image quality for video streaming, a very important challenge for ISP is real-time performance. Real-time processing of 4K or even The massive data of 8K video presents a perfect image effect, the processor performance, excellent algorithm, storage bandwidth and system latency are all challenges.”

AI is the key to improving photo and video performance. Chen Feng pointed out that there are currently two main forms of AI and ISP combination. One is ISP For AI. This combination method is mainly used for products with detection or recognition functions. The goal is to ensure the accuracy of detection or recognition, such as face access control. and assisted driving. The other is AI For ISP. This combination is to replace one or more image enhancement algorithms in the ISP with AI, which has better effects in aspects including HDR (high dynamic range), night scene noise reduction, super resolution, etc.

“In recent years, the mobile phone industry has found that with the support of the current AI computing power, the AI ​​algorithm has surpassed the effect of the traditional ISP algorithm. Therefore, the current mainstream trend is to replace a traditional algorithm with an AI algorithm, or to combine the AI ​​algorithm with the traditional algorithm. Zhang Xing further said, “To give full play to the value of the combination of NPU computing power and ISP, and realize AI ISP in the true sense, it is necessary to combine the two IPs at the beginning of the design. If it is a purchased IP, the two It is very difficult to realize good functions in combination with the above-mentioned, and even if it is realized, there will be a lot of power consumption and waste of chip area.”

Chen Feng specifically pointed out, “In the field of video applications, the AI ​​noise reduction algorithm with slightly better effect requires more than 10Tops of computing power to process [email protected] video in real time. At present, this application scheme is still relatively rare.”

Overall, consumers’ pursuit of better camera and photography functions on mobile phones is the driving force. On the premise that mobile phone manufacturers have accumulated mobile phone imaging technology for many years, self-developed ISP chips have become an achievable goal for mobile phone manufacturers. The ability to extend from photography to video can enhance the competitiveness of mobile phone manufacturers now and in the future. Self-developed ISP chips have naturally become the unanimous choice of mobile phone manufacturers.

The key to high-end success or failure lies in “determination”

The next focus is, what differences can self-developed ISP chips bring? The Xiaomi Pengpai C1 that has been released, the official statement is that the self-developed ISP improves the digital signal processing efficiency by 100%, and can perform more refined and advanced 3A processing.

A senior person in the industry said: “The surging C1 is actually a co-processor, or Pre-ISP. It is relatively easy to process image processing before the SoC ISP, and can assist in processing some applications in specific scenarios. However, , because a single chip is added outside the SoC, both cost and power consumption will increase.”

As for whether the actual user experience has been significantly improved in speed and quality, more user feedback is needed.

The ISP team is very important for the self-developed ISP of mobile phone manufacturers to truly transform from a selling point into a differentiated advantage. “The development of ISP has been progressing, and it is far from a mature stage. To design a competitive ISP chip, it depends on the maturity of the team itself. Some companies have been ISPs for more than ten years, but they have not entered the first-class level in the industry. Of course, this also has a lot to do with a company’s cognition and investment in ISP.” Zhang Xing said.

Industry veterans also said, “The key to the success or failure of mobile phone manufacturers’ self-developed chips lies in their determination to invest, especially SoCs.”

However, the entire domestic chip industry lacks talents, and it is a huge challenge to form an experienced ISP team. Chai Weihua said: “ISP research and development involves algorithms, circuit design, system integration, device adjustment, image quality optimization, etc. Therefore, the corresponding talent requirements are not only specialized, but also broad, in addition to the need to master mathematics, physics, computer, Knowledge of integrated circuits also requires knowledge of optics, color and other disciplines. The domestic accumulation in this field is still relatively limited, so the relevant talents are relatively lacking.”

“The domestic talents with rich experience in the field of mobile ISP are relatively limited, but now Huawei HiSilicon is unstable, and Zhanrui also has some related talents, which are opportunities for other companies to acquire talents.” A senior person in the industry said.

“Even if a competitive ISP can be designed. It will take time for consumers to understand the self-developed chips of mobile phone manufacturers. I think it will take at least 2-3 generations of iterations.” An analyst in the chip industry commented on Lei said, “I think Xiaomi and OPPO’s self-developed chips will have more advantages from the perspective of cash flow, while VIVO will have greater challenges.”

The analyst also said: “The self-developed ISP is mainly used for high-end mobile phones, so the impact on Qualcomm may be greater.”

Interestingly, MediaTek released the Dimensity 5G open architecture platform in July this year. Based on its flagship SoC, it provides mobile phone manufacturers with open resources closer to the bottom, including cameras, displays, graphics, AI processing units, sensors and wireless connections. subsystem. This can be understood not only as complying with the trend of mobile phone manufacturers’ self-developed chips to pursue differentiation, but also as a measure to prevent mobile phone manufacturers from self-developed SoCs.

In any case, self-developed chips are a must for mobile phone manufacturers to stand at the top of the mobile phone market pyramid, but this road is full of challenges in terms of time, technology and capital investment. Both Qualcomm and MediaTek have accumulated decades of experience in the field of mobile phone SoCs. , Apple and Huawei have achieved their current achievements after more than ten years of hard work and iteration, while Xiaomi, OPPO, and VIVO have just begun.

Zuo Kunlong, the architect of Xiaomi’s ISP chip, revealed in the documentary “The Cornerstone of a Powerful Country” that ISP is just the starting point for Xiaomi’s self-developed chips, and its SoC is also under development.

It is reported that both Xiaomi and OPPO are vigorously recruiting chip talents in Shanghai. It is reported that the number of OPPO chip teams has exceeded 1,000. Are they going to directly develop a SoC chip or will they develop other single chips? The above-mentioned analysts believe that power management chips are a good choice, because fast charging is also a major selling point of mobile phones.

Yu Guang pointed out that there are already some good start-up companies in the power management chip market, and mobile phone giants have also invested in many companies in this field.

After the excitement, who will be the next high-end mobile phone player with self-developed SoC?

Author | Bao Yonggang

Editor | Li Shuaifei

Author: Yoyokuo