Manufacturers of mobile phones have gone to extremes to achieve higher image quality and larger screen sizes, despite space limitations. Demand for better image quality in particular is relentless and the technology has already reached the point where some models can display high-definition (HD) images. The main display sizes used at present in mobile phones are WQVGA (240x400 pixels) and WVGA (800x480 pixels). However, with advances in AV equipment such as thin-panel televisions and with HD content services now becoming available on the Internet, we are entering an era in which mobile phones soon are destined to have HD or full-HD video capabilities.

In digital cameras as recently as 2007, 5Mpixels was considered the state-of-the-art for still images from mainstream products. Yet by 2009, resolution capabilities had grown to 8 to 10Mpixels. Further, innovative convenience features such as scene detection and face detection had been introduced, generating considerable consumer interest. The latest indications are that picture resolutions of 16Mpixels are likely to be reached in the near future.

In the area of 3D graphics capabilities of mobile devices in general, support for OpenGL ES 2.0 became common in 2008. Current forecasts indicate that the trend today is toward standards such as OpenVG and OpenMAX (see Figure 1).

Renesas has adhered to a solid technology roadmap for the SH-Mobile series of microcomputers that support the main CPUs in many different mobile products. In particular, we have worked steadily to keep this popular product line ahead of the movement to advanced graphics for mobile phones. Consider the SH-Mobile L3V2, for example. This graphics and GUI solution was developed to support VGA and WVGA displays just as the older QVGA and WQVGA standards were becoming mainstream. Its widespread adoption has clearly demonstrated the wisdom of keeping our aim focused just ahead of the market shift to larger screen sizes and better image quality.

Another example of our success in driving semiconductor technology for mobile devices is the SH-MobileHD1. When this application processor for the mobile phone industry was announced in April 2009, it was the first chip of its type that included support for full HD displays.

The SH-MobileHD1 application processor provides a single-chip implementation for full HD video processing, achieves lower power consumption due to internal dedicated DSPs for audio processing, and has a design that emphasizes interoperation with digital home appliances. Full HD processing is made possible by the 500MHz maximum operating frequency, which is twice the speed of previous Renesas models. The faster clock speed improves the performance of the video processing unit (VPU) IP module six-fold compared to our SH-Mobile L3V2. Also, we manufacture the device using a 65nm process, which further reduces power consumption and allows us to combine the microcomputer with 512Mbit of SDRAM in a small, 10mm x 11mm SiP (system-in-package).

In order to handle full HD video, the architecture for the SH-MobileHD1 obtained more data-transfer speed by expanding the internal bus width. The microcomputer can both record and play back full HD video at 30fps using the H.264/MPEG-4 AVC format (abbreviated below as H.264). To ease integration with external systems, the device also incorporates an industry-standard MIPI-CSI2 (Mobile Industry Processor Interface-Camera Serial Interface 2) interface for inputting video images from a camera module.

The new microcomputer incorporates two 24-bit dedicated audio DSPs that extend the audio functions to complement support for full HD video. These DSPs simultaneously reduce the CPU audio processing load and lower power consumption. Each of them is more than capable of handling stereo audio processing such as Dolby Digital and the AAC-LC (Advanced Audio Coding - Low Complexity) high-quality audio compression standard on its own. Therefore, system engineers can use one DSP for this purpose and the other one for parallel implementation of a sampling rate converter (SRC) or other functions that enhance audio quality, such as an equalizer.

With regard to interoperation with digital home appliances, the system-engineering goal is the exchange of full HD video with external devices. The SH-MobileHD1 incorporates an interface for connecting to an HDMI transmitter LSI to allow full HD video captured by the mobile phone's camera to be played back over a connection to a digital home appliance. Connection to SD memory cards and wireless LAN modules is also enabled, and the microcomputer supports multiplexing of the MPEG2-TS (transport system) and MP4 formats.

Accompanying the introduction of the SH-MobileHD1 was the announcement that Renesas, in cooperation with the System Development Laboratory at Hitachi, had developed and produced the HD Video Middleware. This middleware significantly improves the efficiency of developing applications such as controlling the VPU on the SH-MobileHD1 and implementing recording and playback using full HD video in H.264 format with AAC-LC and Dolby Digital sound. For instance, the middleware simplifies the task of enhancing applications by adding functions such as those described above for integrating high-quality audio playback with full HD video or using MPEG2-TS for interoperation with digital home appliances.

Because Renesas recognizes that the trend in mobile phones is toward higher display resolutions, we have developed a super-resolution technology for boosting the quality of enlarged images. It is now being offered under the name "SuperH Super-Resolution Software." (See the related story, "Technology Update: Correcting Quality Degradation in Enlarged Images."). In simple terms, the super-resolution technology is a correction technique for preventing the coarseness and other image degradation that occurs when a small image is enlarged. The improvement in image quality is particularly noticeable when low-quality images captured with QVGA or VGA resolution, for example, are displayed on an HD or full HD screen.

Renesas intends to continue improving the performance of the microcomputers in the SH-Mobile series. This popular line of devices will be expanded and improved as we work to keep ahead of the advancing technology required for displaying images and implementing GUIs in mobile phones and other portable devices.

Correcting Quality Degradation in Enlarged Images

A software implementation of super-resolution technology for correcting quality degradation in enlarged images is now available from Renesas. It uses digital processing to generate a higher-resolution output signal from an input signal. It delivers clear images with less noise and with fine details that are crisp and vivid.

As high-resolution displays in mobile devices become more common, more instances are encountered in which lower-quality images, such as those that captured with sensors with QVGA resolution, for example, have to be displayed in an enlarged form. Unfortunately, this operation can lead to visible distortion and coarseness in the displayed image. Renesas anticipated that this unwanted side effect of higher display resolutions could be a cause of concern for users. Therefore, we created a solution to the problem by developing a super-resolution technology that prevents quality degradation: "SuperH Super-Resolution Software." It implements a technique that uses digital processing to generate a higher-resolution output signal from an input signal.

Generally, software can achieve this operation in two ways. Either it performs intra-frame processing, whereby each image frame is converted into a single high-resolution image, or it employs inter-frame processing in which one high-resolution image frame is generated from multiple input frames. The Renesas technology uses the former intra-frame method (see Figure 3). Various linear filters and non-linear processing techniques correct the pixels based on the image's characteristics. Thus, the correction method also works successfully on images that include text, computer graphics, or noise. Moreover, our software works effectively on still images as well as on video of different resolutions. The setup procedure for the super-resolution processing is a simple a step-by-step process.

As can be seen in the photographs below, the technique implemented by the SuperH Super-Resolution Software greatly enhances the perceived image quality of the leaves on the trees, the animal's pelt, and the shiny surfaces. Notice that the text in the photographs is also enlarged into a crisp image without blurring.

Because amplification of any noise is kept to a minimum by assessing each image frame in various different ways, the output noise produced by this technique is much less noticeable than the noise that appears on images enlarged using older processing methods such as sharpness enhancement. Our technology produces image quality that is clearer and has fine details that are crisp and vivid.

A major feature of the Renesas technology is that it is implemented as software. In the past, such complex image processing tended to be implemented as hardware due to the heavy load it would otherwise place on the mobile device's processor. Our implementation, however, succeeds in significantly reducing the processing load — to the extent that a CPU operating in the 250MHz range can easily handle it.

By utilizing these advantages, Renesas now offers this technology for a wide range of applications as part of our graphics solutions for mobile devices that use SH-Mobile series devices or other Renesas microcomputers that have the SH-4A core. Beyond the mobile sector, we anticipate that the technology will be used in products that incorporate video processing engines or our MX massively parallel processor with a matrix architecture.