Hi3751V900 MEMC: HiSilicon's Ace in the Hole

[ShangHai, China, May 12, 2020] HiSilicon recently released its latest 8K SoC Hi3751V900 featuring 8K MEMC, which provides users with a newly optimized 8K viewing experience. Proprietary motion estimation and motion compensation (MEMC) technology facilitates a high frame rate, which in turn results in smoother video graphics.

The frame rate is measured in frames per second (fps). Mainstream films tend to feature a standard frame rate of 24 fps, whereas films in the past were often shot anywhere between 16 fps and 24 fps, and played between 22 fps and 26 fps. A typical example is the comedy-drama film Chaplin, which gives the impression that it is being played fast forward, as the director would use frame rate changes to convey different emotions.

When sound films were first launched in 1926, it was found that viewers were more sensitive to audio changes, as opposed to changes to the film frame rate. As most silent films were played at 22 to 26 fps, the intermediate 24 fps value has resulted in the standardized 35 mm sound films that have dominated since about 1930.

Frame rate inconsistency between films and TV programs results in judder

TV series and variety shows are often played at 25 fps or 30 fps. Since the mainstream TV standards are PAL (50 Hz) and NTSC (60 Hz), if the shooting frame rate is 25 fps or 30 fps, which is an integer multiple of the TV scanning frequency, the viewer will not experience any sense of judder when watching TV. However, motion discontinuity or judder can be felt in films or American dramas (typically 24 fps) that are played on TV.

Mainstream consumer-grade display devices (displays, TVs, and projectors) have a typical refresh frequency of 60 Hz, and a few high-end products feature 120 Hz and 144 Hz panels. When a 24 Hz program is played at a 60 Hz refresh rate (which is not an integer multiple of 24 Hz), a 3:2 pull down mode must be used for frame repetition. That is, frame 1 is repeated three times, frame 2 is repeated two times, frame 3 is repeated three times, and frame 4 is repeated two times, and so on. This ensures that the 24 fps video frames are evenly allocated to the 60 Hz playback device. In addition, response times for display devices are limited, which can also lead to motion discontinuity and judder.

Viewing difference in films and TVs

Judder is less likely to occur in films. This is because the projector has a refresh rate of 24 Hz, which means that it does not need to be pulled up or pulled down. That's why the viewing experience in a movie theater is better than that on a computer or TV. Only a few films have been shot at a 48 fps or higher frame rate; these include The Hobbit (48 fps) and Billy Lynn's Long Halftime Walk (120 fps).

Actual trajectory of a football

Football trajectory captured with a 24 fps video camera

Football seen on TV

Actual football trajectory on TV

The above pictures show how the actual trajectory of a football differs from that which is seen on TV. This is even more clear in the following overlay picture:

Motion discontinuity and judder of the football

More noticeable judder in larger displays

It's been determined that the judder of footballs is less obvious on small screens, such as mobile phones or tablets, as opposed to 55-inch or 75-inch large-screen TVs. Since motion discontinuity is magnified on larger screens, people who are sensitive to judder may experience dizziness.

MEMC technology is just the ticket

How then, can judder be eliminated? MEMC technology offers perhaps the best answer yet.

It's capable of predicting an object's motion track and minimizing judder, by inserting an intermediate motion state based on the predicted result.

This technology detects the motion track (motion vector) of an object in an image, and inserts compensation frames to improve the fluidity of the graphics.

There's a handy trick for getting rid of jello effects

Because all new frames are calculated through motion prediction, this solution is prone to prediction error, causing side effects, such as jello effects or halos.

The figures on the left are samples from a TV MEMC product that was launched in China. The figures on the right are samples from the latest HiSilicon MEMC algorithm, which minimizes halo effects by accurately calculating human motion tracks.

Another issue is the failure of the MEMC algorithm to pick up small moving objects.

Why is the football in the interpolated frame missing? In most MEMC modules, motion correlation between objects in adjacent frames is calculated by the block matching method, that is, an image is cut into blocks of 16 x 16 or 32 x 32 pixels. Then interpolation is performed based on the motion vector of the object. The algorithm performs the calculation based on the image blocks, rather than the number of pixels (due to the large numbers involved). As a result, the motion vector estimation for small objects is prone to inaccuracy, and small objects may be missing or overlapping.

HiSilicon's SoC algorithm brings back the "missing football"

In order to improve the motion effects for small objects, HiSilicon's SoC algorithm calculates the motion vector of image blocks based on the current frame rate. This results in dramatically improved accuracy for motion vector calculations of small objects. As shown the third image above, the "missing" football appears on screen.

It's also clear to see that videos are played more smoothly on mobile phones than on TVs. This is because most network video streams have a frame rate of 24, 25, or 30 fps. On the mobile phone screen, the moving distance from location A to location B is far less than that on a TV screen within the same period of time. In general, the size of an LCD TV display is 8 to 10 times larger that on a mobile phone. Therefore, MEMC technology is essential for eliminating judder for low-frame-rate videos played on TVs.

Pioneering integration of 8K MEMC into the 8K SoC

The resolution (pixels) on 8K TVs is four times that on 4K TVs. The size of mainstream 8K TVs ranges from 70–90 inches, and there are even 110-inch models favored by some users. Due to the sheer size of the displays, MEMC is a key technology that has a direct effect on the overall TV viewing experience.

The most recent HiSilicon 8K SoC Hi3751V900 is the first processor to integrate 8K MEMC into an 8K SoC. The MEMC works in concert with the 8K picture quality (PQ) module, producing ultra-immersive 8K graphics. The 8K MEMC PQ effects facilitated by the HiSilicon 8K SoC, is no question, a radically new paradigm, and a game-changer in this field.