Is AV1’s FGS Suboptimal?

AV1 is of particular interest to the industry because it’s royalty-free; it offers better compression efficiency and takes up less bandwidth than HEVC; it suffers fewer instances of buffering and has improved video quality. However, we’re primarily interested in the codec’s film grain synthesis, because as things stand now, film grain just looks like trash on video streaming platforms. Netflix has encoded something like 30% of its library to AV1 but has yet to stream any HDR content with the codec, and for all we know, it never will, because it appears that Dolby is standing firmly behind HEVC. Does it really matter, though? According to one study, AV1’s film grain synthesis is an admirable idea but in subjective tests, it appears to be suboptimal. The following excerpts are from A Subjective Study of Film Grain Synthesis for the Preservation of Creative Intent, authored by Jatin Sapra, Kai Zeng and Hojatollah Yeganeh and presented at IBC 2023. Follow the link to learn about the design of the test, its findings and conclusion.

ABSTRACT

Despite the advancements in digital cinematography, numerous artists and filmmakers still adore the look and feel of the content that is shot on film rolls. Specifically, they believe in true film grain as a signature of motion pictures and thus they treat grain as a key part of their artistic intent. The natural randomness of the true film grain comes from the crystallisation of silver halide when exposed to the light, and this natural randomness of true film grain is what fascinates content creators. However, content distributors like OTT providers and streamers always have trouble with such a high entropy signal since randomness possesses challenges to compression. Content distributors have limited bandwidth; they always try to squeeze videos into the pipes as much as possible. A clever and well thought approach to cope with grainy content is to remove the grain at the source side and then synthesise the grain after decoding the compressed videos. Recently developed codecs such as AV1 and VVC provide end-to-end solutions to achieve this goal; however, the faithfulness of the grain with respect to the creative intent is subject to thorough validation and deep investigation.

We believe that while the proposed framework is technically sound, without looking at the problem from a perceptual video quality point of view, the synthesised film grain will likely not satisfy film makers and content creators’ pursuit for the look and feel they intend to convey. To support this hypothesis, we have conducted a subjective study using content with film grain. In order to create different Hypothetical Reference Circuits (HRCs), standard film grain synthesis techniques like auto-regression models were used to produce different levels of grain with AV1 codec. The subjective data proves that there is still a big gap in the proposed models by the available codec standards.

INTRODUCTION

Film grain is a characteristic texture that appears in traditional film photography, caused by the random distribution of silver halide crystals in the emulsion layer of the film. It is a result of the light-sensitive chemicals on the film reacting to the light that passes through the camera lens during filming (1). This texture can vary in size and intensity, depending on the type of film used, the lighting conditions during filming, and the camera settings. Film grain can be an important visual element for Hollywood directors as it can contribute to the overall look and feel of a film. It can add a sense of texture, depth, and authenticity to the image, and can also help to create a certain mood or atmosphere. In addition to its aesthetic qualities, film grain can also be used strategically by directors to control the brightness and contrast of an image. By adjusting the amount of grain present in the image, a director can subtly alter the look and feel of the scene, enhancing certain elements or drawing attention to specific areas of the frame.

To the best of our knowledge, there is no publicly available dataset that is dedicated to study the preservation of film grain. The purpose of this study was twofold: firstly, to accumulate a library of compressed videos with various levels of artificially generated film grain that could be used as a foundation for developing a texture similarity measure, and secondly, to comprehensively investigate the components of the AV1 film grain synthesis pipeline and their impact on the final decoded frames with the inclusion of film grain. This research was crucial because AV1 was the only available pipeline designed to retain film grain from the original content, and its film grain synthesis feature was met with minimal satisfaction from filmmakers and the video industry, who doubted its capability to conserve creative intent.