HFR – The one UHD technology you rarely hear about
If you’ve been keeping up with Ultra HD TV technology, you may recall that there are essentially six pillars to it:Most of these we’ve got by now. You can go to an electronics store and buy a 4K TV with HDR, WCG, 10-bit color and a Dolby Atmos sound bar or AVR, and they’re not even expensive anymore. The one piece that’s missing? HFR, or High Frame Rate. It’s probably the least understood in terms of benefits and how it works – by consumers but likely also by the creative industry. Even science is still lacking. It’s probably the most controversial of all UHD technologies. Many misconceptions abound, so here’s an attempt to shed some light on what we know and what we don’t know.
- UHD or Ultra HD spatial resolution – 4K or 8K
- HDR or High Dynamic Range
- WCG or Wide Color Gamut
- Deep color resolution
- HFR or High-Frame Rate
- NGA or Next-Generation Audio
HFR for movies
What do we mean by HFR? That depends what we are talking about. When it’s about film, anything above 24fps (frames per second) will be called HFR. It’s not very common. The number of high-profile movies with HFR you can count on one or two hands. First we had Peter Jackson’s Hobbit trilogy, shot in 3D at the double frame rate of 48fps. Since then, we’ve had director Ang Lee taking things further with Billy Lynn’s Long Halftime Walk shot at 120fps and Gemini Man starring Will Smith shot at 120fps and in 3D. Latter movie was screened in different ways: 2D theaters showed it in 24fps; 3D theaters showed it at 60 or 120fps, depending on their capabilities. And with good reason: Ang Lee pursued high frame rate in order to overcome issues inherent to projection of 3D images at 24fps – strobing and flicker, just like Peter Jackson did. James Cameron has a slightly different approach but prefers to use 120fps in certain parts of 3D movies to avoid judder in shots that pan or have lateral movement across the frame. But HFR for movies is a very divisive technique. Some people love it, a majority of people seem (I don’t have any hard numbers here) to hate it. 24fps is sacred.
Director Ang Lee filming ‘Gemini Man’. Picture credit: Paramount
But why? Well, part of the reason is that’s what we’re used to. This video does a good job explaining why we’ve got 24fps, and why we’ve kept it. But if more pixels and more colors and more bits per color and more audio channels are good, why would more frames per second be bad? Doesn’t it add more realism? Yes, it does, and that’s exactly why it’s bad for movies. Realism is not the point – quite the contrary. It’s very similar to the discussion around analog, chemical film grain: Some people dislike it, but like 24fps frame rate it’s something our brains have been conditioned for since almost 100 years. Although most of us will not consciously notice it, our brains register it subconsciously and know we’re watching a proper film, a possibly epic story. It helps with our suspension of disbelief and puts our brains into ‘movie watching mode’ and immerses us, pulls is into the story. Some refer to 24fps as a ‘dream-like cadence’. Heightened realism takes away from that. It breaks the magic spell. As one moviegoer succinctly put it: “I didn't see Gandalf et al - instead I saw a load of actors dressed up in some odd costumes.” Billy Lynn and Gemini Man triggered many of the same type reactions, as you can see on social media. Some compare it to a home video shot on a GoPro or a smartphone, looking “hyper-real” or “like a videogame”; others refer to the “Soap Opera Effect” as it’s called (a bit of a misnomer), and the dreaded motion interpolation that their in-laws have enabled on their TV.Realism is not the pointMuch however is unknown about how our brains process frame rates and motion perception. More scientific research into this area would probably be justified.
Ian McKellen as Gandalf in ‘The Hobbit: The Battle of The Five Armies’. Picture credit: Warner Brothers
- How come we (most of us at least) subconsciously perceive 24fps frame rate without being bothered by judder?
- Are our brains really conditioned into seeing 24fps as ‘epic’ yet when we see 60i or 60p this mechanism shuts down and we go into ‘soap opera’ mode?
- And yet for videogames, which nowadays are also a lot about storytelling, High Frame Rate has evident benefits that gamers appreciate apparently. So do people who play videogames at high frame rates (120fps and above) perceive movie frame rates differently?
- Also, provided the Soap Opera Effect is real, there must be an Inverse Soap Opera Effect whereby TV content converted from 60fps to 24fps suddenly starts to look epic? What points to this is a common technique in sports news shows, where they cut the frame rate to 24, 25 or 30fps, crop the picture to get a wider aspect ratio (adding black bars at the top and bottom) and add dramatic music when they want to make a game summary look epic.
- Does frame rate matter for traditional cel animation movies, and if so, how?
- At what frame rate above 24fps does the magic stop working? 25? 30? 48? 60? Anecdotal evidence (the three Hobbit movies) suggests 48fps is already guaranteed to blow it, but where is the border?
- Although soap operas are never in 3D, 3D does not seem to help with suspension of disbelief or make something more epic. Perhaps even contrary. Is this because 3D adds realism which, like HFR and absence of grain, breaks the spell rather than sustaining it?
- Is this behavior learned? Would someone from another culture who’s never been to the cinema experience frame rates the same way?
Converging technologies
It’s important to bear in mind that Hollywood movies are still made and optimized for cinema. The technical capabilities of cinema guide movie production and as such many of the 'truths' in Hollywood are based on this system. Things could change if suddenly home entertainment becomes a much bigger market for Hollywood, which was actually already happening in 2018 and 2019, and perhaps much more dramatically so in 2020. Movie production optimized for TVs would most likely look different from film production optimized for cinema. After all, what works well at the cinema doesn’t necessarily work well on a TV set. Research by Dolby Labs found that higher brightness makes judder more apparent, as does higher contrast. What looks good in SDR judders too much in HDR, so colorists end up grading HDR darker to avoid this, which defeats the whole purpose. The study also showed that at 50 nits – the typical brightness with traditional theatrical projection – 24fps is the ideal frame rate whereas at 1000 nits – achieved on now fairly common and not too expensive HDR TVs – 32fps would be preferred. Although television came several decades later than cinema, the two technologies have been on parallel paths, going through many of the same innovations. Film went from silent movies to sound, from black & white to color, from mono to stereo to digital surround sound (Dolby, DTS, SDDS, initially all on optical film) with an ever-growing number of channels. These changes have been noticeable but followed an evolutionary approach. But more recently the cinema business has seen some innovations that are major technical changes but entirely behind the scenes and very subtle if at all noticeable to the viewer: from chemical film to digital projection, with movies distributed on HDDs in cartridges and soon possibly online. The next major step in cinema is probably going to be the most drastic one in a century: from projection on a silver screen to ‘direct view’ displays. Of course, no CRT, no plasma, no LCD, no OLED but real LEDs. These bring far greater brightness, contrast and dynamic range, and will bring cinema screens back on par with home cinema – where they are now essentially running behind on spatial resolution, dynamic range and brightness – only way bigger. It will also add the flexibility to use higher frame rates though whether this will be used is doubtful, given the above considerations and the generally conservative nature of the movie business. This switch will also bring new entrants to the market. Samsung, which has never been in the cinema projection business, has launched its ‘Onyx Cinema LED Technology’ – 34-feet (10-meter) diameter screens with true 4K 4096 x 2160 resolution. The first cinemas rolling this out since 2017 were Lotte Cinema World Tower in Seoul, Paragon Cineplex Theatre in Bangkok, Pacific Theatres Winnetka in Chatsworth, California, just north of Los Angeles, Pathé Beaugrenelle in Paris, Sambil Leganés in Madrid, and the Shoudu Cinema in Beijing. Samsung's Onyx Cinema LED direct view display
Size options for Samsung’s ‘The Wall’ Micro-LED TVs
HFR for other TV content
So if it is only for scripted, acted content that low frame rate matters, are higher frame rates better where realism matters i.e. nature documentaries and live sports? Evidence suggests so. HFR adds to the sense of ‘being there’ in a good way. But before we go into that, back to the definition of HFR. Broadcast TV comes in a range of resolutions now. The trend is upward but very slow. A related trend is that slowly but surely we seem to be getting rid of interlaced video, where ‘fields’ (half frames with only the odd picture lines or the even ones) are displayed successively. Sure, at 1080 HD resolution there’s still a lot of 50i and 60i content but at Ultra HD resolutions only Progressive Scan with full frames is permitted. Various frame rates are allowed (including fractional ones), but 50p and 60p are not considered HFR – they’re Standard Frame Rate. When organizations like the Ultra HD Forum speak of High Frame Rate they mean at least double that – 100 or 120fps and beyond. What drives the (very slow) move to higher frame rates? Is it a silly numbers race, like some would argue the move to higher spatial resolution (4K, 8K) became? Not quite. Even if we don’t need or want it for movies, there are definite upsides. First, why are we moving to higher screen resolutions? Just because TV makers can, and they see TVs with higher resolutions having higher margins? No, there’s more behind it than technology push. Since many decades, TVs are getting bigger and bigger. It’s a pretty constant trend, and the average diameter grows by about 1 inch per year across all territories, even if these averages vary from region to region. In the meantime, our viewing distance does not change much. Living rooms (also varying in average size geographically) did not get significantly bigger. That’s why we need more pixels. Now with higher resolutions, the risk of motion blur increases. With 8K, this is particularly visible. 8K sports content, like the Olympic Games, you probably do not want to watch at frame rates lower than 100fps. Although Japanese public broadcaster NHK has announced quite some time ago they’ll shoot and transmit many parts of the Olympics in 8K, they have not yet said at what frame rate.The difficulties with HFR
While shooting, recording and transmitting HFR may be relatively straightforward (arguably more so than HDR), there is a complication: How to achieve backward compatibility with Standard Frame Rate TV sets and transmission systems? At the moment there are two approaches to this, and DVB and ATSC solve this in different ways. Here it’s going to get a little more technical. What the two have in common: Both use a technique called temporal sublayering for backward compatibility of HFR with SFR. ATSC includes optional temporal filtering for enhancing the standard frame rate picture when temporal sublayering is used,HFR: DVB and ATSC - temporal sublayering
How does this work? In ATSC and DVB both, PID (program ID) = 0 is the SFR version, and PID = 1 is the HFR enhancement element, to be used along with PID 0 to reproduce the HFR version. In DVB, it actually wouldn’t matter which PID you viewed, they are just the odd and even frames, so each represents a half-frame rate feed, with just a slight timing offset. In ATSC, the frames are a bit different. The frames in PID 0 are a weighted sum of the odd and even frames of the HFR signal. The result is that the PID 0 content has an artificial motion blur. The HFR camera needs a 360 degree shutter (i.e., photons are being captured essentially 100% of the time; the camera doesn’t blink). The contents of the PID 1 frames are the weighted difference between the two signals. The trick here occurs in the receiver: As in DVB, if you don’t know better, show PID 0, you’ll get a usable SFR signal with full motion blur (depending on the weightings). If you do know better, you recover consecutive HFR frames by summing and differencing the two PIDs frames to reconstitute the original odd & even frames of the HFR. |
High Frame Rate is especially beneficial for sports content. Picture credit: Ultra HD Forum
HFR in practice
So can you go to a store and buy a HFR TV or monitor? One area where this will come in handy is gaming. Current gaming PCs as well as the upcoming PlayStation 5 and Xbox Series X, the ninth generation of consoles, are (going to be) capable of HFR output. PC/gaming monitors already cater to that. High Frame Rate is one of the few feature areas where monitors are ahead of TVs. Most don’t do a too impressive job in terms of high dynamic range, contrast, peak brightness, wide color gamut, resolution, etc. But while TVs currently don’t exceed 60fps (claims about 120Hz, 240Hz, etc. are often marketing overstatement), gaming monitors can now routinely handle 144fps, 165fps and even 240fps. They’re locked in a numbers race, trying to keep up with graphics cards output capabilities. To what extent the human eye can appreciate the difference between 144fps and 240fps remains a question. Mark Rejhon, founder of Blurbusters, argues in favor of a "retina refresh rate" of over 1000fps based on quite extensive research that he’s been doing. HFR demo (with HLG HDR) by LG / EBU / 4EVER Project at IFA 2016
4K HFR gameplay on a 2020 LG OLED TV
Billy Lynn’s Long Halftime Walk. Picture credit: 20th Century Fox
Recommended reading
You can read more facts about and impressions of The Hobbit here on FlatpanelsHD, or read about the aftermath. Filmmaker Vincent Laforet has written an elaborate review. There is plenty of coverage for Ang Lee’s movies but long before the days of internet, Douglas Trumbull – Visual Effects Supervisor for classic movies including 2001: A Space Odyssey, Close Encounters of the Third Kind, Star Trek and Blade Runner – did many experiments with HFR. This article about his movie Brainstorm offers a great summary of that. Trumbull and Cameron were speakers at a 2012 Siggraph panel session on HFR cinema, a report of which you can read here.Standards referenced
Yoeri Geutskens works as a consultant in media technology with years of experience in consumer electronics and telecommunications. He writes about high-resolution audio and video. You can find his blogs about Ultra HD at @UHD4k and @UltraHDBluray.