Camera Tests, Pt. 1: The Theory of Camera Testing
Since the purpose of my Minor Project is to learn about the process of camera testing – and since camera testing in itself is an important part of pre-production and the successful application of the craft of cinematography itself – I will use this blog entry to delve deeper into the matter.
So What Is Camera Testing?
As the name suggests, the procedure of camera testing involves a series of tests designed to establish a camera’s performance, qualities, and limitations in general or in regard to special needs of a certain production. The aim is to create a reference library of footage that can be evaluated in order to find the right camera as well as the optimal solution for a desired look, thus identifying and mitigating possible and/or unforeseen issues early on.
Back in the day, camera tests were originally conducted to test film stock and its emulsions, as well as their interplay with cameras, lenses, lab processes, etc. However, with the digital revolution in film and the emergence of digital cameras, camera testing nowadays has been expanded to testing the digital camera sensor in relation to lenses and post-production.
Since every camera sensor has unique qualities and limitations to itself – and new cameras, and sometimes even new lens kits, codecs, etc. are being released roughly every six months – testing it is imperative. Furthermore, as Elliot Weaver from Reel Deal Filmschool points out, it is estimated that with the elements tested have quadrupled with the digital revolution.
Tested Aspects and Types of Tests
I’ve already roughly outlined what camera testing is supposed to do on a general basis. But what kind of elements are exactly being put to the test? My research hitherto led to this preliminary and non-exhaustive list:
Exposure and Dynamic Range
ISO (Low-light performance and grain/noise)
Colour Space and Colour Rendition (including skin tones)
Shutter Speed and Frame Rate
Lenses and Lens flares
Lighting Ratio (Fill Ratio)
Lighting Styles
Compression artefacts (such as e.g. moiré and step-laddering)
Size and weight of the camera
As I mentioned above, this is only an excerpt of elements that can be put to the test. In reality, only the requirements of your productions – such as e.g. night shoots, candle-lit rooms, harsh and direct sunlight or colour palettes à la Wes Anderson – can tell you which tests you might want to prioritise. Furthermore, some leading questions, as put forward by Suny Behar, might be added to the mix as well:
What is the distribution platform? Will I need high resolution and colour depth for
Are there special deliverable requirements? Resolutions, compression types, etc?
How big is the budget? Can I afford to rent and/or buy a particular camera kit?
What are the ergonomics of production? Will I be shooting e.g. green screen?
Because of this, the procedure of camera testing can entail and/or combine any of the following tests (again, not exhaustive):
Colour Tests
Fill Ratio Tests
Dynamic Range Tests (or Latitude Tests)
Low-Light Tests (or ISO Tests)
Chroma Key Tests
But before I will be able to delve further into the individual tests, I need to add a disclaimer:
Implications for Research
As it might be gleaned from my summery, camera tests are usually part of the pre-production process and a crucial element in finding and creating a certain cinematic look. As finding and refining “the look” can and often does include particulars of a film production, such as e.g. actors, hair, make-up, costume, set design, and lighting tests, camera tests for films and TV-series are often confidential and undisclosed materials that rarely make it out in the open, thus impeding research on the practical execution of said tests.
What quickly became obvious during my research was the fact that – although there are many independent reviews on any camera out there – a lot of reviewers don’t reveal their processes in detail; sometimes only revealing results, but not exactly how these have been obtained. This is especially frustrating regarding aspects such as transparency, comparability and repeatability of the testing process itself as it forces every filmmaker to conduct their own tests.
Even though it could be argued that this would enforce a more informed, individual examination of camera technology per se, the downside is a lack of a coherent, universally recognised procedure to be taught and adhered to. Which is surprising regarding the usually military-like hierarchy in the camera department and set etiquette itself.
Since the execution of these tests differ slightly depending on the DoP, I ventured into some of the more helpful and transparent procedures online in order to find a more versed testing method for some of the more important parameters for my FMP production.
Which is why I will outline some of the DoPs and their procedures in the following:
Camera Testing: Variants
Gavin Finney
DoP Gavin Finney highlights the importance of camera testing. He did a series of tests concerning dynamic range and colour representation by shooting a candlelit room filled with grey scales, colour charts and objects of colour.
With this test, as he elaborates, he wanted to put the footage through post in order to test the latitude of the footage taken as well as some other characteristics of the image itself. And while this seems like a good idea for a test in general, I won’t be able to use it for Pillars of Creation as I do not have a candlelit scene in there. However, it might be worth keeping it in mind for Madame Nomvula.
Shane Hurlbut, ASC
Thus, I was extremely happy to come across the published camera tests of Shane Hurlbut, who helped bring some clarity into the procedure itself, serving as a good initial framework for practical research.
Shane Hurlbut has set out to conduct extensive tests on a variety of camera systems to properly assess their performances, such as e.g. a normal film camera (of undisclosed model), a Sony F5, a Blackmagic Cinema Camera and a Blackmagic Mini Ursa 4.6k, a RED Epic and a Canon C500. In order to getting to know a camera properly and fully understand its functions and limits, he thus advises to test a range of different conditions that are likely to be encountered during production.
Furthermore, he places emphasis on a camera sensor’s ability to render skin tones when he says:
“Skin tone is the vitality of a sensor.” He elsewhere considers the camera sensor as “the brush” of the filmmaker.
Fill Ratio Tests
Because of this, he starts out with so-called Fill Ratio Tests in which he not only tests the skin tones of his actors/stand-ins, but also the progression from light to dark, thereby assessing the skin tone and the fall-off with every 1/3-stop. He does so in his comparative camera test between the Blackmagic Cinema Camera and the Sony F5:
Dynamic Range Tests (“Latitude Tests”)
Hurlbut splits the Dynamic Range Tests into two, called overexposure and underexposure tests. This was somewhat confusing to understand at first, since he does not only want to test the dynamic range of the cameras, but also their latitude. Whilst latitude is based on dynamic range, both terms are not interchangeable, as dynamic range describes, “(…) the range of exposure a camera can cover without losing detail in the brightest and darkest areas of an image”, whereas latitude describes by how many stops you can manipulate a given exposure by stopping up/down and furthermore, how much information can ultimately be recovered in under- and/or overexposed images in post.
Anyways, Hurlbut set about conducting these tests as you can see down below:
Thus, by slowly increasing the exposure, he is able to find what he calls the “breaking point” of the BMCC: At 2 2/3 stops overexposure, the footage is at the brink of clipping in the male stand-in’s face.
Whilst these clips give a good impression as to what and how he is testing these cameras, they sometimes do lack a bit of explanation in terms of the test execution – at least for beginners like me. Whilst I understood his of the imagery analysis, I was confused as to why he would – despite the same mount – pick two different cameras and attach two different lenses to them instead of using the same lens for both.
If anything, I would have argued that this would have added another variable into the equation, making it more difficult to pinpoint certain effects to the camera’s sensor and possibly affecting the outcome. I would have loved a quick disclaimer on his part and an explanation as to why he chose the setup like he did.
Matthew Rosen
In contrast to the above-mentioned DoPs, Matthew Rosen from KINETEK used a different terminology and an additional test in order to assess his camera sensor. He splits the Dynamic Range Test into two, referring to the part of the test concerning underexposure as Grain Test and to the one concerning overexposure as Clip Test. In both tests, he tries to assess just how much detail can be saved in post in the relevant areas. For the Clip Test, the brightness levels will be brought down in the edit in order to flatten the contrast:
But as you can see in the video, he also adds another test to the mixture, one of which I have not yet thought about – the Chroma Key test. While the name of the test suggests that it would assess a camera’s performance in regard to green screen, the true aim of this test is to check for compression artefacts, such as e.g. step-laddering. However, as any good camera test, it allows to test for two different aspects of cinematography at once without influencing the result of either aspect, as he not only successfully ascertains possible compression issues, but also effectively visualises them in the context of green screen application.
Suny Behar
Another cinematographer who was very helpful during my research was Suny Behar. Though his methods of testing a camera and its properties are slightly different to Hurlbut, they deserve to be noted nonetheless.
Thus, Behar emphasises three different tests that he considers to be of importance:
Colour Test
Dynamic Range Test
Low Light Test
Colour Test
If colour and the application of a colour palette are of importance for a production, this is the test to conduct according to Behar. Since every camera sensor has a different colour filter applied to their sensor, which in turn defines the colour gamut, every camera will be prone to record the spectrum of available colours slightly differently. Because of this factor, it is important to test an array of cameras in order to find the right one for your camera.
While a colour test might look like skin-tone tests as described with Hurlbut above, they in fact do entail more. For this test, however, you do not only shoot the skin, but furthermore also the surrounding colours, for you want to assess what happens to them once the skin colour has been balanced in post.
For this, Behar suggests you should conduct the test as follows:
Bring in stand-ins that do have a similar skin tone to your actor.
Surround them with colour patches matching your colour palette (fabrics, etc).
Film and compare how these skin-tones will fare in tandem with the colour swatches.
Organise by shot size: go from WS, to MCU, to CU.
Import the footage in DaVinciResolve.
Only use the primary colour correction controls to correct the skin-tones.
Compare the results of the surrounding colours. How do they differ from camera?
Use colour chart recognition software (if applicable) and check the skintones then.
If you have a 70% match with the desired colour palette, you will be fine.
Dynamic Range Test
Depending on the desired look for your production, you might either be looking for a strong look rich in contrast or for a mellower look. Because of this, getting to know the dynamic range of each tested camera is important.
Thus, for this test, you would want to conduct the test by trying to recreate the dynamic range you want to cover. This, according to Behar, can be done as follows:
Pick a time of day when the natural lighting is the strongest.
Find a window to shoot through and capture something detailed outside.
Take a stand-in that stands next to the window.
Meter the light levels outside.
Meter the light levels on the stand-in’s face without any light on.
Check the difference in stops.
a) If the difference is bigger than the range you want to test, shine a light on your stand-in to reduce the difference.
b) If the difference is adequate, proceed with 7).
Shoot the footage.
Bring the Footage into DaVinciResolve.
Compare the amount of detail in both areas. How does the camera deal with extremes?
a) Are there blow-outs?
b) Are the details filled with image artefacts?
c) Are the shadows noisy?
If a camera happens to be excellent at colour renditions but poorly on dynamic range, then Behar suggests that tweaking the latter by the use of ND-filters, gels and nets to the windows or moving the time of day of the shoot as possible mitigations for the issue.
Low-Light Test
Lastly, Low-Light Tests are called for on productions that require dark and moody scenes. For this, Behar recommends the following procedure:
Find a small room with a lot of objects and details in it.
Darken it (if it isn’t dark yet).
Light a couple of candles or a desk lamp if the room is too big.
Have a stand-in in dark clothes walk through the room with a torch, searching.
Have the stand-in shine light on various objects and the camera lens itself.
Film the footage.
Import the footage into DaVinci and check the digital noise.
Measure the signal-to-noise ratio.
Check how the lit-up objects react to the noise in the shadows.
Check for lens flares and ‘black sun effects’ created by the torchlight.
Behar furthermore recommends to not boost the ISO, as the ISO does not affect the effective dynamic range but only ramp up the light sensitivity of the sensor – which he calls noise or exposure floor. Because of this, shooting in low light will always be a balancing act between digital noise and dynamic range.
Commonalities
What immediately became obvious studying the various approaches to camera testing was an emphasis on colour, dynamic range, and ISO. It was furthermore striking to realise that all DoPs included colour in the background for every test, either by using a colour chart, by using objects of various colours or by lighting the background in a different colour, much like this filmmaker did as well testing the Blackmagic Pocket 6k:
This not only allowed for an assessment of colour rendition and alternative exposure points – as pointed out by Hurlbut – but also for an assessment of colour noise and noise in the dark areas as well as blow-outs in the bright areas.
My Aim and Procedure: Planning My Camera Tests
For my FMP and especially for Pillars of Creation, colour reproduction, dynamic range, and low-light performance will be the most important aspects that need to be assessed by me. As such, I will settle on conducting the relevant tests. My hope is that these will not only help me pick the right camera and look for my FMP, but that it will also teach me more about the interplay of various camera aspects and the intricacies of optics as well.
But because camera tests can quickly become very laborious and extensive, I followed the advice of Behar and limited my selection of potential cameras to a list of maximum three cameras, of which I already blogged about in my blog entry “My Test Subjects: The Cameras I Will Be Testing”.
Since Shane Hurlbut exemplified his procedure of Dynamic Range and Latitude Testing nicely, I will pick his procedure as a template for these aspects of my camera tests. In regard to the Colour Test, I will follow the procedure as laid out by Behar.
I will start with a simultaneous and comparative setup of the Blackmagic Cinema Camera and the Blackmagic Mini Ursa Pro 4.6k, filming a stand-in; including the skin-tone and providing not only a colour chart and objects of colour (provided by Eline) that will most likely be used for our FMP, but also a dark and a bright reference point each in the background. The points in the background will stand as a relative reference for exposure while our stand-in will be our subject we will expose to.
This scene will then be exposed to a neutral lighting setup at 3.200K to help assess the colour output and metered with a light meter to assess exposure. For this, I will use my previous lighting plan and adapt it to fit the brief above. Along with filming the Colour Tests, I will adapt the exposure both in terms of overexposure and underexposure, in order to find the breaking point of each camera by stopping down (underexposure) and stopping up (overexposure) the aperture.
The tested cameras will be logged and stated, along with the lens, shutter speed, filter, colour temperature, native ISO, and used aperture and the obtained footage will then be further processed for both purposes in post-production as described above.
I’d say the only thing left to do is taking out the equipment and transitioning into the practical part of the exercise!
Let’s see how this goes!
References:
Astraatmadja, K. (2007) How & What to do for Camera Test [online] Available at: https://cinematography.com/index.php?/topic/21604-how-what-to-do-for-camera-test/ [Accessed on 25 October 2019]
Behar, S. (2016) How to Perform a Camera Test: Narrow Your Focus to Three Simple Things [online] Image taken from: https://www.moviemaker.com/archives/mm_guide_2016/how-to-perform-a-camera-test/ [Accessed on 25 October 2019]
CookeOpticsTV (2016) Testing your Camera and Lenses || Masterclass - Gavin Finney [online] Available at: https://www.youtube.com/watch?v=kmkYn3_O2UU [Accessed on 25 October 2019]
Giant Screen Cinema Association (2019) GSCA Camera Tests [online] Image taken from: https://www.giantscreencinema.com/Member-Center/Camera-Tests [Accessed on 25 October 2019]
Grunin, L. (2014) How we test: Digital cameras. Learn about CNET's testing procedures for digital cameras [online] Available at: https://www.cnet.com/how-we-test-cameras/ [Accessed on 25 October 2019]
Hurlbut Academy (2013) Red Epic Latitude Test - Over Exposed [online] Available at: https://www.youtube.com/watch?v=nLEBQ7e2cxk&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=44 [Accessed on 25 October 2019]
Hurlbut Academy (2013) RED EPIC Latitude Test - Under Exposed [online] Available at: https://www.youtube.com/watch?v=C9riqeqHYN0&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=43 [Accessed on 25 October 2019]
Hurlbut Academy (2014) Blackmagic Cinema Camera vs Arri Alexa Latitude Under Exposure Test [online] Available at: https://www.youtube.com/watch?v=kHm3ZZdE13I&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=31 [Accessed on 25 October 2019]
Hurlbut Academy (2014) Blackmagic Cinema Camera vs Arri Alexa Latitude Over Exposure Test [online] Available at: https://www.youtube.com/watch?v=WnAzICYqXq8&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=32 [Accessed on 25 October 2019]
Hurlbut Academy (2014) Blackmagic Cinema Camera vs Sony F5 Fill Ratio Test [online] Available at: https://www.youtube.com/watch?v=uWssco9kDHg&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=30 [Accessed on 25 October 2019]
Hurlbut Academy (2014) Canon C500 vs Film Dynamic Range Test [online] Available at: https://www.youtube.com/watch?v=2XXc5qsdEak&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=39 [Accessed on 25 October 2019]
Hurlbut Academy (2014) Canon C500 vs Film Latitude Test Overexposed [online] Available at: https://www.youtube.com/watch?v=BscUveUqcaY&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=38&t=0s [Accessed on 25 October 2019]
Hurlbut Academy (2014) Canon C500 vs Film Latitude Test Underexposed [online] Available at: https://www.youtube.com/watch?v=mBq9V72jYMI&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=37&t=0s [Accessed on 25 October 2019]
Hurlbut Academy (2016) Blackmagic URSA Mini 4.6K Latitude Test | Over Exposure at ISO 800 [online] Available at: https://www.youtube.com/watch?v=GRNoSHysSxM&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=94 [Accessed on 25 October 2019]
Hurlbut Academy (2016) Blackmagic URSA Mini 4.6K Latitude Test | Under Exposure at ISO 800 [online] Available at: https://www.youtube.com/watch?v=tvsbtFC-hH0&list=PLHyU9PnnTe1eK6TDFuJkhUrsRpZB86cKh&index=96[Accessed on 25 October 2019]
Hurlbut, S. (2012) The Importance of Camera Tests [online] Available at: https://www.hurlbutacademy.com/film-education-online-the-importance-of-camera-tests/ [Accessed on 25 October 2019]
Hurlbut, S. (2014) Blackmagic Cinema Camera: Putting It Through the Paces [online] Available at: https://www.hurlbutacademy.com/bmcc/ [Accessed on 25 October 2019]
KINETEK (2016) Tutorial on Cinematography - How To Test Your Camera's Limitations [online] Available at: https://www.youtube.com/watch?v=kVxh-R6XE04 [Accessed on 25 October 2019]
Murie, M. (2014) Cinematographer Shane Hurlbut on Camera Tests and the Joy of Older Lenses [online] Available at: https://filmmakermagazine.com/87556-cinematographer-shane-hurlbut-on-camera-tests-and-the-joy-of-older-lenses/#.Xbh4Py33600 [Accessed on 25 October 2019]
LensProToGo (2019) BLACKMAGIC POCKET 6k - LOW LIGHT and EXPOSURE TESTS [online] Available at: https://www.youtube.com/watch?v=mZUCPIuD4rA [Accessed on 25 October 2019]
The LensPal (2018) Blackmagic Design URSA Mini ISO & Aperture Test by The LensPal [online] Available at: https://www.youtube.com/watch?v=jHyeGqgvO3w [Accessed on 25 October 2019]
Trusted Reviews (2014) How we test cameras and lenses [online] Available at: https://www.trustedreviews.com/info/test-cameras-lenses-3552441 [Accessed on 25 October 2019]
UglyMcGregor (2018) Blackmagic Pocket Cinema Camera 4K Film Look Test [online] Available at: https://www.youtube.com/watch?v=5uw4M_Z4A5I [Accessed on 25 October 2019]
Walters, R. (n.d.) How To Bring a Cinematic Look to a Small Budget Commercial, part 7 in: Indie Cinema Academy (2019) Part 07: Conducting A Camera Test [online] Available at: https://indiecinemaacademy.com/part-07-conducting-a-camera-test/ [Accessed on 25 October 2019]
Weaver, E. (2016) Camera Tests [online] Available at: https://reeldealfilmschool.wordpress.com/2015/09/01/camera-tests/[Accessed on 25 October 2019]