Image-Based Lighting and Hitfilm


  • Triem23
    Triem23 Posts: 20,294 Power User
    edited March 2021

    The following is debate and discussion on whether or not the Diffuse Reflectivity feature in Hitfilm's 3D model shading engines is functionally equivalent to Image-Based lighting in other 3D apps.

  • GrayMotion
    GrayMotion Posts: 1,634 Enthusiast
    edited March 2021

    I found that Hitfilm code must be looking for the word "container" when loading the SMALL map. If you wanna use any other map its name must be "container". . I do find it curious that there are other maps in the folder but no apparent way to access them. Maybe a future feature?

    Mike - along the lines of the two post you've made today maybe you can en- lighten me as to why the environment map function does not emit light directly from areas of the environment directly onto the scene? I believe its call Images Based Lighting. Based on direction and strength. ALL other 3D applications I use have this feature except Hitfilm. What am I missing? Is it that difficult to accomplish with the current coding of Hitfilm. Seems like a no brainer for me but I'm just a lame man.

    EDIT: Irrelevant I know but the only way I can get this type of lighting in Hitfilm is by using the Element 3D plugin....of course with Element the IBL is built-in. Makes me want full image lighting control that much more natively in Hitfilm 😉

  • Triem23
    Triem23 Posts: 20,294 Power User

    @GrayMotion Hitfilm actually DOES do some Image Based Lighting.

    It's controlled via the "DIFFUSE REFLECTIVITY" value, and by the reflections and environment map settings in the model's layer.

    In the 3D model's layer, you need to make certain "Receive Reflections" is on, and that an environment map layer has been assigned.

    Here's a quick demo. I loaded in a simple sphere and set it to Cook-Torrance. I created a embedded comp with a plane with a 4-point gradient effect, added a single light and enabled reflections in the model's layer and set the embedded comp with the gradient as the Environment Map.

    With 0% Diffuse Reflectivity, it looks like this.

    With 75% Diffuse Reflectivity it looks like this.

    So, while we don't have as much control as a dedicated 3D app, and there's room for improvement there, I'd argue it's not actually true that Hitfilm doesn't do any kind of Image Based Lighting.

    I might have gotten better results if I'd used an actual 360 wraparound as the map, but my wifey is making it clear that she's ready to go to the store.

    What do you think, Greg, should we split your comment and this comment off to an "Image Based Lighting in Hitfilm thread?" It doesn't quite fit in the Feature Request Forum.

  • spydurhank
    spydurhank Posts: 3,156 Expert

    That is a vote up for sure and yes. This is a cool idea.

    The Hitfilm environment map only cast reflections and not light.

  • spydurhank
    spydurhank Posts: 3,156 Expert

    Ah, I don't understand your meaning. Help me out? Images in Hitfilm don't actually cast light. It'd need a 3D path tracer for it to really be called image based lighting. If images could actually cast light in Hitfilm, that'd be a whole different ball game right there and it'd be an awesome game. :)

    This is a point light in Hitfilm. That sharp line is the Terminator effect. In Hitfilm the Lights are "almost" represented correctly because all light is a reflection out here in the real world but in the real world, lights also "emit" and cast outwards. The point light however is only a reflection from my POV as it is only casting a reflection and is not calculating or casting actual light. The clipping or terminator effect created by the point light looks and behaves like an image that has too small of a bounding box wrapped around a 3D sphere like you would do for an environment map. If the lights actually emitted light in Hitfilm, then we'd get proper shadows, transparent alpha shadows, glass, caustics, volumetrics and a whole mess of other cool stuff. But yeah, you can call it image based lighting if you really want, actually you can call it anything you like, what matters is what you can do with it. :)

    In Blender though, any image with an emission shader will cast light, shadows, reflections and volumetrics. Hitfilm is a lot closer to EEVEE which is not to say that it isn't cool.

  • GrayMotion
    GrayMotion Posts: 1,634 Enthusiast
    edited March 2021

    What do you think, Greg, should we split your comment and this comment off to an "Image Based Lighting in Hitfilm thread?" It doesn't quite fit in the Feature Request Forum.

    Yep Mike...please do split IBL to another mind was wandering around lighting of a kinds. Sorry to wander off topic.

  • Triem23
    Triem23 Posts: 20,294 Power User

    Before I get into my long debate points, I'd like to tag in Hitfilm devs @CedricBonnier @RobLaycock and @RubenGalvao. Gentlemen, this thread is a debate and discussion on whether or not the "Diffuse Reflectivity" in Hitfilm's 3D model shaders is functionally equivalent to "Image-Based Lighting" in other 3D apps. I know from @TheBenNorris that Rob worked on the Metal/Rough and Spec/Gloss shaders, but I don't know who worked on the Phong and Cook-Torrance code. I'm guessing Cedric or maybe Ruben. This paragraph give you three the basic TL/DR thrust, but, y'know, if you're really bored, you can read the rest of this post. You might be able to explain something I'm not understanding.... Especially since I'm eventually going to make a video tutorial on this topic and I'd rather not give the Hitfilm community inaccurate information.

    Moving on.

    @GrayMotion one thing I didn't address earlier - you noted that the 3D model import viewer grabbed the "Container" image in the folder with the 360-degree images, would grab whatever image in that folder had been named "Container" and wondered if that was setup for a future feature.

    Those maps have all been in the software package since Hitfilm 2 Ultimate (which is the first time I ripped them and put them into a library folder), so now I'm wondering if Hitfilm was supposed to have some way of swapping the images, it was something that wasn't ready for the release, and just got pushed back forever in favor of other features and development?

    @spydurhank When I split off these comments into this thread I missed one of my own comments. This one:

    • "Isn't "Image Based Lighting" picking up the color from the surrounding environment? As far as I can tell the "Diffuse Reflection" in Hitfilm is doing the exact thing Image Based Lighting does!"

    So... I'm not a coder and haven't written shaders, so be ready for a bunch of links to my own research to follow.

    Ye olde Wikipedia says of IBL: "Image-based lighting (IBL) is a 3D rendering technique which involves capturing an omnidirectional representation of real-world light information as an image, typically using a 360° camera. This image is then projected onto a dome or sphere analogously to environment mapping, and this is used to simulate the lighting for the objects in the scene." Then notes ""Almost all modern rendering software offers some type of image-based lighting, though the exact terminology used in the system may vary." [Emphasis mine]

    Wikipedia also says of Diffuse Reflections: "Diffuse reflection is the reflection of light or other waves or particles from a surface such that a ray incident on the surface is scattered at many angles rather than at just one angle as in the case of specular reflection..." and later notes, "The visibility of objects, excluding light-emitting ones, is primarily caused by diffuse reflection of light: it is diffusely-scattered light that forms the image of the object in the observer's eye."

    Of course Wikipedia isn't always the best source, but, Hitfilm's Diffuse Reflections take rough color values from the environment map assigned to the layer - an "omnidirectional [image] of real-world light." which is "projected onto a...sphere." As shown in my images in my earlier post, when I turn up Diffuse Reflectivity on my plain gray sphere the sphere does appear to be lit and is picking up color from the background gradient. To my eyes, this is absolutely what "Image-based lighting" is supposed to do. While Image-Based Lighting traditionally uses HDRI's... Wait, Hitfilm PRO can render in 32-bit color and can import 32-bit EXR files. There's no reason I can't import a true 32-bit HDRI as the environment map!

    But, let's dig deeper into the underlying science. I'll just drop this page here, which contains several scholarly articles from various SIGGRAPHs by the programmers and artists who invented IBL techniques: These are what I have to consider primary sources, and much better than Wikipedia.

    The rest of my points will be pulled specifically from this article (which can be found as a link in the above-linked page): Because this is an encrypted PDF I am unable to directly pull images from the article, so I apologize for the occasional page reference. Maybe open the PDF in another tab to follow along? (Incidentally, the coders here will find code for RADIANCE in the article. May be of use?)

    The opening paragraph: "Image-based lighting (IBL) is the process of illuminating scenes and objects (real or synthetic) with images of light from the real world. It evolved from the reflection-mapping technique."

    Now I'll skip down to page 6 of the 9 page document (it's an excerpt from a long document, so the "Page Number" shown in the doc is "Page 31."

    "Radiance is tracing rays from the camera into the scene, as Figure 7 illustrates. When a ray hits the environment, it takes as its pixel value the corresponding value in the [Environment Map]. When a ray hits a particular point on an object, Radiance calculates the color and intensity of the incident sending out a multitude of rays... in random directions to quantify the light arriving at that point on the object. Some of these rays will hit the environment, and others will hit other parts of the object... After Radiance computes the illumination on the object point, it calculates the light reflected toward the camera based on the object’s material properties and this becomes the pixel value of that point of the object."

    Now, of course, this is talking about the RADIANCE renderer, not Blender or Hitfilm, but the above translates to me as rays from the CAMERA being sent out to measure colors, taking color from the environment wrap, then calculating the final color values based on the material - i.e. reflective properties of the 3D object's materials. There's nothing in that description that reads to my mind as the HDRI environment "Casting Light" on the scene. In fact, to me it sounds exactly like a mathematical model of "Diffuse Reflection."

    And, as the "Diffuse Reflectivity" parameter in Hitfilm is what drives the apparent illumination of the 3D objects based on colors and intensity of the defined Environment Map, as far as I can tell, Hitfilm's Diffuse Reflectivity IS a form of Image-Based Lighting. Hitfilm doesn't have any control over how the environment map lights the image other than the Diffuse Reflectivity percentage slider, so it's certainly a more primitive method that those used in other software, but, as far as I am currently capable of understanding, it's still an image-based lighting situation. It's not that Hitfilm's LIGHTS are doing anything out of the ordinary, it's more the shortcuts taken in Hitfilm's engine with shadow and reflection mapping making Hitfilm's engine less powerful than dedicated 3D apps.

    Then again, I think Hitfilm also does 3D path tracing. After all, a Hitfilm point light emits light in all directions. The Phong shader certainly calculates light/dark based on the angle of incidence of a light ray hitting the object and bouncing off - and, when I was taking 3D Computer Graphics courses in college in the 1980's the Cubicomp manual referred to it's Gouraud and Phong shading as using "Path Tracing" to generate lights/shadows and highlights. Cook-Torrance adds Fresnel equations, which cause changes in object color based on angle of incidence of lighting. Certainly that requires tracing the path of the light ray in 3D space. It's just that Hitfilm isn't ray/path tracing the shadows (choosing to cheat with an image-based Shadow Map calculated from the "viewpoint" of the light), or doing true volumetric refractions (Hitfilm just cheats with an inverted copy of the reflection map, distorted based on a IOR value.

    Still, the techniques I learned in the 80's, using broadcast animation gear of that time, and utterly lacking in raytracing (just Gouraud and Phong, and the Phong was the "high-end" shader) described the lighting model as "Path Tracing." Therefore, to my mind, when people start insisting that "Path Tracing" has to include full volumetric support and volume refractions and ansotropic lighting those people are, in fact, using the term "Path Tracing" incorrectly - in exactly the same way that people now say "PBR" when they're really talking about Metallic-Roughness and Spec-Glossy Shaders based off Disney's BDRF equations... because Phong, Cook-Torrance, and Lambert (among others) are "physically based" shaders that respect conservation of energy and use aspects of surface roughness and other material properties. (With the main thing about modern "PBR" shaders being the accumulation of real-world data about how light reacts with objects is now hard-coded into the shader rather than constantly messing around with maps and multiple parameters to model the material.

    Anyway, the long and the short of it is, as far as my skills, math and understanding allow, Hitfilm's Diffuse Reflectivity is functionally equivalent to Image-based lighting, albeit without advanced controls.

    Frank, if you can show me I'm wrong, please do. To clarify, this isn't a contest about "who's right and who's wrong," I just genuinely try to learn. Also, since I'm the most vocal person in the Hitfilm universe (look at my number of forun posts compared to ANYONE), I want to have correct information because I don't want to give users wrong information. I want us all to learn, grow and improve together. You know I respect the hell out of your skills and knowledge.

    Also, Frank, your comparison of Hitfilm's shaders as being more EVEE than CYCLES? We agree there. You're spot on.

    OK, Laura's waiting for me to stop the heck with the typing so she can serve up dinner. I mean, it's out leftovers from the last meal she cooked, the last meal I cooked and some restaurant leftovers, so it's just nuking our plates for a minute, but it's the principle of the thing.

  • spydurhank
    spydurhank Posts: 3,156 Expert

    Ah dude, you're funny. :)

    I think that you just asked me to, Not prove you wrong but to uh... please prove you wrong? :)

    I'm sitting here laughing so hard that my eyes are watering. :) :)

    It's almost like you're "Double Dog Daring" me to prove you wrong. :)

    Oh man, thanks for that. :) :) :)

    Seriously though, I don't do debates, there's no need for something so dramatic. It's as simple as this...

    I either know what I'm talking about or I don't. Right? :)

  • GrayMotion
    GrayMotion Posts: 1,634 Enthusiast
    edited March 2021

    This is what I see with my eyes - it is not technically based view like yours (Mike).

    I'm going to KISS if I can..

    First. When I use an environment map in Hitfilm it appears I get reflection of the colors....not direct lighting of the bright and dark areas. Now to me the 360 Environment Layer should interact with the model but it doesn't. If you rotate the image it does not reflect the lighting in the image but rather just reflections of colors. Those colors stay in the same spot though. Now I know I'm talking 2 things here...environment map AND environment "wrap". In my mind the environment Layer in the models Material section should be what is controlling light direction not just reflective color.

    Now a Hitfilm test. I used a sphere and adjusted the reflectiveness of the model so I could see the Environment layer on the sphere.. Then I added a 360 environment layer. First things I noticed is that the Environment Layer reflection appears inverted. So Blue sky is reflecting the sky on the bottom of the model and the grass is reflection on the top. This is wonky. Shouldn't the image reflect at the same orientation as the background? If you rotate the camera on X 180 then the reflection is correct but camera is upside down.?? Wonky.

    Now: Element 3D. If I use say a "lightbox" map (with one light box positioned to the left center of image. (Dark on one side, bight on the other.) Then I navigate to the environment section of Element 3D and rotate the background (say on Y) I can see the model being lite brighter/darker depending on the position of rotation. It is not reflective but rather true light from the image.

    Now - iClone. Works just like Element. If I rotate the image the light/dark areas are represented perfectly based on the position of rotation. Blender environment - works the exact same way. That is three software working exactly as I would expect.

    Since I am out of my wheel house as to the technical knowledge like yourself and Frank I cannot articulate what I see as clearly as you two.But like I say..what I see with my own eyes leads me to believe that reflection is at play and not lighting. When I worked on my iSS video and Tony lent me his advice he also mentioned to me that the environment map layer is not working like one would expect to see with true IBL. I believe he said it was based on general refections and I should not expect true lighting (direction) based on the image I used.

    I wish I'd have grown up more like yourself, Tony and Frank because then I'd feel more confident about explaining what I'm trying to say. But alas..since I broke my teeth on Fiber Optics and still doing it mind cant think like you great gentlemen. I'm stuck in the world of lasers , light attenuation and IOR.