Nuke: ModelBuilder
ModelBuilder (Geometry) node:
- Create a card from ModelBuilder node
- Click and drag in viewport to change size
- Move the points so the cards match the scene
Nuke W13 – Projection3D
Project3D node
Purpose:
Project3D is used to project a 2D image onto a 3D object. It’s like shining a slide projector onto a physical model; the image “wraps” around the 3D shape according to the geometry and camera position.
Project on a Match-move Geometry
- Freeze a frame using
Framehold(Choose a frame that is the closest to the camera and appear the clearest) - Input a 2D image into the Project3D node (this can be a texture, or from a premulted rotopaint patch)
- Freeze the frame again (this is to minimize calculation from rotopaint node)
- Premult the patch
- Use a Project3D node that connects to a Match-move Camera
- Project3D > Card > ScanlineRender
- Merge Original Plate with ScanlineRender’s output
MergeMat (Shader): Similar to the Merge node, this is specifically designed for 3D space operations.
Project at different distance
In this setup above, we use 2 framehold nodes, one the closest and one furthest from the camera. Then merge 2 Project nodes together using Mergemat. This approach ensures a more natural result by projecting the patch at different distance.
Projecting Roto
- Roto the 2D element
ModelBuilder (only in Nuke X) – For building geometry. right click and choose mode. right click and change selection mode (like 3D softwares)
Resources:
https://learn.foundry.com/nuke/content/reference_guide/3d_nodes/project3d.html
Nuke Garage (Part 1): 3D tracking
Nuke W11: 3D tracking
3D tracking
1. Preparation of the Footage
- Import: Bring your footage into Nuke.
- Pre-Processing: Ensure the footage is ready for tracking. This includes deinterlacing, stabilizing if necessary, and removing any lens distortion. You can also treat it by brightening or sharpening the shot.
2. CameraTracker Node
CameraTracker analyses the motion in a 2D footage and extrapolates this movement into a 3D camera path. It tracks various points in the footage (usually high-contrast or distinctive features) across frames to determine how the camera was moving when the footage was shot.
- 3D tracking only works on stationary objects
- Roto out area that you want to avoid tracking (things that move/not static, be mindful of reflective objects). Then connect to CameraTracker node via ‘mask’.
- In the use of Roto, change mask type to ‘Mask Alpha’
- In CameraTracker settings, choose the type of source and mask. If you’re unsure about the Lens Distortion and Focal Length, leave settings as default
Several properties in this tab can help achieve a better track:
- Number of Features: The amount of automatic tracking points created by the tracker. If you increase this, reduce Feature Separation.
- Detection Threshold: The higher the number, the more precise the tracker has to be in finding trackable points.
- Feature Separation: The higher the number, the farther apart the tracking features have to be. If you increase Number of Features, reduce this value.
- Camera Motion: This setting tells Nuke whether the camera is handheld or Free Camera, meaning a free-moving camera, or if it’s Rotating Camera, meaning a camera on a tripod.
3. Solving the Camera
After tracking process is done, click ‘Solve’
- Check error figure in AutoTracks tab to evaluate your track
- Click on error-max. Click on graph and press F
- Reduce Max error to 6, click delete unresolved and delete rejected
- Usually solved error is anywhere around 1-2 or below is good
4. Export the scene
Export by choosing ‘Scene’ or ‘Scene+’
Make sure link output is enabled
To check tracking:
- Select some point > Create > Cube/Plane
- Plug the object Card into Scene > move it to match the ground plane
Resources:
https://lesterbanks.com/2020/07/getting-started-with-3d-camera-tracking-in-nuke/
https://lesterbanks.com/2013/09/nuke-3d-tracker-fundamentals/
Final Project: Final video & reflection
Reflection:
Overall, I am happy of how this project turned out, as well as appreciating the knowledge that I have earned through the process of making it. One of the most challenging aspect for me at the beginning that has affected my ability to start was being unsure of what I wanted to make, which got me really anxious and definitely did not help my creativity. I figured I was being too ambitious and got overwhelmed by all the ideas I wanted to make and all the softwares to learn. I eventually had some reflection and reminded myself that the main aim of this project is for me to learn new softwares and practice VFX fundamentals, so I need to lower my expectation for myself and just experiment. I did not have a plan or storyboard at first, and the idea only started coming together as I was creating my Unreal Scene. I have had the vision in mind of the theme that my art usually follows, so every idea after that was by all mean a trust in my own process and experimentation.
Unreal Engine:
In terms of my scene in Unreal Engine, this was the first time that I have used the software and was definitely amazed by how powerful it is in handling such a heavy scene full of foliage (in real time as well). That being said, it took me experimenting on a dozen of projects before I was finally satisfied with what I created, which eventually helped me get more fluent with the software itself. If I have had more time or had a clearer vision in mind, I would have modelled more of my own objects and learned how to texture them realisticly to put into the scene. However, I optimized by playing around with every single material in the scene, from static meshes to foliage to landscape material. This has definitely made me feel more confident on working with models and materials in Unreal. Moreover, I have learned the workflow of importing Alembic into Unreal for animation. The most time consuming part of this was setting up and applying the material for every single parts of the model. In the future, I want to learn how to animate properly and using FBX or USD format to understand the workflow more intensively, also to give myself more freedom in posing and animating the characters.
Zbrush/Substance Painter/Daz
Throughout this project, I have strengthen my skill and love for 3D art when I got to learn my 2 favourite softwares Zbrush and Substance Painter. The leaf boat in particular was something so simple yet I put much effort to make it my own creation. I definitely have spent way more time than I needed to on the sculpting of the boat. If I had to do it again, I would have painted the veins of the leaf with Substance Painter. Yet the workflow that I took luckily taught me valuable lessons in how to work with high & low poly in Zbrush, as well as the baking process before moving to texturing. Considering what I have learned for this, I consider it as a win 🙂
Other used softwares:
Blender: Modelling, Particle system, Animation
Nuke: Compositing the bubbles
Photoshop: Texture creation and editting
Premiere Pro: Final Video editing and rendering
What I want to improve in the future projects:
Apart from strengthening my skill in animation, modelling/sculpting/texturing, I definitely want to be more mindful about the filmmaking/cinematic aspect in the future. I think the final video turned out beautifully, yet it lacks storytelling. I believe if I had solidify a vision earlier on, I would have spent more time on planning, writing a script and making a storyboard. This is typically a workflow known in the industry as well, so eventhough I have always been working purely based on intuition and experimentation, I need to improve on this so I can create more impactful visuals and for anyone who wants to understand my creative process better.
Furthermore, I want to use Nuke more in my future projects since I think it’s a very powerful software. However I have considered its uses in this project and thought it would be better if I know how to to 3D tracking and compositing due to the light and shadow of the scene (which we haven’t fully learned on yet). Luckily the bubble worked out perfectly as it has a complex shader that I have set up, which would be hard to transfer from Blender to Unreal, therefore the use of Nuke in this case is justified and helped blending the bubbles into the scene nicely.
After all, thank you so much for all the help from my tutors to make this project happened!
Final Project P3: Camera Sequence & rendering
Working with sequences and rendering process was pretty easy once I got a hang of how to do it. My decision was to work with a 35mm film format as I was inspired by Wes Anderson’s movie style. I believe it added to the fantasy film look, like a fairytale coming to life. I also chose a prime lens and heavily dependent on camera movement to film my final edits.
How to work with sequences:
- Add level sequence > Save it into a Sequence folder
- Drag camera into timeline
- Animate camera using keyframes
I actually was testing my render, expecting some problems to come up that needs troubleshooting. To my surprise, not only my render came out beautifully, the character hair appeared much nicer compairing to the viewport, which was a problem that I was really trying to fix through shaders. After knowing that the rendering process will be good, I came back and spent some time fixing the character’s materials and makeup, particularly the eyes , which kept being black at first. I resolved this by using a different texture.
I decided to render my scene in EXR sequences format as it ensures high quality render and due to the fact that I will later import the sequences into Nuke for further compositing.
Since I wanted to add some bubbles to the scene, I thought it would be better to try using Nuke for this. So I created some bubble animation using Blender’s particle system, then rendered it with a transparent background.
Resources
Final Project P2: Character and Butterflies Creation
As time was restricted, I decided to use my favourite software Daz for human model creation. I have morphed my own characters so it was very convenient to pose them exactly how I wanted. The model also comes with its own textures that I can use later in Unreal Engine. My idea behind the pose is to later situated the model inside the leaf boat, then add some butterflies into the scene. I also wanted some movement so the model doesn’t look like a static mesh, so I animated the eye blink movement, tried my best to make them dynamic and as lively as possible. If I had more time I would have animated the whole body as well, but I feel good about the result nevertheless.
I was not particularly happy with the fact that Alembic import didn’t name or sort out the material, so it took me quite a bit of time to assemble and set up all the materials one by one. Luckily, I have worked with Daz model before in Blender and there is a particular order to each body part, so it did help me in guessing which one is which. Initially I got frustrated with this so I tried out different technique to import the animation in, from using Daz to Unreal plugin (which didn’t work), to FBX, etc… I decided that nothing looks better than the Alembic route so I went back and tried to finish.
After setting up all the materials manually, there were two problems arised.
- Firstly, the hair was really patchy and not looking great at all, especially at the root and area that has heavier shadow. As I imported the whole model in as a single alembic, there was no way I could separated the hair from the body for modification. I eventually took the time and imported a separated model without the hair first. Then I imported the hair into Blender and converted it to a particle system, exported as Alembic and imported again into Unreal Engine. The hair is now read with UE5 Grooming system, so i have more options to adjust the hair roots and tips until it looks better
- Secondly, the eyes textures turns black after importing into UE5. I resolved this by finding a few different textures to test until it worked out. I also had to set the Cornea material to transparent in order for the Irises to show.
I created two simple butterflies animation in Blender, using pictures I found online. Then I put them into Photoshop to create their normal maps and opacity maps. I would say this is one of my favourite technique to learn and can be used in so many cases in the future.
Resources:
Nuke W10 – 3D tracking
3D classic vs 3D beta
- Classic still incorporate FBX or Alembic. Nodes are green.
- Beta adapt to USD workflow. Nodes are red.
- You can’t mix nodes from the classic 3D system and the new 3D system. There are a few notable exceptions to this rule, like CameraTracker and DepthGenerator. Classic 3D system nodes are green by default and new 3D nodes are red.
- The red new version is still quite unstable, we should use the classic version for now.
3D navigation
Tab in viewer – switch between 2D and 3D
Middle mouse – Pan
Alt + right click – Orbit
Scroll – Zoom
Scene Node: is used to manage and assemble 3D geometry, lights, and cameras. It acts as a container/collector for all your 3D elements. When you connect multiple 3D objects to a Scene node, you can manipulate them as a single unit, which simplifies the process of working with complex 3D composites.
Camera Node: In Nuke, the Camera node is a virtual representation of a real or imaginary camera. It’s used in 3D compositing to define the perspective and projection of a scene. You can adjust parameters like focal length, aperture, and field of view. This node is crucial for match-moving and integrating 3D elements into 2D footage, as it helps in replicating the original camera movements and settings.
ScanlineRender Node: This is a rendering node used to render 3D geometry in a scene. It works by converting the 3D data into a 2D image, using the settings from the Camera node
Lens Distortion
Resources:
Working with Lens Distortion: https://learn.foundry.com/nuke/content/comp_environment/lens_distortion/adding_removing_lens_distortion.html
Lens distortion refers to the way a camera lens can warp or distort the image it captures. This isn’t necessarily a flaw; in fact, it’s often a characteristic of the lens design. There are mainly two types of lens distortions: barrel distortion and pincushion distortion.
- Barrel Distortion: This occurs mostly in wide-angle lenses. It makes the image appear bulged outwards from the center. Think of how a fisheye lens makes things look curved and expansive. It’s often used for artistic effect or to create a sense of immersion.
- Pincushion Distortion: The opposite of barrel distortion, this happens more in telephoto lenses. The image appears pinched at the center, making the edges seem to bow inwards. It’s less common in everyday photography and filmmaking.
LensDistortion Node: This node is used for correcting or applying lens distortion. It allows you to analyze an image for any barrel or pincushion distortion and correct it, or you can intentionally add distortion to match CG elements with live-action footage
It’s important to use LensDistortion (Undistort) before doing many match-move or tracking in order to work with correct information then use LensDistortion (Redistort) afterward to return the plate to correct format.
LensDistortion > Analysis > Detect or Draw (Straight Lines) > Solve
Draw horizontal and vertical lines to help giving as much information to Undistort as possible, especially around the edges. Then click Solve.
LensDistortion (StMap) introduce “forward” and “backward” as an option to view and is lighter to use & faster to view.
3D tracking
- Denoise your plate then treat it (color grade etc)
- Roto out area that you want to avoid tracking (things that move/not static, be mindful of reflective objects)
CameraTracker analyses the motion in a 2D footage and extrapolates this movement into a 3D camera path. It tracks various points in the footage (usually high-contrast or distinctive features) across frames to determine how the camera was moving when the footage was shot.
Source: sequence or still
Autotrack: to improve tracking
Click on error-max. click on graph and press F
reduce Max error to 6, click delete unresolved and delete rejected
Usually solved error is anywhere around 1 or below is good
To check tracking:
- Select some point > Create > Cube/Plane
- Plug the object Card into Scene > move it to match the ground plane
3D: UV Unwrap, Baking & Texturing
UV Unwrap
After sculpting the leaf boat, I imported the low poly model into blender and tried to UV unwrapped it. I found this tutorial that showed how to UV unwrap complex object using a tool called “Seam from island” that helped me create seams from separated parts of the model that I have previously unwrapped individually. Since this is my first time trying out UV unwrapping, I don’t think this was the best result, but it’s good enough for me to import into Substance Painter and start texturing the model.
Texturing using Substance Painter
At this point I was wondering why the holes and cuts on the leaf didn’t show up in the viewport, eventhough I saw that it baked out an opacity map. Tempted by the question, I found out that the solution is to add an opacity channel and also to change the Shader Setting into PBR-metal-rough-with-alpha in order for the opacity map to appear properly.
Leaf texture references
I have decided to go with variation 2 as it matches the overall look of my UE5 scene more.
3D: Modelling & Sculpting Leaf Boat
While searching for inspiration for a boat design to add into my scene, I found this illustration of a Leaf Boat that perfectly matches my vision in terms of simplicity and nature-theme related. I have always wanted to practice with organic shapes and texture, so this can be a fun idea to try out.
Some other illustation that inspired the vibe I want to go for
Keywords:
- Pastel colour
- Nature
- Light Fantasy
- Flow
- Dreamlike
- Fairycore
- Mushroom
For me the most challenging part in the modelling process was figuring out how to make this end bit of the leaf that looks like a tree trunk.
I try to keep this as a low poly version as I will sculpt another detailed high poly version before moving to texturing.
After modelling for awhile, I have an idea of animating the leaf to open up and revealing something inside. So I created a second leaf and put it on top. I might make a character and put her inside the boat.
<– I really like the flow curves of the leaves
I then imported the rough model into Zbrush for further modification before sculpting in the detailed version
At this point, I notice that the edges of the cut-off mesh has these jagged problems. So I have found a solution by using
Deformation > Polish by features (with a closed O)
This has helped cleaned up the holes and cuts.
When I’m happy with the overall mesh, I proceeded to researching on leaf textures. I don’t want to overdo it as it’s only a stylized leaf boat, but adding some textures on it would help it matches with my realistic scene in Unreal better.
After a few trial and errors, I have successful sculpted the details onto the leaf. I used Damstandard for the veins as it resulted in more smoothed lines than Standard or Inflate brushes. Then I used Blob to further texturing the rest of the leaf.
Overall this process took me a lot of time, yet I have learned so much through experimenting with different technique. I learned about how to use Zremesher to retpologize my mesh, tried out projecting details from high to low poly (which I ended up didn’t use), how to sculpt with mask and regular brushes in Zbrush, etc…
This was probably the most fun and satisfying part, sculpting the veins on the leaves is pretty abstract yet I figured that they do follow a sort of pattern.
I am overall satisfied with the result so far, I will then move on to Substance Painter to bake the details and paint on this model.
Reference: