Tangent Basis #
Most systems that render with normal maps use a tangent space, a system of coordinates that specifies the orientation of the surface at each point on the model. Tangent spaces are defined by the tangent, bitangent, and normal vectors at each vertex of the mesh. Normal maps contain RGB values that correspond to XYZ coordinates within this space. The exact orientation of tangent space vectors is, therefore, vital both during baking (when a normal map is created), and rendering (when a normal map is used). If different tangent spaces are used in each step, the orientation of the normals in the map will no longer match those of the original model, and potentially serious shading artifacts will result.
There are various tangent space normal mapping implementations in different 3D applications and game engines. Remember to bake your normal map with the same tangent space your renderer uses to ensure your asset shades correctly. If you’re unsure which tangent space to use, Mikk / xNormal is a good choice, as that is what Unity, the Unreal Engine, and various other applications use.
When baking in Toolbag, the tangent space your mesh is currently set to will be used. Use the Tangent Space dropdown in the mesh properties to set the tangent space. You can also set the default tangent space by going to Edit-> Preferences -> Content -> Default Tangent Space.
Note: this setting will apply to newly imported meshes, not the meshes currently in your scene.
If you don’t have a synced baking workflow, for instance, if you don’t know the tangent space that your renderer uses, or if the tangent space you need is not supported, keep that in mind when creating your low poly mesh. Generally, this means adding bevels and/or hard edges to ensure that the shading errors are less extreme.
Handedness #
Handedness or the orientation of the normal map’s Y axis or green channel can be problematic. When it comes to debugging normal map problems, handedness is perhaps the most common issue that artists face. Whether you’re having trouble getting your normal map to display correctly in Toolbag or you’re seeing errors in your Toolbag-baked normal map in your target renderer, verifying that your handedness is set correctly is a good place to start.
Various 3D applications and game engines use either right-handedness (Y+) or left-handedness (Y-). Right-handedness is sometimes called OpenGL or Maya style, while left-handedness is called DirectX or 3DS Max style. Similar to tangent space, it’s important to ensure you’re using the same handedness when baking and rendering your asset.
In Toolbag, handedness is automatically determined by your tangent space setting. However, handedness can be overridden via the normal map settings in the Baker object by clicking the Flip Y checkbox. Handedness can be overridden for display purposes in the material editor. You can invert the green channel of your normal map in an image editing application to flip the handedness too.