Assignment Goals

The primary goals of this assignment are:

  1. Make larger-scale changes to the core UE4 engine source code.
  2. Find and integrate relevant sample code.
  3. Add a post processing effect that modifies scene color.

For this assignment, you'll be creating a new post processing pass for the Unreal Engine. This pass will filter a rendered scene to produce a cartoon-like outline.

→
Normally rendered scene Scene processed to cartoon look
Grad/Extra Credit version
(These and all images in this project description are linked to full-sized versions)

Though I've given some background below on the functioning of this part of UE4, there are several ways to achieve the desired result, and none of available sample code does exactly what you need. You will need to look through the post processing passes, and come to enough of an understanding of what they do to choose an appropriate model pass and modify it to your own pass.

UE4 Postprocessing Overview

Postprocessing passes take one or more input images (the scene color from passes that have been run already, per-pixel depths, G-buffers, or others). They run a shader on each pixel to produce a new image or images for later passes to consume. The order of postprocessing passes, and which ones to execute, are determined in FPostProcessing::Process(). Before each pass, Context.FinalOutput contains the primary result of the preceeding pass. The new pass is registered with Context.Graph.RegisterPass(), the inputs are hooked up with TRenderingCompositePassBase::SetInput(). After the pass, Context.FinalOutput is updated to the new pass results. To insert a new optional pass, you create a console variable with TAutoConsoleVariable to give you a flag you can turn on and off from the console, then put the code to register the pass at the appropriate place in FPostProcessing::Process(). Our pass will run immediately before temporal antialiasing .

Every postprocessing pass includes one or more classes to set up shaders. You should not need to make your own vertex shader, but will need to set up your own pixel or compute shader. For pixel shader passes, the built-in FPostProcessVS vertex shader will work, and compute shader passes do not use a vertex shader. Every shader setup class includes several key functions. ShouldCompilePermutation tells whether a shader is supported (e.g. if it won't work on mobile). ModifyCompilationEnvironment sets any shader #defines to control how the shader will be compiled. For passes that use this, the shader setup class is usually templated to set up different versions with different choices for #define symbols. Every variant is an extra shader permutation to compile when shader code changes. The constructor hooks up the shader parameters to their name in the shader code, and there's usualy   a separate SetParameters function to set the values for those shader parameters, though you will some passes do this in their Process code. Finally, the auto-generated serialization code doesn't work for these classes, so there's a Serialize function to serialize the shader state. This shader class will be passed into a IMPLEMENT_SHADER_TYPE or IMPLEMENT_GLOBAL_SHADER macro with the name of the shader code file and function within that file to use.

The rest of the code for main class includes a Process function to run the pass, and a ComputeOutputDesc function that reports the name, size, and format of the pass output. You will find examples of Process code using DrawPostProcessPass, DrawRectangle (both for pixel shader passes), or DispatchComputeShader (for a compute shader pass). Any of these is fine.

When testing shader changes, be sure to turn on r.ShaderDevelopmentMode in Engine/Config/ConsoleVariables.ini (there's a line already in there, just uncomment it). This will pop up a dialog box when there's an error in your shader code, rather than crashing the engine. You want to set this in the ini file, since global shaders are compiled when you start the engine, so you won't be able to launch the engine without it if there are any global shader code errors.  Once running, you can recompile shaders with the recompileshaders changed console command (Windows shortcut: control-shift-., Mac shortcut: command-shift-.)

Cartoon Overview

For your cartoon outline look, you will run a Sobel edge detector on the scene depth. This does weighted sums over a 3x3 neighborhood of pixels to find discontinuities. Wherever there is edge in the depth image, you'll use the outline color, otherwise, you'll use the current scene color.

Scene depths Just the depth outlines Depth outlines on original scene

Grad Overview

Grad students should also use a Sobel edge detector on the G-buffer normal. Depth edges alone can't find lines where two objects meet, or at sharp features in a 3D shape, since they're at the same depth there. Normals will tend to have discontinuities at those features, so can catch those extra lines. Normals alone also don't catch everything, since two parallel surfaces (like a table parallel to the floor) will have the same normal, but will have a discontinuity in depth. To catch more edges, you want to draw a line if there is an edge in either buffer.

Scene normals Just the lines from normals Normal lines on the original scene
Both sets of lines Final cartoon scene

Extra Credit

For up to 10 points of extra credit, only available to undergraduates and only available for on-time submisisons: complete the graduate part of the assignment.

Details

Create a project

  1. Create a a Blank Blueprint project with no starter content called assn6.
    • As usual, put the project at the top level of your git repository
  2. Add some shapes to it. For grad students, this should include at least one cube.

Clone a pass

  1. Find a simple existing postprocessing pass to use as a model
  2. Make copies of the .h, .cpp, and .usf files and rename any functions and variables to avoid name conflicts
    • Don't hook it into PostProcessing::Process() yet
  3. Re-run GenerateProjectFiles, then re-launch your IDE and launch your project
    • This will rebuild the UE4 Rendering module, and re-link any other module libraries that use it.
    • If it doesn't build, you probably missed something when renaming (typo or multiply defined symbol). Fix any of these. You should be able to build and run the engine at this stage.
  4. Hook into PostProcessing.cpp
    • Make a CVar to turn your pass on and off
    • Add the code to register the pass
    • If you change the shader to return a solid color (e.g. float4(0,1,0,1)), it'll be easy to tell if it is running when you set your CVar.
    • (Temporarily) turning off optimization in your pass and in PostProcessing.cpp can also be useful to debug whether your pass is being switched on when you want
  5. Adapt the cpp and header code
    • Adjust the pass inputs to what you need (two inputs for color and depth).
    • Adjust the shader parameters to what you need (if any, though it may be handy to create additional CVars to tweak the Sobel thresholds).
    • Remove any unneeded shader variants or compilation environment / define flags.
    • Get rid of any other obviously unnecessary code for your pass.
    • Compile / run / commit a bunch along the way here. At each stage, it should still run.

From this point on, most of your changes will be to the shader code. You should be able to develop by recompiling shaders in the engine without having to rebuild or restart the engine.

Make the outlines

  1. Output scene depth as color
    • This makes sure the depth buffer is hooked up correctly and your buffer lookup is correct
  2. Implement the Sobel edge detector
    • Display as color first to debug
    • Use the step or smoothstep function to adjust the sensitivity and make sure the results are in the range 0-1.
  3. lerp between the base color and outline color based on the detected edge

Grad only: make depth outlines

  1. GBuffers are not hooked up to postprocessing passes in C++ code the same way as other buffers. Find an example and adjust the pass code
  2. GBuffers are not accessed in the shader as PostProcessInputN the way other typical postprocessing are. Helper functions only look up the current pixel, not adjacent pixels. Find shader code that directly accesses the GBuffer textures and adjust the pass shader code
  3. Add Sobel edge detection for normals. You'll need a different threshold than you did for depths to get good looking outlines

Submission

For full credit, you must commit multiple times during your development.

Add an assn7.txt. Tell us what works and what doesn't, and anything else you think we should know for grading. Be sure to include the names of your console variables and what values to try. Include a link to video demonstrating your project.

Push to your repository, and tag your final commit with an assn7 tag.