Scaling is everywhere, pixel-perfect is the past

Long time ago, when computers were slow and screen resolutions were low, everything had to be pixel-perfect. For example, Atari 2600 game console could display only 160x192 pixels. During that time, game characters and all the graphics had to be drawn pixel by pixel, to include all the intended details, like Mario's moustache. This is known as pixel art.

Source: The Evolution of Mario

Years later, with higher screen resolutions, game sprites could be drawn using different methods, or even rendered from 3D models, but icons and other GUI elements were still prepared to be shown pixel-by-pixel. Same applied to web pages.

Nowadays, even GUI icons are scaled. They can be enlarged smoothly and they can be displayed on various monitors, where 4K monitor has 4x more pixels than FullHD. Setting desktop DPI scaling other than 100% scales all the apps in Windows. Modern web pages created according to "responsive design" principles have to look good on all kinds of devices, from little smartphones to huge monitors. Scaling is everywhere.

When programmable cellphones first appeared, making apps and games for them was like going back in time. Just like on retro platforms and first PCs, screens had very low resolutions and pixel art was the way to go when drawing game characters. Now mobile games have to work on all sorts of smartphones, many of them having resolutions like our PC monitors - FullHD or even higher.

What seems like the last relic of pixel-perfection is the rendering of 3D scenes. Since the introduction of 3D graphics, we tend to rasterize and shade our triangles in the same resolution as the image to be displayed on screen, which is ideally equal to native resolution of the monitor. Otherwise, every gamer who cares about image quality would call it looking bad. Or wouldn't he?

Some things can be rendered in lower resolution. There are games that render the layer with alpha-blended, translucent objects (especially particle effects like fire, smoke, clouds) to a 4x smaller texture and then upscale it while compositing with main, opaque geometry. Such elements tend not to have too many high-frequency (small) details anyway, so quality degradation due to lower resolution is not very noticeable, while smaller number of pixels that need to be shaded and blended saves a lot of rendering time.

But that's not the full story. Regardless of resolution, antialiasing is, and always will be, necessary to blur jaggy edges. Ideal solution for it is known as Super Sampling Anti-Aliasing (SSAA), which is nothing else but rendering the scene in higher resolution and then downscaling it to e.g. average 2x2 rendered pixels into a single output pixel. It could be done by a game, or introduced by graphics driver. AMD has this feature in driver under the name "Virtual Super Resolution".

This of course is a slow method, because rendering 4x more pixels requires a lot of computations and memory bandwidth. Various methods exist that provide more efficient antialiasing. Multisample Anti-Aliasing (MSAA), which is supported by GPUs in hardware, lets you shade a pixel (calculate RGB color in a pixel shader) only once, but store it in multiple per-pixel samples, depending on the shape of the edge being rendered. Numerous screen-space postprocessing algorithms exist that intelligently blur already rendered image to smooth the edges, e.g. FXAA, MLAA.

This interchangeability between rendering in higher resolution and higher quality antialiasing, as well as the possibility to do some filtering of the rendered image, is probably best exploited by the engine behind Call of Duty. Jorge Jimenez (Graphics R&D Technical Director at Activision Blizzard) explained it in his talks: "Dynamic temporal antialiasing and upsampling in Call of Duty" (Digital Dragons 2017), "Dynamic Temporal Antialiasing in Call of Duty: Infinite Warfare" (SIGGRAPH 2017). They dynamically scale rendering resolution depending on current game load to maintain sufficient framerate. The scene is then scaled to screen resolution. Their technique "combines dynamic resolution with temporal upsampling". Such techniques are especially useful where high FPS and smooth gameplay is important, even at the expense of graphics quality - in fast-paced games, professional e-sport, and VR.

Screen resolutions become higher, but performance of GPUs don't necessarily scale at the same rate. Single-pixel details are harder to notice. That's why it can make sense to render at resolutions even smaller than output resolution and then interpolate missing pixels. Of course, no interpolation algorithm is perfect and using just bilinear filter would look horrible. That's why techniques are being developed which try to minimize quality loss in this process, e.g. temporal methods (that use image from the previous frame), checkerboard rendering, or new Deep Learning Super Sampling (DLSS) from NVIDIA.

It also makes sense to shade pixels at lower rate in some parts of the image where details are hard to notice, e.g. where the player is not looking (peripheral vision in VR, especially if eye tracking is available), objects are moving fast (based on screen-space motion vectors) or where there are not many high-frequency details (based on analysis of the previous frame). Shading per pixel or per sample is just one option. NVIDIA cards support techniques like Multi-Res Shading or their latest invention - Variable Rate Shading (VFR), where helper texture can locally control shading rate from once per 16 pixels all the way to 8 times per pixel.

Finally, the rate of shading (lighting calculation) can be completely decoupled from the rate of rendering of the final image (rasterization) and done in different space, at different framerate or even completely asynchronously. This is known as Object-Space Shading/Texture-Space Shading. It has successfully been used by Oxide Games in their Ashes of the Singularity and may soon become more widespread.

I think we could say that scaling is everywhere, pixel-perfect is the past. It is not necessarily a bad thing. If the goal of advancements in 3D rendering in games is to look photorealistically like movies, then we should realize that movies are never pixel-perfect - there is always scaling and filtering involved at various stages. Even at the very beginning, camera sensors have some pattern of R, G, B pixels that must be interpolated to fit them into (RGB) triplets.

Then they are often encoded using chroma subsampling (like 4:2:2) and compressed using some video compression codecs. Interpolation and filtering may be involved at many stages of processing, e.g. frame rate conversion, deinterlace, noise reduction, or finally, sharpening commonly applied by modern smart TVs (which I'm very allergic to, but there must be some reason behind it). Recorded videos are never pixel perfect. Rendered 3D games don't have to be as well.