Loading...
PB009: Lecture 1

Introduction

This lecture discusses the scope of computer graphics as a research field, its brief history, and its use today.

The second part of the lecture introduces overview of the typical tasks of computer graphics and their use in image rendering.

What is computer graphics?

There are several definitions of what encompasses computer graphics. In general, it can be defined as the use of computers to synthesize visual information. It is an inverse problem to computer vision, which deals with the extraction of digital information from visual input (i.e., image analysis), though the two fields are complementary and often employ similar methods. Over the years, computer graphics has found application in many areas. Besides movie and gaming industries (which are perhaps the main driving forces in graphical research), it is used in computer-aided engineering and manufacturing, medicine, molecular biology, art or design. With the development of new technologies, its application has expanded beyond standard 2D screens and purely visual information. For example, designs from 3D modeling software can nowadays be 3D printed. Medical simulations can be connected to haptic feedback devices and virtual reality (VR) devices to facilitate the training of medical personnel. Such applications provide not only visual but also other sensory stimuli such as touch and sound. We can thus revise the definition to say that computer graphics is the use of computation to turn digital information into sensory stimuli.

Naturally, with the diversification of computer graphics, several adjacent research fields emerged, including human-computer information (HCI) and visualization. These fields have a large overlap but in general, core computer graphics is typically focused on the process of stimuli synthesis (e.g., digital image generation), while HCI focuses on interaction (e.g., design of suitable user interface), and in visualization, the focus is often on determining the desired visual outcome (e.g., designing suitable representation of high-dimensional data). Nevertheless, these fields often rely on computer graphics principles. Similarly, computer graphics often draws on principles from other research fields, particularly physics, in an effort to mimic real-life stimuli (e.g., simulation of fluids, clothing movement, crowd behaviors for games, creation of realistic light effects., etc.).

History

Modern computer graphics is not limited to 2D screens, but its origins are inherently tied to the development of the first digital displays. These originated in 1897 with the idea of using cathode ray tubes as a display device. CRTs are vacuum tubes containing an electron gun and enable the manipulation of emitted electron beams. When the beam was directed towards a fluorescent screen, the electron-stimulated phosphorus glowed. By directing the beam across the screen, images were traced. The CRT tube used in this way was proposed by Ferdinand Barun and named Braun’s tube after its inventor. It laid the foundations for the first TVs and digital displays. Due to the nature of the early displays the graphical information to be drawn needed to be translated to directional commands such as MoveTo(X,Y)). In some of the displays, commands were cyclically repeated, to avoid fading of the image. The early graphical systems thus used vector graphics, where images are represented by lines that could be easily traced by the electron beam, rather than individual pixels.

crt
Cathode ray tube scheme (© Theresa Knott, CC BY-SA 3.0)
crt
Back of CRT TV (© Raimond Spekking , CC BY-SA 4.0)

Despite the invention of Braun’s tube in 1897, computer graphics did not gather much attention until the 1950s, when the first pointing devices (in the form of a light pen) for interaction with graphical displays were created. This gave rise to the development of graphical user interfaces and showed the potential of graphical applications for many differnet uses. A pioneering program in this area was Sketchpad by Ivan Sutherland presented in 1963. It can be seen in the video below (note the discussion of hidden lines when rotating the object around 6:40 - this problem will come up in later lectures). Fun fact: Mr. Sutherland also invented the first head-mounted display in 1966 but it took another ~50 years for the technology to become mature enough for commercial success.

Demonstration of graphical user interface of Sketchpad program, an ancestor of modern computer-aided deign (CAD) programs.

The vector-based nature of the early displays naturally limited the complexity of the images that could be drawn - e.g., filled areas with gradient fills were not possible. The technology was thus adjusted (in the 1960s) to move the electron beam across the entire screen in a grid-like raster (meaning rake) scan pattern, with the power of the electron beam being modulated as required to create ligter and darker areas. This led to the development of raster graphics, where images are represented as cells (pixels) on a regular grid. Color was later added by mixing primary colors (red, green, and blue with separate electron beams) on a sub-pixel level - more on that in later lectures. The CRT technology was eventually superseded by LCD (introduced in the 1960s but became widely used in TVs and computer screens in the 2000s) and then LED (and OLED) screens, which differ in the way light is emmited, but all modern screens utilize the principle of a raster grid.

raster scan
The term raster comes from raster (meaning “rake”) scan pattern of electron beam in old CRT displays
(Image source: https://www.tutorialspoint.com/)
lcd
Closeup of LCD display (© Ravedave , CC BY-SA 3.0)

Raster vs. Vector Graphics

While vector-based displays all but disappeared, the vector representation of graphical data still has many uses. The main advantage of vector representations is the continuity. Objects are presented with points, connected lines, and curved paths, that can be mathematically described. The representation is precise and independent of resolution. It is also fairly memory efficient. Such representations are very useful, e.g., for storing technical drawings, fonts, and graphics that needs to be scalable, such as logos. Printers and plotters thus often use PostScript protocol, which sends the documents to the printer in vector format if available, so the printer can scale it as needed when printing. Vector formats are also useful in manufacturing, since modern CNC machines work in a similar manner to old vector CRT displays - the cutter/laser receives directional commands where to move and cut. Another area of vector graphics application is web graphics. Here in particular the SVG (scalable vector graphics) is used. This format uses XML syntax to specify graphical elements as shown in an example below. Other image formats supporting vector data include CorelDraw!™ (CDR), AutoCAD™ (DXF), Adobe Illustrator™ (AI), Adobe PDF™, PostScript™, Windows Metafile (WMF).

circle
SVG representation of red circle centered at position (50,50) of 100x100 canvas, with radius 40 and black outline.

Contrary to this, raster graphics is limited by the resolution of the raster grid (also called ‘bitmap’). Image is represented as a grid of pixels storing information about colors. Compression strategies are used to reduce the memory footprint of such representations, however, the size is directly related to the resolution of the grid. Despite these limitations, it is suitable for representing complex images, it is in line with digital image acquisition methods (i.e., digital cameras capture the pictures in the discrete form - more on that in course PA172 Image Acquisition), and with modern displays. There are many different raster formats and representations, which can be split according to criteria, such as employed color model (additive RGB or substractive CMYK, more on that later), pixel format (i.e., how many bits per color channel is allowed and how are they attributed), or compression (loselss vs. lossy). Some of the most common raster image formats include MS-Windows Bitmap (BMP), Portable Network Graphics (PNG), Graphics Interchange Format (GIF), Interchange File Format (IFF), JFIF (JPG), PBM/PGM/PPM/PFM, Macintosh (PICT), Targa (TGA), Tagged Image File Format (TIFF).

raster
Difference between raster and vector representation of font (Image source: https://signalizenj.wordpress.com/)

Since both vector and raster graphics formats are commonly used, we often need a way to convert from one to another (e.g., to display SVG graphics on a modern raster display or to print it). The process of converting vector graphics to raster is called rasterization and it is one of the fundamental problems of computer graphics. It involves sampling of the continuous (vector) data in a regular grid pattern. In later lectures we will cover several algorithms that deal with this problem. The opposite direction is vectorization, which often involves image tracing with continuous contours. It is used e.g., to convert satellite images into maps or by a designer to convert scanned hand sketches to vector images.

Fundamental Tasks of CG

And what about 3D data? Like 2D images, they have many varying representations, which we will discuss in detail in following lectures. However, perhaps the most common representation of 3D models is the polygonal mesh. It consists of points and lines that connect them to create triangles or other polygons. So it is essentially a vector representation and as such also requires rasterization. But besides the rasterization, there are several other issues that need to be resolved to display a 3D scene on 2D screen. In the video demonstrating Sketchpad, one of them is already highlighted - how to decide what should be visible? The research that followed the transition to raster-based displays laid the ground for computer graphics as we know it nowadays and resolved these problems as they were emerging. We can group these issues into several areas:

Modeling and representation of graphical data for storage, processing, and rendering - suitable data formats and data structures, taking into consideration requirements such as precision, intended use (e.g., rendering, computational analysis, 3D printing…), available memory and processing power, etc.

Scene construction - positioning of models in virtual space (model transformation), animation, or simulation of movement

Capturing the 3D scene and its transformation to 2D - camera positioning (view transformation); conversion from 3D coordinates to 2D considering perspective, field of view, or special effects such as fish-eye lens (projection)

Visibility, clipping, and culling - correctly determining drawing order so objects or their parts that should appear on screen in the back do not occlude objects in the front; removal of hidden objects to speed up the rendering process

Rasterization - conversion of continuous (vector-based) data - e.g., mesh triangles given by vertices and lines that connect them - to raster grid

Lighting and shading - modeling realistic illumination of the scene and subsequent shading of the objects; adjacent problems include texturing and shadow generation

They can be roughly organized in the consecutive steps illustrated in the image below. The word ‘roughly’ is key here, as some operations can be performed at different stages of the image synthesis. For example, operations concerning visibility or lighting - while the computations are based on 3D positions of objects, camera, and light in the scene before projection to 2D, for practical reasons some of these steps are often performed after rasterization.

circle
Typicall tasks of image synthesis process.

The following image illustrates how can these tasks be mapped to the rendering pipeline - a series of steps and operations performed by graphical frameworks such as OpenGL to render an image. The complete rendering pipeline is covered in more detail in the course PV112 Computer Graphics API. Here we provide just a brief overview to illustrate how are the operations and tasks that will be discussed throughout this course used in modern graphical pipeline:

circle
Rendering piplene of OpenGL API.

The pipeline starts with the processing of vertex data (positions, normals, etc.) which are transformed to their projected positions in a so-called vertex shader. Computation of some necessary data for lighting and shading on vertex level also happens at this stage, but this information is typically passed down the pipeline for later use.

Once the vertices are transformed to their final positions, they are assembled into graphical primitives (lines, triangles, …). Primitives and parts of primitives located outside of the rendered are removed (by so-called clipping).

The remaining primitives are rasterized, i.e., converted into so-called fragments (essentially pixels of the screen raster grid, but with additional information, such as depth, more on that later).

Fragments are then lit, shaded, and (optionally) textured using information that was passed down from the vertex shader and (if applicable) texture memory in so-called fragment shader.

It should also be noted, that there are different rendering approaches, such as ray tracing, that take a different route to produce rasterized images. They are typically computationally more demanding, but can produce more realistic results. We will discuss those towards the end of this course.