Pixel Data

Overview

Access pixel values directly to process and generate images.

Tools

p5.js

Pixels

Today, most computers use color displays that produce images using a grid of pixels. Conceptually, each pixel is made up of three sub-pixels—red, green, and blue—though the actual hardware may use a different pattern. Because these pixels are very close together, the image appears continuous. Because our eyes can only directly sense red, green, and blue light, the screen can reproduce most of the colors we are capable of seeing.

Pixels Photo by Peter Halasz

When you work with a graphics library like p5.js or the Javascript Canvas API, you don’t have to think about individual pixels. You can use high-level methods like ellipse() and image(), and the library will do the low-level work of setting individual pixel values for you. The process of converting shapes to pixels is called rasterization. Because the p5.js drawing calls handle the details of rasterization for you, you give up some control. When you draw a circle with ellipse() every pixel in the circle will be changed to the current fill() color. ellipse() provides no way to fill the ellipse with random colors, or a gradient, or colors generated by a function. To get this level of control, you need to take control of the rasterization yourself. This is the classic high-level/low-level trade-off. Coding at a low level takes more work but offers more control.

When you work at a high level you are not responsible for the details. Because you are not responsible for the details you tend to consider them less. Working directly with the pixels by reading and writing their red, green, and blue values one-by-one leads to thinking about drawing with code differently.

Vector Monitors

Computer displays don’t have to use pixels at all. One of the earliest video games, Tennis for Two, used a cathode ray oscilloscope as its display. In a cathode ray oscilloscope, signals are drawn by magnetically deflecting a beam of electrons as they race from an emitter—the cathode—towards a phosphor-coated glass screen. The phosphor glows where the electrons strike it, creating an image. This type of display can show smooth lines without any of the aliasing artifacts of a pixel display.

Vector Monitors were used in later video games as well, including the Tempest arcade game and several games for the Vectrex home console.

Tennis for Two

Video Memory

Modern video pipelines are fairly complicated, but at a basic level they work something like this: The red, green, and blue brightness values of every pixel on a display are stored in memory. This data is usually stored on the video card’s VRAM, but might also be stored in the computer’s main RAM. The video hardware repeatedly runs through this data, pixel by pixel, and sends it to the display over a display interface like DVI or HDMI. Hardware in the display receives this data and updates the brightness of each pixel as needed. When you change the values in the RAM, a new picture appears on the screen.

A high-definition display is 1920 pixels wide and 1080 tall, for 345,600 total pixels. Each pixel needs three bytes to represent its color value: one byte each for the red, green, and blue channels. In total that is 6,220,800 bytes—about 6 megabytes—of memory to keep track of the full HD image.

Today, 6 megabytes isn’t much, but many older computers didn’t have enough RAM to keep a full-color image of the screen in memory at all. They used a variety of tricks instead.

The Pixels Array

The memory storing the image shown on the screen is called the video buffer or framebuffer. Direct access to the screen’s framebuffer is pretty unusual on modern computers, and libraries like p5.js don’t (can’t) provide it. P5.js does give you access to a pixel buffer storing the image shown on your sketch’s canvas. When you call drawing functions like rect() and ellipse(), p5.js updates the appropriate values in this buffer. The buffer is then composited into the rendered webpage by the browser and shown in the browser’s window. The browser window is composited onto the display’s framebuffer by the operating system and video hardware.

You can read and write pixel values with the get() and set() methods. These methods are easy to use, but they are really, really slow. With a little bit of math, you can work directly with the pixels array data. This is a little more work but can easily be thousands of times faster.

Writing Pixel Data

A Basic Example

The p5.js library provides two ways to read and write image pixel data. First, you can use get() and set() which are a bit easier but slower. Second, you can access the pixels[] array directly, which is faster but requires some math to find the address of the pixel you want to work with.

This example uses set() to create some random pixel data.

Let’s look at the code in depth.

This example doesn’t write to the canvas pixels directly. Instead, it creates an empty image, writes to its pixels, and then draws the image to the canvas. This approach is more flexible and avoids the complexities of pixelDensity.

Line 10
Use createImage() to create a new, empty 10x10 image in memory. We can draw this image just like an image loaded from a .jpg or .gif.
Line 11
Use loadPixels() to tell p5 that we want to access the pixels of the image for reading or writing. You must call loadPixels() before using set(), get(), or the pixels[] array.
Line 13
Set up a nested loop. The inner content of the loop will be run once for every pixel.
Line 15
Use the color() function to create a color value, which is assigned to c. Color values hold the R, G, B, and A values of a color. The color function takes into account the current colorMode().
Line 16
Use set() to set the color of the pixel at x, y.
Line 20
Use updatePixels() to tell p5 we are done accessing the pixels of the image.
Line 22
Use noSmooth() to tell p5 not to smooth the image when we scale it: we want it pixelated. This resembles Photoshop’s ‘nearest neighbor’ scaling method.
Line 23
Draw the image, scaling up so we can clearly see each pixel.

Gradient Example

This example has the same structure as the first one, but draws a gradient pixel-by-pixel.

Line 15
Instead of choosing a color at random, this example calculates a color based on the current x and y position of the pixel being set.

Random Access Example

The first two examples use a nested loop to set a value for every pixel in the image. The loop visits every pixel in a sequential order. That tactic is commonly used in pixel generating and processing scripts, but not always. This example places red pixels at random places on the image.

In-class Challenge One

Explore using p5’s pixel manipulation functions by modifying the scripts above. Work through the following challenges in order.
Don’t skip any.

Time Comment
< 13 in 20 Minutes You need to put in some extra work
to strengthen your programming understanding.
13 in 20 Minutes Good.
All in 20 Minutes Great.
All in 15 Minutes Hot Dang!

Modify the Basic Example

  1. Change the image resolution to 20x20
  2. Change the image resolution to 500x500
  3. Change the image resolution back to 10x10
  4. Make each pixel a random shade of blue.
  5. Make each pixel a random shade of gray.

Modify the Gradient Example

  1. Make a horizontal black-to-blue gradient.
  2. Make a vertical green-to-black gradient.
  3. Make a horizontal white-to-blue gradient.
  4. Make a vertical rainbow gradient. Tip: colorMode()
  5. Create an inset square with a gradient, surrounded by randomly-colored pixels.

Modify the Random Access Example

  1. Change the image resolution to 50x50, adjust scatter to fill.
  2. Instead of drawing single pixels, draw little + marks at random locations.
  3. Make each + a random color.

Challenging Challenges

  1. Color each pixel with noise() to visualize its values.
  2. Make a radial gradient from black to red. Tip: dist()
  3. Create a diagonal gradient.
  4. Use sin() to create a repeating black-to-red-to-black color wave.
  5. Create a 128x128 image and set the blue value of each pixel to (y&x) * 16

Reading + Processing Pixel Data

The p5.js library also allows you to read pixel data, so you can process images or use images as inputs. These examples use this low-res black-and-white image of Earth.

Read Pixels Example 1

This example loads the image of Earth, loops over its pixels, and multiplies each pixel’s color with a random color.

First we need to load an image to read pixel data from.

Line 3
Declare a variable to hold our image.
Line 5
The preload() function. Use this function to load assets. p5.js will wait until all assets are loaded before calling setup() and draw()
Line 6
Load the image.

With our image loaded we can process the pixels.

Line 18
Set up a nested loop to cover every pixel.
Line 20
Use get() to load the color data of the current pixel. get() returns an array like [255, 0, 0, 255] with components for red, green, blue, and alpha.
Lines 22, 23, 24
Read the red, blue, and green parts of the color.
Line 27
Check if the red value is 255 to see if it is black or white. Since we know the image is only black and white this is enough to check.
Line 28 and Line 30
Set out_color to red or blue.
Line 33
Set the pixel’s color to out_color.
Line 34
Use updatePixels() to tell the image there has been an update. We didn’t need to do this in the loop when we were just setting pixels, but here we mix set() and get(). p5.js requires calling updatePixels() anytime we switch from setting to getting or drawing.

Every time we switch from writing/setting to reading/getting, we have to call updatePixels(). This is because internally, p5.js will call loadPixels() when we call get() which will overwrite our changes in the pixel array. This extra updating and loading is why get() and set() are slower than accessing the pixels[] array directly.

Read Pixels Example 2

This example compares each pixel to the one below it. If the upper pixel is darker, it is changed to magenta.

Image as Input Example

This example doesn’t draw the image at all. Instead, the image is used as an input that controls where the red ellipses are drawn. Using images as inputs is a powerful technique that allows you to mix manual art and procedurally-generated content.

In-class Challenge Two

Explore using p5’s pixel manipulation functions by modifying the scripts above. Work through the following challenges in order.
Don’t skip any.

Time Comment
< 10 in 20 Minutes You need to put in some extra work
to strengthen your programming understanding.
10 in 20 Minutes Good.
All in 20 Minutes Great.
All in 15 Minutes Hot Dang!

Modify Example 1

  1. Make the program turn white pixels green.
  2. Turn the black pixels to a random shade of red.
  3. Turn the black pixels into a vertical, black-to-red gradient.
  4. Comment out line 34 which calls updatePixels(). What happens?

Modify Example 2

  1. Change the lightness() comparison to >.
  2. Change the lightness() comparison to !=.
  3. Add an else block that changes the pixels to black.

Modify Example 3

  1. Tell the program to use the image above by switching which loadImage() call is commented out in preload().
  2. Adjust the expression that determines dot_size to make the result prettier.

Challenging Challenges

  1. Start with the original Example 2 code, without your changes. Set out_color to the average of this_color and below_color. Here is an example you could follow:
    var color_a = color(worldImage.get(0, 1));
    var color_b = color(worldImage.get(0, 2));
    var blended_color = lerpColor(color_a, color_b, .5);
    
  2. Change worldImage.set(x, y, out_color); to worldImage.set(x, y+1, out_color);.
  3. Remove the if statement (but not its contents) so that its content always runs.

Working Directly with the pixels[] Array

Performance

The built-in p5 get() function gets the RGBA values of a pixel in an image. Internally get() calls loadPixels() to make sure it is working with up-to-date information. This means that even when getting the values for a single pixel, every pixel is read. As noted in the reference, this makes get() slower than accessing the values in the pixels[] array directly. In fact, get() can easily be 1000s of times slower.

We can get much faster results by loading all of the pixel values once with loadPixels(), and then reading and writing the pixels[] array directly.

The getQuick() function below reads a pixel’s color value from an image’s pixels[] array. You must call loadPixels() before calling this function. When you are done working with the pixels[] array, you should call updatePixels() to update the image with your changes.

// returns the RGBA values of the pixel at x, y in the img's pixels[] array
// returns values as an array [r, g, b, a]
// use instead of p5s built in .get(x,y), for much better performance (more than 1000x better in many cases)
// see: http://p5js.org/reference/#/p5/pixels[]
// we don't need to worry about screen pixel density here, because we are not reading from the canvas

function getQuick(img, x, y) {

	var i = (y * img.width + x) * 4;
	return [
		testImage.pixels[i],
		testImage.pixels[i+1],
		testImage.pixels[i+2],
		testImage.pixels[i+3],
	];
}

Copy the getQuick() function above into your sketch. You can then replace a built-in p5 get call with a call to getQuick:

Using get()

// in loop
c = img.get(x, y);

Using getQuick()

// before loop
img.loadPixels();

// in loop
c = getQuick(img, x, y);

// after loop
img.updatePixels();

The following example compares the performance of using get() and getQuick() to read and invert the color value of a small image.

The Canvas + Pixel Density

You can work with the pixels in an image using image.pixels[] or the pixels of the canvas with just pixels[]. When accessing the pixel data of the canvas itself, you need to consider the pixel density p5 is using. By default, p5 will create a high-dpi canvas when running on a high-dpi (retina) display. You can call pixelDensity(1) to disable this feature. Otherwise, you’ll need to take into account the density when calculating a position in the pixels[] array.

The examples on this page work with the pixels of images instead of the canvas to avoid this issue altogether. If you need to work with the canvas, the pixels documentation has info on working with higher pixel densities.

Study Example

This example uses an image as an input to control the density and placement of drawn grass.

Input Image

cf.png

Keep Sketching!

Base

Explore working with image pixel data directly. This week, most of your posts should be still images.

Post at least one sketch for each of the following:

  1. Generate an image from scratch: pixel by pixel. Don’t call any high-level drawing function like ellipse() or rect().
  2. Load an image and process its pixels. Show the result.
  3. Use an image as an input source to control a drawing. Don’t show the original image, just the output.

Challenge

Create a pixel Ouroboros. Create code that processes an image. Feed the result back into your code and process it again. What happens after several generations?

Post your source image, the result after one generation, and the result after several generations. Alternately, capture 90 generations as frames and post as a video.

Reaction Diffusion in Photoshop
Create a pattern in Photoshop by repeatedly applying filters.
Factorio
A game in which players gather resources to create increasingly complex technology and factories—sometimes building these structures into pixel art.
Icon Machine
A pixel art web app that randomly generates potion bottle icons.
AtariArchives.org: Memory-Mapped Video: The Scanning Game
An article on ASCII encoding and storage, part of a 1979 primer on computer graphics.