Pixel Data

Overview

Access pixel values directly to process and generate images.

Tools

p5.js

Pixels

Today, most computers use color displays that produce images using a grid of pixels. Conceptually, each color pixel is made up of three sub-pixels—red, green, and blue—though the actual hardware may use a different pattern. Because these pixels are small and close together, the image appears continuous. Because our eyes can only directly distinguish red, green, and blue light, the screen can reproduce most of the colors we are capable of seeing. Today, most computers use color displays that produce images using a grid of pixels. Conceptually, each color pixel is made up of three sub-pixels—red, green, and blue—though the actual hardware may use a different pattern. Because these pixels are small and close together, the image appears continuous. Because our eyes can only directly distinguish red, green, and blue light, the screen can reproduce most of the colors we are capable of seeing.

Pixels Photo by Peter Halasz

When you work with a graphics library like p5.js or the Javascript Canvas API, you don’t have to think about individual pixels. You can use high-level methods like ellipse() and image(), and the library will do the low-level work of setting individual pixel values for you. This process of drawing shapes with pixels is called rasterization. Because the p5.js drawing calls handle the details of rasterization for you, you give up some control. When you draw a circle with ellipse() every pixel in the circle will be changed to the current fill() color. The p5.js library provides no way to fill the ellipse with random colors, or a gradient, or colors generated by a function. To get this level of control, you need to take control of the rasterization yourself. This is the classic high-level/low-level trade-off: coding at a low level takes more work but offers more control.

Vector Monitors

Computer displays don’t have to use pixels at all. One of the earliest video games, Tennis for Two, used a cathode ray oscilloscope as its display. In a cathode ray oscilloscope, signals are drawn by magnetically deflecting a beam of electrons as they race from an emitter—the cathode—towards a phosphor-coated glass screen. The phosphor glows where the electrons strike it, creating an image. This type of display can show smooth lines without any of the aliasing artifacts of a pixel display.

Vector Monitors were used in later video games as well, including the Tempest arcade game and several games for the Vectrex home console.

Tennis for Two

This high/low level tradeoff has some important implications for the creative coder. When you work at a high level you are not responsible for the details. Because you are not responsible for the details you tend to think about them less. Working directly with the pixels leads to thinking about drawing with code differently.

Edward Snowden's Long, Strange Journey to Hollywood, 2016
Adam Ferriss
Edward Snowden's Long, Strange Journey to Hollywood, 2016
Adam pixel-manipulated photos of Edward Snowden for the cover of the New York Times.
Stardust, 2013-14
Sergio Albiac
Stardust, 2013-14
One of more than 15,000 portraits created by a project by Sergio Albiac
Four Generations, 2015
Jon Corbett
Four Generations, 2015
Jon documented four generations of his Cree-Métis heritage by computationally transforming his family portraits into beaded images.
Chromata, 2015
Michael Bromley
Chromata, 2015
JavaScript image-to-art machine.
Sketching in Hardware, 2014
Greg Schomburg
Sketching in Hardware, 2014
An identity system that combines maze generation and pixel-map inputs.
Pixel Art Tutorial, 2013
Derek Yu
Pixel Art Tutorial, 2013
This is a short and useful tutorial on handcrafting pixel art written by Derek Yu. Derek also wrote a great book that looks at the design and implementation of his indie game hit, Spelunky.

Video Memory

Modern video pipelines are complicated, but at a basic level they work something like this: The red, green, and blue brightness values of every pixel on a display are stored in the computers RAM. In comptuers with a dedicated video card, this data is usually stored on the video card’s VRAM. Once per display refresh, the video hardware reads this data from memory, pixel by pixel, and sends it to the display over a display interface like DVI or HDMI. Hardware in the display receives this data and updates the brightness of each pixel as needed. If you change the values in the RAM, you will see the changes reflected on the screen.

The memory used to store the screen’s image is called the video buffer or framebuffer. Direct access to the screen’s framebuffer is pretty unusual on modern computers, and high level libraries like p5.js don’t (and can’t) provide it. But P5.js does give you access to a pixel buffer storing the image shown on your sketch’s canvas. When you call drawing functions like rect() and ellipse(), p5.js updates the appropriate values in this buffer. The buffer is then composited into the rendered webpage by the browser. The browser window is composited onto the display’s framebuffer by the operating system and video hardware. The p5.js library provides two ways to directly read and set the color of a single pixel: get()/set() and the pixels[] array. Using get() and set() is easier, but using the pixels[] array is faster. This chapter has examples for using both.

A high-definition display is 1920 pixels wide and 1080 tall: 345,600 total pixels. Each pixel needs three bytes to represent its color value: one byte each for the red, green, and blue channels. In total that is 6,220,800 bytes—about 6 megabytes—of memory to keep track of the full HD image.

Today, 6 megabytes isn’t much, but many older computers didn’t even have enough RAM to keep an entire full-color image of the screen in memory at all. These computers used lower resolutions, limited palettes, and a variety of tricks to output to screen.

Writing Pixel Data

Random Pixels Example

This example uses set() to set each pixel in a 10x10 image to a random color. This example doesn’t write to the canvas pixels directly. Instead, it creates an empty image, writes to its pixels, and then draws the image to the canvas. This approach is more flexible and avoids the complexities of pixelDensity. This also lets us draw the image scaled-up so you can see the pixels easier.

Let’s look at the code in depth.

Line 10: Use createImage() to create a new, empty 10x10 image in memory. We can draw this image just like an image loaded from a .jpg or .gif.

Line 11: Use loadPixels() to tell p5 that we want to access the pixels of the image for reading or writing. You must call loadPixels() before using set(), get(), or the pixels[] array.

Line 13: Set up a nested loop. The inner content of the loop will be run once for every pixel.

Line 15: Use the color() function to create a color value, which is assigned to c. Color values hold the R, G, B, and A values of a color. The color function takes into account the current colorMode().

Line 16: Use set() to set the color of the pixel at x, y.

Line 20: Use updatePixels() to tell p5 we are done accessing the pixels of the image.

Line 22: Use noSmooth() to tell p5 not to smooth the image when we scale it: we want it pixelated. This resembles Photoshop’s ‘nearest neighbor’ scaling method.

Line 23: Draw the image, scaling up so we can clearly see each pixel.

Gradient Example

This example has the same structure as the first one, but draws a gradient pixel-by-pixel.

Line 15: Instead of choosing a color at random, this example calculates a color based on the current x and y position of the pixel being set.

Random Access Example

The first two examples use a nested loop to set a value for every pixel in the image. The loop visits every pixel in a sequential order. That tactic is commonly used in pixel generating and processing scripts, but not always. This example places red pixels at random places on the image.

Coding Challenges One

Explore this chapter’s example code by completing the following challenges.

Modify the Basic Example

  1. Change the image resolution to 20x20.
  2. Change the image resolution to 500x500.
  3. Change the image resolution back to 10x10.
  4. Make each pixel a random shade of blue. ••
  5. Make each pixel a random shade of gray. ••
  6. Color each pixel with noise() to visualize its values. •••

Modify the Gradient Example

  1. Make a horizontal black-to-blue gradient.
  2. Make a vertical green-to-black gradient.
  3. Make a horizontal white-to-blue gradient.
  4. Make a vertical rainbow gradient. Tip: colorMode() ••
  5. Create an inset square with a gradient, surrounded by randomly-colored pixels. •••
  6. Make a radial gradient from black to red. Tip: dist() •••
  7. Create a diagonal gradient. •••

Modify the Random Access Example

  1. Change the image resolution to 50x50, adjust the code to fill the image.
  2. Instead of drawing single pixels, draw little plus marks (+) at random locations. ••
  3. Make each + a random color. ••

Start from Scratch

  1. Use sin() to create a repeating black-to-red-to-black color wave. •••
  2. Create a 128x128 image and set the blue value of each pixel to (y&x) * 16. •••

Reading + Processing Pixel Data

The p5.js library also allows you to read pixel data, so you can process images or use images as inputs. These examples use this low-res black-and-white image of Earth.

world.png

Read Pixels Example 1

This example loads the image of Earth, loops over its pixels, and white pixels to red and black pixels to blue.

First we need to load an image to read pixel data from.

Line 3: Declare a variable to hold our image.

Line 5: The preload() function. Use this function to load assets. p5.js will wait until all assets are loaded before calling setup() and draw()

Line 6: Load the image.

With our image loaded we can process the pixels.

Line 18: Set up a nested loop to cover every pixel.

Line 20: Use get() to load the color data of the current pixel. get() returns an array like [255, 0, 0, 255] with components for red, green, blue, and alpha.

Lines 22, 23, 24: Read the red, blue, and green parts of the color.

Line 27: Check if the red value is 255 to see if it is black or white. Since we know the image is only black and white this is enough to check.

Line 28 and Line 30: Set out_color to red or blue.

Line 33: Set the pixel’s color to out_color.

Line 34: Use updatePixels() to tell the image there has been an update. We didn’t need to do this in the loop when we were just setting pixels, but here we mix set() and get(). p5.js requires calling updatePixels() anytime we switch from setting to getting or drawing.

Every time we switch from writing/setting to reading/getting, we have to call updatePixels(). This is because internally, p5.js will call loadPixels() when we call get() which will overwrite our changes in the pixel array. This extra updating and loading is why get() and set() are slower than accessing the pixels[] array directly.

Read Pixels Example 2

This example compares each pixel to the one below it. If the upper pixel is darker, it is changed to magenta.

Image as Input Example

This example doesn’t draw the image at all. Instead, the image is used as an input that controls where the red ellipses are drawn. Using images as inputs is a powerful technique that allows you to mix manual art and procedurally-generated content.

Coding Challenges Two

Explore this chapter’s example code by completing the following challenges.

Modify Read Pixels Example 1

  1. Make the program turn white pixels green.
  2. Turn the black pixels to a random shade of red.
  3. Turn the black pixels into a vertical, black-to-red gradient. ••
  4. Comment out line 34 which calls updatePixels(). What happens? ••

Modify Read Pixels Example 2

  1. Change the lightness() comparison to >.
  2. Change the lightness() comparison to !=.
  3. Add an else block that changes the pixels to black. ••
  4. Starting with the original code without any changes, set the out_color to the average of this_color and below_color. Here is an example you could follow:•••
var color_a = color(worldImage.get(0, 1));
var color_b = color(worldImage.get(0, 2));
var blended_color = lerpColor(color_a, color_b, 0.5);
  1. Change worldImage.set(x, y, out_color); to worldImage.set(x, y+1, out_color);. •••
  2. Remove the if statement (but not its contents) so that its content always runs. •••

Modify Read Pixels Example 3

  1. Tell the program to use the image below by switching which loadImage() call is commented out in preload().
  2. Adjust the expression that determines dot_size to make the result prettier. ••

Working Directly with the pixels[] Array

You can read and write individual pixel values with the get() and set() methods. These methods are easy to use, but they are really, really slow. A faster approach is to use loadPixels() and updatePixels() to copy the buffer to and from the p5 pixels[] array. Then, with a little bit of math, you can work directly with the pixels[] array data. This is a little more work but can easily be thousands of times faster.

Performance

The built-in p5 get() function gets the RGBA values of a pixel in an image. Internally get() calls loadPixels() to make sure it is working with up-to-date information. This means that even when getting the values for a single pixel, every pixel is read every time you call get(). As noted in the reference, this makes get() slower than accessing the values in the pixels[] array directly. In fact, get() can easily be 1000s of times slower.

The get() call is slow, and gets slower the larger your image is.

A 10 x 10 image has 100 pixels. Reading each pixel reloads all 100 pixel values. That means 10,000 pixel values are copied from the image into pixels[] array.

A 50 x 50 image has 2,500 pixels. Reading each pixel reloads all 2,500 pixels. That is 6,250,000 pixels copied.

A 1,920 x 1,080 image has 2,073,600 pixels. Reading all of those pixels with get() would require copying 4,299,816,960,000 pixels, but your browser will hang or crash first.

We can get much faster results by loading all of the pixel values once with loadPixels(), and then reading and writing the pixels[] array directly. Since we are reading from pixels[] ourselves, we can make sure we haven’t changed the values we are trying to read and bypass the safety measures that slow down get(). We have to be a little more careful about what we are doing, though, or we might create bugs.

The getQuick() function below reads a pixel’s color value from an image’s pixels[] array. You must call loadPixels() before calling this function. When you are done working with the pixels[] array, you should call updatePixels() to update the image with your changes.

// returns the RGBA values of the pixel at x, y in the img's pixels[] array
// returns values as an array [r, g, b, a]
// use instead of p5s built in .get(x,y), for much better performance (more than 1000x better in many cases)
// see: http://p5js.org/reference/#/p5/pixels[]
// we don't need to worry about screen pixel density here, because we are not reading from the canvas

function getQuick(img, x, y) {
var i = (y * img.width + x) * 4;
return [
img.pixels[i],
img.pixels[i + 1],
img.pixels[i + 2],
img.pixels[i + 3],
];
}

Copy the getQuick() function above into your sketch. You can then replace a built-in p5 get call with a call to getQuick:

Using get()

// in loop
c = img.get(x, y);

Using getQuick()

// before loop
img.loadPixels();

// in loop
c = getQuick(img, x, y);

// after loop
img.updatePixels();

get() vs getQuick()

The following example compares the performance of using get() and getQuick() to read and invert the color value of a small image.

The Canvas + Pixel Density

You can work with the pixels in an image using image.pixels[] or the pixels of the canvas with just pixels[]. When accessing the pixel data of the canvas itself, you need to consider the pixel density p5 is using. By default, p5 will create a high-dpi canvas when running on a high-dpi (retina) display. You can call pixelDensity(1) to disable this feature. Otherwise, you’ll need to take into account the density when calculating a position in the pixels[] array.

The examples on this page work with the pixels of images instead of the canvas to avoid this issue altogether. If you need to work with the canvas, the pixels documentation has info on working with higher pixel densities.

Growing Grass

This example uses an image as an input to control the density and placement of drawn grass.

Input Image

cf.png

Keep Sketching!

Sketch

Explore working with image pixel data directly. Most of your sketches should be still images.

Create at least one sketch for each of the following:

  1. Generate an image from scratch: pixel by pixel. Don’t call any high-level drawing function like ellipse() or rect().
  2. Load an image and process its pixels.
  3. Use an image as an input source to control a drawing. Don’t show the original image, just the output.

Challenge: Pixel Ouroboros

Create code that processes an image. Feed the result back into your code and process it again. What happens after several generations?

Pair Challenge: Generate / Process

Work with a partner.

  1. Make a sketch that generates an image pixel by pixel.
  2. Give your image to your partner.
  3. Create a sketch that pixel processess that image.

Explore