🖼️Lab 3: Image Processing

CS 1004 ~ Prof. Smith

Learning Objectives

This lab puts together a bit of what you’ve done every week so far: variables, conditionals, loops, accepting keyboard input, and drawing to the screen. Additionally, you should begin to gain comfort with:

• Loading images from a file

• Working with arrays

• Accessing pixel values for an image

• Considering computational efficiency/complexity

For this lab, you’ll be creating a set of five “filters” that convert an input photo to some stylized rendering. The sample code includes: dithering, impressionist effects, edge detection, and “glitched” rendering. Each filter will be tied to a key on the keyboard, as in lab 2. The difference is, this time, you will need to turn off the filter as soon as it’s run once (otherwise your framerate will drop significantly), as well as design them so that they don’t take way too long to run. There’s a lot of freedom in how you define your filters, but they should meet the following requirements:

• At least one filter should be glitch art-inspired (e.g. GLITCH in the sample code)

• At least one filter should involve convolving the image with a kernel (e.g. EDGES in sample)

• At least one filter should use randomness and conditionals (e.g. IMPRESSIONIST in sample)

Choose Your Image | Plan Your Lab

Choose an image. I’ve chosen a photo of Natural Bridges State Beach in Santa Cruz, California. When I lived in California, I would go to this beach regularly to relax. I recommend picking an image that you won’t get tired of while working on this lab. Something soothing or funny is nice!

Considerations for which image you pick to work with:

• Don’t make it too big – the smaller the image dimensions (in pixels), the less time each filter will take to process.

• It has to be in either .png, .jpg, .tga, or .gif (single frame only) format

• Color images give you more flexibility to play with filters than greyscale/black-and-white.

You’ll need to upload the file to your project. If you are using the online editor, you can upload files in the file chooser on the far left side of the screen. If you are using the offline Processing editor, go to Sketch -> Add File... to add the image to the data folder.

Working with Images

The first step is to load the image of your choice and store it in a variable, so that you can manipulate it later. loadImage(filename) loads the image data from your computer and stores it in a variable in the program. Remember that this needs to go into a new function, preload(), so that there is time for it to load before the rest of the program runs.

//store the image!
var santacruz;

function preload() {
  santacruz = loadImage("natural-bridges.png");
}

function setup() {
  //manually set the canvas size to be the same as the image dimensions
  createCanvas(650, 429);

  //make a 1-to-1 correspondence between pixel in image and on screen
  pixelDensity(1);

  //initial drawing settings
  background(255);
  noStroke();
}

santacruz now is a variable storing a PImage, which is a custom data type for Processing. A PImage stores information about the image it holds, including its width and height, its color mode, and (importantly for this lab) the color at each pixel. In setup() we create the canvas and manually set its size to be the same as the image dimensions, and then set pixelDensity(1). This is important for modern displays, which will frequently artificially inflate the size of the image to use more of the pixels on your screen! Setting the pixel density to 1 means there is only one display pixel per represented pixel. (If this math is confusing, don’t worry – it’s ok to just put in the magic pixel density code.)

To draw the image, let’s look at the code in the SHOW_IMAGE condition in the draw() function.

//finally, we have the option to just show the image as-is
else if (SHOW_IMAGE == true) {
    image(santacruz, 0, 0);
    SHOW_IMAGE = false;
}

The image(name, x, y) function draws the PImage stored in name (in this case, santacruz) to the screen at the given x and y coordinates (just like rect(...) it draws from the top left corner by default.)

This lets us draw an image to the screen, but to implement image processing filters we need more than to just be able to draw the image, we need to also have access to the color in every pixel in the image. To do this, we’ll use the PImage’s built-in pixels list. For example, we can see in the IMPRESSIONIST filter in the draw() loop in the sample code:

santacruz.loadPixels(); //update pixels in the image's pixels list
//iterate over every 5 pixels in the image width and height
for (var x = 0; x < width; x += 5) {
        for (var y = 0; y < height; y += 5) {
                //santacruz.pixels is a 1-dimensional list with length width*height
                //convert to 1D from 2D with 4 * (x + width*y)
                //colAtPixel is the color at the x,y coordinate
                var ind = 4 * (x + width * y);
                var rp = santacruz.pixels[ind];
                var gp = santacruz.pixels[ind + 1];
                var bp = santacruz.pixels[ind + 2];

                //draw a thick line with random direction that's the same color
                //as the underlying pixel
                stroke(rp, gp, bp);
        }
}

There’s a few important things happening here! Let’s go bit by bit, starting inside the loop.

santacruz.pixels[index] gives us the component of the color that corresponds to the specified index value in the image’s pixel list. You may have noticed that this is a 1-dimensional list, even though images have two dimensions. p5.js uses a 1-dimensional array because it is very slightly more efficient to store values in a 1D array than a 2D array. The 1D array takes each row of the image and places them in a row in the array. So the first width pixels correspond to the first row of the image, the second width set of pixels correspond to the second row, and so on. The total array length is width*height*4. The contents of indices 0-3 are the red, green, blue, and alpha values of the first pixel in the image. The contents of indices 4-7 are the red, green, blue, and alpha values of the second pixel in the image. And so on.

-------------------------------------
|        |        |        |        |
|  0-3   |   4-7  |  8-11  | 12-15  |
|        |        |        |        |
-------------------------------------
|        |        |        |        |
| 16-19  | 20-23  | 24-27  | 28-31  |
|        |        |        |        |
-------------------------------------

A 4px x 2px "image", showing corresponding indices for each pixel.

Notice we have two nested for loops going through the width and height of the image (each incrementing the x and y values by 5). We then index into santacruz.pixels with 4*(x + width*y), which is how we convert from 2D image position back to the 1D index. You could write this as a single for loop, but I find it more intuitive to use a double for loop because that’s the way I think about iterating through the pixels.

Finally, before we can do anything with the pixels, we first need to make sure p5.js has set up the pixels list correctly. This is, again, an efficiency thing – images can get really big, and the pixels list size scales with the size of the image. A lot of people programming with p5.js will never need to access it, so it doesn’t make sense to take up all the space in memory unless it’s needed. p5.js will also “clean up” memory, if it determines that the pixels list hasn’t been accessed recently. That means before you try to access the values in the pixels list, you need to tell p5.js to load the correct data into the list with loadPixels().

Notice that all of these functions are applied to the santacruz variable with a . operator:

santacruz.loadPixels()

santacruz.pixels[...]

If you don’t put the name of the variable in front, p5.js will default to accessing the pixels that are currently active on the drawing canvas.

Implement Your Filters

Note: p5.js has a built in filter(...) function. Do not use it for this lab. The goal is for you to implement your own!

You’re going to be implementing your filters in the draw() function, controlled by variables set in keyPressed() – this should seem familiar to you from Lab 2. Each filter should run once, and then turn off. You can see a small version of this happening with the SHOW_IMAGE setting:

//finally, we have the option to just show the image as-is
else if (SHOW_IMAGE == true) {
    image(santacruz, 0, 0);
    SHOW_IMAGE = false;
}

...and this is the corresponding code in keyPressed() that sets SHOW_IMAGE to true.

if (key == '5') {
    SHOW_IMAGE = true;
}

In keyPressed() we set the variable for that filter to true, and then immediately after running the filter code, we set it back to false.

The reason for this is related to a core concept in computer science: computational complexity. The bigger your image, the more pixels it has. The more pixels it has, the more times the code in the for loop needs to execute. And the more times that code executes, the longer it takes! The more complex that code is, too, the longer it takes: it’s faster to add two numbers together than to multiply them, or raise them to a power. And it’s much faster to do all of those things than it is to iterate over another long array on every iteration of the loop.

As a general rule of thumb, we shouldn’t run time-consuming code unnecessarily: since (most of) the filters you’ll implement only need to run once, we automatically shut it off at the end of its first run. You do have the option to create filters that are animated (if they’re fast enough to support it!), though, in which case you could to implement your filter control code similarly to lab 2.

There are four sample filters included with this lab: impressionism, dithering, edge detection, and glitch art. As a reminder, you should implement five filters, following these guidelines:

• At least one filter should be glitch art-inspired (e.g. GLITCH in the sample code)

• At least one filter should involve convolving the image with a kernel (e.g. EDGES in sample)

• At least one filter should use randomness and conditionals (e.g. IMPRESSIONIST in sample)

Reflect on What You Learned

As usual, the last step is to reflect on what you have learned. In a paragraph or two, describe what you think you’ve learned from this assignment, both in relation to the learning objectives described at the beginning of the lab assignment, and more broadly related to the course goals. Are there any examples you can think of in your project where you came face-to-face with thinking about efficiency and complexity in your code? What do you know now that you didn’t know before starting the lab? If you were to start this lab again knowing what you know now, would you do anything differently? What did you find difficult and/or easy about this lab? Is there more you want to learn?

Turn It In

Save your final p5.js sketch with the naming convention: lastname_firstname_lab3

Then, submit the following:

• A zip file of the directory containing your sketch for your lab assignment (remember to zip the full directory, not just the directory contents) – this will include the image file! If you use the online p5js editor, please also include a link to it saved online!

• Either:

  • screenshots or video demonstrating the output of your lab (if the code works) OR

  • a brief description of what you think is wrong with your code (if the code doesn’t work)

• Your reflection on creating image processing filters and what you have learned this week

Supplemental Files

Full sample code available online or to download:

1MB

cs1004_lab_3_sample_complete.zip

archive