Raspberry Pi Ball Tracking Robot using Processing with Camera module

The field of Robotics, Artificial Intelligence and Machine Learning is evolving rapidly that it is sure to change the lifestyle of mankind in near future. Robots are thought to understand and interact with the real world through sensors and machine learning processing. Image recognition is one of the popular way in which the robots are thought to understand objects by looking at the real world through a camera just like we do. In this project, let use the power of Raspberry Pi to build a Robot that could track ball and follow it just like the robots that plays football.

 OpenCV is a very famous and open source tool that is used for Image processing, but in this tutorial to keep things simple we are using the Processing IDE. Since processing for ARM has also released the GPIO library for processing we will not have to shift between python and processing anymore to work with Raspberry Pi. Sounds cool right? So let us get started.

Hardware Required:

  1. Raspberry Pi
  2. Camera module with ribbon cable
  3. Robot Chassis
  4. Gear  motors with wheel
  5. L293D motor driver
  6. Power bank or any other portable power source

Programming Requirement:

  1. Monitor or other display for Raspberry pi
  2. Key board or mouse for Pi
  3. Processing ARM software
Note: It is mandatory to have a display connected to Pi through wires during programming because only then the camera’s video can be viewed

Setting up Processing on Raspberry Pi:

As told earlier we will be using the processing environment to Program our Raspberry Pi and not the default way of using python. So, follow the steps below:
Step 1:- Connect your Raspberry Pi to your monitor, keyboard and mouse and turn it on.
Step 2:- Make sure you Pi is connected to an active internet connection because we are about to download few things.
Step 3:- Click on Processing ARM, to download the processing IDE for Raspberry Pi. The download will be in the form of a ZIP file.
Step 4:- Once downloaded, extract the files in your ZIP folder in you preferred directory. I just extracted it on my desktop.
Step 5:- Now, open the extracted folder and click on the file named processing. It should open a window as shown below.


Step 6:- This is the environment where we will be typing our codes. For people who are familiar with Arduino, don’t be shocked YES the IDE does look similar to Arduino and so does the program.
Step 7:- We need two libraries for our ball following program to work, to install then just click on Sketch -> Import Library -> Add Library. The following dialog box will open.

Step 8:- Use the top left text box to search for Raspberry Pi and hit enter, you search result should look something like this.

Step 9:-Search for the libraries named “GL Video” and “Hardware I/O” and click on install to install them. Make sure you install both the libraries.
Step 10:- Based on your internet the installation will take few minutes. Once done we are ready with for processing software.

Circuit Diagram:

The circuit Diagram of this Raspberry Pi Ball Tracking Project is shown below.

As you can see the circuit involves a PI camera, Motor Driver module and a pair of motors connected to the Raspberry pi. The complete circuit is powered by a Mobile Power bank (represented by AAA battery in the circuit above).
Since the pins details are not mentioned on the Raspberry Pi, we need to verify the pins using the below picture

To drive the Motors, we need four pins (A,B,A,B). This four pins are connected from GPIO14,4,17 and 18 respectively. The orange and white wire together forms the connection for one motor. So we have two such pairs for two motors.
The motors are connected to the L293D Motor Driver module as shown in the picture and the driver module is powered by a power bank. Make sure that the ground of the power bank is connected to the ground of the Raspberry Pi, only then your connection will work.
That is it we are done with our Hardware connection, let’s go back to our processing environment and start programming to teach our robot how to track a ball.

Raspberry Pi Ball tracking Program:

The complete Processing program of this project is given at the end of this page, which you directly use. Further just below, I have explained the working of the code so that you can use it for other similar projects.
The program concept is very simple. Although the intention of the project is to track a ball, we are actually not going to do it. We are just going to identify the ball using its colour. As we all know videos are nothing but continuous frames of pictures. So we take each picture and split it into pixels. Then we compare each pixel colour with the colour of the ball; if a match is found then we can say that we have found the ball. With this information we can also identify the position of the ball (pixel colour) on the screen. If the position is far left we move the robot to right, if the position is far right we move the robot to left so that the pixel position always stays at the centre of the screen. You can watch Computer Vision video of Daniel shiffman to get a clear picture.
As always we begin by importing the two libraries that we download. This can be done by the following two lines. The Hardware I/O library is used to access the GPIO pins of the PI directly from the processing environment, the glvideo library is used to access the Raspberry Pi camera module.
import processing.io.*;
import gohai.glvideo.*;

Inside the setup function we initialize the output pins to control the motor and also get the video from the pi camera and size it in a window of size 320 * 240.
void setup() {
  size(320, 240, P2D);
  video = new GLCapture(this);
  video.start();
  trackColor = color(255, 0, 0);
  GPIO.pinMode(4, GPIO.OUTPUT);
  GPIO.pinMode(14, GPIO.OUTPUT);
  GPIO.pinMode(17, GPIO.OUTPUT);
  GPIO.pinMode(18, GPIO.OUTPUT);
}

The void draw is like the infinite loop the code inside this loop will be execute as long as the program is terminated. If a camera source is available we read the video coming out of it
void draw() {
   background(0);
  if (video.available()) {
    video.read();
  }}
Then we begin to split the video frame into pixels. Each pixel has a value of red, green and blue. These values are stored in the variable r1, g1 and b1
  for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      int loc = x + y*video.width;
      // What is current color
      color currentColor = video.pixels[loc];
      float r1 = red(currentColor);
      float g1 = green(currentColor);
      float b1 = blue(currentColor);

To detect the colour of the ball initially, we have to click on the colour. Once click the colour of the ball will be stored in variable called trackColour.
void mousePressed() {
  // Save color where the mouse is clicked in trackColor variable
  int loc = mouseX + mouseY*video.width;
  trackColor = video.pixels[loc];
}
Once we have the track colour and the current colour we have to compare them. This comparison is using the dist function. It checks how close the current colour is to the track colour.
float d = dist(r1, g1, b1, r2, g2, b2);

The value of dist will be zero for an exact match. So, if the value of dist is less than a specified value (world Record) then we assume that we have found the track colour. Then we get the location of that pixel and store it in the variable closest X and closest Y to find the location of the ball
if (d < worldRecord) {
        worldRecord = d;
        closestX = x;
        closestY = y;
      }

We also draw an ellipse around the found colour to indicate that the colour has been found. The value of the position is also printed on the console, this will help a lot while debugging.
if (worldRecord < 10) {
    // Draw a circle at the tracked pixel
    fill(trackColor);
    strokeWeight(4.0);
    stroke(0);
    ellipse(closestX, closestY, 16, 16);
    println(closestX,closestY);
Finally we can compare the position of the closest X and closest Y and adjust the motors in such a way that the colour gets to the centre of the screen. The below code is used to turn the robot right since the X position of the colour was found to be in the left side of the screen (<140)
    if (closestX<140)
    {
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.LOW);
     delay(10);
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.HIGH);
     println("Turn Right");
    }
Similarly we can check the position of X and Y to control the motors in the required direction. As always you can refer the bottom of the page for the complete program.

Working of Raspberry Pi Ball Tracking Robot:

Once you are ready with the hardware and program it’s time to have some fun. Before we test our bot on ground, we should make sure everything is working fine. Connect your Pi to monitor and launch the processing code. You should see the video feed on a small window. Now, bring the ball inside the frame and click on the ball to teach the robot that it should track this particular colour. Now move the ball around the screen and you should notice the wheels rotating.
If everything is working as expected, release the bot on the ground and started playing with it. Make sure the room is evenly illuminated for best results. The complete working of the project is shown in the video below. Hope you understood the project and enjoyed building something similar. If you have any problems feel free to post them on the comment section below or help.
Code: 
/*
Processing Raspberry Pi program for Ball following Robot
*/
import processing.io.*;
import gohai.glvideo.*;
GLCapture video;
color trackColor;
void setup() {
  size(320, 240, P2D);
  video = new GLCapture(this);
  video.start();

  trackColor = color(255, 0, 0);

  GPIO.pinMode(4, GPIO.OUTPUT);
  GPIO.pinMode(14, GPIO.OUTPUT);
  GPIO.pinMode(17, GPIO.OUTPUT);
  GPIO.pinMode(18, GPIO.OUTPUT);
}
void draw() {
   background(0);
  if (video.available()) {
    video.read();
  }

  video.loadPixels();
  image(video, 0, 0);

  float worldRecord = 500;
  int closestX = 0;
  int closestY = 0;

  // Begin loop to walk through every pixel
  for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      int loc = x + y*video.width;
      // What is current color
      color currentColor = video.pixels[loc];
      float r1 = red(currentColor);
      float g1 = green(currentColor);
      float b1 = blue(currentColor);
      float r2 = red(trackColor);
      float g2 = green(trackColor);
      float b2 = blue(trackColor);
      // Using euclidean distance to compare colors
      float d = dist(r1, g1, b1, r2, g2, b2); // We are using the dist( ) function to compare the current color with the color we are tracking.
      // If current color is more similar to tracked color than
      // closest color, save current location and current difference
      if (d < worldRecord) {
        worldRecord = d;
        closestX = x;
        closestY = y;
      }
    }
  }

  if (worldRecord < 10) {
    // Draw a circle at the tracked pixel
    fill(trackColor);
    strokeWeight(4.0);
    stroke(0);
    ellipse(closestX, closestY, 16, 16);
    println(closestX,closestY);
 
    if (closestX<140)
    {
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.LOW);
     delay(10);
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.HIGH);
   
     println("Turn Right");
    }
    else if (closestX>200)
    {
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.LOW);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.HIGH);
     delay(10);
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.HIGH);
     println("Turn Left");
    }
    else if (closestY<170)
    {
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.LOW);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.LOW);
     delay(10);
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.HIGH);
     println("Go Frwd");
    }
    else
     {
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.HIGH);
     } 
  }
  else
  {
     GPIO.digitalWrite(4, GPIO.HIGH);
     GPIO.digitalWrite(14, GPIO.HIGH);
     GPIO.digitalWrite(17, GPIO.HIGH);
     GPIO.digitalWrite(18, GPIO.HIGH);
  }
}
void mousePressed() {
  // Save color where the mouse is clicked in trackColor variable
  int loc = mouseX + mouseY*video.width;
  trackColor = video.pixels[loc];
at
Labels: ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)