Image Processing Based Robotics.pdf

Karan Bajaj

IT-BHU VARANASI

IMAGE PROCESSING BASED ROBOTICS

[email protected] [email protected] c.in | karanhindustan.blogspot.com

Karan Bajaj


Getting Started Image Processing comprises of any form of signal processing for which the input is an image. image. Not very surprising! How do we use image processing in Robotics? The Robot we would be using has an eye. eye. Now where is this eye located? It‟s generally a webcam that may be permanently attached at some specific location, or attached to the Robot itself. This webcam inputs images, which are sent to the computer via the webcam connecting wire. The concept of Image processing now comes in, and the acquired image is processed using MATLAB programming (introduced later in the tutorial). Now we need to inform the Robot about the processed details. For this, we use a port of the computer, which sends information from the computer to the Robot. The Robot contains a microcontroller which would decode the information send via the port. This information is worked upon, and signals are sent to the motors. motors. We would work on understanding these aspects STEP by STEP as follows – 1. Image acquisition Image taken in from webcam to be brought into the environment (MATLAB) where it will be processed 2. Image processing Processing of the image using the environment (MATLAB) 3. Using the port Interfacing using Serial ports 4. Motor control Controlling DC motors and Stepper motors 2

Robotics Club, IT-BHU

[email protected] [email protected] n

Karan Bajaj


This diagram roughly depicts the connections.

MATLAB MATLAB stands for Matrix Laboratory. It is a numerical computation environment and 4 th generation programming language that uses matrices to do hell lot of things! As I mentioned we would be using MATLAB, it makes ma kes sense that the work of processing the Image would be done in MATLAB. MATLAB uses matrices to store multimedia entities. Hence, our images imag es would also be stored as matrices.

3



Karan Bajaj


MATLAB environment

We would be using MATLAB 7.8.0 (R2009a) for the tutorial. A typical MATLAB 2009 window would look like the image above. It is divided into four major sections – Command Window We write all the commands in this section Current Directory Shows us the contents of the current directory Workspace Displays the details of variables being used Command History Shows the list of commands recently used The toolbar gives option for setting a custom current directory. You were informed that MATLAB does „hell lotta stuff‟ using matrices. You can yourself find the tutorials to all that using the Help button 4



Karan Bajaj


The Help menu looks like this –

We would be restricting ourselves to the Image Acquisition tool and the Image Processing tool for this tutorial. Now let‟s start working with MATLAB. All the commands would be written on the „>>‟ symbol you can see in the Command Window. Type „k = 9‟ and Press ENTER

5



Karan Bajaj


You can see that MATLAB automatically stores „k‟ as a matrix of size 1x1 with value 9. This action also reflects in the Command Comma nd History. 1D and 2D matrices can be stored as follows –

We can use the „;‟ operator after the statement to avoid display.

To access any element in the matrix, we can use indices corresponding to that location. Indices in MATLAB start from 1. For instance –

The value stored at second row and first column is „5‟ We can do simple mathematical operations operatio ns using the matrix variables -

6



Karan Bajaj


ans is the default variable of MATLAB, and is used if the assignment operator is not present. We can assign the result of this o peration using the „=‟ and a variable on the left.

Result will be stored in „c‟ instead of „ans‟

Trigonometric functions

sin(1) – value in radians is passed sind(45) – value in degrees is passed asin(1) – inverse sin in radians asind(1) – inverse sin in degrees Similarly, we can use cos(), cosd(), tan(), tand(), etc General commands clc – Clear the Command Window clear – Reset the Workspace and delete all the variables stored 7



Karan Bajaj


The colon ‘:’ operator

The colon operator is used for specifying “throughout the range”.

Some illustrations of the use of colon operator are given above. a bove.

Relational Operators in MATLAB

Control statements in MATLAB Conditions statement – „if‟ Iteration statement – „for‟, „while‟ 8



Karan Bajaj


Syntax – if statements else statements end while statements end for statements end

zeros(k) : Returns matrix of dimension k-by-k filled with zeros zeros(p,q) : Returns matrix of dimension p-by-q filled with zeros ones(k) : Returns matrix of dimension k-by-k filled with ones ones(p,q) : Returns matrix of dimension p-by-q filled with ones

dot(A,B) : Returns dot product of two vectors A and B

9



Karan Bajaj


Working with Images We can read images in MATLAB as follows – imread() function

Now the variable „img‟ stores the image cameraman.tif as cameraman.tif as a matrix. One would expect an error message on typing this directly, but MATLAB has some default images present, among which „cameraman.tif‟ is one. If one wonders the value of the matrix formed, try NOT using the semicolon once!

The image can be displayed using the imshow() function as shown. 10



Karan Bajaj


MATLAB supports three types of images – Binary Image (Black and White) – Consisting of only two colours

Grayscale – Black, White and shades of grey

Colour – Contains RED, GREEN and BLUE colours in varied intensities

11



Karan Bajaj


RGB value For each pixel in a colour image, we w e have a value of the RGB colour triplet. This triplet indicates the intensity of Red, Green and Blue on a scale of 0 to 255 (256 colours). RGB value of an image can be very easily obtained o btained using the Data Cursor tool

in the toolbox of the „Figure 1‟ window of MATLAB.

For instance,

has RGB value as 255, 128 and 0.

You can verify this by using the data cursor tool, which shows -

Making m-files It sounds quite sensible when I say that we do not write the complete code in Command Window. MATLAB codes are stored in special s pecial files called m files. files. You can use the

12

icon for opening the Editor –



Karan Bajaj


Type in the code inside this editor –

Save the m file as „sample.m‟ –

For running the m file, write onto the Command Window, the name of the m file without the extension –

And the output (display of image in our case) will appear –

13



Karan Bajaj


FUNCTIONS in Matlab Functions in MATLAB are also stored in an m-file, with an exception that the first executable statement of the file must be of the form – function [out1, out2, ...] = (in1, in2, ...)

The out1, out2, etc are the return values of the function. Unlike C,C++, etc, a MATLAB function can return multiple values. in1, in2, etc are the input parameters required by the function. The function is written in the m-file as follows follo ws –

Save the m-file with the name of the function – „stat.m‟ in our case

The function can be executed from the Command Window in the following manner –

The vector „a‟ is passed to the function „stat‟ as parameter, and return values „m‟ and „std‟ are obtained. 14



Karan Bajaj


Coming on to the aspects of Image processing we would be primarily using – 1. Obtaining the Image 2. Isolating the area of Interest 3. Obtaining the desired outputs

IMAGE ACQUISITIO ACQUISITION N Hardware Information For obtaining an image, we would obviously be having an image ima ge acquisition device (like a webcam) onto our PC/laptop. MATLAB has software called adaptors for accessing these devices. You can obtain this information by using this command – >> imaqhwinfo

Can be visualized as im-aq-hw-info (image-acquisition-hardwareinformation) It produces the following output on my system –

The adapter that MATLAB indicates on my system is „winvideo‟ as shown above. We can now use this adapter to gallop into further details! For getting some more information onto the default colour format, device name, etc, use the following command 15



Karan Bajaj


In case you have multiple image acquisition devices, you may use further DeviceIDs like 2, 3, etc in the statement –

Now we would see the formats that our adapter supports –

These are different formats of capturing the video. The 1024x768 indicates the resolution of the video in that particular mode. Defining Video Input Object

„winvideo‟ is the adapter name, „1‟ is the DeviceID DeviceID of the device we would be using, and „YUY2_320x240‟ is one of the supported format. 16



Karan Bajaj


On pressing ENTER, one would see the following output –

This gives all details of the Video Input Object. Now, how to switch the PREVIEW ON ON? ?

And the Video Preview window opens with the resolution you specified in the object creation statement -

Preview can also be viewed using different formats (Resolutions) from the Supported Formats list. 17



Karan Bajaj


Capturing Image We use the getshapshot() function for capturing an image from our image input object -

The object appears so weird, you think either your webcam has gone berserk, berserk, or it‟s some evil leprechaun!

Nothing to worry, guys! This happened because we took picture in YCbCr family of colour spaces, which is “YUY2_...” format. You can grab some more details about YCbCr here - http://en.wikipedia.org/wiki/YCbCr

We would have to convert this to RGB as follows -

18



Karan Bajaj


The new window shows the snapshot in RGB colour space.

Saving an Image file Similar to imread() command, we have the imwrite() for writing an image variable onto the disk in form of an image i mage file.

Image file stored IMPORTANT – Getting a snapshot requires multiple steps – switching ON the camera, sufficient light entering in it, capturing taking place. So there should be a pause of some duration between both the statements, when the code is written in m-file. 19



Karan Bajaj


vid=videoinput('winvideo',1, 'YUY2_160x120'); preview(vid); while(1) img=getsnapshot(vid); % Do all image processing and analysis here end We now have an idea about how to acquire an image. Let us work on processing the acquired image now!

IMAGE PROCESSING As the image is now with us as input, our main mai n task is to work on it. Typical Image processing problems include locating a particular object with some colour, or counting objects with the same colour, etc. Let us see a simple example – Suppose we have an image „shapes.tif‟

We can store this image in a variable img using the imread() function, and check for the RGB values at different positions. For the image taken, the RGB at Red region is (255, 0, 0), at Green region is (0, 255, 0), and at Blue region is (0, 0, 255). 20



Karan Bajaj


We make an m-file as follows –

Save this file as BB.m Running the program in the Command Comma nd Window –

Figure Window –

21



Karan Bajaj


What we did in the program was to declare a zero matrix with dimensions same as that of the test image. For every pixel location, we test if the region‟s RED value is 255 and GREEN and and BLUE values are 0. If yes, we mark that place as „1‟ (white), otherwise it remains „0‟ (black) in the resultant binary image. A piece of cake! But real time problems consist of not a uniform colour, but a distribution of intensities! Let us take another example – Suppose we have an image – „flower.tif‟ in our MATLAB directory

22



Karan Bajaj


The image is processed using the above program. Lets name the file BB2.m On executing the program –

We have approximately localized the violet colour region from the real world image. This choice of intensity selection has to be done manually and with great care, as the final binary image will wil l be computed accordingly. For -

23

if(im(i,j,1)>50 if (im(i,j,1)>50 && im(i,j,2)>50 && im(i,j,3)>120);



Karan Bajaj


Noise Removal As we saw in the first binary picture of the flower, the resultant image is distorted, with presence of holes, irregularities, etc. This can be treated using the imclose() function. But for using the imclose() function, we need to define a structuring element first, using the strel command as follows –

Combining this with the program –

Modified Output 24



Karan Bajaj


Another important function used for filling all holes in a binary or grayscale image is – imfill() function Using the imfill() function along with the program –

Resultant binary image that is produced does not contain any hole –

Similar to imclose() function, we have the imopen() function, which performs morphological opening of the grayscale or binary image. image . Example – I = imread('snowflak imread('snowflakes.png'); es.png'); imshow(I)

25



Karan Bajaj


se = strel('disk',5); strel('disk',5); I_opened = imopen(I,se); imshow(I_opened)

What actually happens here is that all snowflakes with radius less l ess than 5 pixels were removed by using the structuring element of type „disk‟ with size 5 pixel, which is used. Image Noise removal is very essential for making the data to be computed error free, and hence, our program need to be efficient at it. 26



Karan Bajaj


Some more important commands used for Image Processing are – regionprops() STATS = regionprops(BW, regionprops(BW, properties)

It measures a set of properties for each ea ch connected component (object) in the binary image, BW. The image BW is a logical array; it can have any dimension.

For example, if you want to locate centroid c entroid of any object in a picture you can use the following command s = regionprops(BW, 'centroid');

Web Link: http://www.mathworks.in/help/toolbox/images/ref/regionprops.html

bwlabel() [L, num] = bwlabel(BW, n)

It returns a matrix L, of the same size siz e as BW, containing labels for the connected objects in BW. The variable n can have a val ue of either 4 or 8, where 4 specifies 4-connected objects and 8 specifies 8connected objects. If the argument is omitted, it defaults to 8. 27



Karan Bajaj


[L, num] = bwlabel(BW, n)

It returns in num the number of connected objects found in BW. Web Link: http://www.mathworks.in/help/toolbox/images/ref/bwlabel.html

imcrop() I = imcrop creates an interactive Crop Image tool associated with the

image displayed in the current figure, called the target image. The Crop Image tool is a moveable, resizable rectangle that you can position interactively using the mouse. I2 = imcrop(I) displays the image I in a figure window and creates a

cropping tool associated with that image. I can be a grayscale image, a truecolor image, or a logical array. The cropped image returned, I2, is of the same type as I. Web Link: http://www.mathworks.in/help/toolbox/images/ref/imcrop.html

bwboundaries() B = bwboundaries(BW) bwboundaries(BW) traces the exterior boundaries of objects, as

well as boundaries of holes inside these objects, in the binary image BW. B = bwboundaries(BW,conn) bwboundaries(BW,conn) specifies the connectivity to use when

tracing parent and child boundaries. conn can have either of the following scalar values.

Web Link: http://www.mathworks.in/help/toolbox/images/ref/bwboundaries.html 28



Karan Bajaj


USING THE PORT Now we have a scenario where we have to interface the computer with the microcontroller. When an image is processed, we have a signal for performing some kind of work. This signal needs to reach the uC (short for microcontroller), which is done by the port. port. We will demonstrate Serial and Parallel ports in our tutorial. PARALLEL PORT

SERIAL PORT

In laptops/computers of today, we generally do not have parallel ports, and some modern laptops (like mine!) do not have Serial port also. But they do have multiple USB (Universal Serial BUS) ports. So we use USB to Parallel, or o r USB to Serial port converters – USB to Serial Port

29

USB to Parallel Port



Karan Bajaj


Parallel Port The parallel port has 25 pins, which are numbered as follows –

The pinouts for a Parallel port are –

DB25 pinout 30



Karan Bajaj


Before you being working with parallel port, do check if the parallel port in your system is operational, by following these steps – My Computer>System Properties>Device Manager>Ports You will find the name of your parallel port from here. Let us assume it is – „LPT1‟ >> parport = digitalio (‘parallel’, ‘LPT1’);

We have now defined an object named „parport‟. Port address may be obtained using these statements – >> get (parport, ‘PortAddress’) >> daqhwinfo (‘parallel’);

Now we need to define PINS 2-9 as OUTPUT pins using the addline() function – >> addline(parport, 0:7, ‘out’) Now, depending upon the motion of the robot desired, you can put the data in a matrix using logical() function – >> dout = logical([1 0 0 1 0 0 0 1]);

Using the putvalue() function, we can output this value – >>putvalue(parport, dout);

We can also send the decimal (or octal/hexadecimal) value directly – >>dat = 259; >>putvalue(parport, dat);

While using the parallel port, we do not need any microcontroller, microcont roller, as outputs can be directly received by the motors from the parallel port using H-bridge (L293D or L298 IC – Motor driver ICs) for driving the motor in both directions – clockwise and anticlockwise.

31



Karan Bajaj


Serial Port A Serial port has 9 pins, which are numbered as follows –

Serial transfer means that if you want to transfer a byte of o f information, then Serial port will transfer the 8 bits bit-by-bit bit -by-bit (one bit at a time). As seen from the above diagram, the transmission happens from PIN3 (Tx), and receiving happens at PIN2 (Rx). Now, we would require a microcontroller with UART (Universal Asynchronous Receiver and Transmitter) to be present. Most of the uC that we use in Robotics (like ATMEGA 16L, PIC, etc) have UART, which needs to be initialized for Serial data transfer. The standard used for Serial communication is RS-232 (Recommended Standard 232). This defines the voltage values that would be equivalent to logic ZERO and logic ONE. Valid signals are positive and negative 3 to 15 volts. 32



Karan Bajaj


Now the problem that arises is that our microcontroller works on o n TTL standards, and hence would not work on RS-232. So, here we introduce IC MAX 232, which would help us in this situation. The connection diagram of MAX 232 with Serial port, for our purpose is given is follows –

PINs 11 and 12 can now be used for communication via pins of uC meant for Serial transfer of data. Before you start using the Serial Port, check in the Device Manager (Port section), whether you can see COM ports available. If yes, then your Serial port is ready for some action! Let us assume your Serial port device name to be „COM1‟ „CO M1‟ – We can create a MATLAB object using the following commands – >> ser = serial (‘COM1’, ‘BaudRate’, 9600);

with Baud Rate = 9600 bps (bits per second) 33



Karan Bajaj


Baud Rate - In telecommunications and electronics, baud is synonymous to symbols per second or second or pulses pulses per second . While setting this value in the above command, we must make sure that the value is the same as set while configuring the microcontroller on-board the robot. The Serial data format includes – 1 Start bit 5 to 8 Data bits 1 Stop bit Parity bit (may be there) Additional Stop bit (may be there)

Asynchronous transmission using the UART character So the data that we are sending must be of 8bits. Let us suppose that we keep codes in hexadecimal – „A‟ – Forward „B‟ – Backward „C‟ – Turn left „D‟ – Turn right No we open the Serial port and start sending data through it – >> fopen (ser); >> fwrite fwrite (ser, ‘A’); >> fwrite (ser, ‘C’);

Signal for moving the robot one unit FORWARD, then turning LEFT. 34



Karan Bajaj


This data has to be sent to the microcontroller. So now we w e need to program the microcontroller to „listen‟ to the COM1 port. Let us take ATMEGA16L as our microcontroller. We would generate the program to be burnt, using CodeVision AVR software. A typical window of CVAVR looks like this –

We need to go to File>New This appears –

Select „Project‟ and click on OK. From the new menu, select the first option, and click on OK

35



Karan Bajaj


Now comes the most important part – configuring the microcontroller for use. The window that opens shows several tabs on the left –

Select „ATmega16L‟ (or any chip you are using) from the „Chip‟ tab. We now need to configure the microcontroller for using its USART in asynchronous mode, and along with that set the PORTs of the uC which would be used for signaling the motors. 36



Karan Bajaj


Select the „USART‟ tab from the same window –

Make the necessary settings in this t his submenu – putting a tick on the Transmitter option, and Receiver also if it is being used in your case. The Baud Rate is by default set to 9600. Notice that this is the same as the rate set by us while defining the Serial port object in i n MATLAB. Decide and set the communication parameters, and set the Mode as „asynchronous‟. PORTs can be set using the „Ports‟ tab –

37



Karan Bajaj


Once, all settings are done, Click on Program>Generate, Save and Exit Write the same names in the „Save‟ windows that come. The program now appears in front of you with all configuration settings done.

Browse down the window with the program code to find this –

You can now write your code here for accepting data from Serial S erial port, checking it across a set of operations (like moving left, forward, blinking LED, etc), and perform the required action using the respective PORTs that you already have set as input/output. Output from here would now drive the motors. 38



Karan Bajaj


MOTOR CONTROL Now all the data acquisition and processing is complete. All Al l we need to do is to RESPOND to the instructions that the microcontroller gives. We would do this by interfacing the Output PORT of the microcontroller with the motors. The problem that arises is – the output potential of the port of microcontroller is not enough to drive a motor, hence, we need to use a Motor driver IC (L293D, L298) for doing the same. We would use these with both DC motors and Stepper motors as given below.

DC Motors

As already mentioned above, the DC motor would require an IC for working with the microcontroller output. We can use L293D for this purpose –

39



Karan Bajaj


Pin diagram of L293D Each L293D has four Input-Output pairs. For making a motor run both directions, we need to use two Input-Output pairs per motor. Hence, we can use ONE L293D for controlling two DC moto rs –

40



Karan Bajaj


Shown above is the connection diagram of the two DC motors with one L293D IC.

Vs-pin Vs-pin stands for Motor Voltage and is the voltage with which the motors would be driven. We can use a 12V DC connection on this pin. Enable pins must be set HIGH for enabling the working of both motors. moto rs. These pins can also be used to DEACTIVATE the motor functionality under certain circumstances as directed by the algorithm. For various INPUT conditions at IN1, IN2 and IN3, IN4, we see the following pattern per motor -

DC motors are simple, but they lack accuracy. In case we want to use DC motors, we have to use an efficient feedback mechanism to overcome and compensate the loss of accuracy. 41



Karan Bajaj


Stepper Motors

It is a brushless, electric motor that can divide a full rotation into a large number of steps. The motor's position can be controlled precisely without any feedback mechanism. How it works?

I

II

III

IV

Hence, a complete rotation happens after four such steps. The angle which the Stepper motor rotates with after completion of any step above is called the Resolution of the Stepper motor. Steppers are available in many Resolutions, ranging from 30 to 1.8 degrees per step. Stepper motor can be either Unipolar or Bipolar.

42



Karan Bajaj


As you can see, each stepper has four coils. Hence we have four connections per stepper motor. So, one L293D can drive one bipolar 2 phase stepper motor. The connection diagram of the same is given as follows –

We have the four Controller Pins going to the OUTPUT port of the microcontroller. And the data coming through these pins will be controlling the operation of the Stepper. We saw in a diagram above that there is a specific sequence in which coils should get activated, so as to ensure proper completion of a cycle. The pulse sequence has to be in the following order –

Step Step Step Step

1 2 3 4

Coil 1A High Low Low High

Coil 2A High High Low Low

Coil 1B Low High High Low

Coil 2B Low Low High High

*For reversing the direction, follow the steps in opposite order. 43



Karan Bajaj


Unipolar Stepper Motors A Unipolar stepper motor has one winding with center tap per phase. The special thing about Unipolar Steppers is that we can reverse the direction of rotation of the motor without changing the direction of current.

For Unipolar Stepper, we use IC ULN2003/2004 instead of L293D, as it also has a COM port for serving as the Common connection of the coils.

44



Karan Bajaj


Thank you for going through the tutorial. If you have any queries or suggestions that you wish to share, feel free to contact me at [email protected] [email protected]

45



Image Processing Based Robotics.pdf

Recommend Documents