Visual Robotic Arm Position Identification  using Deep Learning and CNN

    This post is looking into solutions to use Artificial Neural Networks and  Deep Learning  methods to determine the position in the XY plane of a robotic arm end pin ( effector)  using visual recognition and Deep Learning algorithms applied thru Keras and Tensorflow.

   It is a lot to describe and absorb in the above paragraph, so I will start with the target description and work back from there,

  This post will not describe the theory supporting the application but merely a practical approach based on the huge amount of free source available "out there"
   In the associated post I described an I2C robotic arm deives by using I2C IMUs  and  the associated hardware / software bundle.
    The use of the I2C sensors is driving a real messy hardware installation with the I2C network management.
   True is that the most of the complications with harness is coming from the limited range of addresses available for the MPU6050 sensors (cheap) which i use.

   Obvious the arm in the close view is the I2C one usinfg IMU sensors.
   As I was reading more about Deep Learning and AI applications I looked for me that maybe a simpler way to "measure" the parameters defining the arm end effector position.
   I could see it as an application where the "categories" defined in the image recognition process are actually well defined X,Y position of the end effector characterized by a distinct shape of the robotic arm visualized with eventually a wide angle computer connected camera.

This is what the camera would see eventually:

And images taken in different points (and arms angles) look something as :

Note: To list your cameras you can use under Python:
======================================
def list_cameras():
    import cv2
    for cnt in range(0,612,1):
        img=[]
        cap = cv2.VideoCapture(cnt) # device 1
        img=cap.read()[0]
        if img:
            print 'Video', cnt,img
        cap.release()
    return 

========================================================

   

I2C Robotic Arm Project

( - if you do not like I2C do not read this blog
  - it is a lot of information here which relays on your previous knowledge or the wiliness to learn more about Debian , Python and I2C )

Scope :  Create an I2C controllable Robotic Arm based on 

Raspberry PI hardware   and cheap additional hardware - from drives to sensors.

1.  Short Story:  Originally the project started the development using as core system a PI 2 basic hardware and integrating I2C peripherals as needed. 
2. I found fast that for development reasons I needed a faster and more extended hardware - so the project was migrated toward a 4 GB memory  Pentium dual core system with 250 GB HDD.  Thee project can be migrated back to PI without any major modifications.
3. I kept the same basic operating system - by installing Debian 8 64 bits on the machine. 
4. The whole trick is that a regular PC platform has available sometime few I2C networks and one of them is accessible thru the VGA port.see the link (I2C adapter)
5. I need to underline that the whole project  is mostly a result of compiling existing information but I have to say that the I2C availability on the PC hardware really gave a bust to the development.
6. Now I have to say that really the only think which you need from the site is to know that this network exists.
7. The I2C is "naturally" accessible from Debian and also from R-PI.
8. Not being a Linux guru , I can say the installation of Debian 8 was smooth and elegant. I works fine on Intel processors not so much on equivalent Athlon (at least Chromium doesn't like my Athlon)
9. I run the system in standard user mode but by using su (super user ) command and launching the applications from  a su "owned"  terminal  the hardware resources are easier to access. 

==========================================================

It is a lot to say and explain so I will try to start by

using a general schematic of the project

General Schematic:

Note: Probably the above image will see a lot of updates as the project description is evolving.
===========================================================
- The control hardware can by the R-Pi or a PC platform as I described above.
- The operating system for compatibility with PI and also for easiness of hardware access is the free open source Linux Debian 8 (see link: Debian - and thanks to the developers for the great work !)
===========================================================
- The development language is Python 2.7 or if you want you can migrate toward a newer version (3...). (see link:Python). Note: the IDE which I'm using is IDLE - and all this language /environment is install-able very easy from Synaptic Package Manager.
- One of the most useful  application will be the Web Browser - if you need to clarify project details...(Ice weasel) search , search , search...
- So obvious some programming experience is needed here and learning Python can be a funny / frustrating experience pending from which direction are you coming .. or looking.
- Freemat ( see: Freemat ) is an excellent free (thanks !)
===========================================================
alternative to Matlab style scripts and graphic representation - it can be used for post processing data representation.

===========================================================
- GitHub (GitHub) is the main information and sources resource and the Adafriut repository(Adafruit)is essential.
- Due to the migration to PC platform for development some very basic programming sources needed minor modifications for the I2C access  - mostly for pointing toward the correct I2C available network.

==========================================================
The Robot Mechanical Hardware (OWI)

Mechanical dimensions

Range of movements

Original Electrics : 6 V DC ON /OFF (not used in this project)

==========================================================

The I2C PC network from VGA:

- You can ignore the next section if you use a R-PI 2 as there the I2C is available on the 40 pin connector.(SDA , SCL)
- The VGA port has a native I2C connection available.
- The easiest way to access it is if your card has a dual port , by attaching an VGA extension cable, or eventually connecting directly in the card connector "holes". I guess this will work with an laptop by connecting to the extension VGA connector.

- The easy way to identify the pins is by looking for the 5V pin with a voltmeter. This will place you on the "map". In fact in my application I'm connecting to a DVI port with an adapter to VGA and then I'm connecting the extension cable (from an old KVM switch).
==========================================================
How to check your I2C network (Debian 8 and R-PI):
- launch terminal
- go in su mode (Debian only )
i2cdetect -l
// (to list all networks)

- if you have an I2C device attached on the network you should see it on one of the networks with

i2cdetect -y 0
//(or 1 or 2 or 3 .. usually I find   0 or 1  hosting the devices).
Note: it is possible that a computer reset or shoot down the devices will be on a different net as I learned.

The devices for which I'm looking have addresses 0X40 , 0X48 , 0X68, 0X69 , so is the I2C number 1

==========================================================

The I2C devices 
 which I use are already on some small boards so all signals and additional components are on the board - so is no need for the additional resistors and very rarely for 3.3 V

E-bay or Alliexpress are the  main sources for the sensors and devices:

==========================================================
Electrical Hardware:

I guess with  a carefully search  and if you are patient with delivery time, total  it  can get under 50 USD except the robot itself ( around 35).

 ======================================================

- motor driver  PWM driven good for 2 motors (using 3  boards)

Dual H Bridge Stepper Motor Drive Controller Board Module For Arduino L298N UL

==========================================================

- 16 bit AD 4 channel AD converter

16 Bit I2C 4 channel ADS1115 Module ADC with Pro Gain Amplifier for Arduino

==========================================================
- Acceleration / angular sensor :(using 4-5 of them )

MPU6050 3 Port 3-Axis Acceleration Gyroscope 6DOF Accelerometer Sensor

==========================================================

-Connecting wires:

40pcs Dupont 20CM Male To Male Jumper Wire Ribbon Cable Breadboard 

========================================================
- I2C switch (8X I2C) this board is needed to switch between sensors which has the same I2C address .

TCA9548A I2C Multiplexer Breakout board for chaining Modules

or
PCA9548A 8 Channel I2C Bus Switch w/ Reset Breakout for Arduino uControllers

To access the I2C channels thru the board the next code is what I used the following PYTHON code:

********************************************************************
#!/usr/bin/python
# TCA9548A I2C multiplexer
# I2C Address:  70 through 77
# Channel: 0 - 7
#-------------------------------------------------------------------
import smbus

# class for the I2C switch------------------------------------------

class I2C_SW(object):

# init   procedure
    def __init__(self,name,address,bus_nr):
        self.name=name
        self.address=address
        self.bus_nr=bus_nr
        self.bus=smbus.SMBus(bus_nr)
        
# Change to i2c channel 0..7__________________________________________________
    def chn(self,channel):
        self.bus.write_byte(self.address,2**channel)
        
# block all channels read only the main I2c ( on which is the address SW)_____
    def _rst(self):
        self.bus.write_byte(self.address,0)
        print self.name,' ','Switch reset'

# read all 8 channels__________________________________________________________
    def _all(self):
        self.bus.write_byte(self.address,0Xff)
        print self.name,' ','Switch read all lines'

# define the usual sensor 0X70 bus 1

SW=I2C_SW('I2C switch 0',0X70, 1)
SW._all()
SW._rst()

# to enable a channel : SW_chn(channel number - here 0 to 7)

# check with i2cdetect y -1  (if bus_nr=1)

*********************************************************

Because the PWM driver (next component) is broadcasting 0X70 and 0X40 the I2C Switch adress is changed by hardware to 0X74.

Which means to short solder 2 small pads on the board:

and in above code address 0x74 must be used not 0x70
==========================================================
- I2C PWM generator  - this is essential for your drivers command

PCA9685 16-Channel 12-bit PWM Servo motor Driver I2C Module

==========================================================

- Analog 16 Channel mixer

 CD74HC4067 CMOS 16 CH Analog Digital MUX Breakout Board

=========================================================

- USB Joystick (Joypad) - connected to the USB port

==========================================================

 For all I2C devices basic Python application can be found on GitHub or instructions on Instructables, or R-PI associated sources.

 I will fully share all Python sources in this post later.

The robot fully wired .- not pretty yet...

Back to robot philosophy
======================== 

- The basic accomplishment should be to be able to use an USB  joy-pad  to proportionally drive the electric motors Mxx with the displacement of the 2 X  2 axis joystick in the pad.

The association between the joystick movements and the motor movements are presented in the above picture.

- Further a more elaborate accomplishment is to associate the movements of the joysticks with a X/Y 2D space in the plan of the robot arm.
- In fact the whole construction is targeting the scope of being able to move the gripper in a controlled manner in an X/Y coordinate system with the origin in the rotational axis of the arm and ground.
- This control can be moved easy in 3D by extending the coordinate system to a cylindrical one (R,Ro,Fi).

The final hardware setting looks as (the cat is only wondering..) :