What are we doing today
- Deep Learning for Semantic Segmentation
- An image is a collection of pixels
- Can calculate the distances to things
- Finds the same features on the frames
- known distance in between cameras
- Sensitive to amount of features
- We have the knowledge in C++ to describe the logic we might want a robot to
have. But we need to be able to make sense of what the robot sees and
classify it before we can act on this logic.
- Cue OpenCV, an open source computer vision library with bindings for C++
(and a few other languages)
- I guess our ability to see has been ++'d
- Industrial standard for image processing
What does an Image look like to your computer?
- OpenCV stores images in an object called a Mat
- A Mat is an array with rows and columns. Each element
of the Mat is a pixel in the image and its location in the Mat corresponds
to its location in the image
- Computers have no concept of "2d", so Images in memory are continuous.
This means each row of the image is appended onto the end of the last. To
iterate through a Mat you just get a pointer to the beginning of the first
row and keep track of your row number by how far you've traversed.
- There are many different formats for an image
- An image where each pixel is only white to black
- Range [0-255]
- Color images don't embed the color of a pixel in one element. Often, you'll
find each pixel represented in BGR (Blue component, Green Component, Red
Component) form. So now, each row of a color image is 3 times as long as a
row of a black and white image.
- Images don't have to be stored in just BGR format!
- Each Pixel in a color image has a hue, a saturation, and a luminosity.
- Even though our cameras read in images with RGB, converting them to HSV is
easy with OpenCV
- Indicates the amount of grey
Why do we use HSV
- HSV encodes image data in a way that is resistant to changes in color
- To put it another way, on a sunny day an image will contain more red, more
blue, and more green than on a cloudy day. All three channels are affected.
- On a sunny day, the saturation channel will be largely effected, but we can
expect hue to remain mainly stable. This makes it easier to do searches for
colors in the HSV space.
- Sometimes, CV problems present themselves in such a way that Deep Learning becomes the best solution.
- Tensorflow is a library containing machine learning algorithms.
- We use it to provide an implementation of a neural network to perform semantic segmentation for line detection
- Semantic Segmentation is the art of grouping pixels in an image into clusters that share a meaning.
- In our use case, we classify pixels as "Not a line" or "Is a line"
- To do this, we utilize something called a Convolutional Neural Network, or CNN
What is Convolving?
- Convolving refers to the procedural application of a kernel to an image.
- A kernel is a square matrix
- Each pixel value in a region of the image is multiplied by its corresponding element in the matrix, then summed for a result.
- can be used for edge detection, reducing noise, feature extraction, etc
- Returns the original image
- used to reduce noise
- replaces the center pixel with the average of all of it's neighbors.
Back to CNN
- These kernels are used to generate feature maps, which are then subjected to certain preprocessing steps and fed into a neural network as input, which then classifies each pixel as line or not-line.
- To train our network, we manually labelled many images from past competitions and testing runs, which allows our network to preform well for our use cases.
- For the more curious amoung you, here is a beginner's guide to CNNs that goes into more detail: CNN Guide
I bet you all have wondered why I have gathered you here today.
- As many of you might have noticed, we are in the shop rather than on campus.
- In addition, many of you might have noticed our poor documentation habits.
- We are going to be walking you (and us) through different parts of the codebase, explaining and documenting things as we go to help you get onboarded and clean up some shoddy code.
- Welcome to the Shop!