A very simple problem with large language models, which frequently gets highlighted, is their poor ability to accurately do simple grade school math. Something that would be easy for human beings, even just in their head, can be surprisingly difficult for these models to do reliably.

There are many benchmarks for model performance on simple math problems, and their results are often disappointing. This isn’t that surprising, given that their training is based on next token prediction, which isn’t as well suited to number crunching as a calculator. Computers though, should be very good at number crunching. If we could just hook up LLMs to a calculator, we could fix this problem.

LLM Tool Calls

If we can get the LLM to output something that looks like a function call, we can provide some parsing logic, actually execute the request deterministically, and provide the final answer back to the model. This way the model is only concerned with providing the right inputs, for which it can leverage its next token prediction, and then we can be certain that with the right inputs we will also be getting the right outputs.

Calculator Example

Let’s write the simplest possible LLM tool call for a calculator. We can use Python for this. A similar example is also provided by OpenAI’s function calling page.

  1from openai import OpenAI
  2import json
  3
  4client = OpenAI(api_key="your_api_key_here")
  5
  6# 1. Define a list of callable tools for the model
  7tools = [
  8    {
  9        "type": "function",
 10        "name": "calculate",
 11        "description": "Execute a simple arithmetic operation on two integers.",
 12        "parameters": {
 13            "type": "object",
 14            "properties": {
 15                "first": {
 16                    "type": "integer",
 17                    "description": "First number",
 18                },
 19                "second": {
 20                    "type": "integer",
 21                    "description": "Second number",
 22                },
 23                "sign": {
 24                    "type": "string",
 25                    "description": "Symbol to execute the operation",
 26                },
 27            },
 28            "required": ["first", "second", "sign"],
 29        },
 30    },
 31]
 32
 33#2. Define the function logic that will execute when the model calls the tool
 34def calculate(first: int, second: int, sign: str) -> int:
 35    if sign == '+':
 36        return str(first + second)
 37    elif sign == '-':
 38        return str(first - second)
 39    elif sign == '*':
 40        return str(first * second)
 41    elif sign == '/':
 42        return str(first / second)
 43    else:
 44        raise ValueError(f'Unknown sign: {sign}')
 45
 46# Create a running input list we will add to over time
 47input_list = [
 48    {"role": "user", "content": "What is 221 * 645?"}
 49]
 50
 51print("User Question:")
 52print("What is 221 * 645?")
 53print("\n")
 54
 55# 3. Prompt the model with tools defined
 56response = client.responses.create(
 57    model="gpt-4.1-mini",
 58    tools=tools,
 59    input=input_list,
 60)
 61
 62# Save function call outputs for subsequent requests
 63input_list += response.output
 64
 65# 4. If the function call is present in the model's response, attempt to execute it
 66for item in response.output:
 67    if item.type == "function_call":
 68        print("Model Output:")
 69        print(item)
 70        print("========LLM Invoked Function Call============")
 71        print(f"Function Call: {item.name}")
 72        print(f"Arguments: {item.arguments}")
 73        if item.name == "calculate":
 74            # 5. Execute the function logic for calculate
 75            first = json.loads(item.arguments)["first"]
 76            second = json.loads(item.arguments)["second"]
 77            sign = json.loads(item.arguments)["sign"]
 78            result = calculate(first, second, sign)
 79            
 80            # 6. Provide function call results to the model
 81            input_list.append({
 82                "type": "function_call_output",
 83                "call_id": item.call_id,
 84                "output": result,
 85            })
 86
 87            print(f"Function Call Output: {result}")
 88        print("=============================================")
 89        print("\n")
 90
 91# 5. Prompt the model again with function call outputs for final answer
 92response = client.responses.create(
 93    model="gpt-4.1-mini",
 94    instructions="Respond only with the full answer in plain text format.",
 95    tools=tools,
 96    input=input_list,
 97)
 98
 99print("========LLM Recieved Call Answer============")
100print("Final output:")
101print(response.output_text)
102print("============================================")

When the model registers that calculate would be useful it sends a response to us that looks like this:

1[ResponseFunctionToolCall(
2  type='function_call', 
3  name='calculate',
4  arguments='{"first":221,"second":645,"sign":"*"}'
5)]

At step 4 in our code we check for any responses with type function_call. If we find one we parse out the function name. We use the provided paramters to run the appropriate function as a "tool call". Then we simply provide the result of the function to our chat history for the model to observe.

The last part is simply asking the model to format the function’s output as a complete sentence which it can easily do.

Here is the full print output of the code block:

 1User Question:
 2What is 221 * 645?
 3
 4
 5Model Output:
 6ResponseFunctionToolCall(arguments='{"first":221,"second":645,"sign":"*"}', call_id='call_IA3XFzdm5YcamN26Nd7WeISX', name='calculate', type='function_call', id='fc_088a460995856027006a51ee403ea081929c055bb039b6a32c', caller=None, namespace=None, status='completed')
 7========LLM Invoked Function Call============
 8Function Call: calculate
 9Arguments: {"first":221,"second":645,"sign":"*"}
10Function Call Output: 142545
11=============================================
12
13========LLM Recieved Call Answer============
14Final output:
15221 multiplied by 645 is 142,545.
16============================================

Generating Testing Values

In order to test how well a model can predict random arithmetic operations we have to first generate a few random values so that we can compare the results with and without tool calls. We will be using a more difficult examples with decimal points to stress test the model. I picked two ways of generating values randomly:

1. For a linear approach we pick a random magnitude from -5 to 5 as the exponent and a random value from -1 to 1 and we get a random value back of value × 10^exp
2. For a gaussian approach we use a normal distribution with a set standard deviation. This ensures that our values are a bit closer to what we would consider normal, from around 100 to 0.001 with a low chance of outliers.

Both distributions can be seen visualized below for 100 distinct values.

We can also heatmap the values into a 2D grid to see where the values are concentrated. The heatmaps show the values for both the linear and gaussian distributions.

We’ll use both of these distributions to generate random values for testing the model’s ability to do simple math with and without tool calls. The next step is to run a series of tests using these generated values and compare the results.

Testing the Model without Tool Calls

With the test data we have available we can pick two random values and a random operator and ask the model to calculate the result without using tool calls. We can then compare the model’s output with the actual result of the operation and see how well it performs.

Here are the results for the linear distribution of input data using the gpt4.1 mini model, as it yielded the most interesting results.

The mini model performed reasonably on its own, getting a 77.2% accuracy without any tool calling. However with tool calling this number shot up to 99.4% with the only errors coming from the model confusing a negative operator as belonging to the operator itself as well as the second number.

We can also compare the tokens required, time taken and estimated cost for both approaches.

Click on the images to open in a new tab to see the full detailed stats. The results are also temporarily available at this website.

Summary

Adding the calculator made a large difference in these tests. On the log-uniform questions, the pass rate went from 78.2% to 99.8%. On the log-Gaussian questions, all 1,000 calculator-assisted answers passed. That improvement came with a second model request and roughly 7x to 8x the total tokens.

These results are specific to this model, these generated questions, and the way I compared answers at their returned precision. They were enough to answer the question I started with: a calculator tool call helped substantially, but it came at a cost of more time, tokens and money, without a 100% reliability guarantee.

This fear is completely irrational, and I always kick myself for feeling this way because binary search is beautiful. Given a sorted list you can find the index of any value in log(N) time because you get to cut your search space in half with every lookup.

Precisely this operation allows sorted lists to be very space and time efficient at solving problems requiring lookups as well as determining how many entries are greater or less than your target.

Now if you want to take advantage of your sorted lists you may be tempted to write your own binary search and insert functions like I’ve been doing so far and inevitably it takes a few tries to actually get it working right. Thankfully there’s a built in library that already does the same thing, in a single line and a lot faster too.

Bisect

The bisect library is a tiny one, but it does two things and it does them well; search and insert.

For search we have bisect_left and bisect_right which perform binary search to find where to insert a value right before our target on either the left or right. In the case of bisect_right it will overshoot by one index to show you where you should insert your new value to maintain order.

list: [1,2,3,4,5], target = 4

• bisect_left: [1,2,3, _ , 4,5] index= 3
• bisect_right: [1,2,3,4, _ ,5] index= 3+1

For insertion we have insort_left and insort_right which perform binary insert to actually insert our value on either the left or the right of our target and manipulates our list inplace.

• insort_left: [1,2,3, 4 ,4,5] inserted index= 3
• insort_right: [1,2,3,4, 4 ,5] inserted index= 3+1

Apart from bisect and insort which are equivalent in function to bisect_right and insort_right respectivly, these are the only functions available in this library. However they completely replace the need to write your own functions for lookups and inserts.

Bisect is Fast

 1def binary_search(lst, target):
 2    start, end = 0, len(lst) - 1
 3    while start <= end:
 4        middle = (end+start)//2
 5        if target > lst[middle]:
 6            start = middle + 1
 7        elif target < lst[middle]:
 8            end = middle - 1
 9        else:
10            return middle
11    return -1

And here’s a binary race between it and bisect. We’ll be looking for value 0 because ironically it will take the longest to find with binary search since 0 is never in the middle of anything.

 1#list and target
 2input_list = list(range(1000000))
 3target = 0
 4
 5#my binary search
 6start = time.time()
 7index = binary_search(input_list, target)
 8duration = time.time() - start
 9print("search found {} at index {} in {} seconds".format(target,index,duration))
10
11#bisect binary search
12start = time.time()
13index = bisect.bisect_left(input_list, target)
14if index == len(input_list) or input_list[index] != target:
15    index = -1
16duration = time.time() - start
17print("bisect found {} at index {} in {} seconds".format(target,index,duration))

• search found 0 at index 0 in 1.6927719116210938e-05 seconds
• bisect found 0 at index 0 in 2.6226043701171875e-06 seconds

Here we can see that bisect is 6 times faster than my implementation

Because bisect is implemented in C it is much faster, though both functions use the exact same algorithm. Python is just a lot slower at adding numbers than C because all numbers in Python are actually Integer classes which in turn take time to call their addition functions whereas in C integers are just 4 byte data types and are processed very efficiently by the CPU. The difference in time can start to compound into a tangible boost in performance when you start doing many lookups or insertions sequentially.

Conclusion

Bisect is a small but useful library that helps maintain sorted order in lists as well as providing lookups in a quick and efficient way. Knowing about its existence can save on both implementation and runtime and it is an indespensible tool for many coding interviews where questions about sorted lists tend to crop up from time to time.

The real intrigue was the calculated but unexplained time complexity for one of the inefficient solutions which only used recursion, minus the dynamic programming. After all, dynamic programming makes time and space complexity fairly easy to compute, as long as the solution is correct, it cannot exceed the complexity of all possible subproblems. Before I can get ahead of myself though here’s the outline of the original problem which can also be found here https://leetcode.com/problems/regular-expression-matching/

If at some point I lose you during the explanation please leave a comment so I can make a revision.

The Problem:

Given an input string (s) and a pattern (p), implement regular expression matching with support for '.' and '*'.

1'.' Matches any single character.
2'*' Matches zero or more of the preceding element.

The matching should cover the entire input string (not partial).

Note:

• s could be empty and contains only lowercase letters a-z.
• p could be empty and contains only lowercase letters a-z, and characters like . or *.

Some Examples:

Example 1

1Input:
2s = "abcda"
3p = "ab.d"
4Output: false
5Explanation: The . matches the c but the pattern ends before matching to the entire text.

Example 2

1Input:
2s = "aab"
3p = "c*a*b"
4Output: true
5Explanation: c can be repeated 0 times, a can be repeated 2 times. Therefore, it matches "aab".

Example 3

1Input:
2s = "ab"
3p = ".*"
4Output: true
5Explanation: ".*" means "zero or more (*) of any character (.)".

The examples above summarize the kind of cases you can expect to be dealing with during testing. Because the title of the article is “Solving for Recursive Complexity” and not “How to Solve Recursive Problems” I won’t be going too deeply into how to solve this kind of problem. Instead I will resort to flashing a piece of pseudo code that explains the 3 recursive gates that our algorithm will be passing through on its journey to the end of the piece of text. I’d recommend trying to solve this problem on your own by going to the link I included above, and coming back here once you’ve gained a level of appreciation for this problem, because I’m about to spoil it for you.

Solution Pseudo-code:

Quick Explanation

If you’re not very familiar with recursion, this code block might still look very intimidating. The gist is that we’re only doing two things:

1. checking if the current character of the text matches the pattern and

2. asking if the pattern will match the text if we modify the text a bit or the pattern a bit or both for the remaining characters.

**Line 12: keep_asterisk_match is true if we don’t change the pattern but move along the length of the text (text[1:]). This is how * is dealt with in recursion, we simply move on to the next character of the text like the current one never existed and keep the asterisk in the pattern.

Line 17: skip_asterisk_match is true if the text matches the pattern without the asterisk (pattern[2:]) i.e if we match 0 of the preceding element. Remember in Example 2 where c* didn’t have to match anything, and this is also how we jump out of the keep_asterisk_match trap that we made for ourselves earlier.

Line 20: Once we get an answer for whether we get a match when keeping or skipping the asterisk after keep_asterisk_match and skip_asterisk_match are run we return the result as shown in the OR table below.

Line 24: If we don’t see an asterisk we check if that first character matched the pattern with character_match and we keep recusing by reducing the input text and pattern by that first matched character (text[1:], pattern[1:]) and recursively check if the rest of the string matches.

Line 4: The final part of recursion which ironically is usually defined at the top of the function, is our recursion rock bottom. If we have an empty pattern and an empty string as our inputs, we have the divine knowledge that the pattern will match the string because empty matches empty for sure. If only one of them is empty we know that there’s no match.

Considering the Complexity

The time complexity for this question appears to be difficult to pin down. After all, there are two points of recursion and one full stop. And like I said before, the answer to the complexity is given in the solutions with no further explanations. For text of length T and pattern of length P, with the text being indexed at text[i:] and pattern[2j:], the time complexity for this question is….drum roll

Well, that’s quite something.

Normally you’d expect something like O(T*P) or anything else to that effect and in fact, the solutions do propose a more concise upper bound, again without any explanations (not surprisingly), but we’ll get to that later. For now we have enough on our hands as it is with this piece of work.

Linear vs Exponential Complexity

The reason why the complexity isn’t as simple as it is for something like bubble sort, is because we’re not making linear passes over our text and pattern. Instead we’re doing things recursively, diving deeper and deeper through sections of our text and pattern and revisiting the same sub problems for what can feel like an arbitrary number of times. The reason I bring up bubble sort, which has O(n²) complexity, is because its complexity decreases in the shape of a pyramid as we sort more and more digits.

On the other hand, recursion has the classic shape of a tree due to the multiple places where the function can call itself.

The structures look deceptively similar but the crucial difference is that where the pyramid decreases linearly, the tree grows exponentially. This creates a world of a difference when analyzing time and space complexity.

Our Worst Case

When analyzing worst case complexity we have to think of the worst case input. Because we have two inputs, text and pattern, I will focus on the pattern because one depends on the other and at least for me it’s easier to think of the rules rather than the results. In this question each new character we add to our pattern can be either a .* or . where . can be any character. From the code it should be pretty clear which choice causes more computation. If we include the * character, our code splits into two streams which must both be computed until inevitably they should return whether there’s a match, meanwhile if we are matching a single character the fun stops pretty quickly as we are only taking a single route, linearly.

It looks pretty clear that a worst case pattern would consist of many .* to create as many diverging paths as possible, with a final text character which does or doesn’t match the pattern.

Text: aaaaaaaaaab

Pattern: a*a*a*a*a*a*a*a*a*a*

In this scenario we must get both to the end of the pattern and to the end of the string to determine that there is no match, while branching 2¹⁰ number of times starting from the first index of the text.

Visualizing the Computation

In order to figure out the time complexity we have to find how many times each sub-problem had to be computed. Because we are not saving the result like in dynamic programming this will largely contribute to the exploding runtime. Each sub-problem consists of answering whether text[i:] and pattern[2j:] are a match. We use pattern[2j:] because we are using .* repeatedly in our pattern for the worst case, every time we take the skip_asterisk_matched path we skip one of the .* which take up two spaces. In order to visualize the two branches of skip_asterisk_matched and keep_asterisk_matched we will draw a tree diagram to illustrate all the visited sub-problems. Each node will have the format (i,j) for where the text and pattern are indexed at each point in time.

A quick side note on this, when performing recursion, the algorithm will perform a depth-first search by taking the left path first on each iteration. This doesn’t make much of a difference though because in the worst case we’ll have to traverse the entire tree anyways. Let’s take an example sub-problem where we want to see if text[2:] and pattern[2:] were a match. This will appear as (2,1) on our tree:

i=2 because we are at text[2:] for the text j=1 because we are at pattern[2:] which in the worst case means we have only a single .* asterisk that takes up two spaces. Following pattern[2j:] means j=1.

We can see that this sub-problem had been encountered 3 times during computation. The tree format is good for seeing when these sub-problems will be encountered but it’s not very good at determining how many times that will happen.

Combinations

The way to systematically consider how many times a sub-problem will be encountered is to consider how come these sub-problems tend to repeat themselves in the first place. In order to arrive at (2,1) on our tree we have to take two lefts and one right.

As we can see this can happen in three ways, if we traverse the tree in the sequence LLR, LRL and RLL.

In the example of (2,1) we know that we will have 2+1=3 place holders where 2 of the placeholders will be an L and 1 will be an R. In order to find how many combinations are possible we can simply compute 3C2 where out of 3 place holders we will choose 2 to be an L and the rest will automatically be Rs.

That means that for sub-problem (i,j) we have i+j placeholders with i of those being the left side and j being on the right side. This is why the solutions include the combinations symbol for the number of encountered sub-problems (i,j) with (i+j choose i).

Incidentally this is also equivalent to (i+j choose j) because it does not matter whether you are picking which i increments are on the left or instead picking which j increments are on the right, the number of distinct combinations will be the same.

Sub-Problem Complexity

Now that we know how many sub-problems of each kind we will encounter, we should figure out how long those individual sub-problem will take to complete. This sort of thing tends to be very easy to do with recursive problems, like in this case where we only really do one thing, compare a single character and store the result in character_match. This might lead you to believe that the runtime for each sub-problem is constant O(1), but in order to call the function we must first supply text[i] and pattern[j:] arguments. If the strings are long, this procedure is not trivial, especially in python where in order to pass in a sub-string a new string must be created. Therefore for each sub-problem there is a constant time O(1) for checking that the character matched, an O(T-i) for passing in the part of the string that we don’t know is a match yet, and an O(P-2j) for passing in the part of the pattern that we don’t know is a match yet. Adding everything together we arrive at the complexity given in the solution.

The Concise Solution

Remember earlier when I said that the solution proposed a more general bound, without the complex combination notation and sigma signs.

Well here it is.

Let’s go through the idea behind this one before we wrap up.

The idea behind making the time complexity into this simplified form is to go overkill on both how many sub problems we could encounter and how long we plan on those sub problems taking to compute.

Number of Sub-problems

Instead of using combinations to compute precisely how many sub problems we will encounter, which based on our combinations calculation will be only a portion of our tree’s leaf nodes, let’s just take the entire bottom level of the tree.

Let’s also make the tree completely full with a depth of the maximum number that \(i+j\) can take, which is \((T+\frac{P}{2})\).

• T for the length of the text and \(\frac{P}{2}\) for all .* instances in our worst case pattern

• A tree of depth \((T+\frac{P}{2})\) will have \(2^{(T+\frac{P}{2})}\) nodes at the lowest level.

In the image we can see the previous example of text[2:] and pattern[2:] being computed \(\binom{2+1}{2}\) = 3 times, but since we know that all these examples happen on the same level (because # of Ls+ # of Rs is always 3) we can compute the number of all sub problems on that level which is \(2^{(2+\frac{2}{2})}\) = 2³ = 8.

The number of nodes on the entire level (8 in this case) will always be larger than the \(\binom{i+j}{j}\) nodes (3 in this case), so we can use this calculation in our more general upper bound. So we have determined that our upper bound will assume that we will go through \(2^{(T+\frac{P}{2})}\) subproblems, even though we really never will, but it does make our formula more concise.

Sub-problem Complexity

Instead of computing how large the arguments will be before we pass them in, let’s just assume we are passing in the entire strings for both the text and pattern. This will give us O(T+P) time complexity for each sub-problem, eliminating the need to compute what O(T-i) and O(P-2j) are at every step.

Final Answer

Multiplying the Sub-problem complexity by the number of sub-problems gives us the final answer of:

A Case for Dynamic Programming

This was all incredibly tedious and time consuming on both ours and the computer’s part. At some point you probably thought that computing the same problem (i+j choose i ) times was a bit wasteful and if we just stored that result it would save us a lot of trouble. You’d be right.

There’s only T*P distinct combinations of (i,j), and each takes only constant time to verify, so simply storing all of them in a grid will bump our computation down to a whopping.

I hope you enjoyed this exercise in computational complexity. Seeing as you got to the very end, do give yourself a pat on the back because you deserve it.

As a team of three undergraduate students, we were tasked with building a concept self driving car in our winter semester. Not having much experience in data science or machine learning meant that the production process was a mess, although looking back we probably could have made the same amount of progress in a fraction of the time.

This article is for those who are interested in what it takes to build a functioning self driving car in record time, on a budget, without the headaches that we had to go through! I don’t go too much in depth on all the details but there are a few snippets of code to spice things up. Also if you are interested in the code you can help yourself to our Github page, https://github.com/RoboticsCourse, but be warned that it’s not documented and was not created for general use.

What We’ll Need:

• Remote Control/Demo Car
• Arduino Uno board
• Motor Shield
• Bluetooth BLE Module
• 2 Mobile Devices

Cars are meant for driving

First of all, we wanted to finely control the motion of the car, and the best way to do that was to throw away the controller that came with the original car and instead use an Arduino board. Since they’re programmable and compatible with a whole suite of additional modules which we could buy online for cheap, it was amazing for prototyping. The board we used was the Arduino UNO Rev3 which you can get for $20.

Now to actually be able to control the car via the Arduino board we had to hook it up to a Motor Shield which is a useful little add-on that gives you the ability to power a variety of motors from your Arduino board with just a few lines of code. The snippet bellow shows a simple implementation.

 1void Navigation::goForward(int speed) {
 2    frontMotor->setSpeed(speed);
 3    frontMotor->run(FORWARD);
 4    rearMotor->setSpeed(speed);
 5    rearMotor->run(FORWARD);
 6}
 7
 8void Navigation::goBackward(int speed) {
 9    frontMotor->setSpeed(speed);
10    frontMotor->run(BACKWARD);
11    rearMotor->setSpeed(speed);
12    rearMotor->run(BACKWARD);   
13}
14
15void Navigation::SteerSpeed(int speed){
16  if(speed > 0) turnMotor->run(FORWARD);
17  else {
18    speed *= -1;
19    turnMotor->run(BACKWARD);
20  }
21  turnMotor->setSpeed(speed);
22}

An important implementation challenge was that there is no way to specify distances that the motor will be doing useful work over but instead you must pass in the speed the motor will turn at and the time interval. This means that deepening on the amount of power supplied by the motor, you are responsible for fine tuning these measurements.

Ours ran three DC Motors which were well supported by our Motor Shield. This setup guide really helped us set up and run this configuration.

Now that the Arduino + Motor Shield were all hooked up, and the motors were buzzing we gained the ability to control the car through its new small Arduino brain. However this early in the project it could only follow pre-programmed instructions in a loop and it was time to test what we’ve built.

Manual Control

The next step was to hook up a way to control the car manually, which turned out to be really useful in the training phase. A really handy module available in the Arduino arsenal is the Bluetooth Low Energy chip (BLE). The setup is quick and there’s available example source code for mobile devices to BLE connection so it only took us some minor tweaking to get the board connected to one of our phones.

We used an Android phone to interface with the chip, the the help of this BLE Android project. That sample code is an example of Bluetooth text messaging which is very similar to what we’ll be doing to communicate with the car, in fact our implementation will be even simpler.

All it takes is to scan for local Bluetooth devices and connect to the chip, after which we can start sending and receiving messages in a few lines of code! This gave us the freedom to go wild with the input type for the car controller, with my preference being a simple joystick. By sending messages such as S10 (Steer Left at speed 10) or F -100 (Go backwards at speed 100), with S representing the steering motor and F representing the front and back motors, we could encode directions for the car in just a few bytes.

The latency between the controller and the car proved to be nonexistent and even worked for up to 10m away with no issues. At this point the car was controllable in the same way it was before, if not better. The next step was to actually make car drive on its own and after a few failed attempts at coming up with logic for the car, we turned to neural networks.

Machine Learning Needs Data

Since with neural networks we really can’t get anywhere without gathering a large data set, it’s time to roll out the car and gather sweet data. Like any explorer venturing out into the unknown, our car will want to record all of its experiences for future self reflection. This means that we’ll need one mobile device to drive the car (from the implementation above), and another device with a camera strapped to the front of the car.

Because our budget was small we strapped a second phone to the car to record video as the car drove around. The input data we could collect was what the camera on the car can see and the logs left by the controller app as a human pilots the car with the other phone. As it turns out, this setup we had now, was the bare minimum required to train a neural network, featuring the input set and a label set.

The input set was the data captured by the camera on the car, and the label set was the input of the human driver piloting the remote car. Now how does that second set work as a label set exactly?

Imitation Learning

Do you remember the very first steps you took as a child? Chances are you did not. Even before conscious thought develops in the brain, its capacity to imitate the actions of other people are baked into the wiring of our brains and it allows for learning at an astonishing rate. Our car can’t “walk” yet but just like in the case of the baby, it’s our job to give it an example to follow.

By having a method of controlling the car and logging the input from the human, our car can learn what input it should supply to itself over time and become as good as the pilot.

Gathering the Data

We created a diagram of extreme values for the joystick. The first coordinate value being the degree of steering (x axis) and the second value being the amount of motion (y axis). Having a line of (0,0) cut through the middle of the joystick allowed for a gradient of speed between moving full force forward and moving full force backwards, reducing the wear and tear on our motors, and with maximum steering at 45 degree angles.

Below is the trigonometry that constrains the input joystick to a circle

 1float x = event.getX()- (circle.getWidth() >> 1);
 2float y = event.getY()- (circle.getHeight() >> 1);
 3
 4double distance = Math.sqrt(Math.pow(x,2)+ (float) Math.pow(y,2));
 5double angle  = Math.atan2(x,y);
 6
 7if (distance > circle.getWidth()/2){ //if outside the joystick
 8
 9    cursorX = (float) ((circle.getWidth()/2)*Math.sin(angle));
10    cursorY = (float) ((circle.getHeight()/2)*Math.cos(angle));
11
12    cursorX+=circle.getX() + (circle.getWidth() >> 1) - (control.getWidth() >> 1);
13    cursorY+=circle.getY() + (circle.getHeight() >> 1) - (control.getHeight() >> 1);
14}
15else{ //if inside the joystick
16    cursorX = x+circle.getX() + (circle.getWidth() >> 1) - (control.getWidth() >> 1);
17    cursorY = y+circle.getY() + (circle.getHeight() >> 1) - (control.getHeight() >> 1);
18}
19control.setX(cursorX);
20control.setY(cursorY);
21
22final int vector_X = (int)(circle.getX() + circle.getWidth()/2 - control.getWidth()/2 - cursorX) * 255/(circle.getWidth()/2);
23final int vector_Y = (int)(circle.getY() + circle.getHeight()/2 - control.getHeight()/2 - cursorY) * 255/(circle.getHeight()/2);
24
25int final_distance = (int) (dist*distance);
26if(vector_X < 0){
27      final_distance *= -1;
28}
29
30distance = (int) Math.sqrt(Math.pow(vector_Y, 2) + Math.pow(vector_X, 2));
31
32//get scaled values between -255 and 255 for X and between -150 and 150 for Y
33
34float dist = 0;
35if(Math.abs(vector_X) >= Math.abs(vector_Y)){
36    if(vector_X != 0){
37        dist = (float) Math.abs(vector_Y)/Math.abs(vector_X);
38    }
39}
40else{
41    if(vector_Y != 0) {
42        dist = (float) Math.abs(vector_X)/Math.abs(vector_Y);
43    }
44}
45
46int final_distance = (int) (dist*distance);
47
48final int final_X = final_distance;
49final int final_Y = vector_Y * 150/255;

The result was an input range of -150 and 150 for Y and between -255 and 255 for X, the values for Y were smaller because we didn’t want the car to move too fast but a full range of steering was crucial. Below are the results of a training session.

Each dot on the above graph represent the location of the finger on the trackpad at the time that a training image was sliced from the video. The top part represents forward motion, the bottom backwards motion and when the car steered it produced dots that form the diagonals of this hourglass shaped graph. We cut the data into a few logical sectors,

1. Forward

2. 50% Forward

3. Forward Left

4. Forward Right

5. Backward

6. 50% Backward

7. Backward Left

8. Backward Right

And “Stop” which proved to be pretty disastrous because at some point the car would learn to permanently stop.

Collecting the image data was somewhat easier, all we had to do was record a video while the car was being driven. Here’s a link to a sample android project that records a video and saves it to storage when you’re done, we used the same code to capture our videos and even your ordinary camera app works if you don’t want to implement anything custom. The only meta data that’s required is the length of the video and when the recording was initiated.

Combining Inputs and Labels

In order to match the correct actions with what the car is actually seeing we have to bring the two data sets together. By splitting the training video captured by the front facing camera and time stamping it into separate images we can match them to the timestamp of a log from the controller’s input.

The video was split into 4 images per second of video, although with a 30fps recording we could have gone for a lot more detail. On the other hand our mobile device could capture hundreds of actions on the touchscreen per second, requiring us to disregard most of the data for picking one log per image from the video. Both data sets where submitted to a python script to slice the videos and create matching (image, log) pairs.

After driving the car around our building for a bit, we gathered close to 7000 pairs of images and associated human control label pairs. Due our small budget and running the neural network on a phone, we had to down sample our images to 50x50 pixels so that our setup had a fighting chance of working in real time.

Using Neural Networks

This was arguably the most exciting and frustrating part of the project, the machine learning. Because we wanted the car to drive on its own we wanted the on board phone to accomplish both the seeing and the steering for the car. Both were indeed possible. We fully utilized the OpenCV Android library to obtain an image for every time the camera screen was updated as well as running a separate thread to control the Bluetooth chip on the car. We used this android project as an introduction to OpenCV and it got us to a point were we could obtain individual frames of the camera in real time. All that was left to do was to complete a forward pass on our network model with images supplied by the library’s onCameraFrame() function every time there was a new frame.

 1public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
 2  mGray = inputFrame.gray();
 3  Mat mEqualized = new Mat(mGray.rows(), mGray.cols(), mGray.type());
 4  
 5  //normalize the image
 6  Imgproc.equalizeHist(mGray, mEqualized);
 7  
 8  //convert the frame into a scaled bitmap
 9  Bitmap bitmap = Bitmap.createBitmap(mEqualized.cols(), mEqualized.rows(), Bitmap.Config.RGB_565);
10  Utils.matToBitmap(mEqualized, bitmap);
11  final Bitmap scaled_bitmap = Bitmap.createScaledBitmap(bitmap, 50, 50, false);
12
13  //get output from the neural net model
14  int[] shape = model.get_Interpreter().getInputTensor(0).shape();
15  final ArrayList<String> output = inferencer.multiImageInference(model, scaled_bitmap, shape[shape.length-1]);
16}

At this point we could either write our own native neural network , or we could use Google’s Tensorflow Lite library to load a python Keras model directly onto the phone. We went with Keras for the simplicity and existing collection of examples we could reference.

The only downside to this approach was that because the model file could no longer be changed on the Android phone, our car would not be able to use reinforcement learning to learn as it drove on its own. Fortunately this was a sacrifice we were willing to make.

With a basic Keras Sequential CNN model we pumped an input image of 2500 pixels into the series of Convolution and Max Pooling layers. We then converged the model into a dense layer at the end to give us 9 categories of output. For each output we would get a confidence value of how much the car was willing to follow any of the given directions. This was done because it’s not important for the car to make fine movements, the categories we specified above were enough.

Neural Network Architecture

 1model = Sequential()
 2model.add(layers.Conv2D(32, kernel_size=(3, 3),
 3                 activation='relu',
 4                 input_shape=(50,50,10), 
 5                 name='my_layer'))
 6convout1 = Activation('relu')
 7model.add(convout1)
 8model.add(layers.MaxPooling2D(pool_size=(2, 2)))
 9model.add(layers.Conv2D(64, (3, 3), activation='relu'))
10convout2 = Activation('relu')
11model.add(convout2)
12model.add(layers.MaxPooling2D(pool_size=(2, 2)))
13model.add(layers.Dropout(0.25))
14model.add(layers.Flatten())
15model.add(layers.Dense(128, activation='relu'))
16model.add(layers.Dropout(0.5))
17model.add(layers.Dense(numBins, activation='softmax'))

Above is a Keras model we used for our car. It is relatively small and the gist of it is that it converts a 50x50 array of an input image into an array of 9 confidence outputs for each of the joystick colored sectors.

After a few weeks of tweaking we settled on this model that worked fairly well, giving us a 95% success rate with a sizable test set of 30% of our entire training database. The laststep was to save the Keras library as a tflite file and feed it to the Interpreter on the phone.

Tensorflow Lite has the amazing functionality of exporting the python written model into a tflite file which can be used by a Tensorflow Lite interpreter on any other system.

 1model_json = model.to_json()
 2with open("model.json", "w") as json_file:
 3    json_file.write(model_json)
 4
 5keras_file = "model.h5"
 6model.save(keras_file)
 7
 8converter = lite.TFLiteConverter.from_keras_model_file(keras_file)
 9tflite_model = converter.convert()
10open("model.tflite", "wb").write(tflite_model)

This model is simply loaded as a TFLite interpreter object which can then use the simple run() function on an input image to create a prediction.

 1public Interpreter load_internal_model(String model_name) throws IOException {
 2        tfliteModel = loadModelFile(model_name);
 3        tflite_interpreter = new Interpreter(tfliteModel);
 4        return tflite_interpreter;
 5}
 6
 7private ByteBuffer loadModelFile(String model_name) throws IOException {
 8        AssetFileDescriptor fileDescriptor = activity.getAssets().openFd(model_name);
 9        FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
10        FileChannel fileChannel = inputStream.getChannel();
11        long startOffset = fileDescriptor.getStartOffset();
12        long declaredLength = fileDescriptor.getDeclaredLength();
13        return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
14}

Now taking this model we can saved it into the memory of our phone and run it with the TensorFlow Interpreter to simply do a forward pass on the model.

 1float[][][][] float_pixels = new float[1][50][50][num_saved];
 2float[][] outputVal = new float[1][9];
 3
 4for(int i = 0; i < 50; i++){
 5    for(int j = 0; j < 50; j++){
 6        for(int b = 0; b < num_saved; b++){
 7            float_pixels[0][i][j][b] = (float) int_bitmaps.get(b)[i*50+j]/255;
 8        }
 9    }
10}
11
12try{
13    tflite_interpreter.run(float_pixels,outputVal);
14}catch (Exception e){
15    e.printStackTrace();
16}
17
18//outputVal is now contains confidence values for all the sectors

The sector with the highest confidence was taken as the action to execute by the car and finally all the pieces were in place.

Conclusion

At this point we had the car built, data collection set up and a neural network dictating the controls. The car was driving with the expertise of a 6 year old with a controller. It can safely be said that we were not winning any Grand Prix any time soon but the car was able to drive around on its own without crashing itself, with surprising reliability.

In any case we were happy that something that we’ve built over a semester while taking our other classes could actually learn on its own and accomplish such a difficult task, with minimal input from us. As a first project in machine learning this was a tough one to start with but looking back at it, the approach we took was in line with any other standard machine learning task.

If you’re still reading, hopefully you’ve learned something new and if not at least enjoyed our amateur struggles. Again, all the code is available here https://github.com/RoboticsCourse, please consider starring it and checking out the contributors.