Hello Copilot

Github Copilot is a popular discussion topic across the development industry, no exception here at Rocket. From the funny discussions like this one contemplating the effect of AI on hiring practices.

To more serious questions around programming in general, and how things may be positively or negatively impacted by Copilot.

When I got my invite to the technical preview last week, I was excited to see what the AI Pair Programmer could actually do. Below is what I found while trying to work with Copilot, and more anthropomorphism than I intended.

TL;DR

Copilot has some great strengths.

• It is a very capable snippet tool. With access to more snippets than most other extensions on their own.
• It will make solid suggestions to accomplish a task within your code, using your variables, and it can even tell if you’re going with vanilla JavaScript style functions, or modern ES6 style fat arrows functions for example.
• It will pull in libraries, or even setup access to APIs, you may not know of to do something. In the examples below Copilot demonstrates both of those.

There are also some weaknesses with the system, which is to be expected.

• Suggestions may include the variables in your code, but the suggestion may not always use them properly.
• There are some odd places where Copilot gets “stuck” in a sense. A handful of times I had the same suggestion repeated line after line with no end really.
• A new file can give Copilot a little difficulty on suggestions as well. For example, in a new file with nothing but a comment in it, the suggestions given by Copilot will also be comments.

Overall my experience with the Copilot extension was more positive than negative. It’s a great assistant to have on hand. You get some suggestions and/or snippets. That helps you get coding quicker, and in some cases it can help get you unstuck on a problem potentially. I think that’s all we should be asking for from an AI Pair Programmer. Anything beyond that begins to stretch into some uncomfortable topics around replacing developers.

Let’s Give Copilot a Test

After getting the extension installed and authorized in VSCode it seemed like a good idea to just attempt the Python examples from the Copilot documentation. Everything here is in JavaScript, however, because I am (after all) a JavaScript dev.

One of the first examples you see is entering a comment along the lines of // use a web service to determine sentiment of text , so I went with that. Copilot presented me with a couple of comments for the language of the code, and the file path, then the function declaration. Accepting the declaration, Copilot now suggests the internals to the method to accomplish the task.

A few things to note here.

• I am not entirely sure if the Language and Path comments are because of what Copilot is seeing in my workspace, or if it’s because some other repo has those comments. Looking at the suggestions in the Copilot panel suggests the latter.
• In this instance it gave me a suggestion using fetch. In other tests it used $.ajax wrapped in a promise as well. This can be controlled by specifying in the comment such as “using fetch”, or simply having fetch imported in the file already. The selling point of being context aware seems to have hit the mark here for sure.
• In this instance Copilot is using an ES6 fat arrow function. This is because I cheated and already have an ES6 function above this. I have noticed when simply in a lone JavaScript file, with no prior code to draw context from, it would default to vanilla JavaScript.
• Similarly, in a file with no code other than the comment, the code Copilot suggests will also be commented out. This last bit is kind of annoying to be honest, and I have not seen a way out of it, short of having actual code in the file already like I did here.
• The API being suggested here is not one I am familiar with. This is clearly from Copilot’s sample code.

With that example out of the way I tried some other quick things. Starting the declaration for a function to get the current time in milliseconds, const getCurrentTimeInMilliseconds immediately got me a few variations using Date.now(). In that use case Copilot is not much more than a supercharged snippet tool. It’s nice, but it doesn’t really show what Copilot is capable of. For that I decided to try writing a quick app.

Now Witness the Power of This Mostly Armed and Functional AI Pair Programmer

I needed a simple idea to try out a couple of aspects of Copilot. Being a developer, obviously huge geek, and a fan of Big Bang Theory, I decided to make a simple command line version of  "Rock, Paper, Scissors, Lizard, Spock" in Node. Pretty simple game, the algorithm to check the winner can be a fun challenge to try and optimize or over-engineer if you’re looking for something to do. I also can’t remember ever writing anything in Node where I needed user input, so I had no immediate idea how to do that. I’d be relying on Copilot here as my pair programmer pretty much.

I started off with making an array to store the 5 choices for the game. This was interesting in that it was the first time I noticed Copilot was updating it’s suggestions based on what was in my files.

You can see the suggestions are clearly coming from a repo similar to what I am writing here. The suggestion has a description that looks like something from a roleplaying game. After I put in the “winsAgainst” property though, Copilot understands better, and the next suggestion is more in line with what I am looking for. It even includes a typo from the original repo.

Another thing that you don’t see in this clip, but I have noticed is the code formatting Copilot uses. In general arrays and objects end up with trailing commas when using a suggestion. I am not a fan of trailing commas. I started taking them out, and the Copilot suggestions stopped using them. I do not know if that is because Copilot understood the styling change, or just because the context now matched a different repo.

Next up are some quick methods to get a random number, a random choice, and to determine which choice wins. The suggestion for picking a random choice consistently has the choices array being passed into the function here. Sometimes it has the parens for the param, and sometimes it does not have the parens. Again, I assume this is because multiple repos have similar code, with different formatting.

Determining the winner is the key to this game we’re writing here. If the algorithm works, the game works, and I can retire having written this masterpiece. However, Copilot here just does not have a perfect sample to choose from. It also does not have enough context from the app itself to complete this algorithm for me. It makes a good attempt as seen in the clip below, but without more data, the suggestion is pretty far off of the goal. In the end I just wrote a quick and dirty solution myself.

For the last part of this game I just needed to get the user input, let them pick an option, have the computer pick an option, and output the results. This got a little scary in what Copilot suggested for code. The suggestion I got here was more complete than I had expected. Once I started using chalk for output (who doesn’t like a little color?) Copilot knew where I was going, and gave me the exact lines I was looking for.

Now I just need to get the user’s choice from the app. Like I said before, I’ve never done that before. I had no clue how to get user input in Node. Sure, I could hit Stack Overflow or Google. This is one of the things Copilot is supposed to be capable of though, so I let it have a shot at the code.

Again Copilot got really close to the solution, but it missed a couple important things.

• We’re passing the choice directly to the determineWinner method. That’s great and all, but the choice here is a string of user input. We need the actual choice object from the choices array.
• The output message is also using the user input string. Not terrible, but it should use the choice name of course.
• Lastly the message conditional is obviously crap. The determineWinner function always returns a string. This conditional will never be false, and never hit the second case. The player’s choice would always win. This was a bit of a failure. Bonus points for tossing out a ternary though!

What is weird here is that writing this article, and following along in a new project in VSCode, the suggestions I have been getting were different than the first time I wrote this game. Compare that chunk for the prompt (on the left) to the one I got from Copilot initially (on the right).

You’ll notice the same issue with using choice instead of getting the object from the array, and not using the name of the choice for output, but the overall functionality is much better. The code on the right is all Copilot, even the better logging. I’m not really sure why the suggestions change like that. I’ll just blindly trust that whatever learning is going on behind the scenes there was a line of code, filename, or solar flare that caused the change.

I added a couple of extra things to the game just to make it somewhat usable. Prompt can validate the user input, so I added a simple schema for that. Again Copilot managed to output most of that code itself as well. I started defining the schema according to the docs for prompt. Copilot filled in the blanks and came up with this schema. It even understood that there were 5 options to choose from. I assume that similar to the lines to log the 5 choices, the suggestion is inferred from the choices array being imported.

One thing I will not go into too much detail here so nobody falls asleep at the keyboard is that I also tried out unit tests with Copilot. Using Jest, Copilot put out some basic unit tests without any issue. They were very basic though, and in my opinion not the way I would write the test. A good example is testing that an object has the correct properties. The example below illustrates this well.

The assertion on the right is much better here. It will validate the properties and their types. The Copilot suggestion on the left only checks that the properties exist here.

One thing that I did like however was that once I put together a describe block of tests, with variables, and a beforeEach to assign values, Copilot was able to suggest mostly complete describe blocks for different cases. A solid unit test in this game is one to verify that the winner, loser, and draw outputs for that choice. I built a block for where rock is the choice. When I then added a new block to test those 3 cases if Spock were the choice. The Copilot there was an almost perfect copy/paste job. The issue was that Copilot got the indices wrong on the arrays for variable assignment. I’m not sure if the system should know that Spock is the 4th choice, not the 3rd as it put in the code, but that’s not a big deal in my opinion. It also keeps us on our toes for sure.

Randomness

I did try CSS in a Vue app with Copilot. I didn’t know what to expect. While it gave me good class names following the BEM convention, the styling itself was a straight copy of another class in the workspace. I don’t see Copilot being capable of knowing what you want for styling, and how to accomplish that anytime soon, but I may be wrong.

Another thing I tried was the same game in TypeScript. Now I am still new to TypeScript, so I don’t know it as well. What I can say is that by defining a choice type the suggestions I got were more complete there. The determineWinner method for example was almost perfect. The suggestion had the choice.name in the output for me already. Also by using the more fleshed out JSDoc style comments the methods suggested had types and returns on them automatically which was nice.

One thing I would love to see in Copilot is code attribution. Even when looking at the Copilot panel for all current suggestions, there is nothing explaining where it found a piece of code. Maybe someone writes a kick-ass algorithm for determining the winner in Rock, Paper, Scissors. If it gets pulled into another project, maybe we should at least have the ability to drop a comment with the author and a link? Or maybe you’d like to see the unit tests behind that algorithm. You can’t do that without knowing where it came from. Maybe there is a privacy issue in there somewhere, but Copilot is supposed to be using public repos afterall.

I wonder if we need a pool for the first person to blame a major production bug on Copilot suggestions. It will happen one day, and it’ll be funny. At least I hope so.

Final Thoughts

At the end of the day I did come up with a functional game. I can’t say for sure, however, how much of the code I wrote vs how much Copilot “wrote”. I’m not sure if that is a good sign or bad sign. From my experience the suggestions I got were mostly helpful which is good. Copilot is definitely not a perfect system. I don’t think that is the intent though. It does a solid job at being both a powerful snippet system, and suggesting some bigger chunks of code to get you moving again if you’re stuck perhaps.

It will be interesting to see how this system ultimately gets used, and its effects on the industry. The initial reaction of “Copilot will replace developers” however is definitely not the case. With this iteration at least.

There is a repo with the final code from this experiment, if you want to look, or play the game. Maybe find some bugs. You can find it here, https://github.com/CoreyT355/rpsls-with-copilot

You can read more about Github Copilot, and sign up for the technical preview here, https://copilot.github.com/