When Stack Overflow first arrived 15 years ago, it was a huge advance for a lot of programmers. Suddenly, programmers could share knowledge about quirks, bugs, and best practices complete with code ready to copy and paste. I know plenty of engineers who visit multiple times per day. This targeted resource changed how programmers work.
There’s another change happening right now. Unless you’ve been in a coma for the past year, you know that generative AI has taken the programming world by storm. Multiple chat interfaces and plugins are generating code that is going into production systems. GitHub Copilot, ChatGPT, and more have given developers the power of nearly instant code.
But anyone thinking that these tools can make them a super software developer overnight is in for a rude awakening. Being an effective coder with a code generation (codegen) tool still requires you to be an effective coder without one. Codegen tools will undoubtedly bring new people into software development, but they’ll still need to do the work of understanding the code that their chat interfaces produce.
In this article, we’ll talk about the real gains that programmers are seeing today, while also exploring the ways that not even codegen can help an unproductive team.
How codegen can improve productivity
The Codegen tool is trained on your existing code. So much existing code. They know how the code was written and how that code has solved problems in the past. So if you’re writing something that solves common problems like type definitions, connecting to databases or SDKs, or API definitions, codegen makes it easy to solve them.
Some languages have a concept called scaffolding. Create templates that use compilers or other language features to write the actual code. The Codegen tool is great for scaffolding. I asked our internal AI tool for a template to connect to a Postgres database.
python
import psycopg2
from psycopg2 import OperationalError
def create_conn():
conn = None
try:
conn = psycopg2.connect(
database="your_database",
user="your_username",
password="your_password",
host="localhost",
port="5432",
)
print("Connection to PostgreSQL DB successful")
except OperationalError as e:
print(f"The error 'e' occurred")
return conn
connection = create_conn()Voila, we now have a function that can be used anywhere to connect to any Postgres database. I wanted to make it a little more reusable, so I had the AI rewrite the function to take the database name, username, and password as parameters. Since you did not specify a programming language, Python code was provided. Suppose you want JavaScript instead. I rewrote it to tell it to use pg instead.
Typically, you search the web for a connectivity library for your language of choice, then copy and paste the sample code to modify. AI works as a more sophisticated copy and paste in that it tailors the code you paste to your specific problem. There’s no need to tinker with variables or rewrite them manually. Bugs often occur if you forget the closing parenthesis or semicolon.
Two Figma employees talk about how codegen solves the blank page problem in a blog post. You won’t get the finished product, but you can get it closer to 0 to 0.5 instead of 0 to 1. If you have a new API that passes large amounts of JSON blobs to your TypeScript code, you can use AI to set type definitions in seconds instead of entering them manually.
If you know what you need, codegen can be a shortcut. In the blog linked above, Figma developers say, “codegen is most effective when applied in specific ways for specific teams, companies, and workflows.” Codegen works with documentation and sample code to help you build code that applies to specific situations.
Of course, coding was never difficult. That’s a requirement. Understanding specifically what you want from your software so that codegen can write the code for you is a skill in itself. Looking at my example above, I didn’t understand my requirements clearly. I didn’t even know what language to use. In conversations with some of my engineering friends, I was told that thinking through a problem thoroughly to the point where you can write an effective codegen prompt gives you a deeper understanding of your requirements and desired design. There are also local scripts that can help you create more effective prompts before you spend money on an LLM cycle.
Today’s engineering organizations want to create reliable software more than ever. Watch for efforts to create increasingly automated unit tests, implement test-driven coverage, and move test coverage closer to 100%. Many companies are going one step further from automated testing and deploying AI to suggest tests as you write code. You can also write tests with Codegen, but you need to clearly understand what the success and failure conditions are.
These AIs do more than just write code. They can understand it too. If you want to understand your codebase, you can pass your code to an AI chatbot and have it tell you how it works. If you don’t have reliable documentation for your internal code, codegen is a useful shortcut. Once you have an answer, add it to your internal knowledge base so others can learn from your research.
But AI responses aren’t completely reliable
It’s great to be able to use AI to complete coding projects faster. However, it does not solve any specific problem and introduces troublesome new problems. Hallucinations are a well-known side effect of LLM and its chatbot interface. It is debatable whether LLMs have knowledge or are just probabilistic parrots. They can provide very authoritative pieces of code and answers to coding questions, but they can also make something out of the whole thing. One programmer discovered that ChatGPT suggested using functionality from a library that didn’t exist.
As a result, developers have had a hard time trusting AI chatbots. In a 2023 developer survey, only 3% of those surveyed had high confidence in the output of AI tools, and a quarter of respondents had no confidence in it. Some might argue that a healthy distrust of the code you encounter (even your own) is healthy, but in reality, many people would take AI at its word. . This is a recipe for security vulnerabilities. It occurred in one of the most popular code snippets on Stack Overflow, but it will also occur in generated code.
Researchers at Purdue University found that ChatGPT falls short when compared to human-authored answers on Stack Overflow. They asked a set of 517 questions from Stack Overflow and found that ChatGPT answered less than half of them correctly. The study found that 77% of answers were too redundant in addition to being often wrong. However, users preferred these answers about 40% of the time due to their comprehensiveness and clear writing style.
This is dangerous for people who don’t double check their answers. Simple errors are syntax errors, not mistakes made by codegen. Because they don’t understand the context of the question, they make conceptual mistakes, focus on the wrong part of the question, or use the wrong or non-existent API, library, or function. There is a possibility that
In the end, participants preferred Stack Overflow questions the most for their brevity and accuracy. Most LLMs use Stack Overflow as part of their training data, but research shows that the original is generally more accurate than her AI output.
If you’re using AI-generated code, you’ll need to put more effort into documentation. An organization’s long-term success can be hampered by technical debt, and poorly written and understood code bases are a form of technical debt. A junior engineer who turns in a project 5 days early but can’t explain how everything works in documentation is actually setting a time bomb for future productivity losses. .
Functionality added to the codebase using AI systems also carries legal and regulatory risks. Currently, a lawsuit is being held to determine whether AI systems trained on open source code are allowed to make copies of that code or create similar ones without proper attribution and licensing. It’s going through the judicial system.
Imagine a scenario where your organization has the happy outcome of an acquisition or going public. Investors and acquirers who perform due diligence will want assurance that the intellectual property they are purchasing is fully owned, properly copyrighted, and licensed by your company. Investor concerns are much harder to satisfy when the majority of the codebase is written not by engineers, but by AI systems trained on vast tracts of publicly available code.
The long-term health of your tech stack isn’t just determined by how quickly your engineers can submit PRs. It’s also important that different parts of the organization communicate clearly and collaborate to find big-picture solutions to architectural challenges that lay the foundation for future expansion and growth.
One solution to these challenges is to connect AI systems to datasets that they know and trust as ground truth. For example, Stack Overflow uses Search Augmented Generation (RAG) to ensure that queries sent to OverflowAI only see information from public platforms or Teams instances to which the user belongs. By limiting the surface area that AI uses for reasoning, you reduce the chance of errors and hallucinations. Additionally, AI responses can include citations, allowing developers to understand the origin and license of the code they are writing.
AI and AI-enabled conversational interfaces could ultimately become a game-changing new user interface style. Their ability to find and summarize information is unparalleled. However, having access to the original information is key to productivity, especially if it is not public information. And real productivity gains across an organization don’t just come from faster code. These come from improving the way organizations work together.
Real gains will come from improving org as a whole, not just faster code
There’s a reason why organizations that talk about productivity also talk about development velocity, developer experience, and platform engineering. All of this is a big, comprehensive effort that improves not just the productivity of a single developer, but the entire organization. Since software development teams create software together, their productivity must be measured together. There are a few things that all effective organizations do.
Automate wherever possible
First of all, they are looking to automate manual processes as much as possible. They automate software delivery pipelines with effective CI/CD processes that take committed code from build to deployment with minimal effort. They automate testing as much as possible. They focus on platform engineering to make the tools and libraries that everyone uses better. In fact, code generation productivity can be thought of as part of automating boilerplate code and freeing coders to focus on solving more important problems.
Share knowledge company-wide
Then, share the knowledge with everyone in your company who needs it. This is even more important for tasks that require knowledge, such as writing code. Stack Overflow is a great knowledge sharing resource for general coding questions, but for questions about your own codebase, developers should consult their teammates. In our research, we found that around 50% of respondents agreed that waiting for a response causes a flow interruption. Solutions like Stack Overflow for Teams that store and surface knowledge across silos can help reduce these interruptions.
People first, technology second
Third, effective organizations don’t just write code, they think through the process of building software. We’ve seen a lot of people complain about Scrum and Agile because bad processes are easy to spot. However, because these processes were created to improve the lives of developers, successful organizations ensure that their processes work properly, rather than that they are strictly followed. To do. We also have a good design, planning, and requirements process. We mentioned that requirements are the hard part and that requirements variability can be a defining problem in software engineering. If someone keeps moving the target, you can’t attack the target.
Empower, don’t micromanage
Finally, effective organizations focus on enabling developers to do their work, rather than monitoring or micromanaging them. Gone are the days of negative management, where employees are seen as troublemakers who need guidance and management, and where management is used to lessen the negative impact on employees. Managers try to support their employees, give them the breaks and safety they need, and keep them out of the way of their work.
There’s more to productivity than code
Speeding up code created by AI tools can help improve personal productivity. But the real benefits come from improving the organization as a whole. You need to know what you’re doing when you write code, whether manually or through codegen. Otherwise, you could do more harm than good.
