/artificial-intelligence/ai-coding-tools

Read __How enterprise engineering teams can successfully adopt AI__ to explore how GitHub is at the forefront of AI innovation—and how it can help your organization lay the groundwork for successful AI adoption.

Get the ebook to:

- Learn how generative AI has gone from being niche to mainstream and get an overview of common AI-powered developer tools, including GitHub Copilot.
- Understand how AI is reshaping enterprise software development and assess the opportunities that your organization has to benefit from AI.
- Get a roadmap with step-by-step instructions on how to operationalize AI solutions for your engineering teams.


Learn how to bring AI to your engineering teams and maximize the value that you get from it.

Resource Page for /artificial-intelligence/how-enterprise-engineering-teams-can-successfully-adopt-ai

How enterprise engineering teams can successfully adopt AI

## The GitHub Copilot Impact Glossary
This glossary lists a number of measures that an organization can use to help evaluate the impact of Copilot for their organization. Learn:
- Definitions and caveats related to measures
- When (which stage) to use a measure
- How to access a measure (source data location)

For more information on GitHub’s approach to measuring Copilot Impact, check out our [GitHub Copilot Learning Pathway](https://resources.github.com/learn/pathways/copilot/essentials/measuring-the-impact-of-github-copilot/).

Remember 😀: The relevance and usefulness of system-level speed and quality measures vary across organizations. Any Copilot impact beyond coding is indirect, and the downstream possibilities are endless. Thus, we must be careful when searching for causality in speed and quality measures. 

The metrics provided below are representative of processes that often improve with Copilot, but the degree of impact will vary. Based on the strengths and weaknesses of your organization, the impact may be more or less. It's less important to achieve a benchmark than to observe neutral-to-positive improvement. Engineering leaders should only be concerned if a metric regresses. In that case, pause to determine the root cause before proceeding.

### Tips: 
- GitHub recommends looking for improvements in each measure over time rather than setting fixed targets.
- We suggest aligning measurement frequency with your development cadence (e.g. sprint cycles).
- You may choose to collate your data points into existing BI or reporting tools for ease of analysis and review over time for trends. Alternatively, use a spreadsheet to track measures over time.
- As you add developers to Copilot you may see a decrease in some measures as those developers build capability and confidence in using Copilot as part of their daily flow. Consider analyzing measurements according to onboarding groups (set of developers provided with a Copilot seat).
- You may choose to monitor some DevOps metrics over the evaluation and adoption stages for early indicators of likely system- impact areas. But particularly in the evaluation and adoptions stages, we recommend focusing on happiness and engagement measures. Once developers are using Copilot effectively, system level measures are more likely to improve. 

### Category legend:
Developer Happiness - 😃 
Developer Engagement - 👩‍💻 
Speed - 🏎️ 
Quality - ✨

<table>
 <tr>
 <th>Measure</th>
 <th>Definition</th>
 <th>Stage</th>
 <th>Source data location</th>
 </tr>
 <tr>
 <td>Satisfaction using Copilot - 😃</td>
 <td>Number of Copilot users who value Copilot in their workflow.</td>
 <td>All</td>
 <td>Copilot Impact Survey: • How would you feel if you could no longer use Copilot?</td>
 </tr>
 <tr>
 <td>Benefits of using Copilot - 👩‍💻</td>
 <td>Range of areas in which users leverage Copilot and the range/extent of benefits experienced.</td>
 <td>All</td>
 <td>Copilot Impact Survey: • When using Copilot, I… • When using Copilot, my team… • In using Copilot in the last week… • I use GitHub Copilot to…</td>
 </tr>
 <tr>
 <td>Challenges using Copilot - 👩‍💻</td>
 <td>Number of users impacted by potential adoption barriers.</td>
 <td>All</td>
 <td>Copilot Impact Survey: • What challenges have you encountered during your Copilot trial?</td>
 </tr>
 <tr>
 <td>Enablement provided - 👩‍💻</td>
 <td>Number of users with a seat who have received enablement resources/training sessions.</td>
 <td>All</td>
 <td>Copilot Impact Survey: • Which of the following GitHub Copilot enablement materials have you received?</td>
 </tr>
 <tr>
 <td>Average Daily Active Users per Month - Completions - 👩‍💻</td>
 <td>Users who have received 1+ suggestions on a given day.</td>
 <td>All</td>
 <td>Copilot Metrics API: • Take average of Active Users - Completions from API, over 28 days</td>
 </tr>
 <tr>
 <td>Average Daily Active Users - Chat - 👩‍💻</td>
 <td>Users who have 1+ chats on a given day.</td>
 <td>All</td>
 <td>Copilot Metrics API: • Take average of Active Users - Chats from API, over 28 days</td>
 </tr>
 <tr>
 <td>Average Daily Active Users - Total - 👩‍💻</td>
 <td>Unique users who have received and accepted 1+ suggestions OR participated in 1+ chats on a given day.</td>
 <td>All</td>
 <td>Copilot Metrics API: • Take average of Active Users - Total from API, over 28 days</td>
 </tr>
 <tr>
 <td>Suggestions Delivered - Completions - 👩‍💻</td>
 <td>Number of suggestions per day displayed in the code editor for Copilot completions in IDE.</td>
 <td>All</td>
 <td>Copilot Metrics API: • Number of Suggestions</td>
 </tr>
 <tr>
 <td>Number of Acceptances - Completions - 👩‍💻</td>
 <td>Number of suggestions accepted (total of both partial and full) per day.</td>
 <td>All</td>
 <td>Copilot Metrics API: • Number of Chat/code acceptances</td>
 </tr>
 <tr>
 <td>Lines of Code Accepted - Completions - 👩‍💻</td>
 <td>Lines of code accepted (total of both partial and full) accepted per day</td>
 <td>All</td>
 <td>Copilot Metrics API: • Lines of Code Accepted - Completions</td>
 </tr>
 <tr>
 <td>Total Acceptance rate - Completions - 👩‍💻</td>
 <td>Suggestions Accepted (Combined: Full and Partial) / Total Number of Suggestions</td>
 <td>All</td>
 <td>Manual calculation (based on Copilot Metrics API): • Number of Suggestions & • Number of Acceptances</td>
 </tr>
 <tr>
 <td>New seats added in billing cycle - 👩‍💻</td>
 <td>Additional seats purchased for organizational members during this billing cycle</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>Billing and plans (Copilot) (organizational settings) • New seats added during this billing cycle</td>
 </tr>
 <tr>
 <td>Dormant users - 👩‍💻</td>
 <td>Number of users who last used Copilot more than ‘n’ days ago</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>Copilot Access Management (organizational settings): • Sort by last used</td>
 </tr>
 <tr>
 <td>Total completed pull requests - 🏎️</td>
 <td>Total number of pull requests completed in a given week</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>GitHub PRs list (in a repo) can be filtered on closed PRs for a time period</td>
 </tr>
 <tr>
 <td>Pull requests per developer - 🏎️</td>
 <td>Number of pull requests / code contributors</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>Manual calculation: • GitHub PRs list (in a repo) can be filtered on closed PRs for a time period & • Your organization's data on developers contributing to repo</td>
 </tr>
 <tr>
 <td>Time to merge - 🏎️</td>
 <td>Average time from PR opened to merge</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>GitHub: calculation requires consideration of your organization's GitHub set-up</td>
 </tr>
 <tr>
 <td>PR lead time - 🏎️</td>
 <td>Average time from PR opened to deploy</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>GitHub Actions: calculation requires consideration of your organization's GitHub set-up</td>
 </tr>
 <tr>
 <td>Code Review Time - 🏎️</td>
 <td>Average code review time per pull request (time between PR opened to PR closed)</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>GitHub: calculation requires consideration of your organization's GitHub set-up</td>
 </tr>
 <tr>
 <td>Pull request merge rate - ✨</td>
 <td>Total number of successful builds completed in a given week.</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>GitHub: GitHub PRs list (in a repo): The number of PRs merged / The number of PRs submitted</td>
 </tr>
 <tr>
 <td>Total successful builds - ✨</td>
 <td>The percentage of pull requests successfully merged</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>GitHub Actions: calculation requires consideration of your organization's GitHub set-up</td>
 </tr>
 <tr>
 <td>Change failure rate - ✨</td>
 <td>Percentage of code deployments that caused a failure in production</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>Calculation requires consideration of your organization's GitHub set-up and integration with third-party apps</td>
 </tr>
 <tr>
 <td>Open security vulnerabilities - ✨</td>
 <td>Number of unresolved vulnerabilities as identified by GHAS</td>
 <td>Adopt, Optimize, Sustain</td>
 <td><a href="https://docs.github.com/en/enterprise-cloud@latest/code-security/security-overview/viewing-security-insights-for-your-organization#autofix-suggestions">Organizational-level security insights</a></td>
 </tr>
 <tr>
 <td>CI success rate - ✨</td>
 <td>Number of successful CI runs inclusive of test automation / Number of attempted CI runs</td>
 <td>Adopt, Optimize, Sustain</td>
 <td>Calculation requires consideration of your organization's GitHub set-up and integration with third-party apps</td>
 </tr>
</table>

We recommend to first focus on your developers’ adoption and engagement with Copilot. Then, harness developers’ improved productivity to pursue the engineering system improvements that matter most to your organization.

/artificial-intelligence/

How enterprise engineering teams can successfully adopt AI

Glossary: Measuring the impact of GitHub Copilot