Better AI Can Reduce the Risks

If poorly designed AI can create risks, better AI offers a powerful way to reduce the risks of democratized statistics. 

Ideally, any AI helping with statistics will create a guided workflow.  First, it will automate data validation. Tools can flag issues such as missing values, duplicates, and outliers before analysis begins. Imagine an assistant that read in some data and reported: 

Your second variable is called height and appears to be in centimeters.  It also uses “9999” to code for a missing value.  One observation is 999 cm (33 feet) tall. It is likely that this value is a typo. Should I code it as missing? 

As the AI proceeds with the analysis, it can verify if the data satisfy statistical assumptions such as normality or linearity.  For example, before running a correlation or t-test, the AI could automatically check for normality and suggest a non-parametric test if necessary. 

It can also check functional forms. For example, it might ask:  

You requested linear regression, which assumes your variables have a linear relationship. But the residuals in your specification follow a predictable pattern (according to the Ramsey RESET test).  This result is often due to a nonlinear relationship.  I suggest using a more flexible specification such as polynomial regression and also an approach that does not make assumptions about functional form such as random forests. Should I run those analyses instead?

The AI system might also proactively alert users to potential methodological pitfalls. For instance, it could display a cautionary message such as, “Your dataset’s sample size is too small to derive reliable conclusions”—a feature already seen in some advanced software. Additionally, the AI could autonomously adjust P values in contexts involving multiple hypothesis tests, flag the risks of overfitting, or adjust standard errors to more accurately reflect the data characteristics.

One might also imagine the AI engaging interactively with the analyst to solicit further context or supplementary documentation. For example, it could request additional materials—such as codebooks, questionnaires, or survey instruments—to deepen its understanding of the dataset. If the analysis suggests that the available data are inadequate, the system might even recommend collecting additional observations to strengthen the study’s empirical foundation.

Perhaps most importantly, the AI assistant can help interpret the results.  To start, it can point out: “Remember, correlation does not imply causation. Consider additional analysis to understand causality.” 

As the AI system evolves, it will increasingly discern both the intent of the analyst and the contextual nuances of the problem. Such a well-informed assistant can go further in spotting and solving potential concerns.  For example, if the AI knows the analyst is examining the relationship of education and wages, it might add:  

It is important to consider a third factor, currently omitted from the analysis, such as the family’s emphasis on child-rearing that might increase both education and wages.

Similarly, it could warn about selection bias, such as: 

Because your sample consists entirely of individuals who have completed college or higher, these estimates may be subject to selection bias, as those who did not pursue post-secondary education are excluded. To properly interpret your results, it is crucial to account for this non-random selection process.

As the software learns from the user’s evolving expertise—and ideally, as the user’s own analytical skills expand—the degree of handholding and alerting provided by the AI can modify over time. AI systems can contribute to this process by logging each decision they make—from handling missing observations to selecting specific tests. Such an iterative approach can facilitate error detection, enhance understanding of the rationale behind the AI’s suggestions, and foster trust in the system. Just as researchers now routinely submit their Stata or R code with publications, one might foresee a future in which the submission of transcripts detailing LLM prompt-response interactions becomes a requisite component of the peer-review process.

How to implement? 

Many of these capabilities can be integrated simply by refining the prompt provided to a large language model.  For example, ChatGPT 4o implements some of these features if we add to a prompt requesting a correlation: 

To the extent possible, test whether the data satisfy the statistical assumptions of all techniques. Flag potential issues and interpret results with caution. 

Alternatively, it would also be possible to have additional software, module or agent that does this additional testing and explanation. In the context of increasingly popular agentic systems such functionality could be implemented as an adversarial mechanism: where one model is tasked with justifying each decision to another. However, relying on an extra module introduces a significant challenge: novice users, who stand to benefit the most from such extra guidance, may not recognize its relevance.

Thus, the optimal solution would be to embed this supplementary analysis as a default feature in any artificial intelligence tool that facilitates statistical analyses. Developers at organizations such as OpenAI, Anthropic, Google, and Meta should consider incorporating these instructive defaults into the system’s core functionality whenever it identifies that a statistical analysis is being performed. An effective approach might include querying users about their statistical proficiency at the outset and dynamically adapting the system’s feedback based on subsequent interactions and inquiries.

The Balanced Future of Statistics

As statistics become more accessible, the opportunities for innovation and insight are vast. However, the risks of misuse are just as significant. The same AI assistants that empower novices to use statistical techniques should also act as a safety net, helping new users avoid common pitfalls and guiding them to make confident, data-driven decisions.

By combining automated tools with ongoing education and awareness of statistical principles, we can create a world where everyone—not just experts—can harness the power of data responsibly and effectively. We will see users with less initial education producing rapid and credible analyses.  Along the way, the users will learn more statistics and all of society will benefit from more decisions informed by data.

As AI-based statistical tools mature, there is also an opportunity to integrate interactive learning modules. This integrated approach would allow users—regardless of their prior expertise—to progressively build deeper statistical literacy, thereby ensuring that the benefits of democratized data analysis are both far-reaching and sustainable.

We invite developers, educators, and policymakers to join the conversation and ensure that democratized statistics truly benefits everyone.

Note: Authors are listed in alphabetical order. The examples of age over 700 years and height of 999 cm are both from Levine’s experience. Written with the assistance of ChatGPT.

José Ramón Enríquez is a postdoctoral fellow at the Stanford Digital Economy Lab (Stanford HAI) and the Golub Capital Social Impact Lab (Stanford GSB). José Ramón obtained his Ph.D. in Political Economy and Government (PEG) from Harvard University in May 2023.

José Ramón studies the political economy of economic and political development with a focus on political accountability. Specifically, he has worked on understanding the role of information in improving political accountability, with a specific emphasis on misinformation, political polarization, and corruption; the causes and effects of criminal-political violence on democratic representation; and the effects of the lack of coordination across levels of government.

David I. Levine is a professor of business administration at Berkeley Haas and serves as chair of the Economic Analysis & Policy Group. Levine’s research focuses on understanding and overcoming barriers to improving health in poor nations. This research has examined both how to increase demand for health-promoting goods such as safer cookstoves and water filters, and how to change health-related behaviors such as handwashing with soap. He has also written extensively on organizational learning (and failures to learn).

David Nguyen is a research scientist at the Stanford Digital Economy Lab. David’s research explores new and better ways to measure the modern and digital economy. He is particularly interested in advancing economic metrics and statistics on economic output and welfare.

Prior to joining the Stanford Digital Economy Lab, David worked as an economist at the OECD in Paris, and as a senior economist at the National Institute of Economic and Social Research (NIESR). As a research associate, he remains affiliated to the London-based Economic Statistics Centre of Excellence (ESCoE). David received his PhD from the London School of Economics.

The modern-day Federalist Papers

In an effort to guide the critical decisions before us as a society, we borrowed the concept of The Federalist Papers, a series of essays written in the late 18th century that analyzed the great challenges of the day and provided a roadmap for institutional innovation for a young America. We concluded that it was time for another gathering of leading thinkers who could offer ideas about how we should approach governing in the age of AI. We asked 19 authors their thoughts and brought together their responses in a volume of essays we call The Digitalist Papers.

Unlike The Federalist Papers, we did not have a goal of persuading readers to adopt a pre-determined action (in their case, arguing for the ratification of the Constitution). Rather, our aim was to elevate the voices of varied experts who have informed and well-researched ideas and bring out a menu of solutions across a range of related topics. Authors represent diverse fields—including economics, law, technology, management, and political science—along with leaders from industry and civil society. 

This matters because the thoughtful examination of AI-enabled developments in relation to our institutions requires us to turn the spotlight on those institutions and ask whether they remain fit for purpose in all respects, and or if they can benefit from AI or other changes. This is a daunting exploration that demands deep domain expertise and collaboration across academic disciplines.

(One final thing worth noting: Authors did not see each other’s work before publishing.)

Here’s what we learned:

• AI has the power to make governance more inclusive and technologically integrated: A group of essays urged for a significant shift in how AI intersects with the democratic process. Ideas included AI-driven collaborative governance, using AI to scale direct democracy by amplifying citizens’ voices, and using digital tools to advance hyper-local community-focused engagement.
  
  
• AI is forcing us to rethink how we organize our government: Some authors focused on how our government’s failure to deliver the services its citizens have come to expect will lead to civic disengagement, and how the adoption of AI—as well as broader adoption of technology focused on meeting citizens’ needs—can help reinstill faith that government is for the people.
  
  
• The regulation of AI is arguably more important than the tech itself: Three authors focused on the role of regulation, stating that the processes by which we currently regulate don’t stand up to something as dynamic and transformative as AI, that we might be exaggerating on AI’s impact on information and thereby undermining trust in all media, regardless of its accuracy.

Given the broad spectrum of perspectives, at times the authors offered conflicting ideas and differing approaches. That is by design. But the bedrock of every essay was that democracy and AI can and should work well together; that our democratic institutions can be renewed and reinvented for the AI era.  

A technology as fast-changing as AI requires nimbleness, keeping an open mind, and the continual engagement of a diverse group of leaders to debate and guide the technology. It’s not inevitable that AI will lead to more freedom and participation in democracy. If left unchecked, there’s a chance AI would change how we govern in ways the Founding Fathers might have found abhorrent. Therefore, it is our responsibility to approach these urgent questions with the conviction that we have the agency to change it.

What excites you most about the potential for AI to enhance education?

TM: I believe this is the decade when AI can change education for the better. Why? 

First, we have known for a while that human tutoring significantly improves student learning. 

Second, we now have multiple online education platforms that have taught millions of online students, who therefore have more experience teaching than a person could get even if they taught for a hundred years. We now have machine learning methods that use that data to learn and teach better. 

Third, we have just begun to create new computer tutors that use LLMs like ChatGPT to teach in new ways, like having conversations with students about their confusions, and like co-writing essays with students. The opportunity is great, but we need to organize and fund this research if we want to turn this potential into reality.

EB: As an educator myself, I’m acutely aware that my industry has been badly lagging when it comes to using technology–after all, I often use a chalkboard that Aristotle himself would have been comfortable with. But with generative AI, we have the potential to transform education in a fundamental way. In particular, we know from research that students who get individualized tutoring can learn up to two standard deviations faster than students in a classroom where 20 or 30 kids are all instructed at the same pace. Of course, individual tutors have been far too expensive to provide for all kids. 

Now LLMs like ChatGPT have the potential to change that. I already know a lot of kids, and for that matter, adults, including a Nobel Laureate, who use these tools daily for individualized tutoring to learn new subjects and to dive deeper into existing topics. What’s more, LLMs can be incredibly engaging and entertaining. In the coming years, I expect a revolution as developers create even better versions of these tools and millions of stu dents gain their benefits.

Is there anything in particular you’d like readers to focus on?

TM: One of the most important things we can do for the workforce is give them a clear, real-time picture of how the demand for different worker expertise is shifting over time, and what education opportunities are available to them to chart their own career path. A key part of this is actually collecting this real-time information. That will require public-private data partnerships that combine data from different companies that have real-time information about job openings, resumes, salary scales, and more. 

We can do this, but it will require imaginative thinking about how to build these partnerships while respecting the legitimate privacy and competitive concerns of the organizations that have these data.

What would you hope to see if you did another report down the line?

EB: One thing I expect will happen if we do a report like this in the future is that we’re going to not just talk to talk but walk the walk. AI will be a much bigger contributor: helping us with the literature review, gathering the relevant data, analyzing it, and even assisting with the writing. We are acutely aware of some of the weaknesses and risks of AI, but also the real strengths, especially when it’s combined with human oversight.

Lastly, what do you hope people take away from the report?

TM: We strongly expect continuing technical progress in AI, and expect it to significantly impact the economy and the workforce. I hope readers of the report get that the future we’ll live in is not destined by technology—really, it will be determined by the decisions we make about how to use it. That includes decisions made by politicians, technologists, workers, and all of us.

EB: As we worked on this report, it became clear that there are some real opportunities for an amazingly better world when it comes to improvements in science, healthcare, and widely shared prosperity, but also a real risk in areas like privacy, bias, democracy, national defense–even catastrophic risk. We laid out different scenarios and discussed some of the choices that we can make individually and collectively. 

Our most important conclusion was that there is no predetermined future, but rather our choices will determine our future. In fact as the tools become more powerful, our choices become ever more consequential.