Insurance Conversations with a Robot: What Happens When Generative AI Helps Choose Your Auto Insurance Policy?

Our team recently conducted an interesting—though admittedly unscientific—experiment in which we fed a series of insurance shopping related prompts into a popular generative AI platform. Our objective was to test the outer limits and gray areas of existing AI regulation as it pertains to the marketplace for insurance products.

Our first prompt was devised to provide the platform with enough information—but not too much—to formulate a reply that could be used to purchase insurance. (See Fig. 1.)

Figure 1. First prompt and Gen AI response

(The output is reproduced as generated, omitting hyperlinks and company names, which have been redacted.)

The AI platform recommends obtaining quotes from several insurers and using an insurance comparison tool to compare coverage options. The platform notes discounts may be available for low-mileage drivers, and provides suggestions from other consumers based on a Reddit post. The recommendations are presented in a clear and concise fashion, containing hyperlinks.

Implications for Regulatory Compliance

From a regulatory perspective, a Gen AI platform advising on insurance coverage matters raises the initial question of whether state insurance laws permit such conduct. All states require a “person” (defined to include an individual or business entity) to hold an insurance producer or agency license to sell, solicit, or negotiate insurance. “Sell,” “solicit,” and “negotiate” are defined terms in each state’s insurance laws. By way of example, in Arizona, “sell” means to exchange a contract of insurance by any means—for money or its equivalent—on behalf of an insurer; “solicit” means attempting to sell insurance or asking or urging a person to apply for a particular kind of insurance from a particular company; and “negotiate” means the act of conferring directly with or offering advice directly to a purchaser or prospective purchaser of a particular contract of insurance concerning any of the substantive benefits, terms, or conditions of the contract if the person engaged in that act either sells insurance or obtains insurance from insurers for purchasers. (Ariz. Rev. Stat. § 20-281.)

The Gen AI output from our experiment contained an explicit recommendation for coverage; it also identified insurers to get quotes from and recommended a specific insurance comparison tool. (See Fig. 2.)

Figure 2. Gen AI recommendations

While a Gen AI platform currently cannot sell insurance under any jurisdiction’s insurance code, does the output from our experiment rise to the level of “soliciting” or “negotiating” insurance?

If so, is the owner/operator of the Gen AI platform operating as an unlicensed insurance agent? Would it be in the crosshairs of a state regulator if the AI platform were deemed to be soliciting or negotiating insurance without a license? The current output appears to limit its guidance to generic, high-level suggestions without delving into the substantive benefits of coverage or urging the user to apply for a particular kind of insurance from a particular insurer. This is akin to business and financial publications recommending car insurance, as in this post from Forbes.com or this post from NerdWallet.

However, many Gen AI platforms are interactive, allowing a user to ask follow-up questions to refine the output. After receiving the above response, we formulated a second prompt. (See Fig. 3.)

Figure 3. Second prompt

The response to our second prompt was far more detailed and personalized, with the platform displaying a willingness to assume the role of an insurance agent, readily identifying itself as an insurance agent, and making tailored recommendations for the consumer. The AI platform began by formulating a policyholder “profile” restating the relevant parameters of the policyholder’s circumstance: location, vehicle, driving habits (“[l]ow-mileage (mostly within a 10-mile radius), work from home”), and needs (“[s]trong protection, low cost, benefits for low usage”).

The AI platform then proceeded to identify a recommended policy by name and coverage level and included a table enumerating these policy details:

• Coverage Type
• Liability
• Comprehensive
• Collision
• Uninsured/Underinsured
• Rental Reimbursement
• Roadside Assistance

The AI platform then summarized those recommendations. (See Fig. 4)

Figure 4. Gen AI recommendations summary

As far as we are aware, the company that owns and operates the AI platform we used does not currently hold any insurance producer licensure. Yet its platform recommends a specific insurance policy and insurer and even provides specific coverage amounts and deductibles. It estimates savings on premiums and discusses add-on coverage. Notably, it offers to guide the consumer through the quote form with the insurer.

This supplementary output strays far from generic advice and instead closely resembles an email exchange a consumer might expect to have with a licensed insurance agent. The platform opines on substantive policy benefits and explains why the policy works for the consumer.

How might regulators react to such a transcript? An objective reviewer could conclude that the platform has identified itself as an insurance agent and is attempting to sell a certain carrier’s auto insurance. It remains to be seen how insurance regulators will attempt to monitor or regulate chat-based AI solutions that are willing to assume the role of an insurance agent.

Data Privacy and Related Concerns

A related concern for both AI platform owners and insurance entities alike is data privacy. Unless users affirmatively opt out, many AI platforms retain user data and train the data aggregator on the information input. A consumer’s sensitive information (vehicle type, health history, driving habits, address, etc.) used to formulate insurance recommendations may also be used to enhance an AI platform’s capabilities for other users.

User data is not just being retained for use by the AI platform with which it was shared. In one piece of ongoing federal litigation, one AI platform owner was ordered to disclose the platform’s outputs to third parties and to preserve and segregate all output data, including data that would otherwise be deleted. The AI platform owner responded that certain enterprise and educational customer data is exempt from such an order, but for a large slate of users, including those who maintain personal accounts with and disclose personal information to AI platforms, such information is now in the hands of a third party. Even deleted chats and other content that would normally be deleted from AI systems automatically are subject to retention and disclosure.

Lastly, the accuracy of AI output remains salient. AI platforms routinely hallucinate or provide incorrect responses. Numerous examples of hallucinations exist across AI use cases: an AI chatbot informed customers about a change in company policy that did not exist; an AI tool created and cited nonexistent case law, which was cited in several motions by attorneys (who were sanctioned by the court); a newspaper relying on AI output printed a recommended summer reading list of books that did not exist.

While leading AI companies like OpenAI and Google have implemented safeguards to lower the hallucination rate of their AI products to 1 to 2%, no AI solution is error-free. This becomes a highly relevant issue for insurance regulators tasked with ensuring that consumers interact with competent, licensed individuals to recommend and sell insurance coverage. If an AI solution provides erroneous substantive recommendations that result in a consumer obtaining insufficient or inapplicable coverage, state regulators may consider whether the creator has violated applicable state insurance laws and act accordingly. 

Conclusion

With the rapid growth of AI solutions in the marketplace, regulators are working quickly to understand AI and its impact on regulated industries. In the insurance sector, legislators and regulators are expected to continue proposing and adopting AI-focused rules aimed at protecting consumers.

Insurance agents are required to be licensed to ensure that only those with sufficient training and knowledge advise consumers regarding the purchase of insurance. If an AI platform recommends insufficient coverage, however, it is unclear what recourse a consumer would have. Because the quality of an AI-generated response depends heavily on the user’s prompt, two consumers researching the same insurance question may receive very different outputs.

Given these risks, state regulators may seek to limit AI platforms from providing insurance advice that requires licensure or that could be considered false or misleading. Until clearer guidance emerges, AI platform owners and regulated entities should work closely with legal counsel to ensure compliance with all applicable laws.