Call Center Transcripts

Benefitfocus shared 47 call center transcripts, representing eight weeks of conversations about birth/death QLEs and the related appeal process. As a proof of concept to show Benefitfocus how they could analyze the thousands more transcripts they had going back several months, we used AI to sort and analyze them efficiently. 

Three programs were compared by manually reviewing six transcripts and evaluating the findings of each AI program. We used ChatGPT 4.0, a data analyst-tuned version of ChatGPT, and Marvin AI. Findings were categorized as legitimate, questionable, or incorrect. Marvin AI achieved a 100% success rate, accurately assessing the purpose, theme, and problems in each transcript. Therefore, we decided to use Marvin to analyze the 41 remaining transcripts.

Transcript Analysis

Using Marvin AI was straightforward yet powerful. Transcripts, provided as Word documents, were bulk uploaded. Marvin analyzed each transcript and generated notes on the calls. These notes allowed for a quick review of each call's purpose. Marvin also performed further AI-powered analysis to track thematic and emotional trends, creating graphs showing the percentage of calls in each category. Questions were then asked directly to Marvin to identify overall themes, outliers, bottlenecks, and other relevant information. 

Key themes emerged, including confusion with time limits, frustration with the appeal process, and lack of transparency about missing details and errors in submitted materials.

Interviewing Call Center Agents

With these key themes in mind, we drafted an interview protocol to address several points of confusion in the QLE process. We first spoke with nine call center agents at Benefitfocus and Voya Financial, each with various roles and responsibilities.

This provided insights into how employees navigate QLEs using Benefitfocus's current system and offered an inside perspective from individuals familiar with the operational side of the process.

Interview with member of the service recovery team at Benefitfocus

What Call Center Agents Say

From these initial interviews, we found that transparency was a major roadblock, with agents frequently having to provide clarity on documentation, critical deadlines, policy information, and how to navigate the interface. 

Pain points included dependents not being covered and the inability to add or remove a dependent. Agents suggested these problems could be addressed by including quick-access links to the start of the process, texting or emailing employees to notify them of every QLE change, and providing them with an online cheat sheet so it would be clearer what the necessary documents were.

Employee Interviews

In addition to speaking with call center agents, we also wanted to talk with end-users on the other end of the QLE process.

To accomplish this, we recruited 5 friends and family members aged 55+ to understand BenefitFocus's shortcomings during QLEs. We focused on employee experiences, mixed QLE familiarity, open enrollment issues, and policy confusion.

Among these particpants, QLEs experienced included marriage, a dependent turning 26, filing a disability claim, and experiencing the birth of a child at least 20 years prior.

Grief Group Surveys

Finally, in order to address Benefitfocus’s concern about employees who had experienced the loss of a dependent, I recruited 4 participants from a grief group who recently experienced the death of a loved one and/or the birth of a child. This event required either them or a loved one to make changes to their benefits.

Two participants experienced the death of a loved one, one participant experienced the birth of a child, and one participant experienced both.

What End-Users Say