Picture a college student sitting alone late at night, phone in hand, typing a few lines into an anonymous peer-support forum. The message may reflect nothing more than a hard week, or it may be the first time this student has admitted how much they are struggling. What happens next? A concerning post needs to reach someone qualified to respond.
For TimelyCare, a virtual care organization serving students, the challenge was finding a sustainable way to identify which posts needed that level of attention. Its initial approach generated too many false positives. AI offered a way to focus human review, but the team needed guidance on how to build and oversee it responsibly.
TimelyCare became one of the first five organizations to earn URAC’s Health Care AI Accreditation and the first in behavioral health. Its experience shows how accreditation standards can help shape AI implementation while an organization is still developing its approach.
A framework for putting AI to work
TimelyCare’s CEO asked leadership to explore how AI could improve efficiency while maintaining safe, effective care. Heather Geracie, Vice President of Clinical Excellence, already had a quality assurance program supported by URAC’s Telehealth Accreditation. Finding clear guidance for extending that work to AI proved difficult.
When Geracie learned about URAC’s Health Care AI Accreditation, AI was still a new conversation within the company. The accreditation standards gave her a framework to bring to her colleagues in the product and engineering teams as they developed the processes around these tools.
One early application was TimelyCare’s anonymous Peer Community, where students at different colleges can support one another. The first version flagged a significant volume of posts, creating additional work for the team due to false positives. The team developed an AI-assisted review to analyze posts and determine which should be flagged for human review and which could enter the community. Development involved writing prompts, testing the system and adjusting those prompts before testing again.
Geracie identified testing and validation as a particularly valuable part of the accreditation process. A registered nurse with an emergency medicine background and 17 years of experience, she understood clinical quality assurance. Monitoring AI for bias and drift introduced technical considerations that were new to her. The standards helped her work with technical colleagues to establish documented testing and retesting procedures, alongside ongoing quality oversight.
TimelyCare also developed an AI chart-auditing tool to help its small quality team review care at greater scale. In an August discussion with URAC, Geracie said the organization expected to exceed 300,000 telehealth visits that year.
The tool draws on the clinical practice guidelines, policies and scoring methodology behind TimelyCare’s existing Quality Assurance Program. It scores completed charts on a scale of one to five, with higher scores indicating areas that may warrant closer review. This helps the team prioritize records for further examination and apply clinical judgment to provide policy-driven feedback, coaching, or corrective action when appropriate. Each score is a starting point for investigation.
These applications gave TimelyCare concrete ways to focus staff attention. Accreditation helped the organization establish how its teams would test and oversee the technology supporting those decisions.
For leaders considering AI accreditation, that experience offers a useful starting point.
Start with a defined problem
Identify a specific gap in your quality process. TimelyCare needed a more targeted way to review Peer Community posts and a way to prioritize chart audits as visit volume grew. Those needs gave its AI projects a clear purpose.
Define what the tool should help staff do and how you will determine whether it is helping. Use accreditation standards during development to identify the safeguards and evidence your organization will need. That gives clinical and technical teams shared expectations while there is still time to shape the system.
Put clinicians at the design table
Clinical leaders need a role in deciding what these tools should flag and how their performance will be evaluated.
Geracie described roughly six months of work developing policies and procedures with colleagues across the organization. Product and engineering teams brought technical expertise; clinical teams brought experience with evidence-based care and quality review. The accreditation standards gave them a common reference point for deciding what to build and how to demonstrate that it worked as intended.
Organizations pursuing accreditation should budget for that collaboration. Geracie emphasized the need for adequate resources and strong cross-departmental working relationships. Leaders should give those teams time to resolve questions about testing and responsibility before a tool becomes part of everyday work.
Plan the response before the alert
A flag is only as useful as the response behind it. Before deploying an AI review tool, leaders should define who receives each flag, how quickly it must be reviewed and what action may follow. In TimelyCare’s Peer Community, that means a clear path from a concerning post to a professional who can assess whether a student needs support. In chart auditing, it means investigating flagged records and using the findings to improve care.
Organizations must also anticipate what the system may miss. Periodically auditing unflagged posts and lower-scoring charts helps identify blind spots and refine AI detection capability over time.
Ongoing evaluations should review the full workflow, including measuring response times, tracking false negatives, and assessing variances between clinician evaluations and AI outputs to continuously refine system performance and testing protocols.
Finally, maintain thorough, documented oversight. Documenting testing procedures, tracking system modifications, and recording follow-up actions ensures accountability and embeds quality assurance into daily operations.
In behavioral health, clinical attention is a resource worth protecting. The student typing alone late at night deserves a system designed to notice possible distress and a professional prepared to respond. AI can help direct that attention, but clinicians remain responsible for what happens next. When technology empowers clinicians to focus on signals that might otherwise slip through the cracks, the result is more than efficiency; it translates to lives changed and crises averted.
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
