AI-powered professional learning tools that provide teachers with individualized feedback on their instruction have proven effective at improving instruction and student engagement in virtual learning contexts. Despite the need for consistent, personalized professional learning in K-12 settings, the effectiveness of automated feedback tools in traditional classrooms remains unexplored. We present results from 224 Utah mathematics and science teachers who engaged in a pre-registered randomized controlled trial, conducted in partnership with TeachFX, to assess the impact of automated feedback in K-12 classrooms. This feedback targeted “focusing questions” — questions that probe students’ thinking by pressing for explanations and reflection. We find that teachers opened emails containing the automated feedback about 53-65% of the time, and the feedback increased their use of focusing questions by 20% (p < 0.01) compared to the control group. The feedback did not impact other teaching practices. Qualitative interviews with 13 teachers revealed mixed perceptions of the automated feedback. Some teachers appreciated the reflective insights, while others faced barriers such as skepticism about accuracy, data privacy concerns, and time constraints. Our findings highlight the promises and areas of improvement for implementing effective and teacher-friendly automated professional learning tools in brick-and-mortar classrooms.