BACKGROUND: The Adoption Assistance and Child Welfare Act mandates an Administrative Case Review (ACR) for every child in foster care every 6 months regarding their safety, adherence to case planning, and progress towards permanency. Despite limited research, a well-implemented ACR process should improve child welfare outcomes. OBJECTIVE: To systematically examine the reliability of one state's ACR process. PARTICIPANTS AND SETTING: In a large Midwestern state child welfare system, the authors partnered with the state child welfare agency's ACR administrators and five independent, external experts to examine the reliability of their ACR ratings across key priority areas. METHODS: Intraclass correlation coefficients (ICCs) and percentage agreement on ACR ratings between the ACR reviewers and external experts were calculated. In December 2021 - May 2022 (Round 1) and December 2022 - May 2023 (Round 2), a random representative sample of 290 foster care cases were selected whose ACRs were completed by the ACR reviewers and external experts. Results from Round 1 informed documentation and training efforts prior to Round 2. RESULTS: The ICCs and percentage agreement on four combined priority areas in Round 1 was 0.43 and 40 %, respectively, compared to 0.63 and 63 %, respectively, in Round 2 on five combined priority areas. Both rounds showed variability in reliability across priority areas. CONCLUSIONS: Reliability improved from Round 1 to Round 2. Rigorous examination of the reliability of the ACR process, in addition to compliance with federal legislation, should be the basis for states to harness the full capacity of ACR to improve child welfare outcomes.