BACKGROUND: Radiation-associated lymphedema and fibrosis (LEF) is a significant toxicity following radiation therapy (RT) for head and neck cancer (HNC) patients. Recently, the CT Lymphedema and Fibrosis Assessment Tool (CT-LEFAT) was developed to standardize LEF diagnosis through fat stranding visualized on CT. This study aims to evaluate the inter-observer reliability and diagnostic accuracy of the CT-LEFAT criteria. MATERIALS AND METHODS: This study retrospectively evaluated 26 HNC patients treated with RT that received a minimum of two contrast-enhanced CT scans. Qualitative review was conducted by five physician raters to assess the fat stranding observed on CT according to the CT-LEFAT criteria. Fleiss' kappa analysis was used to assess the inter- and intra-rater reliability, and Receiver Operating Characteristic (ROC) Area Under the Curve (AUC) analysis was used to evaluate diagnostic accuracy. RESULTS: The inter-rater reliability across the six CT-LEFAT regions generally indicated a slight to fair agreement across all raters (0.04 ≤ kappa ≤ 0.36). Intra-observer agreement was generally fair to moderate (overall kappa = 0.44). The ROC AUC analysis varied based on aggregation method used (0.60 ≤ average AUC ≤ 0.70). CONCLUSION: This specific use-case evaluating CT-LEFAT criteria displays limited performance. This suggests that additional materials, such as further training, refinement of imaging methods, or other processes may be required before achieving clinically-ready diagnostic performance of LEF diagnosis.