"Fifth-generation-and-beyond" communication technologies have sparked considerable demand for polymer composite materials with low coefficients of thermal expansion (CTE) and low dielectric loss at high operation frequencies. However, the complexity of process parameters and the lack of knowledge about fabrication procedures hinder this goal. In this study, state-of-the-art experiment-in-loop Bayesian optimization (EiL-BO) was developed to optimize a composite of a perfluoroalkoxyalkane matrix with silica fillers. The Gaussian process equipped with an automatic relevance determination kernel that automatically adjusts the scaling parameters of individual dimensions effectively enhances EiL-BO's ability to search for candidates in a complex and anisotropic multidimensional space. This addresses the critical challenge of handling problems with high-dimensional parameters and is capable of managing eight-dimensional parameters, including filler morphology, surface chemistry, and compounding process parameters. The obtained optimal composite shows a low CTE of 24.7 ppm K