The current locations of objects are informative about the temporal order of past events. For example, by simply examining the locations of objects underground, geologists and historians can determine their relative ages. In three experiments, we explored the development of this ability to infer time from structure in children 3-6-years of age (N = 317). In all experiments, children saw pictures of object arrays (e.g., a stack of blocks) and selected the item placed first or last. Children in the final experiment also made judgments about the future (e.g., "Which block will they pick up first?"). By age 5, children were mostly accurate at inferring the order of past events. Children were more accurate when inferring first than last, and when inferring the future than the past. The findings suggest that children infer history by simulating how past events unfolded, and that 3-4-year-olds may struggle to perform these simulations.