We introduce efficient tensor network models for sequence processing motivated by correspondence to probabilistic graphical models, interpretability and resource compression. Inductive bias is introduced via network architecture as motivated by correlation and compositional structure in the data. We create expressive networks utilising tensors that are both complex and unitary. As such they may be represented by parameterised quantum circuits and describe physical processes. The relevant inductive biases result in networks with logarithmic treewidth which is paramount for avoiding trainability issues in these spaces. For the same reason, they are also efficiently contractable or 'quantum-inspired'. We demonstrate experimental results for the task of binary classification of bioinformatics and natural language, characterised by long-range correlations and often equipped with syntactic information. This work provides a scalable route for experimentation on the role of tensor structure and syntactic priors in NLP. Since these models map operationally to the qubits of a quantum processor, unbiased sampling equates to taking measurements on the quantum state encoding the learnt probability distribution. We demonstrate implementation on Quantinuum's H2-1 trapped-ion quantum processor, showing the potential of near-term quantum devices.