Abnormal passive stiffness of the heart muscle (myocardium) is evident in the pathophysiology of several cardiovascular diseases, making it an important indicator of heart health. Recent advancements in cardiac imaging and biophysical modeling now enable more effective evaluation of this biomarker. Estimating passive myocardial stiffness can be accomplished through an MRI-based approach that requires comprehensive subject-specific input data. This includes the gross cardiac geometry (e.g. from conventional cine imaging), regional diastolic kinematics (e.g. from tagged MRI), microstructural configuration (e.g. from diffusion tensor imaging), and ventricular diastolic pressure, whether invasively measured or non-invasively estimated. Despite the progress in cardiac biomechanics simulations, developing a framework to integrate multiphase and multimodal cardiac MRI data for estimating passive myocardial stiffness has remained a challenge. Moreover, the sensitivity of estimated passive myocardial stiffness to input data has not been fully explored. This study aims to: (1) develop a framework for integrating subject-specific in vivo MRI data into in silico left ventricular finite element models to estimate passive myocardial stiffness, (2) apply the framework to estimate the passive myocardial stiffness of multiple healthy subjects under assumed filling pressure, and (3) assess the sensitivity of these estimates to loading conditions and myofiber orientations. This work contributes toward the establishment of a range of reference values for material parameters of passive myocardium in healthy human subjects. Notably, in this study, beat-to-beat variation in left ventricular end-diastolic pressure was found to have a greater influence on passive myocardial material parameter estimation than variation in fiber orientation.