Field test kits are indispensable tools for strengthening community-based water quality monitoring and surveillance programs. However, the reliability of field test kits is important, as these kits are being used to test large number of water sources in developing countries due to insufficient laboratory resources. A field test kit validation protocol is essentially needed to strengthen the quality control mechanism during the kit production and procurement stages. The technical efficiency of a kit is the most important criterion for assessing its sensitivity and specificity to accurately detect the parameter of interest. The adequacy, safety and information to users are also very significant aspects to ensure effective and safe usage of kits. In the present study, arsenic field test kits were evaluated in the laboratory as well as at field by performing in situ testing. Statistical weight was assigned to each parameter, such as technical efficiency, adequacy, safety and information, to estimate the 'individual parameter weight', and then an importance factor was applied to estimate the cumulative parameter weight to assess the effectiveness of the field test kits. The overall arsenic testing efficiency, in the case of the most effective kit when used in the field, was estimated at 55.0% when compared with accurate testing results in the ± 5% range and 90.0% when kit findings were compared with accurate testing results in the ± 10% range. Considering variable and dynamic field conditions, variation tolerance between kit results in the field and accurate laboratory results in the ± 10% range seems to be appropriate. This study provides inputs for the development of a protocol for the reliability assessment of field test kits.