Sea ice motion is triggered by wind and ocean currents. Its magnitude and direction can be automatically retrieved using pairs of satellite images acquired over the same area. However, external reference data for validation of drift retrievals, such as tracks from buoys, are sparse. Information about the reliability of the retrieved ice drift field is crucial for applications such as operational sea ice mapping or validation of computer models for simulations of sea ice dynamics. In this paper, we introduce an intrinsic measure based on the properties of radar image pairs to assess the reliability of the retrieved ice drift vectors. The proposed method combines different parameters, e.g., correlation coefficient and two textural quantities, to provide information about the suitability of subimage regions for pattern matching. In this way, we generate a quality parameter [called confidence factor (CFA)] for the calculated ice drift velocities. The CFA is compared to results obtained by “backmatching.” The latter requires that the drift field is computed twice using the image pair, first in sequential and then in reversed order. For stable ice conditions, the results show that areas regarded as unreliable by the CFA compare well with the areas revealing larger differences from backmatching.