I am running a least squares solver to determine affine matrix between two sets of points.
All the values in points X and Y are identical, except point 1 of Y matrix x-value is changed by 1 pixel

The affine calculation works correctly and I can plug it back in to solve Y = A * X

However, I want to find average translation and rotation between corners, and the affine matrix returns a very large value for translation.

In this case the average translation should be 1 pixel / 5 values = 0.2 pixels, but it returns a value of 5.47

Is this a mistake in calculation, or inherent property of the solution of affine matrix?
Is it better to take difference in position values per corner and divide by number of corners instead to get the average?

I apologize I cannot copy paste this code, only screenshot.