The physical basis of the direction of time