Whether you end up with a $1 million portfolio or a few hundred thousand dollars can come down to the average annual return.