Ancient Jews celebrated a feast called the Day of Atonement. On that day, the high priest took two goats for sacrifice. The first goat was killed and its blood poured into a bowl and taken into the tabernacle and sprinkled upon the Ark of the Covenant, the place where God’s presence rested.
Inside the Ark were the twin tablets of the Ten Commandments, which Israel had never been able to keep. The goat’s blood was sprinkled between the presence of God and the Law which condemned Israel.
When the high priest finished this, he went outside to the second goat, placed his hands on it’s head, and confessed the sins of the nation. The animal was then led far out into the wilderness and let go. The idea was that the sins of the people had been laid upon another, and they were being carried away from the people, never to be seen again.
When Jesus came into the world, He stood as the Mediator between God and sinful mankind. He placed Himself on the altar between the holiness and the justice of God, and the sinfulness of humanity. God the Father then placed our sins upon Jesus, removing them from us forever. Our sins are so forgiven, that God does not even remember them (Jeremiah 31:34).
While we receive the great benefit of forgiveness, God didn’t forgive us for our own benefit, but your sins are forgiven you for His name’s sake (First John 2:12).