Self defense is about stopping a threat that is being directed at you, or a member of your family. There are both violent and nonviolent means of accomplishing this, and the course of action to take is always dependent upon the situation. Retaliation is getting back at a perceived wrongdoing against you.
Shooting an intruder that has kicked in your door is self-defense. Shooting a man whose face that you recognize from a previous burglary is retaliation. One of these is legal (although details vary depending upon the laws of your geographic area). The other will put you in serious legal trouble.
Self-defense stops once the threat stops. The window of opportunity for self-defense stops once the threat ceases. If a transgression against you is in-progress, then a measured response is self-defense. But once the transgression ends, then it becomes a matter of receiving justice (which is a matter of the legal system).
Self-defense can turn into retaliation if one does not engage in a measured response to a threat. If one shoots a home intruder and the intruder falls to the ground incapacitated, then the threat has stopped regardless of whether or not the intruder has survived. If the person in the home goes to "finish off" the incapacitated intruder, then the situation has moved from one of self-defense to one of retaliation.
Once the threat stops, self-defense stops.