Making sure I'm not harming my cat with fostering other cats

 I am a cat foster for our local rescue groups. I am also a foster fail.  My boy is approximately 1 1/2 and after my fosters are  tested, fully vetted and recovered I start the introduction process to my boy because I never know how long I will have my fosters and refuse to leave them in a room.  Once acclimated and playing through the gate, the foster is free to roam. My question is how does this affect my boy when the fosters are adopted? I'm so worried about this. Thank you!