How to stop an abused puppy from being aggressive towards people?

I have a 2 month old pitbull, who I believe has been abused. When I got him he had cuts on his face around his eyes, his belly, legs, and his back it was horrible. He seemed very shy, as did my first puppy when I got her, but after a few days he began to show signs of aggression. He would bark at every dog we ped, which I am not used to because my 5 month American Staffordshire Terrier was the exact opposite. She never barked, it took her about a month to bark and that was only when she wanted something. Well he also began to bark and growl at people. I also noticed that when I picked him up he would'nt look me in the eyes he'd wriggle around avoiding me. I have had him for a couple weeks now, and he has improved as far as barking at other dogs when I walk him, and he no longer has issues with me looking him in the eyes he now licks me like crazy. But he is seriously afraid of my brother and step father and my mom. He growls at them if they approach him, snaps at my brother if he holds him, and my mom has told me that he bit her when she tried to pet him. She said he watched her as she did her laundry snd when she went to pet him he just bit her. I tell him no when he growls at them. They have gotten to the point that they are afraid of him, afraid of what he might do when he gets bigger. He is fine with me, very affectionate, but not with other people. He is afraid, they try to bend down and talk quietly to him, but he still growls. I don't know what to do. I love him dearly, but I also do not want my family to be afraid of him. Please help!