Hi, one of my twins (4.4yrs) is quite shy. Until about 6mths ago we were actually very worried about how she would cope being so painfully shy.
She wouldn't even play with her twin sister at daycare and instead would either stay close to the daycare teacher or play on her own.
We tried to gently encourge her but if she still didn't want to do something then we didn't push her. Then about 6mths ago she started to really come out of her shell.
Now, she is most definately NOT shy at all. She does sometimes take a while to warm up to people though.
With the library both my twins still sit really close to me and hold my hands. They wont sit down by themselves. But we have to sit right at the back and not be right in the middle of everyone. Have you tried sitting right at the back? Or is there a chair nearby that you could sit on and sort of listen which might give him the impression he isn't sitting there with them?
I wouldn't worry too much at his age though.
Personally, I didn't like playgroups so we didn't do them. But that might be an option. Or just the park or playcentre etc where he will get exposed to other children. My girls happily run around with other children now even if they have just met them. Actually they don't meet them, they have no idea who they are.
Then when my girls were just over 3yrs old I put them into Kindy Gym which they love. It probably took them about a year to actually answer their gym teacher though.

They did what he asked but wouldn't answer him and instead would just look down at the floor.