It sounds like you were being too standoffish with them and turned yourself into a threat that they wanted to confront for social points with their friends. For that type of situation, it's always a good idea to use the favor approach first.
Don't tell them to stop, don't be forceful, instead make it sound like they are doing you a favor, and if they respond positively, reward them for it and buy the whole table drinks, it will solidify their commitment to help you by calming down, and they'll all have something in common again (the fact that they don't want to give you a hard time)
Example: "Hey guys, it's sounds pretty lively over here. I know what's going on here is really not my place, and I hope I'm not interrupting something...but you see that guy over there? He's my boss, and if you continue to be as loud as you were just a minute ago, he's gonna kill me. Could I ask you for a favor to tone it down a few notches? Thanks, you are really saving my hide here. How about I get you all drinks? My treat."
Don't tell them to stop, don't be forceful, instead make it sound like they are doing you a favor, and if they respond positively, reward them for it and buy the whole table drinks, it will solidify their commitment to help you by calming down, and they'll all have something in common again (the fact that they don't want to give you a hard time)
Example: "Hey guys, it's sounds pretty lively over here. I know what's going on here is really not my place, and I hope I'm not interrupting something...but you see that guy over there? He's my boss, and if you continue to be as loud as you were just a minute ago, he's gonna kill me. Could I ask you for a favor to tone it down a few notches? Thanks, you are really saving my hide here. How about I get you all drinks? My treat."
