So far as I'm concerned, if the RP is public, it's fair game. Most of them time if I choose to RP in /say, it's because I'm open to others joining in, or I'm inviting it in an indirect way. If for any reason I want the scene I'm doing to be between my character and only a specific character or group of others, it's best to take it to /l, /t or /p.
There's absolutely nothing wrong with sending a /tell to ask if you can join in. If you want to keep it spontaneous, emotes are often the way to go. But, as has been said, if they're too subtle they're difficult to work with. What they do, however, is indicate interest. If I'm RPing in a public place such as The Quicksand and a character emotes nearby that suggests that they may be interested in joining in, it allows me to make a similar emote which acts as an invitation. Just something as simple as "/em regards the newcomer with discreet interest before turning back to her conversation." shows the other RPer that you've acknowledged their presence and interest, and that you're open to their participation.
There's absolutely nothing wrong with sending a /tell to ask if you can join in. If you want to keep it spontaneous, emotes are often the way to go. But, as has been said, if they're too subtle they're difficult to work with. What they do, however, is indicate interest. If I'm RPing in a public place such as The Quicksand and a character emotes nearby that suggests that they may be interested in joining in, it allows me to make a similar emote which acts as an invitation. Just something as simple as "/em regards the newcomer with discreet interest before turning back to her conversation." shows the other RPer that you've acknowledged their presence and interest, and that you're open to their participation.