I find it hard to understand how scientists can make definitive claims of exoplanet size and location using the Radial Velocity method of planet detection.

As an indirect detection method, how can we attribute the impact on the star to only one planet? Doesn't the Radial Velocity method really detect a center of gravity tugging upon the star? As a result, are we not actually measuring the net effect of what could potentially be a larger planetary system on the star rather than the impact of just one large, close orbit planet?

My question is... when it comes to identifying planet size and orbit, how are scientists so sure they have it right?