Knowing the relative position of the viewpoints is equivalent to knowing the relative position of 3D points. Therefore the relative position of 5 viewpoints in general position suffices to obtain a metric reconstruction. This is the principle behind the omni-rig  recently proposed by Shashua (a similar but more restricted application was described in Pollefeys et al. [120,119]).
It is less obvious to deal with the orientation parameters, except when the intrinsic parameters are also known (see below).