This paper is concerned with a topic that is fundamental to understanding the mechanisms of the “displacer” portion of a novel rotary engine, which creates displacement and transmits torque. Since the cam profile appears to be the heart of this displacer system design to assure the mechanisms work correctly, geometric constraints on the cam profile have been studied at first. An analytical approach has been derived to determine a functional cam profile. Then, the relations between the motion and performance of the displacer system and the cam profile have been derived. Finally, an optimization approach has been used to determine an optimal cam profile, which provides the best overall performance in terms of efficiency, motion, motion capability and structural reliability.