Gears are essential parts of many precision power and torque transmitting machines. However, the radiated intensive tonal noise due to the gear meshing is highly undesirable and annoying. In very severe cases, the gear vibrations can reduce the life and performance of the power transmitting components. Typical gearbox vibration and sound spectra contain several dominant narrowband tonal signals that are mixed in with a lower level broadband response signals. Hence, the control of mesh response of gearbox housing belongs to the problem of the rejection or cancellation of periodical disturbance. The frequencies of these tonal signals are related to the number of teeth and rotation speed, and highly predictable. Thus, a feedforward control system was normally adopted. In most of existed applications, an accurate reference based on the frequency information of tachometer pulse train signal is required for this kind of control system. However, in the proposed ANC system, the feedback control with internal model control structure is used. The controller is designed using the inverse modeling technique. In this paper, numerical simulation was conducted based on the recorded gear housing vibration responses and measured dynamic response of the actuation system from an actual high-density gearbox system. Simulation results show that a promising performance of the proposed control system can be achieved at the target gear mesh frequencies.