A hydraulic system is inherently nonlinear and sometimes its performance is greatly affected by those nonlinear factors such as deadband, saturation, hysteresis etc. For a system characterized with badly nonlinear behavior, the predominent nonlinearity must be well determined and compensated, otherwise the performance of the system will suffer. As an example, a hydraulic proportional valve controlled rotary motor servo is analyzed in this paper. Through theoretical analysis and experimentation result, a deadband is determined as the predominant nonlinearity in such system and parameters of the deadband are estimated by using system identification method. A nonlinear controller consisting of a linear pole-placement control algorithm and a deadband compensator is designed for the rotary hydraulic system. Experimental results illustrate that such a system controlled by this nonlinear controller exhibits much better overall performance as compared with that controlled by a conventional PI controller. We have also replaced the low-cost proportional valve by a servovalve to verify the overall system performance, which shows that a proportional valve can challenge a servo valve, if a proper nonlinear compensator is designed for the proportional valve.