The present work proposed to implement oxy-fuel combustion mode into a homogeneous charge compression ignition engine to reduce complexity in engine emissions after-treatment and lower carbon dioxide emission. The combination of oxy-fuel combustion mode with homogeneous charge compression ignition engine can be further optimized by the utilization of direct high temperature and pressure water injection to improve cycle performance. A retrofitted conventional diesel engine coupled with port fuel injection and direct water injection is utilized in this study. A self-designed oxygen and carbon dioxide mixture intake system with flexible oxygen fraction adjustment ability is implemented in the test bench to simulate the adoption of exhaust gas recirculation. Water injection system is directly installed in the combustion chamber with a modified high speed solenoid diesel injector. The combustion characteristics of homogeneous charge compression ignition coupled with oxy-fuel combustion under different oxygen fractions, engine loads and speeds are studied in this work. The experimental results indicate that the minimum applicable oxygen fraction in oxy-fuel combustion with homogeneous charge compression ignition is around 85% limited by cyclic variation performance, and pure oxygen can be utilized under suitable fuel injection quantity without abnormal combustion such as knocking. As the exhaust gas recirculation ratio is improved, the start of combustion is postponed and combustion stability is deteriorated. The potential of utilizing water injection to control abnormal combustion while maintaining system thermal efficiency identical is also illustrated, the feasibility of implementing oxy-fuel combustion into homogeneous charge compression ignition engine is proved.