The mean shift tracking algorithm has become a standard in the field of visual object tracking, caused by its real time capability and robustness to object changes in pose, size, or illumination. The standard mean shift tracking approach is an iterative procedure that is based on kernel weighted color histograms for object modelling and the Bhattacharyyan coefficient as a similarity measure between target and candidate histogram model. The benefits of the approach could not been transferred to monochrome vision systems yet, because the loss of information from color to grey-scale histogram object models is too high and the system performance drops seriously. We propose a new framework that solves this problem by using histograms of HoG-features as object model and the SOAMST approach by Ning et al. for track estimation. Mean shift tracking requires a histogram for object modelling. In the proposed framework a set of high dimensional HoG-features is clustered via K-means and features inside the object area are matched to the cluster-centers via a nearest neighbor search. This procedure is comparable to a Bag of Words algorithm. The proposed system is evaluated for advanced driver assistance systems and it is shown that the framework can be used as a reliable visual tracking system for a pedestrian recognition module.