This paper presents a vision based pedestrian detection system. The presented algorithm is a novel method that accurately segments the pedestrian regions in real time. The fact that the pedestrians are always vertically aligned is taken into consideration. As a result, the edge image is scanned from bottom to top and left to right. Both the color and edge data is combined in order to form the segments. The segmentation is highly dependent on the edge map. Even a single pixel dis-connectivity would lead to incorrect segments. To improve this, a novel edge linking method is performed prior to segmentation. The segmentation would consist of foreground and background segments as well. The background clutter is removed based on certain predefined conditions governed by the camera features. A novel edge based head detection method is proposed for increasing the probability of pedestrian detection. The combination of head and leg pattern will determine the presence of pedestrians. The extracted segments are merged to form the complete pedestrian based on the evident leg and head pattern. The algorithm is tested on real time images and provides good detection capability. The accuracy of the above method can be further improved by using a classifier on the segmented region.