Target detection is essential to the advanced driving assistance system (ADAS) and automatic driving. And the data fusion of millimeter wave radar and camera could provide more accurate and complete information of targets and enhance the environmental perception performance. In this paper, a method of vehicle and pedestrian detection based on the data fusion of millimeter wave radar and camera was proposed. The first step is the targets data acquisition. A deep learning model called Single Shot MultiBox Detector (SSD) was utilized for targets detection in consecutive video frames captured by camera and further optimized for high real-time performance and accuracy. Secondly, the parallel Kalman filter was used to track the targets detected by radar and camera respectively. Since targets information provided by the camera and radar are different, different Kalman filters were designed to achieve the tracking process. Then, the targets of radar and camera were matched by using coordinate transformation. After that, fusion weight was calculated according to the tracking results. Finally, the targets data were fused based on Bayesian Estimation. At first, several simulation experiments were designed to test and optimize the proposed method, then the real data was used to prove further. Through experiments, it shows that the measurement noise can be considerably reduced by Kalman filter and the fusion process could improve the estimation accuracy.