Do I need to remove this gravel?

I am turning my front yard into a vegetable garden. There is an internal border of bricks about 18inchs from the fence that has been mortered in place. In the soil of the brick border are a lot of gravel rocks. Should I sift them out or is it okay to leave it. Next year I plan on building raised beds on top of it. I am also buying some compost/soil to put on top of the area anyway.