Use your seacharts to find out where your yellow arrow is in relation to the "x" and move accordingly. If you aren't on the little map on your treasure chart (the blue side) then you aren't close enough. If you are far away, head for the shining white pillar. When you are close to the X you will hear a shimmering noise.
The louder it is, the closer you are.
I find it helps to compare the map the respective portion of your seachart and look at the treasure chart at the same time.
If you are doing this and still struggling, you might be in the wrong area. The order you obtain the shards is not linear, so you may be in the wrong place if you are following some sort of guide. You remember the IN-credible chart tingle gave you? After you verify that chart, a triforce symbol will glow in the area that you need to be in. Make sure that's where you are.
Hope this helps.