Unless you've encountered a weird glitch of some kind where a key disappeared (very unlikely) then you must be overlooking something. There are six small keys in the dungeon so use a walkthrough to find which one you are missing- in the Water Temple it's very easy to miss a key.
Here's the walkthrough (also available above on the page):
http://www.zeldadungeon.net/Zelda05-ocarina-of-time-walkthrough-10.php
I'd go through all the steps of the walkthrough again if you can't tell which small key you missed just by looking it over.
Since you are as far as the last locked door, I'd guess the key you are missing is the last one which requires you to:
1. while the water is at the top and facing the center chamber from the entrance, swim to the right and enter the passage. It leads to an underwater hallway with a block. Push the block as far as it goes.
2. Lower the water to the bottom, then raise it to the middle. While standing outside the center chamber where you raised the water to the middle, look to the south passage. There is an eye switch and an iron grate blocking the passageway.
3. Shoot the eye switch with an arrow, then quickly longshot the target in the passage. Walk up to the block you pushed earlier and push it forward. A room to your right is opened with the final small key inside.
Hope this helps.
