This is an interesting challenge. I looked at your web page stills, and your work is coming along quite nicely. I see the problem that you perceive - it would be nice to have a more realistic land/water interface.
You mention that you already have a means of determining the depth of the water, and that is the key to the physics, since depth is, by definition, relative to land. Staying close to the physics of the matter is key, so it seems to me that you might consider a coloration function that depends upon depth of the water. Since you also indicate that you are using a sinusoid to generate wave effects, you have means to generate water motion. Then, the coloration function might first look at a normal (or other) distribution, with a mean fixed at a certain water depth. That would give you a gradation, rather than a sharp demarcation. The color itself should pass into white on a scale from whatever color base you are using for water (blue, green, whatever), by increasingly saturating with white, to simulate turbulence. For general shore line, that should work pretty well, I think, and it piggy-backs onto the information that you are already working with. The down-sides would be (1) that any smaller features, such as rocks or sand-bars would display the same effect, which might not be as realistic and (2)unless some local control were exercised, every shore line would have ‘breakers’, or whatever effect you choose. In practice, the amount of turbulence depends upon more factors than just depth, but it’s a good start. Perhaps some random or local variation in saturation would lend more versimilitude. (3) At the scale that you show, this would probably work. I’m not sure how well this would scale, if you start doing closeups with a zoom-in.
As an additional thought, you could also consider simulating tides. If you look at the basic formula for tide amplitude, it is basically just a cosine function with local wrinkles, so it’s not hard to produce a generic tide effect. Whether that is relevant to your game depends upon the time scale, of course, but over the course of 24 hours, it would actually be a factor in the real world, especially for an operation which lands ships or boats on a coastline. Remember Normandy?
More sophisticated treatment would change water color itself in certain areas. Even relatively deep water is not always blue, but can be green. And while I doubt that this matters to your game, the degree of slope for the continental shelf affects water color considerably. On the U.S. western coast, the Pacific is fairly deep right to the shore line. Along the coast of the Gulf of Mexico, on the other hand, the slope is very gradual, which means that water action has a chance to pick up and carry bottom sediment from quite a distance from shore, giving the water a distinctive brownish color.
Hope all of this helps. RTS is my favorite genre. I look forward to seeing more of your product.