Suppose you have a matrix that represents your tiles. For sake of example, suppose your entire landscape is a 2x2 matrix (in your application, maybe 100x100 or 1000x1000). Store an enumerated type or a similar kind of thing in this matrix.
Now, each tile is a square (at least the texture is, suppose). Consider each vertex of that square. Get the type of each vertex by determining what tiles it touches.
Example:
consider a 2x2 world:
R G
G R
where R denotes rock and G denotes grass.
The 3x3 grid of vertexes is then (separated by commas, new rows on new lines):
100% R (1 R divided by 1 total), 50% R and 50% G(1 R & 1 G div. by 2 total, 100% G
50% R and 50% G, 50% R and 50% G (2/4 and 2/4), 50% R and 50% G
100% G, 50% R and 50% G, 100% R
The “type” of a vertex is a floating point vector summing to 1, basically saying: this vertex is about X% rock and Y% grass and Z% water, etc.
Then find where the transitional tiles are. This is easy. All tiles bordering tiles of a different type - and by bordering, I’m using the minesweeper definition: one diagonal step away or bordering on a side - are transitional tiles. You only need to calculate these “vertexes” as I’ve defined them for these transitional tiles.
Now, using the vertexes, interpolate at each pixel of the transitional tile you are creating to find the amount of each of your standard textures’ contributions to that tile.
Example:
Say I want to make a tile that has vertexes labeled
50%R/50%G 100%R
50%R/50%G 100%R
I might wind up with a tile that looks like this (each entry is a pixel, although there’d be more pixels):
50%R/50%G, 51%R/49%G, 52%R/48%G, … , 99%R/1%G, 100%R
50%R/50%G, 51%R/49%G, 52%R/48%G, … , 99%R/1%G, 100%R
…
…
50%R/50%G, 51%R/49%G, 52%R/48%G, … , 99%R/1%G, 100%R
where 50%R/50%G just means take the value at that pixel of the rock tile, multiply it by 50%, and add it to 50% of the value at that pixel of the grass tile.
If I’m confusing the hell out of you, post again at the end of the week. I’m solving the same problem too, so I might be able to hand you a full algorithm.