When planning an aquascape the first thing to do is to sketch it out on paper! It helps to see what will work ahead of time. Taking all of the items below into consideration, I choose plants that I know will be compatible with each other.
Stem plants that grow tall usually occupy the rear and sides of the aquarium. When planting these I bring them in at least two inches from the side and back. It allows plant leaves to spread and not jam the glass and provides a natural corridor for the fish to navigate "behind the scenes". In the midground goes plants that will sit under some of the stem plants(lower light plants) and provide a focal point to the aquascape either with driftwood and rocks or as a stand-alone focal point. I also try to arrange those plants in such a way that lends depth to the composition and allows for future growth by providing spacing between plants. In the foreground I plant those plants that will cover the substrate and stay low to the surface of the substrate.
When planting the background I try to contrast the colors in a manner that is pleasing to the eye. You wouldn't want the same color texture to dominate the back, so different shades are utilized to make each species stand out on it's own and lend interest to the look. Dark next to light, subtle shades next to each other, and usually one dominate red plant and possibly a subtle red somewhere else. The midground then is planted with plants that have slower growth and offer different shades and textures, and a plant group with or without driftwood or rocks as a focal point. I personally think one foreground plant color is best, however it can be a combination of two plants of different colors used to cover the substrate foreground. Uncovered substrate in well chosen foreground areas can lend to the color scheme as well.
Shapes and sizes
Generally speaking small and narrow leaves work better in an aquascape, plants with larger leaves go better next to fine leafed plants, smoothed leaf plants go well next to crinkled leaves, and of course tall in the back but different height background plants that fill spaces and leave spaces is more interesting than all the same height. The same can be said for midground. An Anubias large leaf plant can be a striking addition to a small group of smaller leafed midground plants. The other consideration are plant shapes compatible with the fish population you choose? How do the fish move about in the aquarium? In schools, are there top level swimmers, mid level swimmers or bottom level swimmers? You wouldn't want the surface of the water covered in plants either floating or trailing if you have Hatchet fish, as they need room to swim and play. If you have shy fish by nature you definitely want areas where they can hide or feel safely surrounded by plants in at least one or two locations. Cory's like some bare substrate to rest, schooling fish need some open areas to swim together.
Water and light Conditions
Generally speaking if you have two or more watts per gallon of lighting you wouldn't put an Anubias or other mid to low light plant in an open area, as it would get more light than it likes. Algae will grow on the leaves if it's a slow growing plant. To prevent that you plant it in a shaded section of the aquarium, either in the corners that don't get as much light, or under taller plants. If a plant does well only in soft water then don't use it if you have harder water conditions in your aquarium, as it won't grow as well as it would in soft water. The temperature of the water also plays a factor. If your aquarium is consistently at 80 degrees F don't use cooler water plants, as they eventually expire in the higher temps.
Very important! How much time do you have to spend each week on your plants? If very little, then you might want to go with low maintenance plants like slower growers, less stem plants (fast growers), and more manageable growth. I do my trimming with water changes as it's easier when the water is not completely full. I use a lot of stem plants and have to trim at least once a week. You might not have the time to do that, in which case use slower growing plants to dominate your aquascape.
Look and Preference
Choose plants that you like the look of! I like Hydrocotyle more than any other plant as part of an aquascape but you might not. You are the one that will be looking the most at your aquascape so choose plants, shapes, sizes, and colors that are pleasing to you. There are many plants that can be substituted for others with a slightly different look. There is no hard fast rule for that.
I hope that helps all of you aspiring aquascapers. I can tell you this, a well aquascaped aquarium is a constant conversation piece with all who gaze upon it, and a source of ever changing, interesting living art! Did I mention extremely calming and relaxing? ~ Don Matakis