Call me a skeptic if you must, but I think most would agree that there are fewer players enjoying RTS games and as a result fewer games being developed for the genre.
Currently the last champions of RTS have been reduced to a few significant names, Starcraft 2 that has seen a decline with its e-sports centric audience (most likely mobbing towards the ever popular MOBA genre) and Command & Conquer which had its last official released game panned by Critics and abandoned by fans(Company of Heroes 2 is the other other recent RTS example). The attempt to make a Free to Play C&C apparently failed as well, it’s pretty damning to see the two biggest names in a once popular genre fail to keep a foot hold as they once had.
In a competitive environment they can be quite difficult to play, and quite stressful at times due to how unforgiving they are. More so than team games. This can scare a lot of people away in the long term. The thing to keep in mind is that because there is no reliance on a team, all losses and any mistakes are 100% the fault of the player. This leads to a lot of bruised egos. Also because of this difficulty, it is very difficult to play 1v1 in a casual way, especially right now. Most of the lowest level players have dropped out so anyone just starting is going to be facing some fairly tough competition compared to if they started even a year back.
Most of the people I know who switched from SC2 to DOTA/League did so because they like the team atmosphere and find it less stressful to play.
For non-competitive RTS, it seems like many of the developers have been stuck in a bit of a rut. The Creative Assembly games have been all kinda bland for the last couple, and CnC is basically dead. AoE Online was a bit of a bust so whether or not we will see something new from that series I don't know.
I'm a huge fan for innovation in the genre such as the RPG elements introduced in Warcraft 3 or the focus on capturing bases/checkpoints in Dawn of War / Z and wish to see more of this as I believe it's one of the things to draw players back into the scene. That and perhaps innovating the multiplayer experience to more social for the more casual market that's available.