*** What I'm asking, is if anyone else does something like this, and if they have any advice for tracking it/measuring it
typically, i wont make a change in the first place until i'm sure it a change for the better.
but occasionally one adds a feature and it turns out to not be "all that".
usually, in those cases, i'll simply immediately turn it off, and move on to the next feature. maybe later i'll have an idea that might make it more fun, and then i can revisit the turned off code.
so there's no real need to track "more fun - less fun". everything should add fun/coolness (whatever your metric is), or its outta there! <g>.
i'd say don't over think the problem. go with your gut. if you implement a feature and your gut says "-eh- so so...", just turn it off, and chalk up the time spent to experience. next time, think harder about the dynamics that will result from implementing a new features and how that will affect the overall aesthetic (IE will it enhance it?) before you code. skip the features you're less sure about until all the really cool ones are done.
i find that once you learn how to code games, the really hard part is figuring out what to code (what mechanics / game rules to implement), not how to implement it. so its really about design. code is just how you explain the rules of the game to the computer.
also remember that metrics are very difficult to apply to subjective things like "fun". that may be why a "go with your gut" approach tends to work well.
trust your inner gamer, it will steer you towards the fun. anything you're unsure of, solicit the opinions of others, either through design discussions or via play testing.
for a more scientific approach, videotape play testers, and watch for body movement in sync with the game (IE they duck when the character should duck), and facial micro expressions which give away inner thoughts and emotions. count the type and number of instances of each over a typical hour of gameplay or something like that. then you'll have some hard numbers in the way of metrics to play with.
but me personally - i'd just go with my gut. <g>.