Obvious aspects of any survey experiment
The survey is an experiment:
You first create a hypothesis, what you expect to happen based on your observations while researching and preparing for the survey. You then create an experiment that will either prove or disprove the hypothesis. If you hypothesize that shoe size correllates with hand span, you will design an experiment to collect the associated data of a participant's shoe size and hand span, but you will NOT also measure that person's height, because that has nothing to do with the hypothesis in question.
There is more than one survey:
Your variables are the participants (and any knowledge that they may bring), the environment for the survey, the information you give the participant, and the questions you ask the participant. In order to appropriatly account for each variable, you must create a number of minutely different surveys.
There are certain variables you cannot control:
This is probably the number one consideration of any experiment. All other considerations stem as a necessity from this consideration.
I will not be able to control the participants environment past the framework I create for distributing data. Since this survey will be computer-based, this means that not only do the person's system specs become a consideration, but so does the entire area around the participant. Since I cannot see this area around the participant, I will have to collect data from many participants.
For this reason also, we must select participants for each specific survey at random. This will ideally distribute the variation in the uncontrolled variables across all experiments, giving them the same error.
The knowledge of the participant can effect the outcome of the survey (or, why we need control experiments):
If a survey participant knows what you are trying to test/prove, then the participant will either conciously or unconciously skew their replies to match your expectations. It is vital that you control the amount of information your participants have on the survey. Part of this means that you will also conduct two sets of experiments: one where the participants are provided with no knowledge of the test, and one where the participants are provided with some knowledge of the test.
For example, say I am testing the effects of minute amounts of alcohol on the ability to drive. I hypothesize that, in small quantities, alcohol will act as a mild stimulant and actually increase the participants driving ability. I will conduct three sets of experiments, one in which I tell the participants nothing about the experiment, one in which I tell the participants that small amounts of alcohol are grossly detrimental to driving ability, and one in which I tell the participants that small amounts of alcohol are effective in enhancing driver responce time. In this case, I may find that my hypothesis is correct, but I may also find that knowledge of the situation has a greater effect than the alcohol itself.
I will also double this number of experiments again. This time, one set of experiments will use real alcohol, and one set will use a substitute with no intoxicating effects. Again, I may find that knowledge of the situation is the most important aspect.
aside: as it turns out, minute amounts of alcohol actually DO enhance driving ability. However, knowledge contrary to observation (that is, that it does *not* enhance driving ability) enhances driving to a greater extent (participants are more cautious). Also, the social implications are too great, as common knowledge of this fact may be used as an excuse to drink and drive.
More participants are better:
A larger sample of the population is statistically a better representation of the population as a whole. More participants = more accurate testing.
A very good set of guidelines for preparing online questionaires.