The anwser to your question is simple. Follow the golden rule of programming:
This means that your first focus should be to make your code work. After your code is properly tested, you probably need some refactoring in order to clean up the design.
At this point you should have stable and working code. Because: "Only then, one should look at optimizing".
My experience tells me that premature optimization, especially at the cost of code readability or simplicity always comes back at you. The reason for this is that of the 10000 lines of code you write, probably only a few 100 of them will take up 90% of the processing time. Only optimize where you need to optimize.
Sidenote: I am not saying you should ignore algorithmic complexity until last.