Here are some of my thoughts on how to not worry about performance.
Don't Worry, Be Happy
Sufficient unto the day is the evil thereof.
In other words, there are enough things to worry about. Attend to what is needful. Don't over-react to what may be in the future.
Art of Coding
There will be certain laws of performance that we run up against -- things to measure like complexity, time and space.
We can look out for these, obey rules and observe our heuristic instincts.
But until we run up to hard facts, we apply our preference for certain values in code. It's an art. It's what we live for! It makes what can be a very tedious job sometimes into a more enjoyable journey. More enjoyable as a creative endeavor than "always memoize your calcuations here and here."
There is a worthy order of optimization that some software artist before me has conjured:
Make it work, make it pretty, make it fast.
We get to each -- in turn and in priority. Maybe.
Working: Check. Black or white. Passes acceptance criteria and automated test.
Pretty: Focus on readability and maintainability. These are the next-most important aspects of code after correctness. Humans will read and maintain the code now and over and over again into the future.
Fast: Once we get here, the code is already working in an elegant way. Sometimes adjustments for speed can negatively affect the other two: We make sometimes-arcane optimizations that can unintentionally introduce bugs and make the code difficult to reason about. Thus, it might be worth while to skip any additional speed-related changes in order to maintain simple correctness and readability.
If we have a non-functional requirement like a particular speed metric, or we have observed user-facing issues, we might have our hand forced. We might need to optimize for speed. To do so before the requirement to do so, however, is premature. The tradeoffs do not yet indicate it's a good idea. It's like writing a feature that we aren't sure we'll ever use.
Didn't someone call this the root of all evil at one time?
We often don't know if something needs optimization -- we've never measured it. We don't actually know what needs optimized yet. And our intuition on computer performance optimization is prone to folly.
We sometimes adjust a thing hoping for one result (eg, speed in one place) but create a different, unexpected result.
We want to focus our optimization those parts of code that will yield the greatest, needed payoff -- the biggest problem or most-constricted part of the funnel.
So, in my opinion, most of the time: Make it work. Optimize for readability. This will tend towards simplicity. Optimize for speed only when compelled.