So, my questions is: should I try to avoid dynamic memory allocation.
When you don't need it, don't use it. When you need it, use it. C++ supports using any type as a value-type for a reason. It also supports free-store allocation for a reason. Use each where appropriate.
Roughly speaking, use value types (no allocation) when you can get away with it safely and easily. References in part are meant for this though you can also have pointer to free-store-allocated types just as easily.
The real proper use of these are all based on _lifetimes_. Who is responsible for creating the object? Who is responsible for destroying? Are users of the object just borrowing it for a while or do they need to take over responsibility of the lifetime? If they're just borrowing the object, how long do they need to borrow it for?
Value objects are created in a function's local scope and destroyed at the end of that scope. If that meets your lifetime needs then rejoice for you don't need any free-store allocation. If that does not meet your lifetime needs then you had better figure out which of the many ways of managing heap-allocated objects is best (and there's a lot of them, and all have their appropriate uses).
If you do need to use free-store, always consider std::unique_ptr as your go-to smart pointer. It's not always the right choice but it's the one you should consider first. For some concurrency data structures std::shared_ptr also works, though it and its brother std::weak_ptr are usually over-used and poorly understood and should be avoid unless you know for a solid fact that you need them and nothing else will work.
Integer handles (preferably wrapped to be type-safe) that reference an object maintained by a manager class are another good choice. In this case you replace calls to std:make_unique to something like Handle handle = myManager.createObject(parameters) and you free the objects with myManager.releaseObject(handle) and you access the actual object with MyObject* ptr = myManager.tryGetObject(handle) and remember to check for a nullptr return value (half the point of integer handles it that they can be invalidated safely, unlike pointers).
To me, it seems that dynamic memory allocation is helpful only when one has to create an array where the size of the array is not known.
This is called a std::vector and you should almost certainly never be allocating arrays like these yourself. Even if you do need a traditional stack-allocated array, consider using std::array instead of a C-style array, as it provides a number of niceties that C-style arrays do not.