If you call delete on the base pointer, the destructor needs to be virtual. As far as I'm aware, there is no special rule for if the derived destructor is trivial, so you still need to call the matching destructor of the type that the object was constructed as. Now as frob says, if you don't call delete or explicitly invoke the base constructor, then having a virtual destructor is not required.

Take note: in the vector<Base> example, the destructor does not need to be virtual, because you never destroy the derived class through the base pointer. The temporary derived object has its destructor called after the base object is built without using virtual dispatch.

virtual ~Base(){};

...although I shouldnt need a destructor unless I allocate dynamic memory within the class, right?

No. A simplified real world example I once encountered was

class Base { };

class Derived : public Base { std::string m_myString; }

// ...
Base* p = new Derived;
// ...
delete p;

The std::string inside Derived never gets destroyed. If it allocated memory (which it likely did) then that memory is not freed.

Even beyond indirect memory allocation you can get problems. File handles might not get closed. Other scarce system resources (like sockets) might remain open until your application terminates.