To the extent that design patterns are ever useful at all, they are useful when talking about code. They give you a common vocabulary that you can expect other programmers to know covering many technical abstractions that had no official names before patterns came to prominence. In other words, when used correctly the notion of design patterns is about terminology.
But terminology is only useful if it is providing greater specificity than ordinary language more succinctly. Design patterns often don't do this. It's succinct to call some class X an adaptor, say, but you are not really saying anything. It is less succinct to say that "Class X wraps the old library and exposes the interface that the new version of the software expects" but at least that is actually saying something about class X. In other words, if you say, "Class X is an adaptor that wraps ... blah blah blah" what exactly is the word "adaptor" actually adding?
Also the whole subject seems a little confused in that in some cases patterns are sort of common themes in how algorithms interact with data structures (e.g. the visitor); in other cases they seem to be the data structures themselves (e.g. the composite), and in other cases they seem to be styles of programming (e.g. wikipedia lists RAII as a design pattern. Is RAII even meaningful in languages besides C++?)
Don't worry about design patterns and certainly don't worry about whether you are using one "correctly". Design patterns are an ill-defined topic that don't add much beyond some terminology that is occasionally useful when talking about code.
If having a class do conversion between units polymorphically helps you then do it. If it adds too much complexity then don't do it. Call it an "adapter" if you want to, or don't -- that is all "patterns" are adding.
Honestly I don't understand why "design patterns" haven't gone away already as a thing. Why are young people so interested in them?