When we include libraries from the standard library (like cmath or cstring), what are we actually including?
First, understand that using different brackets in #include "header" and #include <header> only affects the order that the compiler looks through different directories when searching for the files.
Second, understand that you can use different file extensions (in your own headers) and everything will still work fine (as long as the file actually exists with that extension): myHeader, myHeader.h, myHeader.hpp, myHeader.pch, myHeader.meow, whatever - they are just filenames. Occasionally I've even had to #include a .cpp file from one .cpp to another. They're just text files.
Finally, what *actually* happens vary from compiler to compiler, because C++ only enforces behavior not implementation details. Also, the standard library is (mostly) separate from the compiler, even though it is shipped with compilers. For example, Visual Studio and GCC use *different* standard library implementations even though they behave the same and have the same interface. (GCC defaults to using it's own opensource libstdc++ library, Clang defaults to usibng its own opensource libc++ library, and Visual Studio defaults to using a custom-modified version of the third-party proprietary Dinkumware one. There are also other implementations of the standard library available).
The header files actually exist (usually), and (on Windows) they are usually located where you installed your compiler, unless you manually installed the libraries somewhere else.
In my QtCreator install of MinGW, on Win10, I find <cmath> located in:
C:\Qt\Qt5.5.0\Tools\mingw492_32\i686-w64-mingw32\include\c++
At that location, I find a text file named 'cmath' with no extension (I also find 'cstring' and 'iostream' and 'vector' and so on - text files without filename extensions).
It #includes "math.h", undefines some math.h defines (like cos(), tan(), etc...), reimplements them as real functions (instead of macroes), and puts them inside of the std namespace.
Depending on the compiler and header, the compiler can also just pretend behind the scenes that the header file really exists, but really do something else instead as long as the behavior is the same. For example, why recompile the standard library headers over and over again? Maybe the compiler just holds them pre-compiled, if it can detect that you aren't doing something that'd require it to recompile the header (like #defining certain macroes).
I'm assuming they aren't header files since there is no ".h" extension. Also, how can we just include files and be able to access functions from that library? Are the function definitions inside the included file? I thought it was good practice to separate declaration from definition?
First, a huge amount of the standard library classes and functions are templates. Templates in C++ have to be in headers, declaration-and-definition together. It's an unfortunate consequence of how templates work. So for those classes, whether standard library or 3rd party library or your own code, the header is plenty enough, no .cpp needed.
Second, for the functions that aren't templated, and have concrete definitions, they basically fall into several categories: If they are inline, or if the compiler wants to inline them for performance reasons, the compiler will generate the code inside your project's .exe - often those kinds of functions are only a few lines of code, so the performance gains outweigh the minor costs. This is common with compiler intrinsic functions.
For other functions, they can be handled just like regular static libraries (compiled into your code, but not inline), or dynamic libraries.
If they are linked to as dynamic libraries, then you have to make sure that .DLL actually is available for your project to find, when you run your executable. For Microsoft, the standard library DLLs are just shipped with Windows as part of the OS or as part of the Visual Studio runtimes.
For me, using GCC/MinGW, I have to include the standard library DLLs with my .exe. GCC's is named libstdc++-6.dll, which in-turn calls functions from some of Microsoft's libraries, since it has to function on Windows and some code has to run through the OS (like loading files).
You can also tell your compiler to statically include the standard library inside your .exe, but that isn't without tradeoffs.