If this is really a problem, try to keep those includes out of your own headers, perhaps by encapsulating the APIs and only including the headers in very specific source files.
This. If you're including windows.h in one of your headers, you're doing something super wrong.
Use forward declarations and use wrappers. Out of a few hundred non-third-party source files in my latest personal project, only a few include windows.h: filesystem_win32.cpp, threads_win32.cpp, gpu_d3d.cpp, diagnostics_win32.cpp, and logging_win32.cpp. The first three are obvious I think. The diagnostics one is for callstack generation and the assert dialog, and the last one is for OutputDebugConsole support.
Note that not a single header includes windows.h. Of the third-party libraries I use, they only use windows.h inside of their .cpp files or their private .h files, so it never infects my project.
The same is true for just about any large header. D3D/OpenGL headers, sound middleware headers, SDL/SFML/GLFW/etc., bad C++ headers like <algorithm>, anything from Boost, etc.
That said, windows.h is rather special. It's both one of the worst offenders in terms of bloat and one of the most configurable in terms of bloat. For the times that I do need windows.h, I generally wrap it behind a custom windowskit.h that #define's a ton of magic macros before including windows.h that strips out a lot of its cruft and undoes some of its UNICODE macros that annoy me or conflict with identifiers I use.
Here's an example WindowsKit.h