LibreOffice and OpenOffice are free word processors. LaTeX is a free typesetting software which is commonly used for math and science textbooks and articles.

I'm not using it myself, but a friend of mine is doing books editing and he uses some specialised Adobe software for this, he find it superior to all traditional editors (hard to tell for me, but at the first glance it indeed looked more convenient, like ability to save whole chapters in separate files). I don't remember how that software was called (but it should be easy to google). Still, if you are making just one book it might be an overkill...