Most of the mathematics based problems in Competitive Programming that I have solved have been based on Discrete Mathematics. Kenneth Rosen is a good book for that. Learn number theory, Permutations and Combinations, generating functions and Probability.

There is reasoning based Math. Proving things and having strong conjectures that help you solve seemingly tough problems with no pre-requisites. Learn how to prove things properly. It helps you to be sure about what you are doing. Math is a way of thinking more than anything else.

Graph theory is just Math. I recently learnt Max Flow problem, Min Cost Max Flow problem, etc… Analysing the algorithms and proving their time complexities was just an exercise in Math.

FFT is based on complex numbers. The good thing about FFT is even if you don’t know much about the details you can use it a black box for polynomial multiplication. I learnt it using codeforces blogs, with all details, but the details haven’t been too useful when application of it in problems is concerned.

Convex Hull Trick is an optimisation technique generally used in dynamic programming problems. Its based on simple geometry. It won’t require looking into mathematics textbooks.

There are mathematical tricks that you’ll discover while doing competitive programming. I’m not sure if a book will teach you those things. Like some optimisations based on sums, time complexities etc…

Of course I know there is a lot of stuff I don’t know.

I don’t know about Linear Algebra applications as I haven’t solved any related problem. Although I have come across some problems based on Gaussian Elimination. I have heard about some more involved and rare topics and will look into them as and when they arise in some problems that I’m unable to solve. You can’t learn everything in one shot.

I’d say for starters learning Discrete Mathematics should be good. Learning to prove things will equip you with all you need to learn to learn any mathematics based topic.

I’d advise to learn something then solve a few problems, hit a block on some problem where you need to know some more stuff. Then learn that stuff and go on. You can’t learn everything effectively at once. This is not just for Math, its for any topic. Of course, just my opinion.