How to write manual test cases for complex calculations

How do I write test cases for an application which consists of complex mathematical formula?
For example, suppose a user wants to calculate total cost and values coming from more than 3 or 4 modules/forms along with a condition. What will be the best approach for such a situation?

Since you are dealing with complex calculations across multiple modules, your first priority is to test that the calculations of each module are correct. Only then can you properly handle testing across modules.
Let's say your app has 5 modules. Each module can interact with any of the others. We'll call them A, B, C, D, and E
The app also uses two different conditions. Call them x and y. To make things more interesting, let's say that x applies to A, B, and C; and y applies to C, D, and E.
Now let's look at your potential test scenarios  you will of course start with testing each module in isolation and checking that the calculations are accurate. For each module, you should use a range of inputs the more or less follow this pattern:
 Invalid input below the minimum allowed value
 Exactly the minimum allowed value
 Somewhere in the allowed range
 Exactly the maximum allowed value
 Invalid input above the maximum allowed value
Since this set of tests also covers testing without condition x or y, your next set of tests applies the appropriate condition, and checks that you can't apply condition x to modules D & E, and that you can't apply condition y to modules A & B. You would also need to make sure that module C calculations work correctly when both condition x and y are applied.
At this point, you know that each module behaves correctly with and without conditions applied.
After that, it's simply a matter of working out what the output should be for whichever combination you're using, and checking that the application returns the correct output. Ideally there would be automated testing to tell you that the modules are behaving correctly, but if not, you would need to handle the testing of those before starting your more complex test.
That's really all that's involved. Cover each component in isolation, then in combination. I personally would start the combination tests with two components in combination, then move to 3 and so on. I would also, as much as time permits, change only one thing in each new test. So when testing say modules A & B, I'd test:
 A without condition x, B without condition x
 A with x, B without x
 A with x, B with x
 A without x, B with x
And so on.