So let's say quarters = x and dimes = y. You'd then put x + y = 77 (for a total of 77 coins). And your second equation would be .25x + .10y = 12.50 (.25 for the total value a quarter holds and .10 for a total value a dime holds). Then you'd do matrices on a calculator.
