What is wrong with my solution?

It seems you also did the same mistake which I did, you need to count the total number of big and small numbers you encounter during the search(besides swap counters) and check if the count of total big numbers or total small numbers is not greater than the available amount of big and small numbers, if so print -1. (You can find total big and small numbers in log n time using your sorted array by function(s) like lower_bound() or implementing your own binary search).