One thing you can do is set your inner loop ('b' loop) to start at the index one after the current value of a. This way you never compare an element to itself and you don't compare each pair of elements more than once. So the order would be: compare element 1 to element 2, then 1 to 3, then 1 to 4, 1 to 5, 1 to 6, then the 'b' loop ends and the 'a' loop goes to the next iteration and you compare element 2 to element 3, 2 to 4, 2 to 5, 2 to 6, then next 'a' iteration: 3 to 4, 3 to 5, 3 to 6, and so on.
I've made this change to the first method you mentioned (actually the second method is equivalent to the first, if I understand what you are intending to do there, so I'm just using the first method here). Let's see what it gives:
OK, so we got the mode (4). But what happens if I add another 2 and another 4 to the end of x? In this case the mode is still 4, but now 2 is repeated as well. Let's see what the method does:
So it got that 2 and 4 exist more than once in x, and the 2 seems ok, but what's with the three 4's? That's because there are three pairs of 4s in x: (element 3, element 6), elements (3, 8), and elements (6, 8). Each of these pairs is counted separately. Include another 4 to see:
That's six 4s for six pairs of 4s in x. There are four 4s in x, and taking each pair of them gives nchoosek(4,2) = 4!/(2!*(4-2)!) = 6 pairs.
OK, so this method gives extra 4's but still, 4 is the mode, so that's not necessarily a problem. We just have to be able to know that there are more 4's than 2's in the output vector y. And actually, that's the same as finding the mode of y, so this problem is equivalent to the one we started out trying to solve. Basically we've done something to x to get y in an attempt to find the mode of x, and now we just need to find the mode of y, so we're back where we started, right? If we had a method to get the mode of y, we could just apply that method to x and get the mode of x. There would be no need to calculate y at all. So maybe rethinking our approach would be worthwhile at this point.
OK. The mode is the value or values that appear the most, so how about counting the number of times each value occurs in x? (Then the mode would be the value(s) with the maximum count.) To do the counting, we can loop over elements of x, but the key is we'd need to keep track of which values have been counted so far and how many times they've been counted. I won't write down code for this, since you asked for guidance with your logic, but here's an outline of how it might work.
At the end of that process, for this vector x, you would get:
So that you'd know 2 appears 2 times, 3 appears 1 time, 4 appears 4 times, etc.
Then it's just a matter of grabbing the element(s) of counted where counts == max(counts) without using the max() function, and this is/those are your mode(s).