What makes math hard for kids?

Why do some kids have so much trouble with math?
This question has been on my mind a lot because I'm wrapping up running a study on the development of math concepts in four and five year olds.  Part of this study involves assessing their current level of math knowledge using a standard test developed by a leader in this area of research.  It includes questions on counting, comparing two quantities, adding and subtracting, with one and two-digit numbers, with and without manipulatives and visual aids.  Because the children are so young, parents observe the entire study and see how they answer these questions, and are often surprised both by the difficult questions their children answer correctly and the more basic ones they fail even to understand.  
For example, our participants all answer the following question incorrectly: "how many numbers are between 7 and 9?" 
A few of them don't understand the question at all, and guess some random one-digit number.  Most of them reply, "8."  They know the number 8 goes in the middle.  They do not take the next step, realize we're asking about the number of numbers--a second order question--and give the correct answer (1 or 2).  
This is not supposed to be one of the more difficult questions.  It's grouped amongst all the other questions about one-digit numbers presented without visual aids--all assumed to be at the level of the average six year old.  A level that, incidentally, the local population of four and five year olds in this high socioeconomic status area seems to have already reached.
The problem is that they don't understand the word "between."  Ask them the other "between" question on the test--"how many numbers are between 2 and 6"--and they almost never give you a sensible answer, and sometimes they'll look at you blankly.  It doesn't help to use gestures (use a hand on one side of the table to represent 2, another to represent 6, and gesture to the space in the middle).  It doesn't help to use different phrasing, such as "how many numbers do you have to jump over to get from 2 to 6."  They just don't get what you're asking them, probably because it's not the sort of math question they're used to being asked.
Another hard one is "is 5 closer to 6 or 5 closer to 2?"  (where we present a visual aid, a triangle with these numbers at each of the points, with 5 at the topmost one).  The blank looks and wild guesses indicate that some children don't understand the question.  I don't blame them; I'd be confused about whether they were talking about the relative length of the sides of the triangle I'm holding up or the abstract relationship between the numbers.  I've rephrased it "is a five year old closer to a six year old, or closer to a two year old?" to make the distance more palpable, but even this was too confusing for one four year old I tested recently.
In general, the language of math, not math itself, is what trips them up.  They'll answer a question wrong because of how it's phrased when they've already demonstrated through earlier questions that they know the concept being assessed.  
This always surprises parents.  It's never occurred to them that math has its own language, its own jargon.  That kids don't just have to learn how to count and add and subtract, but they have to learn the verbal cues to  perform each of these operations.
Their faces light up in that "wow, I just learned something new today" way and I can see the wheels turning as they think about how to help their child learn the language of math.  (This reaction always makes my day). 
It would be fascinating to do a training study where we demonstrate something like this to parents and then follow them over the next couple of years and see how their kids achieve in math compared to kids with the same educational and socioeconomic background who didn't receive this sort of training. 
Have you or your child struggled with the language of math?  Do you think this would be a productive area for education to focus on?  Let me know what you think!