As I write this, it’s reminding me of how lucky I am that my Broca’s area is working properly. English is complicated, so much so that we usually have to go to school to understand its structure; sitting here writing a book about language, I still can’t honestly tell you that I understand all of English grammar. For the most part I “just know it”, and my brain turns my thoughts into grammatically correct sentences on its own: I speak on autopilot. This is Broca’s area, which has quietly mastered all the complexities of English syntax and is working diligently under the hood, freeing my conscious thoughts to focus on what I want to say, rather than how exactly to put it into words. Without the help of this autopilot I could still get simple points across, but it would be a laborious and frustrating process that was prone to error; the fluidity of speech would be gone.
Patients with aphasia can usually still count to ten. They even tend to know their addition, subtraction and multiplication tables; things like “five times six equals thirty” are actually stored in the brain more as rote phrases, the same way you might memorize a movie quote or a prayer: the series of sounds is not stored along with their meaning, and damage to Broca’s area doesn’t impair this. But ask them to do something they haven’t memorized, like adding 3-digit numbers or solving an algebra equation, and their ability falls flat. The syntactic rules for math are as lost as the syntactic rules for language. They still understand what numbers are, and when they would want to perform, say, addition rather than multiplication; the concepts are all there. But there is a fundamental disconnect between those concepts and syntactical rules.
It may be a bit of an oversimplification, but if you are forced to pick I would say that the “core” ability for any language, including math, is this ability to convey meaning through grammar. Memorizing vocabulary - the mapping between the basic symbols and their meanings - is important but limited. To really go the distance, we need to graduate to a “symbolic system” for knitting the basic symbols together into ideas of arbitrary complexity. And Broca's area seems to be, at least moreso than any other area of the brain, where the magic happens.