In fact, the only ones who don't care about the historicity/geography are a subset of those who are currently active in Church. Let's say 40% of Church members are active. That's about 5 million people. Of them, maybe half don't care about historicity/geography. That's about 2.5 million people in the world of 7.6 billion people.
While it's great that so many active members have such a strong spiritual testimony that they don't care about historicity/geography, the numbers speak for themselves.
We saw recently on this blog that the British Mission in the 1840s baptized more members than they had copies of the Book of Mormon. [With over 150 million copies in print today (plus millions of electronic copies), an equivalent result would be over 200 million members instead of the reported 16 million.]
How did the Apostles accomplish this in the British Mission?
Parley P. Pratt, who published the Millennial Star, explicitly recognized that some people may need to know of physical evidence before they can exercise faith enough to read the book and receive a spiritual witness. Among other things, he published Letter VII, which declared unambiguously that the final battles of the Jaredites and Nephites took place in New York. This grounded the Book of Mormon in reality.
When they wrote Letter VII, Joseph Smith and Oliver Cowdery knew how important it was to ground the text in reality. Ignoring or rejecting what they wrote leads to confusion and doubt. This creates friction that deters people from reading the Book of Mormon.
The other day I heard a comment about friction.
"Any time you add friction to a process, people do less of it. So for example if you have a restaurant that doesn't have good parking, you will get fewer people coming, even if you knew that they could all get parking, but in their minds they thought, 'It's kind of hard to park. It might take me a few minutes.'"
"If you add friction to anything, people change their behavior--and it doesn't even take much friction."
Let's consider friction in terms of missionary, retention and activation work.
We all want people to read the Book of Mormon. That's a major point of doing missionary, retention, and activation work. But as we all know, people have a lot of demands on their time--as well as distractions.
Anyone who considers reading the Book of Mormon compares that activity with their alternatives. E.g., read the Book of Mormon or watch TV?
People make cost/benefit decisions. What are the costs of reading vs the benefits of reading? The higher the cost, the less likely they are to read. Or, the lower the anticipated benefits, the less likely they are to read.
Think of costs in terms of friction. Like in the restaurant example above, additional costs or impediments make people less likely to read the Book of Mormon.
Imagine an investigator who is looking at a copy of the Book of Mormon, wondering whether to read it. She has talked with the missionaries, maybe, or a friend. They told her the book relates the appearance of Jesus Christ in the Americas. Naturally she asked, "Where and when did this happen?"
They told her, "Soon after his resurrection."
"We don't know. Somewhere in the Americas. Probably in Central America, as the painting in the book shows." The missionaries or friends show her the John Scott painting in the book they gave her.
Not knowing any better, she accepts that explanation. Maybe she is curious enough to read. Maybe she likes what she has heard and that's enough motivation. But people today have all been trained to research things on the Internet.
She goes to the Internet and quickly sees a variety of opinions about the Book of Mormon that boil down to the question: Is it fiction or real?
If she looks into it more, she'll discover that while all the prophets and apostles have taught that the Hill Cumorah is in New York, certain intellectuals in the Church teach that such a location is impossible. They teach the prophets and apostles were wrong.
Is that going to help our investigator believe the Book of Mormon is true? Or are the teachings of these intellectuals introducing friction to the process, creating impediments for investigators and less active members of the Church?
When we reject what Joseph and Oliver said in Letter VII, we are introducing friction to the process. We will have fewer people read the book as a result.
And the only reason we're rejecting Letter VII is because of the work of a handful of intellectuals who think they know more than all the prophets and apostles who have endorsed and taught what Letter VII proclaims about the Hill Cumorah in New York.