Original posted on The Notorious R.O.B by Rob Hahn.
Agent ratings are back in the conversation, thanks to this scintillating op/ed by Kris Berg (link is for Inman premium subscribers only) who is one of the best writers in real estate today. I have written about this topic before (here and here) and it continues to fascinate and puzzle me still.
Kris’s point essentially boils down to the fact that providing real estate brokerage service is one fraught with emotion, with unpredictable clients who don’t know what it is that a realtor actually does for them, who cannot make rational evaluations of how good or bad an agent really is. Quantitative metrics don’t provide accurate ratings, in Kris’s view, because those focus on production rather than service. Customer surveys are flawed because customers are ignorant on the one hand, and nuts on the other hand, and are too often influenced by how the transaction itself went down rather than how the realtor performed.
All of her points are, I think, valid and true.
Sadly, they are all irrelevant to some extent.
Fact is, agent ratings are already here in places like Yelp and Angie’s List. Consumers will talk, will evaluate, and will rate realtors (as they do every other service provider) on their FaceBook pages, on blogs, on websites, and with each other in person. It’s going to happen whatever the merits of such ratings.
The real issue, then, isn’t whether such rating systems are good or bad or inaccurate or legitimate, but who will do the rating and how they will do it.
Evaluating Professionals
In an ideal world, evaluating the quality of performance by a professional is something that will be done only by other disinterested professionals. I remember thinking since my law school days that medical malpractice suits should only be heard by a jury comprised entirely of doctors. Only they can say whether a procedure was or was not done correctly, whether negligence was involved or not, and whether the actions of a particular physician was a judgment call or worthy of liability.
When our two year old had to go in for surgery, my wife mentioned the surgeon’s name on FaceBook. A friend of a friend on FaceBook, who is a doctor, posted that our doctor would be the surgeon he himself would use if his child needed similar surgery. The comfort of such an evaluation is immense. After all, if you really want to know who the great doctors are, look for the ones that other doctors go to for their medical needs.
Similarly, it may be the case that only other realtors can accurately rate a realtor’s performance in the course of a transaction.
The best rating system, then, is one in which every realtor would rate his counterpart after each transaction and submit a detailed report as to the reasons for the rating. A website would make it easy for consumers to view the composite score of each realtor, rated by his or her peers, with detailed ratings, trend over time, and reasons for why he got the rating he did on a transaction-by-transaction basis. And in this ideal world, every single realtor would be so professional, so dedicated to upholding the integrity of the profession that he would not hesitate for a moment to rate other realtors with stern rigor.
Sadly, we do not live in such a world. A jury filled with doctors sitting on a medical malpractice case may not exactly be what one might call disinterested. Attorneys who belong to the same Bar Association and rely on each other to refer business back and forth might be motivated by factors other than strict truth in evaluating a fellow attorney. And realtors are not saints.
Furthermore, real estate is less amenable to a peer review system than other professions due to the highly local nature of the business itself. Could a doctor in New Jersey review the work of a doctor in California? Probably so — the human body is not different in NJ than it is in CA (at least on the inside… on the outside is a whole different matter). But the real estate market in one town is different than the real estate market in the next town over; what might be best practice in one area might be a horrible idea in a different area. Which means that the only peers that could accurately rate a realtor are the other realtors in the same market. This is precisely the same group of people that a realtor would have to work with over and over again, and the same group of people who are constantly referring business back and forth to each other. The incentive to be critical of each other, then, is extremely low; the incentive to puff air at each other is extremely high.
So peer ratings is probably not going to work for real estate.
Consumer Reports
What about consumers?
Kris has already pointed out many of the pitfalls in consumer ratings of realtor performance. Too many consumers are emotionally invested and can’t think straight when it comes to their homes. In some cases, a horrible performance by a realtor may get glossed over because the consumer got her dream home (even if she overpaid, got screwed in the negotiation, and so on). A great performance gets downgraded because the transactions went too smoothly, and the consumer has no idea just how masterful a job the agent did.
Fact is, most consumers do not engage in a substantial real estate transaction often enough to be able to evaluate an agent’s performance accurately. The average consumer buys or sells once every seven years; and in between, she’s not thinking about real estate. There is no basis for forming an accurate opinion.
The result, however, is not what Kris fears: negative ratings by consumers who don’t know what they’re talking about. The real result is what we see on the one brokerage website that publishes all of the consumer reviews: far too many positives.
This is the Deals and Reviews page for Leonardo Montenegro, a Redfin Partner Agent, from California. I picked him entirely at random, but having looked through quite a few of the agents, the pattern is the same. Of the 30 reviews on the site, 17 of them are Five Star, 1 is a Four Star, and the remaining are “no response” or “Agent did not provide contact information”. And the comments are simply glowing. For example:
Leonardo was very professional. We followed his advise and got our house. There were multiple offers on it but his fast, intuitive thinking was the reason we won. I highly recommend this young man.
Maybe the reason why this consumer got the house was because Leonardo recommended overpaying by 20% because he was in cahoots with the seller. (NOTE: probably not true, but illustrating a point here! Leonardo most likely is exactly the consummate professional his consumers paint him to be.) How could anyone know, except the selling agent on the other side of the deal?
Now, these stellar ratings and glowing comments would be great news for Leonardo… except for the fact that every other Redfin Partner Agent in the are also has similar stellar ratings and glowing reviews. As a consumer looking for a realtor in the Coachella Valley area, do these kinds of ratings provide any guidance at all?
The only thing a consumer who isn’t completely naive might read into is the non-responses. Ron Jesser, an agent in the same market area, had 29 reviews but only 12 responses. So maybe Leonardo’s 60% response rate compared to Ron’s 41% response rate should tell me that Leonardo’s the better realtor? Yeah, highly dubious, but what else can you go by?
For many other professions or service providers, consumer reports are invaluable because consumers use those professionals more often. A consumer rating of a dentist, for example, might be valuable since he may have been going to a dentist every six months for the past twenty years. A car mechanic could be rated and evaluated more accurately by a consumer who’s brought the car in four times a year for various tune-ups, oil changes, wheel alignment, and the occasional fender bender.
The sad reality is, however, that in the absence of compelling and credible alternatives, consumer reports are what we will end up with.
Why Not Metrics?
So if neither professional peer review nor consumer reports can yield any insight, why not quantitative metrics?
There has been talk since Inman San Francisco, when Diverse Solutions presented their Agent Scorecard concept, of brokerages and MLS’s implementing some sort of a data-driven agent rating system.
Kris takes exception to these data-driven metrics because they don’t reveal all of the details behind a transaction. And again, she has a point. Days on market may not be a reflection of an agent’s skill as much as it is the local market conditions. Number of transactions doesn’t mean that the customer was served with superior skill — or any skill at all for that matter.
And yet… there is something appealing about numbers.
The closest analogy I could think of is comparing NFL quarterbacks. Who is better, Peyton Manning or Tom Brady?
Peyton has thrown for 48,799 yards to date, with 354 TD’s and a lifetime QB rating of 95.2; Tom has thrown for 29,495 yards with 217 TD’s and a lifetime rating of 93.6 with 63 fewer games played. Peyton averages 7.7 yards per throw, while Tom averages 7.3 yards per throw; Peyton averages 1.9 TD’s per game, while Tom averages 1.8 TD’s per game, but Peyton also throws .94 INT’s per game while Tom only throws .75 INT’s per game.
Behind those numbers, however, are all manner of details missing. Peyton Manning has played almost the entirety of his career with quality future Hall of Fame wide receivers like Marvin Harrison and Reggie Wayne, while Tom Brady did not have a bona fide stud receiver until 2007 when Randy Moss and Wes Welker joined the team. There is no way stats can capture how many times Brady has driven his team down the field in the last waning seconds of a game for a winning touchdown or a field goal. Numbers cannot capture how Peyton commands the huddle, how Brady is like in the locker room, or how good or bad the offensive line protecting them is.
And yet… any discussion of quarterbacks will involve numbers, because they are the onlyobjective measure we have. Numbers and statistics are far from perfect, and they do not tell the whole story. But they are objective and importantly, they can be compared.
Brought to real estate, perhaps it is true that DOM is not a reflection of a realtor’s skill or professionalism. But it does mean something, doesn’t it? If Agent A has DOM that is 40% above average, while Agent B has a DOM that is 10% below average, doesn’t that suggest something, whatever that may be? If one realtor sits at 95% sale-to-list over a ten year career, while another realtor is at 80% sale-to-list in the same market, doesn’t that at least suggest something?
Customer service metrics like QSC and NPS may be imperfect — in fact, I’m pretty sure both of those are pretty far from perfect — but don’t they at least suggest some basis for comparison? Redfin’s ratings page may be filled with only five star reviews, but at the very least, they exist — and the non-responses to at least suggest something about the level of service.
In particular, metrics are useful when they show significant deviation from the norm. For example, if the average DOM in a particular market over the past 120 days is 111 days, but Agent X during the same period is selling homes in 85 days, then whatever the reason, it is clear that using Agent X is likely to get your home sold faster. Note that I did not say “get your home sold faster for the price you wanted” — that would require also delving into sale-to-list data. But between those two statistics, the consumer can find an agent who is above the median realtor in the market for getting homes sold faster for close to listing price.
By the same token, the consumer might find that an agent reprices homes way more than the average agent. Perhaps the average agent in a market has two price changes before a sale, whereas Agent Z has seven price changes. That might suggest that Agent Z either doesn’t know the market that well, or gets listings promising the moon, or uses price changes as a marketing tactic. The full story requires detail, but the consumer can find that detail in followup questions to the agent, or by asking around.
Today, however, while the real estate professional has access to these metrics, they are hidden from view from the consumer. Why the fear of numbers?
The Imperfect Solution for an Imperfect World
As a result, I think the real estate industry should embrace some combination of objective metrics with both peer and consumer reviews. The specifics cannot be figured out in a blogpost, even ones as long as mine are. But the rough outline might look something like this:
- Key Performance Indicators from MLS data, such as price-to-list, DOM, and number of transactions.
- Peer Reviews from the MLS. While bias and gamesmanship can be expected, at a minimum, peer ratings would help eliminate the worst of the bunch. There is no incentive for professionals to tolerate individuals who give the entire profession a black eye by their incompetence or unethical behavior. I would include publishing any disciplinary actions by the local Association.
- Consumer Reports a la Redfin.
Combine them all and we may begin to have something approaching the semblance of accuracy in realtor ratings. As a consumer, perhaps I might compare two realtors, both of whom have glowing reviews and five star ratings across the board, then see that while one has better statistics, the other has respect from more of her peers, and go with that. Or I might choose to go with the realtor with more transactions with a better price-to-list track record, even if a consumer has dinged him once or twice.
Provide me the consumer with as much data and as much information as you can, and let me make the decision.
Seems to me that this combined approach is probably what our imperfect world, filled with sinners rather than with saints, needs.
Original posted on The Notorious R.O.B on November 28, 2009 in Marketing, Real Estate and Management.
