I first heard about the concept of Genchi Genbutsu years ago when I first read The Goal (and subsequently The Phoenix Project). Put shortly, it's the concept that you can't really know or understand something well until you see it happening in person.
I have been following this principle, innately, for the vast majority of my career. I don't attribute this to any particular wisdom or even being particularly precocious when it comes to doing user-facing (IT) technical work. Instead, it's because I am extremely stubborn when it comes to the kinds of problems that arise in and around technology, and I will bang my head against as many walls as required until I figure something out. So no, not wisdom, but that stubbornness lends itself well to Genchi Genbutsu. It means I have done things, manually and the hard way, so I have an appreciation for how the basics work, and how the underlying pieces fit together.
It's this experience, this willingness to dive in and get in the weeds and see first-hand how the proverbial sausage gets made that, in my opinion, is the first step toward true differentiation between good technical support and bad technical support.
I have spent most of my technical career in user-facing technical roles: First as a repair tech in a small local computer shop, building and fixing PCs, installing and configuring routers and managing small business networks. Then as a member of the university IT staff, doing regular helpdesk work as well as creating and deploying images for whole computer labs, then as a systems administrator, then as a support engineer for a few pieces of open-source software. These days, I'm not an engineer anymore, but I am responsible for running a global team of support engineers.
As part of that, as you can imagine, I am immensely concerned with the quality of the support that we offer. I think we've all had some pretty awful experiences with support teams in the past, and it's, unfortunately, all too rare to have a truly exceptional experience. While I do think in the high-tech industry at large, quality support is becoming a focus - as the industry has shifted to product-led grown paradigms, the game becomes giving users a top-notch experience from cradle to grave. That often means putting in a lot of work to help make sure they're successful, which means good support is a must.
The best thing a support experience can do for you is not, believe it or not, to just answer the question(s) you ask. That is, really, table stakes. That's the minimum that should happen. What makes for a better support experience is when your question not only gets answered but also the next one you're likely to have gets answered as well. Sometimes that even means finding out the question you asked was the wrong one to begin with!
Software systems are (usually) large and complicated. As you're investigating the usage of something new, it's very common to take a naive approach to the usage of that system. That is normal and expected, and very much a part of the lifecycle of adopting pretty much any new thing. But when you end up working with support, the best outcome is the one that helps you not only solve your immediate problem but also build your own skill and understanding. This is where Genchi Genbutsu enters into the support experience.
The better a support engineer knows their software in ways that are not pure theory, the better they can anticipate your needs and can anticipate the places you're like to go next. They have gotten their hands on the software and used it. They've built things with it, and they have traveled that path from naive to sophisticated. This isn't merely knowing the docs or the syntax. It's understanding what it's like to type a command in, what the response looks like, and what comes next.
Suddenly instead of a lecture or recitation of some piece of technical lore, it's a shared experience. And shared experience breeds empathy. I don't think you can be truly exceptional in any user-facing technical role without empathy. These things are hard - if they weren't, chances are good folks wouldn't be coming to you with the questions they do. And not knowing something is frustrating enough as-is. No one wants to be lectured at or brushed aside and given a link to something that might address their problem. They want to be heard, and they want to learn. (Note: This is not universal. Some users just want an answer, and for you to do it for them. That's another issue, the handling of which will have to be another post)
What happens, when you are intimately familiar with not only the theory but also the practice, of using a software system is you begin to think of problems more holistically. A problem with a feature changes from "try this syntax" to "try this syntax, and you can also use Y feature to make this more efficient." Chances are the user was going to start looking into Y next, anyway. And when you've done it yourself, you know that.
So then the best support experience has become a few things:
1) Learning something new.
2) Having a human connection by shared experience.
3) Having your needs met, even (especially!) the needs you didn't yet know you had.
Any good support organization should try to work themselves out of a job. If you get a user to the point they no longer need you, then you've won. And the company has won, because now that user not only deeply understands your technology, they've almost certainly become successful with it. And that's the kind of story that spreads. "They really helped me get this project off the ground." "I talked with their support, and they not only helped me out but gave me advice on how to better architect our whole approach."
These are true differentiators for your product and your business. They're every bit as important as differentiating features and novel technologies you may also offer, because what good are those features unless folks are successful in using them?
I see no other way to get there than to put yourself in the middle of things. See the reality of using the thing you're trying to support by using it yourself. This should be core to the training and onboarding of, not just all technical employees in the company, but also (in my opinion) all non-technical employees too. The length and depth of such a program will vary, of course, by role, but again, creating these shared experiences creates a conduit for empathy, and empathy between business units is the best lubricant you could ever ask for.