AI Tools for Knowledge Management, Featuring Kelly See, Knowledge Analyst for BI/CI at Ericsson

Kelly See, knowledge analyst for BI/CI at Ericsson, joins “TBR Talks” host Patrick Heffernan for a discussion on the state of AI tools for knowledge management inside one of the world’s leading telecom vendors. See also shares how the demands of global knowledge management are being served with artificial intelligence tools across a diverse global workforce.

Episode highlights:

• Measuring knowledge management success

• How changes in technology and AI have evolved the day-to-day role

• Working with internal AI for the best results

Listen and learn with TBR Talks!

Submit your Key Intelligence Questions for Patrick and his guests

Connect with Patrick on LinkedIn

Learn more about TBR at ⁠⁠⁠⁠⁠https://tbri.com/⁠⁠⁠⁠

TBR Talks is produced by Technology Business Research, Inc.

Edited by Haley Demers

Music by Burty Sounds via Pixabay

Art by Amanda Hamilton Sy

AI Tools for Knowledge Management, Featuring Kelly See, Knowledge Analyst for BI/CI at Ericsson

TBR Talks Host Patrick Heffernan: Welcome to TBR Talks: Decoding Strategies and Ecosystems of the Globe’s Top Tech Firms, where we talk business model disruption in the broad technology ecosystem, from management consultancies to systems integrators, hyperscalers to independent software vendors, telecom operators to network and infrastructure vendors, and chip manufacturers to value-added resellers. We’ll be answering some of the key intelligence questions we’ve heard from executives and business unit leaders among the leading professional IT services and telecom vendors. 

I’m Patrick Heffernan, Principal Analyst, and today we’ll be talking about adopting AI tools and how non-traditional skills are applied in a corporate setting with Kelly See, Knowledge Analyst for BI/CI at Ericsson. When the Ericsson team released their tool to the full organization, the name was established as “Ask BIC” to keep the team branding consistent, but in this conversation, it was referred to as Erica. Please enjoy my chat with Kelly.

Corporate libraries

Excellent. Kelly, thank you so much for joining the podcast. I really appreciate it. I know when we met in Texas a couple of months ago, it was great to see you in person, and I know I sort of threw this out there as an idea, and I’m so glad you jumped on the opportunity.

Kelly See, Knowledge Analyst for BIC at Ericsson: Thanks for having me. I’m looking forward to our discussion.

Patrick: Yeah, and maybe we could start off with just sort of your roles and responsibilities at Ericsson, maybe how it’s changed and like sort of what is most of your job today and maybe a little bit of, you know, how it’s changed from when you first began at Ericsson.

Kelly: Yeah, so I started at Ericsson almost 20 years ago. We had a library at our North American headquarters in Texas, and I was getting my master’s in library science. So, I was an intern and then got hired to work in the library. And so, we did all the traditional library things. We actually had books that we checked out. We had some digital material, and then we did research. And eventually that library went away and I went to work in the Business Intelligence Center, which is similar in that we do research, and we support people that are trying to find information. But we solely have market research as our purview. So, we put market research into a portal that we have online, which we call BIC, or our Business Intelligence Center. And then I specifically support internal knowledge management. So, we have all the external market research that my colleague Elizabeth Roberts is responsible for. And then I work with internal teams to help get internal sources of intelligence, so reports or presentations or any material that they’re generating in their daily work and getting that into our portal so that when our users go out and search for information, they can find both that external and internal material. 

Patrick: Yeah. So, you said a couple things here that really have sparked a lot of interest for me. First, getting a master’s in library science. You- when you started, when you started your sort of academic pursuit in that, you probably weren’t thinking I’m going to work for a great big telecom company, or was Ericsson a company you had targeted that you wanted to work for? I’m just curious.

Kelly: No, not at all. I figured I would probably either work in a public library or an academic library. And I did not have any previous library experience when I started grad school. And I went to a networking session where they brought in librarians from all different aspects. They had academic and public, and then they had several corporate librarians there. And that’s how I met someone who had worked at Ericsson, and she connected me with Elizabeth, who was working in the library at Ericsson, and that’s how I ended up where I am.

Patrick: That’s fantastic. I mean, so I was with Deloitte for a while doing competitive intelligence and the same kind of experience where I didn’t even know that competitive intelligence within enterprises and companies existed. Sounds like you probably didn’t know that there were such things as, you know, these corporate libraries that you could go join.

Kelly: Yeah, exactly. And the great thing was that the aspect of it that I really loved was doing research. And I got the opportunity to do that in addition to the traditional library things. So, it ended up being a perfect fit. And now so even more with working in BIC, getting to support intelligence research and learn more about business and competitive intelligence has been really good.

Measuring knowledge management success

Patrick: And knowledge management, which we talk to companies across the entire technology spectrum, I mean, so, from IT services and consulting and telecom and cloud and software. One of the consistent, so I’ve been here a dozen years, one of the consistent themes is that knowledge management is really hard, often underfunded, never quite measured well, and always something that people sort of fall back on is, oh, we need to do better at it. So how do you, let’s take a couple of those, like how do you measure your own success when it comes to knowledge management? What’s the way that you say, okay, my mission here at Ericsson has been better served this year because we accomplished this with respect to knowledge management.

Kelly: That’s a really tough one because it is something that everybody knows they need to do and that they need to do better, but it’s a challenge to make the time and to change people’s habits and to get people to share, which can be really challenging. So, one of the ways that we have measured it is just counting the number of things that people are sharing. So, just in a really simple way, how many knowledge assets, we call them, are people sharing into the BIC platform. And then we’ve also worked with teams who have found other ways to share not just knowledge assets, but other types of insights. We also do special pages on our platform for different teams so that they can come into the platform on their own dedicated page that has information that’s specific to their business area or their market area. So the more interaction that we have with teams, the number of teams that we’re supporting, the number of questions or projects that we work on with those teams, those are just kind of the basic ways that we measure it. Also, we work a lot to just make people aware of what we’re doing, not just knowledge management, but also just BIC in general and the market research that’s available to the organization. So we do newsletters, we do training, we participate in team meetings and things. So, we keep track of all of our interactions as well.

Who is a good internal knowledge management client

Patrick: Yeah, and so that, I mean, that’s a- you have to start with, sort of, the most fundamental thing, like what are people actually sharing, that makes sense. And keeping track of that’s got to be a challenge. But we’ve had on the podcast a number of analyst relations professionals. And the question I come back to them a lot is, what does a good internal client look like? Like who do you like to work with within your own company because they understand what you do and what value you bring? How about for knowledge management? What’s a good knowledge management client, internal client? What are they like?

Kelly: Just somebody who is eager to share, I would say is probably the best internal client. Somebody who really gets behind the idea that what we’re doing is benefiting other people in the company, and they’re willing to work with us and help us as well as we help them. So, they become actually our biggest cheerleaders within the organization as well. So, they’re sharing in our portal, and then they go out and tell people, hey, we have this great tool. You should be using this as well.

Patrick: Yeah, that’s something that you share in common with the analyst relations professionals that we talk to, is that when they find somebody who’s an advocate for what they’re doing. It’s sort of- it’s great for you to go out and tell everybody about what you’re doing, but it’s even better when somebody internally is like, hey, this is this incredible resource, go use them. So, yeah. 

Kelly: Exactly.

How changes in technology and AI have evolved the day-to-day role

Patrick: I’m curious, so the other thing I’d love to talk about is sort of how much- so of course, technology has changed everybody’s lives, and especially in the last few years the way AI has sort of upended so much. But I’m curious, when you think back to what’s happened over the last five years, has it been that AI and technology has sort of changed your day-to-day work so much that it’s very different, or has it just accelerated what you typically did and just made it easier?

Kelly: I would say it’s accelerated it and made it easier. We’re still working to incorporate a lot of that AI into the work that we do. We have tried to be very careful about the way that we do that because we want to make sure that the AI is helping and not providing the wrong answers or bad sources of information. That’s something that we find to be really important, obviously, in intelligence work is that we want to make sure that we’re providing right answers, the best answers, accurate information. So, we have wanted to embrace AI, but we’ve taken it really slowly to make sure that the ways that we’re using AI are valuable to us.

Patrick: And you’ve hired an intern named Erica to help you do that, right?

Kelly: We have, exactly, yeah. So, Erica is our newest team member in BIC. She’s our AI research assistant. And we launched a little over a month ago with having that service available in our portal. And we’ve gotten some really good feedback about it, I think people really like it, so.

Patrick: Do you think that Erica is going to be most helpful, because it’s not just use cases, I’m anthropomorphizing the AI, but still, we’re all going to do that, so I might as well do it. So, is she going to be most helpful with competitive intelligence, market intelligence, you know, generating sales enablement kind of stuff? Like where do you see five years from now, you’ll say, okay, Erica’s no longer an intern, she’s a full-time digital employee that is super helpful in what?

Kelly: I think that she is most useful for market intelligence right now. I’m not really sure what to say about where she’s going to go. It’s going to be interesting to see. I think that right now, her strong suit is being able to take all of the information that she has available and kind of synthesize that, speed up the process for our researchers and give an answer that kind of synthesizes all that information. She’s not doing a lot of insight work. The technology behind Erica is that she uses RAG. So, she takes the information that we have, the market research, looks into it and brings back an answer. So, she’s not reading it and necessarily creating insights out of that. She’s just pulling out information that the analyst firms have provided us. So, in that sense, she’s providing insights, but they’re not her insights. So, we’ll see what happens if she learns and is able to create her own insights. That sounds a little scary to me, actually.

Patrick: It does. It does. It sounds a little scary to me as well.

Kelly: Yeah, I think it’s really important that our analysts still do their own reading and their own analysis. So, this tool can speed that up for sure, narrow down where they need to look for the insights that they’re gathering. But I think they still need to focus and make sure that they’re doing their own reading and analysis. I think that’s such a valuable resource that we have are the people inside our organization. And I don’t want AI to be seen as a magic tool that will take the place of all of these important minds that we have.

Patrick: Right.

Kelly: Yeah.

Patrick: Right. Yeah. And it’s, I mean, you need the- you need the trusted opinion, you need the contextual analysis. And if Erica can simply find it faster and bring it to the sort of end user quicker, that’s the skill set.

Kelly: Exactly, yeah.

Working with internal AI for the best results

Patrick: I’m curious, and this is going to get a little bit into the weeds on this, and if it’s too much, let me know. But knowing that Erica’s the set of knowledge that she can tap into, the contents of the large language model or small language model, however you want to phrase it, that she’s tapping into is a diverse set of opinions and analysis from a diverse set of analysts and analyst firms. Is there a way over time, I know you just launched it, so man, this is again, maybe too in the weeds and too early to ask, but is there going to be a way over time for you to say, okay, she typically, she’s relying more on these three or four analysts or these three or four firms and relying less, like sort of tier out and say, these are the top, these are the middle, these are the bottom. And if that happens, would there be a way to sort of say, to evaluate that? Is that good or bad? Is she tending to lean toward a particular set of firms because of X, or is it just, we’ll take whatever she- however she evolves in that way?

Kelly: I think that because we have such a vast variety of content. So not all of the firms that she’s looking at provide information on the same things, that she will continue to look at all of the different resources. I think that is an interesting question to look at to see is she depending too much on one source or another or a topic that maybe two firms cover. So, that would be an interesting thing to track. It’s not something that I’ve actually thought too much about, but I do think that just because we have a variety of information and we have people asking a broad range of questions on different technologies or different trends and things that are happening in the industry, that hopefully that won’t happen, but something to watch for.

Patrick: Absolutely. And maybe we can gather again in six months or so, have Elizabeth on with us and talk about where Erica is now. Has she- she’s no longer an intern, she’s now a full-time digital employee, and these are the things that she’s doing really well.

Kelly: Exactly. And that’s something too that we’ve asked our users to provide feedback for us on the tool, because they’re the ones that can point out to us, is there an error. She doesn’t hallucinate very much. Just, there have been a couple of things where she’ll give an error in her answer, as I think all AI LLMs have been shown to do. And so, we have to make sure that we’re finding out about those errors. And so, we have encouraged our users, hey, if there are errors or there’s an answer that’s not quite right, please share that with us. We want that feedback. Because Elizabeth and I use the system but not in the same way that somebody who’s doing research on a topic that maybe we’re not looking at is.

Patrick: Right. Yeah, it’s fascinating. And I think, from your own sort of career trajectory to go from coming in as a trained librarian to being an expert in how to wrangle knowledge management and make knowledge management an asset for the company to now how to make AI even more of an asset internally, it’s just fascinating.

Kelly: It’s a very interesting change for sure. But I think that the knowledge that I have from library science and that you learn all of the search techniques and Boolean logic and all of that, I think that will translate to prompt engineering and figuring out how do we ask the generative AI systems questions so that we get the best possible responses. And that’s something we’re also focusing on is training our users to ask questions in the best way. Because search is not necessarily a natural skill that everyone has. You know, they are used to going out to Google and just asking a question, but with AI, you have to be very specific. You may have to include dates and sources and things like that. So that’s another, hopefully, skill that I can bring to helping our users get the most out of our new AI assistant.

Patrick: Yeah, that’s, and that reminds me of like when GenAI first, when ChatGPT first like, sort of exploded a year and a half, two years ago now, one of the, sort of the ways I thought about it a lot was that we have an Alexa in our kitchen and there are still times when we ask Alexa a question and we get an answer that makes absolutely no sense at all. It’s like, yeah, there’s still a ways to go here with the AI. There’s still a lot of work to be done. And not just in the AI and the technology itself, but in the human component of asking the right question, understanding how to ask the right question.

Kelly: Exactly. Yeah, there’s challenges on both sides.

Final thoughts 

Patrick: Yeah, well good, as long as there’s challenges on both sides then neither of us are going away. So that’s all a good thing. Kelly, this was fantastic. Thank you so much. And I promise we are going to do this again in six/nine months and talk about everything that you all have learned over the last six/nine months with your new intern.

Kelly: That would be fantastic. Thanks so much for having me.

Patrick: Excellent. Thank you, Kelly. 

Tune in next week for another episode of TBR Talks. Don’t forget to send us your key intelligence questions on business strategy, ecosystems, and management consulting through the form in the show notes below. Visit TBRI.com to learn how we help tech companies large and small answer these questions with the research, data, and analysis my guests bring to this conversation every week. 

Once again, I’m your host, Patrick Heffernan, Principal Analyst at TBR. Thanks for joining us and see you next week.

TBR Talks: Decoding Strategies and Ecosystems of the Globe’s Top Tech Firms

Join TBR Principal Analyst Patrick Heffernan weekly for conversations on disruptions in the broader technology ecosystem and answers to key intelligence questions TBR analysts hear from executives and business unit leaders among top IT professional services firms, IT vendors, and telecom vendors and operators.

“TBR Talks” is available on all major podcast platforms. Subscribe today!