Lecture 16 - How to Run a User Interview (Emmett Shear)

all right uh good afternoon today's guest speaker is EMT Shear EMT is the CEO of twitch which was required by Amazon where he now works um and EMT is going to do a new format of class today uh and talk about how to do great user interviews so this is the talking to users part of starting a startup uh should be really useful thank you very much for coming thanks Contex so everyone knows uh where I'm coming from from this uh we started our I started my first startup with Justin KH uh right out of

college um we started this company called Kiko calendar uh it didn't go so well uh it went all right we we we built it we uh we sold it but we sold it on eBay uh so that's not necessarily the end you want for your startup and uh it was uh it was a good time we learned a lot we learned a lot about programming um we didn't know anything about calendars uh neither of us were users of calendars nor did we During the period of time we worked on Kiko talk to anyone who actually

did use a calendar uh so that was uh that was not optimal uh we we got the build stuff part of the uh startup down we did not get the talk to users part uh the second startup we started we used a very common trick that lets you get away with not talking to users which was that we were our own consumer we uh we had this idea for a television show Justin TV a reality show about Justin con's life and uh uh we built a whole set of technology and website around the reality show

we wanted to to run um and so we were the user for that uh for that product and that's actually one way to cheat and get away with not talking to many other users is if you're just building something that literally is just for you you don't need to talk to anyone else because you know what it is you want um and what you need uh but that's actually a really limiting way to start a startup most startups are not just built for the person who is uh who is using them and when you do

that every now and then you get really lucky and you are representative of some huge class of people who all want the exact same thing you do uh but very often also that just turns into a side project that doesn't go anywhere uh so we kept working on Justin TV for a while and we actually achieved a good deal of success because it turned out that there were people out there who wanted to do the same thing we did uh which was broadcast ourselves live on the internet but uh the issue with Justin TV the

thing that the thing that sort of kept us from achieving greatness is we hadn't figured out yet how to uh how to build towards anything beyond that initial TV show we knew how to BU we built a great product actually if you wanted to run a live 247 reality television show about your life we had the website for you we had exactly what you needed but if we wanted to go do more than that if we wanted to open it up to a broader spectrum of people a broader spectrum of use cases we didn't have

uh we didn't have the insight to figure that out because we weren't that user uh and so at some point we decided to Pivot Justin TV we decided we needed to go in a New Direction uh we thought we'd built a lot of valuable technology but we hadn't identified the use case that would let it get really big uh and there were two directions that seemed promising one of them was mobile and one of them was gaming uh and I led the gaming initiative inside of the company and what we did with gaming that was

very very different from what we'd ever done before was we actually went and talked to users because while I loved watching gaming video I was very aware that neither I nor anyone else in the company knew anything about broadcasting video games and so uh I was amped about the content I thought there was a market there that was sort of the Insight that uh that I had that I think wasn't common at the time which is how much fun it was to watch video games uh quick show of hands people know about watching video games

on the internet here okay I'm going just going to assume the people listening to this also know about it if you don't know about watching video games on the internet you should go read about that uh because uh it's sort of important context for the stuff I'm going to talk about but uh the main point is uh I thought that was awesome uh but I didn't know anything about the side of it that was really important which is actually acquiring the content uh to start broadcasting so we went out and we ran a actually a

very large number of user interviews we talked to uh a lot of people and brought that data back and that formed the core of all the decision- making that was for the next three years of product features on Twitch uh was sort many of the insights we got from that and we continued to talk to users and in fact built an entire part of the company whose job it is basically to talk to our users um which is a whole division that we just didn't even have it Jus in TV we had no one at

the company whose job it was to talk to our most important users um so uh so that was twitch and I'm going to I want to give you guys a little bit of a uh uh a little bit of an insight into uh with twitch what uh what that what that meant going to go talk to users so we determined that the broadcasters were the most important people um and the reason we determined that was when we went and looked into the market we I I we looked into what what determined why people watched a

certain uh streamer not went to a certain website they would just follow the content right you had a you had a piece of content you loved uh and the broadcaster would come with you um and that's actually the one really important point about user interviews which is that who you talk to is as important as what questions you ask and what you pull away from it because if you go and talk to a set of users if we'd gone and talk to viewers only we would have gotten a completely different set of feedback than talking

to the broadcasters and talking to the podcasters gave us insight into how to build something for them um and that turned out to be strategically correct uh I wish I could tell you the recipe for figuring out who the target user is uh for your product and who your target user should be but there isn't a recipe uh it comes down to think really hard and and use your use your judgment to figure out who you're really building this for um so uh what I want to do is a littleit something a little bit interactive

now which is uh we're I'm going to I got a bunch of ideas uh from from you guys actually so sort of suggested ideas um and I'm I'm G to pick one of them and I want everyone to sort of sit down and do do step one of this process for me right now which is think about who would you go ask about this like which people where would you go to find the people you needed to talk to about this uh in order to uh in order to learn about what you should build um

and so the idea we're going to use is let me see here of these ideas so here it's a lecture focus note-taking app the idea is I don't think that the state-of-the-art for note taking is good enough yet um and I want to make a note taking app that uh you know improves that experience makes taking notes in class better um or taking notes while listening to a lecture online better um so you know maybe it has collaboration features uh maybe it like helps you focus better somehow it has multimedia enhancements I don't know right

all sorts of possible features but that's the that's the idea so take like take 120 seconds right now and think about not what you would ask ask or what the right features for this app is but who would you talk to who should the who who going to give you that feedback that's going to tell you whether this is good or not I actually mean it right now take your laptop out like type write some stuff down think think about like the you can it's it's good enough to like think of that in your head

but actually like if you actually just write it down and like just come up with the five people you talk to the five types of people you'd talk to um and who you think the most important one was like actually do it because there's nothing like actually running through a practice of something and trying to do it to actually get it into your head the right way to do it um I'm gratified to hear clicking in the of keyboards now um if you're following along at home pause actually do it think about who you who

would you talk to um because uh that's a that is the first question uh for almost any startup that you need to answer is like who is my user and and where am I going to find them all right uh that's like way shorter than you normally used to think about this problem it's actually a really tricky problem and like figuring out where to Source the people is pretty hard but uh uh we're going to move along anyways in the in this highly abbreviated version of learning how to build a product and run a user

interview so um can uh can I can I get one volunteer from the audience to come up and uh tell tell us what uh who you would talk to and we'll talk about it you guys are all pre-selected here you go I don't know how to turn this thing on here we go so who do you talk to um I would definitely talk to college students first obviously because we sit in a lot of lectures and specifically I want to talk to college students studying different subjects to see if maybe um you know if you're

an English major if that makes a difference versus you're studying um math or computer science in terms of how you want to take notes in different lectures um and uh so you you go talk to talk to a bunch of college students would you pick any particular subset of college students like sounds like you want to talk to all college students or like a Brad I I want to talk to college students um like and break down the divisions by like people who study different areas maybe and then also maybe it would make sense for

people who have like different study techniques because some people take a lot of notes some people don't take that many notes but still jot stuff down right so I mean that that's a really good start like that's that is actually obviously a group of users you want to go talk to especially if you're targeting something at you know at college students uh as the consumer um and if you're talking to college students as a consumer uh the you're going to get a lot out of students about what their current notetaking habits are and you know

what they would be excited about um one of the problems with selling things to college students is that college students don't actually spend very much money um it's it's really hard to get you guys to open your wallets especially uh if you want them to pay for a school related thing I mean people don't even want to buy textbooks right uh I think you probably probably all use Che or that you know borrow it from your friend or whatever uh and so uh one of the like one of the things what I think you'd be

missing if if you go after just the students right is you want to figure out who who is the most important person uh to this to this app and if you actually had a not- taking app my guess is for colleges the people most likely to actually buy a Noe ticking app that you guys would use would be College it right I mean presumably for most for the most part if you want to sell software to students like the people who have to get bought into that is usually the school administrators so that would be

that would be one approach if you thought okay well you presumably go talk to the college students and you find out uh they don't actually buy any not taking software right now at all I mean likely uh it's possible they do um in which case I'm I'm completely wrong and this is why you actually have to go talk to the users but uh you then have to try to maybe try other other groups right so I would talk to college I would talk to it administrators as well I think that's a that's another uh area

it's really promising you might talk to parents right who who who spends money on their kids education it's like willing to pull their wallet out like the you know parents of kids um parents of kids who are freshman are going up to college for the first time you need this app to make your kid productive so they don't fail out of college um and and there's actually a lot of groups that are potential that aren't necessarily the obvious user but who are critical critical to your app success potentially um and you when you when you're

at the very beginning of a startup like this when you're like you have this idea that you think is awesome uh you want to have that broadest group you possibly can you don't just want to talk to one type of person and and learn that you want to get familiar with the space you want to get familiar with the various kinds of people who could be contributing all right so uh uh let's uh let's have someone come up and we're gonna we're going to pretend uh we're going to run this new interview so we're going

to talk to a college student um and try to find out uh what we should build you know what we should get uh into this note taking app so so some another volunteer please uh for for running an interview yes all right so uh hello hi I'm Stephanie hi Stephanie nice to meet you um welcome thank you for agreeing to do this user interview with us so uh I wanted to hear from you about you know what are your note- ticking Habits Like how do you take notes today sure so um I take notes in

a variety ways I like to um now because of speed and efficiency and just to come back to it later it's easy for me to just take notes on my laptop um and so a lot of those notes will be primarily text based um but in certain classes so for example if I'm taking a history class most of it will be in text but if I'm taking it Taking a physics class for example they going to be more complex diagrams different angles that I have to draw and so that's a little harder harder for me

to get what software do you use for this stuff today I just do pen and paper for that you do pen and paper so you do a combination you take notes with pen and paper you take notes with your computer sometimes um and uh when you take notes with uh when you take all these notes at the end like do you actually review them like do you be honest do you actually go back and ever actually look at these notes the pen and paper not so much but yes to the um software based because it

it's more easy to access and it's easier for me to to share and collaborate and maybe like even merge notes with classmates and friends so what do you uh what do you use take notes today on your computer um Google Docs um and Evernote Google Docs and Evernote and uh tell me more about like why two things at the same time um so evern note is easy if I'm trying to just collect it for myself I think um and yes you can share but I think Google docs for me is um easier to share and

it depends also if you know a friend has already created a folder for example on go and I just have to add to that folder if it's a group project for example versus if it's for my personal use I I tend to um go more toward so it sounds like you do a lot of like not- taking collaboration yeah I wish she was integrated what uh uh tell me more about that like like do you take a do you wind up taking most of the notes most of the value of the notes out of notes

other people take or is it mostly your own notes you review at the end of the semester how does that work it's mostly mine because I'm pretty picky about the way I like things organized um like design wise or formatting um even color I'm really particular with and like the font that we use and that really affects the way I study so um I tend to like it to like to like to personalize it even after I merge so you you pull in notes from other people but then you merge them into into the main

what works for me right um awesome and if you uh if you have Evernote notes and and you have uh Google Docs notes and you have pen and paper notes once the semester's over do you ever go back to any of that stuff or is it like a quarter you guys are in quarters here right once the quarter is over uh do you ever go back to any of that stuff do you ever um for classes not so much um but if it's notes that I've taken for like talks like these for example if it's

like interview prep that I'm doing um I tend to go back because it's things that I like to kind of keep fresh in my mind um and to help me prep for for future things so that's interesting tell me more about that like you take notes not just in class yeah um so I take notes to also just summarize main points so if it's like inspirational quotes for example from talks that I go to like these and then like maybe I'm going to an event where I'm actually going to meet someone and it it helps

to actually to think about and to remember and recall um what was shared at the time that you know I attended the talk or something awesome all right well normally I'd actually dig into a lot more detail uh there's a huge amount of like open questions that are still in my mind after hearing that stuff um questions about which people do you collaborate with questions about uh whether or not you like like what the volume of notes are and like how how long of of note taking stuff and just sort of digging into like what

the current behavior is but like in the interest of time and not like keeping everyone here hearing about the intricacies of one person's note taking habits forever uh we're going to move on but thank you very much Stephan appreciate that so uh so that that's like that kind of stuff you notice we're not talking about the actual content of the app at all like I'm not I'm not really interested in features I don't really want to know about uh what they the specific feature set in Google Docs or Evernote I might start digging in a

little bit more into which features actually get used like if she's actively collaborating you know is how does that work I heard some interesting things about oh we we use folders that's interesting to me uh but the main thing you're trying to do when you're running these first set of interviews is not necessarily get like questions about like user flows and like optimizing that or questions about uh like the specifics of of um of of any of that stuff uh kind of can be distracting because users think they know what they want but like you

you get the uh you get the Horseless Carriage effect where you're you're uh uh you're you're getting asked for a faster horse instead of trying to design the actual uh real solution to the problem if you start asking people about features you want to stay as far away from features as possible because the things they tell you want end up almost feeling overwhelmingly real um when you have a real user asking you for a feature it's it's very hard to say no to them uh because uh here's a real person who really has this problem

and they they're saying build me this feature but as you start to talk to lots of people and really get a sense for what what their problems are you figure out if this is actually a promising area or not and like based on what I heard there it's like starting from that user interview I'm not necessarily positive there is a problem or there's at least there's a there's a big enough problem that it's worth building a whole new product for uh because I didn't hear a lot of like things where I'm where where there was

a a big blocker where there's something really wrong with the way it the way it uh it was working and unless I had some big idea uh I would take that as a you know maybe a negative sign um but it doesn't necessarily mean that you can't uh you can't move forward and keep talking to more people because just because you talk to the first person you don't get anything out of it doesn't mean there's not going to be uh a ton more people who actually have a problem and you once you talk to about

six seven eight people uh you're usually about done uh it's unlikely you're going to discover a bunch of new information there um which is why it's important to talk to different extremes of people right go go find people who are at different uh different points because if you talk to six or six or seven Stanford college students you're going to get a very different response than if you talk to six or seven uh high school students or six to seven six to seven parents um all right once again look at the so um based on

that though right uh I think the I think it's possible you could come up with a set of ideas right you have this information about how someone takes notes you've you've come up with uh potentially when you came up with this idea you had you had some ideas as when you heard this idea you had some ideas as to like how you could build something cool um and so if you're going to build just one feature on top of Google docs uh what would that feature be right and that's for for for a new product

like this it might be a good way to like get started thinking about where to go which is uh okay they're extensively using this thing right now how could we make that experience just one Quantum better something that would be really exciting to this person to be one uh one step ahead and so want take two minutes right now and think about what that feature might be uh actually like try to try to come up with uh What uh what you might do based on what you heard from uh from Stephanie that could convince her

to switch or from her current collaborative multi-person all working together workflow on Google Docs to your new your new thing that is has all the features of Google docs plus this one special thing that's like going to make it uh that's going to make it more uh more useful and convince them to stop using the thing they're already using for awesome all right so I'm gonna invite our our third guest if you if you if you have something up uh I I don't want to put you on the spot if you feel like you don't

you're not sure but uh yeah so what I what I is it on yes okay what I thought about was like the the reason she uses evernot is because like of like sticky note type notes like like more thoughts and like details so I feel like GOOG Google dogs has like documents and not like smaller notes so I feel like a feature that would be like super like a mobile version of drive that doesn't like isn't that clunky and like doesn't make you make real documents could be like really useful awesome so right that's a

that's a good Insight right that's exactly that's one of the things you get out of that user interview and now you have this idea right you've gotten this uh this us your feedback you got this idea what if we had a uh Google Docs that had the collaborative aspects and the group aspects of that but where you uh you could pull in more little one-off notes and it was it was designed more around note taking and so the question is now once you have this idea which I think it's that it's a actually perfectly reasonable

approach is this enough is this something people would actually switch uh just to have and the way to validate there's two ways to validate that one is if you're uh quick at programming you can literally just go build it and throw it out in the world and see what happens uh and that's a that's great and if that when that works that's a that's an excellent way to approach it but a lot of the time that one little thing that's just a little bit better uh might take you 3 months to actually build something that's

worthy of actually using and so you actually want to go out and validate that idea further before you go ahead and start building it um and so you might take that idea and you might go back uh go back out and you know you can sit down with uh with diagrams you can you can draw what the uh what it looks like um draw the workflow and go bring that in front of people but uh the one thing you really don't want to do is ask them uh this is this is just a sort of

a trap I want to warn you against doing is just don't go out and say come up with the feature idea and go out and ask people are you I got this great idea for a feature are you excited about it because the the feedback you get from users if you tell them about a feature and ask them is this feature good is often oh yeah that's great like that sounds like such a good idea uh but when you actually take that in front of people uh and you actually build it you then find out

that well they thought it was such a clever idea no one actually like cares to switch to get it and so the one question you can't ask is is this feature actually good or not yes Sam what is the minimum that you can do in your experience to actually get real data on that question if asking you know between asking and actually building the whole thing yeah so Sam's asking if what's the what's the minimum you can actually uh get away with to validate given that you can't actually just go and ask them is this

good or not um and it's it's highly dependent uh the answer that's highly dependent on the particular feature but usually the the best thing you can do is uh is is really just hack something together right it's you find if you're if your idea is to build something on top of Google docs don't for your V1 go rebuild a an awesome Google Documents but for note taking application uh find a way to write a browser extension that that that stuffs just that little bit of incremental feature in and and see if it's actually useful for

people um go like act go go find a way to cheat is what it comes down to because if you can't actually put it in front of people um it's really really hard to uh to find that out for bigger things where you're actually trying to uh get people to spend money it actually gets a lot easier so if you're selling it uh it's great actually sales is this cure all for this problem get people to put give you their credit card and I guarantee you they're actually interested in the future uh it's it's one

of the most validating things you can do for a product is go out there and actually get them to commit to pay you upfront and the problem is when you're working on a student notetaking app that's going to be relatively hard uh because you probably unless your idea is that you're actually going to sell it it's probably something where you're thinking at least the the trial version is free and you're not necessarily going to learn uh that much by trying to charge people money but if you go out there and you if you can get

people to say hey I'm going to I'm going to give you money the money test is amazing it really does uh clarify whether or not they're really excited about about it or not because if you're not $5 excited about it you're probably not very excited about it um so the last thing I wanted to do is actually work through with you guys uh what happened at twitch um so I I brought some slides of feedback um that I'd like to get put up that's my my only slides for the uh for the thing and it's

it's what it is it's it's it's representative excerpts of twitch feedback I had a whole like 26 page document full of all the feedback and then I realized that reading that was going to be a little bit tedious and there was no way I'd make it through it in a lecture so pretend that like this is stuff is all representative of uh like lots of people said this kind of thing out to to us when we ask them questions and I've already pre condensed it for you into the real feedback you got so when we

were working on Twitch to go launch it we uh we went we talked to a bunch of existing Justin TV broadcasters and asked them about their experience broadcasting what they liked about broadcasting why they broadcasted what they broadcasted what else was going on in their life and the interesting thing is when you talk to users of your product who are who are detail users of your product they come back to you with actually very detail things about features because they actually get mired in the feature and you have to sort of read between the lines

but um they ask for us things like I want to be a way to clear the band list in my chat room like this that was actually a very common request uh because there was a particular issue with how our chat rooms worked people would ask for the ability to edit the uh titles of highlights after creating them and and it's it's this was like this stuff was was really consistent as you talked to broadcasters you probably talked to 12 14 something like that uh broadcasters on the Justin TV gaming platform um we got we

got all this feedback and you know what else do we have we have your your competitors have all these cool features like polls and scrolling text I can personalized chat there and uh uh we have some positive feedback they're like oh you guys don't have ads that's great um I need to be able to ban trolls there a bunch of stuff about chat a bunch of stuff around uh interactivity with with uh uh interactivity with the uh with their viewers and that was all really interesting right so this is what the this is what the

Justin TV broadcasters uh wanted us to build and this is what they what what where they felt pain using the using the product and uh so if you thought that what we did was go and address these problems you would be wrong because actually people who are using your service already and are willing to put up with all these issues kind of kind of means that these are probably not actually the biggest problems because if you're willing to ignore the fact that you can't edit the band list and titles are uneditable and there's no way

to get trolls out of your channel and you're using the service anyways maybe those aren't huge problems and so that sort of brings up another really important point which is you have to compare uh you to compare groups of people and compare the level at which they uh uh they argue with each other so if you go to the next slide um yes nice uh we got competitor broadcaster feedback which is really interesting so this is stuff that you we heard a lot from people who are using other broadcast platforms um they wanted to be

able to switch multiple people onto their Channel at the same time uh they they complained about us not having a REV share program um where they talked a lot about how they were trying to make a living they really wanted to make money uh pursuing this uh pursuing this gaming broadcasting thing they talked about a lot about video stability our service wasn't good in Europe um specifically but but just globally video stability was this huge huge issue for them um and if you compare and contrast actually it was really different like the things that people

who didn't use our service said about what they cared about was completely different than the things that people who were using the service cared about um and we focused on this stuff because this was the stuff where it was so bad they weren't even willing to use our service because of it um and most of them actually had thought about it because we were our user base happened to be a very uh well educated user base in this in the area who knew about all their options for for this and they would they you know

reaching out to them uh meant that they they'd probably already tried all four services and actually had an opinion it's great when you can get users who are that that informed and that that they understand the space that well um and uh if you go to the I'm just going to go to the next slide here we go um the other big thing we did that I thought was really important was we talked to non-b broadcasters um so we went out there and we talked to all the people who weren't using us or competitors um

and in many ways those are the most important people right talking to your competitors that's a short-term win right if someone's using a competing piece of competing software um unless your your piece of competing software or something like Google which is a search engine which everyone uses okay maybe then then there are no nonusers to convert but in the case of gaming broadcasting almost everyone's a non-user right the the majority and this is true for most new products the majority of people you're competing with are non-users um there there are people who have never used

your service before and what they say is actually the most important what they say is is the thing that blocks you from expanding from uh uh expanding the size of the market with your features right if you all you do is look at your competitors uh and yourself and all you do is talk to your your you know people who use your competitor's products people who use your products you can never expand not never but you're not learning the things that help you expand the size of the market you want to talk to the people

who aren't even trying to use one of these things yet who who have thought about it maybe but who aren't uh who aren't into it so what did they say um my computer isn't fast enough uh I'm focused on training 12 hours a day for the next tournament um I like making the perfect video and like editing it uh and so I just upload things to YouTube I don't do live streaming I don't I I uh I have no desire to to go into that space um or uh or this is actually particularly in Korea

this is a big problem uh once our strategy gets broadcast in a major tournament we have to start over we have to like come up with an entirely new strategy and so the last thing we ever want to do would be broadcast our practice sessions are you crazy uh that's going to hurt us in the next big tournament and so this became this became a big outreach program for us trying to figure out how we can get people over this and we bought people computers we uh uh we worked really closely with gaming broadcast uh

software companies to help the people who made the broadcasting software to make that better um we started building broadcasting into games and into platforms like we built broadcasting into the Xbox we built broadcasting the PlayStation 4 uh because we want need to overcome this issue that like it was too hard uh broadcasting wasn't uh wasn't possible and so you sort of combine these for us these are the three three big groups we looked at for broadcasting uh and you combine that feedback um and what it tells you is not the features to build right because

the features they asked for uh things like polls um things like a uh um you know the the ability to have child account like child accounts on your account we haven't built most of that stuff um but what was important were the were the the issues like the goals they were trying to accomplish there people wanted money people wanted stability and quality um people wanted universal access for viewers all around the world um to be able to watch them and so that became our Focus actually and we dumped almost all of our resources into things

that uh none of no one ever mentioned uh in an interview but those were the things that actually addressed the problem and the way you could tell that it worked is as we we would build these things then we'd go back to the exact same people we interviewed and we'd say Hey you told us you really cared a lot about making money well we built you the subscription program that will let you make money and uh it it's it's astonishing cuz most most people aren't have never had that experience actually they've never talked to someone

and uh said it would be really great if your product had feature X and then and then like two months later or a month later your product actually has feature X um or at the very least a feature that addresses the problem that they brought up and so uh it was actually the the people we converted first to our product were the people that we talk to about user research they were the ones who were actually the most impressed which is kind of fun uh but it really worked because those we picked people who were

representative and we picked big broadcasters small ones medium ones and we we made sure we were addressing their concerns and that that was completely different from how we'd approached the problem on Justin TV because in Justin TV when we tried to do this we' we' sat down we we trolled through huge amounts of data like we we spent tons of time looking at Google analytics looking at mix panel looking at in-house analytics tools figuring out how people use the service looking at where our traffic came from uh completion rates on flows we spent all this

time doing that and that that's good I mean you can learn things from that I'm not telling you not to look at your uh at your data but uh it doesn't tell you where you need to go it doesn't tell you where what the problems are you need to address um and so we would just sort of invent these ideas in Justin TV and then nine times out of 10 without talking to someone that idea turns out to be bad um and that's actually one of the most disappointing things about doing user interviews and user

feedback which is why I think so many people don't do it which is that you're G to get Negative news about your your favorite pet feature most of the time like you're gonna have this great idea and you're going to talk to a user and it's going to turn out that uh that nobody actually wants that like no no one's actually they're actually completely concerned about completely different things and they don't care about what you thought was important at all and uh and that's a little bit sad but just just think about how sad you'd

be in four months uh when you launch that feature and it turns out no one actually wants to use it so um I think that's about it for my the lecture section of what we're we're talking about I want to take some questions from from the audience what do you see startups get most wrong about doing interview like most startups don't do them at all but the ones that do what are the most common mistakes um that's say the the most common mistakes are showing people your product um don't don't show them your product it's

it's sort of like telling them about a feuture um you want to learn about what's already in their heads you want to avoid putting things there the other thing is uh asking about your your pet feature Direction so if you think you want to add add subscriptions to your product uh going and asking people would you pay for a subscription going and asking them would you use this feature um and I'd say the uh the other big mistake people make is talking to who's available rather than talking to who they need to talk to there's

certain users that are really easy to get at because they are uh say members of your uh Forum already right you have you have some product forum and you go and you you talk to the users on that Forum because they're they're easy to get access to um we we spent like weeks digging for identity information and figuring out who these people were uh so we could contact them so we could talk to them uh because a lot of these people weren't it wasn't obvious they were just some user on a on a site and

that site didn't support messaging there was like no obvious way to interact with them um and so we spent a bunch of time trying to Network and and find those users and bring them on because if you if you just talk to who's easy to talk to you're not really getting uh uh getting the best data the the fortunate side there is that almost everyone is flattered to be asked what they think uh and so uh most of them will actually talk to you and tell you things yeah how hard was it to get Buy

in from the rest of your company I mean like you can go and be like whatever I'm in charge so you're doing what I say but that's probably not the best way of doing it so how did you get them to that's a good question so the the question is uh how hard is it to get Buy in from the rest of the company and how do you do it um getting Buy in if you just go to them and say I figured I talked to the users I figured it out uh we have to

build this uh is really hard um because people don't trust you uh there's something magic about showing the interview though so I really recommend you record interviews um recording interviews is like magic a it stops you from taking notes in the middle um and taking notes is a little bit disruptive it makes it hard for you to feel like you're actually engaged in the conversation um and B you can then play that recording for people so when they don't have to be there for the entirety of all the interviews but when you want to make

a point about what what what we should be building and why you can just play back for the rest of the company that interview and it's like magic the influence it has on people's uh thoughts and what's uh what the right thing to build is yes so did you um since you mentioned recording did you uh try to insist on doing Skype interviews rather than over email or what was your impression so you definitely want to do Skype uh or sorry the question was um did we insist on Skype interviews for recording um you don't

want to do interviews over email if you can avoid it uh because interviews over email are non interactive and the most interesting things you learn in interviews come from from the interesting tell me more uh cuz the instant you you H this vein of they'll say something you didn't expect and the instant they say something you didn't expect or didn't already know you should drop into detective mode and detective mode is huh that's interesting can you tell me more about that um people don't like silence so they'll keep talking to fill the void uh and

the best part about doing it over Skype or doing it in person is you have that interactive uh feedback and you can actually pull a lot more out of people email interviews are they're okay but they're they're B basically useless if you're in person over Skype they're actually also easy to record um make sure you ask them if it's okay to record it uh it's not polite to record people without their consent but if they're willing to like give you an user interview they're probably willing for you to record it as well sorry but what

about the international market like you mentioned it that you had a lot of USS in Korea and I don't know like maybe they didn't feel comfortable speaking English or yeah um so uh the question is like what about people in the international market where you're trying to do user interviews with people who don't speak your language that's just really hard and actually to this day twitch works way better in English speaking countries than it does in non-english speaking countries and I think a big part of that is we are much better at talking to people

in English speaking countries and learning what their needs are and we're not as good at it in other countries we've tried to address that by hiring people who speak Korean um and having them translate we've tried to address that by uh finding representative people in those countries who speak both English and Korean and reaching out to them but the problem with that is like uh the you're not actually getting a representative sample no matter how hard you try the very fact that they are a fluent English speaker means they're not representative of all the people

who don't speak fluent English um it's just a hard problem uh it's why companies find it easier to W build markets that win in their home in their home country uh much more easily than abroad uh because it's really hard to talk to users abroad yes um what channels you use to reach out to them and you ever compensate that um so the channels we use to re what channels do we use to reach out to them did we ever compensate them the channels used to reach out to them uh were uh on-site messaging systems

so like if your most site websites have some way of contact a user so if they're a visible user of another website you use that site's messaging system and say hey I was watching your stream or whatever this person was doing on the site uh I'd love to ask you some questions about uh your use um would you mind hopping on a Skype call and as for the other thing we do is we'd find out who people's were and we'd send them email um we'd like run into them at event events cuz a lot of

these people go to the same events and we like would go to the events and like get we wouldn't run the user interview at the event but you get to know them you exchange business cards or you know whatever it is you actually do now that aren't isn't business cards and uh uh and you youd get in touch with them we tended not to compensate people uh I think that if you if no if people don't care enough about the problem to like talk to someone who's trying to solve it uh you're probably barking up

the wrong tree we never had any trouble getting people to talk to us without paying them what about user feedback tools do you get feedback from that so so there's this whole second set of user feedback that's really important um that I should talk about um the question was what about like on-site user feedback tools um and I think the stuff you're talking about is where you you have like a uh a new product and you want to see how if it's actually going to work or not and so you put it in front of

people and you see how they use it or not that's really important that kind of work is super super important and it can tell you lots of things about where you went wrong building something before you launch it uh which is great it doesn't tell you what to build it it helps you iron out the The Kinks and edges of the thing you did build but generally speaking we uh that wasn't the kind of user feedback we were getting I mean that stuff's good it's good it's like uh it's much more similar though to the

to the data driven approach right you're finding out why are people dropping off in this flow you're not finding out what problem should I really be solving for them and what what are they care about as a human and for this kind of like really early stage user interview which is the kind of user interview that's crucial startups do uh that's the that's where you want to focus so we didn't bring anyone on site actually it was almost all over phone or Skype yes so for the three different groups of people they have different kinds

of feedback so as you start you have limit of time and resources is there good you focus on first yeah so with the three different kinds of people uh did we focus on one of them uh given that we had very limited resources yes uh we focused on the competing uh people using competing products because we knew that they already were interested in the behavior that we needed and uh they were willing to do it at all and therefore all we had to do is convin them the switch which is a much easier thing to

do than to try to create a new Behavior where none existed before um and we had to do that because we had to get some quick wins because my gaming project inside of Justin TV would have been killed if it wasn't showing 25% month-of month growth every single month so uh we did and that meant focusing on shortterm get the people in right now and that turned out to be good in general because uh it turns out that building something that some people want generally generalizes uh and so I want to bring in people who

weren't even users of the service as well yes switch has been around from the beginning to like build up for example the video game industry and in the beginning this industry was very like decentralized like there wasn't a lot of cohes with like you know different video game companies consolidating where tournaments are and stuff but now that's very different so you said originally you spoke to like broadcasters and um you know streamers El how has that changed when like for example like Riot has you know banned users from or professional players from streaming their own

stuff you tried to you know gain leverage with that or yeah so the question is what about the game Publishers basically right the game Publishers are these huge important people in the space um a the game Publishers and any big company for that matter isn't going to give you the time of day as a small startup um which is both good and bad uh it means you don't really need need to talk to them because they're uh they're not interested in you but it means you actually just can't talk to them I mean we tried

but no one wanted to talk to us and uh uh they did once we started getting some traction and and becoming a little bit slightly bit of a player in the space I don't want to like talk that bad about them because they they were nice about enough about it it's just that you know when you're when you're a tiny little startup there's lots of tiny little startups and they they don't have the time to talk to all of you um as we've gotten bigger actually uh the point that you know game Publishers have become

increasingly important uh consistency for us and if I was to talk about who twitch does user interviews with now who we uh who we pulled information from now uh it would include game Publishers definitely uh because they be they've become much more active in the space it was something that they weren't uh particularly active three or four years ago as much as they are now and uh uh that's another really important point about user interviews in general which is that the pool of people you care about is going to shift over time uh the people

who get you started like The crucial people to get your product started for the first six months are not who will be using it 3 years later and it's very important you keep doing this stuff because one of the thing it's really easy to do is do a little bit B of it in the beginning and and achieve some level of success and then just sort of stop talking to new people um and that's a good way to make the the next set of features you build be not as good as the first ones how

about one more question yeah yes how do you give good user feedback if you're a user um so how do you give good user feedback it's really good question so uh I think what I what I want a user to do is I want a user to tell me about uh what they like what they're really thinking right and what what what their problems really are and to just sort of ramble like I want someone to like just tell me about stuff in their life because the the more you learn about them as a person

and sort of that the their what's going on in the context of what they're doing uh the easier it is to understand why they want the things they want and that's really the critical question so I'd say like you know what I'm looking for in a in someone when I'm doing a user interview is someone who's going to uh be willing to talk a lot and be willing to to really give me a full give me a full picture um and so that's what I guess on the flip side if you want to be a

good if you want to help people out with good user interview feedback uh uh ramble like be just just talk about stuff uh and everything all right great well thank you very much thank you very much