Building an Engineering Team by Ammon Bartram and Harj Taggar

as a slides of loading there there is no topic that should occupy your minds more as you build your company then bring it on the team that's going to make your company successful as you move forward Hajin amin from triple byte YC alumnus are going to talk about building in today's day and age perhaps the key portion of that team the engineering team so please welcome harsh you're starting yeah harsh alright thanks for having us everyone so I'm Hodge I'm one of the cofounders of trip I along with Arman previously I used to be a

partner at Y Combinator and part of the inspiration of sighing triple buy it was noticing how after graduating y Combinator and raising the first investment around everyone's number one problem was hiring and specifically hiring engineers because the hardest hiring challenge so three working on to a bite which is a hiring marketplace that's used by engineers to find new places to work Armen and I have gathered lots of data on what works well when it comes to hiring engineers I've personally focused a lot on spending time with companies helping them think through their strategies for finding

engineers and obviously getting the most out of triple buy it and Arman spent a lot of time thinking through the details of how do you evaluate an engineer how to evaluate the engineering skill and answer the question of are they a good engineer or not so we're going to share that and sort of have a divide and conquer strategy going on here the four main topics we're going to cover are where to look for engineers and when you should start thinking about using recruiters and that's what I'm going to start with then we'll talk about

evaluating technical skills which are Mons gonna cover and then I'll finish up by talking through the process of making offers and closing which is getting people to actually join your team but before we start on any of that I want to just issue a warning and make sure you're well prepared for the fact that hiring really well and truly sucks it's like incredibly painful process because for many reasons which I'll describe in detail the first is it takes a lot of time like it takes a lot of time just to convince someone good to eat

and have a conversation with you and as a founder as you know time is a very scarce resource there's bugs to fix there's sales customers to close various things going on and hiring will never feel like it's your top most urgent priority and it's very easy to procrastinate and push it back but if you do that for too long you won't scale and you won't grow your startup and someone else will come along do that and take the market to hiring involves a lot of repetitive work so actually as Tyler was giving his presentation I

was talking to Jeff about this back there there's a lot of similarities between sales and hiring and actually there's a lot of similarities between sales and hiring and fundraising and buy lots of things that you do as a start-up founder large part of it is effectively selling all of the time and selling as Tyler pointed out involves a lot of repetitive work this so hiring will involve lots of messaging and revolve taking lots of coffee meetings lots of phone calls lots of interviews and most of those will result in the dead end and be a

complete waste of your time but you have to keep going and finally you will get your heart broken you will inevitably end up getting rejected by people you really wanted to hire who would have been the perfect fit to help you hit your growth goals but it turns out they were never really that serious about leaving their comfortable job at a big company to join your exciting but risky startup so be prepared for all of this and as you're thinking through building a hiring process I encourage you to think about this as a funnel that

you're creating that has three parts to it the top of the funnel is sourcing and that's finding people who could be good fit the second is screening that's answering the question of do you want to hire this person or not and the final part of the funnel is closing making the offer and getting the offer accepted I'm going to start by talking through some strategies for building the top of your hiring funnel these are the five places I'd recommend that you look for making engineering hiring for making engineer hires I'll talk through the pros and

cons of each of these they are personal networks hiring marketplaces LinkedIn github job boards and meetups I'll talk about how you can the most out of each of these and this list is ranked or it's sorted in order of where I think you should start out focusing most of your attention and energy down to where you focus the least so we start with personal networks in my opinion personal networks are the best place to hire especially when you're early and making your first few hires by far and the reason is anytime you're deciding if you

want to hire someone you're essentially asking yourself two questions one does this person have the skills that you that you need for them to do the job and two can you personally work effectively with this person when you're a big company you can mostly focus on answering just the first question because they're your large enough there's enough people there's enough teams that it's likely that someone one team somewhere will be able to work effectively with anyone but when you're small that's not true whether you can work effectively with someone or not is a big determinant

of your success and if you hire the wrong person early on that could literally be fatal so when you hire someone that you've worked with previously or you hire someone that's worked with the person you trust you do risk the chances that you won't be able to work well together which is a big thing to consider early on so that probably sounds like somewhat obvious advice and yet I'm surprised by how often founders still don't really use their personal networks effectively when they're hiring and I think there's two reasons for this the first is they

don't use a process to exhaustively search through everyone that could potentially hire and the second is they don't actually make the ask that's usually comes down to being afraid of being rejected by your friends it's somewhat easier actually to be rejected by a complete stranger than it is to ask your best friend to come join you and they say I'm not sure the idea is that great there's there's also like you can also worry about what happens to the friendship of the startup doesn't work out like it's the sort of more that goes into it

when you're talking someone you know personally that when you're talking to someone you don't but the truth is you just have to sort of suck it up and do it but you won your startup to be successful hiring from people you know it's a tremendously valuable resource and you just have to make that ask so I'd recommend you follow a strict process here start with just making a list of every good engineer you know whether you think they're available or not like that's actually irrelevant doesn't matter if they just sold their company for a billion

dollars put them on the list then ping each and every one person on that list to meet up and commit to asking them if they would join if you have a crazy you think it is however unlikely commit to making the ask if they say no they're hesitant so deviously a little B and say will you at least come by the office and see what we're working on and if the office is your apartment that's totally fine to you but just keeps pushing until you've at least shown them something that you've done keep working on

convincing them if it doesn't work out if they say no and you get a definitive no thence then ask them if they were in your position who would they try and hire make a list and go out and repeat this exact same process with them and this process just never ends like I know public company startup founders know long startups pod like probably company founders who kids still do this on a daily basis like this is just like a key thing you have to embed in yourself as a start-up founder as your company does scale

and grow and the tea and you start putting two team together you want to start tapping into the personal networks of your team and the way I'd recommend doing this is team events where people brainstorm potential hires and this is commonly referred to as a sourcing party the way I had to recommend going about this is get everyone together send out a shared spreadsheet and describe the role that you're hiring for so if it's an engineering role sort of described in detail like who you're looking for who are examples candidates what are skills and qualities

you're you'd be excited about and then literally have everyone spend 30 to 45 minutes going through their LinkedIn or their Facebook or whatever right there and then thinking of everyone that could be a fit and putting them into the spreadsheet once that's done a trilobite I'll actually then personally follow up with anyone on that list who seems like a good candidate or not and we've made several really great hires doing this it works works really really well and you can kind of make it like a fun thing to write so we'll do it at the

end of the week just before our Friday all-hands food and drink and you can also offer referral bonuses to your team to incentivize them to do this so really make sure that you use the great you're sticking to an exhaustive process you're making the ask of people you know and then as you scale tap into the personal networks of your team once you sure that you've exhausted your personal network for leads the next place I'd start looking would be hiring marketplaces hiring marketplaces are actually relatively new they've become more popular over the last few years

as has become harder to hire engineers by using traditional methods like reaching out on LinkedIn or github and I'm gonna talk more about that next the way I think about hiring marketplaces is actually work a lot like dating sites the idea is there's engineers who create profiles companies that create profiles and both are advertising their best selves you message each other and you figure out if it's worth meet up in person and you know if it all works out you make a hire the dynamics the marketplaces those such that the demand for good engineering talent

far exceeds supply and so typically it's the companies that are being a lot more proactive in terms of reaching out first to the candidates the candidates are getting multiple of inquiries and the candidates or the engineers are the ones that are choosing who they want to speak to and who they don't a big benefit of using a marketplace especially in the early stages is that they can help you hire very quickly because everyone most candidates are on the marketplaces are actively looking for a place to move right now it's very quick to get on a

phone call with them and start pitching and if you run a good closing process you can significantly reduce the amount of time you'll spend as a founder on hiring which is obviously great the downsides though are that they tend to be quite competitive so engineers are being reached out to do by multiple companies at the same time so you'll have to be very effective at convince them to join if you want to make hires and the second is that they can be expensive so most marketplaces will work on a fee per hire basis which can

be 15 to 20 percent of the first year salary that's cheaper than a recruiting agency but still a significant cost if you're in early-stage startup I'm obviously biased Aircast requires a hiring marketplace but I'd say like the three main ones that come up in conversation at least when we're pitching customers we be trilobite higher than Vetri you should try they're all free to use to get started so you're welcome to try I'd say provide the way we differentiate ourselves is essentially by having better candidates and we measure that by what percentage of candidates that companies

interview through to provide do they make an offer to you and that tends to be twice the rate that they make through other sources and just as a general note hiring as a funnel you're optimizing your funnel so you should pay attention to what percentage of candidates are making it through each step in your funnel when you're early you won't have that many candidates so you can't be that scientific about it but start sort of capturing that data and just building that into the habits you have when you're thinking about hiring the third source that

I recommend you go to and Mehcad is LinkedIn and github and these are effectively the biggest online directory of engineers in the world and most of hiring that's done at big companies is through teams of technical recruiters reaching out to engineers on LinkedIn or github finding the ones that fit sort of certain keyword criteria and sudden cold messages the reason they and they go for a very high volume approach here so technical Kreutzer may well sound over 100 messages a day just in the hope that they get a few replies and a dynamic that occurred

especially over the last few years is there's more technical recruiters there's more messages going out on these platforms so response rates are dropping for everyone which means for early-stage startups in particular it's going to require a lot of your time sending a lot of messages in order to get a few interested candidates so making this work for you requires in my opinion not playing to a high volume approach like a big company recruiting team word and instead spending the time actually researching the reading the details of profiles or reading through someone's Linkedin looking through their

github looking at the details of the work that they've done and sending a smaller number of personalized targeted messages and emphasizing when you send the message why would someone be a good fit for your company specifically and give them clear evidence that you've read their profile and you're interested in them as an individual as opposed to sending a spam message but that doesn't mean the message has to be super long I still advise keeping it short and concise just the key fact is there's proof that you've read their profile final note here send emails instead

of messages so if you sign up for LinkedIn recruiter light which is about $120 a month that will give you access to connect to fire it's a chrome plug-in that makes it really easy to pull out email addresses from anyone's LinkedIn profile and you'll consistently see much higher response rates through emails and you want messages so definitely definitely go for that okay the fourth place I try looking for engineers would be job postings or job boards these the two main ones for startup so stack overflow jobs and AngelList I haven't included I can use jobs

on there because it's only available to YC companies but how can use jobs is somewhat unique and just it has a particularly high quality of engineer and I rank it second on this list actually if it were sort of stand-alone source of Engineers job board in general do suffer from a quantity of a quality problem so what's good about them is you don't have to spend a huge amount of time posting to one of them the downside though is that the time suck can come in later because most of the applicants you'll get will be

vastly under qualified and you'll get a lot of applications from people who aren't even really software engineers and it will take a lot of your time reading through all of these resumes and applications to find the one or two good applicants so to maximize your return and maximize the number of good applicants you do get I'd recommend focusing on making your job listings unique and interesting and bear in mind that the majority of job descriptions on the internet are written by someone in a recruiting or marketing department that's using corporate boilerplate language that isn't going

to especially using an appeal to an engineering audience right so as a start-up founder you can serve experiment to try and bring through a bit of your personality in the job listing so one thing you might try is writing the first person about the personal story for why you started the company why you're excited about the mission and making it seem like you get that sort of excitement and passion across other things you could try is there anything unique about the culture is there anything specific about the technical challenges or product challenges you're facing so

if you put in more details that again will stand out because big companies tend to be very generic and vague when they're talking about what you actually get to work on they're the final source I talk about is actually just like physically in person meet ups so I don't think that these are actually going to be very effective and they're sort of a long shot like I just the numbers don't really work out like in-person meetups don't have that large of a number of people in attendance and truth is most of the time people are

there for free food and drink more than trying to actively find somewhere to work so it's unlikely that you're gonna find both a really qualified candidate who's actively looking to move who's excited about your company and you also personally need to be very effective at talking to strangers and convincing them of things it all for this to work at all right so I've accrued it on there because I do know startup founders who have had success through meetups but they are few and far between if you do try this approach I'd recommend focusing on technical

meetups and by that I mean the local closure programming group where people come together and meet up and bring laptops and work on things is more likely a better source of engineers than going to Dreamforce the final thing you could try here is hosting meetups at your own your own office right so you could also combine this with the personal network hiring and use as an excuse to have friends come by the office or have their friends who are engineers come by the office at a Tribble bite for example one of our engineers is a

fanatical Emacs user and he hosts the Bay Area Emacs meets up at our office and it's sort of a it's a it hasn't worked for us for hiring but it is a good way to just meet and build a network of good engineers that could come invaluable at some point in the future so it's worth the definite worth considering okay now I'm going to talk about when you should think about using recruiters truth is there's there's not really any hard rule on when you should hire a technical recruiter I've seen companies of less than ten

people hire one I've seen companies wait until they're 50 people on plus it's sort of I treat these are my opinions or when you should and I treat these as a rule of thumb right so first I think you absolutely should wait until you've at least you're first engineer before you consider bring you on a recruiter in any capacity the reason for that is actually I think general startup advice applies here so one is just in general when you're hiring it's good for you to do the job yourself for a bit so you feel the

pain and you understand it and you sort of understand the details of what makes someone good at that role at your startup in particular before you go and hire someone you'll be better able to assess them so if you feel the pain of recruiting yourself I think you have a better shot of hiring the right recruiter for you and the second the second reason I say is just it's it's like sales like it's actually good for you to go out and try and pitch and convince people to join your startup so you can just understand

yourself like what message works and what doesn't because as a startup founder you are always selling and you never know when you might bump into someone who could be a really great hire and if you've already sort of practiced the pitch for convincing engineers to join you and you'll have those ready to go so I'd recommend making sure you always make your first hire before you try and delegate this to a technical recruiter second I'd expect to have a good hiring cadence so somewhere around hiring an engineer a month for the next six months or

so and before you start bringing on a recruiter otherwise it's likely they'll run out of work to do a quickly and then finally as a rule firm if you're spending more than 50% of your time sourcing that's like all the things I mentioned before and doing initial phone calls and screens and getting people to come and meet you in person over 50 percent is probably about the time where you want to start thinking about bringing on help because 50% is about the amount of time you want to be spending on hiring so recruiters themselves come

in roughly three types you have contract recruiters who you pay by the hour and they can do anything from just : messaging on LinkedIn all the way to doing initial phone screens this in-house recruiters it's just hiring a full time technical recruiter and that works as a member of your team and the final would be agency and agencies are essentially teams of salespeople that are paying lots of engineers on LinkedIn or wherever they can and then sending their resumes out to as many companies as they as possible and they tend to charge 25 to 30

percent or first year salary if you hire an engineers maybe they sent you they do tend to be fairly high touch so they're quite involved in trying to give you information that will help you close an engineer but they're also sending that information to multiple companies at the same time so my recommendations here would be when you get to the point where you feel it's time to bring on help or bring on a recruiter start with a contract recruiter and have them focus on just the sourcing piece of it so have them just focus on

reaching out to engineers on LinkedIn and github and their main deliverable for you should be filling your calendar with calls with promising candidates then you're doing the pitching and practicing or yielding the pitching and convincing when you get to a point where that becomes too much work for just you then I'd consider hiring a full-time in-house recruiter and training them to do the pitch cool so now they're both sourcing and doing the initial calls and sort of setting up the on-site interview process for you so to sum up most of startup hiring plan if I

were just getting going and building an engineering team would be start by making sure you've exhausted your personal Network spend lots of time taking people out for lunch coffee making the ask to you experiment with the hiring marketplaces your mileage will vary on these depending on how effectively you can pitch your company but even if you don't make hires from them you'll get valuable experience pitching real engineers and real candidates and learning what resonates about your company to that audience third spend some amount of time doing personalized and targeted outreach to engineers on LinkedIn and

github making those messages really personalized and finally treat job boards and meetups into meeting people in person as a background process you run where you're not expecting to make any highs from them but you're sort of building a general pipeline that could be valuable in the future cool that's the first part of this almonds are now going to talk about how to screen and evaluate a technical school of engineers then I'll jump back in a wrap-up with making offers and closing awesome thank you hard so I'm Aiman I'm hard as co-founder of a triple byte

and before this I started Socialcam with Michael Seibel and I was an early employee at twitch and I'm gonna talk about just the screening step so how to identify skilled engineers at your company so the first question here is just you know why you should believe me about any of this and one answer is that I've done a lot of interviews so since starting trilobite I've interviewed over a thousand engineers personally I but I think a better answer is that you know trilobite has a pretty special vantage point we're able to you know see how

the views do see how candidates rather do you know in the interviews at multiple different companies are we can see how the same candidate performs in multiple companies and that gives us a data set that I think no one else has and you know that data is is you know where the advice I'm gonna give today comes from before I jump in I want to just go over the the basic hiring process that most companies use and this is actually pretty standardized so probably 95% of tech companies use these basic steps to screen campaigns so

first is a resume screen someone applies to a company they send in a resume any recruiter looks over the resume and decides if this looks like someone who's a basically good fit then a recruiter call so this is a usually a 30-minute phone call with the recruiter I just ask about the candidate background judge culture fit and make sure they're interested in the company then a technical phone screen and so this would be between 30 minutes and an hour with an engineer usually solving a single programming problem so this is sometimes something like fizzbuzz or

a little bit harder often this point done over over a synchronized text pad of some sort then this is optional sometimes I take home project so a substantial project the candidate completes on their own time and then you know sends in to the company to be evaluated then finally the on-site interview so it can they comes into the office and does between three and six one-hour sessions with engineers at the company I you know covering covering individual problems and then finally a decision meeting so you so usually the next day after the candidate has gone

home everyone who interviewed the candidate and the hiring manager gets together in a room and talks about sort of their perceptions and make you know they they as a group make a hire no hire decision some stats on this companies make offers to between two and eight percent of all of the engineers who apply but interestingly exactly where in the process that drop-off happens is is very different between companies so we so we see companies where you know seven five percent of people who apply get screened out you know at the first step on a

culture fit call and then we see companies where almost all applicants make it through to the final interview and that's where all this reading happens and about ninety five percent of people who are hired work out that is about five percent of technical hires result in someone being being fired within a few months and then from the candidates side what we see is a distribution of success in interviews so the the the top you know few percentage points of programmers by skill receive job offers after most of their interviews but then the bulk of programmers

are somewhere in the middle where they receive you know the you know they do interviews and they receive offers after somewhere between you know fifteen to thirty percent of the interviewers they do and an interesting point is that no one passes all their interviews so there are not magical engineers who receive offers after every interview they do and yeah this gets at what I think is the the the major challenge when designing any of you process and this is inconsistency so this is noise in the interview process it's kind of the idea you know is

your interview fundamentally repeatable and meaningful you know if you had to if you could somehow you know reading to view your co-workers your employees or everyone you who passed review in the last year if you could be any of those people how many of them would pass again and it's a pretty scary question but kind of interesting thing is that we've able to add some data on this that's what I did was I calculated a stat called the inter-rater reliability between all of the interviewers at all of the companies on trilobite and so what that

means is this this is the statistical measure of the extent to which different interviewers tend to agree about which candidates are best and it's on a scale of zero to one where zero would be no agreement or the amount of agreement you would expect to see you know in random data from chance alone and one would be perfect agreement and what I found was an agreement of just over 0.1 so so the first point is that that's obviously much closer to to no agreement than it is to perfect agreement but for some context than that

I calculated the same stat on a data set of online movie reviews and what I got was a agreement of very similar but it's actually slightly higher so it ends up that you know interviewers agree about which engineers are the best at about the same rate that Netflix viewers agree about which movies are best and that's yeah that's scary I interviews are fundamentally noisy and they are they are more noisy than the data shows them to be more noisy than most hiring managers sort of want to believe so I know the first the first question

here is sort of why do invested all that infinitives are noisy why do them at all why can't we just use something like trial periods atashi engineers and yeah I think this is actually a great idea it is almost certainly much more you know if you can work with someone for a week you can almost certainly get a much sense of if they're a good employee then and then you could in in a you know if we are interview with them the problem is that most engineers don't actually want to do trial periods so we

did some research on their sexual bite and it ends up that only 20% of engineers are willing to to do trial periods and there's actually some there's some adverse selection there so the you know most of the best programmers are in the 80% who would prefer to do a standard technical interview because it takes less their time and is faster so I think that trial employment is an excellent option and you can offer that as an option but you know if you don't want to scare away most good programmers you do have to still run

a standard traditional interview process so we're gonna talk about sort of today is you know specific pieces of advice that you can use to try to reduce the noise in a traditional interview and I'm gonna go over seven points so point one the first way to reduce the noise in an interview is to decide what skills matter for your company there are a lot of different ways that a programmer can be skilled so you know someone for example can be very productive or they can be you slow careful write great tests and and make sure

they don't commit bugs you know someone can be strong in math and computer science or they could be you know deep knowledge about the internals of the Linux kernel and and you know scheduling and real-time operating systems or something and if you don't decide as a founder which of those skills matter for your company then your interviewers are going to decide that for you so they're going to come up with questions that they asked you in the interview process and they will fail people who answer poorly on those questions whether or not those are areas

that that that should matter for your company and this is actually a major source of noise in interviews every you know engineer has this bias where they think the things that they know the best are the most important skills one could know an absent specific direction from above about what to look for they will fail people for areas that are not important for your company so my first piece of advice here is that you should go through and ask yourself these questions before you start hiring us the first is I need to clarify these are

these are the axes along which we see most disagreement between interviewers the first question is you know do you need sort of fast iterative programmers or someone who's careful and rigorous do you want to look hire someone who's motivated by solving hard tech problems or someone who is motivated by building you know beautiful products for users I important that someone comes in with skill in a particular technology pick your language say or can you hide you want hire a smart person and let them learn your tech stack on the job is academic computer science or

algorithm ability something which is important for you or is that an irrelevant skill and then is there any other sort of specific expertise that you need in people you're hiring it's actually so you know it's fine to answer both some of these questions you don't have to specify only a single profile of person you're hiring the important thing is to decide what matters even if that's multiple profiles um that sets you up to design the rest of your process and make sure that you're not you know failing people for being bad in relevant areas okay

point number two second way you can reduce noise in interviews is to use structured interviews so to define this a free-form unstructured interview is an interview where a new viewer gets in a room with a candidate and they ask questions and they follow their intuition they they you know based on the answer the critic a that gives they ask follow-ups and they try to get a sense what the candidate feels like if that person can be a good fit at the company and at the end to make a global yes/no I don't know how a

decision based on that Daddy interview and that contrasts with a structured interview where the interviewer comes into the room with a question which they're going to ask and a defined criteria they're trying to evaluate and the very interesting point here is that everyone thinks that that freeform interviewers feel better almost all interviewers prefer freeform interviews and they think they are more accurate in many candidates if you ask them actually say they prefer to be entered in that way but the things that this is completely opposite to all of the available data on this structure knickers

are simply more predictive they are better at predicting success on the job and there's no excuse not to use them we should all be using structure interviews and so in the purse exams what that means um the first thing is that you have to hold the process constant so the goal of an interview is essentially to evaluate variants and candidates and you know if you do not hold the rest of your process constant between candidates you're introducing noise so there's simply no excuse for not asking every candidate for the same job the exact same set

of questions I think the reason this is not more common is that the interviewers themselves tend to find this boring and you know I can say it suck it up like hey you like it you have to do that so second point on structure interview is that you want to give your interviewers defined criteria to evaluate so rather than putting in the room and saying decide if this K this person to be a good fit for the company say we care about coding productivity and you know knowledge of back-end web systems and so your goal

is to ask this question the interview and grade the candidate on coding pregnat ivities and knowledge of back-end web and there's actually some great research on this it ends up that a lot of the sort of worst biases that can come up in interviews are made worse when the interviewer is trying to make a global decision so if something's making a global - this person feel like a good employee decision that's where things like you know do they look like someone got you in the past or you know their race their gender that's what those

things come into play and interviewers are much better at ignoring those criteria ignoring those attributes when they're given a defined criteria to evaluate and then final point is that you want a unified decision-making on this and this applies posted to larger companies but you want to make sure that one person or one group of people is involved of all the final decisions so rather than viewing interviewers as people making decisions view them as you know people taking notes grading candidates on criteria and that all those notes and criteria go to a centralized person or a

centralized group who makes the final decision and the idea is just that again the goal is consistency and it's far more easy to be consistent if one person making all the decisions point three for how to reduce noise is that it's just used better interview questions and so I have some tips on that the first is that you want to avoid questions that require a leap of insight from the candidate and rather you are questions where there is a gradual sort of ramp of steps that they can follow that lead to a solution and this

is the general idea the rule of thumb I like is that you can ask yourself can this question be given away so if it's a question that has a sort of a single fact in the solution which a candidate could know in advance and perhaps be communicate them by a friend or by anything they read in Glassdoor then that means that it's probably a bad question and the example of that is there's a plastic question the question is you know imagine you're staying at the bottom of a flight of stairs and every step you can

either take a single step up one stair or a double step up to stairs you know how many unique paths are there at the top of the flight of stairs and you know it ends up the solution of that is the Fibonacci sequence kind of strangely if someone knows that obviously it's you know that's the solution if they don't know it they may well struggle then think about it and go down some some rabbit hole and so that's exactly what is the solution example of a good question is you know can you please implement the

game Connect four for me so there you know it's a series of steps each one relatively straightforward but that leads to a solution and and there's nothing F summons friend could tell them in ten minutes which would give them no massively unfair advantage you know in a game Connect four so another idea here this is the kind of related but you want multi-step problems and there's tend to lead to problems I don't have insight but also candidates will often get stuck in an interview even good candidates and if your problem has multiple steps to it

you can you can get them a hint you can help them out in one portion and it still have enough left for them to go on to do well and be themselves and demonstrate skill whereas if your problem is all in one one one one one sort of nugget of difficulty if they can't solve that you have to help them at that point they basically failed or done poorly on that section you want to avoid specialized knowledge so this is you know if your goal is to assess general program ability you probably want to ask

questions that evolve you know things like list and hash tables and strings rather than questions that evolve you know tries or prefix trees you know this list of court you know unless you've decided that you really care about algorithm you know structure knowledge and your goal is to make sure that me knows about tries in that case it's totally fine to ask about them but if you're if you're measuring something else if you if a try the portion of the problem you know some portion of your your people your your candidates will be familiar with

us others well in that will introduce noise so in general I think it's good to stick with the sort of classic most basic CS concepts another route you want you run a budget about three times the amount of time it takes you to solve a problem for the candidate so if you come up with a question and you solve it yourself it takes you ten minutes that's probably a good question for a 30 minute interview session and the reason here is just that it's it's far easier to be the interviewer that is to be the

candidate it's far easier to ask the question and we tend to downplay how hard the questions actually are and we can actually research on this we went through all the questions we asked we asked a terabyte and we looked which are most correlated with candidates going on to succeed and it ends up that the most effective questions tend to be much easier than what you know our intuition going in was predicting so it ends up that most interviewers think that you know the optimal question is quite a bit harder than the optimal question actually is

you wanna make sure that you ask four or more questions in an interview they hear is that you know each individual question carries a certain amount of noise as the person seems questioned before is it you know did they have whatever they lucky answering it if you ask more questions you'll get a a more consistent signal out and then finally just one one tip type of question we like a lot a trilobite is a question where we we give the candidate a problem and then rather than wanting them to devise a solution we tell them

a solution to the problem and then their goal is to take that idea and implement that in the code and so we give them an algorithm and we see if they can implement that algorithm rather than require them to devise an algorithm a fourth way to reduce noise is to ignore credentials during your interview so by credential I mean things like I you know did someone study at a well-known school or has someone worked at a well-known company and I'm not claiming that credentials are meeting are meaningless credentials are important right so the productivity case

the group of people who you know our ex new colors are indeed better engineers the group of people who have not worked at Google I'm so it's totally you know legitimate to take that fact into account when when when deciding about the hire or someone however it's not relevant to their actual programming skill so my advice here is to make sure that you're not biased by the fact that someone comes from Google when you are narrowly evaluating the programming skill so I recommend you hide that credential from the interviewers and the reason is we found

that interviewers are actually pretty biased by this if they know someone has you know what were your strong credentials they're more likely to interpret the result of the coding screen as in a positive fashion or this person you know yeah they didn't know that answer but I'm sure they you know I'm sure there was a temporary slip-up and so how do pencils from your interviewers let them assess program ability and then when making the final decision in the decision meeting consider credentials and also the performance in the interview and yeah I this will help you

find the programmers who are skilled and who lack traditional credentials and those are the undervalued people in the market and as a start-up if you can you know get if you're good at finding those people that is a big advantage okay point five is that you want to think about the false negative rate in your interview so a false negative is when someone fails your interview who could have gone on to do the job well and the opposite of false positive is when you have someone past if you hire someone who then goes on to

do poorly and probably be fired and both false positives false negatives are very costly so if you hire a bad person if the fire of them that's terrible it'll hums the morale for the team and also also very expensive its actual money big problem but you know if you're a start-up and you're really hungry you know you're held back but not having engineers if you pass up a person who could have joined your team and couldn't productive that's also very expensive and I'm making a pretty subtle point here but I think there's a bit of

a cognitive bias where false positives are very we're very aware of them we're very like we hire a bad person we feel that pain for a month or a month plus best-case and we've seen comes into a bite they're generally they generally are have too much faith that the folks who fail their process must have been bad engineers and couldn't have done the job and that's empirically not the case right if you watch you know as I said earlier no engineer will pass all interviews right so a significant portion of people who fail interviews do

go on to be employed very productively at other companies and so I just recommend that that folks designing any of you processes think about the false negative rate and try to give that some way in in in their calculations of setting studying hiring bars um one of my goals would apply to actually just get to a point where you can actually measure this rate because cuz no one knows what it is no one knows what the false negative rate on the interview is because to measure that you have to just hire people randomly and see

how they perform that's very expensive my goal is to get to where we can do that let's see okay point six is that you want to genuinely calibrate on the maximum skill that each candidate brings rather than their average skill or their minimum skill so someone comes in an interview they're very strong in one area or the weaker and others what matters the most is what they were strongest in everyone can look stupid right everybody you know if you ask me the right question I will definitely look very stupid that's true for everyone out there

and so the problem is that someone might go through an interview and do very well on some things that would be you know useful for the company but look stupid on one question and you know if that interviewer gives a blocking know that that person was stupid that's interesting noise in the process perhaps now again if someone does poorly in a crack in the area that is important for the job totally failed them but be open to the fact that everyone does look stupid sometimes and don't fail someone just because they look stupid on one

portion of the interview okay a final last point here is that you want to think about the candidate experience when designing an interview you know you want to you know make sure that every candidate who goes through I likes you likes your company and this is true for a few reasons one is just you know if they enjoy the process they have a higher probability of accepting an offer you make and so this will help in the closing step but a interesting point is that it will also actually make your screening itself more accurate because

stress has a big impact on performance high percentage of candidates get very stressed in interviews and underperform their actual peak ability and so some tips here to help you do stress include just letting everyone bring in their lap and work in their own environment you know what throw and throw but their own language their own tools they will be they will be able you're much more productive much less stressed and then coaching your interviewers in just some soft skills so being friendly providing breaks for the candidates and you know when they are doing poorly on

a section training them and how to intercede anyway which isn't too stressful insulting to the candidate rule of thumb here that comes from the the old Jolan software blog is that you you you want every candidate no matter how they do to finish your interview wishing that they had you know wanting join your company you know even people who fail your any very poorly you want them to end your interview wanting to join you being signs about the opportunity and the final point I you want to avoid hazing so this is this is rare but

results in some of the worst horror stories of interviews this is where you know I mean who takes on the role of you know a ritual of acceptance into a group you know think if this if this happens it's terrible and as a hiring manager you can just stay totally away from that okay those are my points um I want to emphasize that I'm not saying that you should lower the bar for who you hire I think if you follow this advice you will get the more accurate signal you can then set your barbecue higher

RF you want on that signal i but you'll still be making at that point better hires so just to summarize here i recommend that the first step you follow is to decide what skills matter for your organization and make sure that you're screening on things that you actually care about and then design a structured interview around those skills so come up with ways to assess each skill and stretch the criteria for the interviewers so they are you know less likely to be biased by by outside factors then you want to use good interview questions you

know multiple parts in the lives of insight you want to hide credentials from the technical interest because that interest is noise in the process you want to think about the false negative as well as the false to positive cost in your interviewers and you want to genuinely calibrate around the maximum skill that each candidate brings well doing all that you want to try to provide a positive experience for the candidate so those are those are my my tips so far I think this all applies to both big and small companies so I'm gonna go over

a few points here that I think are specific to let's say Series A and smaller companies so one one point here is winning what if your what if you're so small you don't have the scale to standardize your process so your seed stage start off your hiring your interviewing your first few people you obviously can't run an extremely standardized process because the first two candidates have seen those questions that's a totally real point I think it still is worth trying to run instruction interviews so you know you won't have you'll still you know your play-doh

pie you simply Google Doc with some tips tips written down but it still is helpful to think about what skills matter and try to design questions that assess those skills that will still reduce you know the bad bias in the interview let's say that you're in trouble sourcing so you're already staged start up and your number one problem is we know how do you get enough qualified applicants for your company so in that scenario it's fairly obvious that a false negative costs more so if you're struggling to get people to apply to your company screening

out someone who's bad stress all right so here someone who could have been good is a very expensive more expensive than for a large company but the flipside is that if you're a small company hiring a bad person is also way more cost you know from the way more expensive so it's not cream if the ratio of those two costs changes so I think you still have to care a lot about both I think what you can do is be less aggressive about screen folks out early so if you're an early stage companies struggling to

source candidates I recommend you are less aggressive in speeding out you're on site you know pass more folks through and accept a lower final interview you know success rate in exchange for you know better screening all the applicants let's say that you're a small start-up in use don't know what skills matter so you're not sure if you if you want hire someone who's you know very CCS focus or someone who's very you know web focused and there's no crisp answer here so we have plenty of examples of you know billion-dollar companies that have taken various

routes here my personal advice from from trilobite and from Socialcam and from twitch when it was small is that yeah I believe strongly that that that that the two most important skills for the first few hires are productivity and ownership so being able to basically take a project figure out has to be built and just make that happen and I recommend applies for that at the expense potentially of code quality so I think the first few hires at a company you should accept that perhaps the writing crappy code but you know by god they're writing

it quickly and they're getting stuff done and I yeah just say that was the case during the early stages of you know all through comes I've been involved with and I think that's normal and probably good so let's say that you're hiring for an area where you do yourself don't have technical expertise so you're you're a web developer and you're you need to hire an iOS engineer what do you do so what enters you can you sure you can use a stick neutral byte I will help you do that I you can call them friends

did you any before you if you have them but I think a good point here is that a trick that I've used in the past is to just ask and asking this to explain things so your web developer you ask the iOS engineer question they give an answer you have no idea if it's a good answer or a bad answer what you can do is just ask them to explain you say oh that's interesting can you explain why that why that's a good idea and for the most part if someone truly is a skilled iOS

engineer they should be capable of explaining their answer in terms out of web screen flow much get to understand it's not always the case sometimes someone come along who's you know good at assumption but the bad communicator and that that might fail them but this is a pretty good trick to hire outside your area of expertise okay that's basically I want to end act with an ask for you guys so this is this is a we've been developing an exercise for interviewers that we use when training new interviewers patrol bite and I want to see

if I get you guys to try this and I sort of email me and let me know how it goes because I'm curious but this works more broadly and the idea is that and you know interviewers tend to sniffily underestimate how noisy interviews are they they overestimate their own ability to distinguish to engineer some bad engineers and part of becoming a better interviewer is to is to sort of become a little more humble and more aware of your limitations and so this exercise is designed to sort of highlight that and so what I want you

to do is to do a mock interview with one of your co-workers so have one of your co-workers interview you where they're role-playing you know they're asking questions they're the interviewer and you're role-playing a candidate you're giving answers and I want you to tell them in advance that you're going to be giving bad answers to some of the questions you're gonna be roleplay I can't think who makes mistakes and then during the interview they ask questions I want you to half the time go ahead and give a bad answer roleplay a poor answer to the

question and the other half the time giving your best shot and actually try to answer the best possible answer you know to that question and then after the interview is done do a debrief session where you know your coworker goes through and sort of goes over all the mistakes you made and gives you feedback and what's great here is that they don't know what answers that you gave were intentionally bad and what were you trying your best and if they MIT look if they don't highlight the mistake you made where you were trying to make

a mistake they're gonna look a little bit bad so they're fully incentivized to be completely honest and rigorous when putting out what the flaws they saw in the interview and though you know the criticism they give will apply both of the points where you're intentionally making mistakes but also to your attempts to give your best possible answers this is being really interesting we've been doing this a lot a trilobite and it's great highlighting how interviewers disagree so if you do this with your critic or worker and you do it a few times reverse roles you

know the result is that you all well you'll highlight your area so you're both weak but you'll also highlight a bunch of areas where you have legitimately strong disagreements about what constitutes the best answer to it to it to a technical question there will almost certainly be more disagree more than you're expecting going in yeah I found this super insightful I'd love you guys could try this and I email me let me know how it goes awesome that's a thing for me hardies gonna talk about closing and then we'll both answer questions okay I'm gonna

wrap up here by going through best practices and point to optimize your chances of people actually accepting the offers you make them okay I have five main pieces of advice here the first is just understand as a start-up in particular speed is a huge hiring advantage that you have over bigger companies so through two of why we work with startups and larger companies and we ourselves are surprised by how often a big company can take weeks just to deliver the actual final offer details and when someone's getting to the end of their job search process

that's when there's most keen to just make a decision and move on and know what they're doing so as a start-up if you're quick at every step in your hiring process from the moment you have the first contact through to when you make the offer you increase your chances of closing a candidate so just be fast responsive at every step second Almond already mentioned like train your interviewers to have a good bedside manner with interviewees and so we ask candidates or engineers on trigger by what were some of the the reasons you had bad experiences

when you went and interviewed in person at companies and all of the reply centered around their experience with the interviewers and the top two complaints were interviewers not being familiar with the technical questions they were asking and second the interviewer being determined to show how smart they were specifically smarter than the interviewee so if someone has a bad experience interviewing with you they are highly unlikely to accept your offer three be prepared to talk in some detail about your company culture because almost every candidate is going to ask you that question what I like to

work here what's your company a culture like and I think we've seen a particular uptick in this question over the past year anything frankly a lot of stuff around uber and just a lot of the focus on the culture in the technology industry in Silicon Valley is making people especially individuals think more about the kind of culture they want to work in so two specific bits of advice here I definitely be prepared to answer questions about how you think about diversity if your teams already diverse or not I think you should have some thought around

how you plan on incorporating that into your hiring practices going forward and second when you're describing your culture be aware that almost every company like a significant majority essentially use three adjectives I open transparent and collaborative and we've seen this on trim why because we see them carrying their profiles right and so that can be an effective strategy because those all three of those things are good aspirational qualities right like no one would not want to work at an open transparent collaborative place and but it's not very good for differentiating yourself right so one strategy

you could take when you're talking about your culture is take more risk and the example the risk might be talk about the trade-offs in your culture so if you are if you believe yourself to be an open culture talk about the trade-off there right like openness means like being honest about things and being honest about things can be difficult and uncomfortable for people so if you're willing to kind of go in that direction you may a lien ate some candidates but you may also really increase your closing rate on the candidates are most excited about

you forth get your team and investors involved in the process so once you made an offer make sure your team reaches out afterwards offers to meet up with the candidate again answer more questions and generally be involved in their decision making process and do the same with investors and in particular pick any investors you have who would be particularly good at closing that candidate so if they're on the fence about whether to leave a big company for your startup do you have an investor that successfully left a big company and joined a successful startup and

finally present full and transparent offers with all the details the candidate needs to understand how much they're being compensated this sounds against probably somewhat obvious but I've been surprised by seeing companies engage in this sort of hypothetical battle where instead of just telling in candidate here's how much salary and equity you get so posing questions like how excited are you about us if we made you an offer do you think you join and this always creates a bad awkward candidate experience it's also unfair because the truth is no matter how powerful you powerful your mission

is compensation is a big factor in where someone's going to work and so asking them to commit before giving them details it just puts them in an uncomfortable position and decreases the chances that they would accept your offer to join so when you do make the offer provide full details this slide has I take the minimum of what you should put on an offer letter salary is fairly self-explanatory with equity make sure that when you present the details of an equity offer so the number of stock options exercise price etc that you're also checking into

a Sikandar how comfortable are they with this right if they're if they're a senior engineer and they've worked at 6 or 7 different startups they probably understand this you don't have to go into too much detail if they've spent most of their life working at Intel they may not be familiar with how startup equity works right so ask them how comfortable they are don't overwhelm them when you present the offer details if they're not just send them follow-up resources and that they can read on their own time and also this sounds somewhat obvious but make

sure you yourself as founders really understand how stock options work because people will ask you questions about it and you actually don't end up spending much time on this because you yourself have restricted stock and fundraising doesn't usually include options but you should understand even down to level all I'd say tax implications of different types of options a final thing I'm doing close on here is a common thing that comes up with startups is how do you compete for engineers against the tech giants like Google and Facebook in particular right so Google and Facebook obviously

can offer much larger liquid compensation packages than a start-up can and frankly like the public tech company stocks have been doing great over the last few years so it's becoming more challenging to compete against them still we routinely see startups do it and win out against much larger companies offering more money think there are four pictures they make that have been quite compelling first is emphasized learning the the case I'd make here is that you learn the most or you learn the fastest when you're given real decision-making it's a responsibility which means if you make

a mistake like bad happens and there's very very little chance practically no chance that that's going to happen to a big company right like big companies have checks and balances in place to make sure that no one can create too much damage in particular for new hires right so if you tell someone that hey like we just don't have that luxury we're growing quickly we've got to get everything shipped yesterday you're gonna get thrown straight into making a real decision making authority and if you screw up that's going to be bad for the company and

that's the way that you learn and build real experience so emphasize that too I talked about career progression and one bottom two macro comment I'd make is if you look at executives at public technology companies they tend to be you're often can be much younger than you see at the counterparts that like public deactive or like a public utilities company right and the the reason for that is you can grow up and be like chief product officer Facebook at 38 or however old the chief product officer is because you join the startup when it was

really early and you grew along with it right and the tech industry is really the only place that happens like it's more of a meritocracy you don't have to pay your 20 years dues at a company before they consider you for a senior role you can rise up as quickly as a start-up grows so emphasize that third talk about opportunity cost and in particular the way I'd frame this or talk about this would be at any given moment in time there's always going to be a basket of big safe technology companies to go work at

the names might change sort of every five years or so but the experience of working at one is largely fungible whereas the experience of working at a start-up varies wildly right so I would emphasize the fact that your startup right now the team you have the opportunity is a unique one that's a uniquely good fit for that candidate and that they won't get again and if it doesn't work out they can always go back to generic big company and final point this one specific to hiring more junior candidates talk about mentorship so one thing that

we see is especially for new grads people started off college is that they're worried they won't get the right level of mentorship they won't learn how to best practices work if they join a start-up and maybe they should go work at Google for a couple of years get that under their belt and then join a startup I mean first decide if that's actually true or not right like if you're if you yourself have only junior engineers on your team and frankly you just own one to mentor or get someone up to speed and then don't

make that case and maybe don't hire someone like that and but if you have experienced engineers on your team emphasize that and emphasize the fact that they will get to learn alongside experienced engineers who have worked at places and learn best practices etc etc so that's everything I think we're gonna open up for questions I will ask the questions about the topics we've covered you wanna come up like background things are things that rule out the candidate background background checks frankly most startups I know don't do them they're especially easy to do now though because

you can integrate Checker in with your payroll service so sure but that's not something I would like focus on I focus more on like if someone can be a good fit or not yeah [Music] that's a hard question so the majority of computer science engineers are probably not gonna be a to a pic Euler good job of doing the work you want that said this sounds like a case where you could probably just evaluate past work this person has done so you could probably ask this person to take to give you a portfolio or other

examples of work they've done and if that is at a level and of a type that you're happy with hiring them would probably work yeah probably so the thing we focus on is background blind hiring so we focus very narrowly on just direct skills assessment rather than looking at other things that that correlate in our predictive so you know personality type is real people vary in their personality so that stuff matters for performing jobs I think it's more research on like the big five rather than big five percent of traits rather than - Briggs all

relevant but I would try to keep it pretty separate from technical assessment so it's assess technical attributes yeah and then you know separately try to assess you know culture for it friendliness soft skills and then combine that all and they try to make a global decision yep sorry back in the sunglasses [Music] that's the extreme version and I said you probably can't so you okay so the question is if you're a non-technical founder how can you assess an applicant technical skills and I think the answer in fortune is that you probably can't so you can

fall back in the third device I try to look at past work but I think this is why it's very helpful to have a technical co-founder is because you are going to be at a fundamentalist advantage there yeah I okay so the question is how do you find a technical co-founder and you figure out they're good or not I think sort of Arlen's advice basically applies here they the most thing is the even if you use a friend or some that you know who is an engineer to talk to them and figure out they're good

engineer or not otherwise there's there's not really any other options yep yeah honey I'm not precise with my pointing yes but I think this is largely focused I'm sorry moving the question the question the question is other ways to work with the universities and colleges to generate like a pipeline of talent that you could hire from that accurate I'd say yes there certainly are on you know they exist like career fairs and sponsored events at colleges et cetera et cetera but I think those most there's probably work better for larger companies that can fit themselves

around the college graduation deadlines and that schedule for start-up is kind of tough because usually you want someone that's available right now to hire as opposed to waiting for them to graduate so I probably wouldn't recommend investing too much time in that as a start-up and just to clarify big companies at this point make offers a full year in advance with large signing bonuses you know you deliver it a year in advance so it's very hard for startups to compete with college hire [Music] yeah the question is to clarify unlike my last point about role

playing a bad interview so the key point here is that you're gonna roleplay need to do in the cat you know the co-worker who's playing the candidate who's getting the answers they're going to intentionally make some mistakes and what that does is it frees up their coworker to be critical so normally if you say if you do a mock interview between two co-workers the problem is that you know in the end everyone's gonna pull their punches they're not gonna honestly give their opinion of the performance that that their co-worker gave it just doesn't quite work

due to all kinds of social pressure if you tell them I'm gonna be intentionally making some bad I'm saying I'm gonna tell you things and mistakes that frees them up where they're taking they're then free to point out the flaws in your interview but only free to they have to because if you say I'm making mistakes and you ask them what's your opinion of not even the results if they don't point out a mistake you made that means you got something past them right so it flips the social pressure and creates pressure where there then

incentivize to be critical which is extremely useful because you know as experienced engineers like weird like me it's been a while your senior engineer on a team after small company it's been a while since you've had someone really brutally critique how good you are at answering your own questions and that you know what will happen is that they will point out legitimate flaws you'll be trying to give your best answer and they will point out things you said that are wrong or things that could be better and that it's very humbling it's very humbling and

I recommend that they do that's I think this is exercise you can give yourself that experience of being humble and having flaws and even your own performance pointed out in the middle yes got it you talked about a 95% plus success rate on one of the slides and I have given the the fact that the interviewers tend not to have a very high rate of reliability it seems time to believe that there's a 95% and also doesn't match my experience that 95% of the hires successful so when you think is the real success rate where

you know you're after hire something's still there and they're pop before yep that's gonna be so this comes from some so that number I 5 comes from some surveys you get upon companies okay sorry so the question was I my slides I said that that there is that tires are successful ninety-five percent of time but that doesn't mesh with you know the experience of being a hiring manager height people many hire are not not great so the question is what's the actual rate at which you know candidates end up being top performers and yeah questions

totally right so that that number comes from some surveys we did where we asked companies of all their hires what percentage got fired and we actually asked web stands our top performers and that five times the fire rate so five percent actually get fired you know an additional 30% are people who stick on but are not particularly great employees and then about 10 percent are in the bucket that cut that companies said were there their best hires and top performers thank you [Applause] you