Building an engineering team

As the slides are loading, there is no topic that should occupy your minds more as you build your company than bringing on the team that's going to make your company successful as you move forward. Harsh and Aman from Triple Byte, YC alumnus, are going to talk about building, in today's day and age, perhaps the key portion of that team, the engineering team.

So please welcome Harsh, you're starting, yeah, Harsh. All right. Thanks for having us, everyone. So I'm Harsh. I'm one of the cofounders of Triple Byte, along with Aman.

Previously, I used to be a partner at Y Combinator, and part of the inspiration of starting Triple Byte was noticing how after graduating Y Combinator and raising the first investment round, everyone's number one problem was hiring and specifically hiring engineers, because it's the hardest hiring challenge.

So through working on Triple Byte, which is a hiring marketplace that's used by engineers to find new places to work, Aman and I have gathered lots of data on what works well when it comes to hiring engineers. I've personally focused a lot on spending time with companies, helping them think through their strategies for finding engineers and obviously getting the most out of Triple Byte.

And Aman spent a lot of time thinking through the details of how do you evaluate an engineer, how do you evaluate their engineering skill, and answer the question of are they a good engineer or not. So we're going to share that and sort of have a divide and conquer strategy going on here.

The four main topics we're going to cover are where to look for engineers, and when you should start thinking about using recruiters, and that's what I'm going to start with. Then we'll talk about evaluating technical skills, which Aman's going to cover. And then I'll finish up by talking through the process of making offers and closing, which is getting people to actually join your team.

But before we start on any of that, I want to just issue a warning and make sure you're well prepared for the fact that hiring really well and truly sucks. It's an incredibly painful process, because for many reasons, which I'll describe in detail, the first is it takes a lot of time. It takes a lot of time just to convince someone good to even have a conversation with you.

And as a founder, as you know, time is a very scarce resource. There's bugs to fix, there's sales, customers to close, various things going on. And hiring will never feel like it's your top, most urgent priority, and it's very easy to procrastinate and push it back.

But if you do that for too long, you won't scale and you won't grow your startup, and someone else will come along, do that, and take the market. Two, hiring involves a lot of repetitive work. So actually, as Tyler was giving his presentation, I was talking to Jeff about this back there, there's a lot of similarities between sales and hiring.

And actually, there's a lot of similarities between sales and hiring and fundraising, and by lots of things that you do as a startup founder, a large part of it is effectively selling all of the time. And selling, as Tyler pointed out, involves a lot of repetitive work.

So hiring will involve lots of messaging, it will involve taking lots of coffee meetings, lots of phone calls, lots of interviews, and most of those will result in the dead end and be a complete waste of your time. But you have to keep going. And finally, you will get your heart broken.

You will inevitably end up getting rejected by people who you really wanted to hire, who would have been the perfect fit to help you hit your growth goals. But it turns out they were never really that serious about leaving their comfortable job at a big company to join your exciting but risky startup. So be prepared for all of this.

And as you're thinking through building a hiring process, I encourage you to think about this as a funnel that you're creating that has three parts to it. The top of the funnel is sourcing, and that's finding people who could be a good fit. The second is screening, that's answering the question of, do you want to hire this person or not?

And the final part of the funnel is closing, making the offer and getting the offer accepted. I'm going to start by talking through some strategies for building the top of your hiring funnel. These are the five places I'd recommend that you look for making engineering hiring, for making engineer hires. I'll talk through the pros and cons of each of these.

They are personal networks, hiring marketplaces, LinkedIn, GitHub, job boards, and meetups. I'll talk about how you can get the most out of each of these. And this list is ranked or it's sorted in order of where I think you should start out focusing most of your attention and energy down to where you should focus the least. So we start with personal networks.

In my opinion, personal networks are the best place to hire, especially when you're early and making your first few hires by far. And the reason is anytime you're deciding if you want to hire someone, you're essentially asking yourself two questions. One, does this person have the skills that you need for them to do the job? And two, can you personally work effectively with this person?

When you're a big company, you can mostly focus on answering just the first question because you're large enough, there's enough people, there's enough teams that it's likely that one team somewhere will be able to work effectively with anyone. But when you're small, that's not true. Whether you can work effectively with someone or not is a big determinant of your success.

And if you hire the wrong person early on, that could literally be fatal. So when you hire someone that you've worked with previously, or you hire someone that's worked with a person you trust, you de-risk the chances that you won't be able to work well together, which is a big thing to consider early on. So that probably sounds like somewhat obvious advice.

And yet, I'm surprised by how often founders still don't really use their personal networks effectively when they're hiring. And I think there's two reasons for this. The first is they don't use a process to exhaustively search through everyone they could potentially hire. And the second is they don't actually make the ask. That usually comes down to being afraid of being rejected by your friends.

It's somewhat easier, actually, to be rejected by a complete stranger than it is to ask your best friend to come join you, and they say, eh, I'm not sure the idea is that great. You can also worry about what happens to the friendship if the startup doesn't work out. There's more that goes into it when you're talking to someone you know personally than when you're talking to someone you don't.

But the truth is, you just have to suck it up and do it. But if you want your startup to be successful, hiring from people you know is a tremendously valuable resource, and you just have to make that ask. So I'd recommend you follow a strict process here, and start with just making a list of every good engineer you know, whether you think they're available or not.

That's actually completely irrelevant. It doesn't matter if they just sold their company for a billion dollars. Put them on the list. Then ping each and every one person on that list to meet up and commit to asking them if they would join you. However crazy you think it is, however unlikely, commit to making the ask.

If they say no, or they're hesitant, de-risk it a little bit and say, will you at least come by the office and see what we're working on? If the office is your apartment, that's totally fine too. But just keep pushing until you've at least shown them something that you've done. Keep working on convincing them.

If it doesn't work out, if they say no, and you get a definitive no, then ask them if they were in your position, who would they try and hire? Take a list and go out and repeat this exact same process with them. And this process just never ends. I know public company startup founders, well not startups, but public company founders who still do this on a daily basis.

This is just a key thing you have to embed in yourself as a startup founder. As your company does scale and grow and you start putting the team together, you want to start tapping into the personal networks of your team. The way I'd recommend doing this is team events where people brainstorm potential hires. This is commonly referred to as a sourcing party.

The way I'd recommend going about this is get everyone together, send out a shared spreadsheet and describe the role that you're hiring for. So if it's an engineering role, describe in detail who you're looking for, who are examples, candidates, what are skills and qualities you'd be excited about.

And then literally have everyone spend 30 to 45 minutes going through their LinkedIn or their Facebook or whatever, right there and then, thinking of everyone that could be a fit and putting them into the spreadsheet. Once that's done, at TripleBind, I'll actually then personally follow up with anyone on that list who seems like a good candidate or not.

And we've made several really great hires doing this. It works really, really well. And you can kind of make it like a fun thing too, right? So we'll do it at the end of the week, just before our Friday, all hands, food and drink. And you can also offer referral bonuses to your team to incentivize them to do this.

So really make sure that you're sticking to an exhaustive process, you're making the ask of people you know, and then as you scale, tap into the personal networks of your team. Once you're sure that you've exhausted your personal network for leads, the next place I'd start looking would be hiring marketplaces. Hiring marketplaces are actually relatively new.

They've become more popular over the last few years as it's become harder to hire engineers by using traditional methods like reaching out on LinkedIn or GitHub. And I'm going to talk more about that next. The way I think about hiring marketplaces is they actually work a lot like dating sites.

The idea is there's engineers who create profiles, companies that create profiles, and both are advertising their best selves. You message each other and you figure out if it's worth meeting up in person. And if it all works out, you make a hire. The dynamics of the marketplaces, though, is such that the demand for good engineering talent far exceeds the supply.

And so typically, it's the companies that are being a lot more proactive in terms of reaching out first to the candidates. The candidates are getting multiple inquiries, and the candidates or the engineers are the ones that are choosing who they want to speak to and who they don't.

A big benefit of using a marketplace, especially in the early stages, is that they can help you hire very quickly because most candidates who are on the marketplaces are actively looking for a place to move right now. It's very quick to get on a phone call with them and start pitching.

And if you run a good closing process, you can significantly reduce the amount of time you'll spend as a founder on hiring, which is obviously great. The downsides, though, are that they tend to be quite competitive. So engineers are being reached out to by multiple companies at the same time. So you'll have to be very effective at convincing them to join if you want to make hires.

And the second is that they can be expensive. So most marketplaces will work on a fee per hire basis, which can be 15% to 20% of the first year salary. That's cheaper than a recruiting agency, but still a significant cost if you're an early stage startup.

I'm obviously biased here because TripWide is a hiring marketplace, but I'd say the three main ones that come up in conversations, or at least when we're pitching customers, would be TripWide, Hired, and Vetri. You should try it. They're all free to use to get started, so you're welcome to try it. I'd say TripWide, the way we differentiate ourselves is essentially by having better candidates.

And we measure that by what percentage of candidates that companies interview through TripWide do they make an offer to. And that tends to be twice the rate that they make through other sources. And just as a general note, hiring is a funnel. You're optimizing your funnel. So you should pay attention to what percentage of candidates are making it through each step in your funnel.

When you're early, you won't have that many candidates, so you can't be that scientific about it. But start capturing that data and just building that into the habits you have when you're thinking about hiring. The third source that I'd recommend you go to and look at is LinkedIn and GitHub. These are effectively the biggest online directory of engineers in the world.

Most of hiring that's done at big companies is through teams of technical recruiters reaching out to engineers on LinkedIn or GitHub, finding the ones that fit certain keyword criteria, and sending them cold messages. And they go for a very high volume approach here. So a technical recruiter may well send over 100 messages a day, just in the hope that they get a few replies.

And a dynamic that's occurred, especially in the last few years, is as there's more recruiters, there's more messages going out on these platforms, so response rates are dropping for everyone. Which means for early-stage startups in particular, it's going to require a lot of your time sending a lot of messages in order to get a few interested candidates.

So making this work for you requires, in my opinion, not playing to a high-volume approach like a big company recruiting team would, and instead spending the time actually researching and reading the details of profiles, so reading through someone's LinkedIn, looking through their GitHub, looking at the details of the work that they've done, and sending a smaller number of personalized, targeted messages, and emphasizing when you send the message, why would someone be a good fit for your company specifically, and give them clear evidence that you've read their profile and you're interested in them as an individual, as opposed to sending a spam message.

But that doesn't mean the message has to be super long. I'd still advise keeping it short and concise. Just the key fact is there's proof that you've read their profile. Final note here is send emails instead of messages.

So if you sign up for LinkedIn Recruiter Lite, which is about $120 a month, that will give you access to Connectify, which is a Chrome plugin that makes it really easy to pull out email addresses from anyone's LinkedIn profile, and you'll consistently see much higher response rates through emails than you will on messages. So definitely go for that.

The fourth place I'd start looking for engineers would be job postings or job boards. The two main ones for startups are Stack Overflow Jobs and AngelList.

I haven't included Hack and Use Jobs on there because it's only available to YC companies, but Hack and Use Jobs is somewhat unique and just has a particularly high quality of engineer, and I'd rank it second on this list, actually, if it were sort of a standalone source of engineers. Job boards, in general, do suffer from a quantity over quality problem.

So what's good about them is you don't have to spend a huge amount of time posting to one of them. The downside, though, is that the time suck can come in later because most of the applicants you'll get will be vastly underqualified.

You'll get a lot of applications from people who aren't even really software engineers, and it will take a lot of your time reading through all of these resumes and applications to find the one or two good applicants.

So to maximize your return and maximize the number of good applicants you do get, I'd recommend focusing on making your job listings unique and interesting, and bear in mind that the majority of job descriptions on the internet are written by someone in a recruiting or marketing department that's using corporate boilerplate language that especially isn't going to appeal to an engineering audience.

So as a startup founder, you can experiment to try and bring through a bit of your personality in the job listing. So one thing you might try is write in the first person about the personal story for why you started the company, why you're excited about the mission, and get that excitement and passion across. Other things you could try, is there anything unique about the culture?

Is there anything specific about the technical challenges or product challenges you're facing? So if you put in more details, that again will stand out, because big companies tend to be very generic and vague when they're talking about what you actually get to work on there. The final source I'll talk about is actually just like physical in-person meetups.

So I don't think that these are actually going to be very effective, and they're sort of a long shot. Like the numbers don't really work out, like in-person meetups don't have that large of a number of people in attendance, and the truth is most of the time people are there for free food and drink, more than trying to actively find somewhere to work.

So it's unlikely that you're going to find both a really qualified candidate who's actively looking to move, who's excited about your company, and you also personally need to be very effective at talking to strangers and convincing them of things in order for this to work at all.

So I've included it on there because I do know startup founders who have had success through meetups, but they are few and far between.

If you do try this approach, I'd recommend focusing on technical meetups, and by that I mean the local closure programming group where people come together and meet up and bring laptops and work on things is more likely a better source of engineers than going to Dreamforce. The sort of final thing you could try here is hosting meetups at your own office, right?

So you could also combine this with the personal network hiring and use it as an excuse to have friends come by the office, or have their friends who are engineers come by the office.

At TripleByte, for example, one of our engineers is a fanatical Emacs user, and he hosts the Bay Area Emacs meetup at our office, and it hasn't worked for us for hiring, but it is a good way to just meet and build a network of good engineers that could come in valuable at some point in the future. So it's definitely worth considering.

Now I'm going to talk about when you should think about using recruiters. Truth is, there's not really any hard rule on when you should hire a technical recruiter. I've seen companies of less than 10 people hire one. I've seen companies wait until they're 50 people and plus. These are my opinions on when you should, and I treat these as a rule of thumb.

So first, I think you absolutely should wait until you've at least hired your first engineer before you consider bringing on a recruiter in any capacity. The reason for that is actually, I think general startup advice applies here. So one is just in general, when you're hiring, it's good for you to do the job yourself for a bit.

So you feel the pain and you understand it, and you sort of understand the details of what makes someone good at that role at your startup in particular, before you go and hire someone, you'll be better able to assess them. So if you feel the pain of recruiting yourself, I think you have a better shot at hiring the right recruiter for you. And the second reason I say is just, it's like sales.

It's actually good for you to go out and try and pitch and convince people to join your startup. So you can just understand yourself like what message works and what doesn't, because as a startup founder, you are always selling and you never know when you might bump into someone who could be a really great hire.

And if you've already sort of practiced the pitch for convincing engineers to join you, you'll have that sort of ready to go. So I'd recommend making sure you always make your first hire before you try and delegate this to a technical recruiter.

Second, I'd expect to have a good hiring cadence, so somewhere around hiring an engineer a month for the next six months or so before you start bringing on a recruiter. Otherwise, it's likely they'll run out of work to do quite quickly.

And then finally, as a rule of thumb, if you're spending more than 50% of your time sourcing, that's like all the things I mentioned before, and doing initial phone calls and screens and getting people to come and meet you in person. Over 50% is probably about the time where you want to start thinking about bringing on help, because 50% is about the amount of time you want to be spending on hiring.

So recruiters themselves come in roughly three types. You have contract recruiters who you pay by the hour, and they can sort of do anything from just cold messaging on LinkedIn all the way to doing the initial phone screens. There's in-house recruiters, it's just hiring a full-time technical recruiter that works as a member of your team. And the final would be agency.

And agencies are essentially teams of salespeople that are paying lots of engineers on LinkedIn or wherever they can, and then sending their resumes out to as many companies as possible. And they tend to charge 25% to 30% of first year salary if you hire an engineer's resume that they send you.

They do tend to be fairly high touch, so they're quite involved in trying to give you information that will help you close an engineer, but they're also sending that information to multiple companies at the same time.

So my recommendations here would be when you get to the point where you feel it's time to bring on help or bring on a recruiter, start with a contract recruiter and have them focus on just the sourcing piece of it. So have them just focus on reaching out to engineers on LinkedIn and GitHub, and their main deliverable for you should be filling your calendar with calls with promising candidates.

Then you're doing the pitching and practicing, or you're doing the pitching and convincing, and when you get to the point where that becomes too much work for just you, then I'd consider hiring a full-time in-house recruiter and training them to do the pitch call. So now they're both sourcing and doing the initial calls and sort of setting up the on-site interview process for you.

So to sum up, my sort of startup hiring plan, if I were just getting going and building an engineering team, would be start by making sure you've exhausted your personal network, spend lots of time taking people out for lunch, coffee, making the ask. Two, experiment with the hiring marketplaces.

Your mileage will vary on these depending on how effectively you can pitch your company, but even if you don't make hires from them, you'll get valuable experience pitching real engineers and real candidates and learning what resonates about your company to that audience.

Third, spend some amount of time doing personalized and targeted outreach to engineers on LinkedIn and GitHub, making those messages really personalized.

And then finally, treat job boards and meetups and sort of meeting people in person as a background process you run, where you're not expecting to make any hires from them, but you're sort of building a general pipeline that could be valuable in the future. Cool, that's the first part of this.

Armand's now going to talk about how to screen and evaluate technical school of engineers, then I'll jump back in and wrap up with making offers and closing. Awesome, thank you, Harj. So I'm Armand, I'm Harj's co-founder at Dribblebyte. And before this, I started SocialCam with Michael Seibel, and I was an early employee at Twitch. And I'm going to talk about just the screening step.

So how to identify skilled engineers at your company. So the first question here is just, you know, why you should believe me about any of this. And one answer is that I've done a lot of interviews. So since starting Dribblebyte, I've interviewed over 1000 engineers personally. But I think a better answer is that, you know, Dribblebyte has a pretty special vantage point.

We're able to see how interviews do, see how candidates rather do, you know, in the interviews at multiple different companies, I can see how the same candidate performs in multiple companies. And that gives us a data set that I think no one else has. And, you know, that data is, is, you know, where the advice I'm going to give today comes from.

Before I jump in, I want to just go over the basic hiring process that most companies use. And this is actually pretty standardized. So probably 95% of tech companies use these basic steps to screen candidates. So first is a resume screen. Someone applies to a company, they send in a resume, and a recruiter looks over the resume and decides if this looks like someone who's a basically good fit.

Then a recruiter call. So this is a usually a 30 minute phone call with recruiter, just asked about the candidate background, judge culture fit, and make sure they're interested in the company. Then a technical phone screen. And so this would be between 30 minutes and an hour with an engineer, usually solving a single programming problem.

So this is sometimes something like fizzbuzz or a little bit harder. Often this point done over over a synchronized text pad of some sort. Then this is optional, sometimes a take home project. So a substantial project the candidate completes on their own time, and then sends into the company to be evaluated. Then finally, the onsite interview.

So the candidate comes into the office and does between three and six one hour sessions with engineers at the company, covering individual problems. And then finally, decision meeting.

So usually the next day after the candidate has gone home, everyone who interviewed the candidate and the hiring manager gets together in a room and talks about sort of their perceptions, and they as a group make a hire or no hire decision. Some stats on this. Companies make offers to between two and eight percent of all of the engineers who apply.

But interestingly, exactly where in the process that drop off happens is very different between companies. So we see companies where 75% of people who apply get screened out at the first step on a culture fit call. And then we see companies where almost all applicants make it through to the final interview, and that's where all the screening happens. And about 95% of people who are hired work out.

That is about 5% of technical hires result in someone being fired within a few months. And then from the candidate side, what we see is a distribution of success in interviews. So the top few percentage points of programmers by skill receive job offers after most of their interviews.

But then the bulk of programmers are somewhere in the middle where they do interviews and they receive offers after somewhere between 15 to 30% of the interviews they do. And an interesting point is that no one passes all their interviews. So there are not magical engineers who receive offers after every interview they do.

And yeah, this gets at what I think is the major challenge when designing an interview process. And this is inconsistency. So this is noise in the interview process. It's kind of the idea, is your interview fundamentally repeatable and meaningful?

You know, if you had to, if you could somehow, you know, re-interview your co-workers, your employees, or everyone who passed your interview in the last year, if you could re-interview those people, how many of them would pass again? And it's a pretty scary question. But it's kind of an interesting thing is that we've been able to get some data on this.

So what I did was I calculated a stat called the inter-rater reliability between all of the interviewers at all of the companies on Trailbite. And so what that means is this is a statistical measure of the extent to which different interviewers tend to agree about which candidates are best.

And it's on a scale of zero to one, where zero would be no agreement, or the amount of agreement you would expect to see, you know, in random data from chance alone. And one would be perfect agreement. And what I found was an agreement of just over 0. 1. So the first point is that that's obviously much closer to no agreement than it is to perfect agreement.

But for some context on that, I calculated the same stat on a data set of online movie reviews. And what I got was an agreement of very similar, but actually slightly higher. So it ends up that, you know, interviewers agree about which engineers are the best at about the same rate that Netflix viewers agree about which movies are best. And that's, yeah, that's scary.

Interviews are fundamentally noisy, and they are more noisy, the data shows them to be more noisy than most hiring managers sort of want to believe. So, you know, the first question here is sort of why do interviewers at all then? If interviews are so noisy, why do them at all? Why can't we just use something like trial periods to actually be engineers?

And, yeah, I think this is actually a great idea. It is almost certainly much more, you know, if you can work with someone for a week, you can almost certainly get a much better sense of if they're a good employee than you could in a, you know, in a three-hour interview with them. The problem is that most engineers don't actually want to do trial periods.

So we did some research on this at Trailbite, and it ends up that only 20% of engineers are willing to do trial periods. And there's actually some, there's some adverse selection there. So the, you know, most of the best programmers are in the 80% who would prefer to do a standard technical interview because it takes less their time and is faster.

So I think that trial employment is an excellent option, and you can offer that as an option. But, you know, if you don't want to scare away most good programmers, you do have to still run a standard traditional interview process. So what I'm going to talk about sort of today is, you know, specific pieces of advice that you can use to try to reduce the noise in a traditional interview.

And I'm going to go over seven points. So point one, the first way to reduce noise in an interview is to decide what skills matter for your company. There are a lot of different ways that a programmer can be skilled. So, you know, someone, for example, can be very productive, or they can be slow, careful, write great tests, and make sure they don't commit bugs.

You know, someone can be strong in math and computer science, or they could be, you know, deeply knowledgeable about the internals of the Linux kernel and, you know, scheduling and real-time operating systems or something. And if you don't decide as a founder which of those skills matter for your company, then your interviewers are going to decide that for you.

So they're going to come up with questions that they ask you in the interview process, and they will fail people who answer poorly on those questions whether or not those are areas that should matter for your company. And this is actually a major source of noise in interviews.

Every, you know, engineer has this bias where they think the things that they know the best are the most important skills one could know. And absent specific direction from above about what to look for, they will fail people for areas that are not important for your company. So my first piece of advice here is that you should go through and ask yourself these questions before you start hiring.

So the first is, I need to clarify, these are the axes along which we see most disagreement between interviewers. So the first question is, you know, do you need sort of fast iterative programmers or someone who's careful and rigorous? Do you want to look for someone who's motivated by solving hard tech problems or someone who is motivated by building, you know, beautiful product for users?

Is it important that someone comes in with skill in a particular technology, particular language, say? Or can you hire, you know, do you want to hire a smart person and let them learn your tech stack on the job? Is academic computer science or sort of algorithm ability something which is important for you or is that an irrelevant skill?

And then is there any other sort of specific expertise that you need in people you're hiring? It's actually, so, you know, it's fine to answer both some of these questions. You don't have to specify only a single profile of person you're hiring. The important thing is to decide what matters, even if that's multiple profiles.

That sets you up to design the rest of your process and make sure that you're not, you know, failing people for being bad in irrelevant areas. Okay, point number two, second way you can reduce noise in interviews is to use structured interviews.

So to define this, you know, a free-form unstructured interview is an interview where an interviewer gets in a room with a candidate and they ask questions and they follow their intuition. They, you know, based on the answer the candidate gives, they ask follow-ups and they try to get a sense of what the candidate feels like, if that person can be a good fit for the company.

And at the end, they make a global yes-no, hire-no-hire decision based on that interview. And that contrasts with a structured interview where the interviewer comes into the room with a question which they're going to ask and a defined criteria they're trying to evaluate. And the very interesting point here is that everyone thinks that free-form interviews feel better.

Almost all interviewers prefer free-form interviews and they think they're more accurate. And many candidates, if you ask them, actually say they prefer to be interviewed in that way. But the interesting thing is that this is completely opposite to all of the available data on this. Structured interviews are simply more predictive.

They are better at predicting success on the job and there's no excuse not to use them. We should all be using structured interviews. And so I'm going to go over some examples of what that means. The first thing is that you have to hold the process constant. So the goal of an interview is essentially to evaluate variance in candidates.

And if you do not hold the rest of your process constant between candidates, you're introducing noise. So there is simply no excuse for not asking every candidate for the same job the exact same set of questions. I think the reason this is not more common is that the interviewers themselves tend to find this boring. And all I can say is suck it up. You have to do that.

So second point on structured interview is that you want to give your interviewers defined criteria to evaluate. So rather than putting them in the room and saying, decide if this person is going to be a good fit for the company, say, we care about coding productivity and knowledge of back-end web systems.

And so your goal is to ask this question in the interview and grade the candidate on coding productivity and knowledge of back-end web. And there's actually some great research on this. It ends up that a lot of the worst biases that can come up in interviews are made worse when the interviewer is trying to make a global decision.

So if someone is making a global, does this person feel like a good employee decision, that's where things like, do they look like someone I knew in the past? Or their race, their gender, that's where those things come into play. And interviewers are much better at ignoring those criteria, ignoring those attributes when they're given a defined criteria to evaluate.

And then final point is that you want to unify decision-making. making. This applies mostly to larger companies. But you want to make sure that one person or one group of people is involved in all the final decisions.

So rather than viewing interviewers as people making decisions, view them as people taking notes, grading candidates on criteria, and that all those notes and criteria go to a centralized person or a centralized group who makes the final decision. And the idea is just that, again, the goal is consistency. And it's far more easy to be consistent if one person is making all the decisions.

Point three for how to reduce noise is just to use better interview questions. And so I have some tips on that. The first is that you want to avoid questions that require a leap of insight from the candidate. And rather, you want questions where there is a gradual ramp of steps that they can follow that lead to a solution. And this is a general idea.

The rule of thumb I like is that you can ask yourself, can this question be given away? So if it's a question that has a single fact in the solution, which a candidate could know in advance and perhaps be communicated to them by a friend or by a thing they read in Glassdoor, then that means that it's probably a bad question. And an example of that is there's a classic question.

The question is, imagine you're standing at the bottom of a flight of stairs. And every step, you can take a single step up one stair or a double step up two stairs. How many unique paths are there at the top of the flight of stairs? And it ends up the solution of that is the Fibonacci sequence, kind of strangely. If someone knows that, obviously that's the solution.

If they don't know it, they may well struggle and then think about it and go down some rabbit hole. And so that's the example of a bad question where this leap of insight is the solution. An example of a good question is, can you please implement the game Connect Four for me? So it's a series of steps, each one relatively straightforward. But that leads to a solution.

And there's nothing someone's friend could tell them in 10 minutes which would give them a massively unfair advantage implementing the game Connect Four. So another idea here, this is kind of related, but you want multi-step problems. And those tend to lead to problems that don't have leaps of insight, but also candidates will often get stuck in an interview, even good candidates.

And if your problem has multiple steps to it, you can give them a hint. You can help them on one portion and still have enough left for them to go on to do well and redeem themselves and demonstrate skill. Whereas if your problem is all in one sort of nugget of difficulty, if they can't solve that and you have to help them at that point, they've basically failed or done poorly on that section.

You want to avoid specialized knowledge. So if your goal is to assess general programmability, you probably want to ask questions that involve things like lists and hash tables and strings rather than questions that involve tris or prefix trees.

This is, of course, unless you've decided that you really care about algorithm data structure knowledge and your goal is to make sure that everyone knows about tris. In that case, it's totally fine to ask about them.

But if you're measuring something else, if a tri is a portion of the problem, some portion of the people, your candidates will be familiar with this, others won't, and that will introduce noise. So in general, I think it's good to stick with the sort of classic, most basic CS concepts. You want to budget about three times the amount of time it takes you to solve a problem for the candidate.

So if you come up with a question and you solve it yourself and it takes you 10 minutes, that's probably a good question for a 30-minute interview session. And the reason here is just that it's far easier to be the interviewer than it is to be the candidate. It's far easier to ask the question. And we tend to downplay how hard the questions actually are. And we did some actual research on this.

We went through all the questions we asked at Trollbyte, and we looked which are most correlated with candidates going on to succeed. And it ends up that the most predictive questions tend to be much easier than what our intuition going in was predicting. So it ends up that most interviewers think that the optimal question is quite a bit harder than the optimal question actually is.

You want to make sure that you ask four or more questions in an interview. The idea here is that each individual question carries a certain amount of noise. Has the person seen this question before? Were they lucky answering it? If you ask more questions, you'll get a more consistent signal out. And then finally, just one tip.

A type of question we like a lot at Trollbyte is a question where we give the candidate a problem. And then rather than wanting them to devise a solution, we tell them a solution to the problem. And then their goal is to take that idea and implement that in the code. And so we give them an algorithm, and we see if they can implement that algorithm rather than require them to devise an algorithm.

OK, the fourth way to reduce noise is to ignore credentials during your interview. So by credential, I mean things like did someone study at a well-known school, or has someone worked at a well-known company? And I'm not claiming that credentials are meaningless. Credentials are important. So they're predictive in any case.

The group of people who are ex-Googlers are indeed better engineers than the group of people who have not worked at Google. So it's totally legitimate to take that fact into account when deciding whether to hire someone. However, it's not relevant to their actual programming skill.

So my advice here is to make sure that you're not biased by the fact that someone comes from Google when you are narrowly evaluating their programming skill. So I recommend that you hide that credential from the interviewers. And the reason is we've found that interviewers are actually pretty significantly biased by this.

If they know someone has strong credentials, they're more likely to interpret the results of the coding screen as in a positive fashion. Oh, this person, yeah, they didn't know that answer, but I'm sure there was a temporary slip up.

And so hide credentials from your interviewers, let them assess programmability, and then when making the final decision in the decision meeting, consider credentials and also the performance in the interview. And yeah, this will help you find the programmers who are skilled and who lack additional credentials. And those are the undervalued people in the market.

And as a startup, if you're good at finding those people, that is a big advantage. OK, point five is that you want to think about the false negative rate in your interview. So a false negative is when someone fails your interview who could have gone on to do the job well.

And the opposite of false positive is when someone passes your interview, you hire someone who then goes on to do poorly and probably be fired. And both false positive and false negatives are very costly. So if you hire a bad person and you have to fire them, that's terrible. It harms the morale of the team. It's also very expensive and there's actual money, big problem.

But if you're a startup and you're really hungering, you're held back by not having enough engineers, if you pass up a person who could have joined your team and could have been productive, that's also very expensive. And I'm making a pretty subtle point here, but I think there's a bit of a cognitive bias where false positives, we're very aware of them.

If we hire a bad person, we feel that pain for a month or month plus, best case. And we've seen companies on Shopify, they generally have too much faith that the folks who fail their process must have been bad engineers and couldn't have done the job. And that's empirically not the case. If you watch, as I said earlier, no engineer will pass all interviews.

So a significant portion of people who fail interviews do go on to be employed very practically at other companies. And so I just recommend that folks designing interview processes think about the false negative rate and try to give that some weight in their calculations of setting hiring bars.

One of my goals for Shopify, actually, is to get to a point where you can actually measure this rate, because no one knows what it is. No one knows what the false negative rate on the interview is, because to measure that, you have to just hire people randomly and see how they perform. That's very expensive. My goal is to get to a point where we can do that. Let's see.

OK, point six is that you want to generally calibrate on the maximum skill that each candidate brings, rather than their average skill or their minimum skill. So if someone comes in an interview, they're very strong in one area or they're weaker in others, what matters the most is what they were strongest in. Everyone can look stupid.

If you ask me the right question, I will definitely look very stupid. That's true for everyone out there. And so the problem is that someone might go through an interview and do very well on some things that would be useful for the company, but look stupid on one question. And if that interviewer gives a blocking no, that that person was stupid, that's introducing noise in the process, perhaps.

Now again, if someone does poorly in an area that is important for the job, totally fail them. But be open to the fact that everyone does look stupid sometimes, and don't fail someone just because they look stupid on one portion of the interview. OK, finally, last point here is that you want to think about the candidate experience when designing an interview.

You want to make sure that every candidate who goes through your process likes your company. And this is true for a few reasons. One is just if they enjoy the process, they have a higher probability of accepting an offer you make. And so this will help in the closing step.

But an interesting point is that it will also actually make your screening itself more accurate, because stress has a big impact on performance. A high percentage of candidates get very stressed in interviews and underperform their actual peakability.

And so some tips here to help reduce stress include just letting everyone bring in their laptop and work in their own environment, their own language, their own tools.

They will be able to be much more productive, much less stressed, and then coaching your interviewers in just some soft skills, so being friendly, providing breaks for the candidates, and when they are doing poorly on a section, training them on how to intercede in a way which isn't too stressful and insulting to the candidate.

Rule of thumb here that comes from the old Jolan software blog is that you want every candidate, no matter how they do, to finish your interview wishing that they had, you know, wanting to join your company. You know, even people who fail your interview very poorly, you want them to end your interview wanting to join and being excited about the opportunity.

And the final point, you want to avoid hazing. So this is rare, but results in some of the worst horror stories of interviews. This is where, you know, an interviewer takes on the role of, you know, a ritual of acceptance into a group. You know, if this happens, it's terrible, and as a hiring manager, you want to stay totally away from that. Okay, those are my points.

I want to emphasize that I'm not saying that you should lower the bar for who you hire. I think if you follow this advice, you will get a more accurate signal. You can then set your bar for who you hire if you want on that signal, but you will still be making, at that point, better hires.

So just to summarize here, I recommend that the first step you follow is to decide what skills matter for your organization, make sure that you're screening on things that you actually care about, and then design a structured interview around those skills.

So come up with ways to assess each skill and structured criteria for the interviewers so that they are, you know, less likely to be biased by outside factors. Then you want to use good interview questions, you know, multiple parts, no leaks of insight. You want to hide credentials from the technical interviewers because that introduces noise in the process.

You want to think about the false negative as well as the false positive cost in your interviews, and you want to generally calibrate around the maximum skill that each candidate brings. While doing all that, you want to try to provide a positive experience for the candidate. So those are my tips.

So far, I think this all applies to both big and small companies, so I'm going to go over a few points here that I think are specific to, let's say, series A and smaller companies. So one point here is, what if you're so small that you don't have the scale to standardize your process? So you're a seed stage startup. You're hiring. You're interviewing your first few people.

You obviously can't run an extremely standardized process because it's the first few candidates to have seen those questions. That's a totally real point. I think it still is worth trying to run structured interviews. So your plan will probably be a Google Doc with some tips written down, but it still is helpful to think about what skills matter and try to design questions that assess those skills.

That will still reduce the bad bias in the interview. Let's say that you're having trouble sourcing. So you're an early stage startup, and your number one problem is, how do you get enough qualified applicants for your company? So in that scenario, it's fairly obvious that a false negative costs more.

So if you're struggling to get people to apply to your company, screening out someone who could have been good is very expensive, more expensive than for a large company. But the flip side is that if you're a small company, hiring a bad person is also way more expensive. So it's not clear yet the ratio of those two cost changes, so I think you still have to care a lot about both.

I think what you can do is be less aggressive about screening folks out early. So if you're an early stage company struggling to source candidates, I recommend you are less aggressive in screening out prior to your on-site. Pass more folks through and accept a lower final interview success rate in exchange for better screening all the applicants.

Let's say that you're a small startup, and you just don't know what skills matter. So you're not sure if you want to hire someone who's very CS focused or someone who's very web focused. And there's no crisp answer here, so we have plenty of examples of billion dollar companies that have taken various routes here.

My personal advice from Trailbite, and from Social Cam, and from Twitch when it was small, is that I believe strongly that the two most important skills for the first few hires are productivity and ownership, so being able to basically take a project, figure out what has to be built, and just make that happen. And I recommend optimizing for that at the expense, potentially, of code quality.

So I think the first few hires at a company, you should accept that perhaps they're writing crappy code, but by God, they're writing it quickly, and they're getting stuff done. Let's just say that was the case during the early stages of all three companies I've been involved with, and I think that's normal and probably good.

So let's say that you're hiring for an area where you yourself don't have technical expertise. So you're a web developer, and you need to hire an iOS engineer. What do you do? So one answer is you can use us, you can use Trailbite. We'll help you do that. You can call in friends to do the interview for you if you have them.

But I think a good point here is that a trick that I've used in the past is to just ask candidates to explain things. So you're a web developer, you ask the iOS engineer a question, they give an answer. You have no idea if it's a good answer or a bad answer. What you can do is just ask them to explain. You say, oh, that's interesting. Can you explain why that's a good idea?

And for the most part, if someone truly is a skilled iOS engineer, they should be capable of explaining their answer in terms that an experienced web engineer can understand. It's not always the case. Sometimes someone will come along who's a good iOS engineer but a bad communicator, and that might fail them. But this is a pretty good trick to hire outside your area of expertise.

I want to end, actually, with an ask for you guys. We've been developing an exercise for interviewers that we use when training new interviewers for Toolbite. And I want to see if I can get you guys to try this. And email me and let me know how it goes, because I'm curious about if this works more broadly.

And the idea is that interviewers tend to significantly underestimate how noisy interviews are. They overestimate their own ability to distinguish good engineers from bad engineers. And part of becoming a better interviewer is to become a little more humble and more aware of your own limitations. And so this exercise is designed to highlight that.

And so what I want you to do is to do a mock interview with one of your co-workers. So have one of your co-workers interview you where they're asking questions. They're the interviewer. And you're role-playing a candidate. You're giving answers. And I want you to tell them in advance that you're going to be giving bad answers to some of the questions.

You're going to be role-playing a candidate who makes mistakes. And then during the interview, they ask questions. I want you to, half the time, go ahead and give a bad answer. Role-play a poor answer to the question. And the other half of the time, give it your best shot and actually try to give the best possible answer you know to that question.

And then after the interview's done, do a debrief session where your co-worker goes over all the mistakes you made and gives you feedback. And what's great here is that they don't know what answers that you gave were intentionally bad, and what were you trying your best?

And if they don't highlight the mistake you made, when you were trying to make a mistake, like they're going to look a little bit bad. So they're fully incentivized to be completely honest and rigorous when pointing out the flaws they saw in the interview.

And the criticism they give will apply both to the points where you were intentionally making mistakes, but also to your attempts to give your best possible answers. This ends up being really interesting. We've been doing this a lot at Trailbite. And it is great at highlighting how interviewers disagree.

So if you do this with your coworker, and you do it a few times, reverse roles, the result is that you will highlight your areas where you're both weak, but you'll also highlight a bunch of areas where you have legitimately strong disagreements about what constitutes the best answer to a technical question. There will almost certainly be more disagreement than you're expecting going in.

Yeah, I found this super insightful. A lot of you guys could try this and email me and let me know how it goes. Awesome. That's everything from me. Harjit's going to talk about closing, and then we'll both answer questions. OK. I'm going to wrap up here by going through best practices and points to optimize your chances of people actually accepting the offers you make them.

OK, I have five main pieces of advice here. The first is just understand, as a startup in particular, speed is a huge hiring advantage that you have over bigger companies. So through Triplebyte, we work with startups and larger companies, and we ourselves are surprised by how often a big company can take weeks just to deliver the actual final offer details.

And when someone's getting to the end of their job search process, that's when they're most keen to just make a decision, move on, and know what they're doing. So as a startup, if you're quick at every step in your hiring process, from the moment you have the first contact through to when you make the offer, you increase your chances of closing a candidate.

So just be fast and responsive at every step. Second, Armon already mentioned, train your interviewers to have a good bedside manner with interviewees. And so we ask candidates or engineers on Triplebyte, what were some of the reasons you had bad experiences when you went and interviewed in-person at companies? And all of the replies centered around their experience with the interviewers.

And the top two complaints were interviewers not being familiar with the technical questions they were asking. And second, the interviewer being determined to show how smart they were, specifically smarter than the interviewee. So if someone has a bad experience interviewing with you, they are highly unlikely to accept your offer.

Three, be prepared to talk in some detail about your company culture. Because almost every candidate is going to ask you that question. What's it like to work here? What's your company culture like? And I think we've seen a particular uptick in this question over the past year.

And I think, frankly, a lot of the stuff around Uber and just a lot of the focus on the culture in the technology industry in Silicon Valley is making people, especially individuals, think more about the kind of culture they want to work in. So two specific bits of advice here. I'd definitely be prepared to answer questions about how you think about diversity.

If your team's already diverse or not, I think you should have some thought around how you plan on incorporating that into your hiring practices going forward. And second, when you're describing your culture, be aware that almost every company, like a significant majority, essentially use three adjectives, open, transparent, and collaborative.

And we've seen this on TrimbleWide because we see them creating their profiles. And so that can be an effective strategy because all three of those things are good aspirational qualities. No one would not want to work at an open, transparent, collaborative place. But it's not very good for differentiating yourself.

So one strategy you could take when you're talking about your culture is take more risk. And an example of risk might be talk about the trade-offs in your culture. So if you believe yourself to be an open culture, talk about the trade-off there. Openness means being honest about things. And being honest about things can be difficult and uncomfortable for people.

So if you're willing to go in that direction, you may alienate some candidates. But you may also really increase your closing rate on the candidates that are most excited about you. Fourth, get your team and investors involved in the process.

So once you made an offer, make sure your team reaches out afterwards, offers to meet up with the candidate again, answer more questions, and generally be involved in their decision-making process. And do the same with investors. And in particular, pick any investors you have who would be particularly good at closing that candidate.

So if they're on the fence about whether to leave a big company for your startup, do you have an investor that successfully left a big company and joined a successful startup? And finally, present full and transparent offers with all the details a candidate needs to understand how much they're being compensated. This sounds, again, probably somewhat obvious.

But I've been surprised by seeing companies engage in this sort of hypothetical battle, where instead of just telling a candidate, here's how much salary and equity you get, posing questions like, how excited are you about us? If we made you an offer, do you think you'd join? And this always creates a bad, awkward candidate experience.

It's also unfair, because the truth is, no matter how powerful your mission is, compensation is a big factor in where someone's going to work. And so asking them to commit before giving them details just puts them in an uncomfortable position and decreases the chances that they would accept your offer to join. So when you do make the offer, provide full details.

This slide has the minimum of what you should put on an offer letter. Salary is fairly self-explanatory. With equity, make sure that when you present the details of an equity offer, so the number of stock options, exercise price, et cetera, that you're also checking in to ask the candidate, how comfortable are they with this?

If they're a senior engineer and they've worked at six or seven different startups, they probably understand this. They don't have to go into too much detail. If they've spent most of their life working at Intel, they may not be familiar with how startup equity works. So ask them how comfortable they are. Don't overwhelm them when you present the offer details if they're not.

Just send them follow-up resources that they can read on their own time. And also, this sounds somewhat obvious, but make sure you yourself as founders really understand how stock options work, because people will ask you questions about it. And you actually don't end up spending much time on this, because you yourself have restricted stock, and fundraising doesn't usually include options.

But you should understand, even down to the level of, I'd say, tax implications of different types of options. Final thing I'm just going to close on here is a common thing that comes up with startups is how do you compete for engineers against the tech giants, like Google and Facebook in particular, right?

So Google and Facebook obviously can offer much larger liquid compensation packages than a startup can. And frankly, the public tech company stocks have been doing great over the last few years. So it's becoming more challenging to compete against them. Still, we routinely see startups do it and win out against much larger companies offering more money.

I think there's four pitches they make that have been quite compelling. The first is emphasize learning. The case I'd make here is that you learn the most, or you learn the fastest when you're given real decision-making responsibility, which means if you make a mistake, bad shit happens.

And there's very, very little chance, practically no chance, that that's going to happen at a big company, right? Like, big companies have checks and balances in place to make sure that no one can create too much damage, in particular for new hires, right?

So if you tell someone that, hey, like, we just don't have that luxury, we're growing quickly, we've got to get everything shipped yesterday, you're going to get thrown straight into making real decision-making authority, and if you screw up, that's going to be bad for the company. And that's the way that you learn and build real experience. So emphasize that.

Two, I talk about career progression, and one sort of macro comment I'd make is, if you look at executives at public technology companies, they tend to be, or often can be, much younger than you'd see at their counterparts at like a public detective or like a public utilities company, right?

And the reason for that is you can grow up and be like chief product officer at Facebook at 38, or however old the chief product officer is, because you joined the startup when it was really early and you grew along with it, right? And the tech industry is really the only place that happens. Like, it's more of a meritocracy.

You don't have to sort of pay your 20-year dues at a company before they consider you for a senior role. You can rise up as quickly as a startup grows. So emphasize that. Third, talk about opportunity cost. And in particular, the way I'd frame this or talk about this would be, at any given moment in time, there's always going to be a basket of big, safe technology companies to go work at.

The names might change sort of every five years or so, but the experience of working at one is largely fungible. Whereas the experience of working at a startup varies wildly, right? So I would emphasize the fact that your startup right now, the team you have, the opportunity is a unique one that's a uniquely good fit for that candidate that they won't get again.

And if it doesn't work out, they can always go back to generic big company. And final point, this one's specific to hiring more junior candidates, talk about mentorship. So one thing that we see, especially for new grads, people straight out of college, is that they're worried they won't get the right level of mentorship. They won't learn how to best practices work if they join a startup.

And maybe they should go work at Google for a couple of years, get that under their belt, and then join a startup. I mean, first decide if that's actually true or not, right? Like if you yourself have only junior engineers on your team, and frankly, you just don't want to mentor or get someone up to speed, then don't make that case and maybe don't hire someone like that.

But if you have experienced engineers on your team, emphasize that and emphasize the fact that they will get to learn alongside experienced engineers who have worked at places and learn best practices, et cetera, et cetera. So that's everything. I think we're gonna open up for questions. I will ask the questions about the topics we covered. Do you wanna come up? Okay. Thank you so much. Yeah.

So, this is obviously a more engineering and tech problem discussion, but can you maybe talk about things like background checks and things that totally rule out the candidate. Like background, things that rule out the candidate? I mean, background checks or. .. Background checks, frankly, most startups I know don't do them.

They're especially easy to do now, though, because you can integrate checker in with your payroll service, so sure, but that's not something that I would focus on. I'd focus more on, like, is someone gonna be a good fit or not? Yeah. Yeah. I have a computer science engineer who wants to intern for me.

I don't have any engineers on the team, but even given that I only need to get a, like, three credit time, what would you do? That's a hard question. So, the majority of computer science juniors are probably not gonna be doing a particularly good job of doing the work you want. That said, this sounds like a case where you could probably just evaluate past work this person has done.

So, you could probably ask this person to give you a portfolio or other examples of work they've done, and if that is at a level and of a type that you're happy with, hiring them would probably work. Is there any metadata you could actually use to help make a better decision when it comes on to, like, hiring people that have higher grades or probably a Geek of Legends ELO score? Yeah, probably.

So, the thing we focus on is background-blind hiring. So, we focus very narrowly on just direct skills assessment rather than looking at other things that correlate and are predictive. So, you know, personality type is real. People vary in their personality. Some of that stuff matters for forming jobs.

I think there's more research on, like, the Big Five, rather than, the Big Five personality traits rather than Myers-Briggs. All relevant, but I would try to keep it pretty separate from technical assessment.

I'd say assess technical attributes, and then, you know, separately try to assess, you know, culture, friendliness, soft skills, and then combine that all, and then try to make a global decision. Yeah, sorry, back in the sunglasses. So, okay, I'm a, like, a non-technical commenter, right?

So, if I want to find the right technical, I know you guys just went through, and you guys told us, like, you know, the ways that you can test, or have an engineer on board, and give them, like, a talk, so, but if I'm non-technical, how can I engage if the person is actually qualified if I don't actually know how to do this? That's the extreme version, and I'm just gonna say you probably can't.

So, you, oh, yeah, so the question is, if you're a non-technical founder, how can you assess an applicant's technical skills? And I think the answer, unfortunately, is that you probably can't.

So, you can fall back on the same advice of trying to look at past work, but I think this is why it's very helpful to have a technical co-founder, is because you are gonna be at a fundamental disadvantage there. I mean, as far as finding a technical co-founder, I think that would be the dad part, to know, like, they're the right person to be able to do that.

Yeah, I, yeah, you have your thought there? I was like, so the question is, how do you find a technical co-founder and figure out if they're good or not? I think sort of Armon's advice mostly applies here. The most thing is that even if you, use a friend or someone that you know who is an engineer to talk to them and figure out if they're a good engineer or not.

Otherwise, there's not really any other options. Yeah. Yeah. Apparently, I'm not precise with my pointing. Need to work on that. A question about talent generation, so like a step before sourcing.

You know, is there any methods around kind of working with universities and doing project-based challenges with them or some kind of mechanism for sourcing that might be a good way to kind of build an account rather than sourcing? Yes, but I think this is largely focused, oh, sorry, yeah, repeating the question.

The question is, are there ways to work with the universities and colleges to generate like a pipeline of talent that you could hire from? Is that accurate? I would say, yes, there certainly are, and they exist, like career fairs and sponsored events at colleges, et cetera, et cetera.

But I think those probably work better for larger companies that can fit themselves around the college graduation deadlines and that schedule. For startups, it's kind of tough because usually you want someone that's available right now to hire as opposed to waiting for them to graduate. So I probably wouldn't recommend investing too much time in that as a startup.

And just to clarify, big companies at this point make offers a full year in advance with large signing bonuses delivered a year in advance. So it's very hard for startups to compete with college hiring. Question? Yeah. So you talked about role-playing a bat in your college career, so why a bat is good other than just basically face-saving in front of your friends and making mistakes.

Yeah, so the question is, for me to clarify on my last point about role-playing a bat interview. So the key point here is that you're gonna role-play an interview and the coworker who's playing the candidate who's giving the answers, they're going to intentionally make some mistakes. And what that does is it frees up their coworker to be critical.

So normally, if you say, if you do a mock interview between two coworkers, the problem is that in the end, everyone's gonna pull their punches. They're not gonna honestly give their opinion of the performance that their coworker gave. It just doesn't quite work due to all kinds of social pressure.

But if you tell them, I'm gonna be intentionally making some bad, I'm saying, I'm gonna be intentionally making some mistakes, that frees them up where they're then free to point out the flaws in your interview. They're not only free to, they have to. Because if you say, I'm making mistakes, and you ask them, what's your opinion of my results?

If they don't point at a mistake you made, that means you got something past them, right?

So it flips the social pressure and creates pressure where they're then incentivized to be critical, which is extremely useful because as experienced engineers, like it's been a while, you're a senior engineer on a team at a successful company, it's been a while since you've had someone really brutally critique how good you are at answering your own questions.

And that, what will happen is that they will point out legitimate flaws. You'll be trying to give your best answer, and they will point out things you said that are wrong or things that could be better, and that it's very humbling. It's very humbling, and I recommend that.

I think this is an exercise where you can give yourself that experience of being humbled and having flaws in your own performance pointed out. Okay guys, sorry, due to the timing, just one more question, please. In the middle. Yes, got it.

You talked about a 95% plus success rate on one of the slides, and I, given the, the fact that the interviewers tend not to have a very high interview rate or reliability. It seems hard to believe that there's a 95%, and it also doesn't match my experience, that 95% of the hires were successful.

So, what do you think is the real success rate where, you know, a year after hire, somebody's still there and they're a top performer? Yep, that's going to be, so that number 95 comes from some surveys we did among companies. Oh, okay, sorry.

So the question was, in one of my slides I said that hires are successful 95% of the time, but that doesn't mesh with, you know, the experience of being a hiring manager, hiring people. Many people you hire are not great. So the question is, what's the actual rate at which, you know, candidates end up being top performers? And, yeah, the question is totally right.

So that number comes from some surveys we did where we asked companies, of all their hires, what percentage got fired? We actually asked what percentage are top performers, and that 5% is the fire rate. So 5% actually get fired. You know, an additional 30% are people who stick on but are not really great employees.

And then about 10% are in the bucket that that company said were their best hires and top performers. Thank you.