Speaker 1: 0:00

What predicts the success of your business is how many times you can do that experiment. You have to be incredibly open-minded about what the world wants versus what you want to build.

Speaker 2: 0:22

Hello and welcome to another episode of the Accelerometer. I'm Dr Matthew Merman, CEO and founder of Anarchy. Today, we're going to be discussing AI programming languages and dev tools with Nicholas Carrier, the founder of Axflow, a TypeScript framework for AI. Nick, I guess the first question I'm wondering is why we need this. What's happened in the?

Speaker 1: 0:43

last few years is that AI went from being a niche domain that very specialized, dedicated data science teams would do on very specific domains like self-driving or ad ranking. It's become like this commoditized engineering practice that can be introduced in any software company, and so we're seeing this. We're seeing with the release of ChatGPT and Midjourney, and like the heightening of delays, of abstraction, it's becoming easier to use and so we're seeing teams in every single company prototype new features and systems with AI. So because of this, we want to make sure that the interfaces and the abstractions match that new developer type. It's no longer the same one.

Speaker 2: 1:24

What's wrong with the existing abstractions, like the Python, for example?

Speaker 1: 1:28

So I don't think there's anything wrong with them per se. They're very useful, and for 20 years we've had like kind of fantastic tooling built in the Python ecosystem for ML, but there's a shortage of people, of ML engineers that are familiar with PyTorch or the complex kind of. They're lower level abstractions, so it's not that it's wrong. It's that it requires much more deep AI knowledge to manipulate, and really now AI is elevated to a product engineering level. So it's kind of like giving a product team a new magical API that can do cool stuff. They don't need to understand anything about how it works, but they do need to be able to manipulate it, and these teams, more often than not, are writing higher level languages. Pyp script is a very common one. Like at Facebook, the product team is writing hack, like a version of PAP. Even if it is Python, it typically is a different layer of abstraction where things are easier to manipulate, and so it's less lower level primitives and it's more higher level actions that you can take.

Speaker 2: 2:23

So can you give me an example of some of these actions?

Speaker 1: 2:26

So I mean at a very high level. It's the difference between manipulating a model transformation pipeline with PyTorch and saying I want to generate an image and I'm going to describe it to you in English language, and so that latter is just a very different way of interacting with the AI system and therefore it requires different interfaces. This is all changing very quickly, by the way, like multi-modality kind of shows this. Historically, a lot of models are like single modality and we're seeing, like at the time of release of this, chatgpt is just about releasing, like, image plus text. We expect maybe more in the future, and so with that comes a need to evolve these interfaces even more so that developers can easily upload images alongside with text, alongside with, maybe, audio and video, and then merge these in their application. And so that's why we need to kind of invent or improve the interfaces to make all of these developers more productive Can we do things like train with your systems.

Speaker 2: 3:21

No systems.

Speaker 1: 3:28

No, so what we're building is really because our target customer is this higher level product developer. They're not really interested in training. Actually, the industry has kind of stratified in such a way that the models are being trained by single companies, single entities that have all of the experts. So this is like OpenAI, midjourney, anthropic, and then at these companies, really all you do is you either use a foundational model directly and then you might-tune it, which, by the way, you don't do this with the lower level pytorch primitives. You do this with an api. Openai has built an api. You post about you http, post a bunch of examples, you get a new url. It's your fine-tune model. So they they have to understand a lot less about how it works and they're given these like higher level primitives. And that's what our framework wants to expose. Because our target customer is these high-level product developers that care much more about business value, we try and abstract away so that the AI system is very easy to reason about and manipulate.

Speaker 2: 4:15

How did you get started working on this?

Speaker 1: 4:17

So I've worked on ML platforms and systems for like six years. At Pinterest, I was on the protect team doing anti-abuse real-time systems, so preventing bad stuff porn, spam, fraud from appearing on Pinterest. And then at Cruise, I built a lot of ML tools and platforms for offline self-driving and I always found it very fun because it's like this problem where the theory has been there for a while and it's really exciting, but bringing it to production is incredibly hard. We had self-driving cars driving in campuses of Stanford in the 80s and it's only in 2020 that we had them in the streets. That's because the chasm between prototyping ML and putting it in production is just really, really large and I like that challenge. I found that the gap is like pragmatic, just software engineering building tools that make engineers productive and allow them to build systems that they control and can manipulate and have confidence before they deploy to production.

Speaker 1: 5:07

I built this at Cruise. It enabled self-driving. That was all very exciting. And then when the LLM wave, when ChatGPT happened, it was just super exciting to see everybody get excited by AI and want to add it to their application. And they all hit the same thing, which is they built a demo. It was super excited about that demo and then nobody was able to go from like 80% to 95, 98% and I could see that need that we built tools for at Cruise and Pinterest, like all over the place.

Speaker 2: 5:31

So the tools that you were building at Cruise I think this is a particularly interesting example because we're like in a self-driving system you need your AI, your models, very close to the robotics, like physically on device, on edge, literally running on the car, right Yep, and in this case, with these models, most of the time people are working with APIs, like you said, yep. So the tools that you're using, they must be quite different in many ways.

Speaker 1: 6:05

Yeah, so they are different and they're also the same and it depends on how much you zoom in or you zoom out. If you zoom out enough, they're very similar because you have this thing that I like to call the data loop. You have production data in self-driving it's cars driving around and then you want to somehow take that data, do stuff and then improve the system somehow and all AI systems have this loop and and then improve the system somehow, and all AI systems have this loop. And this loop goes through creating a data set, maybe labeling that data set, maybe retraining or fine tuning, then evaluating and then redeploying, and so everybody has that loop. So in that sense it's very similar.

Speaker 1: 6:33

But to your point, at Cruise, it's such a hard and deep problem that they want to control every single aspect vertically. So, cruise, but they've still abstracted things away. So at Cruise, ml engineers are still working in PyTorch, which is kind of high-level Python I was calling it low-level before, but compared to GPUs it's high-level right and they're training models and writing the model architectures. And then we have another team that is called Acceleration that takes Python code and it kind of compiles it down to CUDA, which is like the NVIDIA language and optimizes it there. But you see how what we did there is build tools to make each different team as productive as we can, and so we also had to build labeling tools, evaluation tools, dataset management tools all of the steps I mentioned we built tools for and then integrated them together.

Speaker 2: 7:14

So what's your favorite part been of running a startup?

Speaker 1: 7:17

So we've been doing this since June. By far my favorite part is why I wanted to do this, which is like extreme accountability and ownership. So I find that I do my best work when I'm 100% accountable and I cannot pass the buck and complain. It's just something's got to get done and I have to do it, and that's been like. It's just incredibly empowering, and empowering. That means very motivating and that means I work hard, and when I work hard, I'm actually happier, so so it's this virtuous circle, so that's my favorite part.

Speaker 2: 7:45

Oh, I love that. I feel very similarly. When I was working for, you know, apple, I found it very hard to motivate myself to get up in the morning because you know it's hard to know if what you're doing actually matters at the end of the day Agency. How do you know that what you're doing?

Speaker 1: 8:02

matters, who does it matter to, is like the way to frame that, like as long as it matters within your value system, as long as you care right, that means it matters to you. I think that's enough. And then the goal is of every startup is that, like other people care and pay and you can make a great business. And that's like external validation, which is even better. But in my opinion, if you can build something that you're proud of and you're happy building, you've already kind of won. And then there's like a next level of winning, which is like the public, external validation.

Speaker 2: 8:28

Do you feel like you have that public external validation?

Speaker 1: 8:31

No, no, like at YC we call this PMF. We don't think we have PMF. We think that the problem we're solving is really important and people have it. We know this because we validated the problem by talking to lots of customers. But we don't have the traction. True pmf is like so blindly obvious that you can't miss it. You know chat gpt is. This is like it's down all the time. It's terrible interfaces, they break it and everybody uses it all the time. That's a true pmf. And your graphs look hockey, hockey, sticky, etc. So we don't have any of that. We do have pockets of very encouraging feedback, like we have users organically telling us parts of our framework that they really like and that they think we're really thoughtful and we put a lot of effort into them. So that's really rewarding and people organically sharing to their friends. So that's great, but it's not quite PMF.

Speaker 2: 9:14

How do you?

Speaker 1: 9:14

think about going about and finding PMF. To me it's all about. This is actually something that very much ties back to a lesson I learned at Cruise you want many, many shots on goal. So at Cruise, like the whole in an AI system, let me just tie it back to AI. In an AI system, it's a non-deterministic system, and you want to try things and see how the global output of the system is and if it's better or worse, and actually what predicts the success of your business-like value is how many times you can do that experiment, and so I think that in a startup, it's the same thing. So it's all about velocity how quickly you can run a hypothesis experiment result loop, which typically is like the customers want this. That's the like, the experiment goal, and so that's what we focus on a lot.

Speaker 1: 9:55

How can we build things quicker? How can we validate faster? Can we get an answer without even building anything? And when we do build, where should we spend time? My co-founder and I spend a lot of time debating this right, because time is the biggest constraint we have. Where do you spend most of your time? It's changed. Right now we're in a phase where we're building a lot, so 80% of our time is building and that's actually super fun, but before that there was a lot of customer interviews. It was much more like get external validation to confirm the problem we're solving and also get some early partners to make sure that they will integrate as we build what they want. So we're not building in a vacuum when we confirmed it before, and so those different phases had very different time schedules for me.

Speaker 2: 10:33

How did you go about getting those early partners?

Speaker 1: 10:37

I don't think there's one trick here. I actually think that the amount of energy you put in directly kind of affects how many you get, because you should try and target as many channels as you can. So I have networks like the yc network was helpful. My network of my friends, of developers, of companies I've worked at, was helpful. Uh, reddit has been helpful.

Speaker 1: 10:53

Slack I go into ai slacks or discord communities and I talk about things. I go to events, try and make friends and do like word to mouth. So I try everything, yeah, which I think is the only way, because I think it's a bit of a hit industry. Like you want to hit, you want like a perfect symbiosis partnership where you learn a lot from them and you provide a lot of value for them. And when you find that I've seen this before in previous startups that I've been part of when you find that like you get this like virtuous circle of motivation and success and so I'm kind of chasing that at all times, what is the most interesting thing that somebody has used xflow for so far?

Speaker 2: 11:27

I?

Speaker 1: 11:27

think the use case we like the most is we have a company called code story that's building like they're doing it, they're forking vs code and they're making it like an ai editor. They want to change the way you code, and we like that use case because so we're typescript right, and so the reason why we're building typescript is mostly because a lot of people are building in TypeScript for the web, targeting the web or mobile with React Native. But these guys are cool because they're targeting like a TypeScript runtime, which is VS Code, which is like native apps, which is pretty rare, and so we learn a lot from what they're doing and we also believe a lot in their product. Also, because they're doing pretty intense code understanding and generation. They need to be able. Our framework needs to be very powerful, which is what we want. We want to cater to hard, deep problems as opposed to easy, shallow, zero to one weekend projects, and so that's probably my favorite one.

Speaker 1: 12:13

So what do you mean by powerful? I think I mean the layer where the layer of abstraction is. So maybe let me give you an example you could have a high level of abstraction where all you do is describe what you want. Openai's new GPT is this. It's like you are a code interpreter I will give you some code and you will give me answers. That's very easy to set up, but you can't really edit it and make it. If you're iterating on that problem, you want knobs and things you can manipulate, and so it's too high right. And so when I say powerful, I mean you can manipulate, and so it's too high right. And so when I say powerful, I mean lower, a lower level of abstraction, and so this is a system where developers can tune, they have a lot more control and they can build a lot more of that deep expertise using the tool, as opposed to using a high level tool.

Speaker 2: 12:57

When you said you want to solve these deeper problems. Maybe you can elaborate a little bit about what those problems look like deeper problems.

Speaker 1: 13:05

Maybe you can elaborate a little bit about what those problems look like.

Speaker 1: 13:10

Yeah, actually, my co-founder and I spend a lot of time talking about who our customer is, and so we think our customer shape are product teams who are mostly developer.

Speaker 1: 13:14

First, like it's mostly developers maybe an engineering manager, maybe a PM that wants to also be involved and they're solving what we call a non-trivial AI problem. This means that it's going to take them months or maybe even years to solve it and they're going to spend a lot of time on it, and we specifically care about this for the reasons I mentioned before, which is it's very easy with, like the AI APIs to get an impressive demo. It's very hard to get a reliable, robust, rich system that can understand complex real world problems. It's like with self-driving you can drive on a fixed track really easily, but deploying cars in San Francisco, where you have an infinite long tail of unique scenarios, is incredibly difficult, and so we want to build tools for those people. The tools you need when you're solving for the long tail are very different tools than the tools you need when you're capability first trying to do a proof of concept.

Speaker 2: 14:09

And so we're very much focused on the long tail hard problems. So what's been something that you really wish you could solve but you haven't been able to address yet. So that one's easy.

Speaker 1: 14:15

It's multimodality. So I just deeply believe that the future of these AI APIs is multimodal. I think that the reason why human intelligence is so fun is that it's multimodal, is that you can feel and touch and smell and talk and hear, and I think these models are very clearly going in that direction and that's where we will end up. I don't know if it'll be six months, two years, 10 years. It will probably be faster than I think, so probably a year or two from now. The problem with that is very hard to design APIs for multimodality. The problem with that is very hard to design APIs for multi-modality. Let me give you a basic example. Tokens come in slowly, so we use streaming. You can stream tokens.

Speaker 1: 14:51

What does streaming an image mean? That's super unclear, and so do you make one response for each modality? Do you have one request where you're sharing the payloads, like when you get into the weeds? It's actually not at all obvious how to architect this. Do you have the model that knows about multiple modalities, or should the framework axflow help route to different, specialized, single modality models like this is a really important question and I don't have the answer yet, um, and we're paying a lot of attention to try and figure it out as fast as we can I think mid journey has a good solution to streaming, where it gives you like a partial and or like a very fuzzy image and just resolves the image, as the diffusion would resolve the image I think they do a good job, but they're single modality, so how would you combine that if it was responding text as well, like what would that interface even look like?

Speaker 1: 15:36

it's not obvious, although with mid journey, I guess the discord bot could be streaming text at the same time. Um, but you add, add one more modality audio and all of a sudden are you supposed to synchronize them? Is that concept of a tick like these are all questions that get difficult did you see the uh paper?

Speaker 2: 15:55

it was just posted in an ai chat but the paper that was withdrawn that talks about diffusion models for code generation, I did not see it. No, I'm wondering like if you, if mid journey, can have a diffusion model for uh images and that resolves like part by part, what if you have the it for images and text and you like resolve the text and image at the same time, part by part? First out with like nonsense text and every time you read it it's a little bit clearer.

Speaker 1: 16:26

It's it's definitely possible. I mean, uh, it's hard to know how these models work internally and there are different methods, uh. But you see how this is an open research problem, like I'm kind of eagerly waiting for this research to stabilize. So my job has always been to let the researchers do their thing, figure the hard problems out, and then I help bring things to production and it's not clear what the right answer is. I think it's very much an open problem. Multi-modality is very open.

Speaker 2: 16:48

How do you decide when the research is solved enough that you can start implementing it and then you can start helping people work with it?

Speaker 1: 16:55

Yeah, this is not an obvious question. So I'll go back to Cruise for a second. Cruise was very much an R& right so at all times. For example, we had a car where the architecture was single modality models, but there was always a team working on a multi-modality alternative For them to prove that they are better. You typically do what you do in any non-deterministic system. You kind of come up with a set of metrics or benchmarks, some kind of scorecard where you can compare things, and you try and prove that you're beating it on the scorecard. Invariably, the problem is people debate the scorecard. You know the problems with benchmarks and LLMs, for example. They're a very unpopular way to measure performance. So in practice it's just as much influence peddling as it is actually technology peddling. If you're open AI, you can set a standard that others can't, even if it's worse at this point. So there's no right or wrong answer. It's more like where's the community going to go? So you went through.

Speaker 2: 17:48

YC right. Yes, Did you enjoy it.

Speaker 1: 17:52

I really did. I moved to America because of Paul Graham's essays, so this has been something on my list for a long time. How long ago was that? So I was in college in 2009 to 2012, doing my master's in Paris, and I was going to become a consultant. I had a standing offer for BCG and I started reading Hacker News and Paul Graham's essays about why you should build a startup and why it's important to build, and it was incredibly inspiring and my goal has been so.

Speaker 1: 18:19

I moved to San Francisco as soon as I graduated and my goal was to start a startup right then and then, but I didn't have the green card yet, so I joined YC companies instead, and then I kind of always found a good reason not to do it. Like, one of my YC companies got acquired by Pinterest, and that was a fun, high-scale place to work at. And then Cruise was a bit of a difficult opportunity to pass up because it's a kind of generational problem to work on get to go from humans driving to robots driving like I'll be able to tell my grandkids that and that's really cool. Uh, it's a very engineer, engineers, engineer company. Um. But as soon as that finished, I was like, okay, it's time, and so it's been 10 years, but I'm finally doing it how did you know when it was time for you first?

Speaker 1: 18:59

I stayed there four years and by the end my job was. So I was in ic but my job was no longer to build. It was to spend a lot of time mentoring, interviewing, doing design reviews, helping other people do design reviews, and just being in meetings and that kind of burns you out. It doesn't. I wasn't burnt out in the in a true sense of like I was miserable, but it was very clear that I was. There was a cost opportunity of my creativity and so, like I kind of my boredom kind of pushes me out. This is typically what happens.

Speaker 2: 19:31

Have you always been certain about your startup?

Speaker 1: 19:34

No, not at all. I think you I mean my stance is you have to be incredibly open-minded about what the world wants versus what you want to build. So this is back to like. Really, what I would like to do is build a culture where, one, we enjoy ourselves and, two, we put out experiments as quickly as possible so that we can do this for a long time and get many, many I'm optimizing for shots on goal and so I'm not certain. Right now we're actually releasing a new. We're staying on the same problem, but addressing it and tackling it in a different way. We're building like a SaaS UI for this called Studio and like that's actually the way I like to do things is I like to fall in love with a problem or pick a problem that's deep enough and interesting enough that I know I'm willing to work on this for years and then be very flexible on the solution space.

Speaker 2: 20:19

So is this the uh first uh pivot that you guys had? Was it always this?

Speaker 1: 20:26

It wasn't the first pivot. During YC we pivoted so we joined yc with one week of like. We're going to do this like. There was no history of the company behind us. We were brand new and I applied to yc with a, an idea that was similar to what I was doing at pinterest, which is like a policy enforcing engine for llms, like a middleware that would sit between llms and any application to make sure that you could enforce the moderation policies of that application, which are very application specific. I still think that's a pretty good idea, but I couldn't validate it. Nobody needed this today because nobody's in production, so nobody has policy enforcement issues yet. I think they will have them, but because I just couldn't validate it, we moved away.

Speaker 1: 21:01

Pivot number one and then we started building something similar to Axlo but in Python and then doing a lot of customer interviews, we realized that actually these teams building AI are majority TypeScript teams and there's a big gap in tooling there. So that was another pivot. We kind of switched to TypeScript and then we pivoted our strategy from being top-down, trying to be sales first, trying to find big sales deals at larger companies. We switched from that to open source and being for developer first, because that's our DNA and we feel much more comfortable. And now we're going, we're moving slightly from like the framework to maybe a UI application that it actually is very complimentary to the framework because it generates the code for the framework for you. But you have this GUI where you can jump in and out of to do troubleshooting, and so the onboarding flow to get new users is very different. You can try it out in a second in a visual way which is much, much lower of a barrier to entry than like understanding a GitHub repository, and I expect there'll be more pivots.

Speaker 2: 21:56

So I think that first pivot that you were, or the pre-first pivot what you got to YC is very interesting because of what's been going on recently with the executive order. Yeah, have you ever thought about including these sorts of features in your current framework?

Speaker 1: 22:10

Yes, I actually think about it constantly. I think eventually the idea will come back because it has never left me. So I think it's going to be very important To give the background here. Social networks all have this problem. When you have user-generated content, that means you don't control the content. That means there's abuse. Just that's what happens, and so you need to build these anti-abuse real-time systems. It's important, and Facebook or Meta has a great system for this. So does Twitter, so does Pinterest, the one I built, and not a lot of other companies do, but all these companies, they have to solve this. It's an important problem.

Speaker 1: 22:40

I think LLMs are going to have the exact same problem. We have content that's being generated in a non-deterministic fashion, and you want to control it. You want to make sure that the tone is right, that it's not leaking PII data. There's many, many things you want to make sure, and so I think eventually this will be a problem. Regulation would exacerbate it, but even without regulation, if you're building a tutoring app for kids or a dating app that's using LLMs or like a corporate search, you have incredibly different moderation policies and therefore, like you can't expect OpenAI's moderation or fine-tuning to be right for you. You need to have your own, and it's not enough to fine-tune, you want to guarantee it, so there needs to be a middle layer that's catching bad stuff. So I think it'll come back and I would like to put it into the framework for sure.

Speaker 2: 23:21

What's your vision for the future of AI?

Speaker 1: 23:52

no-transcript assumed and normal and part of the world. That being said, there's one exception, which is there might be these very, very powerful and expensive oracles things like GPT-10, that are very expensive and know a lot and are maybe making some big decisions, and that that's maybe the agi question, which I. I don't love that debate because I think it's very theoretical, but that might still always be called ai and I don't really know where that's going.

Speaker 2: 24:20

I think time will only tell how far does this concept of us like, uh, rebranding ai actually go, though? Because I feel like we've gotten to this point with chatbots. That was always in my head what I thought AI would look like, and now that we have them and they work, I'm not like well, that's not AI anymore. At least, I don't feel that way.

Speaker 1: 24:44

Yeah, I think that's right, but I think what might happen is that it won't be. If I'm building a chatbot leveraging LLM technology, it probably won't the best product probably won't be all LLM based. So I'm typing in a text box and the LLM is making all the decisions. Rather, that's probably like part of the project that your software developers have an API and they decide to sometimes use it in like one area in an if clause, like this is, and they decide to sometimes use it in one area in an if clause. This is a hard sentiment to pass. That's used the other day for this, but if we can control things in a deterministic manner, we'd rather do that, and so I think that that's why it will be harder to call it AI, because you'll call it the great chatbot. Maybe the chatbot has a brand name and you'll associate that brand name with the service, but not AI.

Speaker 2: 25:30

What are some of the challenges that you're facing on a day-to-day basis currently?

Speaker 1: 25:32

I wish there were more hours in a day.

Speaker 2: 25:34

That's my biggest challenge um, there's a startup out there that wants to, you know, work on slowing down the earth.

Speaker 1: 25:41

Yeah, I would buy I would buy this product, it would have to end up with me, uh no, I think the hardest thing is context switching and prioritization. So, like I have a lot of ideas, our customers want a lot of things, we don't have time to do everything and prioritizing is hard and context switching is costly, and so we have to be extremely disciplined, and we do this with like a semi-rigorous framework of like one to two week goals to make sure that we don't get too distracted. But we're building something and it's going on for two weeks and death day happens and opening eye releases exciting new stuff. Like do you get distracted and try and participate or do you keep building?

Speaker 1: 26:17

These are the kinds of things that are actually difficult, particularly in AI. It's moving really fast, but I'm of the opinion that you can't build anything like good without spending lots of time on it. It's important to focus, and so this is really hard. As a startup, you can't spend. It's not a phd. You can't spend six months or years on something. You have to be ruthless in your planning to make sure you have just enough time to focus, but not wasting time and doing the wrong experiments.

Speaker 2: 26:41

You guys are an open source product, right? Yes, do you ever get the community to help you with that?

Speaker 1: 26:46

we. We have had a couple contributions not enough and this is something that we're looking to like kind of improve. I think the AI open source is just very crowded, so there's a lot of noise and so we're kind of competing for attention. But the people that do use us do contribute back and are very helpful. They also give us a lot of feedback in private or public slacks. We have a slack channel for the community and that's been super helpful, but we're looking into ways to engage the community more and we're considering doing bounties.

Speaker 2: 27:20

Well, that's wonderful. I found the bounty programs to be incredibly successful. You know, I was surprised early on. I was like, are people going to like this or not? And then, yeah, everybody that was messaging was like thank you so much. This is wonderful. I feel like people don't have an easy way to get involved in open source or anything you can do to help people get involved and help people learn. It's primarily about learning.

Speaker 1: 27:45

Yeah, I think that's a really good tip. I also think that doing the bounty forces you to do another exercise, which is to create the right tickets, which I think if you do that, it actually also helps. When I've done it, I've maintained open source products in the past at Pinterest and previously, and when we did the work and created the tickets and tagged them, the contributions went up dramatically.

Speaker 2: 28:06

You know, actually what I did was I didn't have time to create tickets, so I was like you know what? I'm gonna create a bounty for creating good tickets. People came up with good ideas recursive bounty solutioning. That's great, yeah, yeah, I mean, this is how things worked at apple too. Like everything was a ticket internally. Like you wanted your door fixed on your office, you would create a ticket ticket. Somebody would triage that, send it to maintenance. Like you needed to get an exam at the doctor, would you send the tickets through iMessage? No, no, they have an internal tool. I don't remember what it was called.

Speaker 1: 28:39

That was so long ago, because I was told that this is just an apathe. But I was told that you guys use iMessage to communicate with an Apple communicate with an apple, which seems crazy to know.

Speaker 2: 28:52

We I mean, we're an open source team, so we might have been weird, but we used, uh, irc. Oh, I love our ilc. It's been so long since I even forgot. I think that was the last time I used irc.

Speaker 1: 29:01

Yeah I still it's still got the best dev support communities you could still go talk to like postgres core maintainers. They respond on irc in a way that you would not get them to respond anywhere else. So the best open source projects are still, in my opinion, very active on IRC. There's this barrier to entry that prevents like city questions from getting there.

Speaker 2: 29:18

Oh, I love that. Yeah, I think I'm going to move my Discord community to IRC.

Speaker 1: 29:22

Exactly Discord great UX, but that has downsides.

Speaker 2: 29:26

Yeah, you don't want to have too good a UX. I mean, what we just did was we created separate levels for, like different like types of maintainers, people who have proven themselves a little bit.

Speaker 1: 29:35

But IOC is the perfect filter for developers that you want to contribute, because it's like technically hard to get there. So you're putting a bit of a barrier to entry there.

Speaker 2: 29:44

I don't exactly want to put barriers to entry. I want to help people. I want to help people like learn how to, but I guess you need a barrier to entry, like for for them, for like, would you learn anything if you didn't have barriers? Well, so I am.

Speaker 1: 29:57

I couldn't agree more with you. It's just uh, if the barriers are entry too low, you encourage not learning but just asking for the solution. Right, learning uh kind of implies trying, uh, and so that sometimes that's missing, and nowadays is support slacks and discords. People haven't tried anything, they just come. They're like here's my stack trace, it's failed. What should I do?

Speaker 2: 30:16

yeah, that's true. We did get a lot of people, like you know, submitting one line, a solution, or one line, uh, a pull request, that like they were changed like one character. Like okay, yeah, you technically made a pull request, but like they were changed like one character. Like okay, yeah, you technically made a pull request, but what have you learned? That's not of the spirit of this project. Yeah, so you, uh, you previously said your core or your dna was open source. Can you tell me a little bit more about that? Yeah, so, I guess, a little bit of background when?

Speaker 1: 30:48

Yeah, so I guess a little bit of background. When I grew up, my father was a hobbyist developer his whole life a chemist by training, but a hobbyist developer and he kind of taught me about free software and so he was like giving me lectures about free software when I was 10 and the importance of Linux over Windows. So it was kind of baked into me and I just think that technology is what it is because of open source, like it's straight up, it's helped us collectively achieve amazing growth and technical capabilities by sharing what we have, and so I believe deeply that it's the best way. It's kind of like research is better when it's open. It's the same idea Code and technology is better when it's open, and so I've always believed that I also have.

Speaker 1: 31:31

I've generally liked the people in the open source community.

Speaker 1: 31:33

So when I was younger and I was an engineer like and I was trying to let you just described it I got some mentors who I never met in person, right, but just would help me through tickets and I was always very impressed at the time they took to explain complicated, complex concepts as soon as I showed some motivation to help, and so I always grateful and so I've always felt that like I owe it back.

Speaker 1: 31:52

I've always felt that way, and so I'm very happy to contribute, which is why, at Pinterest, I was on the open source committee. I was maintainer of like PyMemcache, snappass a couple of the open source projects. I have a bunch of things on my GitHub that I've built over the years by default. When I I make it public, you know I don't believe in hiding your code. I don't think anybody cares, so most of the time it's silly, and so my co-founder is similar to me, and it was very obvious to us that we want Axflow to be very open source first. So even this Studio UI app I just told you about, we're going to open source it so you can run it yourself if you want to. That's totally fine.

Speaker 2: 32:34

We don't think that that kind kind of prevents much in terms of our company's success. If it has PMF, we'll manage it anyway. I mean, I think this is wonderful and I'm going to ask you the question that literally everybody has asked me since the beginning of time which?

Speaker 1: 32:41

is how do you plan to monetize? I, right now it's not a huge focus. We've picked, we're building a startup with the venture-backed model, right. So really, we have money to survive for a few years and we've picked a problem and we're tackling it. Really, what we're trying to do is build something people love.

Speaker 1: 32:54

In my opinion, once you've built something people love, it doesn't matter if it's open source or closed source. There's a path to monetization and typically it's not even very hard to find it. If you haven't built something people love, you could still monetize, but it's hard, but that's not what we're trying to do, and so we're just going to be ruthless in finding that hockey stick graph and when we have it I don't think it'll be. There's plenty of examples out there open source projects Mongo, docker that have maybe had their issues, but they've done great overall. It's very possible and I believe that you can typically have an open source component and then a closed source component. That complements it beautifully, for the people willing to pay. Enterprises have very different requirements from developers trying things out or the one to learn, and therefore you can kind of give both of them what they want and have a business and open source at the same time.

Speaker 2: 33:39

I'm very similar Like when I was growing up I was contributing to Blender, I was writing plugins and the community was so helpful and then I decided to make my own Ray Tracer and the open source Ray Tracer community like got together and I was like a 16 year old kid and they were like, yeah, yeah, this is how you do things. And I was like, okay, well, you know, everything I do is going to be open source. I never thought about it, yeah, and then I got into a startup and everybody was like no, no, you shouldn't open source things. It's bad, but I think it really doesn't matter. I think you just do good things and the world will eventually figure out, like, how to reward you.

Speaker 1: 34:18

Look at Meta and everything Meta's open sourced. Meta's actually got an incredible track record of open source projects and they have one of the best businesses in the world.

Speaker 1: 34:34

Doesn't hurt them in the slightest, react or these things. It's good, if anything, it helps them hire ideas from your community. Do we get ideas from our community? Yeah, I don't think we so far no, or not very good ones, and I'm always a as someone who's been building like my customer has always been developers. Right, I've been building him out tooling for years and so, like, developers want to give you solutions all the time. So you go and have a meet with them. You're like what do you need? And they try to describe the solution. So I'm very, very well trained to like kind of flip them back and be like no, but let's go over like your workflow and like where you're spending time and I would like to identify problems. Um, if people can give me a thing they're spending and wasting a lot of time and energy on every week, that's much more valuable to me than if they give me, like, an idea. I think ideas are cheap compared to real world data on the problems that people are spending time on.

Speaker 2: 35:18

So where do you get the data from if you're open source? This is also something that's like been a particular conundrum to us. We have an open source project. I don't want to put analytics in it because I kind of like, yeah, it feels like an overreach.

Speaker 1: 35:27

We have the exact same problem. We think, especially given what our framework is, which is right now it's just code that runs in your application it feels very wrong to put analytics in there. If it was like a self-hosted, Dockerized application and you could opt out, it would feel differently. But as a library that typically is part of a bunch of just running in your environment, it feels wrong. So we've added no analytics. So we have like NPM downloads and that's about it and that's not great and we hate it. So we haven't solved this.

Speaker 1: 35:53

So this new product one of the reasons why we like it is we get to instrument the hell out of it and people can opt out if they're running it open source and if they're using the hosted version. We have like a terms of service privacy policy that they're agreeing to. We're very much looking forward to that. I don't have the answer I would love the open source community to come up with just accept, like even if maybe it's default opted out and you opt in and people just do it more regularly. But we're not there yet, I feel.

Speaker 2: 36:19

Maybe this is an opportunity for a startup build an open source analytics engine for open source libraries. Yeah, like you could just like default import OSS analytics and the entire computer like will have a defaulted like or like opt out, but like people who are in the know will opt in. I think that's a great idea.

Speaker 1: 36:38

I think that I actually think that that's one of the things that people don't build because it won't be a good business but would be very useful. So I hope it does get built. But it might not get built for post. Hog is open source analytics and that's pretty good, but they're business and they're trying to make money, so like they're not really optimizing for that use case, I think we worry too much about what's going to be a good business.

Speaker 2: 36:57

At least you know, I think, when you're raising money you're you're thinking entirely like, is this going to be? There's one idea that I'm coming up with I'm going to be, but like how many companies out there like they were that one thing and they kept being like dropbox? Like they were that one thing. They created a different pricing plan and that that was it, and most other companies.

Speaker 1: 37:17

There are a million things I maybe it's because I I don't know. I'm this way, but I like to simplify things. I like to mold the world in a simple way. I like to think the world is actually simple, um, and so I see companies as they do one thing I think most successful companies do actually one, like Google does indexing and searching the web, and everything else to do is bad. It doesn't matter because they do one thing.

Speaker 1: 37:35

Well, apple does email and great hardware, youtube is pretty good. Yeah, I mean not great, not good businesses, not good businesses, that's for sure. And so Dropbox is another great example. If you follow that logic, really, what you should try to do if you want to build a generational company is find a one thing, and that one thing could be open source. It doesn't really none of it matters If you find a thing that people OpenAI was a nonprofit Like. If you find a thing, you will get pulled by the markets, and so that's like this, the shots on goal thing, right, like we just want to find our one thing no-transcript journey.

Speaker 1: 38:35

So I think that taste is really important in life. Underrated and mid journey is like just a product with taste and that's great. So I like that. Outside of ai, my girlfriend and I are big fans of linear, which, like they take like you were talking about the ticketing system at apple they take a pretty boring problem and they make it just like they make the experience awesome, both design and engineering wise, um, so that's one of my favorite products right now. This is a very nerdy answer, uh, but yeah, I recent ones, old ones, I use vim and I love vim. I've been in love with vim for like 20 years now. So that's, that's a. That's another small one, but I'm trying to think of recent, recent tools did you ever try emacs?

Speaker 2: 39:15

I tried it for like an afternoon.

Speaker 1: 39:16

But I was like no I, why would I use emacs when I love vim and everybody in my vim community is telling me that emacs is evil?

Speaker 2: 39:24

uh, so yeah, is it evil? I had to learn to use vim. Yeah, I have to do this because vim is on servers, right, so it's everywhere. I have to know just enough to be able to move the cursor around and get out of Vim.

Speaker 1: 39:37

When I came to San Francisco after my master's, I was kind of this green, I want to be a software developer and build companies, but I have no idea what I'm doing. And I joined this startup and there was this developer who's a great friend of mine now and he's super strong, and I was coding one day I think I was using, uh, sublime text, and he sat behind me. He was like what is this crap? Uninstall it. I'm gonna set you up with vim and you're only gonna use vim, all right. And he made me promise and I used vim for like two weeks and then he was like you like it? And I was like, yes, I like it. He's like there you go. And he would be this like very opinionated, like this is what you do. In the first two years of my career, which was great, he set me on the right path for a few few different things.

Speaker 2: 40:09

I feel like I'm this person for a lot of people like you're gonna use vim, you're gonna use emacs, we're gonna sit there and like, no, I see you opening sublime, I see you opening vs code.

Speaker 1: 40:19

It's bad, it is not the way it was helpful to have such an opinionated view at the time, because I was just like, okay, I'm gonna learn this thing I feel a little bit bad because I've since, like started using vs code.

Speaker 1: 40:30

I think the last couple of months. This is a fight my co-founder and I have. We've had it for years, since cruz. Uh, he's a vs code user and he keeps telling me that it's great and I should use it. And I'm being stubborn, but every time I try to use it I feel like I'm fighting. I've tried using it for like two or four weeks and I just feel like I'm fighting it the whole time and it does many, many things that I don't need it to do and therefore it feels like overkill, which I always hate. I prefer minimalism.

Speaker 2: 40:51

So actually the thing that got me into it was I was developing an extension for VS Code, and the only way to develop an extension for VS Code really I mean this might be possible in other ways, but was to in VS Code Got it, so I had to use it. What was the extension? It was an early like code generator tool. But like since then, since then, I've been like doing quite a bit of web development and it's it's pretty good for web development. It's like it is like it's based in typescript, so like you guys are building a typescript app, don't you find it a little bit that's?

Speaker 1: 41:28

so that's the reason why my co-founder says I should Like. Basically, microsoft has been hitting home runs recently. The world has changed since the 2010s and now Microsoft is cool and shipping great open source projects like TypeScript and VS Code that work really well together, and so, like they have this fly, I think because TypeScript is just eating the world. This is what we TypeScript and JavaScript this is what we learned at YC doing custom interviews is just dominating. It's 70% of developer teams. It's 75 in the US and growing, and they're all using VS Code more and more and these things actually work really well together. They're being built by Microsoft and they have this crazy symbiosis and so I think this is all. Yeah, microsoft has got some tailwinds behind the open source products they're building. It's impressive.

Speaker 2: 42:10

I think specifically with typescript, what's really interesting is that the language is like almost built for type imprints, uh, and vs code is like integrated so into the type of press. It feels like there's this uh, I used to be a haskler and there was this tool for haskell called hoogle yeah, and back in 2000 then there wasn't a TypeScript to call that.

Speaker 2: 42:32

It made Haskell the most beautiful language in the world to program in because, anytime you're like I need something that solves this problem, you could describe the problem in type language and in search for the code that did that thing for you, rather than just being like well, what could the name of this problem be? What, like assuming that I knew the name of the function, like, what is that? That's a really cool way to solve problems.

Speaker 1: 43:00

It reminds me a bit of like in physics. Sometimes you reason about the units to find, like, the equations that are ruling the world.

Speaker 2: 43:05

It's exactly that.

Speaker 1: 43:08

That's cool. Yeah, I wrote Elixir back in the day at Pinterest for a couple of years and that was. I miss functional programming a lot. It's just honestly, it's better, but people don't use it, so you have to meet the people.

Speaker 2: 43:17

But TypeScript is functional programming. It's kind of you can mutate things, the data is you can. But that doesn't make it not functional.

Speaker 1: 43:25

It makes a difference in what you can assume. Yeah, types modern typeshift feels very similar to elixir actually, and so in that sense it's kind of very fun to write. I'm enjoying myself these days, so I remember like the early days of programming javascript.

Speaker 2: 43:40

By the early days I mean like 10 years ago, so it's not really the early days, yeah, but back then, without too much frameworks, it was still like create an asynchronous function for literally everything and pass it to everything was closures, yeah, and hold back hell. You know, you'd speak to beginning programmers who were learning in javascript and they were like yes, I love imperative languages, I'm programming in the imperative language of javascript. What is this closure thing? I'm like that's a closure. You're using a closure in your code right now. Like no, no, I'm not. Functional programming is bad.

Speaker 1: 44:15

They have to exactly. I mean, the front-end world has come such a long way. People criticize it, but I think it's just like it's PMF right. It's so used by. Browsers are so important, the runtime is so important and now with React Native, you can target mobile. So TypeScript is not going anywhere. Therefore, it has to evolve, and it evolves really fast. Where it's come from in 10 years, like jQuery and the callback hell to what it is today with TypeScript and all of the new paradigms, is just really impressive.

Speaker 2: 44:47

So are you guys primarily a front-end framework, or does your framework work on the back-end too, or is it both? It's both, so this is a good question.

Speaker 1: 44:59

Fundamentally the AI lifecycle. Remember, you're collecting production data and then there's some data management and that's not language-specific, it's infrastructure. There's a language-to-interface, but it's infrastructure. And then there's labeling, which is probably a UI, so that's a product, so it's infrastructure. And then there's labeling, which is probably a UI, so that's a product. So it's actually not language specific, it's a UI. And then you go back to like fine tuning and evaluation, so all of these things actually are language agnostic.

Speaker 1: 45:20

And then the framework in which you're writing your application is TypeScript. So we think of TypeScript more as a wedge, which is where we're meeting, where you're writing your application, and over time we're hoping that it's kind of going to encroach on the other domain. So right now, data processing is heavily Python. Their system, like Spark and all of the data warehousing software, is kind of like a very mature Python ecosystem. But we hear TypeScript developers complaining about this.

Speaker 1: 45:45

Like the teams we're talking to are like, yeah, we're doing all the application in TypeScript. We kind of wish we could have like some kind of processing. We want to massage the data before we send it to the evaluation pipeline. Right now we're dumping it in a table somewhere and then extracting it in Python. We wish we didn't have to do that. So over time I'm hoping that data processing kind of also becomes a TypeScript thing, but this is not. This is, we see, typescript more of the wedge of meeting people where their applications are. The rest of the infrastructure is going to be infrastructure and products and it's pretty language agnostic, to be honest.

Speaker 2: 46:13

What do you wish? More people knew.

Speaker 1: 46:16

I wish that what more people knew is that working in non-deterministic systems is just fundamentally, it's much more different than traditional software than it seems. So in traditional software you have a bug, you flip a bit, then it's correct. You can write a test, it's always correct. And so all of the tools and practices and frameworks are built for. Like the software that we've been building for the past 20 years is built for that like imperative kind of style, like the world is true or false. Ai is non-deterministic and that's just really weird. So troubleshooting is hard, making guarantees is hard. You need to think in this probabilistic way, which is like if you've done math, stats feels very different than the rest of math. It's kind of the same thing Like AI feels very different than the rest of software engineering, probably because it's based on stats, and so you need your tools and your frameworks and your state of mind to be different. You need to reason in terms of probabilities, which people don't do. You see this time and again with the fact that they build a demo. They get excited and they pitch it and they haven't thought about the downsides. Let me actually I have a catchy way of saying this Research AI is excitement-driven or capabilities-driven. Oh, what can the AI do? It's so exciting. Look at this demo.

Speaker 1: 47:22

Production AI is all about it's fear-driven. It's all about minimizing what it shouldn't do, and that's what people don't know and underestimate. People say production AI is different. Everybody says that, but I don't think everybody has actually felt that pain. If you do self-driving, you've really felt that pain, because it's a safety-critical environment and so the tooling just needs to be different. The practices need to be different. Software engineers that are building AI features need to think differently, and so, as we're trying to convince teams to adopt AxeFlow, what I wish they knew that was your question, right what I wish they knew is that they need their tooling to look and feel different, and that's why our tooling sometimes looks and feels a little different. We're trying to help you reason in this like data, probabilistic way, and so that's I think it's going to happen. It's just going to take time. All software engineers are becoming like AI engineers in a way, because they're being asked to build features and they're going to have to adapt and they will, but I wish it was happening faster.

Speaker 2: 48:15

Well, thank you very much for coming on, Nick. This was really great. Thank you very much.