Speaker 1: 0:16

Hello and welcome to another episode of the Accelerometer. I'm Dr Matthew Merman, ceo and founder of Anarchy. Today we have Daniel Lenton, ceo and founder of Unify. Unify is a company that's working to simplify the fragmented AI stack. I'd love to hear a little bit more about what Unify is working on.

Speaker 2: 0:36

Yeah, awesome. Well, first of all, really great to be here, very excited to chat with you today. High-level Unify is, as you say, looking to Unify and simplify the fragmented AI stack. One of the main things we've released is this framework called IV. Iv is an open source machine learning framework which effectively makes it possible to write code, ones that supports all frameworks, but also makes it possible to convert code between the frameworks. You can take TensorFlow code and convert it to PyTorch, you can take Jack's code and convert it to TensorFlow and whatever, and that operates at the level of individual functions but also layers of a network, full libraries, networks and so on, so you can take the whole thing and convert it between frameworks. And that's very good when you're doing R&D and trying to navigate these fragmented tools.

Speaker 2: 1:20

Unify more broadly, aside from just the framework IV, we're now focusing very much on our deployment engine where, effectively, you can upload your model and we can then get the deployment performance much faster. And I mean I'll try not to monologue too long, otherwise I'll talk forever. But the high level here is that the fragmentation in the field goes beyond just the frameworks. There is fragmentation in the orchestration tooling, which is kind of Kubernetes and it is serverless GPUs and all this kind of stuff. It's then also the compression techniques available, the compiler infrastructure and the hardware, and for us to really get deployment pipelines running as fast and cheaply as they possibly can, you need to take a very holistic approach and understand all those levels of the stack and how they interact.

Speaker 2: 2:08

And this is what our engine is able to do by taking the very, very best of the open source tooling, the proprietary tooling, and synthesizing these things together to get recipes that are very fast and better than the sum of the parts effectively so high level. That's what we're doing Unify. What gave you the idea to work on this? So I was in the Dyson Robotics Lab working on 3D vision robotics. We had a lot of people in the lab, the Dyson Robotics Lab, that's what it was called.

Speaker 1: 2:33

Was it like Dyson vacuum robotics Exactly?

Speaker 2: 2:35

So it was a lab that they basically are sponsoring and so on, so doing robotics and vision research under their umbrella. We had a lot of people in the lab who were doing internships at various places. We had people go to OpenAI and DeepMind and Meta and so on. Everybody would then come back using totally different frameworks. I personally interned at Amazon working on their drone program.

Speaker 2: 2:58

Quick chronology of machine learning frameworks. Like in 2015, all of us were using CAFE, the C++ library, 2016,. It became much more popular to use TensorFlow, this kind of Python and API, but it still had this static graph. I will get too technical. Then PyTorch came out in 2017, something that everybody's familiar with. I was using MXNet at Amazon.

Speaker 2: 3:18

A lot of people moved over to Jax when they intended DeepMind, so there was this big fragmentation problem in the framework space throughout 2015 to 2020. This made it very hard to collaborate on projects together when everyone was using different frameworks. I found this very annoying, wanted to do something about it, and what I thought would be a few weeks project to start with, kind of just building some simple tooling to enable people to convert between these frameworks A few weeks became a few months, a few months became a few years and here I am still deep in the engineering rabbit hole. So it was basically birthed out of a frustration in the PhD lab I was working in and the inability to collaborate seamlessly.

Speaker 2: 3:57

The breadth of unification we're striving to solve has gone beyond the frameworks and is now, as I said, all at levels of the stack. But it came from a personal need during my research. So that's kind of where it started. It wasn't a company, it was just a personal project I made to alleviate a personal problem and actually the VCs started to reach out. It was cold inbound from VCs who saw we had some open source traction and we're interested if we wanted to make a company, and and lo and behold, I did and that's kind of where things have gone. That, that's more or less how things started.

Speaker 1: 4:27

So when you first released the project, how did you initially get users?

Speaker 2: 4:31

Initially, we just did a Twitter release in. This isn't a.

Speaker 1: 4:34

Twitter release.

Speaker 2: 4:35

Yeah, I don't know if that's a phrase, but we did a Twitter release, a Twitter launch, a Twitter launch, exactly. So it was in like February 2021 initially and I just put it out there on Twitter. I mean, I did ask a few friends if they could like it or repost it and so on, but not many kind of maybe 10 friends. I asked if they could amplify it and that did pretty well. We got like 250 likes or something and got like I don't know 14,000 views or something on the tweet. So so, considering my Twitter profile was non-existent, I was quite pleasantly surprised with the attention it got. We got like 200 stars and we got people starting to use it and download it and play with it. We're not kind of crazy. It was kind of in the dozens at the beginning.

Speaker 2: 5:13

I think one of the things that's really enabled us to grow the users was actually. So we've done this big hiring push. We reached out to every computer science department we could find pretty much in the world, got our internships advertised at these computer science departments, put ourselves on a lot of job ads and on a lot of job boards, and my thesis is that exceptional engineering talent is universal. It's all over the world. You don't need to have gone to Harvard or MIT or Stanford to be a top computer science well, top engineer, and that's something that we've really found during hiring.

Speaker 2: 5:50

So we wanted to just put the word out everywhere do totally merit-based hiring, trust our own instincts, pretty much ignore the CV, and that's why we wanted to cast a wide net. We've since had more than 50,000 applications to our roles, so a ton of people have applied and that also helped spread awareness. To be honest, that has also been helped along that line and a lot of people that have applied, a lot of people that have contributed to the project, have then gone on to continue to use it, tell their friends about it, and that's really enabled this word of mouth spreading that we're seeing at the moment. So I think, yeah, a few things, but certainly just getting the word out there and the hiring volume has certainly helped.

Speaker 1: 6:27

I've found the same thing with our project. We decided to go open source. We got contributors from all over the world. In fact, contributors have previously contributed to Ivy.

Speaker 2: 6:35

I noticed, actually, I've seen a few, yeah, yeah.

Speaker 1: 6:39

And they all talk super highly of Ivy and had a great experience there and when they come into our project they also know a lot more in that bring a lot of really good ideas. Yeah, hiring all over the place, it's been really effective.

Speaker 2: 6:54

I think people miss out on just how powerful that is. I mean, I know there's a big push now to come back into the office and everything, and there are downsides. I mean, you do need to think quite carefully about how you keep people feeling engaged and how you keep people motivated when they're working from what might be their bedroom or their living room and so on, and over time, it does require effort to keep a fully remote workforce engaged and so on. But the number of exceptional engineers we've brought from Vietnam, from Egypt, from I've just got all over the world, from it is just like and there's not that many top, top engineers. And I know that having spoken, having worked in top research labs, having worked at Amazon and so on. And you just miss out on such a global talent pool if you limit yourself to one city or one location, and I therefore think that you need to really think hard about whether you want to close the door there to the global talent pool.

Speaker 1: 7:47

So when you started Ivy, was your intention to be an open source project, or did you? Weren't even thinking about being a company, or are you at that point?

Speaker 2: 7:55

Yeah, no, not at all. We weren't thinking about being a company. I say we, it was just me as well. So just me with my In your PhD, with your cool project, exactly, yeah. So there was no intention at all, I just wanted to build something useful.

Speaker 2: 8:07

But then the motivation to do the company I suppose came from a motivation to let the project reach its full heights. Like I mean, obviously I never, I never really wanted to be a founder. I honestly didn't know what YC was until about three weeks before applying and I didn't know what the VC acronym even meant. I didn't know what venture capital really was.

Speaker 2: 8:26

So, totally, kind of often in my own world with the research and what I really cared about, and then the motivation to make it into a company was well, fully open source projects can never quite, or often can't, reach the full potential of how big the problem they can really solve is, and things that can fund themselves go on to solve much bigger problems. So I was always very problem focused and wanted to actually solve this fragmentation that I needed somehow to monetize that. Otherwise it's just going to remain. I'd say just, I don't want to be. There's a lot of amazing open source projects. But the motivation to turn it into a company was very much about the motivation to solve the problem in the biggest way, and it's been, yeah, a lot of fun. A lot of fun so far, for sure.

Speaker 1: 9:07

Did you ever have any doubts about whether to open?

Speaker 2: 9:10

source it. No, there was no doubt about open sourcing it and in fact it was an open source from day one. So I think with the project, with this particular project, it wouldn't work unless it was open source because, unusually, just the scope of engineering that's required is massive. Like, most projects are nowhere near as modular as Ivy, we literally have thousands of totally independent functions, each of which can be done independently, and I mean we have 1500 contributors and that's not like a gimmick that they have literally each of them contributed meaningful functions to our front ends and our back ends.

Speaker 2: 9:47

I don't want to get too technical, but just to give a scope of why that is. It's because what we do with Ivy is we recreate and mimic the functional API of every single machine learning framework out there. So we totally have to recreate that behavior and we do that building out of Ivy functions, which themselves then need to have a back end for every framework. So Ivymatmul needs a TensorFlow implementation, needs a PyTorch implementation, needs a Jax implementation. These can be done totally independently and therefore this modularity is just insane and we have 1000 plus independent tasks on this to-do list that anyone can pick up. The power of open source has really what's enabled Ivy to take flight, and it wouldn't be possible otherwise. So yeah, no doubts about that.

Speaker 1: 10:32

How has the transition been from academia to your work now?

Speaker 2: 10:37

Personally they're very different and both have their pros and cons. So when you're doing your PhD, you have a lot of time to think deeply about problems, kind of uninterrupted, and you can go away for four days and no one knows where you are and you're just there deeply thinking, reading research papers and kind of really wrestling with the cutting edge of the field, which I really like. The thing is with doing a PhD in academia in general it's a much more solitary thing and you don't have the same camaraderie, or at least in the lab I was in it was a little bit more individualistic. So I really liked that, but I think it was always. There's always a risk that it's a bit academic and you're not actually seeing it deployed in the real world and solving real problems.

Speaker 2: 11:19

I personally much prefer now doing the company A because I really like working with people. I talk to the whole team every day. I'm trying to project, manage what everyone's doing and, I suppose, trying to make sure everyone's happy and satisfied and enjoying their work and everything, and just the need to talk to people so regularly really, really. I like that a lot and also having a tangible impact in the real world is obviously very important and that really keeps me going and drives me. So personally, I prefer the founder journey, but both of them have uncertainty. When you're submitting a paper and you don't know if it's going to get into Europe or not, or you don't know if it's going to get in, like there's kind of this big gap between success and nothing. And the same is true in startups. Like a lot of startups fail, you've got to wrestle and deal with that risk and be comfortable with that risk and what you're doing is risky. But I think that's the case in both, in both worlds, to be honest.

Speaker 1: 12:14

So yeah, so how much project management do you find yourself actually doing?

Speaker 2: 12:19

Yeah, quite a lot.

Speaker 2: 12:20

I mean, we have a team of 30 engineers right now, so that requires quite a lot of project management.

Speaker 2: 12:25

Making sure there's enough paralyzation so everyone has something to do that's not trading on each of the toes Definitely requires quite a lot of thought. I think one of the things I've really learned over the last 18 months is to make sure you delegate effectively as well. So I think there was a point in time where I was trying to be in the loop on too many decisions and was kind of like having my someone that's trying to spend too many plates. That was a little bit of a problem. And getting the very, very best people and elevating them into decision making positions quickly so that then they can take full ownership and you can entrust that on them, is very important, I think. So there's certainly a lot of project management, but there's also just being good at bringing on the right people and knowing when is the right time to empower them with a lot of responsibility, because you can't be a crazy puppet master for too long, otherwise it gets too much.

Speaker 1: 13:16

So actually one of the things for me and my startup journey is I'm trying to figure out this open source. Right now, we have a lot of contributors like a lot of contributors coming in and more people who want to contribute than we even have tasks or people to review. How do you manage a thousand contributors?

Speaker 2: 13:34

I guess the first thing is to come back to the fact that I think our project is just unusually modular. So I would like to take credit for us being incredibly insightful and thoughtful about how we manage this. If you have a project where you have thousands of independent tasks that can be done, it's very easy, first of all, to get them making the PRs. Now, in terms of getting them reviewed again, we basically ramped up hiring relatively quickly. So we started off with four interns and actually those four interns a couple of them are now senior roles with us but we started out with this general intern hiring process where then the very best people that join as interns go on and get promoted and become full-time and then leaders and so on. So basically every single intern that joined and let's say at some point in time we have 20 interns. There was never 20 interns, but we have like 10 full-time staff, 10 interns or something like this.

Speaker 2: 14:24

Everybody is responsible for reviewing PRs on a random basis. It's a relatively simple DevOps task to randomly assign one of these 20 people to one of these PRs, and it basically scaled up like this. So at the beginning, I mean, we now have something like 35 new PRs coming in every day. So that requires 35 engineers to review a PR every day. So that's a lot of that's a big volume obviously.

Speaker 2: 14:49

But it wasn't like that overnight, like when the project started and when we first got our funding and we first started hiring, we had four or five PRs a day and I was doing them all and that was fine. It took me a few hours and then we hired the first people and then they started. I up-skilled them, they then learned to do it and it just went like that basically. And then obviously now we have 30 engineers all responsible for doing at least one PR a day. But again, I feel as though there's a lot less ingenuity on my side and it's much more just. The project is like so beautifully modular that we can just do this basically.

Speaker 1: 15:22

That's really cool. So what's been the hardest part of being an open source company so far?

Speaker 2: 15:29

I think the hardest part of being an open source company. There's lots of things that are obviously good, which is just the amount of awareness and passion that the community brings, and the eyeballs and everything, and that's all great. I guess every open source company has the conflict which their open source community doesn't necessarily directly overlap with the enterprises that they need to sell to, and this is something that we're seeing to some extent and trying to, and you end up needing to try to balance the two things. I mean I even think amazing open source companies like Hugging Face the community doesn't always necessarily directly lead to the sales correction that they get, and you do still need to kind of make a bit of top-down sales and so on. So I think, just trying to balance the two.

Speaker 2: 16:07

We want to keep growing the community, we want to keep getting, we want to have the community remain loving the project, loving the product, using it, getting a lot of benefit from it, even while knowing that that won't generate necessarily all of the revenue.

Speaker 2: 16:21

It does help with sales leads. A lot of the sales that we have then done have come from inbound, coming from the community, but still it doesn't directly correlate with revenue, obviously, and I think what you get is a slightly longer term. Talking purely economically, I personally just want to build something that people find useful, to be honest. But in terms of how these things fit into the revenue growth eventually, I do think that if you build something that the open source community loves, one day these people are decision makers and companies and if they love what you've built and at some point then there's an enterprise version and they can be the biggest advocates in the company to use it, it does pay off eventually. But I think it's juggling the two, like we want to make sure we listen to, love and build all the best stuff for our open source community, but we also need to become revenue obsessed as well, and it's a bit of a mind splitting your mind a bit when you're trying to do both of those things sometimes.

Speaker 1: 17:14

Have you ever considered starting it as a non-for-profit foundation?

Speaker 2: 17:19

I haven't. It's an interesting idea and maybe I will Not especially I mean to us this is maybe exposing my naivety a bit here but structurally, what would that fundamentally do? It would kind of make it detached from the enterprise itself. It would become this thing that is, yeah, I mean sorry, maybe.

Speaker 1: 17:43

Just what would that actually mean like in practice when, when things so I there are a few like I think pipe torches structure this way and Apache. There's like an Apache organization and then a separate company which was like spun out from the Apache foundation and like, since I'm not sure if it's Apache, so maybe don't quote me on that. Yeah, like it's not, like it's recorded. Yeah, I'm sure, whatever it can go on the internet, that I'm wrong about something. I don't mind being wrong about a thing, everyone's wrong about things every now and then. But yeah, I think what you, a lot of open source companies, do because they want like some open source stewardship and like fundamentally, open source is not there to make money is they spin out a foundation and the company, like it, provides resources to the foundation, but then because it's like a separate foundation, you don't have to worry about like conflicting interests. Yeah, yeah, yeah.

Speaker 2: 18:37

I think I think actually that's very interesting point and structurally that it kind of already is how we behave. I mean, to some extent I guess that the people at the top of the foundation are then typically one step removed from the directors of the company or something. So I guess that maybe is where the structural difference comes from. I know that, for example, mlir is this open foundation which is not the same as modular or mojo, for example, and same with LLVM, but of course they have very strong interrelations. Same with open XLA now, I think, is a foundation and is not directly a product of Google. Yeah, I think that's. I can imagine that be where we go at some point when we have the time and headspace to kind of do the restructuring or something. But I would like IV to be in this kind of vein, but let's see how it all unfolds. But yeah, very definitely an interesting proposal.

Speaker 1: 19:29

So you just mentioned, like mojo and XLA, what are some projects that you're particularly excited about at the moment, other than your own? I'm sure you're very excited by.

Speaker 2: 19:38

IV.

Speaker 1: 19:38

I love to talk about my own all the time.

Speaker 2: 19:40

No, there's lots. I mean, vllm is amazing to do inference on LLMs and they're doing a very good job of keeping up to date with all of the latest state of the art attention layers and page attention and all this kind of stuff. So I think that's great. I was excited with TensorFlow RT LLM getting open sourced. I think there's a lot of really interesting infrastructure in this space which is great to see, and I a big open source advocate. So I think what mojo doing is also amazing and I do think that bringing compiler level representations and structs and being able to do kind of you know optimizations around tiling and things like this, bringing that to a Python like language, is a big improvement, because right now what you have are these machine learning frameworks which are a Python thinly wrapped API around heavy C++ and CUDA which you kind of need to rewrite and recompile the entire thing to benefit from, whereas with something like mojo, you could just step inside the function and literally just change the you know in the same language and modify it and so on. So yeah, I'm very excited about mojo, although I would like obviously, as much of that to be open source as possible.

Speaker 2: 20:44

Modular's engine won't be open source. But again, the more transparent this technology is, the better. That's why I'm loving initiatives like flash attention that you know ask the question whether we are going to have the cutting edge stuff open source and not just behind kind of CUDA and CUDA and things like this Long answer, I'm just listing loads of things. But open AI Triton is also very exciting, but it has a very similar flavor to mojo in that it's kind of bringing like GPU programming to a high level. Representation that is, and just how performant it is is incredibly impressive. Torch inductor now uses Dynamo by default on sorry, sorry. Torch inductor now uses Triton by default and the performance is amazing. So I think if we can unlock you know, I mean, I love Nvidia as a company, it's doing amazing things but if we can unlock the CUDA mode a little bit, that would be a good thing for the world as well. So yeah, these are some of the things that are exciting me.

Speaker 1: 21:38

So I think mojo is an interesting example because of that like partially not open source aspect of it. Do you think it's really possible to build something that's going to be an integral part of the infrastructure of, you know, the rest of the developer community? That's not open source?

Speaker 2: 21:56

I think not. I mean, it's hard to say. Cuda has been a fundamental part of every machine learning stack for a long time and that is proprietary, obviously. But whether that still is possible in light of things like open AI, triton and so on is yet to be seen. Mojo intends to open source. That that's what they've said. So I think they won't. I mean, hopefully they won't have that problem and it will be open source. That'll be a very good thing, but no, I think it is very difficult and I think I mean it's all well and good saying that. Obviously I don't know.

Speaker 2: 22:25

And you could say the same about models, and I would have thought the same about models pre-chatagibatini and GPT4 that, like open source, will always have the cutting edge stuff. That's not the case in terms of large language models. Let's see if, what time, you know how that will be over time. But compiler infrastructure is a bit different, because you need massive data, massive compute. You need tens of millions of dollars or, you know maybe now hundreds of millions of dollars to really get the best performance. I think to get very good compiler infrastructure and tooling you need deep technical knowledge of how these things work, and these come from top PhD labs and so on. So the gate is is a different gate and I think when it just becomes kind of intuition of you know top compiler engineers, then they'll always be very strong open source variants and and I hope they they win in the end.

Speaker 1: 23:12

Yeah so let's maybe switch topics a little bit and talk about just AI in general, sure? Sounds good is there anything that worries you about the state of AI right now?

Speaker 2: 23:25

good question, maybe not the state of AI. I mean, to be honest, I've watched enough videos of people talking about AI to feel as though it's hard to say anything that doesn't feel a little bit cliched. But my, I think my immediate concern is a bit less about the AGI concerns and it's just a bit more about whether we have the economic setup to deal with the job displacement and things like this. Broadly speaking, the world has seen increasing inequality. In many countries there's obviously increasing nationalism and everything. And like AI, will you know, regardless of AGI, in the trajectory that goes on, ai will replace drivers, it will replace people doing data entry, and that's a lot of people. There's a lot of kind of admin, assistant style roles that could become, could disappear.

Speaker 2: 24:11

So I'm concerned about the general social unrest that could come from poorly managed policies around job displacement. That that's my probably top concern. But on the more existential side, I mean, yeah, I do believe I guess I don't quite maybe somewhere between Hinton and LeCun where it's like I do think that it's potentially the beginnings of kind of AGI and and it wouldn't surprise me if in five years I mean, I'm not saying that's what it is, but it wouldn't surprise me if the trajectory wrong does mean in five years there are, like AGI, that genuinely can engage with the internet in a way that's as sophisticated as a person could, if not more, and could then all the threats that come along without, like you know, releasing a super drug or super bug or super drug would be pretty nice. Super drug, we nice, yeah, super drugs. To combat the super bug we'll have to have like a good rogue AI and a bad rogue AI battling out I mean, I think this is also like.

Speaker 1: 25:05

This is one of the current theories. You know, people want to accelerate faster because we're going to end up with good EGI, which will combat the bad issue.

Speaker 2: 25:13

Yeah, I also do bind to that camper bit, to be honest, like I feel as though you can't close, you can't stop the flood, like there are, you know, autocracies in the world that are not going to stop.

Speaker 2: 25:27

I mean, you know China are doing amazing research, russia will continue to research, and I think you, I think just putting a blanket stop on research is not the right answer, but I do think I think putting a lot of effort into AI safety research is important. I do think we need to keep going, because we need to understand this and know what it can and can't do, and and I just I do genuinely believe that the more open the better, though I know that then, if it's open source, anybody can then do fake news, but if it's so widespread, like people I mean I don't know you can do arguments in both sides and and I don't have maybe enough kind of justifications here but my sense is that open source is is just better, because then at least the people can see when it's BS and when it's not, and and can kind of moderate it to some extent, whereas if it's closed behind a small number of companies, that to me doesn't feel, doesn't feel great. Yeah, so I'm slightly meandering answer.

Speaker 1: 26:22

I feel like with IV, you have a pretty unique opportunity. Like I can imagine, it's something that's used to buy a lot of very disparate research organizations, and you probably do get a lot of feedback about what people are actually doing behind the scenes that not many other people have at the moment.

Speaker 2: 26:40

Yeah to some extent, although what I would say is that, while the whole hype is LLMs and that's all you hear about, we're working with a lot of organizations that are, you know, doing computer vision, are doing, you know, text-to-speech, are doing video generation and all kinds of things, and I think one of the main particularly for Ivy. So so yeah, again, that the two things that Ivy enables you to do is to write code once that supports all frameworks, which is TensorFlow, jax, pytorch, etc. And also to convert code between the frameworks. One of the big ways in which Ivy is currently being used is to convert code, especially between PyTorch and TensorFlow. Pytorch is getting better at deployment and there's Torch 2. Obviously came out end of 2022. There's now execute Torch, which is very good for getting this running on, you know, on edge devices and things. Bit of a success of Torch Mobile. Sorry, again, a bit of a meandering answer. Basically, one of the one of the big use cases is taking the cutting edge models implemented in PyTorch and deploying those in TensorFlow, because TensorFlow still has superior deployment features and a lot of very large enterprises you know, billion dollar companies are using TensorFlow to deploy on a large scale and that's where we've got a lot of usage with Ivy.

Speaker 2: 27:49

What I would say is Our broader engine, where we're not only unifying the frameworks but unifying the orchestration layers, the compression, the compilers and the hardware. We have a lot of companies that are Using LLMs there and and actually they don't know how to start. They, they can spin something upon, replicate, they can get something going quickly. They, before they know, at spending 20 or 30k a month on compute costs. We can bring that down to five or 10k because we take this holistic approach. Looking at the orchestration I'm just keep listing them over again but looking at all levels of the stack and getting this, you know, holistic approach that considers all of them and and really gets things optimized for their needs.

Speaker 2: 28:25

Because People's needs depend, you know. You might care more about latency, you might care more about cost or accuracy or cold start time All of these things. It depends on the application. What's your actual API request rate and how much do you need it to be very quick and so on. And given the number of tools, the number of different use cases, it's a Complex mess to deal with all of that and this is where we shine and and yes, so we're learning a lot about those companies doing LLMs as well, and that's amazing. I mean it's a privilege to kind of be in there, you know, deep in the weeds, and actually talking to these companies doing really exciting stuff. Yeah.

Speaker 1: 29:01

So yeah, given that you've spoken and you've worked with so many companies, I guess what's the most interesting thing that companies have been using IV for?

Speaker 2: 29:10

Most interesting thing they've using IV for. So I'll try avoid I won't do any name-dropping but there's this companies, this company's in the semiconductor industry like doing testing and you know things like this. There's companies doing texture speech, companies doing dubbing, real-time dubbing, companies doing image generation with stable diffusion, and things like this companies doing Robotics, like actually in a robotic factory, like picking things up in object section. So I mean, in the companies we've worked with, yeah, it really varies. There's some doing like robotic farming and like automated, you know, picking up or planting seeds and things. There's no robotics in factory there's. There's all these kind of things that. There's things that are very B2C, customer-focused and they're processing 5 million images a day. It really varies.

Speaker 2: 29:55

But but the kind of models we're seeing we've done things with whisper, we've done things with stable diffusion, obviously llama and but also just resnets and units and like there's a lot of. There's a big world of computer vision that maybe the newspapers are forgotten about, but you know, cnns are deployed on a massive scale and when we're making deployment cheaper, we kind of see. I think that's also a good thing because, to be honest, I I do think that it's good to be reminded that the world of AI is. It's huge, it's bigger than just just LLMs. I mean, just just to round that off, we're also working with companies using extra boost and scikit-learn, which we can accelerate, which is obviously, you know, dinosaur age ML. As far as you know, you might think, but I mean not really like seven or eight years ago this is being used on a large scale and it still is on a massive scale. So I guess the short answer is a lot.

Speaker 1: 30:42

Given that you're open source, you don't really have very much control over who's using IV and using it for what. And this is something that like really shocked me about Open source is the number of other founders that, and not just founders. But when I first open source, that people who I was working with my Employees were just like but we should be, we should need to control who's using it. We don't want them to use this for the various purposes. How do you deal with that?

Speaker 2: 31:07

I mean. So I think, first of all, I guess you just need to know what you're charging for and what you're not, and who you can realistically so. So with all open source companies, you end up having a small number of enterprises make up the vast majority of revenue in most cases and, I think, acknowledging the benefit of wide usage and interest and what that means in terms of, like, natural word of mouth marketing, what that means in terms of the long-term value you get when they eventually, you know, I mean like students using that, using your tool, that eventually does pay off, because if they tool is so good and they become hooked to it at some point, if they're in an enterprise, there are going to be enterprise features that can be gated and can be, you know, withheld, so that then you need to have, you know, the security levels improved, you need to be able to have like big number of teams or sharing and collaborating on it and so on. So I've never been concerned with Mass usage a because I really just want to solve people's problems is like where it all started. But be because, anyway, this isn't lost revenue, like in many cases at least, like I mean people doing their PhDs they're not going to become customers, so you might as well let them just. You might as well create that value, because you were never going to capture it anyway, so you might as well create it for them, where you can capture the value, which is where companies actually have the money to spend.

Speaker 2: 32:23

I guess you just need to, like you know, draw the boundaries sensibly around the kinds of features that actually make sense, and and and, and I think that's what we try to do. So you know, when you want to deploy on a large scale and save a lot of money, that's when it costs. But if you're just trying to convert between frameworks and so on, then we have a, you know, for all students and hobbyists it's totally free. There's actually some gating, so not everything is open source.

Speaker 2: 32:48

You, you create an account with us and then you can do unlimited conversions, unlimited usage of the compiler, provided it's not for commercial reasons, and if it is, then you need a, you know, modest license with us to do it. Basically, and yeah, and I think I've never been concerned about hobbyists and students using something great for free, because you know more than more than Well, two things more than pace for itself, long-term and be. You got to want to create value anyway, and if you know if they were never going to be able to afford it, you might as well let them have it, because then you just make the world a better place. You make people solve problems quicker, and that's all good, obviously.

Speaker 1: 33:22

Have you been keeping up with you the executive order and the European regulations on AI?

Speaker 2: 33:28

I haven't that much. I do know on a high level but unfortunately I won't be the best sparring partner to discuss. But happy to Be to kind of share high levels if you distill it down, but not too much, unfortunately.

Speaker 1: 33:39

I mean, I've heard a lot of like concern that Open source AI is gonna be regulated. Is that something that you're worried about at all?

Speaker 2: 33:47

I think that would be a concern. It's very difficult with these things like. It's also very hard to comment on some of these things without sounding political or something as well, to be honest, but I'm genuinely generally an advocate for lack of regulation, and that even extends into things like Twitter. Now, there's a lot of downsides to having you know hate speech and everything that is. That is not a good thing. But also Trying to become the arbiter of what is and isn't acceptable, both in terms of speech but also in terms of AI, is an impossible task. Like who's gonna take that responsibility on and I Don't know.

Speaker 2: 34:26

Regulating open source seems the wrong approach. Again, I unfortunately I don't know the details in the context too much, so I'm kind of just taking what you've said basically. To me it seems like more Maybe regulating the usage of it and the product of it, but regulating the neural network architecture and the weights or something Feels wrong. It still feels like it's science and science. It's also different to like building a nuclear bomb on nuclear reactor, where that's like a very physical, tangible thing, like when you're talking about Software and the abstract of like. It just feels like it would start to tread on the toes of research as well, and I think that Some police in the world that will not be regulated and we better be like understanding and learning about this stuff otherwise.

Speaker 1: 35:10

This is my, this is my feeling, basically I think, now that you mentioned, I really want to see regulations that require the usage of dropout. Think everybody should use dropout in their neural networks at some layer. I think so true, probably yeah I have at least at least 30% of the network yeah, I have dropout in it. Yeah, I don't be very good for research that would be very good.

Speaker 2: 35:29

That's true, maybe. That's. Maybe that's what they should come with that. That's the final sentence on that. Their policy, maybe.

Speaker 1: 35:35

So, given given that you're so in the loop right now, where do you think research is headed on a I?

Speaker 2: 35:41

I mean I'm glad that I appear in the loop. I'm not as in the loop as I was doing my PhD. To be honest, I'm in the loop of the Industry side of it way more obviously than the research side. But in terms of where I was heading, I do think that one of the main stories of the last few years has been that with enough data compute you can do anything. So I think the, the architecture seems to not matter that much. I mean, transformers are very good, but mlp mix is a very good those recent paper showing that with enough data and compute, cnn is actually a competitive with visual transformers. So it seems like if we can keep getting more computer, more data, things will keep getting better. And that is Kind of scary because it does suggest what. When is that cut off, like? How close is it? So I think that's where is going. Let's make a load more cheap use and try to generate lots more data. Like the architecture, let's not stress about it too much. I mean, of course Research is well and should continue to do that, because we're not suddenly gonna magically turn up like how does not suddenly gonna become 10 times more abundant and data is not going to do the same either. So you need to get shortcuts to do the best of what you've got now. That's how i's been right, like i's been right, this very simple network is going to be amazing, but we don't have the computer data. So let's do tree boosting and let's do Decision decision trees that work well in the regime run but actually know. It's very simple idea from the 80s prevails and and I feel like In reality, to get a I really, really intelligent, my gut feeling is we don't need that much more. Who knows? But my gut feeling is we might not need that much more algorithmic insight if we had enough data compute I mean evolutionary algorithm of life Generate it, as that's an incredibly simple algorithm running with a lot of data and a lot of time. My gut feeling is that's how very complex emergent intelligence will always arise and it won't come through. It won't come through like too much human heuristics. I mean that's a very broad sweep of where is heading. But I think just to just add one more point, that's a bit more, a bit less big bit more rained in.

Speaker 2: 37:38

I think that compilers, which I'm very interested in, I think they will continue to be driven and I think there's been some really interesting research, such as alpha tensor with the mind, discovering matrix multiplication algorithms that are superior to any human heuristics. And there's also great research was. There was a paper recently by meta we did it in our paper reading group actually that was like it was just called, large language models for compiler optimization and they showed that you can just take a llama to seven be and predict the llvm optimal flags and it can actually improve upon the best heuristics in llvm for compiling just by training one ll, just by training one llm On this very simple task of predicting optimal flags for the compiler. And I think at some point it might be the case where we have a model and we have this mathematical model and then we can have an optimization algorithm. That's a driven that spits out assembly and it's better than anything we can comprehend.

Speaker 2: 38:34

We don't know how it's done this, but it's maybe implicitly done. Some quantization and some compression and some sparsification and some kind of algorithmic wizardry like flash attention just emerges and we don't even understand. Not only do we not understand the math, that the mathematical model, but we don't even understand how this relates to assembly and it's just a hundred times faster. This is kind of where I think things might go in the compiler space, which is scary and exciting. These are two, two thoughts of mine on that. On that question, I mean, you guys are, you guys are kind of writing a pilot is right.

Speaker 1: 39:05

We are kind of the reality is so interpreters maybe. No, we are. No, we absolutely are so so IVs.

Speaker 2: 39:13

We've now called it the tracer. But we've written a tracer and a transpiler and a compiler which means you can take, yeah, pytorch code, you then trace the computation graph, for example, and then you can take that graph and kind of transform it into TensorFlow or so on. But but on the deployment side we're working in the land you were working with XLA and open it right and and so on, and sometimes pure kuda kernels to push the performance of the model Jumping ahead a bit. So so there's effectively two products. One of them is IV, which is this framework which you can use to write new code and then convert code between frameworks, and this kind of has a compiler. It certainly does function tracing to trace a graph which has a compiler flavor.

Speaker 2: 39:59

On the engine side of things, right now we are heavily optimizing specific models, both that are very popular but also that come through from kind of requests, from the sales leads and so on, and then we're building this heavily optimized model hub and the abstractions that enable us to automate this process where you upload any model and then that model is automatically made much faster. That's something that we've pushed back a little bit because what we found is to really really get models going as fast as possible. Right now you do still need to think about them as point solutions if you want a model to be running as fast as possible. It's very hard to get a compiler that can spit out llama CPP or can do kind of what ggml does off the shelf. So we're kind of focusing on point solutions in a model hub with the intention of then Abstracting this logic with compiler logic over time. Again, I'm unpacking quite a lot, maybe on a high level. That makes sense. No, it absolutely makes sense.

Speaker 1: 40:55

What I was originally wondering, though, was, like you know, there has been a lot of research in the last few years on using AI for compiler, and you have this interesting problem. You know where you have currently have, you know, a thousand contributors like trying to solve your problems. Have you ever considered utilizing some of this AI researchers in order to do some of that work for?

Speaker 2: 41:16

Yes, so so that's a great well, not not quite related to AI compilers, but one thing that we have done is built an AI developer which can actually do the work of the AI developers, the work of the contributors, to some extent. So basically, as I say, there's thousands of functions that need to be implemented in the front end, thousands that need to be implemented In the back end. We've taken I think we took code llama, we fine tuned it on our own data, where we've already got thousands that have been implemented is ground truth and then generate these new functions and we get like 55, 60% that are correct on the first pass. So we've kind of taken the like co pilot flavor but then, you know, fine tuned it to very specifically solve the problem we have in the open source space In terms of AI generated compilation. It's something we find really interesting and want to keep exploring.

Speaker 2: 42:10

But I think that is it's something that Focusing too much on that too soon I feel like could be obsessing more about the solution, and we're effectively now I think. I think it's a big research question still to really get an AI driven compiler that can massively outperform the very good heuristics there are Is a huge research, research project and deep mind of publishing papers every year on this topic and they get 3% improvements in XLA. Like we're not there to suddenly have that step change moment, but I think it's coming, that let's see when.

Speaker 1: 42:46

Given that you have a PhD, have you considered, you know, at least allocating some of your communities resources towards working on research?

Speaker 2: 42:56

Yes, we have, and the research we would like to do is this direction actually compiler driven research with spoken with I won't drop names, but spoken with various academics working on really cool stuff. We actually stay very much in the loop with the kind of whole mls community and so on and we've considered dedicating time to do out in our research and dedicating some of our own investment into doing that, and we certainly have the engineers in house that you know more than qualified to do that research. It's, I guess it's a case of. The difficult thing is like I guess right now it's challenging enough just taking the research that's done and kind of finding the way to really integrate that, to get the performance for the end users and really push the boundary. I think at some point that would make sense.

Speaker 2: 43:50

But, truth be told, everyone is inundated with no one has a free moment any day just kind of getting things over the line for customers. I feel like once we're very confident that we have product market fit and once we're confident you know true, true product market fit, once we're confident that we're in a state of like impending profitability, then I think that's when you can go for like bigger moonshot things a little bit. We're trying to be somewhat careful with that, although I have to say it's very tempting because I want to just like, I would love to just like spend lots of time and effort dedicated doing pure research. But we're trying to be somewhat sensible with how we allocate time at the moment.

Speaker 1: 44:29

Yeah, that makes a lot of sense. I we tried allocating a bit of time towards research and it always feels like such a distraction, but it's a lot of fun, yeah, yeah, I mean the good thing is, at the very least.

Speaker 2: 44:40

At the very least, we do stay on the pulse of the ongoing research. We have a reading group every week. Tomorrow I'll be live streaming our next reading group on a few papers from DeepMind on compiler optimization. We do that every week. Everyone in the team joins, we all read the paper and go through it, so we're kind of investing time on staying up to date with it. But it's still, you know, it's still not the same as literally contributing to it, and I think it is a bit of a distraction. Yeah, but I would love, obviously, when, at some point, I would love to be doing that, but I would only do it when the company is, you know, in a state where it makes sense to put that investment, which is a high risk investment, obviously, yeah.

Speaker 1: 45:17

So, yeah, it's really interesting that tomorrow you're live streaming this. I found also in our community that live streaming a lot. It helps yeah, Like it helps getting people involved. What would you suggest for people who want to get involved in the IV project or?

Speaker 2: 45:33

unify. Yeah, so if they want to get involved in the IV project, the first thing to do is to visit unifyai. So visit our website and click on the Discord link right at the top and join our Discord server and you'll see all kinds of things there. I mean, the simplest thing to get involved is to read our Open Tasks guide. So if you go into the docs and I mean if you just do a search, you'll find it just open tasks part of the contributor guide. That gives a series of YouTube videos explaining how you can contribute and create new functions, and then somebody in our community will review that and get you listed as one of our kind of 1300 contributors. And the other thing would be to just, you know, join the reading groups, listen to those.

Speaker 2: 46:11

We have a reading group channel on Discord where we discuss the papers every week and you can ask questions and you can also apply to become a what we call a core contributor. So you actually apply to this. You have to apply, yeah, you have to apply. So we're very selective. Actually, we have about 30 core contributors at the moment who so it is a voluntary, unpaid role.

Speaker 2: 46:31

It's also something that we offer sometimes to people that don't quite make the mark for internships or full-time roles, but you can also apply directly and what you get is direct mentorship. You're assigned somebody on our team to work directly with you and give you lots of hands-on feedback, and you could get access to all of our internal meetings, all of our internal tasks, and the idea is it's mutually beneficial. Like, obviously, you know it's something that people choose to do, and if they don't want to do it, that's fine. But actually we have probably about a 20% success rate of those that apply the GEDICS. We also want to make sure that when we bring people on, you know they are very good engineers and they're prime to learn from the team and so on. So I would also recommend anybody that you know wants to get involved on a deeper level they're very welcome to you know get involved in that sense as well. But yeah, these are just some of the ways you can get involved.

Speaker 1: 47:15

I know a lot of people who'd probably want to get involved. So what's the biggest thing you've learned over the course of building Unify?

Speaker 2: 47:26

I don't know the biggest thing I've learned. I mean there's a few. I've already mentioned, I think, earlier, that you do need to delegate quite quickly and you want to avoid becoming the bottleneck as the team grows, because you need to trust people and empower them and let them make mistakes, let them grow from those mistakes and give them support. So be comfortable letting things go and not being the micromanager that needs, you know, the finger in every play. That's been very important. I think that didn't take too much learning. I think I knew that anyway, but still probably took a bit longer than it could have done. But I think one of them has been it's kind of the YC mantra, to be honest, but it is very true that, at least in my mind as someone that's a bit of a perfectionist, there is a tendency to become obsessed with solutions instead of problems, and I think that's something that I constantly feel the temptation to do and I think like it didn't necessarily come totally naturally to me to constantly talk to their users, constantly ask them what their problems are, think deeply about what the shared problems are between these dozens of people and then work backwards from those problems and you know, I have this tendency say we're building this beautiful, elegant, unifying solution. You know, nothing else is like this. This is the way it needs to be, and you know there's truth to that.

Speaker 2: 48:39

Obviously, you need to have a starting point of where you're going to go, but then sometimes it's really important to just be, to remain anchored, because also, things change, like the field changes so quickly and I think even the very best founders, even when they're years and years in, they stay right down at the low level. They talk to the users regularly. They really know, you know, what their biggest problems are. Because you don't know, because you've been managing a company for, you know, a year and a half or two years or whatever. I've not been doing research or coding on the same deep level, you know, for at least 18 months. Now things change in even that amount of time and unless you stay grounded to the people that are actually using your product, you can easily like get a bit lost.

Speaker 2: 49:15

So it's very basic. It's not that exciting answer, but for me it wasn't totally natural and that's been, you know, incredibly important and I guess sorry just to top that off, also just the really important one as well to say the only metric of success as a company is how much your users love what you're doing. Like nothing else is, yeah, stars and kind of investment and all these things yeah, they're great, but you need to make sure your users love what you're doing and if you're doing that, then you're on a good path. And again I sound like I'm just like like regurgitating YC points, but that's true. They're very good. No, I love that.

Speaker 1: 49:49

Yeah yeah. I personally hate the stars metric. I feel like people put so much attribution onto it and my first couple of projects that I ever released in GitHub just accidentally somebody found them, posted them on Reddit. They got a ton of stars in the first week and then nobody ever used them again. They weren't intended for people to be used, but you know it's sticky the stars don't go away. Yeah, yeah, yeah, yeah. Like it looks like I have like active views projects. Yeah, yeah.

Speaker 2: 50:13

No, exactly, I agree. I mean I probably just got a decent number of stars, but in the end, like, what really matters is how many people are using and how much they love it and how much they keep coming back. And I agree, I think some of these kind of surface level metrics that people obsess about optimizing it's not really what's going to make the difference. Like you need to have you know again. I sound like I'm just regurgitating YC. You know 10 users that love your project and asking you questions constantly is better than a thousand. That kind of used it once or twice and know about it but didn't really care that much about it. Like I think you know internalizing that is very important.

Speaker 1: 50:45

Yeah, I like this idea of like people asking you questions as a measure of yeah, yeah, like people using it, because I've also had this problem in the open source project. Because it's an open source project, I don't want to put analytics that send like who's using it back to us. Yeah, yeah, like somebody might be using it for their nefarious purposes. Yeah, yeah, we want to enable all nefarious actors or projects. But, yeah, I guess like just checking, like are people asking interesting questions?

Speaker 2: 51:12

Yeah, exactly, are people asking questions or people like having problems with it and saying, hey, could you have this feature? Just this kind of thing is a big indicator, much more than any of these other surface level metrics, I think.

Speaker 1: 51:22

So I think we have time for one more question. Yeah, what's some advice that you would have to any aspiring open source founders?

Speaker 2: 51:30

Advice for aspiring open source founders. Good question, I guess. Well, first of all, advice to founders in general is obviously just try to build something that you're passionate about. I think the novelty of being a founder for the sake of being a founder wears off incredibly quickly. I mean, I love being a founder, but I love it because of the problem that we're solving and kind of the vision that that inspires in me every day. So I think it's good if you can work on something that you genuinely are passionate about. I mean, that's important On the open source side.

Speaker 2: 52:03

I guess, to make a bit more specific to the question asked, I would say yeah, like I mean, I guess you need to think quite carefully about where you draw the line between the open source and the enterprise side, and I guess you do need to spin two plates a little bit. This is what I'm finding. You need to keep the open source community engaged because they are, you know what's. They're an incredibly important part of your whole story and what you're building, and they're going to be your biggest backers and you need to, like you know, give them the love and respect that they deserve as the people that are really like your champions, really out there. So you need to make sure you, you know, give them an amazing experience, give them something they can use, that they love using. They don't need to pay for it and everything. But you can't only optimize that. You need to make sure that what you're building isn't just something that's free and that's all it's ever going to be. So you need to make sure there is an enterprise component as well and think very carefully about that. And to be honest, for me I've.

Speaker 2: 53:04

I think you also need to accept that, even for an open source company which has these new distribution channels with potential for kind of bottom up adoption and you should leverage those you need to be comfortable with the fact that.

Speaker 2: 53:15

I mean, in my mind, you need to be comfortable that you should also be thinking about top down sales as well. I think people ask is it top down, is it bottom up? I personally think both is better than one or the other, and we have had leads come in from us. We've also done quite a lot of outreach, trying to get introductions to, you know, investor network and their portfolio companies and so on, and we've had deals close in both regimes and I think you can't shy away from that. You still are trying to be an enterprise and you're trying to close big enterprise contracts and they're not always going to come finding you because of your open source projects. So it's quite hazy advice, but I guess you need to accept that you're spinning two plates and you need to keep spinning them both. In my mind that's how it feels like. Yeah.

Speaker 1: 53:54

Yeah, I love that. Actually, when I finished YC, their comment to me was you spent all of YC philosophizing too much. My heart is broken. And after that I put at the top of my discord, like in my constitution like just do things, yeah, just do more things and don't think about it. And ever since then I'm just like yeah, I don't care whether I'm doing top up or bottom up or pop down or bottom up sales, I'm just going to do that and you need to try as well.

Speaker 2: 54:22

Exactly, I think there's this whole philosophy of just take more shots on goal, launch things quickly, release things, try doing some top down, see if it doesn't work. Try, you know, build the community. And I think you just need to be trying lots of things and getting out there and not like dilly-dallying about decisions and you learn from experience. You then get the signals just from having done it and seeing what happens. So again, another piece of somewhat cliched but very valuable wicy advice, I think.

Speaker 1: 54:45

Yeah, I think we can't. We can't hear it enough times. Yeah, yeah, anyway, this has been really wonderful. Thank you so much for coming. Yeah, likewise had a lot of fun. Thank you very much for having me.