Joel: this thought around web scraping. Every time I like say web scraping to, to most developers, you can kind of see 'em like curl up a little bit like, oh, that, don't say that around me. You know, it's kind of a dark part of the internet, but it serves a big purpose and I think,
you know, kind of bringing that to light a little bit more would do it well.
[00:00:20] Introduction
Andrew: Hello, welcome to Dev Tools fm. This is a podcast about developer tools and the people who make 'em. I'm Andrew, and this is my co-host.
Justin: Hey everyone. Uh, we're really excited to have Joel Griffith on today. So Joel, you are the founder and CEO of Browserless, um, which is pretty cool company. Uh, I've actually used browserless a few times, so it's like fun to have you on, uh, browse Animation kind of sucks and y'all do a good job at it. So, you know, browse for that.
Uh. We're excited to chat more about what you're doing today, but before we dig into that, I would love to just hear a little bit more about you. So would you like to give us an intro?
Joel: Yeah, I'll give you the, the TLDR, hopefully. Thanks for the intro and the kind words. Uh, really happy and excited to be on the podcast. Um, so yeah, Joel, founder and CEO of browsers io. Um, uh, wasn't an engineer when I started. 12 ish, 13 years ago, and then became an engineer, you know, wanting to solve my own problems a little bit with websites and whatnot for bands and stuff I was playing in. And then similar sort of thing happened with Browserless, to be honest. I was working on a few things and I was like, Hey, this really sucks. It could be better. And then kind of had an aha moment when I was building something else at the same time and kind of reran into another web automation problem. I was like, why? Why does this still suck? Um, so yeah, I kind of did a pivot then and, you know, launched browserless and it was a good fit, you know, as an kind of a engineer myself in knowing the language and the people and you know, where they hang out and all those things. Definitely made it a lot easier to kind of like, you know, find first users had an easier time talking to customers.
I think especially in tech, you get used to this like weird metaphor rising of like technical problems such that, you know, layman or laypeople can understand what you're saying. Um, and I don't have to do that, which is great. Uh, it's not my best skill, I try. But, um, anyways, so yeah, that's the quick, very, very quick about me.
Um, and yeah, we could, we could go any direction from there, to be honest. So, yeah.
Andrew: ~~So what. What, ~~so before you were a developer, you were a jazz trumpet player. That's, uh, that's pretty out there. What, like, how did that switch happen? It seems like a seismic shift almost.
Joel: It is. Yeah, there was, so there was two things. One of them is kind of sad and one of 'em is just kind of like. Opportunistic. So, um, at the time as a jazz trumpet player was about to have my first kid. Um, and you know, we go in at 20 weeks for this ultrasound and realize there was some like abnormalities,
you know, and they, they weren't good ones, uh, to be quite honest.
And so, um, we kind of, my wife and I were looking at this long, long list of things that we needed to do, and I was like, I don't know if as a musician I can. We can afford those things that might need to come up. Um, and so it was somewhat out of, you know, external pressures or external things that had happened that kind of like, I don't wanna say forced me into technology, but like strongly encouraged me to get back into technology.
I always loved hacking on the side, you know, even in like high school was writing programming calculator functions for friends and selling those for like. Dollar or two or soda or something. So there's that kind of like trait I had for a long time. And then, you know, this happened with, you know, a kid. Um, and so that really kind of was like, okay, I really should see, you know, what I'm capable of here and, and if I can make it happen.
Um, so that was the big function. And then the second thing was. The store that I was working at Music Store, actually you can go look it up on my LinkedIn profile, you can see I was director of internet services at a music store. Um, they were getting pitched a website and they were getting charged a lot for side website. And I remember looking at this with the owner at the time and I was like, man, if you could give me like a couple days, a couple weeks, I think I could probably build this, you know, if, if I'm allowed to on company hours. So, um, it was kind of those two things kind of fusing together at the same time. The timing was. Fortunate and, um, yeah, so did that, this was 2012, so we didn't have flexbox, we didn't have much no ai. So it was a lot of like, you know, the, the Peter Griffin GIF of him, like trying to get the blinds to go in the right way with, you know, horizontally and vertically centering things in cs. That was like my life for the first five years, um, in, in a nutshell.
So, um. But I loved it. It was fun. I came to it, you know, every day after work I was still like, oh, let's do some more. Like, let's figure, you know, this other thing out. So, you know, having that I guess part of you is really, really, really important in tech. Um, even with ai, it's still, I think, a really, you have to be kind of like a tinkerer at heart or curious by heart to get into it.
So yeah. Anyways, that's, that's the longer form version of how I got into technology.
Andrew: Yeah, I would. I wouldn't wish block or table layout on my worst enemy.
Justin: Yeah. Do you remember having to like float left and then clear?
Joel: Both, I think was the hack.~~ Yeah. Sorry.~~
I'm, I'm really dating myself not to, but like
even nowadays, I'll start writing a class and it's like clear. No, I gotta stop that, Joel. Like we, we, we've got flex.
Just use Flexbox.
Justin: IE six I eight support. Awful. Awful. Anyway, moving on to brighter topics.
[00:05:21] Building Browser Automation as a Service
Justin: Uh, so you've done a lot since then, uh, and, you know, prompts to you for that. So you've got, you've got browsers now, which is a bootstrap company, which I think is, is incredible. That's the hard mode for building a company. Uh, good on you for, for taking that on.
Uh, could you kinda walk us through, like you, you'd mentioned your aha moment earlier, like, what was that like? What, what was the inspiration? It's like, ah, I need to do browser automation.
Joel: Yeah, so I was working on a site that lets you like, create a wishlist for birthdays or anniversaries or whatever. You know, I'm, I'm sure most people on listening will understand that. Like, Hey, it's your birthday, what do you want? Like, oh, lemme go to Amazon or whatever and create a wishlist. But I was like, I don't want just stuff from Amazon.
I don't love Amazon, like they're great and everything, but like, I'd rather like venture out, you know, to other things too. So anyways, I was building an app that kind of just did that so you could just. Paste a link into something that you wanted and it would, you know, grab price pictures, metadata, all those things. Um, and then at the time it was target.com was running as like a spa single page app. And so none of that metadata was available unless you parsed and ran JavaScripts to get that data. And so, uh, this was back in like 20 18 20 17, 20 17. Um, and so. I, I was honestly at the right place in time, ran into this problem. Um, puppeteer was still like, pretty new onto the scene, I think. Uh, what was it? That was before like puppeteer and headless chrome. There was a phantom, phantom js, just like that maintainer had said, Hey, see, I'm done. Goodbye. You know, Google's got this now. Um, and so right time and place, probably a little early to be honest.
Um. So the first few years were, were pretty slow growth. I mean, I ran the company by myself for the first four years roughly, um, and worked full-time. So, yeah, it is kind of hard mode. You either gotta like subsidize your, your, your living with you, either ramen profitability is, you know, bootstrappers like to say, or you know, you work a another job and fund your. Living with that. So, um, so yeah, kinda worked, kinda worked two jobs for a long time until it got to a place where it was, you know, predictable enough, profitable enough that I could, you know, do my own thing. But yeah, I think that wishlist was like the turning point, you know, thinking about if I ran into this, other people are gonna run into this.
And then I was even looking at like puppeteer issues, like sorted by the most commented. And the first one was, how do I get this to run in Linux? And I think that was actually the aha moment was like, oh man, if this is the number one pain point. Maybe I could do that and start a service that, you know, that's all it does is just browsers as a service.
So, um, yeah, that was the, that was the start and, you know, the big realization. So, and then, you know, looking back now on eight-ish years, like a lot of other things just kind of happened that I wasn't even aware of or, you know, working towards, it just happened organically a little bit. Love developers.
Kind of like how I started at the top of this was like I, I spoke the language. I knew who they were, I knew their struggles. I knew what it was like to be in the trenches with them. So that helped a lot. Um, just kinda like knowing and what the struggles are and you know, speaking the jargon and all that.
So, yeah.
[00:08:31] Running Headless Browsers at Scale: Gotchas, Use Cases, and Scraping Ethics
Andrew: Um, so for devs who might not have like, tried to run browsers, uh, like in the cloud at scale, what problems are there? Why can't I just string together the tools that are out there now and do something similar?
Joel: Oh man. Where to begin? Um, there's a lot, uh, like out of the box. It's gotten a lot better out of the box. Like you could grab play right now. It even has like a sample docker file, and that'll get you like 90% of the way there. There were some funky things a few years ago where it was like you ran the Docker file and inlined the code that you wanted to run. So there's just kind of like some weird API, you know, boundaries. Um, you know, my mental model for like how this was gonna work was treating it almost like a hosted database in a sense like, you know. You get an A URL username password, you connect it, but the database doesn't do everything for you. Like you still need to tell it like, Hey, I want this structure, you know, these rows, those sorts of things. Um, and even today, like the, you know, the playwright Docker files, other ones don't feel like that to me. They're still kind of like very opinionated on how you use that docker file. And I wanted it to be a little more open and, you know, generally available and just kind of have some like best practices and code into it.
But. So that was the biggest problem, I think. And then, you know, all sorts of other things like running into system, fonts, other system packages you might need for this weird part of Chrome to work, or WebGL or, you know, whatever it is. Um, there's just a lot of like gotchas, a lot of sharp edges that you run into having done it for a few years.
And if, yeah, like over a weekend, sure you can get up and running perfect, but then you're gonna be like. Constantly maintaining that baby and it going for the next, I dunno, I'd say six months probably. Um, until you've kind of like, okay, this is not fun to deal with anymore. I, I, I should not be thinking about this problem.
But, um, yeah, that's just a start to be honest. Um, and then there's, you know, new binaries that come out all the time, and so you're constantly having to update that and like, do you have a CI to test that? Like the new binary runs the same way that the pre prior one did and like, no, I mean, we do, we have like. Several hundred specs now that run on every binary change. So, um, yeah, I guess it's like, you know, how much of this problem do you want to own? You know, or do you want to outsource? So that's kind of like the ultimate trade off, I think is like deciding I'm done with this, I'll just outsource it. And that's kind of where we are a natural fit and a good fit.
Justin: Yeah, I imagine there's like all these other sort of like system properties, like browsers are, are very complicated. Uh, you know, you're talking about switching out, uh, out binaries. I mean, I know that you've, uh, sort of written about like. JavaScript memory leaks and like other like challenges that you have at like, running these at scale?
Um, I'm curious, can you talk like in more depth about some of like concrete challenges, um, so, you know, be it memory leaks or you know, process isolation, whatever. Like what have, what have been the big meaty things that's like, been hard to work through?
Joel: Yeah, it's the, honestly, like, if I could put, put it into a sentence, the hardest parts are the things you don't easily find on the internet. You know, there's like weird combination of chrome CLI flags that, you know, help, help it run better or in a more, uh, like user. Ish type way, you know, um, and you know, they're really e even chrome CLI flags there, there isn't a centralized doc site for chrome launch flags.
There's like the, I can't remember the guy's name. Peter something. Peter Lou. It's like some, some developer tried at one point to do a description for all of 'em, but so many of 'em still have like, no, no comments. So. A lot of it's trial and error and like, you know, again, you'll get an update to playwright.
There'll be new, you know, Chrome binaries or whatever, and they break for whatever reason you don't know. And so now you gotta go figure out why that is. And usually it's like, oh, this right combination of node. You, BUN two and playwright, there's a missing package that just isn't there yet. So you're like too far on the bleeding edge and you gotta like, downgrade node or downgrade, uh, you know, you BUN two or whatever the base OS is.
So, um, just like hundreds of those kinds of things really, to be honest. They just pile up and it's like, ah, this again. So you gotta go, you know, dig it back up. But, um, honestly, that's one of my favorite parts. I, I, you know, I love taking something that. Maybe shouldn't work or, you know, isn't supposed to.
And then like coming through, you know, and being excited about that and having that aha moment, like, oh yeah, I did get it. You know, it just took a lot of trial and error. So, um, those parts, but then, you know, that's the big picture node memory leaks are always a fun one. Um, like as an example, we had this interesting. Uh, bug where we were splatting, uh, objects into an array and, you know, splatting from one JavaScript object into a new one and like. Probably creates a shallow clone and then there's a memory leak there. And so it was like better not to do that in a like map reduce function. It's better to just keep the original one and mutate it. So there's a bunch of stuff that you run into like that where it's like, this doesn't look good. It's not a good practice, but man does it perform well. And I don't run outta memory when I'm doing it. So that was a pretty big one. It was like, okay, all the new fun array methods. Sometimes they don't work out, they look good and they, you know, run well, but like the compilers or whatever, haven't caught up to, you know, handling those performantly.
Um, yeah, that was a big one. Whole bunch of other, whole bunch of other war stories too. We've had fun with Nginx and trying to get it to do things sanding and, you know, I don't know. Yeah. It, it, it can go on and on and on.
Andrew: Yeah, lots of not fun issues to deal with there because being on the bleeding edge, uh. Has its, its pros and its cons, but there's usually a lot of cons.
Joel: Yeah, it's funny 'cause now I get people come up and be like, oh, like you're so, you know so much about these things. It's like, no man. I just, like, somebody gave me a hammer and said, learn how to use a hammer. And I just did a bunch of hammering and have bruises all over. And now I'm the expert at hammering is like, no, I just, somebody gave it to me and I just had fun with it and now I'm, I guess I'm an expert or something.
So, yeah, that's how it goes.
Andrew: So you made a pretty useful hammer and it's a pretty generic hammer as well. So like what are, like the types of use cases that you've seen, uh, people use browserlesss for and like, have there been any that just like completely surprised you, you're like, oh, I would never have thought of that thing.
Joel: Yeah, there, there have been, and there's ones that people still haven't used it for and I'm like, man, that would actually be kind of a cool product to build. Um, so maybe I'll give one out here in a minute, but, uh, like the three big use cases we see, I mean like data scraping, which is what I sort of had built it for initially. Automating third party systems that are like either antiquated, don't have a rest, API, you know, those kinds of things, which seems to come up a lot in like third party logistics. Oddly enough, I never thought I would be in third party logistics, but here I am doing stuff with third party logistics and shipping ports and everything. Um, testing's a big one. And then, you know, uh, like asset generation is what I call it, but like screenshots or PDFs or videos. Like, we had an ad tech company that was taking like HTML five ads and converting those to video and so they would like screen record or stream those. Um. Honestly, it's like every day I'm like, oh, that's cool.
I didn't know you could do that. Like, that's actually kind of a smart idea or good way of going about that. Um, but the one sorry that I kind of like leaded up with, uh, was, uh, you know, there's been a big kind of a problem recently with, um, you know, security like NPM security and like package takeovers and, you know, things of that nature. Um, and there's like a, a really good amount of like dev tools in the browser. Like you can. Fire up a browser, go to a site, and you know, just by all the APIs that that has, you can look at the package chain a little bit more and see like, hey, you've, you're running this version of, you know, whatever, which had a takeover at some point and you should probably fix that.
So there's like a lot of really cool security things I think you can solve with, with headless Chrome. And there's been a few, you know, companies that have done that in more of like a different way, but, um, just like. You know, that's a company I think in my mind is like loading up a website and just doing like a quick security posture snapshot and like, okay, you're using these versions of this library, which are bad, or, you know, you're not using S-S-L-H-T-P-S cookies or, you know, whatever it may be.
Um, and that's just, it's kind of low hanging fruit, I almost think. 'cause it's really easy to see that in, you know, puppeteer playwrights. So, um, yeah, all sorts of stuff. Uh, load testing websites to, you know. Governments testing for like, compliance on, you know, federal sites. And it's, it's nuts. It's crazy. Um, every day it's something new, um, which honestly keeps me coming back.
It's a lot of fun to see what people are building with it. For sure.
Justin: Well, going beyond the use cases a little bit and just like to, you know, how the technology works. Uh, I think there's this. Interesting challenge that you're probably facing today. So, um, as AI started kicking up and, you know, companies with LMS are crawling more and more, sites like folks started locking down their sites a lot more services like CloudFlare started offering, um, you know, bot protection solutions, scraping protection solutions.
There's been a long time of like folks like, oh, you know, we don't want people like scraping our data or whatever. I worked for Food Network for a while and we had this problem with like. Recipe data is like, oh, do we like, want people scraping? Like do we try to block them or whatever? And I'm sure as you're building a product for this, you, you kind of go through that same thing.
You're on the other side of it. It's like we're enabling people to like get information and we want to allow that, but the companies are trying to fight us. 'cause you know, the data is their moat and they're like really holding onto it for dear life. So how do you, how do you navigate that? Like what is that
like?
Joel: It depends is the answer. Um, yeah. I mean. I, I've done a lot of soul searching on this topic a little bit just personally and, you know, like, I kind of go back to like, the founding of the internet. You know, like when the internet came about, like the whole point of it was like free open information, you know, like it wasn't locked behind encyclopedias or libraries or books you had to buy.
Like it was accessible to just about everybody, you know, and it was out there for you to, to take and do what you wanted. And so. That's kind of the, like the motive I, I sort of like try to base assumptions on is like, you know, trying to lock portions of the internet down feels almost antithetical to the reason why the internet is the way it is. And so I definitely, I mean, obviously I run browsers, so of course I'm gonna be more on the like, yeah, it should be free and open, accessible, blah, blah, blah. Um. AI does bring a particularly interesting spin now. 'cause now it floats on from like, okay, if, if, if everything's open and free and accessible, like who owns the ideas?
Is, is there, is there an owner? You know, and it's then you get even further out into the philosophical weeds. Like, who are we? What is, is, you know, what a do we, you know, we come with nothing. We leave with nothing. So do what? Do we actually own things anymore? Um, but, you know, anyways, so. Trying to bring it back a little bit. Um, everybody does it, everybody does web scraping at some point or other. And, and it's just, you know, people say they do or they don't, and a lot of people just say they don't. But like, I can't think of a single business that at some point didn't do some kind of like, you know, competitor analysis. You know, obviously like e-comm, this is big, you know, they wanna make sure that their pricing is competitive.
And so in order to do that, you gotta see what other prices are on the internet. So, um. It's just funny, like people will pull that lever to get data, but they're like, oh, we we're not gonna put our data back into that pot. No, no sir. We'll block all that stuff out. And it's like, well, you're saying one thing and doing another now.
So like, which one is it? You know, which side of the fence are you on, I guess. But anyways, I'm, I'm all for free and open. Um, and for the last kind of like example I'll use for this, which is interesting, we're starting to see like cer certain, you know, federal bodies mostly in the eu. Where they actually do need to get through CloudFlare on like fake or, you know, false consumer sites.
Um, I can't remember what country, but that they have like a whole program where they, they literally scrape bad actors on the internet and those bad actors use CloudFlare to, to bot block people checking them. So it's like it does flip on its head at some point. It's like, okay, now you have. Legal bodies that are trying to use this technology to get through to see if, uh, you know, something is legitimate or not.
So anyways, just kinda a fun example to kinda like contrast against. 'cause usually, yeah, it's always about like breaking through CloudFlare or whatever, and you're not supposed to, well, it's like, well no, there's, there's times where that case is actually legal and justified. So I don't, I don't think it's all totally black and white.
Andrew: Uh, so I guess the next question after that is like, do you think there's like a too far in that, like there's lots of legitimate use cases for the product. Do you view any use cases as like, ah, that we, we should be doing that?
Joel: Yeah, I mean, intent is always, you know, an interesting one. Right? So like trying to suss out at the end of the day, like, what is it you're after? You know? Does that, I. Is it good? Is it bad in like a moral sense, a legal sense? You know? And so like there is a bit of a KYC process for us just to kinda like know what you're up to if you're gonna do bad things. Fun example was like one of our first high paying users. Like when we started, this guy was trying to like swing bets on, uh, I think like a poker site. And so he was firing up bots to do it. And I was like, uh, no man, that's, that's not cool. I don't, I don't think that's good. Um, so yeah, no, you can't. Do that on the platform.
I'm sorry. Go, go find business elsewhere. Uh, and, uh, uh, best of luck, I guess in, in a sense. So, um, yeah, it's, it really comes down to intent and that's, you know, that's where it gets tricky and more nuanced, you know, like, what's a good thing? What's a bad thing? Um, so yeah, we, we try to weigh each one in kind and then, you know, make sure that it aligns to, you know, like our own moral and ethical code.
So now, like, now we've gotta go figure out what that is too. So. I dunno, a lot of nuance, you know, some of it's good, some of it's bad, not all of it is bad. Is, is I think, the biggest takeaway. I, I would stress to people that are like, kind of in this space or idiom for whatever reason. So, um, yeah, I think
that, I dunno, to be honest, like that's a big thing for me this year is to kinda like, try to flip a little bit the um. kinda like this thought around web scraping. Every time I like say web scraping to, to most developers, you can kind of see 'em like curl up a little bit like, oh, that, don't say that around me. You know, it's kind of a dark part of the internet, but it serves a big purpose and I think, you know,
kind of bringing that to light a little bit more would do it well.
Justin: Yeah, and I think it's worth acknowledging that like they're just like a lot of companies that don't have APIs, like you have your data, you're a user, you've got stuff in there, and you just can't access any of it. You know, be it from a technical merit of like the, the.
Joel: like,
Justin: Company or product, like, just doesn't have the capabilities to build the APIs or, you know, they
don't want you getting your data because it's valuable to them or whatever the reason be.
Like I, I can definitely see, um, all like a, a wide range of reasons why this is like, good and necessary. I'll use one example that my company uses is, um, we're a legal tech company and we watch the SEC marketing rules. So it's like when they publish a new update, we like, you know, need to know, you know, and, um.
You know, sometimes it's just like, we'll just hit the page and kind of scrape some of the data and get that. And I feel like that's a very like normal Okay. Like use case of that. And if they had better ways of us getting that data, then we'd do it. You know? And like, just like sometimes, especially if it's government entities or whatever, you just, that's the only way you can stay up to date.
You know, you gotta do what you gotta do.
Joel: Yeah, they're
not gonna move fast enough to have rest APIs. And, and to your point, you know, like it seems easy for engineers. Like, oh yeah, I'll just make a API to do that. But like, there's so many things you gotta think about with
that. You know, like, are you, you know, doing tokens the right way?
Are you
making sure that those are all secure? You know, is the whole stack ready for it? You know, I don't know. There's just, you open yourself up to more potential security vulnerability things with those kinds of things. So I get it, you know? It is funny. I have actually, there's been a bunch of new, like legal e startups and they're in it to save hours, you know, they're trying to save
like human effort, you know, by scraping these sites and, you know, yeah, it, it makes a lot of sense to me, but yeah, it's hard to
sometimes justify it to the person depending on who you're talking to.
Justin: Yeah, for sure. Uh, I wanted to, to sort of like tie all this back to a point that you made earlier, which I thought was really interesting. You're, you're, you're, you're saying you're like. You're thinking about, uh, browserlessed initially is like, oh, well I'm thinking about this as like a database service, and it's kind of cool that you have this feature called, uh, browser QL or, uh, like this kind of query language for scraping or, or like more like turning the internet into an API, which I like, kind of like frame it as, which is really cool.
Um, I'd love to hear more about that. How did that come around? Like what does it look like? Kind of, yeah, just more details there.
[00:25:46] Why BrowserQL: turning the web into a typed, SQL-like API (GraphQL + automation)
Joel: I had the thought of this. 'cause that was our kind of like, company thematic was, you know, turning the internet into an API, you know, and then we, you know, tried to like, make, you know, make the other part of me was like, well, let's make getting data on and off the internet as easy as like. A SQL statement, right?
Like all these te other technologies, databases in particular, it's pretty easy to like, based upon like what you're asking, you kind of have an understanding of what you're gonna get back and, you know, that didn't, doesn't kind of, didn't exist until, you know, browser QL or BQL as we call it. Um, and so that was the whole point was like, one, I want to make it more structured and easy and like. It's based on GraphQL, which, sorry if I've immediately lost half of your audience when I said that word. Um, but the nice thing about GraphQL is like the shape of what you ask is the shape of what comes back. You know, there's, it's strongly typed, there isn't, uh, a language or framework barrier. Um, and so there was just a bunch of like nice things you get off the shelf with it that I was like, man, this feels like a pretty interesting thing.
I wonder if I can mix, you know, headless automation with um. Like a structured language, and that way we could offer it to, you know, people that have to run PHP or Scala or you know, go that don't have either like a really nice language to run, um, or a nice framework to, you know, do this type of work with. Um, and so it just kind of opens the doors for those folks, uh, was a big, another big reason. Um, but. There's also just like a lot, you have to, you know, puppeteer and playwright to their points are really good for what they're kind of focused on, which is seems to be more and more towards testing. Um, but there's a lot that, you know, those libraries could handle for, you know, users that they don't today.
And it's 'cause they've taken a much more agnostic approach to, you know, why you're using them. Um, and so kind of having our own first class language was our way of like, trying to do a little bit more and kind of getting at what you know. Outcomes people wanted without having to be so. Kinda like o overly dogmatic about like, oh, we're not gonna do that because it's too opinionated or whatnot.
So yeah, I think those are like the big benefits. And then like having a great editing experience is nice. Like if you play with our editor, it's like you write the statement, you run it, you see what the browser's doing. You got dev tools. All these things, it tries to like help coach and recommend a little bit, like what could be better, what couldn't be.
Um, obviously it's a new tool so it does take a little bit of a hurdle to get over to learn it, but I think it's worth a trade off 'cause we do try to simplify a lot of things and not have to like, make you think so much about like promises and, you know, race conditions like it that that all's kind of handled for you, which is nice.
So, um. I love it. It's my favorite part of the pro the product, to be honest. I love, like that's the first place I go and I need to like figure something out. So it's kind of nice to have it all in one place and docs and everything and you know, like one stop so you're not having to like context switch a bunch.
So, um, yeah. Yeah, that, that's a lot. We'll, we could dive into any of any, any of those things. Um, but that's, yeah, the kind of the big teal or very long did read part of it.
Andrew: Yeah. I find it super interesting. You guys created a whole language around this, like an IDE, like how far did you guys go? Do you have like a language server? Like how, how, how, how languagey is it?
Joel: Oh man, I wish, uh, no thank, thank God we didn't write Alexa and a parser though. I'm kind of debating if we should. There's some like bottlenecks, especially in the node implementation for like GraphQL js. Just, you know, being single threaded in some of the. I think some of the lexing and the parsing is still ran on the main thread.
I don't know if they do it in the background, but they may have a seal library now that does that Anyways, neither here nor there it is. That stuff is thankfully off, off of our plate. Um, yeah, those are, those are really challenging to solve sometimes. And I think, you know, anytime you adopt a new technology, it's like, how much do you wanna like put in this to yourself?
How much ownership do you want versus how much can you just get off for free, you know, if it meets your needs. So, um. Kind of another reason for going GraphQL is like their editor is all componentized already in React. So it made it really easy to like build this remote viewer for the browser and the dev tools part was really easy 'cause we could just swap out those components.
So, um, yeah, big props to like MATA and their dev team for like, kinda like getting the fundamentals right and getting the building blocks right, such that, you know, we can come and do this pretty easily, which is amazing to me, to be honest.
Justin: Yeah. I think it's a really interesting use case for GraphQL. I, it's all a random PR for like someone wanting to do, or, or an RFD or something for like GraphQL and chrome dev tools as a protocol. I don't know. It is something like that. I don't, I don't remember exact context. It just like cropped up on, on, on Twitter and I was like, oh, that's interesting.
Joel: ~~Huh That, ~~
yeah, that is interesting.
We'll see if that happens.
Justin: maybe relevant and useful.
Joel: Yeah, it's GraphQL some somewhere, some layer.
There's GraphQL Yeah.
all the way down.
Justin: Um, all the way down. Uh, so yeah. U
[00:30:41] Browserless 2.0
Justin: h, kind of wanna talk about your, uh, the browserless 2.0. Uh, so y'all done a, a, a lot of work. It's been a long time, you know, kind of working on this like, revamp of browserless. It's been, um, it's been a few years now, I guess since that, uh, point though. Um, and I'm curious like.
What sort of like led up to that and like, where are y'all at today? Um, have you like done another major version since then? Are you still sort of like rolling on those too and like, I don't know what's going on? What's new in the ecosystem? Uh, Yeah,
Joel: so 2.0 came, we wanted to have support for other browsers just besides Chrome and chromium. So we do support, uh, WebKit and uh, Firefox as well Now. Um, but supporting those made us rethink a lot about like, how routing worked. Um, and then like, you know, what would need to change to, in order to support those two.
And like, you know, making it work in the, you know, 1.0 branch would've been probably possible, but there was just. 1.0 was so focused on Chrome and chromium that it felt like odd to have these other two vendors as well in there. And to be like, it just feels kinda like left-handed, didn't feel like, uh, sort of like intentional.
That's what we wanted. Um, so it was like, I'm, I'm not a huge fan of breaking changes, I feel like. You should spend the time. You know, think about it, a long time, the 2.0 release probably took me eight months to just like going back and forth on like, how do I want routing to work? What should those look like file wise?
You know, what should the system take care of for you and what should it not? You know? And then like honestly like building that up, using it and being like, Nope, this isn't it. This doesn't feel right. I'm gonna go back and rethink that part of it again. Um, and so. I mean it's, it was, you know, the culmination of all the years of building the first product and like going back to the drawing board knowing that we have big breaking changes we wanna make, like what, what other things do we want to do? Um, so yeah, we, as one does, we built our own router. It supports web socket and rest at the same time, which was a big one that was hard to find and still kind of is even in like next js I think it still feels a little awkward or left-handed to have a web socket server. 'cause you kind of like. Bind it to a global object, if I remember right.
And then it's just kind of like accessible via, um, you know, outside references to, to get access to that object. And it's like, no, that doesn't, none of that feels right. So, um, anyways, just seeing like all of that and wanting to use TypeScript and really like, leverage a lot of typescripts. So we have this fun build process where, you know, routes can define what they accept, what they return, and like query parameters to. At build time, you know, we look at all the routes, we take all those definitions and we convert 'em into open API schemas, and then those are verified at runtime. Um, this one took quite a while and actually that build file is kind of insane for that, but, um, it gets us to a lot of stuff for free. So like if we have a property on a post body that is like directly corresponds to maybe like a puppeteer, API, any puppeteer. Documentation comments they have on that API actually get passed through all the way to our route definitions and in our open API, um, build as well. So you can like go and see exactly like the puppeteer notes for that particular property, which I thought was like, oh, this is sweet man. Like, I got this for free.
Like, I don't have to go and like pains painfully like annotate, like what are, what is a cookie in, you know, puppeteer parlance, like, it just passes through for free. Which is, which is awesome. Um, and so yeah, anyways, all those things, you know, multi token work as well, like, wanted to just do a bunch of different, you know, features and stuff, which was just really hard into it, uh, to get into 1.0. So 2.0 is like rebirth of everything, and like building on what made it successful, but also like keeping it more open for, you know, browsers and other, you know, better DevX performance boosts. Um, and yeah, honestly, I'm happy with it. Like it's been, like you said, I think a couple years now, maybe a year and a half.
I don't really envision a 3.0 to, to be honest, like, yeah, don't break change, don't make breaking changes unless you really have to. I think we actually have a, a blog post about this coming out soon. It's like, yeah, like don't, don't break if you really don't need to. And, and if you do break all the things because you might as well just get it done, you know, when, when that event happens. So, um, yeah, fingers crossed there'll be, there'll never be a 3.0 is my, is my go-to. So we'll see what happens though.
Andrew: A good goal. It's a, a lofty goal. It's, it's surprising. We work on a platform, the web that does that. Like, it's kind of crazy to think that, like they try to make no breaking changes forever. Like, I can't even. Imagine as as a library author.
Joel: yeah, that is a good point. You, you bring up a very great point, like the fact that you can run JavaScript from IE five days today and it still parses and executes is like, that is a miracle that that ever happens. So, um. Yeah, very true. Very true.
Andrew: Yeah. It kind of goes back to the data access stuff we were talking about. Like the web doesn't feel like a platform that can die as easily as some like iOS apps. It's like Apple can update the the SDK and then eventually you have an app that can't run on anything but an emulator that you can't run unless you have an old computer.
It's like multiple layers of like, I can't do it anymore.
Joel: Yeah, no that, sorry, I, I had callbacks to running like the IE VMs from back in the day. I'm like, virtual box. I'm not sure if any of you have had to do that. It was like, run the IE six VM and it's like, yeah. And then those stopped working after a while Microsoft pulled pulled the plug on those and it's like, whoa, shoot.
Now what am, what am I gonna do? I have to go to like a secondhand store to find a pen three or whatever to run this operating system again, like, uh, yeah. It is crazy. It is crazy.
Andrew: Cool.
[00:36:20] Open source, licensing, and AI’s impact on the business
Andrew: So moving on, uh, to the next topic. Uh, we like to talk about open source and business models a lot on this podcast. Uh, you have a lot of open source out there. It seemingly you can run your own browserlesss. So, uh, can you walk us through like how you think about open source and how you think about your product in relation to that?
Joel: Yeah, so I, you know, I, as a, as an engineer myself, I got tons and tons of value from open source, like I. You almost can't give back enough for the amount of, you know, benefit there is to open source. Um, and so, you know, thinking about that and also, you know, kinda like a personal sort of skill trait motive that I love is like showing, not telling necessarily. Um, and so that was another reason for like trying to do open source. So like, I get it, people are in very different stages of their journey where it's like, I just need to do this thing on a personal project or whatever. It's like, great. Yeah. Don't even, don't even, don't even talk to us. Like just grab the docker file and go, go have a good day.
Um, you know, like I don't wanna have to like loop you into a bunch of sales calls and like do licensing and blah, blah, blah. So the motive here, you know, for me to do open source was like. One to show people that like, yeah, you can just, this is how it, how it's done, how we do it. And so if that's what you need, awesome.
I'm glad that we could have like, helped you and hopefully, you know, someday you might need us or whatever. So, great. We've had a good experience. Um, the second part too is like, I, I was, I was never an engineer, you know, I, I never went to school for this and then like, I kind of wanna like show people like, hey. You can do really cool things, like really important things, and maybe they're outside of your reach and like, use me as an example. And like, this is kind of like, I don't know, a way of showing a story or telling a story too that's like a little more, at least grounded for engineers to like, you know, Joel, he, he, he went from never having like opened up Git or a CLI to like doing this.
So, um, I think it's just another way to like illustrate what's possible for people. Um, and especially now with AI too, it's like, it's even more accessible to just about any audience. You don't have to be an expert. I think there probably still is a, a reason to be an expert, but like you can get a lot done and not know a lot, which is pretty cool.
Um, obviously be careful with that, but, um, yeah, so those are kinda like the motives. Um. So, yeah, the open source Docker image is free. You can download it, you can run it. It's never gonna ask you for a license, it'll never bug you about it. It doesn't call back home. Um, it'll log, I think like a couple links and that's it.
So you can feel good about it that way, that you don't have to like drive, you know, go to a website, sign up for a license, keep put it in, you know, all those sorts of things. But, um, yeah, so on the commercial side, it's like, yeah, if you're gonna use this for like, something proprietary, then that's when you, you wanna contact us and get a license or, you know, it's hard to put that in like legal terms.
There's always. I think we've, we've tried two licenses at this point. I like the second one. Okay. But I think there's still some nuance I would try to tweak, but, you know, not a lawyer. It's really hard to get that out and like not piss off the wrong crowd, so to speak. Um, 'cause yeah, you know, at the end of the day it's like obviously got bills to pay and stuff.
So, you know, if we helps you be successful in your journey with whatever, we would love to partake in that success with you. So yeah, that's kind of the overall, I guess, motivation. Thematic ness around open source. But, um, yeah. I'm actually kind of curious. Ha ha. Has it varied very much from folks that you've interviewed, or has it been, it seems like a crazy, you know, to me it still is messy.
It's like kind of a problem, to be honest. It's like funding open source projects doesn't really
have like great answers. There's some, but they're like, you know, they're too far on one side of the spectrum or another. I'm kind of curious if you've, if, if both of you or either of you have seen,
you know, changes in that.
Justin: yeah, I mean, we've seen all sorts of things. I, I do think like you see more and more of the like open core model, like collapsing a little bit. Um, you know, uh, this has like, been very classic around like database companies, for example. So you use the Mongo DBE like server side public license, which came about because their is being cannibalized by other cloud providers, right.
And, and of course like, um. You know, license, drama around licenses. It continues to be a thing where people feel like they deserve. Yeah. And, and to any engineers listening to this, uh, I'm sorry for, you know, hopefully this doesn't come off wrong, but like, we are a privileged class and broadly. Can feel like we, you know, like if a, a maintainer decides to change a license from one to another for their personal, like, benefit to make a project sustainable or whatever, people take that personally, you know, like, I mean, and it is especially true if it's done by a company, um, because you know, people feel like it's a rug pull or whatever.
And like, also, like all understandable and, you know, we can empathize with it, but it is, it's fraught, you know. Um, and I think the really interesting thing is we haven't actually had anybody on the podcast to talk about this recently, but like. There's a lot of open source projects that are actually closing to contribution because of like a bunch of like drive by vibe coded prs and like just massive amounts of like new contributions coming in and they're like, look, I'm not even gonna try anymore.
Like, open an issue, open a dis or like open a discussion on GitHub or something like that. And then maybe we'll sort of like talk about it, but like, I'm gonna close your pr, I'm gonna close your issue or whatever. And. That's also the really interesting thing for me that I'm seeing a lot more of. Um,
Joel: did not,
that is crazy. Yeah. We've um. We haven't had a ton like that necessarily, but um, we have had, you know, like punctuation changes in prs. It's like kind of gaming, like how many prs have you opened up in the last month? A little bit. So there's like a, a little bit of a gamification going on. Um, so it's like white space changes.
Punctuation changes, like, what the heck? Fortunately for me, it hasn't been too much of a problem just yet. But man, that would be. I always think it's always just a good practice to open up an issue and talk about it. Like, Hey, you know, this is what, you know, I'm seeing or, or would like to see, here's the strategy, I'm gonna go down, thumbs up, thumbs down, you know, before I go
and like, waste my time building out this poll request that you're not gonna even review.
So,
um, yeah, I guess I'm not shocked really to hear that, to be honest, but kind of sucks to have to like close down all of that, you know, just because of the amount of abuse.
Justin: I think the other thing I'll add, and I know that you've talked about this in other forums, is that like we're also seeing the sponsorship model is just like not scaling. Um, there are very, very, very, very few and usually only big names, like maybe like Tanner and maybe like Evan U of View, and like you could probably count them on one or two hands.
The number of people that are like really successful. Doing sponsorship and they're also really, really freaking good. Like top, you know, top 0.001% or something of like developers. So it's like, I think that's an incredibly hard thing to do well to get any sort of like actual value out of. So.
Joel: Yeah, I
agree. Uh, even back in 2018 or whatever, when we was trying to think through some of these things and like go a sponsorship route and do like, you know, consulting or whatever, like even then it wasn't. On a good trajectory was my thought at the time. And to your point, it's like, yeah, you really had to be in the top point, 1.01% of, you know, authors out there to make it work.
And then, then you gotta deal with like consulting potentially to like subsidize what it doesn't do. Um, yeah, I don't know. I've even thought about like, oh, it'd be cool to have like a marketplace of like apps, like Dockerized apps and then you could charge, you know, I'm actually kinda surprised that Docker didn't do this, to be honest.
'cause we needed at one point, like a way to host like. Kinda like private images and like, oh yeah, like other users wanna license this from us. So like why can't Docker just like have a paid model where, you know, like they're verified, they get all the security stuff or whatever, and you, I dunno, pay for poll or, I dunno how the model would necessarily work, but like. It seems like there's a need. I, I, I know I not have it, of course, but I don't know, just kinda an interesting business tactic I thought would be, you know, and it would help with other people that like, Hey, I built this really cool database engine. It's over here, you can pay for it, blah, blah, blah. You know, and like, yeah, I guess it depends on what part of the stack you're working software wise, but if it was like a fundamental layer or like a, you know, service of some kind, then that, that route seems to. Could have been a lot more sustainable, I think, going in the future, but we'll see.
Andrew: Yeah, I've often thought the same thing of like MPM, like there's this like kind of shadow ecosystem of like packages I can pay for, but like never know about. It's you just kind of like happen upon them. You're like, oh wow, that here's a private registry thing that I. Could maybe do, uh, but that it's not baked in.
Seems like a obvious miss for developers and open source in general.
Joel: Yeah, I mean, just think about the amount of duplicative effort going on right now for like just supply chain verification. You know, like anytime we get, you know, get through a company and we're going through like a vendor onboarding, it's like. They have to do that all over again. And it's like if there was just some like supplier or whatever out there that just said, okay, like these are all verified.
You don't have to think about it. Here's like a certificate or something for it. And then like it could just like get you through a lot faster. It would be super nice 'cause every business like has to do the same process over and over and over and over and over again for every new vendor and it just like, ah, this could be consolidated in some way and mass streamlined.
But I dunno, if you're listening, go build that. I'll be your first customer maybe.
Justin: ~~You wanna do the maybe Andrew, uh, given we got like 10 minutes or something, maybe we do just do a wrap up question and. ~~
Andrew: Uh, yeah. Uh, w the, we kind of already covered the data access one a whole lot, so,
uh, we can skip that one. Uh, let's talk about ai. Um, so we said the magic words a little bit while ago. Ai, it's changing the industry, really changing everything. Uh, I wouldn't have said that a year ago. It's kind of crazy for me to come full circle on this.
Uh, but how has AI played into browserless and like, what changes have you made to kind of accommodate this new world that we live in?
Joel: Yeah, it's affect us on two fronts. I mean, we're consumers and producers in a sense for, for AI and you know, models. A lot of those, you know. Things out there that, you know, need real time access to the internet, need a web browser at the end of the day to do that part. So, you know, we are, I, I guess, a cog in the wheel, so to speak, for a lot of, um, AI ambitions going forward. Um, you know, it does have a lot of same symptoms to like the data access problem. I know we kind of touched upon that a little bit earlier, but it's kind of the same questions I ask myself at the end of the day, like. Did who, who owns these ideas? Do we own ideas? You know, who, who gets the dollar for it?
And, you know, those are big, like bigger than me, philosophical questions to a degree. Um, but you know, I think at the same time, genie's outta the bottle, you know, this is
like the internet again, but potentially even more so. And so, like, it's open, it's out there, so you know, what are you gonna do about it or how, how are you gonna react to it?
And so, you know, kind of looking back on my history with, you know, getting into dev, getting into like business running. I really stress to people that are like listening that are kind of scared about it. 'cause you know, I've went through that like depressed sorrow of like, shoot, all my skills I've worked so hard for are kind of like replaced.
I'm like, no, you still need those. You still need like that compass, that internal like. I dunno, we called it like your spider sense for bad code or whatever. You know, there's that, like that thing that's inside of you that like, this is good, this is bad. Like that is still very much needed. 'cause you know, obviously you're not gonna get perfect code from an AI as it stands today. Um, but the amount of just like boring problems it can solve for you is just crazy. And so, like once you go through that valley of depression with it, you, you'll come out and like, oh man, I can do so much more. I don't have to think about like, wiring all these things up and. You know, as long as you're okay or good at reviewing code, like you'll get a lot of benefit out of it.
Um. So, you know, in, in terms of like the, the discipline of software engineering, I think it's, it's still a tool. I don't think it's gonna replace anybody soon. I think, you know, they're still gonna need somebody to like contextually understand things. And, you know, there's a limit in, in my mind of like, language, you know, you can only like articulate so much about how a system can work and sometimes it's like you just need to know a little bit and it's hard to put into words exactly like behaviors or whatever.
So, um, at the end of the day, I had never met a PM that just couldn't. Tell me in perfect terms to get a prompt to work. So I feel like, you know, your jobs are still safe. 'cause you can like, Hey, what about this, what about this, what about this? And you know, AI is not gonna replace that anytime soon. Um, so yeah, anyways, people are doing really cool stuff with it.
I think, you know, it does feel like early internet times to me a lot so. I think trying to leverage it as best you can and being part of the narrative on like what's gonna happen with it is more important than just sitting there frozen. Like, oh God, what are we, are we gonna turn in the matrix? You know, like, is everybody a machine?
Am I married to a real person anymore? Like, no, this is a chance, like, you have the time, you can go do something with that, you know, for good or for for bad or for nothing and, and just let happen to you what happens? So, um, anyways, I always like having a little bit of autonomy and like dictating like what happens and what doesn't.
So. I think that's kind of where I veer is like, I'd rather make use of these tools, leverage them, well talk about them. Well and like good in a good sense how to use them too. So, um, yeah, I mean it's crazy. It is a fast, it is fast paced. Every day I wake up and there's like a new N eight N or lovable or something like that that we're, we suddenly gotta be a part of.
And I was like, oh, that's cool. How come nobody told me about this and we're like, uh, an adapter into their system now. So I guess I better figure out how this works. So, um, yeah, it's fun. It's a lot of fun. Um, I hope it continues to be fun and I hope it does veer in a good direction, societally. Otherwise, we may all be robots someday.
I don't know.
Andrew: Yeah, it's, it is so many unknowns. Like, I've thought about this so much, and like, I go, my, my brain goes back to like the printing press and like all these other innovations that kind of like expanded access to
information.
like the internet as well. Like, there's so many good things that comes from it, but like the dark side, the bad things that we can't predict and how it'll change our society. So it's like it's, and the speed this time is just like unprecedented too. So it's like, it's a, it is a crazy time to be alive, honestly.
Joel: Yeah, no, I think, yeah, the speed at which it goes and like, I, I, I, I, I just keep coming back. It's a tool at the end of the day, you can use a tool for a good thing or a bad thing, so, you know, but like never using the tool and letting it, you know, it or somebody else do something with it, you know, I think is not good either.
So, I don't know. Yeah. Use, use the tools for good things, I guess, is the, is the answer there.
Andrew: Okay.
Uh, well that wraps it up for our questions. Uh, thanks for coming on, Joel. This was a fun, uh, conversation about everything, browserless and all the data access and our lives and ai. It was, it was a great time. So thanks for coming on.
Joel: Hey, appreciate it so much. Thanks for having me. It was great chatting with you both. Um, and we'll see you on the flip side of the AI storm.
Justin: Yeah. Thanks again, Joel. Uh, it's, it's really cool to see what you're building. Uh, I think this is a hard space. Uh, I have a lot of appreciation for it, having had to do a few of these things in the past and, uh, yeah. Y'all keep up the good work.
Joel: Awesome. Thanks.