Paul: What thing could we even build that allows us to not have this incredible complexity? And the incredible complexity I'm referring to is, is the infrastructure side of things.
So dark was really, How do we build without the user having to think about any of this stuff?
Andrew: hey, before we get started, we like to direct you towards our merch store. You get some cool dev tools FM merch, or even this TypeScript hoodie. Show how close to the metal you are with this one. If you remember of our Patreon, you get a 20% discount code. Now let's get onto the episode but remember the full episode is only available to our Patreon members.
Hello, welcome to the Dev tools FM podcast. This is a podcast about developer tools and the people who make them. I'm Andrew, and this is my co-host Justin.
Justin: Hey everyone. Our guest today is Paul Bigger. Paul was the founder of Circle CI and spends time these days working on dark, uh, an integrated development environment for building serverless backends.
I'm sure that's not really doing dark justice. Uh, so Paul, I'll let you describe that better, but before we dig into dark, uh, maybe you can tell us or tell our listeners a little bit more about yourself.
Paul: Uh, thank you. Thank you for having me on the podcast. Um, so yeah, I'm, I'm a developer basically. I, I've been a developer for about 20 years. I did a CS degree, I did a PhD in, in, um, compilers in so analysis, and then I'd mostly done startups. So, you know, I, I started Circle ci, uh, I was the, I was the CEO there for the, for the first four years.
Paul: Kind of the first one of that. It's, it's writing code and the rest of it is, is not the much writing code. Um, and then I've been working on dark now for about, uh, for about five years. Um, and that as of recently, it's, uh, it, you know, as the last couple of years it's been an awful lot of, of writing code and, and just working on making the product, uh, really good for.
[00:02:04] CircleCI
Andrew: Let's, let's start with Circle CI for a little bit because, uh, I personally have been a very avid Circle CI user. Uh, when, when I was, when I was Intuit, I was very gung-ho on trying to switch from Jenkins to Circle ci cause the experience is just so much better. Uh, what prompted you to found Circle ci? Uh, and was it your first startup?
Paul: Uh, it was my third startup, um, the, the other two having just not gone anywhere. But the, the experience of having done them real, you know, kind of, kind of stood with me. And, and so, or circle was, you know, kind of my, my first, uh, startup that went to anywhere, um, where that, that, that really got off the, uh, off the
Paul: bottom rung of the ladder,
Paul: um, started it largely because. Uh, I, I was working at Mozilla. I was a, I was an engineer on the JavaScript engine, and so all the time I'm using the, the tool that they had at the time, the, the, you know, what we would now call, uh, a CI server. Um, and, and it kind of sucked. And so I, I kind of spent a year banging my head against this thing.
Well, one thing in particular is that if something failed in the CI server and worked locally on your. You know, there's nothing you could do. You, you could, you could make a, um, um, a ticket and then they might be able to get you access to the server in the next few days to sort of see what went wrong.
And so one of the first things we did at Circle was we, we built the ability to, to retry, uh, a build with SSH enabled. Um, so it, it, it's that sort of thing, just sort of like, you know, the experience of using this thing is terrible. Um, and, you know, I think we can make something.
Justin: Absolutely. That retry with SS H's feature is, is was gold. I use that many, many
Paul: Yeah. PE people, people keep, uh, keep building, um, uh, ci. Competitors that don't have that, I'm like, What, what are you doing?
Justin: For sure. So you, you're talk, you mentioned like, you know, the first year of circle was like a lot of programming and then to sort of get after that and it's like less and less programming, more and more business stuff. So what's the experience like shifting from, you know, this like developer hands on, You're just the whole team.
It's just getting together to build product to this like higher level. You're now, you know, executive board member, you know, like out of the, the day to day product delivery. What does that, what's that transition like?
Paul: Well, I, I, I, I guess it's sort of the. The company transition, like the, you know, the, the, the very early stage is fun and with circle it didn't, it didn't last very long. It was, it was maybe six months really before we started having, having a decent number of users where there was a novel lot of user discovery.
And, you know, like, uh, literally six months after, wrote the first line of code I was spend. Significant portions of the day, just like calling up users and saying like, What went wrong? You know, what can we fix? What can we improve? Will you give us money? Um, that, that sort of thing. Um, and, and then, you know, the next six months after that, I largely did support.
So I was, I was, you know, answering every, every support email. And it was only then that we started to have a team. Um, so that's, that transition actually wasn't too bad. You know, I really enjoyed. The small team time, we, we maybe had a year of just kind of, um, everyone, everyone sort of figuring out what to work on for themselves.
And then that broke down and nine people, and then between nine and 25 people was when I was just in disarray. I just had no idea what to do, where to where to spend my time, you know, all this sort of thing that, that, that you read about in, in startups about like, you know, setting the culture and setting the mission.
You know, um, more obvious things like product roadmaps and, and management structures and that sort of thing. You know, just like, and you know, part of the thing was we weren't really into some of that and there, there was a lot of like flat company stuff going on. Don't do it, it's a bad idea. Um, but, you know, at the end of, at the end of four years of that, I was just like, I don't like this job anymore.
This, this being CEO of a 25, 30 person company where. You know, the, the, the time that you're spending is like building this company. Um, you know, it's, it's not a super fun job. You know, I, I, I really, I really enjoy coding. I really enjoy working on, on the product and, and making improvements to the product and things that a user facing and, you know, hiring salespeople and drawing a commission plan. Do not.
Justin: Yeah, yeah, yeah, for sure. Uh, so now that you're no longer at Circle, do you still use it?
Paul: Yeah, I, I use it like literally every day. Um, yeah.
Andrew: So is is is dark built upon Circle ci?
Paul: I mean, it's not built on it, but we, you know, we, we, we have, uh, traditional code bases. You know, we, we, we have a client and a server. Um, And there's also a little bit of other stuff, but, you know, that's basically it. And so we, we have a, you know, a monorepo and a, we have a CIC CD pipeline for it. And there is, you know, uh, 11 steps and there's a, you know, deployment step as well.
And, and you know, the kind of things that, that, that you're expecting in a standard sort of webby SaaS kind of project.
Andrew: Awesome.
[00:07:46] What is Dark?
Justin: That's awesome. So maybe we can switch over and start talking about dark. Uh, so Dark is a super, super exciting project. I, I've, I've followed it for a while. Um, so how would you describe Dark to someone who has never heard of it?
Paul: Uh, I'm assuming they're a coder.
Justin: Yes.
Andrew: Yeah.
Paul: Yeah. Cuz otherwise the first step is explaining what the cloud is. Um, so the way I really like to think about dark is, is getting rid of all the bullshit from back end development. Um, I don't think that, that, I approach it from the idea of like, Oh, I've got this, I've got this great idea for how software should be written.
It really came from the idea that why do we do all this bullshit? You know, why do we have to deal with all this shit? And surely there's a way of writing code that that
Paul: doesn't have. to
Paul: Um, and, and so it, it's really about like taking
Paul: things away. and The, you
Paul: know, you described earlier kind of an ide for, for
Paul: building cloud applications So, you
Paul: know, that's, that's, it's not entirely wrong. Um, but how we approach it is like, you know, what thing could we even build that allows us to not have this incredible complexity? And the, the incredible complexity I'm referring to is, is the infrastructure side of things. So like, you know, everything from, uh, spinning up more instances and kind of.
The serverless side of
Paul: things to CICD pipelines
Paul: to, um, you know, database migrations. Uh, anything where, you know, kind of should be obvious and instant. And instead we, we've kind of built up this, this legacy and whether that legacy is like containers or Unix or, um,
Paul: you know, IDEs and syntax
Paul: and then that sort of
Paul: thing, it.
like None of this
Paul: stuff is, is actually a necessary part of coding. Like the only thing that really matters for coding is that you've got, like, you've got input, you've got output, you've got some computation that you want to do on it. Um, you know, tho those are kind of the essential things. Um, and, and we have this, you know, 50, 60 years of stuff that we've built up, layers that we keep adding stuff on top of.
Um, and so dark was really, How do we build without the user having to think about any of this stuff? The, the developer having to think about any of this stuff?
Justin: Yeah, that reminds me. Uh, so back in episode 26, uh, we talked to the founder of, or the creator of arc.codes, which is a serverless, sort of like rain mark. And we talked a lot about like incidental complexity and, and how more and more paradigms are, are evolving to sort of get us past that complexity and focus more directly on like, what is the product we're building, not as, what is all the stuff we have to duct tape together to even get to the point to build the product. Um, so yeah, it's a, it's a, it's a super admirable goal and I think highly needed.
Paul: Yeah. And I think, I think they're very much on the, that is arch codes is, is on the same page as us. As, as like, you know, we, we, we have to remove things from the. How developers think about what they're building because there's a lot of people out there who are solving this by adding more things. Um, and, um, those are probably, you know, in many ways, more practical, um, you know, solving, they're solving the problem today.
Um, and I, I, you know, have a lot of, have a lot of love and affinity for the people who are doing the, this will not actually be really good for five. . Um, but we managed to remove this, this, and this. From, from it.
[00:11:35] Moving Past the Past
Andrew: So I. Dark Lang has this theme of like shedding the past and making a new way to develop your backend services. How does that differ from traditional methods? Like what does writing a dark application feel like?
Paul: Yeah. So, The, you know, I'll kind of talk about it from, from the, the way of, you know, that we're removing stuff. So one, one of the obvious things that we're removing is, is infrastructure. So when you set up a database in dark, you just add the database, like you click a button or you, you use
Paul: the keyboard.
shortcut
Paul: or, or whatever. And. Data stores there, and there was no, you know, there's, there's no requiring a server from somewhere to put it on. There's no, you know, going to, you know, some other service, you know, inserting a, a URL
Paul: for your postgres config, whatever.
Paul: Um, The, uh, s SCHs are just, you know, specified in, in the same language as, as the rest of dark.
So there isn't SQL there. Uh, when you make a request, uh, or when you make a query, you query in the dark language, which is the same as the rest of the language that you're using. And it's just like these functions are com, you know, behind the scenes compiled down to, to the SQL that, that you would need.
In, you know, to the, to the developer. It just, it just looks like the, the, the same language and the things that you pull out of it are again, the same language. Um, so there's no, there's no orm, there's no impedance mismatch between the values that you store in the database and the values that, that, that, that you store in and the language.
Similarly, when you're, you're
Paul: setting up a HTTP request,
Paul: you're not
Paul: setting up a HTTP server,
Paul: you're just add an end point, um, and the end point is directly connected to the code and you write them in the same place. There's no spinning up servers. Um, yeah, I think there's a lot
Paul: of serverless for, there's
Paul: no spinning up servers, but there's also like no API gateway.
There's no, you know, it's just all kind of native. The values that come in are, are native and they're in the dark language as well. And then when you're, when you're writing code, um, you know, you wanna remove of. The, the latency between when you write code in your computer and it going into production for your users.
And so we, we said, you know, that time is gonna be zero. There is, you know, There's not gonna be any building containers, there's not gonna be any like, um, pre-deployment steps. It is just you write code and it's immediately in production and you can, um, and this one I think is, is one of the, one of the hardest ones.
And it's still something that, that we continue to work on because you need that to be safe. So the language is designed in such a way that every key character you type can safely go into production without production, just having a syntax error or the server
Paul: shutting down or , um, or
Paul: that, you know, there needs to be a way to write new code without breaking the old code.
And so feature flags are baked into the language. They're baked into the editor, they're baked into the infrastructure. It's all kind of like designed around this zero second. Um, we've been calling
Paul: that deploy less
Paul: uh, someone calling that a couple of years ago. Um, and, and that's kind of the, the word that we've been using for it.
And it's just, You know,
Paul: every key stroke goes straight
Paul: in production. How can that be done safely? And so as a result, you have no, you know, you've no, you've no ci pipelines, you've no like extensive unit tests or integration tests that gets run. There's no deployment process like with, with Kubernetes or, or whatever.
It's just sort of like, you know, it's all built into this one environment and it's all like cut down to the absolute minimum that it could
Paul: possibly be.
Justin: Yeah, the structured editor of Dark was like one of the really exciting things that I saw in the beginning because it's like that model I've wanted to take off in some way for a long time. It's like, you know, we, we deal a lot with like formatters and you know, have flame wars over like tabs versus spaces and you know, just all this stuff and it's gotten better over the years. Right. With like auto formatters and stuff, but.
Paul: Mm.
Justin: Still, you can make mistakes and it's like time wasted. Just, you know, typing characters, you know.
Paul: I mean there, there, there's quite a number of, there's quite a lot to be said about kind of auto formats and code formatters and kind of how they've made the world, uh, a little bit better. Um, I, I, I think they, um, you know, if you have a repo, you should have an auto formatter in it, which of course becomes more complexity that you need to
Paul: set up and then you know,
Paul: you have the issue that some of your users have one version of the auto format or, and then some have another version, or you need to, you need to migrate between versions of them and there's a different formatting change. And so now there's, there's, you know, something throughout the code base or there's a code base wide diff that you need to apply to do this thing.
And the people whose branch has existed before that need to be rebased onto it. And so, That's the kind of thing that that, that we're trying to evolve even, even, you know, very small things like, like code formatters bring in complexity. And I totally agree that the world is better with code formatters rather than without it.
Um, but even then, like can you remove that, that complexity? Can you abstract over? And one of the, one of the nice things I, I've met, I meant to implement this a couple weeks ago since we now are, are at a place that we can do it. Um, but you know, why does every user have to indent
Paul: the same level? Some people
Paul: like indent two.
Some people like indent four. Some people like to line up the, the right hand sides of, of their, um, of their records that they're using or something like that. Some people don't. These should be, these should be settings for users, um, that they set. And making the setting shouldn't affect what
Paul: it is for everyone else.
And because
Paul: we store code and text and we put that text in a GI repository and every user has the same version
Paul: of that git repository.
Paul: Um, and this, you know, it pulls down the same text. We don't really have any sort of like, advanced stuff and we still have to deal with kind of all this bullshit as a result.
Andrew: So I, Is there any sort of version control built in? So like it, I, I'm assuming it doesn't, doesn't work with git Right.
Paul: It does not work with gi. Um, very, very much intentionally git, uh, git, uh, like many things get is wonderful, but also get is very complex and, um, I'm not sure that the model that it brings to the world, um, is super valuable. Um, and I think that, that we do a lot better without it. Um, so there is version control.
Paul: Version control is. let me talk about what exists because most of, most, the dark is, is still speculative and, and still is sort of designed, but
Paul: it's not done. So right
Paul: now there's, it's, you know, dark is sort of a multiplayer canvas. Um, and it has infinite undo, so, you know, you can undo any, any changes to the thing.
Um, where we are going with this is that feature flags are essentially branches. So they, they are checkpoints that you can go back to, um, to, to view the old code. Um, they provide the benefits
Paul: of branches. Um, so I,
Paul: if you think of feature flags as, as a branch, right? The, you're building something in production, and so there isn't really this whole separate branch over here where, you know, at some point you're gonna merge that branch over the, the main branch.
Um, that, that model doesn't really apply when you're building in production. Instead, what it feels more like, You have, you know, kind of like an an ab test slash feature flag sort of thing where you're building your branch separately in this little box over here instead of that thing over there. And the language is designed to allow that with, you know, versioned, uh, version functions and that sort of thing.
So, um, a lot of the things that, that we
Paul: assign to git you know,
Paul: long running future
Paul: branches, um, diffing, history,
Paul: reversion, they look different, but they, they, they will exist, um, or in some cases do. Um, so I think, I think reversion is a really good idea. Like, it's, it's not safe to just revert something, right? ~~You, you, uh, the, ~~when you committed it the first time, you probably had like a, a safe roll.
Um, but the world has changed since then. So like, going backwards is something that you probably haven't tested. So if you want to revert something, you really have to put it like in a feature flag. And you know, if you, if, if it's, if it's a hurry, you know, you, you can press the, the forced this shit through from, you know, button and, and, and force it through.
But really, you, you wanna slowly roll out that thing to make sure that, that the reversion doesn't break anything as well. And you don't wanna just like, force push onto, onto the main branch and, and force that deployment out and hope that those containers, uh, with that version of the application work well.
Um, yeah, it, it doesn't really make sense.
Andrew: So I, I assume there's no environments. Also, those are all just also done through this feature flagging system.
Paul: Uh, yes. And I, I think, you know, one area that, that we haven't fully figured out is, uh, is what to do with, with databases. Like, you know, people like to have a, a staging version of the data of their database or a local version of it. Um, and that's something that, that we haven't quite figured out, but I think a lot of, um, lot database.
Have versions of that, that, that work where you can take a, um, you know, you can take a snapshot work on that. Um, so the, the model isn't necessarily that difficult, it's just sort of like, how does it fit in with all the rest of it?
Justin: Planet scales approach to this is really interesting. Where they actually
Paul: Yeah.
Justin: has this really sophisticated schema versioning thing under the hood. It's pretty cool. Um,
Paul: so we, we, we have plans around schema versioning for, for the data store. Um, but you know, that's sort of separate from a, you know, we would like this data to not be corrupted by, by these changes that we're trying to make.
Justin: Yeah. Going back to to, you know, removing some of the overhead and traditional applications. So, uh, Dark has, is, is a pretty visual environment that you're developing in. You're, you're, you're generally writing code in this, Does the campus still exist in the new version of Dark where you're
Paul: The, the canvas still exists. It won't, it won't fully exist in, in the, um, In the future version or it will be a different sort of paradigm, but um, you will still be typing text.
Justin: Yeah. The, I I think the, the really great thing about that is, is you're not focused on. It's literal text file structure. You know, you're not focused on how to organize your project from like a hierarchical perspective. You're just thinking like, Oh, I need an HTTP handler, I'm just gonna create one right here.
And, you know, it's like the, the organization, how that sort of frees you up from not having to like, think about this, like relational, like organizational stuff. I, I find just like makes the mental model of just developing an app so much easier.
Paul: Yeah. I mean, often you have something like, You know, this, this directory is managed by this person or, or this team, you know, works on this repo and this other team works on, on this repo And, and this sort of idea where, where your, uh, you know, the structure of your application, uh, in a certain sense follows the, the structure of your, of your team or, or your organization. And I think it makes a lot more sense to just accept that from the very start and. To design this application around it. So, you know, as, uh, so dark has this, this concept again, this, this bit isn't built, but, um, or at least we built a version of it wasn't good. So, you know, it's been removed. Um, We, we want people to, to group around, around this, this sort of organization.
So, you know, groups, uh, represent teams, groups represent products, groups, represent microservices, sort of, um, you know, just like what I said with the, you know, everything kind of being in the same language. It's you, you want to use. , um, a single concept to represent multiple things. So, um, you no longer need separate repos, separate directory, separate files.
Um, you know, there's just one concept of separation. Um, and, and you can use it in sort of nested in multiple ways and applied permissions of various different levels and that sort
Paul: of thing to it.
Justin: That's really fascinating. It's like, you know, working with Conway's won law instead of working against it.
Paul: Yeah, I mean it's, it's kind of funny that we, that we keep. , um, you know, how, how much files exist to us and that, I mean, they exist because there's nothing else, right? You know, you, you need to store text into place, and this is the place that the text gets stored. Um, but then we build our editors around the files rather than around the languages, you know, in, you know, various languages have, you know, you can only have one class in a file, or you can have multiple classes in a file.
Then it's like, if you could have multiple classes, why does the file, why? Why do
Paul: we have a file? Well because
Paul: we have to, because you know, we write code on Munix. Um, but it, you know, it doesn't really bring anything to, to, to the, to the table.
Justin: Yeah, there are a lot of like semantics that are just like baked into this structure and it, and it adds, you know, naming a file and where the file lives and the folder structure like has, is like communicating this thing and. It's like a part of the pat pattern of shaping your code. I've been really excited by languages like Unison, Unison Web, which is like all your codes in a database, and instead of like functions don't have names, they have these globally unique identifiers that are like what their actual identity are.
And you can apply a label to it and just see the label. And if you wanna rename it, you can rename it. But like that's not a, that's the non-destructive operation to the actual code and stuff like that is just incredibly exciting. And I think dark sort of like plays into that, that model as well of like, you know, being less tightly coupled to, you know, you know this, this base structure.
Paul: yeah. I mean, you know, same sort of idea that the code is started in a database. The, um, it's,
Paul: it's an, AST b, um, functions
Paul: are versioned, you know, kind of, uh, I think that's the, uh, we agree on a lot of things.
[00:26:35] Challenges
Andrew: So dark is, is a pretty expansive project. It's it's own language. So, uh, what, what are some of the challenges you've faced Building this completely new language?
Paul: A lot of the advantage of dark comes from the fact that we own the whole thing, right? So we own the editor. We, we, we own the database. We, um, we own the infrastructure. We own, we own the language. And that means that the language needs to be designed for these things. So a very obvious one is the language has to take into account the ex, the existence of feature flags, whereas in non. You know, in a, in a normal language, you would use like the, the Launch Darkly sdk and, you know, it'd be, there'd be an if state or something like that, but our language needs semantics for what is a feature flag and how does it differ from an if statement. Then we, we need something like, um, um, partials. So a, a partial is, is what you get when you backspace over a function.
Until you have filled in a new function name. Right. So it's, it's this sort of like the intermediate state. Uh, and that needs semantics as well. And then the, the semantics of that need to intersect with the semantics of the feature flag. Um, and there, there, there's a couple of things like that, that there's four or five different things where, um, there's a little bit of Oh, exactly.
A complexity explosion. Since, since, you know, we've managed to like shave. Uh, sh shave each of the features a little bit, but, you know, sometimes they intersect and then they intersect with the editor and they intersect with like the, you know, the, the SQL compiler. Um, and so a lot of times there's like, oh, there, there's this entirely new thing that we invented and we invented it because it needs to run in, um, you know, to allow users have
Paul: this deployless environment.
Paul: But now we need to define the semantics for it. The s QL editor or like, what happens if someone tries to save this value to a database? So, you know, what does that mean? Um, so I, I think that that's kind of like our, you know, probably our most interesting challenge. Um, the other, the other major thing is, um, Dark is still a language under development.
So we, we still want to change some of the language semantics, but we don't want to break anything that our users have done. Um, so we need to, we need to migrate our users from old versions of the things to the new versions of the things. Um, and we need to figure out a way that, that the semantics either don't change or that we can validate that they don't change for any particular users, or we can give 'em some way to just like not be. Uh, that's, that's a little bit tough.
Andrew: Cool. So dark has like kind of like automatic migrations built in for old language fe features to new feature.
Paul: Well, if they didn't, we would have to keep all the old language features, uh, forever. And then we would have to, you know, every time we built another language feature, that's another thing for it to understand. Every time we build a new editor, it's like, Oh, what happens if, if, you know, the person uses version one of of records, which.
You know,
Paul: were invented
Paul: four years ago and you know, now everyone's on, on version three. And what happens if they switch to version two from version one, but they didn't switch to version. Yeah. That's just, it becomes a mess. And it's similar with, with functions like, you know, we like to delete old code, um, we want to move people from old versions to new versions.
Um, remove deprecated things from, from documentation. So we built deprecation into the language. We built it into the standard library. know, the most obvious thing there is that if you, uh, if you've never used a function, we will only show you the latest version of it. So, for
Paul: example, HTTP client is
Paul: now at version five.
We won't show you version one to four or zero to four if you, uh, if you never added it. But, you know, we would also like to remove version zero version one, version two. And so, you know, we, we, we would like to migrate to, to the new versions and we'd like to know that it's safe. So in, in, in a sense, um, part of what Dark takes on board, part of the complexity burden that gets taken on by Dark the product is, is migrating users from old versions to new versions.
And that's not always gonna be,
Justin: considering the, the language and sort of the integrated language features, how much of the developer experience of dark really is from the design of the language itself, and I mean, what other programming languages might you compare it.
Paul: Um, yeah, so, so dark was, was very much designed with sort of developer experience in mind. Um, we, we wanted to keep it really simple. Um, we wanted to, to avoid a lot of the things that, that we felt were kind of wrong directions for the industry. So, for example, inheritance. Um, I don't think inheritance was a good direction for the industry to go.
Kind of moving away from it as, as an industry. Certainly, you know, languages that, that are created today don't have the sort of, you know, in depth inheritance at c plus plus and Java had. Um, the other thing I really like is immutability. Uh, I've been coding in immutable languages for over 10 years. It's phenomenal.
Um, the, like, literally every language I've used in the last decade has been, has been immutable except when I have to write JavaScript. So, um, you know, Dark is a statically typed functional language. It is very similar to Elm. Uh, has some similarities to Rust. It's similar, a little bit to Haskell O Camel, F Sharp, um, rescript, you know, these are, you know, a lot of these languages.
Very similar, um, has some types. It's mutable. Um, and that's very much designed because you can understand programs a lot better that, uh, if they're immutable,
Andrew: So does, does Dark come with any type of type safety? Like what? With the, The whole like shift in code organization. Like where, Where do I store AULs file? Is it me just like talk serverless functions, talking back and forth to each other? Like how does that all work?
Paul: So, um, so I'll adjust your, your, your, um, your first question. So like, type safety. Um, dark is designed to be a statically typed functional language. However, we don't actually have a type checker and we put a lot of the type checking into the run time, which. Problems. So dark will be gaining a type checker in, in the future, and that will be part of the editor.
So, um, you know, in the same way that that like type check checking is done at the start of a compile, um, similar, you know, every, every time you press a character, there will be, there will be a type check done and it will be designed around this kind of, you know, language that could be partially written sort. Um, and there's, there's a lot of advantages that, you know, the, right now, if you want something to type check you, you know, you have to get the stuff in front of the type checking. You can't do sort of this partial type checking. Um, and there's a couple of languages in academia that, that have this concept called type tos.
Um, that, um, Hazel and, um, uh, the, there's another that, uh, there's another couple that, that are like it. Um, So, you know, you know, type checking will be, will
Paul: be built i n.
Andrew: So from what I understand of dark, you're just kind of like, Uh, bdeclaring like, Oh, here's like my get request and how that works. Sharing code between that, like if I wanna have like just a general utility function that I use in two routes, is that other route represented as, or is that utility represented as a route, or can I share, Just share the code in some way.
Paul: Uh, it's, it's just functions so, so dark is, is designed like mono repos are. Everyone wants monorepo. Mono repos are much better than. Than anything else, than than microservices. But the reason that people cr you know, move away from Monte repos have nothing to do with the fact that Monte repos aren't great.
They're the fact that they don't scale particularly well once, once you get to a certain size. Um, and so people, you know, carve it off into teams or they carve it off into a particular function, or they carve it off something that has a different deployment velocity or, or deployment of frequency to, to something else or. whose tests, you know, take much longer or, you know, don't change very often. And none of those actually matter in dark. You know, all of those can be subs student, so, so dark is going to be, you know, a single monorepo, um, for your entire company, your entire organization. Um, and then we can, we can carve things off into, you know, use groups for different organizations, different divisions, different teams.
You know, there's no need for, uh, for something to be like, Yeah, put in another route. It, it, it, it's not serverless like Lambda is serverless. It's really just like, you know, there are functions, there are types, they're shared across the thing, but they're only loaded for the things that, that need them to be loaded.
So, for a particular route, you know, that route knows, knows what it needs. And, and so, you know, the things are, are loaded.
[00:36:11] Testing
Justin: So going back to something you talked about earlier. So you mentioned, you know, feature flags being a built in part of the language, uh, and having this deploy list sort of environment where you're putting things into production sort of immediately and trying to do that safely, you know, with like the feature flagging systems.
But at some level you need to be able to validate your application, need to be able to ensure that, you know, the logic that you've implemented is what you expected it to be or, you know, whatever. Uh, so how does testing work or validation in general work within the dark ecosystem?
Paul: dark has this idea of trace driven development. So what that means is you, you know, let, let's suppose you're building a client, building, you know, standards, react client, let's say, right? You build a button, you press the button, that button then goes to the dark server, right? And there's no, there's no end point for it.
So it appears in, in the four oh fours, uh, and you click and button run the shortcuts. Uh, you now have, uh, an API route for that request that you just made, and you also have the data. And in fact, every time that any user makes a request against a an endpoint, we store a trace of of that data. So for testing, what we're probably gonna do a couple of things, but the first of them is we're
Paul: going to take traces and
Paul: turn them into tests, right? These were the things that you typed yourself in your editor, um, or in your, in your client. Um, so, you know, there, there's a list of them. You can go through them. Uh, and over time we will find ones that are interesting. We'll find ones that are broken, we'll, you know, apply some funky machine learning to say this one is, is is an important one to become a test and will allow you
Paul: convert them into tests Um, and because
Paul: dark is, is immutable, um, most tests are just gonna be, here's an input. We expect this output. Um, for the un immutable things like, you know, saving to a data store. Uh, our analysis framework, which is what we would use for testing, uh, already has a built in, uh, concept of, uh, what you would call mocking.
Um, so we, we would have the, the results stored as part of, of that trace, which is kind of the same as as mocks built. So then once you've got that concept right, you've got, you've got your inputs, you've got your marks, you've got your outputs, and you've got types for the things that are coming in and the things that are going out, uh, we'll, we'll add
Paul: automatic fuzzing So we'll,
Paul: um, suggest things that increase code coverage, um, suggest these paths that you haven't gone down before. So not, you know, not done yet. And all these things exist in, in the real world, but people don't really use them because the overhead of them is, is so high. Um, but I think that the overhead of them will be really low and dark and we'll be able to, um, to make them really sort of usable in people's workflows.
Andrew: I, uh, that's a super cool idea. Just having like, you basically have snapshots of all the inputs that you could possibly want, and just being like, Oh, yeah, click a few boxes. That those are the tests that I want for this. That's a, that's a super powerful concept.
Paul: Yeah.
Justin: This reminds me a little bit tangentially of, uh, there's a rails library out there called vcr, which is like, Oh, we'll record your inputs and outputs and you can use those for testing. It's like, a lot of times in testing, the hardest thing is getting like semi real data that like simulates what the, you know, production code would, would simulate.
So that's, yeah. That'd be super, super helpful.
Paul: Yeah. Yeah. Um, the, the other thing that that's sort of related to that is, um, you, you have an exception tracker in a, in a language or in a, you know, in your infrastructure. Uh, maybe it's connected to your, to your logging slash observability, you know, whatever. But, you know, for the most part, there's, there's something there that when an exception happens, it
Paul: captures the stack trace.
That rarely
Paul: feeds back into your, your test cases, right? The, some of them at some point had buttons that you could generate a unit test, but like, obviously, you know, unit tests are, are kind of custom for your framework and that sort of thing, but, but it should, you know, every, every exception you have should turn into a unit test of some sort.
Um, and again, you know, uh, exception tracking is something that will be part of dark. So obviously, you know that that will feed back into, um, into testing and, and traces and, you know, just having a small number of primitives. They will be the same primitives throughout the the application.
[00:40:46] Packages
Andrew: it, it Seems like a lot of things are built into dark right now. Uh, one thing I read about that I don't think is quite implemented yet is a package manager. Uh, do you, what, what are your guys' plans for a package manager and what type of code do you think would be shared through a package manager?
Paul: Yeah, so, so the obvious one is, uh, functions will be, you know, most, like re re really in dark. The, everything is kind of a function. It's, it's a functional language. There's, there's not really very many other constructs in it. Um, the, uh, types are, are, are the other one that, that, you know, your, um, your functions are gonna have types.
So you know, you're gonna have some custom types. Those types are gonna have to be shared as well. Um, and, and you know, all of those are gonna be versioned separately. So the, um, there's not gonna be versions of a package. There's gonna be versions of a function and versions of a type. Um, so you might even have types that, or you might even have functions that, um, use two or the, that take parameters of two different versions of the same type, perhaps that that's something, you know, that, that, um, to help people migrate from the old version of the package, the new version of the package or, or something like, So that's an obvious one.
The other, the other obvious one is kind of templates. Um, so people, uh, you know, if you're building, let's say user package, you're gonna want some templates for a HTML page, for a signup form, a password reset form, uh, maybe some, um, um, what do you call them? Uh, third party sign-in single, single sign-on, or, you know, social media
Paul: sign-ons or whatever.
You're gonna
Paul: need some forms for that. Um, so those, those are sort of be, be put in, in the same, um, in the same, uh, packaging thing. The, the ones that get a little less traditional, um, refactoring tools are gonna be part of the package manager. So refactoring tools are gonna be written in dark. You're gonna be able to use them in the. Um, and then you're gonna be able to put 'em into a package. And so you, we want people to distribute packages that have refactoring tools in them. Um, and maybe these are, these are, you know, people have, you know, in essence DSLs, uh, domain specific languages, you know, often that they, that they ship. And, and there are, there are, um, perhaps some static checks that, that you can do on your types.
Yeah, it's not possible to have to have certain things or, or whatever. Tho those are things that, that are gonna be, um, distributed as, as part of it as well.
Justin: It's really awesome. It, it seems like packages are like similarly related functionalities to, to groups almost, you know, And that given that all of this is integrated, that the concepts have a lot of overlap, you know? Yeah.
Paul: Right, Right. Yeah. So you, um, Down the road. I want people to distribute whole applications in dark. So I want, you know, I, I think that dark is a good place to build your, your entire company's infrastructure in. One of the, one of the reasons that, that I've started dark was, um, I was using this application tracking system and. I can't remember what it was called. Recruiter Box, I think. Uh, and it was, you know, it had some problems. So we were gonna switch to Lever, and then Lever had some other problems as well, and we wanted to, , you know, the, the concepts, the, the workflow of this actually isn't that complicated. We wanted to use an api.
We wanted to, to have access to all the things that were happening in the api, and we wanted to like, pipe them into some custom things. Um, some emails, um, some Slack channels and, and they, they just didn't have integrations for this. And, you know, kind of contemplated like, should we build this ourselves?
It's like, no, of course not. That's ridiculous. And. I realized that we're now at this point where writing anything yourselves, you know, writing your own product that, that you could buy instead is almost always a bad choice because it's because the overhead of writing code is so high. The overhead of like running services, maintaining things is so high.
And so I felt that if we reduced that complexity enough that people would actually be able to write
Paul: tools, themselves again. And we'd
Paul: be able to write our own applicant tracking system. And perhaps the people in the, in the talent side of the company, the recruiting side of the company, would be able to, you know, add their own custom notifications, add their own custom toolings, add their own custom workflows on top of that, to, to represent what they actually do day to day.
Um, and, you know, perhaps, perhaps it would be something. You know, the, if the team wants a, wants a weekly report about what's coming in, you don't just get what the tool has built in, um, new gets to actually build your own stuff in it. So, um, I imagine that, that at some point in the future, someone will build, you know, an applicant tracking
Paul: system in dark.
And they
Paul: will put that in the package manager and your company will be able to install that. We'll be able to contribute back to it. We'll be able to, you know, build your own custom workflows on top of it.
Andrew: And that wraps it up for Tool Tips this week. Uh, thanks for coming on the podcast, Paul. This was a super interesting conversation about, uh, a potential future for programming. And I hope,
Paul: Yeah. Thank you so much for
Andrew: yeah. I wish you guys the best and hope for your success.
Paul: Thank
Justin: Yeah, Paul, awesome as always. Uh, really, really excited to see Dark's Future. It's such a cool project and such a huge inspiration. So yeah, just keep doing good work. It's awesome.
Paul: Awesome. Thank you. Well, you know, Dark is, is open to, to contributors and, and for people to try it, you know, Dark mind.com would love people to, to give it a go.
Andrew: Well, that's it for this week's free portion of the episode. If you want to listen to the full conversation, head over to our Patreon and become a member for just $5 a month, you can help support the show and help us get to our goal of delivering content weekly. Thanks for listening!