The AI Fundamentalists
A podcast about the fundamentals of safe and resilient modeling systems behind the AI that impacts our lives and our businesses.
The AI Fundamentalists
Principles, agents, and the chain of accountability in AI systems
Dr. Michael Zargham provides a systems engineering perspective on AI agents, emphasizing accountability structures and the relationship between principals who deploy agents and the agents themselves. In this episode, he brings clarity to the often misunderstood concept of agents in AI by grounding them in established engineering principles rather than treating them as mysterious or elusive entities.
Show highlights
• Agents should be understood through the lens of the principal-agent relationship, with clear lines of accountability
• True validation of AI systems means ensuring outcomes match intentions, not just optimizing loss functions
• LLMs by themselves are "high-dimensional word calculators," not agents - agents are more complex systems with LLMs as components
• Guardrails provide deterministic constraints ("musts" or "shalls") versus constitutional AI's softer guidance ("shoulds")
• Systems engineering approaches from civil engineering and materials science offer valuable frameworks for AI development
• Authority and accountability must align - people shouldn't be held responsible for systems they don't have authority to control
• The transition from static input-output to closed-loop dynamical systems represents the shift toward truly agentic behavior
• Robust agent systems require both exploration (lab work) and exploitation (hardened deployment) phases with different standards
Explore Dr. Zargham's work
- Protocols and Institutions (Feb 27, 2025)
- Comments Submitted by BlockScience, University of Washington APL Information Risk and Synthetic Intelligence Research Initiative (IRSIRI), Cognitive Security and Education Forum (COGSEC), and the Active Inference Institute (AII) to the Networking and Information Technology Research and Development National Coordination Office's Request for Comment on The Creation of a National Digital Twins R&D Strategic Plan NITRD-2024-13379 (Aug 8, 2024)
What did you think? Let us know.
Do you have a question or a discussion topic for the AI Fundamentalists? Connect with them to comment on your favorite topics:
- LinkedIn - Episode summaries, shares of cited articles, and more.
- YouTube - Was it something that we said? Good. Share your favorite quotes.
- Visit our page - see past episodes and submit your feedback! It continues to inspire future episodes.
The AI Fundamentalists a podcast about the fundamentals of safe and resilient modeling systems behind the AI that impacts our lives and our businesses. Here are your hosts, andrew Clark and Sid Mungalik.
Speaker 2:And today we're very excited to bring on a special guest. We have with us Dr Michael Zargum. He is the founder and chief engineer at BlockScience, which is a systems engineering firm focused on digital public infrastructure. He's also a board member and research director at the US-based nonprofit MediGov, a community of practice focused on digitally mediated self-governance. He's also an advisory council member at NumFocus. Dr Zargum received his PhD from general robotics, automation and sensing and perception at the University of Pennsylvania. Glad to have you on, mike.
Speaker 3:Thank you very much. I'm happy to be here.
Speaker 2:And so, since I mean, it's really awesome to have people that are, you know, really expertise in the systems engineering domain, because we talk a lot about systems engineering and we know a lot about it, but we're not systems engineers ourselves, so we'd love to hear a little bit about your perspective on the current landscape of agents and agent modeling. So I guess, like if we're going to just start from the beginning here, how would you characterize or define what an agent is?
Speaker 3:Yeah. So for me and this may not be a systems engineer's opinion, but I think it's a useful opinion Agent implies the existence of a principle. So everyone's heard of the principal agent problem where there's like a potential mismatch between the interests of the agent and the principle that they're the agent of. I sort of associate the concept of agent with that relationship. So anytime I hear agent I ask okay, but like, who is the principal?
Speaker 3:The way that relates to my kind of systems engineering and robotics experience is that in systems engineering there's always an accountable stakeholder.
Speaker 3:There are many affected stakeholders, but the accountable stakeholder is sort of, as they say, the throat.
Speaker 3:You choke when something goes wrong and it's important that those accountable stakeholders they might be government administrations or they might be the cognizant engineers, the people who actually own the. Okay, yeah, we have enough confidence in this thing within its operating parameters to actually put it in the field to operate and, insofar as it's imperfect, it's expected to behave within the tolerances implied by those those confidences. And I know that's kind of technical and jargony, but like the gist of it is like cool, you know, or a person or an entity, you're deploying a piece of machinery to act autonomously in an environment and you're like cool, but like, what does it do? What kinds of environments is it prepared to operate in, and those kinds of higher level questions. They belong to the principle. The principle provisions or deploys the agent into its operating environment with certain constraints and objectives, even if it acts autonomously within that operating environment. So I don't know if that's helpful. It's kind of a grounding that moves away from this, like kind of deification of agents as some kind of like magical new, I don't know what.
Speaker 2:No, I think that's a really interesting framing and so I guess, like can you pick apart from me a little bit what a principle might look like, so we can kind of chew on? What is the implication of this idea that agents are built around a principle?
Speaker 3:So when I was an undergraduate I was doing my senior design project at Dartmouth and it's down the road from the Army Corps of engineers cold research research, cold regions research, engineering lab krell and um, I got to work on, uh, an autonomous, um like rover, an arctic rover, whose job was to basically ferry around a ground penetrating radar to pull together maps of ice flows so that, um like researchers could go out on the ice and, like, do whatever they do. But you needed to know that it was a safe environment for the humans. And you did that by doing this kind of mapping exercise so that you knew the current state of the ice and whether it was safe. But you achieved that by deploying this autonomous robot that did kind of mowing the lawn style grids, which then allowed you to stitch together the maps. And so I got to write the navigation system. So I got to do some instrumentation and some like kind of robustness analysis to ensure that the navigation was capable of maintaining those tight grids under circumstances of, you know, wind and bumps in the ice and crappy GPS, because it was the, you know, mid 2000s and GPS in the Arctic had miserable tolerances and expectation.
Speaker 3:But the reason I bring all that up is because there's a kind of higher level kind of principle there which is what's the let's say, the administration or body or entity that was responsible for deploying that agent. So like here I would say, well, it was deployed to McMurdo station, but it was under the auspices of the army corps of engineers. And so you're like okay, well, at the institutional level, the army corps of engineers is responsible for this, you know mission or this capability, but then of course that's an abstract entity. So then you're like mission or this capability, but then of course that's an abstract entity. So then you're like okay, but like who? Well, there would be, there was, at least when I was working on the design and testing, there was an engineer from the Army Corps who was the project lead and there were a bunch of liaisons to others and they had all the student teams. But like they weren't gonna just take a project that students did and stick it in a production environment without putting it through. I mean, we did design, design analysis, we did lab testing, we did like local kind of range testing. So out in the snow in Hanover, like it went on a field testing expedition with one of my teammates to Greenland before it was finally like, considered safe and ready to be put into production, to do its little autonomous routes, to clear, like ice area, as safe for researchers and so like.
Speaker 3:The way that I think about this is that, like you know, at the actual project level, there almost certainly was on the ground in Antarctica an operator and technicians that were maintaining it.
Speaker 3:I think one of my teammates did go to McMurdo and I anticipate that they were subject to a more senior engineer who had direct accountability to the institution that funded the expedition and that there was like a layering of accountability all the way down to the field where we're, like, we're doing individual runs, someone's on the ground and they're the operator, even though things are autonomous, because someone has to like validate that it's working as intended and if, like, for some reason, it clearly didn't do its job right, you still needed to be able to like, not give the clear, even if it seems like it should have um, like from a data perspective.
Speaker 3:And that oversight, human validation kind of quote human in the loop bit doesn't need to be limiting the automation. It just means that, above whatever automations you have, you're going to have human oversight, calibration and validation, and so that's how I'm thinking about it here, but that's coming from robotics view of what an automated agent is, not necessarily from this. That's how I'm thinking about it here, but that's coming from robotics view of what an automated agent is, not necessarily from this kind of I'm going to call it newfangled AI agents everywhere. Meme.
Speaker 2:So this is really interesting. I think that it sounds like there's this play happening between, like, true autonomy and basically a set of controls or a set of you know users or roles who then enact policies on these autonomous systems. So it's not like it's strictly the autonomous system out in the world. If we want to truly govern it and have its principle match our intended goals, we may actually be doing a little bit more in direct intervention, and and so I guess, like, if we expect agents to be autonomous and we expect co-pilots to kind of be managed systems, where, like where is this in between and how would we distinguish them then?
Speaker 3:Yeah, so I mean when I hear agent versus co-pilot, like I hear, co-pilot is a decision support system and an agent is something that you give kind of, as I said before, like a principal gives an agent resources, it gives them constraints, it gives them objectives and it deploys them into an operational environment and expects them to, you know, pursue their mission subject to those constraints right, but like you don't have a hand on command and control. The autonomy is that they're in that environment doing the thing you asked them to do. So. So there's another analogy here that might be helpful was just sticking to agents for a second before I go on to co pilots.
Speaker 3:So my wife is a master chief in the US Coast Guard and she's she's a bosun's mate. So when she was younger she was a tactical small boat driver, and so what's interesting is that like in a sense, she's like an agent of the Coast Guard equipped with a team and an expensive vessel and a mission and some parameters of that mission. So like she's literally like an agent of that institution and has a command structure. So she was given that mission and those resources, like to do a thing, and so in a way, I think it's helpful to like, set aside for a moment whether the quote unquote autonomous agent is a human or a machine or a unit that's a mix of humans and machines with various capabilities, and just say a unit that's a mix of humans and machines with various capabilities. And just say, like, identify the dotted line boundary, and ask, all right, like what are the resources available to the agent? Who provisioned them, who gave it its constraints, who gave it its objectives? Like, and that is essentially at the level of moving the action to, you know, basically provisioning and calibrating the agent and giving it its instructions and saying go here, do your thing, and the party that did that is the one that's accountable for it.
Speaker 3:Now there are limits where, like again to the Coast Guard example, as a human being you could totally do things that are out of your parameters, but you might get court-martialed. And so here we're dealing with a situation where these non-human agents were like but like what if they don't do what we told them to do? And that's where I think a lot of things get interesting, and I know we're going to talk a little bit later about deontics and guardrails and such, and so I'll pin this topic and talk about co-pilots for a moment. But does that make sense? It's like kind of agents as a concept, independent of whether they're human machine, like cyborg unit, they're still agents.
Speaker 4:Yes, thank you for that. I really like how you laid out those definitions. One other thing not to muddy the waters too much, but I'd really like your thoughts on I know you do a lot of economic modeling and things like that as well of there's the traditional agents from the economic world and like for agent-based modeling and those other types of things. How do you really reconcile or think about the difference? Like there's different definitions of agent depending on, like, what type of modeling and things.
Speaker 3:So in agent based modeling, basically the, the agent is placed into a world with other agents and the world has laws of motion.
Speaker 3:And so, if you think about the constraints not being imposed on the agent by its principle but potentially by the world itself when you built the simulation, but then you're generally giving agents either heuristics they might have norms, depending on which techniques you're using, you might have reinforcement learning agents but what's happening is that those agents exercise their autonomy in those environments but they're interacting with the state of that environment and potentially any utilities or benefits that they're deriving based on their local optimization objectives.
Speaker 3:Again, it kind of depends on which agent-based modeling framework and technique you're following. But those agents, they don't necessarily have a principle in the sense that we're talking about now, but they're actually. They're kind of like representations of some principle. We're saying like well, someone cares, and we wrote a little computer program that emulates the behavior of those actors. And I think in this ontology, I think that agents and agent-based models are actually representations of actors. And actors are just, you know, they have state, they have the ability to act and they may or may not have utilities that determine their choice of action when faced with a choice.
Speaker 2:Yeah, and I really like this framing because it kind of gets at this idea that, like agents, isn't a split between humans and non-humans. It's not like agents are replacing humans, and even in an economic simulation, these agents are basically optimizers and they're optimizing for things that are within the field of what they can perceive and act upon, can be construed as a principle, maybe an abstracted one that's one layer away from what's actually happening, that they're trying to optimize some outcome. So you know, thinking about this a little bit for a second, we're talking about considering LLMs as agents, but agents don't act on a singular principle or in a very clear optimizing way towards the environment and tasks that they exist in. So you know, in what ways do LLMs fit into this? Are they agents? Are they not agents? Do they fail as agents in some places?
Speaker 3:Yeah. So I think the next question is about, like what you are thinking of, the operational environment and their parameters. So there's a lot of confusion around LLMs versus agents. An LLM, just like raw, is in my opinion like a really high dimensional word calculator. It's an X token predictor they have in their individual. Like you go grab a model and say fine, tune it and use it standalone. It is in a sense a generative model that predicts probabilistically the most likely next token, which is really just a point in a high dimensional embedding space. That, from a dynamical systems perspective, feels very natural. But it's also not immediately obvious how and when it would be useful quote as an agent. But then if you think about it as being part of a higher level workflow, you step out one and so, like I use LLMs a bit like a graduate student.
Speaker 3:For example, my favorite LLM tool is the Control Systems Master, GPT. The reason I love this one is because it speaks control theory. So if I explain an idea in like half control theory jargon and ask for feedback, or I ask it to structure it and repeat it back to me, what it says actually makes sense to me because it's got a whole bunch of knowledge trained into it, or I don't know whether it's fine-tuned or whether it's a rag or a mix of both, because I use it as a black box. I use it because I like the way that it behaves, but I use it in a very specific way. Like a grad student, I'm like you know. I want to know how constraint satisfaction problems might be relevant to my, you know, network optimization problem. I'm working on Like, like. Tell me what you think, and it'll. It's the equivalent of sending my student off to go read some books and come back and tell me do I totally trust that they got it right? No, Am I doing it within a realm? I have a lot of expertise, so it's easy for me to validate whether they're going off the deep end Absolutely, and so my preference for using, at least as an LLM, this particular model is because we have some common jargon. It's knowledgeable about things that I care about, and because I have both the ability to give it instructions in a way that's natural for me and I feel confident that I can evaluate whether it's bullshitting me.
Speaker 3:Then I feel relatively comfortable using it as a sort of research facilitator, research assistant agent, but it's not an agent in the sense that it's particularly agentic right. It doesn't go off and solve things on its own. When we get into agents in a more autonomous sense, based on LLMs, that's usually when people are getting into higher level prompts that invite like iterative, almost more optimized play where you're like go try to do this. If it doesn't work, catch the error, try again and you can see these playing out. But those aren't just LLMs. Those are systems composed with LLMs as one of their components. They often have things like a sensor fusion via a model context protocol with availability of a variety of different resources. They might have guard rails.
Speaker 2:You end up assembling a more complex system where the LLM word calculator bit is just one piece complex system where the LLM word calculator bit is just like one piece, and I think that absolutely makes sense in this idea that, like, llms aren't basically the decision module itself, they're a piece that's used to allow humans to very naturally interact with these types of systems. Right, you look at stuff like tool former, which is still using classic API calls, but you don't have to write the API call anymore. You're just a human, you speak in human language to it and I'll spit something out and you can validate, like JSON, because it's extremely highly structured. And you're kind of alluding at, really, what I want to talk about next, which is how can we add these types of mechanisms into our LLMs to turn them into agents where the LM can now be compartmentalized, but now we can validate what's coming in and what's coming out, similar to how we would expect from a more principled agent system. Like what does that you know, duct tape and WD-40 solution look like?
Speaker 3:Right, so yeah. So two things. First, before the duct tape and WD-40, which I have a lot to say about. I think of guardrails as being like constraint enforcers, like I like deterministic guardrails as tools that say, like you know, they're like behavioral constraints, like, given this input, is this output acceptable? Or, given this input and given this history, is this output acceptable? This input and given this history, is this output acceptable Just like as a tool or a component?
Speaker 3:To me that feels like kind of natural, kind of constraint-based control where you want to get properties of a non-deterministic system but you can deterministically enforce what is effectively like protocols or sets or, like you know, I call them admissible actions. So you don't know what they're going to do, but you know that you can enforce that it's part of an admissible action set, where the admissible actions that itself is state dependent. But that means you have to have a state model of your system from which you can derive what is admissible given that state, and that starts to look like policy engines, which is something that is a more traditional kind of expert system AI use case, and so you start marrying things like policy engines with things like non-deterministic text in text out, next token predictor things, along with things like knowledge basis, which is where the retrieval, augmented generation, rag stuff comes in, and obviously recently we've seen much more sophisticated kind of resource organization through the model context protocol. As you're taking those pieces and assembling them into something that can kind of act on its own. I see the fundamental difference being in moving from like a static input output to a closed loop dynamical system model. Like in a closed loop dynamical system, you initialize it and it runs until it hits an end state. So that's like a discrete time dynamical systems model.
Speaker 3:So if you're trying to make something agentic, then you're kicking off a job where your initial prompt kicks, starts something that does not just go input output, it goes. Okay, let me try something. Okay, depending on how it worked out, do I try something else, do I dive into the details? And so you're running through what is effectively, uh, almost like a um emergent causal graph of activity, where it's both figuring out what to do, trying it whether it works and iterating through that process until it achieves a satisfactory end state, which is honestly a lot like what engineering is in the first place right. It's a set of structured patterns for going from a less defined problem, breaking it down into smaller, better defined problems, trying to solve those better defined problems, getting solutions, integrating them back up into a whole solution, validating that it solves the requirements in the original prompt and if it does, you say here's my solution.
Speaker 3:And so, regardless of what it is that you're doing, the like agentic workflows have that kind of heterogeneous dynamism which again starts to feel like you know a team member, maybe a junior team member, because you're still responsible for their behavior, because you're the one provisioning them and deploying them. But, like you know, you send your teammates, you give them kind of high level, fuzzy instructions and say like, hey, go figure this out, and they bring you back a solution that they think solves what you said. And then sometimes you go not quite that, try something else. Or sometimes you go great, thanks, that's. But again, I think you get into fuzzy territory.
Speaker 3:If you don't have clarity about who said hey, go into this environment and do this, I don't think that you can ever really remove the accountability from the actor responsible for provisioning the agent into an operating environment. Because again, this is a recurring theme with the narratives around AI is that we're treating them not as autonomous systems, but actually we're talking about them as autopoietic systems. It's like self-creating systems, systems that emerge from and maintain themselves. And that's actually not what they are, because we still need to give them computational resources, Depending on what they're doing. We would have to give them financial resources, we might give them access credentials to read or write, read from sensors and write to actuators, like up to a point.
Speaker 3:Whoever's doing that is still provisioning and calibrating and deploying the agent. And so I think actually military-based models of autonomous systems and accountability make more sense, because, okay, we don't really necessarily know in advance what these systems are going to be used for, but someone somewhere was the point in the chain of command that said all right, this is our mission, we're going to pursue it, we're going to deploy this unit or this agent or this like actual autonomous agent into this environment with these parameters and these instructions, and I, at least in my opinion, those people um are still responsible and the determination on whether that behavior was appropriate, at least at the um like legal level, is gonna map back to that chain of authority, and if something extremely problematic took place, then you're going to look for where in that chain of command the behavior deviated from the thing that's acceptable morally.
Speaker 4:Thank you so much for that explanation. That's the best explanation of what a proper agentic AI I think I've ever heard is. Thank you for that. Like this is where, like what you described and what normally you're seeing Harvard Business Review, ibm and everybody talking about how agentic works it's an LLM in the middle making all the decisions. We've had issue previously with like agentic ai on our previous podcast and things. Versus like the true definition you define, with even like the, the like agent of somebody and all like the accountability stuff we started with. And then like there's so many ways of using the llm for where it makes sense. It's like we our previous uh conversation with christoph molnar and some others. It's like there's t-shaped model of what are the different paradigms, where does it fit, using the right tool for the job, and then there there's that rigorous and this is where you always speak in mathematical language and using the precision so much matters, though this is where my beef sometimes in the computer science world is. It's just kind of like just add more GPUs and just put an LLM in front of it and predict the next word. It's somehow going to magically get there.
Speaker 4:Versus, as you're outlining, autonomous systems and things. None of this is is news. New quote unquote it's been around a long time. It takes a lot of work and technical nuance. It's possible to get where, like the idea of agentic ai is great. Mainstreaming that outside of like military systems is fantastic. But there's that level of expertise that a lot of people don't even realize exists or know how to execute on that precision of like the dynamical system but all those sorts of things.
Speaker 3:In the business world, I think we get caught going meta. I'm like I run a business, like I do a lot of digital systems, architecture and analysis, so like I can kind of relate to this. But like, when I say going meta, I mean like losing sight of the interfaces between the representation and the represented. So with robotics, like you know hands-on physical systems, you have sensors and actuators that read from and write to the world. But the question of what is the world when you're talking about, you know economics or compliance or you know some of the stuff that we're using LLM-based tools to at least analyze, I don't really think make decisions. I'm pretty skeptical of LLMs making decisions at all. I think at best we can engineer autonomous systems where LLMs are components that help handle the mismatches between things. I think Sid mentioned earlier about like formatting, taking things in natural language and formatting them to match API specs. That's what I think of when I think of duct tape and WD-40. It's like we had all the information. We just kind of had to like, make it fit, it almost fit, like we had everything, it just didn't quite fit. And so we're finding tools that we can use to like, make it fit and I think as we move to these applications though, as I was starting to say about going meta, we don't do as good of a job with validation I am. I like to tell this story about another undergraduate robotics like class, where you you're first learning and I've seen this happen, I've TA'd these classes like you're first learning to program a robot and you're like sitting at your laptop and you're like looking at some you know terminal feed that is telling you that it's crushing it. Like your utility function is going up, you know your loss function is going down, however you coded it, and like it's working. But you like look at the robot and it's like banging into the wall and you're like, oh right, those aren't the same thing. So like you have a bug or a mistake or a mismatch somewhere and you could very easily it's you know see something in the digital representation and the representation that like it. Of course, it thinks it's doing a great job. Literally, its only job is to do a great job, and so it's going to do a great job.
Speaker 3:Now, whether its notion of doing a great job matches up to your notion of it doing a great job in real life, that's a different thing, and so we have difficulty making those separations in business applications. And you see this, with people, for example, being unwilling or having difficulty with techniques that will like reduce the quality of your like validation performance, but like reduce bias, for example, because you're introducing constraints and based on first principles of optimization. If you constrain the system, it's going to do worse than its objective period. That's a theorem in optimization. So, like, the act of constraining your AI to avoid a particular like problematization that you've identified will make it do worse on its scores and validation.
Speaker 3:And people have such a hard time doing that because they're like no, I need my loss function number to go down. It went up. I couldn't possibly be okay to do this. And you're like dude, like that's not actually the point. I get it. It's like the proxy, it's the representation, but like that's not the point.
Speaker 3:And so I see this as being kind of the same problem as the robot bouncing into the wall. We're like not especially in these digital business systems. We're like not willing to accept that the, the robot bumping into the wall, is the thing that matters and that if my object, even if I make my loss functions worse, if the robot's not bumping into the wall. It's actually doing the thing that's better, and I don't. I realized that's kind of getting a little away from LLMs, but like it's just a concept of how we think about automation and business processes and like even like basic ML classifiers, regressors, whatnot we were like we've gotten so good at basically meta-parameter optimization and hacking our systems to make our loss functions go down that we sometimes forget that that wasn't actually the point.
Speaker 2:Absolutely, and this kind of gets at the problem of just goal alignment right that, like, we build these systems to operate in a very nice toy environment with very nice toy objectives and when we deploy them in the real world we find that they often don't always match with what our intention was, or that this problem is unspecifiable, or what you have is not very clearly written. It can be any number of things, and you know so. I think this I mean I want to cue back in on what you were saying on deontics If you want to align these systems and you want them to act in a very specific way, how might you end up in a situation where you end up putting in rules which are not respected, and how much you put in rules that are actually respected and actually are validated by the model?
Speaker 3:Yeah, so I do love my deontics. So let's talk for a second about constitutional AI, and this is where this idea of should versus can't came up. So when you make a long prompt that sort of basically constitutes and kind of tells an AI agent you know who they are and what their goals are and what they're supposed to do and what they're not supposed to do, that because it's in a prompt like structure, it's at best a should. Right, it's not, it's not a must. I mean, even if you write the word must, in practice it's a should because it's in a prompt word must, in practice it's a should because it's in a prompt. And so you know we deal with jailbreaking and trying to get things, subvert them, get them to behave differently than they've been intended. If you want a must or shall type deontic, then you need to use constraints, and in practice in LLM-based systems, this shows up through guardrails, because your aggregate system will simply not respond if it violates a constraint that was implemented via a guardrail. It's going to say, nope, I can't respond to that.
Speaker 3:Now, obviously, people do more jailbreaking to try to get workarounds. What they're doing, then, is trying to basically trick the the more llm kind of text generator thing into producing a result that can slip through the rules that are written in the guardrail, which is, you know, one of these kind of red teaming activities which is actually quite fun. If you've ever played, there was this old uh war, uh llm war gaming thing called gandalf. That was super fun, um, it was maybe a year or two ago now. But there's this kind of exercise of saying like, well, even if you have a guardrail based system and those rules are deterministic, they're actually still computing over text based domains and co domains, and so it's possible to slip something through that is in its meaning to a human, something that a human interpreting that rule might have said, hey, that shouldn't have made it through. So we are still stuck with some incompletenesses, some imperfections here, certain amount of quote-unquote epistemic gap, um gap between the representation and the represented. That's unavoidable here, and so it's not like we're actually suggesting that you should try to completely remove it. What we're saying is that you need to try to characterize it and respect it, and I have a friend, um, who likes to talk about this in civil engineering analogies, and you know we have this saying um, I actually don't remember which one of us originated it.
Speaker 3:But like, if you compare it to like a bridge, when you drive up to the bridge and you see a sign that says like how many tons the bridge can carry, it doesn't say bridge cannot carry infinite weight, like gpt says gpt can be. Like check your answers as if you could like open it up and like I mean you're using it, you expect it to be able to perform the bridge. On the other hand, it has a lot of complexity to it. I mean I live in the Northeast United States. A common thing is like flex plates because we have seasonal variation. So you've got materials that expand and contract and you have like all sorts of interesting structural and materials engineering that goes in to making a bridge both safe and reliable and predictably so. So like you know under which parameters, like you can expect it to be safe and hopefully we won't leave those parameters. But like there are circumstances that can destroy a bridge but we have a reasonably good sense of what operating conditions it was designed for and that those designs are relevant for the usage and the environment, the climate, etc.
Speaker 3:I actually sometimes like to point out that by that description a bridge is an autonomous system.
Speaker 3:You design it, you develop it, you provision it, you deploy it, you do the ribbon cutting and people use it, and it adapts to changes in temperature, it vibrates when cars drive over it.
Speaker 3:All of these things happen according to the laws of motion, of physics and the behavior of humans around it and it just works and, again importantly here, up to its safety specifications.
Speaker 3:It might have a weight rating and you're going to have real issues if someone tries to take something over it that doesn't meet its weight ratio or weight rating. This is why on small back roads, they'll say things like no trucks. Part of that is because there might be, you know, culverts within that back road that are not actually rated for big heavy trucks. And so, anyway, I think there's a lot of it, of digital, there's a lot of civil engineering analogies here that I tend to and I think I said this and this came up in the introduction like, I think, of a lot of my work in the context of digital civil engineering and as designing and developing digital systems that support and enable a wide range of human behavior, but also with, like, realistic expectations around. You know, accountability, oversight, calibrating and governing mechanisms, and that could be market mechanisms, but that can also be decision and decision support mechanisms.
Speaker 2:And maybe this is like too big of a question. So I mean, feel free to, like you know, take the easy way out of this question. But we have these systems where we understand the materials, like we can understand the physical properties of a bridge, we can understand the physical properties of most linear based machine learning models. How can we try and understand the materials and the interfaces and the tolerances of bigger systems like this, where the evaluation methods is just kind of like an Ouroboros?
Speaker 3:I think that's a really good question. I'm going to make another example to physical materials, because for me this is the only way to make this make sense used to systems that were like deterministic and like fully specified that we could understand them completely from mathematics and computer science that we, I sometimes think we forget that most of human history is working with stuff we don't actually understand how to evaluate. And so my my first engineering job when I was in high school was at an aerospace materials startup, and so I got to do cool stuff with silicon carbide and carbon fiber ceramics. And it was, you know, these complicated like you mixed up slurries of polymer and silicon carbide crystal and you like soaked carbon fiber fabric in it, and then you did these like you know, ridiculously complicated heat cycles over the course of two weeks and at the end of the day you'd get like this, like composite material, and then you're like okay, cool, but like is it good? Like what is it? Did it work? What do we, you know, can we use it? Well, then you would actually cut it into little strips, take it into the machine shop and then spend the next week like doing tons and tons of painfully repetitious, like failure mode tests until you like sort of understood it. And even doing that once didn't necessarily guarantee anything. You had to do it repeatedly. And then even then, if you managed to get to a point where you had a strong hypothesis that you had a material with interesting properties that could be used in a higher order system in their case mostly aerospace you would then have to have like process engineers, try to further stabilize that to the point that you could reliably produce materials that reliably pass these checks.
Speaker 3:But all of this was entangled with real world measurement and like. Real world measurement models, data, scientific theories, gap between like how do we theorize it? How do we measure it? How do we eventually bind the measurements and the theories and the practice in such a way that one would be comfortable putting that component into, let's say, a spaceship and you're like okay, cool, but also even then you're not like relying on it being perfect. There are safety factors, there are redundancies, and so when we do engineering, it's not that we eliminate those uncertainties, is that we design with them in mind.
Speaker 3:Another great example of that is telecommunication networks. Like most of Shannon, information theory emerged from the problem that communication networks actually weren't that great, but we still wanted reliable communication. So I think the answer here is less about like, like making them perfect or perfectly measurable, and remembering that we do actually know how to make stuff reliable with unreliable stuff. But that means moving out of the computer science and mathematics can perfectly, you know, formally verify everything back into the realm of systems engineering and validation and redundancy and like specifications and like. It's just a.
Speaker 3:It's a different practice, it's up a level and I think that difference between verification and validation is something that a lot of like.
Speaker 3:The formal difference between validation and verification is something that more ML engineers should re-familiarize themselves with, because we've kind of tended to I mean, we've collapsed the concept of validation in those communities as against the concept of validation more generally in a systems engineering lifecycle. I'll clarify that. So when before we were talking about validation and like cross-validation and making your loss functions go down, that's like what's commonly understood as validation in the ML data science world but in the control systems and systems engineering world, if you're doing that and your thing is not performing in its operating environment, then you are not doing validation correctly. You are validating the correspondence of the intents and the outcomes, not making a number go down Now. You're probably using some measurements to do that. So I'm not saying it's just watch the thing and say it's working. I just mean that your attention is on whether the outcome matches the intents, rather than your attention is just on whether you've made a particular measurement, move in a particular direction.
Speaker 2:That's totally right. It asks and it puts the onus of verification back on the modeler to make sure that they're not just optimizing for loss and not just optimizing for MSC. They're actually optimizing for outcomes, and that requires rigorous testing, repetitive testing, very cleanly specified use cases either business use cases or civil use cases and thinking of validation more broadly as a set of tests that allow us to trust systems, rather than verification as a method of making sure that we optimize for some small piece of what we want out of the system.
Speaker 4:Yeah, we're trying to use the terminology a little bit like fit for purpose for the systems. So it's like is it fit for the purpose? You're using it for A little bit less than like you're just optimizing that function. So we're trying to figure out that happy medium as well. But no love, all of the detail you're providing here and it's always such a pleasure to always learn so much whenever I talk to you at the end of the day, when the model fails because all models will fail, hopefully only in testing who takes responsibility for that right?
Speaker 2:we may play the blame game in game if there's like a large fallout in the real world. But in the testing world, where it's still in the lab, who is responsible for making sure that the model is better? Is it the role of the model developers, the role of the QA? Is it the role of the business designers who, you know, takes responsibilities for the concerns? And maybe that's a different answer for different scenarios and it depends. Answer is reasonable here.
Speaker 3:Yeah, so it definitely depends. But again, I think we could be a little bit concrete about how we figure this out. So let's imagine that you're let's say you're a product manager for a data science team or a director of data science within an organization where you are accountable to say the CTO and the CTO says to you hey, I need a system that can, you know, basically perform the following estimations. We're going to interrupt with some other stuff other people are doing but, like now, you're the party who is accountable for bringing that, saying it's ready to be deployed. You know, you may not be the DevOps person, but you can look at the data outputs about its performance and make a call, for example, to say, hey, we need to rerun it, something's weird. Or you might be the person who has to say look, even though I was responsible for putting in this production. Who has to say, look, even though I was responsible for putting in this production, I'm going to recommend that we revert to a previous version because something's not right about the way it's behaving. It's like the person who has the authority has the accountability. That's the way that I like to think about it and that that recurses nicely. So, like within a team. You can't be that person if you don't also trust your teammates, but then to an extent they need to be accountable to you for doing their jobs, and so you end up with this kind of arrangement of accountabilities and authorities that give rise to.
Speaker 3:On one hand, the buck usually needs to stop with someone who has a I'll call it like a kind of boundary accountability, which I was using this kind of director of data science, you know, product manager type person to embody that role, because, like, up to a point, the teammates are their agents and they are the principal and therefore they are having those teammates like do their jobs, and if something's not working or if it's not ready to go or it's not safe, then they have to kind of stick their neck out a little bit and be like hey, I know we're going to be behind by two weeks, but it's not ready to go. Or they're the ones that have to make the call that says, hey, like it's doing pretty well, we think it's good to go, we're going to put it into production, but actually we're going to run it in parallel with another system and that way we can fall back quickly and easily if it doesn't hit the mark. And these kinds of decisions, they tend to live at the point of operationalization, so at deployment, at calibration, at oversight, and so for the most part, I think that the work in the lab, it's lab work. You're supposed to screw up, that's how you learn.
Speaker 3:But you need to know the difference between explore and exploit. You need to know the difference between coming up with a new idea and hardening it so that it's safe to be deployed in its operating environment. And also it depends on what the operating environment is. Some operating environments are more safety critical or mission critical than others. Some operating environments might just be not super sensitive and they're not a bad place to try new things.
Speaker 3:And so knowing whether it's a fit for purpose solution again is going to fall on some decision maker, some actor with both accountability and authority. But that accountability has to be to someone and that authority has to be the ability to make the decisions, because I don't know about you, but I don't want to be accountable for something I don't have authority over. Like, if you don't give me the resources, if you don't give me the time and you don't give me the power to make the decisions, I don't want to be held accountable for the outcomes. But if you do give me those things, then I think it's reasonable for you to expect me to be accountable for what I output.
Speaker 2:And I think that's a really good heuristic to use right, because it's tractable, it's understandable and it empowers the people building bottom-up solutions. Right. Rather than this being like a business demand and everyone below is just being told what to do, you empower your engineers to actually think critically hard on their systems and be able to take ownership of what they've created. And I mean, I'm cheating a little bit here, but I love this answer because it really just echoes that we hear time and time and again in this podcast, which that if you ask an expert how do you do this the right way, they're going to say you got to go back to basics. We've already solved this problem.
Speaker 2:These aren't new problems. We have the frameworks, we have these ideas. Let's go back into our toolkit and really try and solve these problems with the hard methods, with the robust methods, with the methods that have worked for a long time. And you know we don't tell our guests to say these things right, but time and time again we see this exact same message pop up over and over again. So I mean thank you a lot for all these responses and your time today.
Speaker 3:Well, it's been great hanging out with you guys. Thank you so much for having me. Thank you so much.
Speaker 1:Thank you for joining us. If you have any questions about this topic or any of the other topics that we cover on the podcast, please leave us a comment or send us a note at AIfundamentalists at monetarai. Until next time.
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
The Shifting Privacy Left Podcast
Debra J. Farber (Shifting Privacy Left)
The Audit Podcast
Trent Russell