Production Considerations for Generative AI Apps
So you’ve convinced your employer to allow you to play around with generative AI, and you’ve managed to put something neat together. What’s the next steps? The gap from “neat experiment on my laptop” to “production-ready application” is often wide, and with the new landscape of generative AI models it can be even more daunting than before. Here’s a few tips to keep in mind.
Development Considerations
Have a killswitch
If/when your generative AI app goes viral for the wrong reasons, having a quick way to pull the plug without having to go through a deployment or something similar is likely a good idea. You’ll want to make sure you gracefully support this “temporarily unavailable” state as well on the client to avoid any big ugly errors from showing up.
Log Everything
You’ll want to keep track of a few different things:
- How often a model is getting called - each call often costs money, so if there’s areas where you can optimize / cache responses it’s a good idea to keep an eye on it. Not to mention you’ll likely want alerts against a high volume of calls (rather than trying to explain to accounting why the bill is so high at the end of the month)
- The prompts and replies - this is crucial data to keep track of the performance of your generative AI application. Maybe users are asking questions you didn’t anticipate, or your application doesn’t have access to the data it needs to answer the questions fully, or your bot is hallucinating and giving incorrect responses - none of these problems will be known to you unless you log the interactions users have with the model.
- User feedback - you should ensure there’s a way for users to indicate if the response that was generated for them was good or bad. Having this subjective measure gives a quick way to identify and potentially flag problems.
Business Considerations
Beta test internally, and then with a small trusted external group, first
One of the biggest things with generative AI is the new edge cases that you may not even be aware of. Language models have a whole bunch of responses baked into them through their various training sources, and it’s likely that your users will be able to get it to say and do things that you hadn’t thought of. It’s crucial that you trial out your new generative AI app with a trusted group of users first - likely internal users first, and then a trusted group of external users. Once who aren’t afraid to try things, but more importantly will come talk to you first before posting on social media the hilarious / offensive / morally questionable thing they got your new chat bot to say. This is one of those cases where you’re never going to think of every possible question on your own, so farming out the work to trusted individuals is your best bet.
Have a terms of service / user agreement
As annoying as they are, having a consistently available terms of service / usage agreement is the best way to cover yourself from any legal ramifications your chat bot may lead your company into. It doesn’t take a lot of effort to find examples where generative AI has caused companies to offer vehicles for $1, misinform airline passengers, or tell business owners to break the law. Having a legal document between your chat bot and the end user’s first click is a good strategy to offset at least some of the ramifications of this.
Review and implement your brand’s identity
Something that often goes unnoticed (by developers, anyways) is the tone and style you can apply to generative outputs. If your brand has a certain persona that it relies on, you can try to craft your prompts so that the generated content reflects that persona in its replies. At the very least, even if you aren’t going for a specific style, you should ensure that your generated content doesn’t portray an unwanted identity - you likely don’t want to respond to customers in a rude, unhelpful, or brutish way. You can include this information in your system prompt if needed.
Constantly review the generated outputs
Since the output of generative AI can be so variable, and is often based on end user input, keeping an eye on what your new generative AI application is showing end users is a necessary practice. This shouldn’t be a “oh, save things in the DB and we’ll get around to it” type of commitment, but rather a dedicated job function of an existing employee. Not only will you be able to quickly identify and respond to red flags, but you’ll also be able to identify areas where your supporting information (i.e. the sources of data for retrieval-augmented generation) may be falling short or absent, or areas where the bot’s reply could be tweaked to be more helpful.
UI Considerations
Make those terms of service agreements front and center of each new interaction
As highlighted above, those legal documents can potentially save your company’s pocketbook, so ensuring there’s no way an end user could say “oh, I didn’t see that” is important.
Clearly label AI interactions
It should be apparent to end users that they’re working with a generative AI assistant. As much as some users may hate this, it’s important to be clear that the interactions aren’t with a human on the other side. This tempers expectations and gives a bit more leeway to the potentially incorrect responses end users may encounter.
Provide a way back to a human
For the users who don’t want to interact with generative AI, or those who find themselves unable to be helped by it, there should be a clear path out of the interaction and into a human one. Even if it’s just a simple “contact us” form, giving users a clear way out can help alleviate some of the potential frustration (real or imagined) that they may have with the initial interaction. Bonus points if you can link a submitted “contact us” form to the AI interaction, allowing someone to review and see where things may have gone wrong.