Generally, when we write APIs, we expect them to return the response in a structured format (such as JSON or XML) so we can populate the response in a model object and utilize the information in other parts of the application. The same is true for APIs interacting with LLM models. This is especially useful when extracting information from the report documents, emails, or other long-form texts in an unstructured way.
This LangChain4J tutorial will help to start requesting the structured output in a specific format from LLM and populating the model objects with the response.
We are assuming that you have completed the initial setup for LangChain4J in your application.
1. The Response Model
Let’s say we have a Person record in our application and we want to create a new Person for each response from the LLM model. The record Person has four fields (name, age, city and country) that we want to extract the provided text.
record Person(String name, int age, String country, String city) {}2. Extractor Interface and AiServices
Similar to HttpExchange or Retrofit, in LangChain4j also, we can create declarative interfaces with the desired API and let LangChain4j provide an object (proxy) that implements this interface. The main purpose of such proxy objects is to hide the complexities of interacting with LLMs and perform some common and repeated tasks such as:
- Formatting inputs for the LLM
- Parsing outputs from the LLM
In the following example, we are creating the interface PersonExtractor that has a method to request for the structured JSON output provided an unstructured text in the request. The @UserMessage annotation provides instructions or hints for how the extraction should be performed i.e. return the details in the form of a JSON document.
interface PersonExtractor {
@UserMessage("""
Extract the name, age. city and country of the person described below.
Return only JSON, without any markdown markup surrounding it.
Here is the document describing the person:
---
{{it}}
""")
Person extract(String text);
}We can not create a proxy of this interface using the AiServices.create() method which uses Java’s dynamic proxy mechanism.
//Create model instance
ChatLanguageModel model = OpenAiChatModel.builder()
.apiKey(OPENAI_API_KEY)
.modelName(OpenAiChatModelName.GPT_3_5_TURBO)
.logRequests(true)
.logResponses(true)
.build();
PersonExtractor personExtractor = AiServices.create(PersonExtractor.class, model);Now we can use the PersonExtractor object to call its extract() method which method takes a text description of a person and returns a Person object.
String inputText = """
Charles Brown, aged 56, resides in the United Kingdom. Originally
from a small town in Devon Charles developed a passion for history
and archaeology from an early age, which led him to pursue a career
as an archaeologist specializing in medieval European history.
He completed his education at the University of Oxford, where
he earned a degree in Archaeology and History.
""";
Person person = personExtractor.extract(inputText);
//Verify the person object and populated information.
System.out.println(person);The program output shows that the input prompt automatically includes the expected JSON derived from the Person record.
2024-06-20T01:03:00.455+05:30 DEBUG 19236 --- [ restartedMain] d.a.openai4j.RequestLoggingInterceptor : Request:
- method: POST
- url: https://api.openai.com/v1/chat/completions
- headers: [...]
- body: {
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "user",
"content": "Extract the name, age. city and country of the person described below.
\nReturn only JSON, without any markdown markup surrounding it.
\nHere is the document describing the person:
\n---
\nCharles Brown, aged 56, resides in the United Kingdom. Originally
\nfrom a small town in Devon Charles developed a passion for history
\nand archaeology from an early age, which led him to pursue a career
\nas an archaeologist specializing in medieval European history.
\nHe completed his education at the University of Oxford, where
\nhe earned a degree in Archaeology and History.
\n\n\nYou must answer strictly in the following JSON format:
{\n\"name\": (type: string),\n\"age\": (type: integer),
\n\"country\": (type: string),\n\"city\": (type: string)\n}"
}
],
"temperature": 0.7
}
----
2024-06-20T01:08:09.751+05:30 DEBUG 8888 --- [ restartedMain] d.a.openai4j.ResponseLoggingInterceptor : Response:
- status code: 200
- headers: [...]
- body: {
"id": "chatcmpl-...",
"object": "chat.completion",
"created": 1718825889,
"model": "gpt-3.5-turbo-0125",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "{\n\"name\": \"Charles Brown\",\n\"age\": 56,\n\"country\": \"United Kingdom\",\n\"city\": \"Devon\"\n}"
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 161,
"completion_tokens": 28,
"total_tokens": 189
},
"system_fingerprint": null
}
----
Person[name=Charles Brown, age=56, country=United Kingdom, city=Devon]3. Handling Response Parsing Errors
Not all the LLM models will honor your request for a strict JSON response. For example, Gemini likes to wrap the JSON content within a Markdown code block. Such a response will fail when the JSON to POJO conversion happens behind the scenes.
There are a couple of ways to handle such situations.
Extract the response in String (in place of Person record) and strip such markup or other unwanted formatting.
// Assume response is the raw response string from the model
String cleanedResponse = response.replaceAll("^```json\n|```$", ""); // Remove Markdown code block
Person person = objectMapper.readValue(cleanedResponse, Person.class);We can try setting the 'temperature' and 'topK' parameters to low values which often causes language models like Gemini to increase the determinism of the output i.e. the model follows instructions more closely.
Note that lowering temperature and topK limits the model’s ability to generate diverse or creative responses. This trade-off is necessary when strict adherence to instructions is more important than creativity.
4. Complete Example
The complete working example for getting the model response in strictly JSON format and populating the model POJO is given below. The example is intended for getting started purpose and you are expected to write the modular code with proper packaging and logging.
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.openai.OpenAiChatModel;
import dev.langchain4j.model.openai.OpenAiChatModelName;
import dev.langchain4j.service.AiServices;
import dev.langchain4j.service.UserMessage;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
@SpringBootApplication
public class StructuredResponseApplication {
public static void main(String[] args) {
SpringApplication.run(StructuredResponseApplication.class);
}
@Value("${OPENAI_API_KEY}")
private String OPENAI_API_KEY;
record Person(String name, int age, String country, String city) {}
interface PersonExtractor {
@UserMessage("""
Extract the name, age. city and country of the person described below.
Return only JSON, without any markdown markup surrounding it.
Here is the document describing the person:
---
{{it}}
""")
Person extract(String text);
}
@Bean("structuredResponseApplicationRunner")
ApplicationRunner applicationRunner() {
return args -> {
ChatLanguageModel model = OpenAiChatModel.builder()
.apiKey(OPENAI_API_KEY)
.modelName(OpenAiChatModelName.GPT_3_5_TURBO)
.logRequests(true)
.logResponses(true)
.build();
PersonExtractor personExtractor = AiServices.create(PersonExtractor.class, model);
String inputText = """
Charles Brown, aged 56, resides in the United Kingdom. Originally
from a small town in Devon Charles developed a passion for history
and archaeology from an early age, which led him to pursue a career
as an archaeologist specializing in medieval European history.
He completed his education at the University of Oxford, where
he earned a degree in Archaeology and History.
""";
Person person = personExtractor.extract(inputText);
System.out.println(person);
};
}
}5. Summary
In LangChain4j, we can use the AiServices approach to operate with strongly typed objects. This helps in not interacting directly with the LLM, instead, we are working with concrete classes such as PersonExtractor and Person. This approach takes away the complexity of interactions with LLMs from day-to-day development tasks and focuses more on business logic.
Happy Learning !!
Comments