AsyncLlamaInfrence Class Reference

Provides an interface for asynchronous inference operations on a model, such as GPT. More...

#include <lib.h>

Collaboration diagram for AsyncLlamaInfrence:

Public Member Functions
	AsyncLlamaInfrence ()
	Default constructor for the AsyncLlamaInfrence class.
void	ClearAllTasks ()
int	GenerateAsync (const std::string &prompt)
	Asynchronously generates output based on the provided prompt.
std::string	GetGenerationResults (int task_id)
	Retrieves the generated text associated with a task ID.
bool	GetSetupResults (int task_id)
	Retrieves the result of the setup operation associated with a task ID.
bool	IsGenerationReady (int task_id)
int	SetupAsync (const LoggingContext *logging, const gpt_params &cfg_params)
	Asynchronously sets up the model with specified configuration LLama.cpp parameters.
	~AsyncLlamaInfrence ()
	Destructor for the AsyncLlamaInfrence class.

Data Fields
LlamaAsyncImpl *	pImpl
	Pointer to the implementation class, used by the PImpl idiom.

Detailed Description

Provides an interface for asynchronous inference operations on a model, such as GPT.

The AsyncLlamaInfrence class provides methods for asynchronously setting up the model with configuration parameters, generating predictions, and retrieving results. This class uses the PImpl (Pointer to Implementation) idiom to hide implementation details and manage asynchronous tasks safely.

Constructor & Destructor Documentation

AsyncLlamaInfrence()

AsyncLlamaInfrence::AsyncLlamaInfrence

(

)

Default constructor for the AsyncLlamaInfrence class.

Initializes an instance of the AsyncLlamaInfrence class, setting up internal structures and preparing the environment for asynchronous operations. The implementation details are hidden within the PImpl structure, ensuring that the public interface remains clean and simple.

~AsyncLlamaInfrence()

AsyncLlamaInfrence::~AsyncLlamaInfrence

(

)

Destructor for the AsyncLlamaInfrence class.

Responsible for cleaning up resources used by the instance, including terminating any ongoing asynchronous tasks and releasing memory allocated for the implementation details. This ensures that no resources are leaked and that the application remains stable and efficient.

Member Function Documentation

ClearAllTasks()

void AsyncLlamaInfrence::ClearAllTasks

(

)

GenerateAsync()

int AsyncLlamaInfrence::GenerateAsync

(

const std::string &

prompt

)

Asynchronously generates output based on the provided prompt.

Parameters

prompt

The input prompt to the model.

Returns

An integer representing the task ID for the generation operation.

GetGenerationResults()

std::string AsyncLlamaInfrence::GetGenerationResults

(

int

task_id

)

Retrieves the generated text associated with a task ID.

Parameters

task_id

The task ID of the generation operation.

Returns

A string containing the generated text.

GetSetupResults()

bool AsyncLlamaInfrence::GetSetupResults

(

int

task_id

)

Retrieves the result of the setup operation associated with a task ID.

Parameters

task_id

The task ID of the setup operation.

Returns

A string representing the result of the setup operation.

IsGenerationReady()

bool AsyncLlamaInfrence::IsGenerationReady

(

int

task_id

)

SetupAsync()

int AsyncLlamaInfrence::SetupAsync	(	const LoggingContext *	logging,
		const gpt_params &	cfg_params )

Asynchronously sets up the model with specified configuration LLama.cpp parameters.

Parameters

cfg_params

A reference to a gpt_params struct containing the setup parameters for the model.

Returns

An integer representing the task ID for the setup operation.

Field Documentation

pImpl

LlamaAsyncImpl* AsyncLlamaInfrence::pImpl

Pointer to the implementation class, used by the PImpl idiom.

---

The documentation for this class was generated from the following file:

• wrapper/infrence-lib/lib.h