Google Cloud Native is in preview. Google Cloud Classic is fully supported.
Google Cloud Native v0.32.0 published on Wednesday, Nov 29, 2023 by Pulumi
google-native.dialogflow/v2.getEvaluation
Explore with Pulumi AI
Google Cloud Native is in preview. Google Cloud Classic is fully supported.
Google Cloud Native v0.32.0 published on Wednesday, Nov 29, 2023 by Pulumi
Gets an evaluation of conversation model.
Using getEvaluation
Two invocation forms are available. The direct form accepts plain arguments and either blocks until the result value is available, or returns a Promise-wrapped result. The output form accepts Input-wrapped arguments and returns an Output-wrapped result.
function getEvaluation(args: GetEvaluationArgs, opts?: InvokeOptions): Promise<GetEvaluationResult>
function getEvaluationOutput(args: GetEvaluationOutputArgs, opts?: InvokeOptions): Output<GetEvaluationResult>
def get_evaluation(conversation_model_id: Optional[str] = None,
evaluation_id: Optional[str] = None,
location: Optional[str] = None,
project: Optional[str] = None,
opts: Optional[InvokeOptions] = None) -> GetEvaluationResult
def get_evaluation_output(conversation_model_id: Optional[pulumi.Input[str]] = None,
evaluation_id: Optional[pulumi.Input[str]] = None,
location: Optional[pulumi.Input[str]] = None,
project: Optional[pulumi.Input[str]] = None,
opts: Optional[InvokeOptions] = None) -> Output[GetEvaluationResult]
func LookupEvaluation(ctx *Context, args *LookupEvaluationArgs, opts ...InvokeOption) (*LookupEvaluationResult, error)
func LookupEvaluationOutput(ctx *Context, args *LookupEvaluationOutputArgs, opts ...InvokeOption) LookupEvaluationResultOutput
> Note: This function is named LookupEvaluation
in the Go SDK.
public static class GetEvaluation
{
public static Task<GetEvaluationResult> InvokeAsync(GetEvaluationArgs args, InvokeOptions? opts = null)
public static Output<GetEvaluationResult> Invoke(GetEvaluationInvokeArgs args, InvokeOptions? opts = null)
}
public static CompletableFuture<GetEvaluationResult> getEvaluation(GetEvaluationArgs args, InvokeOptions options)
public static Output<GetEvaluationResult> getEvaluation(GetEvaluationArgs args, InvokeOptions options)
fn::invoke:
function: google-native:dialogflow/v2:getEvaluation
arguments:
# arguments dictionary
The following arguments are supported:
- Conversation
Model Id This property is required. string - Evaluation
Id This property is required. string - Location
This property is required. string - Project string
- Conversation
Model Id This property is required. string - Evaluation
Id This property is required. string - Location
This property is required. string - Project string
- conversation
Model Id This property is required. String - evaluation
Id This property is required. String - location
This property is required. String - project String
- conversation
Model Id This property is required. string - evaluation
Id This property is required. string - location
This property is required. string - project string
- conversation_
model_ id This property is required. str - evaluation_
id This property is required. str - location
This property is required. str - project str
- conversation
Model Id This property is required. String - evaluation
Id This property is required. String - location
This property is required. String - project String
getEvaluation Result
The following output properties are available:
- Create
Time string - Creation time of this model.
- Display
Name string - Optional. The display name of the model evaluation. At most 64 bytes long.
- Evaluation
Config Pulumi.Google Native. Dialogflow. V2. Outputs. Google Cloud Dialogflow V2Evaluation Config Response - Optional. The configuration of the evaluation task.
- Name string
- The resource name of the evaluation. Format:
projects//conversationModels//evaluations/
- Raw
Human stringEval Template Csv - Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
- Smart
Reply Pulumi.Metrics Google Native. Dialogflow. V2. Outputs. Google Cloud Dialogflow V2Smart Reply Metrics Response - Only available when model is for smart reply.
- Create
Time string - Creation time of this model.
- Display
Name string - Optional. The display name of the model evaluation. At most 64 bytes long.
- Evaluation
Config GoogleCloud Dialogflow V2Evaluation Config Response - Optional. The configuration of the evaluation task.
- Name string
- The resource name of the evaluation. Format:
projects//conversationModels//evaluations/
- Raw
Human stringEval Template Csv - Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
- Smart
Reply GoogleMetrics Cloud Dialogflow V2Smart Reply Metrics Response - Only available when model is for smart reply.
- create
Time String - Creation time of this model.
- display
Name String - Optional. The display name of the model evaluation. At most 64 bytes long.
- evaluation
Config GoogleCloud Dialogflow V2Evaluation Config Response - Optional. The configuration of the evaluation task.
- name String
- The resource name of the evaluation. Format:
projects//conversationModels//evaluations/
- raw
Human StringEval Template Csv - Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
- smart
Reply GoogleMetrics Cloud Dialogflow V2Smart Reply Metrics Response - Only available when model is for smart reply.
- create
Time string - Creation time of this model.
- display
Name string - Optional. The display name of the model evaluation. At most 64 bytes long.
- evaluation
Config GoogleCloud Dialogflow V2Evaluation Config Response - Optional. The configuration of the evaluation task.
- name string
- The resource name of the evaluation. Format:
projects//conversationModels//evaluations/
- raw
Human stringEval Template Csv - Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
- smart
Reply GoogleMetrics Cloud Dialogflow V2Smart Reply Metrics Response - Only available when model is for smart reply.
- create_
time str - Creation time of this model.
- display_
name str - Optional. The display name of the model evaluation. At most 64 bytes long.
- evaluation_
config GoogleCloud Dialogflow V2Evaluation Config Response - Optional. The configuration of the evaluation task.
- name str
- The resource name of the evaluation. Format:
projects//conversationModels//evaluations/
- raw_
human_ streval_ template_ csv - Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
- smart_
reply_ Googlemetrics Cloud Dialogflow V2Smart Reply Metrics Response - Only available when model is for smart reply.
- create
Time String - Creation time of this model.
- display
Name String - Optional. The display name of the model evaluation. At most 64 bytes long.
- evaluation
Config Property Map - Optional. The configuration of the evaluation task.
- name String
- The resource name of the evaluation. Format:
projects//conversationModels//evaluations/
- raw
Human StringEval Template Csv - Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
- smart
Reply Property MapMetrics - Only available when model is for smart reply.
Supporting Types
GoogleCloudDialogflowV2EvaluationConfigResponse
- Datasets
This property is required. List<Pulumi.Google Native. Dialogflow. V2. Inputs. Google Cloud Dialogflow V2Input Dataset Response> - Datasets used for evaluation.
- Smart
Compose Config This property is required. Pulumi.Google Native. Dialogflow. V2. Inputs. Google Cloud Dialogflow V2Evaluation Config Smart Compose Config Response - Configuration for smart compose model evalution.
- Smart
Reply Config This property is required. Pulumi.Google Native. Dialogflow. V2. Inputs. Google Cloud Dialogflow V2Evaluation Config Smart Reply Config Response - Configuration for smart reply model evalution.
- Datasets
This property is required. []GoogleCloud Dialogflow V2Input Dataset Response - Datasets used for evaluation.
- Smart
Compose Config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Compose Config Response - Configuration for smart compose model evalution.
- Smart
Reply Config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Reply Config Response - Configuration for smart reply model evalution.
- datasets
This property is required. List<GoogleCloud Dialogflow V2Input Dataset Response> - Datasets used for evaluation.
- smart
Compose Config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Compose Config Response - Configuration for smart compose model evalution.
- smart
Reply Config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Reply Config Response - Configuration for smart reply model evalution.
- datasets
This property is required. GoogleCloud Dialogflow V2Input Dataset Response[] - Datasets used for evaluation.
- smart
Compose Config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Compose Config Response - Configuration for smart compose model evalution.
- smart
Reply Config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Reply Config Response - Configuration for smart reply model evalution.
- datasets
This property is required. Sequence[GoogleCloud Dialogflow V2Input Dataset Response] - Datasets used for evaluation.
- smart_
compose_ config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Compose Config Response - Configuration for smart compose model evalution.
- smart_
reply_ config This property is required. GoogleCloud Dialogflow V2Evaluation Config Smart Reply Config Response - Configuration for smart reply model evalution.
- datasets
This property is required. List<Property Map> - Datasets used for evaluation.
- smart
Compose Config This property is required. Property Map - Configuration for smart compose model evalution.
- smart
Reply Config This property is required. Property Map - Configuration for smart reply model evalution.
GoogleCloudDialogflowV2EvaluationConfigSmartComposeConfigResponse
- Allowlist
Document This property is required. string - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart compose model. - Max
Result Count This property is required. int - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- Allowlist
Document This property is required. string - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart compose model. - Max
Result Count This property is required. int - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist
Document This property is required. String - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart compose model. - max
Result Count This property is required. Integer - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist
Document This property is required. string - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart compose model. - max
Result Count This property is required. number - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist_
document This property is required. str - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart compose model. - max_
result_ count This property is required. int - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist
Document This property is required. String - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart compose model. - max
Result Count This property is required. Number - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
GoogleCloudDialogflowV2EvaluationConfigSmartReplyConfigResponse
- Allowlist
Document This property is required. string - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart reply model. - Max
Result Count This property is required. int - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- Allowlist
Document This property is required. string - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart reply model. - Max
Result Count This property is required. int - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist
Document This property is required. String - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart reply model. - max
Result Count This property is required. Integer - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist
Document This property is required. string - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart reply model. - max
Result Count This property is required. number - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist_
document This property is required. str - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart reply model. - max_
result_ count This property is required. int - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
- allowlist
Document This property is required. String - The allowlist document resource name. Format:
projects//knowledgeBases//documents/
. Only used for smart reply model. - max
Result Count This property is required. Number - The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
GoogleCloudDialogflowV2InputDatasetResponse
- Dataset
This property is required. string - ConversationDataset resource name. Format:
projects//locations//conversationDatasets/
- Dataset
This property is required. string - ConversationDataset resource name. Format:
projects//locations//conversationDatasets/
- dataset
This property is required. String - ConversationDataset resource name. Format:
projects//locations//conversationDatasets/
- dataset
This property is required. string - ConversationDataset resource name. Format:
projects//locations//conversationDatasets/
- dataset
This property is required. str - ConversationDataset resource name. Format:
projects//locations//conversationDatasets/
- dataset
This property is required. String - ConversationDataset resource name. Format:
projects//locations//conversationDatasets/
GoogleCloudDialogflowV2SmartReplyMetricsResponse
- Allowlist
Coverage This property is required. double - Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
- Conversation
Count This property is required. string - Total number of conversations used to generate this metric.
- Top
NMetrics This property is required. List<Pulumi.Google Native. Dialogflow. V2. Inputs. Google Cloud Dialogflow V2Smart Reply Metrics Top NMetrics Response> - Metrics of top n smart replies, sorted by TopNMetric.n.
- Allowlist
Coverage This property is required. float64 - Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
- Conversation
Count This property is required. string - Total number of conversations used to generate this metric.
- Top
NMetrics This property is required. []GoogleCloud Dialogflow V2Smart Reply Metrics Top NMetrics Response - Metrics of top n smart replies, sorted by TopNMetric.n.
- allowlist
Coverage This property is required. Double - Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
- conversation
Count This property is required. String - Total number of conversations used to generate this metric.
- top
NMetrics This property is required. List<GoogleCloud Dialogflow V2Smart Reply Metrics Top NMetrics Response> - Metrics of top n smart replies, sorted by TopNMetric.n.
- allowlist
Coverage This property is required. number - Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
- conversation
Count This property is required. string - Total number of conversations used to generate this metric.
- top
NMetrics This property is required. GoogleCloud Dialogflow V2Smart Reply Metrics Top NMetrics Response[] - Metrics of top n smart replies, sorted by TopNMetric.n.
- allowlist_
coverage This property is required. float - Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
- conversation_
count This property is required. str - Total number of conversations used to generate this metric.
- top_
n_ metrics This property is required. Sequence[GoogleCloud Dialogflow V2Smart Reply Metrics Top NMetrics Response] - Metrics of top n smart replies, sorted by TopNMetric.n.
- allowlist
Coverage This property is required. Number - Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
- conversation
Count This property is required. String - Total number of conversations used to generate this metric.
- top
NMetrics This property is required. List<Property Map> - Metrics of top n smart replies, sorted by TopNMetric.n.
GoogleCloudDialogflowV2SmartReplyMetricsTopNMetricsResponse
- N
This property is required. int - Number of retrieved smart replies. For example, when
n
is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model. - Recall
This property is required. double - Defined as
number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply
divided bynumber of queries with at least one smart reply
. Value ranges from 0.0 to 1.0 inclusive.
- N
This property is required. int - Number of retrieved smart replies. For example, when
n
is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model. - Recall
This property is required. float64 - Defined as
number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply
divided bynumber of queries with at least one smart reply
. Value ranges from 0.0 to 1.0 inclusive.
- n
This property is required. Integer - Number of retrieved smart replies. For example, when
n
is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model. - recall
This property is required. Double - Defined as
number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply
divided bynumber of queries with at least one smart reply
. Value ranges from 0.0 to 1.0 inclusive.
- n
This property is required. number - Number of retrieved smart replies. For example, when
n
is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model. - recall
This property is required. number - Defined as
number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply
divided bynumber of queries with at least one smart reply
. Value ranges from 0.0 to 1.0 inclusive.
- n
This property is required. int - Number of retrieved smart replies. For example, when
n
is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model. - recall
This property is required. float - Defined as
number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply
divided bynumber of queries with at least one smart reply
. Value ranges from 0.0 to 1.0 inclusive.
- n
This property is required. Number - Number of retrieved smart replies. For example, when
n
is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model. - recall
This property is required. Number - Defined as
number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply
divided bynumber of queries with at least one smart reply
. Value ranges from 0.0 to 1.0 inclusive.
Package Details
- Repository
- Google Cloud Native pulumi/pulumi-google-native
- License
- Apache-2.0
Google Cloud Native is in preview. Google Cloud Classic is fully supported.
Google Cloud Native v0.32.0 published on Wednesday, Nov 29, 2023 by Pulumi