Evaluate an Annotation store against a ground truth Annotation store. When the operation finishes successfully, a detailed response is returned of type EvaluateAnnotationStoreResponse, contained in the response. The metadata field type is OperationMetadata. Errors are logged to Cloud Logging (see Viewing error logs in Cloud Logging).

Scopes

You will need authorization for at least one of the following scopes to make a valid call:

• https://www.googleapis.com/auth/cloud-healthcare
• https://www.googleapis.com/auth/cloud-platform

If unset, the scope for this method defaults to https://www.googleapis.com/auth/cloud-healthcare. You can set the scope for this method like this: healthcare1-beta1 --scope <scope> projects locations-datasets-annotation-stores-evaluate ...

Required Scalar Argument

• <name> (string)
  • Required. The Annotation store to compare against golden_store, in the format of projects/{project_id}/locations/{location_id}/datasets/{dataset_id}/annotationStores/{annotation_store_id}.

Required Request Value

The request value is a data-structure with various fields. Each field may be a simple scalar or another data-structure. In the latter case it is advised to set the field-cursor to the data-structure's field to specify values more concisely.

For example, a structure like this:

EvaluateAnnotationStoreRequest:
  bigquery-destination:
    force: boolean
    schema-type: string
    table-uri: string
    write-disposition: string
  eval-info-type-mapping: { string: string }
  golden-info-type-mapping: { string: string }
  golden-store: string
  info-type-config:
    evaluate-list:
      info-types: [string]
    ignore-list:
      info-types: [string]
    strict-matching: boolean

can be set completely with the following arguments which are assumed to be executed in the given order. Note how the cursor position is adjusted to the respective structures, allowing simple field names to be used most of the time.

• -r .bigquery-destination force=true
  • Use write_disposition instead. If write_disposition is specified, this parameter is ignored. force=false is equivalent to write_disposition=WRITE_EMPTY and force=true is equivalent to write_disposition=WRITE_TRUNCATE.
• schema-type=gubergren
  • Specifies the schema format to export.
• table-uri=lorem
  • BigQuery URI to a table, up to 2000 characters long, must be of the form bq://projectId.bqDatasetId.tableId.

• write-disposition=gubergren

  • Determines if existing data in the destination dataset is overwritten, appended to, or not written if the tables contain data. If a write_disposition is specified, the force parameter is ignored.

• .. eval-info-type-mapping=key=eos

  • Optional. InfoType mapping for eval_store. Different resources can map to the same infoType. For example, PERSON_NAME, PERSON, NAME, and HUMAN are different. To map all of these into a single infoType (such as PERSON_NAME), specify the following mapping: info_type_mapping["PERSON"] = "PERSON_NAME" info_type_mapping["NAME"] = "PERSON_NAME" info_type_mapping["HUMAN"] = "PERSON_NAME" Unmentioned infoTypes, such as DATE, are treated as identity mapping. For example: info_type_mapping["DATE"] = "DATE" InfoTypes are case-insensitive.
  • the value will be associated with the given key
• golden-info-type-mapping=key=dolor
  • Optional. Similar to eval_info_type_mapping, infoType mapping for golden_store.
  • the value will be associated with the given key
• golden-store=ea
  • Required. The Annotation store to use as ground truth, in the format of projects/{project_id}/locations/{location_id}/datasets/{dataset_id}/annotationStores/{annotation_store_id}.

• info-type-config.evaluate-list info-types=ipsum

  • These infoTypes are based on after the eval_info_type_mapping and golden_info_type_mapping.
  • Each invocation of this argument appends the given value to the array.

• ..ignore-list info-types=invidunt

  • These infoTypes are based on after the eval_info_type_mapping and golden_info_type_mapping.
  • Each invocation of this argument appends the given value to the array.

• .. strict-matching=true

  • If TRUE, infoTypes described by filter are used for evaluation. Otherwise, infoTypes are not considered for evaluation. For example: * Annotated text: "Toronto is a location" * Finding 1: {"infoType": "PERSON", "quote": "Toronto", "start": 0, "end": 7} * Finding 2: {"infoType": "CITY", "quote": "Toronto", "start": 0, "end": 7} * Finding 3: {} * Ground truth: {"infoType": "LOCATION", "quote": "Toronto", "start": 0, "end": 7} When strict_matching is TRUE: * Finding 1: 1 false positive * Finding 2: 1 false positive * Finding 3: 1 false negative When strict_matching is FALSE: * Finding 1: 1 true positive * Finding 2: 1 true positive * Finding 3: 1 false negative

About Cursors

The cursor position is key to comfortably set complex nested structures. The following rules apply:

• The cursor position is always set relative to the current one, unless the field name starts with the . character. Fields can be nested such as in -r f.s.o .
• The cursor position is set relative to the top-level structure if it starts with ., e.g. -r .s.s
• You can also set nested fields without setting the cursor explicitly. For example, to set a value relative to the current cursor position, you would specify -r struct.sub_struct=bar.
• You can move the cursor one level up by using ... Each additional . moves it up one additional level. E.g. ... would go three levels up.

Optional Output Flags

The method's return value a JSON encoded structure, which will be written to standard output by default.

• -o out
  • out specifies the destination to which to write the server's result to. It will be a JSON-encoded structure. The destination may be - to indicate standard output, or a filepath that is to contain the received bytes. If unset, it defaults to standard output.

Optional General Properties

The following properties can configure any call, and are not specific to this method.

• -p $-xgafv=string

  • V1 error format.

• -p access-token=string

  • OAuth access token.

• -p alt=string

  • Data format for response.

• -p callback=string

  • JSONP

• -p fields=string

  • Selector specifying which fields to include in a partial response.

• -p key=string

  • API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.

• -p oauth-token=string

  • OAuth 2.0 token for the current user.

• -p pretty-print=boolean

  • Returns response with indentations and line breaks.

• -p quota-user=string

  • Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

• -p upload-type=string

  • Legacy upload protocol for media (e.g. "media", "multipart").

• -p upload-protocol=string

  • Upload protocol for media (e.g. "raw", "multipart").