Skip to main content

job.proto

path mgmt/v1alpha1/job.proto

package mgmt.v1alpha1


Messages

ActivityFailure

NameTypeDescription
messagestring

ActivityOptions

Config that contains various timeouts that are configured in the underlying temporal workflow(s) and activities

NameTypeDescription
schedule_to_close_timeoutoptional int64Total time that a workflow is willing to wait for an activity to complete, including retries. Measured in seconds
start_to_close_timeoutoptional int64Max time of a single Temporal Activity execution attempt. This timeout should be as short as the longest psosible execution of any activity (e.g. table sync). Important to know that this is per retry attempt. Defaults to the schedule to close timeout if not provided. Measured in seconds
retry_policyRetryPolicyOptionally define a retry policy for the activity If max attempts is not set, the activity will retry indefinitely until the start to close timeout lapses

AiGenerateSourceOptions

NameTypeDescription
ai_connection_idstringThe connection id that corresponds with an AI-based Neosync connection
schemasrepeated AiGenerateSourceSchemaOptionThe list of schemas (and their tables) along with any configuration options that will be used to generate data for.
fk_source_connection_idoptional stringAn optional connection id that will be used as the basis for the shape of data to be generated.
model_namestringThe name of the model to use
user_promptoptional stringOptionally provide a user prompt to give more context to the schema
generate_batch_sizeoptional int64The batch size that will be used to generate X number of records. This is global and will be applied to all tables configured.

AiGenerateSourceSchemaOption

NameTypeDescription
schemastringThe dataabase schema
tablesrepeated AiGenerateSourceTableOptionThe list of tables (and their configuration) that reside within the schema to receive generated data

AiGenerateSourceTableOption

NameTypeDescription
tablestringThe table that will be used to. 1. The schema of the table will be injected into the prompt, of which the resulting data will then be inserted.
row_countint64The total number of records to be generated.

AwsS3DestinationConnectionOptions

NameTypeDescription

AwsS3SourceConnectionOptions

NameTypeDescription
connection_idstring

CancelJobRunRequest

NameTypeDescription
job_run_idstring
account_idstring

CancelJobRunResponse

NameTypeDescription

ColumnError

NameTypeDescription
schemastring
tablestring
columnstring
errorsrepeated string

CreateJobDestination

NameTypeDescription
connection_idstring
optionsJobDestinationOptions

CreateJobDestinationConnectionsRequest

NameTypeDescription
job_idstring
destinationsrepeated CreateJobDestination

CreateJobDestinationConnectionsResponse

NameTypeDescription
jobJob

CreateJobRequest

NameTypeDescription
account_idstringThe unique account identifier that this job will be associated with
job_namestringThe unique, friendly name of the job. This is unique per account
cron_scheduleoptional stringOptionally provide a cron schedule. Goes into effect if the job status is set to enabled
mappingsrepeated JobMapping
sourceJobSource
destinationsrepeated CreateJobDestination
initiate_job_runboolInitially trigger a run of this job regardless of its status or cron schedule
workflow_optionsWorkflowOptionsSpecify timeouts and other workflow options for the underlying temporal workflow
sync_optionsActivityOptionsSpecify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table
virtual_foreign_keysrepeated VirtualForeignConstraint

CreateJobResponse

NameTypeDescription
jobJob

CreateJobRunRequest

NameTypeDescription
job_idstring

CreateJobRunResponse

NameTypeDescription

DatabaseError

NameTypeDescription
errorsrepeated string

DeleteJobDestinationConnectionRequest

NameTypeDescription
destination_idstring

DeleteJobDestinationConnectionResponse

NameTypeDescription

DeleteJobRequest

NameTypeDescription
idstring

DeleteJobResponse

NameTypeDescription

DeleteJobRunRequest

NameTypeDescription
job_run_idstring
account_idstring

DeleteJobRunResponse

NameTypeDescription

GcpCloudStorageDestinationConnectionOptions

Configuration for Google Cloud Storage Connection Job Options

NameTypeDescription

GenerateSourceOptions

NameTypeDescription
schemasrepeated GenerateSourceSchemaOption
fk_source_connection_idoptional string

GenerateSourceSchemaOption

NameTypeDescription
schemastring
tablesrepeated GenerateSourceTableOption

GenerateSourceTableOption

NameTypeDescription
tablestring
row_countint64

GetJobNextRunsRequest

NameTypeDescription
job_idstring

GetJobNextRunsResponse

NameTypeDescription
next_runsJobNextRuns

GetJobRecentRunsRequest

NameTypeDescription
job_idstring

GetJobRecentRunsResponse

NameTypeDescription
recent_runsrepeated JobRecentRun

GetJobRequest

NameTypeDescription
idstring

GetJobResponse

NameTypeDescription
jobJob

GetJobRunEventsRequest

NameTypeDescription
job_run_idstring
account_idstring

GetJobRunEventsResponse

NameTypeDescription
eventsrepeated JobRunEvent
is_run_completebool

GetJobRunLogsStreamRequest

NameTypeDescription
job_run_idstring
account_idstring
windowLogWindowThe time window in which to retrieve the logs
should_tailboolWhether or not to tail the stream. Note: only works with k8s-pods and is not currently supported with Loki logs
max_log_linesoptional int64Optionally provide a max log limit
log_levelsrepeated LogLevelProvide a list of log levels to filter by. If any of these are UNSPECIFIED, all log levels are returned.

GetJobRunLogsStreamResponse

NameTypeDescription
log_linestring
timestampoptional google.protobuf.Timestamp

GetJobRunRequest

NameTypeDescription
job_run_idstring
account_idstring

GetJobRunResponse

NameTypeDescription
job_runJobRun

GetJobRunsRequest

NameTypeDescription
job_idstring
account_idstring

GetJobRunsResponse

NameTypeDescription
job_runsrepeated JobRun

GetJobStatusRequest

NameTypeDescription
job_idstring

GetJobStatusResponse

NameTypeDescription
statusJobStatus

GetJobStatusesRequest

NameTypeDescription
account_idstring

GetJobStatusesResponse

NameTypeDescription
statusesrepeated JobStatusRecord

GetJobsRequest

NameTypeDescription
account_idstring

GetJobsResponse

NameTypeDescription
jobsrepeated Job

IsJobNameAvailableRequest

NameTypeDescription
namestring
account_idstring

IsJobNameAvailableResponse

NameTypeDescription
is_availablebool

Job

NameTypeDescription
idstringThe unique identifier of the job
created_by_user_idstring
created_atgoogle.protobuf.Timestamp
updated_by_user_idstring
updated_atgoogle.protobuf.Timestamp
namestringThe unique, friendly name of the job
sourceJobSource
destinationsrepeated JobDestination
mappingsrepeated JobMapping
cron_scheduleoptional string
account_idstringThe account identifier that a job is associated with
sync_optionsActivityOptionsSpecify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table
workflow_optionsWorkflowOptionsSpecify timeouts and other workflow options for the underlying temporal workflow
virtual_foreign_keysrepeated VirtualForeignConstraint

JobDestination

NameTypeDescription
connection_idstring
optionsJobDestinationOptions
idstring

JobDestinationOptions

NameTypeDescription
postgres_optionsPostgresDestinationConnectionOptions
aws_s3_optionsAwsS3DestinationConnectionOptions
mysql_optionsMysqlDestinationConnectionOptions
mongodb_optionsMongoDBDestinationConnectionOptions
gcp_cloudstorage_optionsGcpCloudStorageDestinationConnectionOptionsDestination Connecton options for Google Cloud Storage

JobMapping

NameTypeDescription
schemastring
tablestring
columnstring
transformerJobMappingTransformer

JobMappingTransformer

NameTypeDescription
sourceTransformerSource
configTransformerConfig

JobNextRuns

NameTypeDescription
next_run_timesrepeated google.protobuf.Timestamp

JobRecentRun

NameTypeDescription
start_timegoogle.protobuf.Timestamp
job_run_idstring

JobRun

NameTypeDescription
idstringThe id of the job run. This will currently be equivalent to the temporal workflow id
job_idstringThe unique identifier of the job id this run is associated with
namestringThe name of the job run.
statusJobRunStatusthe status of the job run
started_atgoogle.protobuf.TimestampA timestamp of when the run started
completed_atoptional google.protobuf.TimestampAvailable if the run completed or has not yet been archived by the system
pending_activitiesrepeated PendingActivityPending activities are only returned when retrieving a specific job run and will not be returned when requesting job runs in list format

JobRunEvent

NameTypeDescription
idint64
typestring
start_timegoogle.protobuf.Timestamp
close_timegoogle.protobuf.Timestamp
metadataJobRunEventMetadata
tasksrepeated JobRunEventTask

JobRunEventMetadata

NameTypeDescription
sync_metadataJobRunSyncMetadata

JobRunEventTask

NameTypeDescription
idint64
typestring
event_timegoogle.protobuf.Timestamp
errorJobRunEventTaskError

JobRunEventTaskError

NameTypeDescription
messagestring
retry_statestring

JobRunSyncMetadata

NameTypeDescription
schemastring
tablestring

JobSource

NameTypeDescription
optionsJobSourceOptions

JobSourceOptions

NameTypeDescription
postgresPostgresSourceConnectionOptions
aws_s3AwsS3SourceConnectionOptions
mysqlMysqlSourceConnectionOptions
generateGenerateSourceOptions
ai_generateAiGenerateSourceOptions
mongodbMongoDBSourceConnectionOptions

JobSourceSqlSubetSchemas

NameTypeDescription
postgres_subsetPostgresSourceSchemaSubset
mysql_subsetMysqlSourceSchemaSubset

JobStatusRecord

NameTypeDescription
job_idstring
statusJobStatus

MongoDBDestinationConnectionOptions

NameTypeDescription

MongoDBSourceConnectionOptions

NameTypeDescription
connection_idstring

MysqlDestinationConnectionOptions

NameTypeDescription
truncate_tableMysqlTruncateTableConfig
init_table_schemabool
on_conflictMysqlOnConflictConfig

MysqlOnConflictConfig

NameTypeDescription
do_nothingbool

MysqlSourceConnectionOptions

NameTypeDescription
halt_on_new_column_additionbool
schemasrepeated MysqlSourceSchemaOption
connection_idstring
subset_by_foreign_key_constraintsbool

MysqlSourceSchemaOption

NameTypeDescription
schemastring
tablesrepeated MysqlSourceTableOption

MysqlSourceSchemaSubset

NameTypeDescription
mysql_schemasrepeated MysqlSourceSchemaOption

MysqlSourceTableOption

NameTypeDescription
tablestring
where_clauseoptional string

MysqlTruncateTableConfig

NameTypeDescription
truncate_before_insertbool

PauseJobRequest

NameTypeDescription
idstring
pausebool
noteoptional string

PauseJobResponse

NameTypeDescription
jobJob

PendingActivity

NameTypeDescription
statusActivityStatus
activity_namestring
last_failureoptional ActivityFailure

PostgresDestinationConnectionOptions

NameTypeDescription
truncate_tablePostgresTruncateTableConfig
init_table_schemabool
on_conflictPostgresOnConflictConfig

PostgresOnConflictConfig

NameTypeDescription
do_nothingbool

PostgresSourceConnectionOptions

NameTypeDescription
halt_on_new_column_additionbool
schemasrepeated PostgresSourceSchemaOption
connection_idstring
subset_by_foreign_key_constraintsbool

PostgresSourceSchemaOption

NameTypeDescription
schemastring
tablesrepeated PostgresSourceTableOption

PostgresSourceSchemaSubset

NameTypeDescription
postgres_schemasrepeated PostgresSourceSchemaOption

PostgresSourceTableOption

NameTypeDescription
tablestring
where_clauseoptional string

PostgresTruncateTableConfig

NameTypeDescription
truncate_before_insertbool
cascadebool

RetryPolicy

Defines the retry policy for an activity

NameTypeDescription
maximum_attemptsoptional int32Maximum number of attempts. When exceeded the retries stop even if not expired yet. If not set or set to 0, it means unlimited, and rely on activity ScheduleToCloseTimeout to stop.

SetJobSourceSqlConnectionSubsetsRequest

NameTypeDescription
idstring
schemasJobSourceSqlSubetSchemas
subset_by_foreign_key_constraintsbool

SetJobSourceSqlConnectionSubsetsResponse

NameTypeDescription
jobJob

SetJobSyncOptionsRequest

NameTypeDescription
idstringThe unique identifier of the job
sync_optionsActivityOptionsThe sync options object. The entire object must be provided and will fully overwrite the previous result

SetJobSyncOptionsResponse

NameTypeDescription
jobJob

SetJobWorkflowOptionsRequest

NameTypeDescription
idstringThe unique identifier of the job
worfklow_optionsWorkflowOptionsThe workflow options object. The entire object must be provided and will fully overwrite the previous result

SetJobWorkflowOptionsResponse

NameTypeDescription
jobJob

TerminateJobRunRequest

NameTypeDescription
job_run_idstring
account_idstring

TerminateJobRunResponse

NameTypeDescription

UpdateJobDestinationConnectionRequest

NameTypeDescription
job_idstring
connection_idstring
optionsJobDestinationOptions
destination_idstring

UpdateJobDestinationConnectionResponse

NameTypeDescription
jobJob

UpdateJobScheduleRequest

NameTypeDescription
idstring
cron_scheduleoptional string

UpdateJobScheduleResponse

NameTypeDescription
jobJob

UpdateJobSourceConnectionRequest

NameTypeDescription
idstring
sourceJobSource
mappingsrepeated JobMapping
virtual_foreign_keysrepeated VirtualForeignConstraint

UpdateJobSourceConnectionResponse

NameTypeDescription
jobJob

ValidateJobMappingsRequest

NameTypeDescription
account_idstringThe unique account identifier that this job will be associated with
mappingsrepeated JobMapping
connection_idstring
virtual_foreign_keysrepeated VirtualForeignConstraint

ValidateJobMappingsResponse

NameTypeDescription
column_errorsrepeated ColumnError
database_errorsDatabaseError

VirtualForeignConstraint

NameTypeDescription
schemastring
tablestring
columnsrepeated string
foreign_keyVirtualForeignKey

VirtualForeignKey

NameTypeDescription
schemastring
tablestring
columnsrepeated string

WorkflowOptions

Config that contains various timeouts that are configured in the underlying temporal workflow More options will come in the future as needed

NameTypeDescription
run_timeoutoptional int64The timeout for a single workflow run. Measured in seconds

Enums

ActivityStatus

NameNumberDescription
ACTIVITY_STATUS_UNSPECIFIED0
ACTIVITY_STATUS_SCHEDULED1
ACTIVITY_STATUS_STARTED2
ACTIVITY_STATUS_CANCELED3
ACTIVITY_STATUS_FAILED4

JobRunStatus

An enumeration of job run statuses.

NameNumberDescription
JOB_RUN_STATUS_UNSPECIFIED0if the job run status is unknown
JOB_RUN_STATUS_PENDING1the run is pending and has not started yet
JOB_RUN_STATUS_RUNNING2the run is currently in progress
JOB_RUN_STATUS_COMPLETE3the run has successfully completed
JOB_RUN_STATUS_ERROR4the run ended with an error
JOB_RUN_STATUS_CANCELED5the run was cancelled
JOB_RUN_STATUS_TERMINATED6the run was terminated
JOB_RUN_STATUS_FAILED7the run ended in failure
JOB_RUN_STATUS_TIMED_OUT8the run was ended pre-maturely due to timeout

JobStatus

NameNumberDescription
JOB_STATUS_UNSPECIFIED0
JOB_STATUS_ENABLED1
JOB_STATUS_PAUSED3
JOB_STATUS_DISABLED4

LogLevel

NameNumberDescription
LOG_LEVEL_UNSPECIFIED0
LOG_LEVEL_DEBUG1
LOG_LEVEL_INFO2
LOG_LEVEL_WARN3
LOG_LEVEL_ERROR4

LogWindow

NameNumberDescription
LOG_WINDOW_NO_TIME_UNSPECIFIED0
LOG_WINDOW_FIFTEEN_MIN1
LOG_WINDOW_ONE_HOUR2
LOG_WINDOW_ONE_DAY3

Services

JobService

GetJobs

MethodGetJobs
RequestGetJobsRequest
ResponseGetJobsResponse
Description

GetJob

MethodGetJob
RequestGetJobRequest
ResponseGetJobResponse
Description

CreateJob

MethodCreateJob
RequestCreateJobRequest
ResponseCreateJobResponse
Description

DeleteJob

MethodDeleteJob
RequestDeleteJobRequest
ResponseDeleteJobResponse
Description

IsJobNameAvailable

MethodIsJobNameAvailable
RequestIsJobNameAvailableRequest
ResponseIsJobNameAvailableResponse
Description

UpdateJobSchedule

MethodUpdateJobSchedule
RequestUpdateJobScheduleRequest
ResponseUpdateJobScheduleResponse
Description

UpdateJobSourceConnection

MethodUpdateJobSourceConnection
RequestUpdateJobSourceConnectionRequest
ResponseUpdateJobSourceConnectionResponse
Description

SetJobSourceSqlConnectionSubsets

MethodSetJobSourceSqlConnectionSubsets
RequestSetJobSourceSqlConnectionSubsetsRequest
ResponseSetJobSourceSqlConnectionSubsetsResponse
Description

UpdateJobDestinationConnection

MethodUpdateJobDestinationConnection
RequestUpdateJobDestinationConnectionRequest
ResponseUpdateJobDestinationConnectionResponse
Description

DeleteJobDestinationConnection

MethodDeleteJobDestinationConnection
RequestDeleteJobDestinationConnectionRequest
ResponseDeleteJobDestinationConnectionResponse
Description

CreateJobDestinationConnections

MethodCreateJobDestinationConnections
RequestCreateJobDestinationConnectionsRequest
ResponseCreateJobDestinationConnectionsResponse
Description

PauseJob

MethodPauseJob
RequestPauseJobRequest
ResponsePauseJobResponse
Description

GetJobRecentRuns

MethodGetJobRecentRuns
RequestGetJobRecentRunsRequest
ResponseGetJobRecentRunsResponse
DescriptionReturns a list of recently invoked job runs baseds on the Temporal cron scheduler. This will return a list of job runs that include archived runs

GetJobNextRuns

MethodGetJobNextRuns
RequestGetJobNextRunsRequest
ResponseGetJobNextRunsResponse
DescriptionReturns a list of runs that are scheduled for execution based on the Temporal cron scheduler.

GetJobStatus

MethodGetJobStatus
RequestGetJobStatusRequest
ResponseGetJobStatusResponse
Description

GetJobStatuses

MethodGetJobStatuses
RequestGetJobStatusesRequest
ResponseGetJobStatusesResponse
Description

GetJobRuns

MethodGetJobRuns
RequestGetJobRunsRequest
ResponseGetJobRunsResponse
DescriptionReturns a list of job runs by either account or job

GetJobRunEvents

MethodGetJobRunEvents
RequestGetJobRunEventsRequest
ResponseGetJobRunEventsResponse
Description

GetJobRun

MethodGetJobRun
RequestGetJobRunRequest
ResponseGetJobRunResponse
DescriptionReturns a specific job run, along with any of its pending activities

DeleteJobRun

MethodDeleteJobRun
RequestDeleteJobRunRequest
ResponseDeleteJobRunResponse
Description

CreateJobRun

MethodCreateJobRun
RequestCreateJobRunRequest
ResponseCreateJobRunResponse
Description

CancelJobRun

MethodCancelJobRun
RequestCancelJobRunRequest
ResponseCancelJobRunResponse
Description

TerminateJobRun

MethodTerminateJobRun
RequestTerminateJobRunRequest
ResponseTerminateJobRunResponse
Description

GetJobRunLogsStream

MethodGetJobRunLogsStream
RequestGetJobRunLogsStreamRequest
ResponseGetJobRunLogsStreamResponse stream
DescriptionReturns a stream of logs from the worker nodes that pertain to a specific job run

SetJobWorkflowOptions

MethodSetJobWorkflowOptions
RequestSetJobWorkflowOptionsRequest
ResponseSetJobWorkflowOptionsResponse
DescriptionSet any job workflow options. Must provide entire object as is it will fully override the previous configuration

SetJobSyncOptions

MethodSetJobSyncOptions
RequestSetJobSyncOptionsRequest
ResponseSetJobSyncOptionsResponse
DescriptionSet the job sync options. Must provide entire object as it will fully override the previous configuration

ValidateJobMappings

MethodValidateJobMappings
RequestValidateJobMappingsRequest
ResponseValidateJobMappingsResponse
Descriptionvalidates that the jobmapping configured can run with table constraints