How to copy data from On-Prem PostgreSQL to Azure SQL Database use Data Factory

  1. Set up a self-hosted integration runtime on a server that has access to the On-premises PostgreSQL database. This integration runtime acts as a bridge between your on-premises data source and Azure Data Factory.
  2. Create a linked service for the On-premises PostgreSQL data source in the Azure Data Factory portal, and specify the integration runtime you set up in step 1 as the integration runtime for the linked service.
  3. Create a linked service for the Azure SQL Database destination in the Azure Data Factory portal.
  4. Create a pipeline in Azure Data Factory that includes a copy activity.
  5. In the copy activity, select the On-premises PostgreSQL data source as the source, and the Azure SQL Database destination as the sink.
  6. Configure the source and sink datasets with the necessary connection details and credentials.
  7. Map the columns in the source dataset to the corresponding columns in the sink dataset.
  8. Optionally, you can apply transformations to the data using Azure Data Factory’s built-in data transformation activities, such as mapping, aggregation, and filtering.
  9. Finally, you can run the pipeline to move the data from On-premises PostgreSQL to Azure SQL Database.

##JSON##
{
    "name": "CopyPostgreSQL2SQL",
    "properties": {
        "activities": [
            {
                "name": "CopyPostgreSQL2SQL",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "PostgreSqlSource"
                    },
                    "sink": {
                        "type": "AzureSqlSink",
                        "writeBehavior": "insert",
                        "sqlWriterUseTableLock": true,
                        "disableMetricsCollection": false
                    },
                    "enableStaging": false,
                    "translator": {
                        "type": "TabularTranslator",
                        "mappings": [
                            {
                                "source": {
                                    "name": "employee_id",
                                    "type": "Int32"
                                },
                                "sink": {
                                    "name": "employee_id",
                                    "type": "Int32",
                                    "physicalType": "int"
                                }
                            },
                            {
                                "source": {
                                    "name": "first_name",
                                    "type": "String"
                                },
                                "sink": {
                                    "name": "first_name",
                                    "type": "String",
                                    "physicalType": "varchar"
                                }
                            },
                            {
                                "source": {
                                    "name": "last_name",
                                    "type": "String"
                                },
                                "sink": {
                                    "name": "last_name",
                                    "type": "String",
                                    "physicalType": "varchar"
                                }
                            },
                            {
                                "source": {
                                    "name": "birth_date",
                                    "type": "DateTime"
                                },
                                "sink": {
                                    "name": "birth_date",
                                    "type": "DateTime",
                                    "physicalType": "date"
                                }
                            },
                            {
                                "source": {
                                    "name": "hire_date",
                                    "type": "DateTime"
                                },
                                "sink": {
                                    "name": "hire_date",
                                    "type": "DateTime",
                                    "physicalType": "date"
                                }
                            }
                        ],
                        "typeConversion": true,
                        "typeConversionSettings": {
                            "allowDataTruncation": true,
                            "treatBooleanAsNumber": false
                        }
                    }
                },
                "inputs": [
                    {
                        "referenceName": "PostgreSqlTable",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "Connect2AzSQLDB",
                        "type": "DatasetReference"
                    }
                ]
            }
        ],
        "annotations": [],
        "lastPublishTime": "2023-02-27T12:56:48Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}
Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s