Current Version: v.1.7.0

Version Release Schedule

• v0.1.0 (Pilot) August 3, 2024

• v1.0.0 August 17, 2024

• v1.1.0 August 31, 2024

• v1.2.0 September 14, 2024

• v1.3.0 September 28, 2024

• v1.4.0 October 12, 2024

• v1.5.0 October 26, 2024

• v1.6.0 November 9, 2024

• v1.7.0 November 23, 2024

Supported Connectors:

• MySQL (since v0.1.0 Pilot)

• IBM DB2 (since v0.1.0 Pilot)

• Oracle (since v0.1.0 Pilot)

• Postgres (since v0.1.0 Pilot)

• Snowflake Data Warehouse (since v1.0.0)

• Salesforce (since v1.0.0)

• MongoDB (since v1.0.0)

• Azure Cosmos MongoDB (since v1.0.0)

• Azure Cosmos PostgreSQL (since v1.0.0)

• Azure PostgreSQL (since v1.0.0)

• Azure MySQL (since v1.0.0)

• Azure SqlServer (since v1.0.0)

• Google CLoud MongoDB Atlas (since v1.0.0)

• AWS Aurora MySQL (since v1.1.0)

• AWS Aurora PostgreSQL (since v1.1.0)

• AWS RDS PostgreSQL (since v1.1.0)

• AWS RDS MySQL (since v1.1.0)

• AWS RDS SqlServer (since v1.1.0)

• AWS RDS Oracle (since v1.1.0)

• AWS RDS MariaDB (since v1.1.0)

• Clickhouse DB (since v1.1.0)

• ElasticSearch (since v1.1.0)

• Fauna DB (since v1.1.0)

• Cockroach DB (since v1.1.0)

• Amazon S3 (since v1.2.0)

• Amazon S3 Csv (since v1.2.0)

• Amazon S3 Excel (since v1.2.0)

• Azure Data Lake (since v1.2.0)

• Azure Data Lake Csv (since v1.2.0)

• Azure Data Lake Excel (since v1.2.0)

• FTP (since v1.2.0)

• FTP Csv (since v1.2.0)

• FTP Excel (since v1.2.0)

• Google Sheets (since v1.2.0)

• Google Firebase (since v1.3.0)

• Google Firestore (since v1.3.0)

• Heroku Postgres (since v1.3.0)

• Google Cloud Platform SqlServer (since v1.3.0)

• Google Cloud Platform MySQL (since v1.3.0)

• Google Cloud Platform PostgreSql (since v1.3.0)

• CSV File for Agent (since v1.4.0)

• Excel File for Agent (since v1.4.0)

• Rest Api Connector (since v1.5.0)

Installation and Minimum Hardware Requirements

...

Follow these steps to install the Secure Agent on your system.

Prerequisites

(Python 3.9)

• Install python latest 3.9.x in the environment which is the current supported version for the secure agent using pyspark

  • https://www.python.org/downloads/ for python

  • https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-python.html for conda

• set PYSPARK_PYTHON env to point to 3.9 python binary executable for pyspark or switch to the conda environment using the same version see https://spark.apache.org/docs/latest/configuration.html

(Install JDK)

For Windows:

1. Download JDK:

   • Download the latest JDK from the official Oracle or OpenJDK Website

2. Install JDK:

   • Run the installer and follow the on-screen instructions to install the JDK.

3. Set JAVA_HOME Environment Variable:

   • Open the Start menu and search for "Environment Variables".

   • Click on "Edit the system environment variables".

   • In the System Properties window, click on the "Environment Variables" button.

   • Under "System variables", click "New" and set the variable name to JAVA_HOME and the variable value to the JDK installation path (e.g., C:\Program Files\Java\jdk-11).

   • Find the Path variable in the "System variables" section, select it, and click "Edit".

   • Click "New" and add %JAVA_HOME%\bin to the list.

   • Click "OK" to close all dialogs.

...

• base_socket_url

  • Description: The URL used by the Secure Agent to establish a connection with the EazyDi platform.

  • Purpose: This is a crucial setting that should not be altered. It specifies the endpoint for all communication between the agent and the cloud service.

• username

  • Description: The username associated with your EazyDi account.

  • Purpose: This parameter is used for authenticating the Agent with the EazyDi platform. It ensures that the agent operates under the correct user context.

  • Example: username=myusername@eazydi.com

• token

  • Description: The authentication token provided by EazyDi.

  • Purpose: This token is used in conjunction with the username to authenticate the Secure Agent. It is essential for secure and authorized access to the platform.

  • Example: token=320af1e1-5786-4779-bf20-1ecadb92c900

• spark_memory

  • Description: The amount of memory allocated to Apache Spark.

  • Purpose: This setting determines how much memory is available for Spark jobs executed by the Agent. Adequate memory allocation is crucial for performance, especially when handling large datasets. see https://spark.apache.org/docs/latest/configuration.html

  • Example: spark_memory=6g

• spark_master

  • Description: The Spark master URL.

  • Purpose: This specifies the master node for the Spark cluster. In a standalone or local mode, it indicates that Spark jobs should run on the local machine. see https://spark.apache.org/docs/latest/submitting-applications.html

  • Example: spark_master=local

• spark_cores

  • Description: The number of CPU cores allocated to Spark.

  • Purpose: This parameter sets the number of CPU cores available for Spark processing tasks. More cores can improve performance by allowing more parallel processing. see https://spark.apache.org/docs/latest/configuration.html

  • Example: spark_cores=2

For Windows

...

• Once Job is complete it will reflect in Eazydi’s Monitor → Archived Listings

Pre and Post Environment Script execution (since v1.6.0)

See https://eazydi-doc.atlassian.net/wiki/spaces/EazyDIDOC/pages/132907059/Pipeline+Pre+and+Post+Script+Executions#Agent-Scripts-Execution