753 Data Sync

This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. The script performs the following tasks:

• Truncate the specified layer in ArcGIS to clear any previous features before adding new ones.
• Fetch data from an API in paginated form.
• Save data from each API response to individual JSON files.
• Aggregate all data from all pages into one JSON file.
• Add the aggregated data as features to an ArcGIS feature service.

Requirements

• Python 3.6 or higher
• Required Python packages (see requirements.txt)
• ArcGIS Online credentials (username and password)
• .env file for configuration (see below for details)

Install Dependencies

To install the required dependencies, use the following command:

pip install -r requirements.txt

Alternatively, you can install the necessary packages individually:

pip install requests
pip install python-dotenv

Configuration

Before running the script, you need to configure some environment variables. Create a .env file in the root of your project with the following details:

API_URL=your_api_url
AGOL_USER=your_arcgis_online_username
AGOL_PASSWORD=your_arcgis_online_password
HOSTNAME=your_arcgis_host
INSTANCE=your_arcgis_instance
FS=your_feature_service
LAYER=your_layer_id
LOG_LEVEL=your_log_level  # e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL

Environment Variables:

• API_URL: The URL of the API you are fetching data from.
• AGOL_USER: Your ArcGIS Online username.
• AGOL_PASSWORD: Your ArcGIS Online password.
• HOSTNAME: The hostname of your ArcGIS Online instance (e.g., www.arcgis.com).
• INSTANCE: The instance name of your ArcGIS Online service.
• FS: The name of the feature service you are working with.
• LAYER: The ID or name of the layer to truncate and add features to.
• LOG_LEVEL: The desired logging level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL).

Script Usage

You can run the script with the following command:

python 753DataSync.py --results_per_page <number_of_results_per_page>

Arguments:

• --results_per_page (optional): The number of results to fetch per page (default: 100).

Functionality

1. Truncate Layer:

Before fetching and adding any new data, the script will call the truncate function to clear out any existing features from the specified layer. This ensures that the feature layer is empty and ready for the new data.

2. Fetch Data:

The script will then fetch data from the specified API in pages. Each page is fetched sequentially until all data is retrieved.

3. Save Data:

Data from each page will be saved to an individual JSON file, with the filename including the page number and timestamp. The aggregated data (all pages combined) is saved to a separate file.

4. Add Features:

After all the data has been fetched and saved, the script will send the aggregated data as features to the specified ArcGIS feature layer.

Example Output

• Individual page files are saved in the data/ directory with filenames like enforcement_page_1_results_100_2025-03-26_14-30-45.json.
• The aggregated file is saved as aggregated_enforcement_results_2025-03-26_14-30-45.json.

Logs will also be generated in the 753DataSync.log file and printed to the console.

Example Output (Log)

2025-03-26 14:30:45 - INFO - Attempting to truncate layer on https://www.arcgis.com/...
2025-03-26 14:30:50 - INFO - Successfully truncated layer: https://www.arcgis.com/...
2025-03-26 14:30:51 - INFO - Making request to: https://api.example.com/1/100
2025-03-26 14:30:55 - INFO - Data saved to data/enforcement_page_1_results_100_2025-03-26_14-30-45.json
2025-03-26 14:30:56 - INFO - No more data to fetch, stopping pagination.
2025-03-26 14:30:57 - INFO - Data saved to data/aggregated_enforcement_results_2025-03-26_14-30-45.json
2025-03-26 14:31:00 - INFO - Features added successfully.

Error Handling

The script handles errors gracefully, including:

• If an error occurs while fetching data, the script will log the error and stop execution.
• If the truncate or add_features operations fail, the script will log the error and stop execution.
• The script handles HTTP errors and network-related errors gracefully, ensuring that any issues are logged with detailed information.

Troubleshooting

• If the script stops unexpectedly, check the logs (753DataSync.log) for detailed error information.
• Ensure your .env file is correctly configured with valid credentials and API URL.
• Make sure the specified ArcGIS layer is accessible and has the correct permissions for truncating and adding features.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.