Update README file.
This commit is contained in:
Nick Heppler
parent
54e188bc55
commit
2e17eac57d
105
README.md
105
README.md
@ -1,2 +1,105 @@
|
|||||||
# 753-Data-Sync
|
# 753 Data Sync
|
||||||
|
|
||||||
|
This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. The script performs the following tasks:
|
||||||
|
|
||||||
|
1. **Truncate the specified layer** in ArcGIS to clear any previous features before adding new ones.
|
||||||
|
2. **Fetch data** from an API in paginated form.
|
||||||
|
3. **Save data** from each API response to individual JSON files.
|
||||||
|
4. **Aggregate all data** from all pages into one JSON file.
|
||||||
|
5. **Add the aggregated data** as features to an ArcGIS feature service.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.6 or higher
|
||||||
|
- Required Python packages (see `requirements.txt`)
|
||||||
|
- ArcGIS Online credentials (username and password)
|
||||||
|
- `.env` file for configuration (see below for details)
|
||||||
|
|
||||||
|
### Install dependencies
|
||||||
|
|
||||||
|
You can install the required dependencies using `pip`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Before running the script, you'll need to configure some environment variables. Create a `.env` file with the following details:
|
||||||
|
|
||||||
|
```env
|
||||||
|
API_URL=your_api_url
|
||||||
|
AGOL_USER=your_arcgis_online_username
|
||||||
|
AGOL_PASSWORD=your_arcgis_online_password
|
||||||
|
HOSTNAME=your_arcgis_host
|
||||||
|
INSTANCE=your_arcgis_instance
|
||||||
|
FS=your_feature_service
|
||||||
|
LAYER=your_layer_id
|
||||||
|
```
|
||||||
|
|
||||||
|
### Variables
|
||||||
|
|
||||||
|
- **API_URL**: The URL of the API you are fetching data from.
|
||||||
|
- **AGOL_USER**: Your ArcGIS Online username.
|
||||||
|
- **AGOL_PASSWORD**: Your ArcGIS Online password.
|
||||||
|
- **HOSTNAME**: The hostname of your ArcGIS Online instance (e.g., `www.arcgis.com`).
|
||||||
|
- **INSTANCE**: The instance name of your ArcGIS Online service.
|
||||||
|
- **FS**: The name of the feature service you are working with.
|
||||||
|
- **LAYER**: The ID or name of the layer to truncate and add features to.
|
||||||
|
|
||||||
|
## Script Usage
|
||||||
|
|
||||||
|
You can run the script with the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python 753DataSync.py --results_per_page <number_of_results_per_page>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Arguments
|
||||||
|
|
||||||
|
- **--results_per_page** (optional): The number of results to fetch per page (default: 100).
|
||||||
|
|
||||||
|
## Functionality
|
||||||
|
|
||||||
|
1. **Truncate Layer**: Before fetching and adding any new data, the script will call the `truncate` function to clear out any existing features from the specified layer. This ensures that the feature layer is empty and ready for the new data.
|
||||||
|
|
||||||
|
2. **Fetch Data**: The script will then fetch data from the specified API in pages. Each page is fetched sequentially until all data is retrieved.
|
||||||
|
|
||||||
|
3. **Save Data**: Data from each page will be saved to an individual JSON file, with the filename including the page number and timestamp. The aggregated data (all pages combined) is saved to a separate file.
|
||||||
|
|
||||||
|
4. **Add Features**: After all the data has been fetched and saved, the script will send the aggregated data as features to the specified ArcGIS feature layer.
|
||||||
|
|
||||||
|
### Example Output
|
||||||
|
|
||||||
|
- Individual page files are saved in the `data/` directory with filenames like `enforcement_page_1_results_100_2025-03-26_14-30-45.json`.
|
||||||
|
- The aggregated file is saved as `aggregated_enforcement_results_2025-03-26_14-30-45.json`.
|
||||||
|
|
||||||
|
Logs will also be generated in the `753DataSync.log` file and printed to the console.
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
- If an error occurs while fetching data, the script will log the error and stop execution.
|
||||||
|
- If the `truncate` or `add_features` operations fail, the script will log the error and stop execution.
|
||||||
|
- The script handles HTTP errors and network-related errors gracefully.
|
||||||
|
|
||||||
|
## Example Output (Log)
|
||||||
|
|
||||||
|
```
|
||||||
|
2025-03-26 14:30:45 - INFO - Attempting to truncate layer on https://www.arcgis.com/...
|
||||||
|
2025-03-26 14:30:50 - INFO - Successfully truncated layer: https://www.arcgis.com/...
|
||||||
|
2025-03-26 14:30:51 - INFO - Making request to: https://api.example.com/1/100
|
||||||
|
2025-03-26 14:30:55 - INFO - Data saved to data/enforcement_page_1_results_100_2025-03-26_14-30-45.json
|
||||||
|
2025-03-26 14:30:56 - INFO - No more data to fetch, stopping pagination.
|
||||||
|
2025-03-26 14:30:57 - INFO - Data saved to data/aggregated_enforcement_results_2025-03-26_14-30-45.json
|
||||||
|
2025-03-26 14:31:00 - INFO - Features added successfully.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- If the script stops unexpectedly, check the logs (`753DataSync.log`) for detailed error information.
|
||||||
|
- Ensure your `.env` file is correctly configured with valid credentials and API URL.
|
||||||
|
- Make sure the specified ArcGIS layer is accessible and has the correct permissions for truncating and adding features.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the **GNU General Public License v3.0** or later - see the [LICENSE](LICENSE) file for details.
|
||||||
Loading…
Reference in New Issue
Block a user