nick/753-Data-Sync
1
0
Fork 0
Code Issues 4 Pull Requests Releases 2 Activity

feature/add-debug-logging #3

Merged
nick merged 5 commits from feature/add-debug-logging into master 2025-04-01 19:54:34 -04:00
Conversation 0 Commits 5 Files Changed 3 +40 -25
1 changed files with 40 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit da4ef4dc55 - Show all commits

63
README.md
View File

@ -2,11 +2,11 @@
This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. The script performs the following tasks:
1. **Truncate the specified layer** in ArcGIS to clear any previous features before adding new ones.
2. **Fetch data** from an API in paginated form.
3. **Save data** from each API response to individual JSON files.
4. **Aggregate all data** from all pages into one JSON file.
5. **Add the aggregated data** as features to an ArcGIS feature service.
- **Truncate** the specified layer in ArcGIS to clear any previous features before adding new ones.
- **Fetch** data from an API in paginated form.
- **Save** data from each API response to individual JSON files.
- **Aggregate** all data from all pages into one JSON file.
- **Add** the aggregated data as features to an ArcGIS feature service.
## Requirements
@ -15,17 +15,24 @@ This script fetches enforcement data from an external API, truncates a specified
- ArcGIS Online credentials (username and password)
- `.env` file for configuration (see below for details)
### Install dependencies
## Install Dependencies
You can install the required dependencies using `pip`:
To install the required dependencies, use the following command:
```bash
pip install -r requirements.txt
```
Alternatively, you can install the necessary packages individually:
```bash
pip install requests
pip install python-dotenv
```
## Configuration
Before running the script, you'll need to configure some environment variables. Create a `.env` file with the following details:
Before running the script, you need to configure some environment variables. Create a `.env` file in the root of your project with the following details:
```env
API_URL=your_api_url
@ -35,9 +42,10 @@ HOSTNAME=your_arcgis_host
INSTANCE=your_arcgis_instance
FS=your_feature_service
LAYER=your_layer_id
LOG_LEVEL=your_log_level # e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL
```
### Variables
### Environment Variables:
- **API_URL**: The URL of the API you are fetching data from.
- **AGOL_USER**: Your ArcGIS Online username.
@ -46,6 +54,7 @@ LAYER=your_layer_id
- **INSTANCE**: The instance name of your ArcGIS Online service.
- **FS**: The name of the feature service you are working with.
- **LAYER**: The ID or name of the layer to truncate and add features to.
- **LOG_LEVEL**: The desired logging level (e.g., `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`).
## Script Usage
@ -55,36 +64,34 @@ You can run the script with the following command:
python 753DataSync.py --results_per_page <number_of_results_per_page>
```
### Arguments
### Arguments:
- **--results_per_page** (optional): The number of results to fetch per page (default: 100).
- `--results_per_page` (optional): The number of results to fetch per page (default: 100).
## Functionality
1. **Truncate Layer**: Before fetching and adding any new data, the script will call the `truncate` function to clear out any existing features from the specified layer. This ensures that the feature layer is empty and ready for the new data.
### 1. **Truncate Layer**:
Before fetching and adding any new data, the script will call the `truncate` function to clear out any existing features from the specified layer. This ensures that the feature layer is empty and ready for the new data.
2. **Fetch Data**: The script will then fetch data from the specified API in pages. Each page is fetched sequentially until all data is retrieved.
### 2. **Fetch Data**:
The script will then fetch data from the specified API in pages. Each page is fetched sequentially until all data is retrieved.
3. **Save Data**: Data from each page will be saved to an individual JSON file, with the filename including the page number and timestamp. The aggregated data (all pages combined) is saved to a separate file.
### 3. **Save Data**:
Data from each page will be saved to an individual JSON file, with the filename including the page number and timestamp. The aggregated data (all pages combined) is saved to a separate file.
4. **Add Features**: After all the data has been fetched and saved, the script will send the aggregated data as features to the specified ArcGIS feature layer.
### 4. **Add Features**:
After all the data has been fetched and saved, the script will send the aggregated data as features to the specified ArcGIS feature layer.
### Example Output
## Example Output
- Individual page files are saved in the `data/` directory with filenames like `enforcement_page_1_results_100_2025-03-26_14-30-45.json`.
- The aggregated file is saved as `aggregated_enforcement_results_2025-03-26_14-30-45.json`.
Logs will also be generated in the `753DataSync.log` file and printed to the console.
## Error Handling
- If an error occurs while fetching data, the script will log the error and stop execution.
- If the `truncate` or `add_features` operations fail, the script will log the error and stop execution.
- The script handles HTTP errors and network-related errors gracefully.
## Example Output (Log)
```
```text
2025-03-26 14:30:45 - INFO - Attempting to truncate layer on https://www.arcgis.com/...
2025-03-26 14:30:50 - INFO - Successfully truncated layer: https://www.arcgis.com/...
2025-03-26 14:30:51 - INFO - Making request to: https://api.example.com/1/100
@ -94,6 +101,14 @@ Logs will also be generated in the `753DataSync.log` file and printed to the con
2025-03-26 14:31:00 - INFO - Features added successfully.
```
## Error Handling
The script handles errors gracefully, including:
- If an error occurs while fetching data, the script will log the error and stop execution.
- If the `truncate` or `add_features` operations fail, the script will log the error and stop execution.
- The script handles HTTP errors and network-related errors gracefully, ensuring that any issues are logged with detailed information.
## Troubleshooting
- If the script stops unexpectedly, check the logs (`753DataSync.log`) for detailed error information.
@ -102,4 +117,4 @@ Logs will also be generated in the `753DataSync.log` file and printed to the con
## License
This project is licensed under the **GNU General Public License v3.0** or later - see the [LICENSE](LICENSE) file for details.
This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE](LICENSE) file for details.