From da4ef4dc550d7c398538b640161f6505970b5449 Mon Sep 17 00:00:00 2001 From: Nick Heppler Date: Tue, 1 Apr 2025 19:19:30 -0400 Subject: [PATCH] Update README --- README.md | 65 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index a99519c..61ce81a 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. The script performs the following tasks: -1. **Truncate the specified layer** in ArcGIS to clear any previous features before adding new ones. -2. **Fetch data** from an API in paginated form. -3. **Save data** from each API response to individual JSON files. -4. **Aggregate all data** from all pages into one JSON file. -5. **Add the aggregated data** as features to an ArcGIS feature service. +- **Truncate** the specified layer in ArcGIS to clear any previous features before adding new ones. +- **Fetch** data from an API in paginated form. +- **Save** data from each API response to individual JSON files. +- **Aggregate** all data from all pages into one JSON file. +- **Add** the aggregated data as features to an ArcGIS feature service. ## Requirements @@ -15,17 +15,24 @@ This script fetches enforcement data from an external API, truncates a specified - ArcGIS Online credentials (username and password) - `.env` file for configuration (see below for details) -### Install dependencies +## Install Dependencies -You can install the required dependencies using `pip`: +To install the required dependencies, use the following command: ```bash pip install -r requirements.txt ``` +Alternatively, you can install the necessary packages individually: + +```bash +pip install requests +pip install python-dotenv +``` + ## Configuration -Before running the script, you'll need to configure some environment variables. Create a `.env` file with the following details: +Before running the script, you need to configure some environment variables. Create a `.env` file in the root of your project with the following details: ```env API_URL=your_api_url @@ -35,9 +42,10 @@ HOSTNAME=your_arcgis_host INSTANCE=your_arcgis_instance FS=your_feature_service LAYER=your_layer_id +LOG_LEVEL=your_log_level # e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL ``` -### Variables +### Environment Variables: - **API_URL**: The URL of the API you are fetching data from. - **AGOL_USER**: Your ArcGIS Online username. @@ -46,6 +54,7 @@ LAYER=your_layer_id - **INSTANCE**: The instance name of your ArcGIS Online service. - **FS**: The name of the feature service you are working with. - **LAYER**: The ID or name of the layer to truncate and add features to. +- **LOG_LEVEL**: The desired logging level (e.g., `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`). ## Script Usage @@ -55,36 +64,34 @@ You can run the script with the following command: python 753DataSync.py --results_per_page ``` -### Arguments +### Arguments: -- **--results_per_page** (optional): The number of results to fetch per page (default: 100). +- `--results_per_page` (optional): The number of results to fetch per page (default: 100). ## Functionality -1. **Truncate Layer**: Before fetching and adding any new data, the script will call the `truncate` function to clear out any existing features from the specified layer. This ensures that the feature layer is empty and ready for the new data. +### 1. **Truncate Layer**: +Before fetching and adding any new data, the script will call the `truncate` function to clear out any existing features from the specified layer. This ensures that the feature layer is empty and ready for the new data. -2. **Fetch Data**: The script will then fetch data from the specified API in pages. Each page is fetched sequentially until all data is retrieved. +### 2. **Fetch Data**: +The script will then fetch data from the specified API in pages. Each page is fetched sequentially until all data is retrieved. -3. **Save Data**: Data from each page will be saved to an individual JSON file, with the filename including the page number and timestamp. The aggregated data (all pages combined) is saved to a separate file. +### 3. **Save Data**: +Data from each page will be saved to an individual JSON file, with the filename including the page number and timestamp. The aggregated data (all pages combined) is saved to a separate file. -4. **Add Features**: After all the data has been fetched and saved, the script will send the aggregated data as features to the specified ArcGIS feature layer. +### 4. **Add Features**: +After all the data has been fetched and saved, the script will send the aggregated data as features to the specified ArcGIS feature layer. -### Example Output +## Example Output - Individual page files are saved in the `data/` directory with filenames like `enforcement_page_1_results_100_2025-03-26_14-30-45.json`. - The aggregated file is saved as `aggregated_enforcement_results_2025-03-26_14-30-45.json`. - + Logs will also be generated in the `753DataSync.log` file and printed to the console. -## Error Handling - -- If an error occurs while fetching data, the script will log the error and stop execution. -- If the `truncate` or `add_features` operations fail, the script will log the error and stop execution. -- The script handles HTTP errors and network-related errors gracefully. - ## Example Output (Log) -``` +```text 2025-03-26 14:30:45 - INFO - Attempting to truncate layer on https://www.arcgis.com/... 2025-03-26 14:30:50 - INFO - Successfully truncated layer: https://www.arcgis.com/... 2025-03-26 14:30:51 - INFO - Making request to: https://api.example.com/1/100 @@ -94,6 +101,14 @@ Logs will also be generated in the `753DataSync.log` file and printed to the con 2025-03-26 14:31:00 - INFO - Features added successfully. ``` +## Error Handling + +The script handles errors gracefully, including: + +- If an error occurs while fetching data, the script will log the error and stop execution. +- If the `truncate` or `add_features` operations fail, the script will log the error and stop execution. +- The script handles HTTP errors and network-related errors gracefully, ensuring that any issues are logged with detailed information. + ## Troubleshooting - If the script stops unexpectedly, check the logs (`753DataSync.log`) for detailed error information. @@ -102,4 +117,4 @@ Logs will also be generated in the `753DataSync.log` file and printed to the con ## License -This project is licensed under the **GNU General Public License v3.0** or later - see the [LICENSE](LICENSE) file for details. \ No newline at end of file +This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE](LICENSE) file for details. \ No newline at end of file