Compare commits

...

9 Commits

Author SHA1 Message Date
Nick Heppler
072358aae9 Update README: Added information about Windows executable compiled with PyInstaller. 2025-05-22 12:44:59 -04:00
b58653f3f7 Merge pull request 'Add --reload flag to data synchronization script' (#12) from feature/add-reload-flag-for-loading-data into master
Reviewed-on: https://git.nickhepler.cloud/nick/753-Data-Sync/pulls/12
2025-05-22 12:27:44 -04:00
Nick Heppler
3afe292b23 fix: Ensure feature layer is truncated at the start of script execution
- Moved the call to the truncate function to always execute at the beginning of the main function.
- Ensured that the feature layer is cleared before loading data from a JSON file or fetching from the API.
2025-05-22 12:12:01 -04:00
Nick Heppler
cad8d64c23 feat: Add --reload flag to data synchronization script for loading JSON data
- Introduced a new command-line argument `--reload` to allow loading data from a specified JSON file.
- Updated `parse_arguments` to include the `--reload` flag.
- Modified `main` function to truncate the feature layer and load data from the JSON file when the flag is used.
- Enhanced documentation to reflect the new functionality and usage examples.
2025-05-22 11:25:41 -04:00
b48cade0df Merge pull request 'Add --test Flag to Fetch Only the First Page of Results' (#10) from feature/add-test-flag-for-first-page-results into master
Reviewed-on: https://git.nickhepler.cloud/nick/753-Data-Sync/pulls/10
2025-05-22 10:51:20 -04:00
Nick Heppler
5be8659264 docs: Update README to include --test flag functionality
- Added documentation for the new `--test` command-line argument.
- Updated the "Script Usage" section to reflect the new flag.
- Enhanced the "Functionality" section to mention the test mode feature.
- Included examples of how to use the `--test` flag in the "Usage Examples" section.
2025-05-22 10:32:35 -04:00
Nick Heppler
b7ca534211 feat: Add --test flag to fetch only the first page of results
- Introduced a new command-line argument `--test` to limit data fetching to the first page.
- Updated `parse_arguments` function to include the `--test` flag.
- Modified the pagination logic in the `main` function to break after the first page when `--test` is enabled.
- Enhanced documentation to reflect the new functionality and usage examples.
2025-05-22 10:27:36 -04:00
364e365e9c Merge pull request 'Add Configurable Log and Data Retention' (#8) from feature/logfile-purge-support into master
Reviewed-on: https://git.nickhepler.cloud/nick/753-Data-Sync/pulls/8
2025-04-15 18:18:15 -04:00
55816054fc Merge pull request 'Add Elapsed Time Logging to 753 Data Sync Script' (#7) from enhancement/log-execution-time into master
Reviewed-on: https://git.nickhepler.cloud/nick/753-Data-Sync/pulls/7
2025-04-15 17:31:46 -04:00
2 changed files with 57 additions and 8 deletions

View File

@ -12,20 +12,22 @@
## 🚀 Overview
This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. It also logs the operation, saves data to JSON files, and optionally purges old files.
This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. It also logs the operation, saves data to JSON files, and optionally purges old files. Additionally, it supports reloading data from a JSON file without making API calls.
---
## 📦 Requirements
- Python 3.6 or higher
- Python 3.6 or higher (if using the Python script)
- Required packages in `requirements.txt`
- `.env` file with your configuration
- ArcGIS Online credentials
---
## 🔧 Installation
## ⚙️ Installation
### Python Script
```bash
pip install -r requirements.txt
@ -37,6 +39,10 @@ Or install packages individually:
pip install requests python-dotenv
```
### Windows Executable
A Windows executable is available for users who prefer not to run the script directly. You can download it from the [releases page](https://git.nickhepler.cloud/nick/753-Data-Sync/releases). This executable is compiled using PyInstaller and can be run without needing to install Python or any dependencies.
---
## ⚙️ Configuration
@ -78,15 +84,27 @@ PURGE_DAYS=5
## 🧪 Script Usage
### Python Script
```bash
python 753DataSync.py --results_per_page 100
```
### Windows Executable
Simply double-click the executable file to run it. You can also run it from the command line with:
```bash
753DataSync.exe --results_per_page 100
```
### CLI Arguments
| Argument | Description |
|----------------------|---------------------------------------------|
| `--results_per_page` | Optional. Number of results per API call (default: `100`) |
| `--test` | Optional. If set, only fetch the first page of results. |
| `--reload` | Optional. Load data from a specified JSON file instead of fetching from the API. |
---
@ -99,6 +117,8 @@ python 753DataSync.py --results_per_page 100
5. **📤 Add Features** — Sends data to ArcGIS feature layer.
6. **🧹 File Cleanup** — Deletes `.json`/`.log` files older than `PURGE_DAYS`.
7. **📑 Dynamic Logs** — Logs saved to `753DataSync_YYYY-MM-DD.log`.
8. **🧪 Test Mode** — Use the `--test` flag to fetch only the first page of results for testing purposes.
9. **🔄 Reload Data** — Use the `--reload` flag to truncate the feature layer and load data from a specified JSON file.
---
@ -112,7 +132,6 @@ python 753DataSync.py --results_per_page 100
📄 753DataSync_2025-03-26.log
```
---
## 📝 Example Log
@ -152,6 +171,12 @@ python 753DataSync.py
# Run with custom page size
python 753DataSync.py --results_per_page 50
# Run the Windows executable with default page size
753DataSync.exe
# Run the Windows executable with custom page size
753DataSync.exe --results_per_page 50
```
---

32
app.py
View File

@ -146,10 +146,16 @@ def parse_arguments():
# Add arguments for results per page
parser.add_argument('--results_per_page', type=int, default=100, help="Number of results per page (default: 100)")
# Add a test flag
parser.add_argument('--test', action='store_true', help="If set, only fetch the first page of results.")
# Add a reload flag
parser.add_argument('--reload', type=str, help="If set, load data from the specified file instead of fetching from the API.")
# Parse the arguments
args = parser.parse_args()
return args.results_per_page
return args.results_per_page, args.test, args.reload
def generate_token(username, password, url="https://www.arcgis.com/sharing/rest/generateToken"):
"""Generates an authentication token."""
@ -305,8 +311,8 @@ def main():
purge_old_files(purge_days)
# Parse command-line arguments
results_per_page = parse_arguments()
logger.info(f"Parsed arguments: results_per_page={results_per_page}")
results_per_page, test_mode, reload_file = parse_arguments()
logger.info(f"Parsed arguments: results_per_page={results_per_page}, test_mode={test_mode}, reload_file={reload_file}")
# Load environment variables
logger.info("Loading environment variables.")
@ -330,9 +336,22 @@ def main():
fs = os.getenv('FS')
layer = os.getenv('LAYER')
# Truncate the layer before adding new features
logger.info("Truncating the feature layer.")
truncate(token, hostname, instance, fs, layer)
# If --reload flag is set, load data from the specified file
if reload_file:
logger.info(f"Reloading data from file: {reload_file}")
# Load data from the specified file
with open(reload_file, 'r', encoding='utf-8') as f:
aggregated_data = json.load(f)
# Add the features to the feature layer
response = add_features(token, hostname, instance, fs, layer, aggregated_data)
logger.info("Data reloaded successfully from the specified file.")
return
all_data = []
page_number = 1
@ -356,6 +375,11 @@ def main():
logger.info("No more data to fetch, stopping pagination.")
break
# Break the loop if in test mode
if test_mode:
logger.info("Test mode is enabled, stopping after the first page.")
break
page_number += 1
except Exception as e:
logger.error(f"Error fetching or saving data for page {page_number}: {e}", exc_info=True)