Add --reload flag to data synchronization script #12

Merged
nick merged 2 commits from feature/add-reload-flag-for-loading-data into master 2025-05-22 12:27:44 -04:00
2 changed files with 22 additions and 7 deletions
Showing only changes of commit cad8d64c23 - Show all commits

View File

@ -12,7 +12,7 @@
## 🚀 Overview
This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. It also logs the operation, saves data to JSON files, and optionally purges old files.
This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. It also logs the operation, saves data to JSON files, and optionally purges old files. Additionally, it supports reloading data from a JSON file without making API calls.
---
@ -88,6 +88,7 @@ python 753DataSync.py --results_per_page 100
|----------------------|---------------------------------------------|
| `--results_per_page` | Optional. Number of results per API call (default: `100`) |
| `--test` | Optional. If set, only fetch the first page of results. |
| `--reload` | Optional. Load data from a specified JSON file instead of fetching from the API. |
---
@ -101,6 +102,7 @@ python 753DataSync.py --results_per_page 100
6. **🧹 File Cleanup** — Deletes `.json`/`.log` files older than `PURGE_DAYS`.
7. **📑 Dynamic Logs** — Logs saved to `753DataSync_YYYY-MM-DD.log`.
8. **🧪 Test Mode** — Use the `--test` flag to fetch only the first page of results for testing purposes.
9. **🔄 Reload Data** — Use the `--reload` flag to truncate the feature layer and load data from a specified JSON file.
---
@ -114,7 +116,6 @@ python 753DataSync.py --results_per_page 100
📄 753DataSync_2025-03-26.log
```
---
## 📝 Example Log

24
app.py
View File

@ -149,10 +149,13 @@ def parse_arguments():
# Add a test flag
parser.add_argument('--test', action='store_true', help="If set, only fetch the first page of results.")
# Add a reload flag
parser.add_argument('--reload', type=str, help="If set, load data from the specified file instead of fetching from the API.")
# Parse the arguments
args = parser.parse_args()
return args.results_per_page, args.test
return args.results_per_page, args.test, args.reload
def generate_token(username, password, url="https://www.arcgis.com/sharing/rest/generateToken"):
"""Generates an authentication token."""
@ -308,8 +311,8 @@ def main():
purge_old_files(purge_days)
# Parse command-line arguments
results_per_page, test_mode = parse_arguments()
logger.info(f"Parsed arguments: results_per_page={results_per_page}, test_mode={test_mode}")
results_per_page, test_mode, reload_file = parse_arguments()
logger.info(f"Parsed arguments: results_per_page={results_per_page}, test_mode={test_mode}, reload_file={reload_file}")
# Load environment variables
logger.info("Loading environment variables.")
@ -333,8 +336,19 @@ def main():
fs = os.getenv('FS')
layer = os.getenv('LAYER')
# Truncate the layer before adding new features
truncate(token, hostname, instance, fs, layer)
# If --reload flag is set, load data from the specified file
if reload_file:
logger.info(f"Reloading data from file: {reload_file}")
truncate(token, hostname, instance, fs, layer) # Truncate the layer
# Load data from the specified file
with open(reload_file, 'r', encoding='utf-8') as f:
aggregated_data = json.load(f)
# Add the features to the feature layer
response = add_features(token, hostname, instance, fs, layer, aggregated_data)
logger.info("Data reloaded successfully from the specified file.")
return
all_data = []
page_number = 1