nick/753-Data-Sync
1
0
Fork 0
Code Issues 4 Pull Requests Releases 2 Activity

Merge pull request 'Add --test Flag to Fetch Only the First Page of Results' (#10) from feature/add-test-flag-for-first-page-results into master

Browse Source 
Reviewed-on: https://git.nickhepler.cloud/nick/753-Data-Sync/pulls/10
This commit is contained in:
Nick Hepler 2025-05-22 10:51:20 -04:00
commit b48cade0df
2 changed files with 13 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -87,6 +87,7 @@ python 753DataSync.py --results_per_page 100
| Argument | Description | | Argument | Description |
|----------------------|---------------------------------------------| |----------------------|---------------------------------------------|
| `--results_per_page` | Optional. Number of results per API call (default: `100`) | | `--results_per_page` | Optional. Number of results per API call (default: `100`) |
| `--test` | Optional. If set, only fetch the first page of results. |
--- ---
@ -99,6 +100,7 @@ python 753DataSync.py --results_per_page 100
5. **📤 Add Features** — Sends data to ArcGIS feature layer. 5. **📤 Add Features** — Sends data to ArcGIS feature layer.
6. **🧹 File Cleanup** — Deletes `.json`/`.log` files older than `PURGE_DAYS`. 6. **🧹 File Cleanup** — Deletes `.json`/`.log` files older than `PURGE_DAYS`.
7. **📑 Dynamic Logs** — Logs saved to `753DataSync_YYYY-MM-DD.log`. 7. **📑 Dynamic Logs** — Logs saved to `753DataSync_YYYY-MM-DD.log`.
8. **🧪 Test Mode** — Use the `--test` flag to fetch only the first page of results for testing purposes.
--- ---

14
app.py
View File

@ -146,10 +146,13 @@ def parse_arguments():
# Add arguments for results per page # Add arguments for results per page
parser.add_argument('--results_per_page', type=int, default=100, help="Number of results per page (default: 100)") parser.add_argument('--results_per_page', type=int, default=100, help="Number of results per page (default: 100)")
# Add a test flag
parser.add_argument('--test', action='store_true', help="If set, only fetch the first page of results.")
# Parse the arguments # Parse the arguments
args = parser.parse_args() args = parser.parse_args()
return args.results_per_page return args.results_per_page, args.test
def generate_token(username, password, url="https://www.arcgis.com/sharing/rest/generateToken"): def generate_token(username, password, url="https://www.arcgis.com/sharing/rest/generateToken"):
"""Generates an authentication token.""" """Generates an authentication token."""
@ -305,8 +308,8 @@ def main():
purge_old_files(purge_days) purge_old_files(purge_days)
# Parse command-line arguments # Parse command-line arguments
results_per_page = parse_arguments() results_per_page, test_mode = parse_arguments()
logger.info(f"Parsed arguments: results_per_page={results_per_page}") logger.info(f"Parsed arguments: results_per_page={results_per_page}, test_mode={test_mode}")
# Load environment variables # Load environment variables
logger.info("Loading environment variables.") logger.info("Loading environment variables.")
@ -356,6 +359,11 @@ def main():
logger.info("No more data to fetch, stopping pagination.") logger.info("No more data to fetch, stopping pagination.")
break break
# Break the loop if in test mode
if test_mode:
logger.info("Test mode is enabled, stopping after the first page.")
break
page_number += 1 page_number += 1
except Exception as e: except Exception as e:
logger.error(f"Error fetching or saving data for page {page_number}: {e}", exc_info=True) logger.error(f"Error fetching or saving data for page {page_number}: {e}", exc_info=True)