753 Data Sync

A Python-based data ingestion tool for syncing enforcement data from a public API to ArcGIS Online.

---

🚀 Overview

This script fetches enforcement data from an external API, truncates a specified feature layer in ArcGIS, and adds the fetched data as features to the layer. It also logs the operation, saves data to JSON files, and optionally purges old files.

---

📦 Requirements

• Python 3.6 or higher
• Required packages in requirements.txt
• .env file with your configuration
• ArcGIS Online credentials

---

🔧 Installation

pip install -r requirements.txt

Or install packages individually:

pip install requests python-dotenv

---

⚙️ Configuration

Create a .env file in the root of your project:

API_URL=https://example.com/api
AGOL_USER=your_username
AGOL_PASSWORD=your_password
HOSTNAME=www.arcgis.com
INSTANCE=your_instance
FS=your_feature_service
LAYER=0
LOG_LEVEL=DEBUG
PURGE_DAYS=5

Required Variables

Variable	Description
API_URL	The API endpoint to fetch data from
AGOL_USER	ArcGIS Online username
AGOL_PASSWORD	ArcGIS Online password
HOSTNAME	ArcGIS host (e.g., www.arcgis.com)
INSTANCE	ArcGIS REST instance path
FS	Feature service name
LAYER	Feature layer ID or name

Optional Variables

Variable	Description
LOG_LEVEL	Log level (DEBUG, INFO, etc.)
PURGE_DAYS	Number of days to retain logs and JSONs

---

🧪 Script Usage

python 753DataSync.py --results_per_page 100

CLI Arguments

Argument	Description
--results_per_page	Optional. Number of results per API call (default: 100)
--test	Optional. If set, only fetch the first page of results.

---

📋 Functionality

1. 🔁 Truncate Layer — Clears existing ArcGIS features.
2. 🌐 Fetch Data — Retrieves paginated data from the API.
3. 💾 Save Data — Writes each page to a time-stamped JSON file.
4. 📦 Aggregate Data — Combines all pages into one file.
5. 📤 Add Features — Sends data to ArcGIS feature layer.
6. 🧹 File Cleanup — Deletes .json/.log files older than PURGE_DAYS.
7. 📑 Dynamic Logs — Logs saved to 753DataSync_YYYY-MM-DD.log.
8. 🧪 Test Mode — Use the --test flag to fetch only the first page of results for testing purposes.

---

📁 Example Output

📁 data/
├── enforcement_page_1_results_100_2025-03-26_14-30-45.json
├── enforcement_page_2_results_100_2025-03-26_14-31-10.json
└── aggregated_enforcement_results_2025-03-26_14-31-15.json

📄 753DataSync_2025-03-26.log

---

📝 Example Log

2025-03-26 14:30:45 - INFO - Attempting to truncate layer...
2025-03-26 14:30:51 - INFO - Fetching page 1 from API...
2025-03-26 14:30:55 - INFO - Saved data to data/enforcement_page_1_results_100_...
2025-03-26 14:30:57 - INFO - Aggregated data saved.
2025-03-26 14:31:00 - INFO - Features added successfully.
2025-03-26 14:31:01 - INFO - Deleted old log: 753DataSync_2025-03-19.log

---

🛠 Troubleshooting

• Set LOG_LEVEL=DEBUG in .env for detailed logs.
• Ensure .env has no syntax errors.
• Make sure your ArcGIS layer has permission for truncation and writes.
• Check for internet/API access and expired ArcGIS tokens.
• Logs are written to both console and daily log files.

---

🧪 Testing

Currently, the script is tested manually. Automated testing may be added under a /tests folder in the future.

---

📖 Usage Examples

# Run with default page size
python 753DataSync.py

# Run with custom page size
python 753DataSync.py --results_per_page 50

---

💬 Support

Found a bug or want to request a feature?
Open an issue or contact @nick directly.

---

📜 License

This project is licensed under the GNU General Public License v3.0.

💡 You are free to use, modify, and share this project as long as you preserve the same license in your changes.