Metadata-Version: 2.4
Name: succulent
Version: 0.6.0
Summary: A lightweight framework for collecting and processing data from HTTP POST requests
License-Expression: MIT
License-File: LICENSE
Keywords: data collection,data science,ESP32,sensor measurements,smart agriculture,smartwatch
Author: Iztok Fister Jr.
Author-email: iztok@iztok.dev
Requires-Python: >=3.10
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: Flask
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Hardware
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: flask (>=2.3.2,<4.0.0)
Requires-Dist: lxml (>=5.2.2,<7.0.0)
Requires-Dist: numpy (>=1.26.0,<3.0.0)
Requires-Dist: pandas (>=2.0.1,<3.0.0)
Requires-Dist: pyyaml (>=6.0,<7.0)
Project-URL: Bug Tracker, https://github.com/firefly-cpp/succulent/issues
Project-URL: Changelog, https://github.com/firefly-cpp/succulent/blob/main/CHANGELOG.md
Project-URL: Documentation, https://succulent.readthedocs.io/en/latest/
Project-URL: Homepage, https://github.com/firefly-cpp/succulent
Project-URL: Repository, https://github.com/firefly-cpp/succulent
Description-Content-Type: text/markdown

<p align="center">
  <img alt="logo" width="300" src=".github/images/logo.png">
</p>

<h1 align="center">
succulent
</h1>

<h2 align="center">
  A lightweight framework for collecting and processing data from HTTP POST requests
</h2>

<p align="center">
  <img alt="PyPI Version" src="https://img.shields.io/pypi/v/succulent.svg">
  <img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/succulent.svg">
  <img alt="PyPI - Downloads" src="https://img.shields.io/pypi/dm/succulent.svg">
  <a href="https://aur.archlinux.org/packages/python-succulent">
    <img alt="AUR package" src="https://img.shields.io/aur/version/python-succulent?color=blue&label=Arch%20Linux&logo=arch-linux">
  </a>
  <a href="https://src.fedoraproject.org/rpms/python-succulent">
    <img alt="Fedora package" src="https://img.shields.io/fedora/v/python3-succulent?color=blue&label=Fedora%20Linux&logo=fedora">
  </a>
  <a href="https://pepy.tech/project/succulent">
    <img alt="Downloads" src="https://static.pepy.tech/badge/succulent">
  </a>
  <a href="https://repology.org/project/python:succulent/versions">
    <img alt="Packaging status" src="https://repology.org/badge/tiny-repos/python:succulent.svg">
  </a>
  <a href="https://github.com/firefly-cpp/succulent/blob/master/LICENSE">
    <img alt="GitHub license" src="https://img.shields.io/github/license/firefly-cpp/succulent.svg">
  </a>
  <a href="https://github.com/firefly-cpp/succulent/actions/workflows/test.yml">
    <img alt="Build" src="https://github.com/firefly-cpp/succulent/actions/workflows/test.yml/badge.svg">
  </a>
  <a href="https://succulent.readthedocs.io/en/latest/?badge=latest">
    <img alt="Documentation status" src="https://readthedocs.org/projects/succulent/badge/?version=latest">
  </a>
</p>

<p align="center">
  <img alt="GitHub repo size" src="https://img.shields.io/github/repo-size/firefly-cpp/succulent">
  <img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/w/firefly-cpp/succulent.svg">
  <a href="http://isitmaintained.com/project/firefly-cpp/succulent">
    <img alt="Average time to resolve an issue" src="http://isitmaintained.com/badge/resolution/firefly-cpp/succulent.svg">
  </a>
  <a href="http://isitmaintained.com/project/firefly-cpp/succulent">
    <img alt="Percentage of issues still open" src="http://isitmaintained.com/badge/open/firefly-cpp/succulent.svg">
  </a>
  <a href="#-contributors">
    <img alt="All Contributors" src="https://img.shields.io/badge/all_contributors-5-orange.svg">
  </a>
</p>

<p align="center">
  <a href="https://doi.org/10.5281/zenodo.10402365">
    <img alt="DOI" src="https://zenodo.org/badge/DOI/10.5281/zenodo.10402365.svg">
  </a>
</p>

<p align="center">
  <a href="#-detailed-insights">🔍 Detailed Insights</a> •
  <a href="#-installation">📦 Installation</a> •
  <a href="#-container">🐳 Container</a> •
  <a href="#-usage">🚀 Usage</a> •
  <a href="#-configuration">🔧 Configuration</a> •
  <a href="#-cite-us">📄 Cite us</a> •
  <a href="#-license">🔑 License</a> •
  <a href="#-contributors">🫂 Contributors</a>
</p>

Do you ever find it challenging and tricky to send sensor measurements 📏, data 📊, or GPS positions from embedded devices 📱, microcontrollers, and [smartwatches](https://github.com/firefly-cpp/AST-Monitor) to a central server? 📡 Setting up the primary data collection scripts can be a time-consuming ⏳ process, involving selecting a protocol, framework, API, and testing them out. Moreover, these scripts are often tailored for specific tasks, making them difficult to adapt to different scenarios.

But fear not! Introducing succulent 🌵, a pure Python framework that simplifies the configuration, management, collection, and preprocessing of data collected via POST requests. This framework draws inspiration from real-world data collection challenges in [smart agriculture](https://github.com/firefly-cpp/smart-agriculture-datasets/tree/main/plant-monitoring-esp32) 🧠🌿, specifically plant monitoring using ESP32 devices. The main goal behind succulent is to streamline the process of configuring various data parameters and provide a range of useful functions for data transformations. By leveraging succulent, you can set up your entire data collection endpoint within minutes, freeing you from the hassle of dealing with server-side scripts. 🚀🔧

* **Free software:** MIT license
* **Documentation:** [https://succulent.readthedocs.io/en/latest](https://succulent.readthedocs.io/en/latest/)
* **Python versions:** 3.8.x, 3.9.x, 3.10.x, 3.11.x, 3.12.x, 3.13.x
* **Tested OS:** Windows, Ubuntu, Fedora, Alpine, Arch, macOS. **However, that does not mean it does not work on others**

## 🔍 Detailed Insights

The current version of succulent comes packed with exciting features, including, but not limited to:

- **Hassle-free generation of request URLs** for seamless data collection 🌐
- **Effortless data retrieval** from POST requests 📥
- **Versatile data storage options**, such as CSV, JSON, SQLite, XML, and even images 🗂️📊🖼️
- **Customisable boundaries for collected data**, allowing you to set minimum and maximum thresholds ⚙️
- **Web dashboard for exploring the data** with charts, data quality and image gallery (optional) 📊

With succulent, the process of collecting, managing, and preprocessing data becomes a breeze, empowering you to focus on what truly matters—gaining valuable insights from your embedded devices, microcontrollers, and smartwatches. ⌚ So why waste precious time? ⏳ Dive into the world of succulent and unlock the true potential of your data! 💪📈

## 📦 Installation

### pip

To install `succulent` with pip, use:

```sh
pip install succulent
```

### Alpine Linux

To install `succulent` on Alpine Linux, use:

```sh
$ apk add py3-succulent
```

### Arch Linux

To install `succulent` on Arch Linux, use an [AUR helper](https://wiki.archlinux.org/title/AUR_helpers):

```sh
$ yay -Syyu python-succulent
```

### Fedora Linux

To install `succulent` on Fedora, use:

```sh
$ dnf install python3-succulent
```

## 🐳 Container
Create a `docker-compose.yml` file with the following content in the root directory:

```yml
version: '3.8'

services:
  app:
    image: codeberg.org/firefly-cpp/succulent:v6
    ports:
      - "8080:8080"
    volumes:
      - ./run.py:/succulent-app/run.py
      - ./configuration.yml:/succulent-app/configuration.yml
    environment:
      - GUNICORN_WORKERS=2
```

Next create a `configuration.yml` file in the root directory. Here's an example of a configuration file:

```yml
data:
  - name: 'temperature'
  - name: 'humidity'
  - name: 'light'
results:
  - enable: true
  - export: true
timestamp: true
```

More information regarding the configuration file and its settings can be found in the [configuration](#-configuration) section.

Then create a Python file named `run.py` with the following content in the root directory:

```python
from succulent.api import SucculentAPI

api = SucculentAPI(config='configuration.yml', format='csv')

# Flask app instance, called by gunicorn
app = api.app
```

Once you have set up the configuration file and the Python file, build the Docker image with the following command:

```bash
docker compose build
```

Finally, run the Docker container with the following command:

```bash
docker compose up
```

## 🚀 Usage

### Example

```python
from succulent.api import SucculentAPI
api = SucculentAPI(host='0.0.0.0', port=8080, config='configuration.yml', format='csv')
api.start()
```

## 🔧 Configuration
### Data collection
In the root directory, create a `configuration.yml` file and define the following:
```yml
data:
  - name: # Measure name
    min:  # Minimum value (optional)
    max:  # Maximum value (optional)
```

Measurements are stored in the `data` folder next to your script (e.g. `data/data.csv`). Each measurement is appended to the file (also for `json` and `xml`, where it is inserted before the closing `]` or `</data>`), so storing it takes the same time however much data is already stored, which keeps succulent fast on limited hardware such as a Raspberry Pi. The files keep the same format as written by earlier versions. Measurements that arrive at the same time, also from several server processes (e.g. gunicorn workers), are all stored. A small lock file (e.g. `data/.data.csv.lock`) is used for this; it can be ignored.

To collect images, create a `configuration.yml` file in the root directory and define the following:
```yml
data:
  - key: # Key in POST request
    allowed_extensions: ['jpg', 'png'] # Optional, see below
```

Only images are accepted. By default, the allowed extensions are `jpg`, `jpeg`, `png`, `gif`, `bmp`, `webp`, `tif` and `tiff`; use `allowed_extensions` to narrow or extend this list. For these image types, the content of the file is checked as well, so a file renamed to `.jpg` that is not a JPEG image is rejected. Other extensions you allow (e.g. `heic`) are checked by extension only. A file without an extension (e.g. from an ESP32-CAM) is accepted if its content is an allowed image, and the matching extension is added. Rejected uploads return a `400` response with the reason. Images are stored as `<timestamp>_<filename>`; an image with the same name in the same second gets a counter (e.g. `_1`) instead of overwriting the earlier one.

To store data collection timestamps, define the following setting in the `configuration.yml` file in the root directory:
```yml
timestamp: true # false by default
```

To access the URL for data collection, send a GET request (or navigate) to [http://localhost:8080/measure](http://localhost:8080/measure).

To store data, send a POST request to `/measure` with the values in one of the following ways:

- query parameters, e.g. `POST /measure?temperature=21.5&humidity=40` (works with every format),
- a JSON object body, e.g. `{"temperature": 21.5, "humidity": 40}` (works with every format),
- an XML body (with `format='xml'`), where each child of the root element is one value; the name of the root element is not checked:
  ```sh
  curl -X POST http://localhost:8080/measure -H "Content-Type: application/xml" -d '<data><temperature>21.5</temperature><humidity>40</humidity></data>'
  ```

A malformed XML body or a JSON body that is not an object (or not valid JSON) is rejected with a `400` response. All errors are returned as JSON with a `message` field, e.g. `{"message": "Invalid JSON: the request body could not be parsed."}`. With `format='xml'`, column names must be valid XML element names: letters, digits, `_`, `-` or `.`, starting with a letter or `_`, without spaces, and not starting with `xml`.

To restrict access to the collected data, define the following setting in the `configuration.yml` file in the root directory:
```yml
password: 'password' # Password for data access
```

The password can be sent with any request (`/measure`, `/data` and `/export`) in one of the following ways:

- `Authorization: Bearer password` header (recommended),
- `X-API-Key: password` header,
- `"password": "password"` field in a JSON body or `<password>password</password>` element in an XML body (`/measure` only),
- `?password=password` query parameter (supported for backward compatibility).

Prefer one of the headers, since URLs (including query parameters) are often written to server and proxy logs. For example:
```sh
curl -X POST http://localhost:8080/measure -H "Authorization: Bearer password" -H "Content-Type: application/json" -d '{"temperature": 21.5}'
```

### Data access
To access data via the Succulent API, define the following setting in the `configuration.yml` file in the root directory:
```yml
results:
  - enable: true # false by default
```

To access the collected data, send a GET request (or navigate) to [http://localhost:8080/data](http://localhost:8080/data). To access password-protected data, send the password as described above (e.g. the `Authorization: Bearer password` header).

### Data export
To export the data, enable the export option in the configuration file:
```yml
results:
  - export: true # false by default
```

To export the data, send a GET request (or navigate) to [http://localhost:8080/export](http://localhost:8080/export). To export password-protected data, send the password as described above. The data is downloaded as a CSV file (`data.csv`), whichever storage format (`csv`, `json`, `sqlite` or `xml`) you use. Exporting is not available for the `image` format.

### Data exploration
succulent includes an optional web dashboard for monitoring the collected data in a browser. To enable it, define the following setting in the `configuration.yml` file in the root directory:
```yml
explore:
  - enable: true # false by default
  - refresh: 10  # auto-refresh interval in seconds (optional, 0 disables it)
```

Then navigate to [http://localhost:8080/explore](http://localhost:8080/explore). The dashboard shows:

- the latest value of each column, with its position between `min` and `max`,
- data quality: the number of measurements, missing values, values outside `min`/`max` and the typical interval between measurements,
- a chart for each numeric column, with `min` and `max` as dashed lines (long ranges are averaged so the page stays fast),
- the latest measurements in a table,
- for the `image` format, a gallery of the uploaded images.

With `timestamp: true`, you can choose the time range (1 h, 24 h, 7 d, 30 d or all). If export is enabled, the data can be downloaded as CSV from the dashboard.

<p align="center"><img alt="succulent explore dashboard" width="800" src=".github/images/explore.png"></p>

If a password is set, the dashboard asks for it once and keeps the session in a cookie, so the password never appears in the URL. Sessions stay valid across restarts and in every server process (e.g. several gunicorn workers); changing the password ends them. To choose the key for the session cookie yourself, set a `secret_key` in the configuration. Scripts can read `/explore/api/data` with the `Authorization: Bearer password` or `X-API-Key: password` header instead.

The dashboard only reads the stored data; data collection works exactly as without it. It needs no internet access (no external scripts or fonts), so it also works on a local network, and values sent by devices are always shown as text.

## 📄 Cite us
Fister Jr., Iztok, and Tadej Lahovnik. Succulent. 0.4.0, doi:[10.5281/zenodo.10402365](https://doi.org/10.5281/zenodo.10402365).

## 🔑 License

This package is distributed under the MIT License. This license can be found online at <http://www.opensource.org/licenses/MIT>.

## Disclaimer

This framework is provided as-is, and there are no guarantees that it fits your purposes or that it is bug-free. Use it at your own risk!

## 🫂 Contributors

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<table>
  <tbody>
    <tr>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/lahovniktadej"><img src="https://avatars.githubusercontent.com/u/57890734?v=4?s=100" width="100px;" alt="Tadej Lahovnik"/><br /><sub><b>Tadej Lahovnik</b></sub></a><br /><a href="https://github.com/firefly-cpp/succulent/commits?author=lahovniktadej" title="Code">💻</a> <a href="https://github.com/firefly-cpp/succulent/issues?q=author%3Alahovniktadej" title="Bug reports">🐛</a> <a href="#ideas-lahovniktadej" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/firefly-cpp/succulent/commits?author=lahovniktadej" title="Documentation">📖</a> <a href="#tutorial-lahovniktadej" title="Tutorials">✅</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/AyanDas348"><img src="https://avatars.githubusercontent.com/u/53610626?v=4?s=100" width="100px;" alt="Ayan Das"/><br /><sub><b>Ayan Das</b></sub></a><br /><a href="https://github.com/firefly-cpp/succulent/commits?author=AyanDas348" title="Code">💻</a> <a href="https://github.com/firefly-cpp/succulent/commits?author=AyanDas348" title="Tests">⚠️</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://www.iztok-jr-fister.eu/"><img src="https://avatars.githubusercontent.com/u/1633361?v=4?s=100" width="100px;" alt="Iztok Fister Jr."/><br /><sub><b>Iztok Fister Jr.</b></sub></a><br /><a href="https://github.com/firefly-cpp/succulent/commits?author=firefly-cpp" title="Code">💻</a> <a href="#ideas-firefly-cpp" title="Ideas, Planning, & Feedback">🤔</a> <a href="#mentoring-firefly-cpp" title="Mentoring">🧑‍🏫</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://carlosal1015.github.io"><img src="https://avatars.githubusercontent.com/u/21283014?v=4?s=100" width="100px;" alt="Oromion"/><br /><sub><b>Oromion</b></sub></a><br /><a href="https://github.com/firefly-cpp/succulent/issues?q=author%3Acarlosal1015" title="Bug reports">🐛</a> <a href="#platform-carlosal1015" title="Packaging/porting to new platform">📦</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/rhododendrom"><img src="https://avatars.githubusercontent.com/u/3198785?v=4?s=100" width="100px;" alt="rhododendrom"/><br /><sub><b>rhododendrom</b></sub></a><br /><a href="#design-rhododendrom" title="Design">🎨</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/zala-lahovnik"><img src="https://avatars.githubusercontent.com/u/105444201?v=4?s=100" width="100px;" alt="Zala Lahovnik"/><br /><sub><b>Zala Lahovnik</b></sub></a><br /><a href="https://github.com/firefly-cpp/succulent/commits?author=zala-lahovnik" title="Documentation">📖</a></td>
    </tr>
  </tbody>
</table>

<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->

<!-- ALL-CONTRIBUTORS-LIST:END -->

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!

