I Wanted to Keep Track of My App Reviews Without Expensive Tooling — So I Built My Own

TL;DR: I maintain a couple of apps on both stores. I kept losing track of what users were saying, and every tool I found either cost too much or did too many things I didn't need. So I wrote a CLI that pulls my reviews, classifies them with an LLM, and tells me what broke. It runs locally, costs me ~$0.50/month in API calls, and the code is on GitHub if any of this is useful to you.

---

What Was Bugging Me

I maintain a few apps on Google Play and the App Store. Nothing huge — just apps I've built over the years that have real users. Every few days I'd remember to open Play Console, scroll through reviews, and try to mentally sort them — "ok that's a bug, that's a feature request, that person is just frustrated, that one is a crash I already fixed three releases ago."

I kept missing stuff. A user would report something important, and I'd only see it two weeks later buried under a dozen "great app 5 stars" reviews. I knew what I actually needed:

• Fetch my reviews from both stores without me having to open two dashboards
• Fetch ratings so I'd notice a drop before it snowballs
• Figure out what's a bug and what's a feature request — because I don't have time to read every review
• Categorize them properly — not just sentiment, but actual categories: bug, crash, feature request, performance issue
• Correlate crashes if possible — when someone writes "app keeps closing," I want to know if there's a matching crash in Sentry
• Just alert me — a morning summary in Slack or my terminal. I don't need a dashboard. I need a nudge.

I Looked Around for a While

Before writing any code, I spent a fair amount of time looking for something that already existed. Surely someone had solved this.

I tried AppFollow, Appbot, AppTweak, and a few others. They range from $50 to $200+ a month. Some of them are genuinely good products — if you're a team with a PM who needs dashboards, competitor tracking, ASO tools, and review assignment workflows. But I'm one developer trying to keep tabs on my own apps. I don't need competitor intelligence. I don't need ASO. I just need to know what broke and what people are asking for.

The classification in most of these tools was also frustrating. They mostly use keyword matching, which means a review like "this used to crash all the time but the update fixed it" gets flagged as a crash report. That's noise, not signal.

I also tried just being more disciplined about checking Play Console and App Store Connect manually. That lasted about two weeks before I was back to checking once every few days and missing things.

What I really wanted didn't seem to exist: something lightweight, something I could run locally, something that actually understood what a review was saying — not just pattern-matching on keywords. And ideally something I wasn't paying a monthly subscription for.

So I figured I'd just build it.

What I Ended Up Building

I called it AppPulse. It's a Python CLI — no web app, no Docker, no hosted service. Just pip install and run it on your machine.

Here's what happens when I run apppulse run (or when cron triggers it at 7 AM):

1. Pulls new reviews from Google Play and/or the App Store via their official APIs
2. Classifies each review using an LLM — bug, feature request, crash report, performance complaint, praise, or general frustration
3. Pulls crash data from Sentry so I can see if user complaints match real crashes
4. Stores everything in a local SQLite database
5. Generates a digest — top bugs, top feature requests, crash health
6. Sends it to my terminal, Slack, or email

pip install -e .
apppulse setup    # guided wizard, ~5 minutes
apppulse run      # pull → classify → report

No servers, no cloud accounts. Everything stays in a SQLite file on my machine at ~/.apppulse/data.db.

How It Fits Together

           apppulse run (or daily at 7 AM)
                      │
         ┌────────────┼────────────┐
         ▼                  ▼                  ▼
       Google Play       App Store             Sentry
      Publisher API     Connect API           REST API
         │                   │                   │
         └────────────┼────────────┘
                      ▼
              ┌──────────────┐
              │  Raw Reviews │
              │ + Crash Data │
              └──────┬───────┘
                     ▼
              ┌──────────────┐
              │ LLM Classify │  ← GPT-4o-mini
              │ (batch of 10)│
              └──────┬───────┘
                     ▼
              ┌──────────────┐
              │  SQLite DB   │  ← ~/.apppulse/data.db
              └──────┬───────┘
                     ▼
           ┌─────────┼─────────┐
           ▼             ▼             ▼
        Terminal         Slack         Email

I kept each step as a separate module, mostly so I could swap things out as I went. If you want to add a different crash source or use a different LLM, you'd change one file.

The Part That Made It Worth Building

I could have just written a script to dump reviews into a CSV. But the whole point was that I didn't want to read every review myself. I wanted the tool to tell me: "these 4 are bugs, these 2 are feature requests, and the rest are just people saying thanks."

Keyword matching doesn't cut it. A review saying "the app never crashes anymore" is praise, not a crash report. So AppPulse sends reviews to an LLM in batches of 10 and gets back structured results:

# From src/apppulse/analyze/classifier.py
# Each review gets classified into:
{
    "category": "bug",           # bug, feature_request, crash, performance, praise, complaint
    "severity": "major",         # critical, major, minor, none
    "sentiment": "negative",     # positive, negative, neutral, mixed
    "summary": "App freezes when uploading photos larger than 10MB",
    "keywords": ["freeze", "upload", "photos"],
    "functional_area": "media"   # which part of your app is affected
}

The batch approach matters for cost. One API call handles 10 reviews instead of 10 separate calls. With GPT-4o-mini, my apps generate maybe 1,000-2,000 reviews a month, and the LLM bill comes to around $0.50. That's the entire monthly cost of running this.

If you're not comfortable sending review text to OpenAI or Anthropic, there's an Ollama option — run Llama 3.2 locally and classification is completely free.

The Setup (Not as Bad as It Sounds)

You need Python 3.10+, API credentials for the stores (a Google Cloud Service Account for Play, a .p8 key for App Store Connect), and an LLM provider (OpenAI API key, or just Ollama running locally).

Install

git clone https://github.com/Mr-Ashish/AppPulse.git
cd AppPulse
pip install -e .

Run the Setup Wizard

apppulse setup

The wizard walks through 4 steps — adding apps, choosing an LLM, setting up notifications, configuring the schedule. I put inline instructions in each step because I kept forgetting which Play Console page to visit for the service account. Now it just tells me.

After the wizard, you set a couple of environment variables (API keys — these are never stored in the config file):

# ~/.zshrc or ~/.bashrc
export OPENAI_API_KEY="sk-..."
export SENTRY_AUTH_TOKEN="sntrys_..."           # optional
export APPPULSE_SLACK_WEBHOOK="https://hooks.slack.com/..."  # optional

Then verify and run:

apppulse test   # checks all credentials exist
apppulse run    # pulls reviews, classifies, prints digest

The first run grabs the last ~7 days of Google Play reviews (that's a hard API limit — more on that below) and all available App Store reviews.

The 7-Day Problem (This One Annoyed Me)

This caught me off guard. The Google Play Publisher API only returns the last 7 days of reviews. There's no parameter to go further back. It's a hard limit baked into the API.

For an app that's been live for two years, that means thousands of reviews you can never pull through the API. I wanted the full history — especially to see patterns over time and to have a proper baseline for trend detection.

After digging around, I found that Google has an official GCS bulk export option. Play Console can export your entire review history as monthly CSV files to a private Google Cloud Storage bucket. These go back to the day the app was first published.

So I added a backfill command that:

1. Connects to your GCS bucket using the same service account
2. Lists all available monthly CSV files
3. Downloads and parses them (handling Google's UTF-16 encoding)
4. Deduplicates against reviews already in your database
5. Optionally classifies them with your LLM

# Install the GCS dependency
pip install -e ".[gcs]"

# Backfill everything
apppulse backfill --app "MyApp Android"

# Backfill from a specific date
apppulse backfill --app "MyApp Android" --since 2024-01-01

# Preview without importing
apppulse backfill --app "MyApp Android" --dry-run

To enable it, add the GCS bucket ID to your config (~/.apppulse/config.yaml):

apps:
  - name: "MyApp Android"
    platform: "google_play"
    package_name: "com.example.myapp"
    gcs_bucket: "pubsite_prod_rev_12345678901234567890"
    credentials:
      service_account_json: "/path/to/key.json"

Watch out: The GCS bucket permission requires account-level access in Play Console, not just app-level. And it can take up to 24 hours to propagate after you grant it. If you get a storage.objects.list permission error, wait a day before debugging further.

What I Actually Use Day to Day

It's all terminal-based. Here's what I have:

Command	What it does
apppulse setup	Interactive wizard — credentials, LLM, notifications, schedule
apppulse run	One-shot: pull + classify + digest + notify
apppulse start	Start the daily scheduler (runs at your configured time)
apppulse backfill --app "X"	Import full review history from GCS
apppulse reviews --category bug	Browse reviews filtered by category
apppulse reviews --rating 1 --days 30	Find all 1-star reviews in the last month
apppulse crashes	Show top crashes from Sentry
apppulse reply 42 "Thanks!"	Reply to a review by its DB ID
apppulse export --format csv	Export reviews to CSV or JSON
apppulse test	Verify all credentials are working

What It Doesn't Do

I should be honest about this — it's a CLI tool I built for my own use. There's no web dashboard (yet). There's no team collaboration, no review assignment workflow, no competitor tracking. If you need to share review reports with a PM who doesn't use a terminal, this isn't the tool for that right now.

If you have a team that needs shared dashboards, role-based access, or competitor intelligence, the paid platforms are genuinely good at that. I just didn't need any of it.

What I do get is: my data stays on my machine in a SQLite file I can query directly, the LLM-based classification actually understands context (not just keywords), and I can hook in my Sentry crash data — something none of the tools I looked at could do. And the whole thing costs me about $0.50/month in OpenAI API calls. If even that feels like too much, Ollama runs locally for free.

What I'm Working on Next

This is still a work in progress. Here's what I want to add for my own use — and if any of this is useful to you, PRs are welcome.

Things I'm actively building:

• Review-crash correlation — automatically match 1-star reviews with Sentry crash reports by keyword overlap, device info, timing, and version
• Trend detection — notice when bug reports spike after a release, or when ratings start dropping
• Background daemon mode — so the scheduler runs without blocking my terminal

Things I'd like to get to eventually:

• A simple web dashboard — probably Streamlit. I don't need something fancy, just a visual overview of what's happening across my apps
• Review clustering — so when 20 people report the same login bug, it shows up as one item, not 20
• Auto-drafted replies — generate a reply based on the category, let me approve before sending
• GitHub issue creation — when a cluster of reviews hits a threshold, auto-create an issue with excerpts and device breakdown

If you want to pick up any of these, the issues page is the best place to start.

If You Want to Try It

The whole thing is on GitHub: github.com/Mr-Ashish/AppPulse

git clone https://github.com/Mr-Ashish/AppPulse.git
cd AppPulse
pip install -e .
apppulse setup

It's MIT licensed — fork it, extend it, gut it and use the parts you need. If you find bugs or have ideas, open an issue. I built this to solve my own problem, but if it saves you some time or money too, that's great.

---

Ashish Mishra — I build apps and occasionally build tools to avoid reading app reviews manually.