Debugging & Troubleshooting

Some fields are missing or wrong

Check the parser status first

curl -X GET \
  "https://aiparser.scraperapi.com/parsers/<PARSER_ID>?api_key=API_KEY"

import requests

url = "https://aiparser.scraperapi.com/parsers/<PARSER_ID>"

params = {
    "api_key": "API_KEY"
}

response = requests.get(url, params=params)

print(response.status_code)
print(response.text)

const parserId = "PARSER_ID";
const apiKey = "API_KEY";

const url = `https://aiparser.scraperapi.com/parsers/${parserId}?api_key=${apiKey}`;

async function getParser() {
  try {
    const response = await fetch(url);

    const data = await response.json();

    console.log("Status:", response.status);

    // IMPORTANT: force full expansion in console
    console.log(JSON.stringify(data, null, 2));
  } catch (error) {
    console.error("Error:", error);
  }
}

getParser();

require 'net/http'
require 'uri'

parser_id = "PARSER_ID"
api_key = "API_KEY"

uri = URI("https://aiparser.scraperapi.com/parsers/#{parser_id}?api_key=#{api_key}")

response = Net::HTTP.get(uri)

print(response)

import java.io.*;
import java.net.*;

public class Main {
    public static void main(String[] args) {
        try {
            String parserId = "PARSER_ID";
            String apiKey = "API_KEY";

            URL url = new URL(
                "https://aiparser.scraperapi.com/parsers/" +
                parserId +
                "?api_key=" +
                apiKey
            );

            BufferedReader in = new BufferedReader(
                new InputStreamReader(url.openStream())
            );

            StringBuilder response = new StringBuilder();
            String line;

            while ((line = in.readLine()) != null) {
                response.append(line);
            }

            in.close();

            System.out.println(response.toString());

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

The parser extracted most fields correctly but a few are off

Field is missing → It might not exist on the page or wasn't recognized. Use PATCH with add_fields and a clearer description.
Incorrect formatting (price shown as $12.99 instead of 12.99) → Use modify_fields with a more specific description: "Price in USD as a number, no currency symbols".
Field shows wrong value (SKU and product code are mixed up) → Use modify_fields with a better description or add a selector to target the correct element.

Results are inconsistent across runs

Parsing the same URL twice returns different results - This usually means the target site itself returns different HTML on different requests: common with A/B tests and geographic variations. Solutions:

Geotarget to a specific country, by using country_code in scraper_params
Try a different URL of the same type. Some pages may have edge-case layouts. If others work, the parser is fine.

The Parser worked yesterday but is failing today

The target website probably introduced structural changes to its HTML.

Quick fix - use PATCH and modify_fields to update the affect fields' descriptions. Use selectors to target the correct elements.
If changes to the HTML are major - Create new parser with fresh example URLs. The old parser will still be accessible, unless you decide to DELETE it.

Parser generation keeps failing

If parser generation keeps failing, this usually means that the target server is blocking the requests. Bot detection, rate limiting or geo-restrictions can prevent us from reaching the example URLs. Adding additional ScraperAPI parameters to the scraper_params can help bypass those detections/restrictions. Try the following options one at a time.

Geotargeting - if the contents are geo-locked to a specific country, geotarget with country_code

{ "scraper_params": { "country_code": "us" } }

Use premium proxies - premium=true ensures that residential IPs are used, which decreases the chance of the request getting flagged

{ "scraper_params": { "premium": true } }

Advanced unblocking mechanisms - add ultra_premium=true when targeting websites with a robust anti-bot protection.

{ "scraper_params": { "ultra_premium": true } }

premium and ultra_premium should be used separately. Each adds more credits to the final cost, so starting light and adding parameters only when needed keeps your usage efficient.

PreviousStatus Codes & Errors NextWhat is DataPipeline?

Last updated 17 days ago