Home

GoScrapy: Wiki (v0.26.0)

Prerequisites

GoScrapy requires Go version 1.22 or higher.

GoScrapy CLI

Scaffold your project and components instantly using the CLI.

Usage

go install github.com/tech-engine/goscrapy@latest
gos startproject my_project

---

Base Concepts

GoScrapy operates around three core concepts:

• Job: Target input for your spider (must implement Id() string).
• Record: The output data structure produced by your spider (must implement core.IOutput[OUT]).
• Spider: The main logic of your scraper, embedding gos.ICoreSpider.

Record (Auto-Generated)

The Record is generic. Its Record() method returns the specific type OUT rather than a hardcoded pointer, allowing for type-safe pipeline processing.

[record.go]

type Record struct {
    Title string `json:"title"`
    Price string `json:"price"`
}

func (r *Record) Record() *Record { return r }
// ... implements RecordKeys(), RecordFlat(), Job() ...

Spider (Auto-Discovery)

In v0.26.0, defining these methods on your spider struct allows the engine to auto-discover and call them without manual wiring:

• Open(context.Context): Called when the engine starts.
• Close(context.Context): Called when the engine shuts down.
• Idle(context.Context): Called when the scheduler is empty and workers are idle.
• Error(context.Context, error): Called on spider-level failures or panics.

[spider.go]

func (s *Spider) StartRequest(ctx context.Context, job *Job) {
    req := s.Request(ctx)
    req.Url("https://books.toscrape.com")
    s.Parse(req, s.parse)
}

func (s *Spider) parse(ctx context.Context, resp core.IResponseReader) {
    // Extraction logic...
    s.Yield(&Record{Title: "The Great Gatsby"})
}

---

Settings (Auto-Generated)

GoScrapy uses settings.go as a configuration hub. It maps constants to environment variables via an init() function to tune the engine without modifying the core.

[settings.go]

// --- Engine & Tuning ---

// Default: 16
const SCHEDULER_CONCURRENCY = ""

// Default: 1000
const PIPELINEMANAGER_OUTPUT_QUEUE_BUF_SIZE = ""

// --- HTTP Transport (client.go) ---

// Default: 10000 (ms)
const MIDDLEWARE_HTTP_TIMEOUT_MS = ""

// Default: 1000
const MIDDLEWARE_HTTP_MAX_IDLE_CONN = ""

// --- Middlewares & Pipelines ---

var MIDDLEWARES = []middlewaremanager.Middleware{
    middlewares.Retry(),
    middlewares.DupeFilter,
    middlewares.Stats(), // Enables real-time metrics for TUI
}

var PIPELINES = []engine.IPipeline[*Record]{
    csv.New[*Record](csv.Options{Filename: "results.csv"}),
}

---

Base & Usage

[base.go]

func New(ctx context.Context) (*Spider, error) {
    // Initialize the engine with generics
    app, err := gos.New[*Record]()
    if err != nil {
        return nil, err
    }

    app.WithMiddlewares(MIDDLEWARES...).
        WithPipelines(PIPELINES...)

    spider := &Spider{ICoreSpider: app}
    
    // Auto-Discovery: Connects spider methods to the internal signal bus
    app.RegisterSpider(spider)

    go func() { _ = app.Start(ctx) }()

    return spider, nil
}

[main.go]

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    spider, _ := my_project.New(ctx)
    spider.StartRequest(ctx, nil)

    // Wait(true) initiates a graceful shutdown when the spider becomes Idle
    if err := spider.Wait(true); err != nil {
        log.Printf("Engine exit: %v", err)
    }
}

---

Selectors

GoScrapy supports CSS and XPATH out of the box using a fluent API.

func (s *Spider) parse(ctx context.Context, resp core.IResponseReader) {
    // CSS Selector
    titles := resp.Css(".product_pod h3 a").Text()
    
    // XPATH Selector
    links := resp.Xpath("//article[@class='product_pod']//h3/a").Attr("href")
    
    // Chaining
    first_title := resp.Css(".product_pod").Css("h3").Get()
}

---

Built-in Components

Middlewares

• Retry: Smart exponential back-off for 5xx/429 status codes.
• DupeFilter: Memory-efficient request de-duplication.
• Stats: Non-blocking atomic metric collection for monitoring.
• AzureTLS: Advanced fingerprint spoofing to bypass bot detection.

Pipelines

• Export2CSV / Export2JSON: Fast local file exports.
• Export2GSHEET: Direct Google Sheets integration.
• Export2MONGODB / Export2FIREBASE: Production-ready database connectors.

---

Get in touch

Discord | GitHub Issues