Skip to content

Conversation

@xakep666
Copy link

This may be useful i.e. when your service should download file and send it inside json to other service without storing whole file in memory.
In my case service downloads a file from S3-like service and makes request to our email provider to send it as attachment (base64 encoded).
Currently I'm using this snippet:

func (a *Attach) MarshalJSONObject(e *gojay.Encoder) {
	e.StringKey("name", a.Name)
	e.StringKeyOmitEmpty("encoding", a.Encoding)

	// attach content handling
	if a.Content != nil {
		e.AppendByte(',')                                 // begin new field
		e.AppendBytes([]byte(`"content"`))                // appends "content" key
		e.AppendByte(':')                                 // delimiter between key and value
		e.AppendByte('"')                                 // string begins
		_, err := io.Copy((*encoderWriter)(e), a.Content) // big string (unsafe, no escaping)
		if err != nil {
			panic(err) // caught by caller
		}
		e.AppendByte('"') // string ends
	}
}

My changes allows to shorten it to this:

func (a *Attach) MarshalJSONObject(e *gojay.Encoder) {
	e.StringKey("name", a.Name)
	e.StringKeyOmitEmpty("encoding", a.Encoding)
        e.ReaderToBase64Key("content", a.Content, base64.StdEncoding)
}

This PR adds similar methods for decoding (may be useful i.e. when you get response where file goes as base64-encoded string and this file should be uploaded to somewhere).

Side changes:

  • decoder: better buffer reusability when parsing escape sequences
  • encoder: flush buffer when it's size reaches a limit provided in SetBufFlushThreshold if target writer provided (to reduce memory consumption). Should close Periodically flush the buf to the Writer. #78

This made to reuse existing data buffer without (re)allocations
when decoder tries to parse escape sequence
Buffer flushing happens when target writer provided
and buffer size reaches value provided in SetBufFlushThreshold
@xakep666 xakep666 requested a review from francoispqt February 3, 2020 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Periodically flush the buf to the Writer.

1 participant