Skip to content

Added Tensor.dataToString() #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cowwoc wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Added Tensor.dataToString() #272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
9e90751
Added Tensor.dataToString().
cowwoc Apr 3, 2021
b3a2ab7
Added missing import.
cowwoc Apr 3, 2021
e30d9b5
Documentation fix.
cowwoc Apr 3, 2021
8bf627a
Documentation fix.
cowwoc Apr 3, 2021
61e7cad
maxWidth was truncating text mid-element.
cowwoc Apr 3, 2021
ffdf879
* Added Operand.dataToString().
cowwoc Apr 4, 2021
da04cd4
Fixed test.
cowwoc Apr 4, 2021
5821e99
Leaving out comma after closing brackets.
cowwoc Apr 4, 2021
6f8dedc
Data type tests, wrap strings in quotes
rnett Apr 5, 2021
98d320d
Cleanup in response to PR comments.
cowwoc Apr 6, 2021
921e077
Merge pull request #1 from rnett/feature/tensor-datatostring
cowwoc Apr 6, 2021
7e56987
Merge remote-tracking branch 'upstream/master' into feature/tensor-da…
cowwoc Apr 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/Operand.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

package org.tensorflow;

import org.tensorflow.Tensor.ToStringOptions;
import org.tensorflow.ndarray.Shape;
import org.tensorflow.ndarray.Shaped;
import org.tensorflow.op.Op;
Expand Down Expand Up @@ -65,6 +66,17 @@ default T asTensor() {
return asOutput().asTensor();
}

/**
* Returns the String representation of the tensor elements at this operand.
*
* @param options overrides the default configuration
* @return the String representation of the tensor elements
* @throws IllegalStateException if this is an operand of a graph
*/
default String dataToString(ToStringOptions... options) {
return asTensor().dataToString(options);
}

/**
* Returns the tensor type of this operand
*/
Expand Down
149 changes: 100 additions & 49 deletions tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/Tensor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@
* A statically typed multi-dimensional array.
*
* <p>There are two categories of tensors in TensorFlow Java: {@link TType typed tensors} and
* {@link RawTensor raw tensors}. The former maps the tensor native memory to an
* n-dimensional typed data space, allowing direct I/O operations from the JVM, while the latter
* is only a reference to a native tensor allowing basic operations and flat data access.</p>
* {@link RawTensor raw tensors}. The former maps the tensor native memory to an n-dimensional typed
* data space, allowing direct I/O operations from the JVM, while the latter is only a reference to
* a native tensor allowing basic operations and flat data access.</p>
*
* <p><b>WARNING:</b> Resources consumed by the Tensor object <b>must</b> be explicitly freed by
* invoking the {@link #close()} method when the object is no longer needed. For example, using a
Expand All @@ -49,15 +49,15 @@ public interface Tensor extends Shaped, AutoCloseable {
* <p>The amount of memory to allocate is derived from the datatype and the shape of the tensor,
* and is left uninitialized.
*
* @param <T> the tensor type
* @param type the tensor type class
* @param <T> the tensor type
* @param type the tensor type class
* @param shape shape of the tensor
* @return an allocated but uninitialized tensor
* @throws IllegalArgumentException if elements of the given {@code type} are of variable length
* (e.g. strings)
* @throws IllegalArgumentException if {@code shape} is totally or partially
* {@link Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
* @throws IllegalArgumentException if {@code shape} is totally or partially {@link
* Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
*/
static <T extends TType> T of(Class<T> type, Shape shape) {
return of(type, shape, -1);
Expand All @@ -67,27 +67,27 @@ static <T extends TType> T of(Class<T> type, Shape shape) {
* Allocates a tensor of a given datatype, shape and size.
*
* <p>This method is identical to {@link #of(Class, Shape)}, except that the final size of the
* tensor can be explicitly set instead of computing it from the datatype and shape, which could be
* larger than the actual space required to store the data but not smaller.
* tensor can be explicitly set instead of computing it from the datatype and shape, which could
* be larger than the actual space required to store the data but not smaller.
*
* @param <T> the tensor type
* @param type the tensor type class
* @param <T> the tensor type
* @param type the tensor type class
* @param shape shape of the tensor
* @param size size in bytes of the tensor or -1 to compute the size from the shape
* @param size size in bytes of the tensor or -1 to compute the size from the shape
* @return an allocated but uninitialized tensor
* @see #of(Class, Shape)
* @throws IllegalArgumentException if {@code size} is smaller than the minimum space required to
* store the tensor data
* @throws IllegalArgumentException if {@code size} is set to -1 but elements of the given
* {@code type} are of variable length (e.g. strings)
* @throws IllegalArgumentException if {@code shape} is totally or partially
* {@link Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
* @throws IllegalArgumentException if {@code size} is set to -1 but elements of the given {@code
* type} are of variable length (e.g. strings)
* @throws IllegalArgumentException if {@code shape} is totally or partially {@link
* Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
* @see #of(Class, Shape)
*/
static <T extends TType> T of(Class<T> type, Shape shape, long size) {
RawTensor tensor = RawTensor.allocate(type, shape, size);
try {
return (T)tensor.asTypedTensor();
return (T) tensor.asTypedTensor();
} catch (Exception e) {
tensor.close();
throw e;
Expand All @@ -111,16 +111,17 @@ static <T extends TType> T of(Class<T> type, Shape shape, long size) {
* <p>If {@code dataInitializer} fails and throws an exception, the allocated tensor will be
* automatically released before rethrowing the same exception.
*
* @param <T> the tensor type
* @param type the tensor type class
* @param shape shape of the tensor
* @param dataInitializer method receiving accessor to the allocated tensor data for initialization
* @param <T> the tensor type
* @param type the tensor type class
* @param shape shape of the tensor
* @param dataInitializer method receiving accessor to the allocated tensor data for
* initialization
* @return an allocated and initialized tensor
* @throws IllegalArgumentException if elements of the given {@code type} are of variable length
* (e.g. strings)
* @throws IllegalArgumentException if {@code shape} is totally or partially
* {@link Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
* @throws IllegalArgumentException if {@code shape} is totally or partially {@link
* Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
*/
static <T extends TType> T of(Class<T> type, Shape shape, Consumer<T> dataInitializer) {
return of(type, shape, -1, dataInitializer);
Expand All @@ -130,27 +131,30 @@ static <T extends TType> T of(Class<T> type, Shape shape, Consumer<T> dataInitia
* Allocates a tensor of a given datatype, shape and size.
*
* <p>This method is identical to {@link #of(Class, Shape, Consumer)}, except that the final
* size for the tensor can be explicitly set instead of being computed from the datatype and shape.
* size for the tensor can be explicitly set instead of being computed from the datatype and
* shape.
*
* <p>This could be useful for tensor types that stores data but also metadata in the tensor memory,
* such as the lookup table in a tensor of strings.
* <p>This could be useful for tensor types that stores data but also metadata in the tensor
* memory, such as the lookup table in a tensor of strings.
*
* @param <T> the tensor type
* @param type the tensor type class
* @param shape shape of the tensor
* @param size size in bytes of the tensor or -1 to compute the size from the shape
* @param dataInitializer method receiving accessor to the allocated tensor data for initialization
* @param <T> the tensor type
* @param type the tensor type class
* @param shape shape of the tensor
* @param size size in bytes of the tensor or -1 to compute the size from the shape
* @param dataInitializer method receiving accessor to the allocated tensor data for
* initialization
* @return an allocated and initialized tensor
* @see #of(Class, Shape, long, Consumer)
* @throws IllegalArgumentException if {@code size} is smaller than the minimum space required to
* store the tensor data
* @throws IllegalArgumentException if {@code size} is set to -1 but elements of the given
* {@code type} are of variable length (e.g. strings)
* @throws IllegalArgumentException if {@code shape} is totally or partially
* {@link Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
* @throws IllegalArgumentException if {@code size} is set to -1 but elements of the given {@code
* type} are of variable length (e.g. strings)
* @throws IllegalArgumentException if {@code shape} is totally or partially {@link
* Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated
* @see #of(Class, Shape, long, Consumer)
*/
static <T extends TType> T of(Class<T> type, Shape shape, long size, Consumer<T> dataInitializer) {
static <T extends TType> T of(Class<T> type, Shape shape, long size,
Consumer<T> dataInitializer) {
T tensor = of(type, shape, size);
try {
dataInitializer.accept(tensor);
Expand All @@ -167,18 +171,19 @@ static <T extends TType> T of(Class<T> type, Shape shape, long size, Consumer<T>
* <p>Data must have been encoded into {@code data} as per the specification of the TensorFlow <a
* href="https://www.tensorflow.org/code/tensorflow/c/c_api.h">C API</a>.
*
* @param <T> the tensor type
* @param type the tensor type class
* @param shape the tensor shape.
* @param <T> the tensor type
* @param type the tensor type class
* @param shape the tensor shape.
* @param rawData a buffer containing the tensor raw data.
* @throws IllegalArgumentException if {@code rawData} is not large enough to contain the tensor
* data
* @throws IllegalArgumentException if {@code shape} is totally or partially
* {@link Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated with the given parameters
* @throws IllegalArgumentException if {@code shape} is totally or partially {@link
* Shape#hasUnknownDimension() unknown}
* @throws IllegalStateException if tensor failed to be allocated with the given parameters
*/
static <T extends TType> T of(Class<T> type, Shape shape, ByteDataBuffer rawData) {
return of(type, shape, rawData.size(), t -> rawData.copyTo(t.asRawTensor().data(), rawData.size()));
return of(type, shape, rawData.size(),
t -> rawData.copyTo(t.asRawTensor().data(), rawData.size()));
}

/**
Expand All @@ -191,6 +196,33 @@ static <T extends TType> T of(Class<T> type, Shape shape, ByteDataBuffer rawData
*/
long numBytes();

/**
* Returns the String representation of elements stored in the tensor.
*
* @param options overrides the default configuration
* @return the String representation of the tensor elements
* @throws IllegalStateException if this is an operand of a graph
*/
default String dataToString(ToStringOptions... options) {
Integer maxWidth = null;
if (options != null) {
for (ToStringOptions opts : options) {
if (opts.maxWidth != null) {
maxWidth = opts.maxWidth;
}
}
}
return Tensors.toString(this, maxWidth);
}

/**
* @param maxWidth the maximum width of the output in characters ({@code null} if unlimited). This
* limit may surpassed if the first or last element are too long.
*/
static ToStringOptions maxWidth(Integer maxWidth) {
return new ToStringOptions().maxWidth(maxWidth);
}

/**
* Returns the shape of the tensor.
*/
Expand All @@ -212,4 +244,23 @@ static <T extends TType> T of(Class<T> type, Shape shape, ByteDataBuffer rawData
*/
@Override
void close();

class ToStringOptions {

/**
* Sets the maximum width of the output in characters.
*
* @param maxWidth the maximum width of the output in characters ({@code null} if unlimited).
* This limit may surpassed if the first or last element are too long.
*/
public ToStringOptions maxWidth(Integer maxWidth) {
this.maxWidth = maxWidth;
return this;
}

private Integer maxWidth;

private ToStringOptions() {
}
}
}
Loading