Unknown enum value causes plugin to write JSON it can't read back in

[elwaxoro]

Found an issue with unknown enum values which results in ProtobufModule writing JSON it can never read back in.

Using proto3, jackson 2.9.5, jackson-datatype-protobuf 0.9.10-jackson2.9-proto3

Example message V1

message TestEnum {
    TestType type = 1;
    enum TestType {
        UNKNOWN = 0;
        FIRST = 1;
    }
}

Example message V2

message TestEnum {
    TestType type = 1;
    enum TestType {
        UNKNOWN = 0;
        FIRST = 1;
        SECOND = 2;
    }
}

Steps to error:

1. Upgrade proto from V1 to V2, but don't deploy to all apps (undesirable, but supported by protobuf spec)
2. Write proto with new enum value as bytes from app with V2 jar
3. Read bytes in app using V1 jar (supported by protobuf: enum value is held as UNKNOWN_ENUM_VALUE_TestType_2)
4. Write proto to json using ProtobufModule, results in json looking like:

{"type":"UNKNOWN_ENUM_VALUE_TestType_2"}

5. Read json, get exception:

com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.google.protobuf.Descriptors$EnumDescriptor` from String "UNKNOWN_ENUM_VALUE_TestType_2": value not one of declared Enum instance names
 at [Source: (String)"{"type":"UNKNOWN_ENUM_VALUE_TestType_2"}"; line: 1, column: 9]

For now, the only way to recover from this error is to upgrade the outdated application and hand-edit the JSON to the correct enum value. Even once the jar is upgraded, the JSON value UNKNOWN_ENUM_VALUE_TestType_2 can never be read by ProtobufModule.

Simple unit test demonstrating issue (kotlin):

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.readValue
import com.hubspot.jackson.datatype.protobuf.ProtobufModule
import com.google.protobuf.util.JsonFormat
import org.junit.Test
import java.io.File
import java.nio.file.Files

import foo.TestEnum

class ProtoReaderTest {

    private val mapper = ObjectMapper().registerModule(ProtobufModule())

//    @Test
//    fun genV2Binary() {
//        val path = File("/tmp/newbinary.bin").toPath()
//        val proto1 = TestEnum.newBuilder().setType(TestEnum.TestType.SECOND).build()
//        Files.write(path, proto1.toByteArray())
//        val proto2 = TestEnum.parseFrom(Files.readAllBytes(path))
//        assertEquals(proto1, proto2)
//    }

    @Test
    fun readV2WithV1Jar() {
        val path = File("/tmp/newbinary.bin").toPath()
        val proto1 = TestEnum.parseFrom(Files.readAllBytes(path))
        val json1 = mapper.writeValueAsString(proto1)
        // Broken! this throws InvalidFormatException
        val proto2: TestEnum = mapper.readValue(json1)
    }

    @Test
    fun readV2WithV2JarJsonFormat() {
        val path = File("/tmp/newbinary.bin").toPath()
        val proto1 = TestEnum.parseFrom(Files.readAllBytes(path))
        val json1 = JsonFormat.printer().print(proto1)
        val proto2 = TestEnum.newBuilder()
        JsonFormat.parser().merge(json1, proto2)
    }
}

Reading with protobuf's JsonFormat.printer() produces json for the unknown value like:
{ "type": 2 }
Reading back from json using JsonFormat.parser() restores the enum as UNKNOWN_ENUM_VALUE_TestType_2

Related issue: the json written by JsonFormat.printer() for this unkown value is also unreadable by ProtobufModule:

com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.google.protobuf.Descriptors$EnumDescriptor` from number 2: index value outside legal index range [0,1]
 at [Source: (String)"{
  "type": 2
}"; line: 2, column: 11]

Edit: added example of how JsonFormat is handling unkown enum

[cowtowncoder]

Are you aware of DeserializationFeature.READ_UNKNOWN_ENUM_VALUES_AS_NULL and READ_UNKNOWN_ENUM_VALUES_USING_DEFAULT_VALUE options to deal with such situations?

[elwaxoro]

@cowtowncoder I am, yes. the issue is more that that json serializer makes something that the deserializer can't handle at all without resorting to options that cause data loss (null / default). Ideally, the jackson plugin would parse the unknown enum back into the protobuf in the same way it pulled it out.