how to tell jackson serializer to stop continuing to serialize and write when object is big enough

in my recent project, there are a lot of log statement like below:

Bar bar = foo();
logger.info("something happened: param:{}", JSON.toJSONString(bar));

as you can see, just json serialize the object into log. sometimes bar is very big, and serializing the bar object consumes too much time, and log file expands a lot. the json serializing framework is fasterxml-jackson

So my question is: is there a way ,like customizing JsonSerializer to implement:

  1. count the chars/bytes that has already been serialized(by invoking method #toJSONString())
  2. if the count has exceed the MAX_SIZE, say 2000, then stop
  3. return just the 2000 char String and print it in log file

2 answers

  • answered 2020-11-25 10:49 MichaƂ Ziober

    If we do not want to fall into Jackson and how it works we can just limit Writer where objects are serialised into. It is simple approach which does not require to implement custom serialisers with limits checked inside. Downside of this approach is we do not skip serialisation process we just ignore it's result.

    You can write your own Writer with a maximum buffer size. Simple implementation based on StringBuilder:

    class LimitedStringBuilderWriter extends Writer {
    
        private final StringBuilder buffer;
        private int remaining;
    
        public LimitedStringBuilderWriter(int limit) {
            if (limit <= 0) {
                throw new IllegalArgumentException("Limit must be positive number!");
            }
    
            this.remaining = limit;
            this.buffer = new StringBuilder(limit);
        }
    
        @Override
        public void write(char[] cbuf, int off, int len) {
            if (len == 0 || this.remaining <= 0) {
                return;
            }
            if ((off < 0) || (off > cbuf.length) || (len < 0) ||
                    ((off + len) > cbuf.length) || ((off + len) < 0)) {
                throw new IndexOutOfBoundsException();
            }
            final int size = len - off;
            if (this.remaining >= size) {
                this.remaining -= size;
                buffer.append(cbuf, off, len);
                return;
            }
            buffer.append(cbuf, off, this.remaining);
            this.remaining = 0;
        }
    
        @Override
        public void write(int c) {
            if (this.remaining <= 0) {
                return;
            }
            this.remaining--;
            this.buffer.append(c);
        }
    
        @Override
        public void flush() {
        }
    
        @Override
        public void close() {
        }
    
        @Override
        public String toString() {
            return this.buffer.toString();
        }
    }
    

    Usage:

    import com.fasterxml.jackson.databind.ObjectMapper;
    import com.fasterxml.jackson.databind.json.JsonMapper;
    
    import java.io.IOException;
    import java.io.Writer;
    import java.util.Arrays;
    import java.util.LinkedHashMap;
    import java.util.Map;
    import java.util.function.Supplier;
    
    public class LimitJsonLogApp {
    
        public static void main(String[] args) {
            Map<String, Object> value = new LinkedHashMap<>();
            value.put("array", Arrays.asList(1, 2, 3));
            value.put("string", "Value");
            value.put("int", 23);
    
            for (int limit = 11; limit < 30; limit += 3) {
                System.out.println(limit + " => " + JSON.toJSONString(value, limit));
            }
        }
    }
    
    class JSON {
    
        private static final ObjectMapper mapper = JsonMapper.builder().build();
    
        public static String toJSONString(Object value) {
            return toJSONStringSupplier(value).get();
        }
    
        public static String toJSONString(Object value, int limit) {
            return toJSONStringSupplier(value, limit).get();
        }
    
        public static Supplier<String> toJSONStringSupplier(Object value) {
            return toJSONStringSupplier(value, 1000);
        }
    
        public static Supplier<String> toJSONStringSupplier(Object value, int limit) {
            if (value == null) {
                return () -> "null";
            }
            return () -> {
                try {
                    LimitedStringBuilderWriter writer = new LimitedStringBuilderWriter(limit);
                    mapper.writeValue(writer, value);
                    return writer.toString();
                } catch (IOException e) {
                    throw new IllegalArgumentException(e);
                }
            };
        }
    }
    

    Above code prints:

    11 => {"array":[1
    14 => {"array":[1,2,
    17 => {"array":[1,2,3],
    20 => {"array":[1,2,3],"st
    23 => {"array":[1,2,3],"strin
    26 => {"array":[1,2,3],"string":
    29 => {"array":[1,2,3],"string":"Va
    

    Notice, some methods in JSON class returns Supplier<String>. You should use them if your logger allows to provide suppliers. It allows to postpone serialisation to the moment when it is really needed. And in case you disable INFO log in configuration it would really disable execution of this code.

  • answered 2020-11-25 14:43 seiyas

    after doing some research, the below thought comes up. this is not that elegant. wish someone could improve this!

    import java.io.IOException;
    import java.io.Writer;
    import java.util.stream.IntStream;
    
    import com.fasterxml.jackson.core.JsonFactory;
    import com.fasterxml.jackson.core.JsonGenerator;
    import com.fasterxml.jackson.core.json.WriterBasedJsonGenerator;
    import com.fasterxml.jackson.databind.JavaType;
    import com.fasterxml.jackson.databind.JsonMappingException;
    import com.fasterxml.jackson.databind.JsonSerializer;
    import com.fasterxml.jackson.databind.MappingJsonFactory;
    import com.fasterxml.jackson.databind.ObjectMapper;
    import com.fasterxml.jackson.databind.SerializationConfig;
    import com.fasterxml.jackson.databind.SerializerProvider;
    import com.fasterxml.jackson.databind.jsontype.TypeSerializer;
    import com.fasterxml.jackson.databind.ser.BeanSerializerModifier;
    import com.fasterxml.jackson.databind.ser.SerializerFactory;
    import com.fasterxml.jackson.databind.ser.Serializers;
    import org.jetbrains.annotations.NotNull;
    
    public class JSON {
    
        static int MAX_SIZE = 10;
        final static ObjectMapper OBJECT_MAPPER;
        static {
            OBJECT_MAPPER = new ObjectMapper(new WrappedJsonFactory(new MappingJsonFactory()), null, null);
            OBJECT_MAPPER.setSerializerFactory(new WrapperSerialFactory(OBJECT_MAPPER.getSerializerFactory()));
        }
    
        static String toJSONString(Object value) {
            try {
                return OBJECT_MAPPER.writeValueAsString(value);
            } catch (IOException ex) {
                throw new RuntimeException(ex);
            }
        }
    
    
        public static void main(String... args) throws Exception {
            int[] array = IntStream.range(1, 100).toArray();
            System.out.println(JSON.toJSONString(array));
            MAX_SIZE = 20;
            System.out.println(JSON.toJSONString(array));
            MAX_SIZE = 30;
            System.out.println(JSON.toJSONString(array));
        }
    
        static class WrapperSerialFactory extends SerializerFactory {
    
            private SerializerFactory wrapped;
    
            public WrapperSerialFactory(SerializerFactory wrapped) {
                this.wrapped = wrapped;
            }
    
            @Override
            public SerializerFactory withAdditionalSerializers(Serializers additional) {
                return wrapped.withAdditionalSerializers(additional);
            }
    
            @Override
            public SerializerFactory withAdditionalKeySerializers(Serializers additional) {
                return wrapped.withAdditionalKeySerializers(additional);
            }
    
            @Override
            public SerializerFactory withSerializerModifier(BeanSerializerModifier modifier) {
                return wrapped.withSerializerModifier(modifier);
            }
    
            @Override
            public JsonSerializer<Object> createSerializer(SerializerProvider prov, JavaType baseType)
                throws JsonMappingException {
                JsonSerializer<Object> serializer = wrapped.createSerializer(prov, baseType);
                return new WrappedTypeSerializer(serializer);
            }
    
            @Override
            public TypeSerializer createTypeSerializer(SerializationConfig config, JavaType baseType)
                throws JsonMappingException {
                return wrapped.createTypeSerializer(config, baseType);
            }
    
            @Override
            public JsonSerializer<Object> createKeySerializer(SerializationConfig config, JavaType type,
                JsonSerializer<Object> defaultImpl) throws JsonMappingException {
                return wrapped.createKeySerializer(config, type, defaultImpl);
            }
        }
    
    
    
        static class WrappedJsonFactory extends JsonFactory {
            private JsonFactory wrapped;
    
            public WrappedJsonFactory(JsonFactory wrapped) {
                this.wrapped = wrapped;
            }
    
            @Override
            public JsonGenerator createGenerator(Writer w) throws IOException {
                w = new LimitingWriter(w);
                return wrapped.createGenerator(w);
            }
        }
    
        static class LimitingWriter extends Writer {
            private int count;
            private boolean finished = false;
    
            private Writer delegate;
    
            public LimitingWriter(Writer delegate) {
                this.delegate = delegate;
            }
    
            private void finish() throws IOException {
                if (!finished) {
                    delegate.write("...");
                    finished = true;
                }
            }
    
            @Override
            public void write(int c) throws IOException {
                if (!finished) {
                    if (count >= MAX_SIZE) {
                        finish();
                    } else {
                        delegate.write(c);
                        ++count;
                    }
                }
            }
    
            @Override
            public void write(@NotNull char[] cbuf) throws IOException {
                if (!finished) {
                    if ((count + cbuf.length) > MAX_SIZE) {
                        delegate.write(cbuf, 0, MAX_SIZE - count);
                        count = MAX_SIZE;
                        finish();
                    } else {
                        delegate.write(cbuf);
                        count += cbuf.length;
                    }
                }
            }
    
            @Override
            public void write(@NotNull String str) throws IOException {
                if (!finished) {
                    if (count + str.length() > MAX_SIZE) {
                        delegate.write(str, 0, MAX_SIZE - count);
                        count = MAX_SIZE;
                        finish();
                    } else {
                        delegate.write(str);
                        count += str.length();
                    }
                }
            }
    
            @Override
            public void write(@NotNull String str, int off, int len) throws IOException {
                if (!finished) {
                    if (count + len > MAX_SIZE) {
                        delegate.write(str, 0, MAX_SIZE - count);
                        count = MAX_SIZE;
                        finish();
                    } else {
                        delegate.write(str, off, len);
                        count += len;
                    }
                }
            }
    
            @Override
            public Writer append(CharSequence csq) throws IOException {
                if (!finished) {
                    if (count + csq.length() > MAX_SIZE) {
                        count = MAX_SIZE;
                        delegate.append(csq, 0, MAX_SIZE - count);
                        finish();
                    } else {
                        count += csq.length();
                        delegate.append(csq);
                    }
                }
                return this;
            }
    
            @Override
            public Writer append(CharSequence csq, int start, int end) throws IOException {
                if (!finished) {
                    if (count + end - start > MAX_SIZE) {
                        delegate.append(csq, start, start + MAX_SIZE - count);
                        count = MAX_SIZE;
                        finish();
                    } else {
                        count += (end - start);
                        delegate.append(csq, start, end);
                    }
                }
                return this;
            }
    
            @Override
            public Writer append(char c) throws IOException {
                if (!finished) {
                    if (count >= MAX_SIZE) {
                        finish();
                    } else {
                        ++count;
                        delegate.append(c);
                    }
                }
                return this;
            }
    
            @Override
            public void write(@NotNull char[] cbuf, int off, int len) throws IOException {
                if (!finished) {
                    if (count + len > MAX_SIZE) {
                        delegate.write(cbuf, off, MAX_SIZE - count);
                        count = MAX_SIZE;
                        finish();
                    } else {
                        delegate.write(cbuf, off, len);
                        count += len;
                    }
                }
            }
    
            @Override
            public void flush() throws IOException {
                delegate.flush();
            }
    
            @Override
            public void close() throws IOException {
                delegate.close();
            }
        }
    
        static class WrappedTypeSerializer extends JsonSerializer {
            private JsonSerializer wrapped;
    
            public WrappedTypeSerializer(JsonSerializer wrapped) {
                this.wrapped = wrapped;
            }
    
            @Override
            public void serialize(Object value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
                WriterBasedJsonGenerator writerBasedJsonGenerator = (WriterBasedJsonGenerator)gen;
                LimitingWriter writer = (LimitingWriter)writerBasedJsonGenerator.getOutputTarget();
                if (writer.finished) {
                    return;
                }
                wrapped.serialize(value, gen, serializers);
            }
        }
    }
    

    output:

    [1,2,3,4,5...
    [1,2,3,4,5,6,7,8,9,1...
    [1,2,3,4,5,6,7,8,9,10,11,12,13...