Decode Avro decimal-as-bytearray string
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
add a comment |
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
For comparison:struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'
– z0r
Nov 21 '18 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 '18 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 '18 at 23:01
add a comment |
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
I'm using Kafka with Avro messages. One of my fields is defined like this:
{
"name": "a_number",
"type": "bytes",
"logicalType": "decimal",
"precision": 4,
"scale": 4
}
Using the Avro console consumer, I see a message like this:
{"a_number": "tu0000°"}
Which I expect to equal 59.
Supposedly, the bytearray should be the twos-compliment of the number. I've tried using Python's struct module to decode it, but the values I get don't make any sense:
bs = 'tu0000°'.encode('utf8') # b'tx00xc2xb0'
struct.unpack('>l', bs)[0] / 1e4 # 15104.4784
How can I validate the message? Can I decode the string somehow, or has the Avro console consumer corrupted it?
python python-3.x apache-kafka avro
python python-3.x apache-kafka avro
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
edited Nov 21 '18 at 22:46
cricket_007
84.2k1147119
84.2k1147119
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
asked Nov 21 '18 at 22:14
z0rz0r
4,91713655
4,91713655
For comparison:struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'
– z0r
Nov 21 '18 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 '18 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 '18 at 23:01
add a comment |
For comparison:struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'
– z0r
Nov 21 '18 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 '18 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 '18 at 23:01
For comparison:
struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'– z0r
Nov 21 '18 at 22:16
For comparison:
struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'– z0r
Nov 21 '18 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 '18 at 22:38
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 '18 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 '18 at 23:01
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 '18 at 23:01
add a comment |
1 Answer
1
active
oldest
votes
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 '18 at 2:43
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421197%2fdecode-avro-decimal-as-bytearray-string%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 '18 at 2:43
add a comment |
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 '18 at 2:43
add a comment |
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
You seem to be going about this the Hard Way. The approach suggested by How to extract schema for avro file in python is to use:
reader = avro.datafile.DataFileReader(open('filename.avro',"rb"),avro.io.DatumReader())
schema = reader.meta
Single stepping in a debugger to see how the reader decodes your messages should get you closer to assembling a "raw" hand engineered decode.
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
answered Nov 22 '18 at 2:07
J_HJ_H
4,4441922
4,4441922
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 '18 at 2:43
add a comment |
Yep fair point. I don't have a.avrofile to read, but maybe I should just write a little Python script using anAvroConsumerinstead of the (presumably) Java-based console consumer to test it.
– z0r
Nov 22 '18 at 2:43
Yep fair point. I don't have a
.avro file to read, but maybe I should just write a little Python script using an AvroConsumer instead of the (presumably) Java-based console consumer to test it.– z0r
Nov 22 '18 at 2:43
Yep fair point. I don't have a
.avro file to read, but maybe I should just write a little Python script using an AvroConsumer instead of the (presumably) Java-based console consumer to test it.– z0r
Nov 22 '18 at 2:43
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421197%2fdecode-avro-decimal-as-bytearray-string%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
For comparison:
struct.pack('>l', int(59 * 1e4)) == b'x00tx00xb0'– z0r
Nov 21 '18 at 22:16
Are you sure that you want to encode high ASCII values as UTF-8? They will gain at least one additional byte, then, which will influence the total value. That said, at least you will get 4 bytes. Your sample string defines only 3.
– usr2564301
Nov 21 '18 at 22:38
@usr2564301 Yeah, I'm not sure - it does seem weird. The reason I chose UTF-8 is that that's what JSON uses to encode strings, and the output of the Avro console consumer is (apparently) JSON. I am a bit suss on that string; I would have expected it to write something in Base64 or so.
– z0r
Nov 21 '18 at 23:01