Is there a portable way to provide localization of a package distributed on PyPI?
Context:
This is kind of a followup to another question of mine.
I would like to provide localized versions of a package. Following the Python documentation, I have extracted a .pot file with pygettext, prepared a translation in a .po file, compiled it in a .mo file.
Everything is fine till there, and my package displays the translated messages.
But my final goal would be to make it available on PyPI. So I have done some research and found:
setuptools documentation: not even one single word about localization...
The Format of GNU MO Files
It explains that the format depends on the endianness of the platform where the file was generated. My understanding is that only the po files are portable...
What is the correct way to include localisation in python packages?
The answer is fully relevant and speaks of the setuptools/babel integration but:
- that integration only allows to build mo file and does not speak of their distribution
- author describes how they use it, no references for portability across systems
Babel: compile translation files when calling setup.py install
An interesting way, even if it requires the babel module on the target platform. Not so heavy but way heavier than my own package... In fact, the distributions contain only po files and they are compiled with babel at install time.
Question:
Is there a way to build platform specific wheels containing compiled mo files?
If not I will have to require babel on target and try to find my way through mo compilation at install time.
python localization setuptools
add a comment |
Context:
This is kind of a followup to another question of mine.
I would like to provide localized versions of a package. Following the Python documentation, I have extracted a .pot file with pygettext, prepared a translation in a .po file, compiled it in a .mo file.
Everything is fine till there, and my package displays the translated messages.
But my final goal would be to make it available on PyPI. So I have done some research and found:
setuptools documentation: not even one single word about localization...
The Format of GNU MO Files
It explains that the format depends on the endianness of the platform where the file was generated. My understanding is that only the po files are portable...
What is the correct way to include localisation in python packages?
The answer is fully relevant and speaks of the setuptools/babel integration but:
- that integration only allows to build mo file and does not speak of their distribution
- author describes how they use it, no references for portability across systems
Babel: compile translation files when calling setup.py install
An interesting way, even if it requires the babel module on the target platform. Not so heavy but way heavier than my own package... In fact, the distributions contain only po files and they are compiled with babel at install time.
Question:
Is there a way to build platform specific wheels containing compiled mo files?
If not I will have to require babel on target and try to find my way through mo compilation at install time.
python localization setuptools
add a comment |
Context:
This is kind of a followup to another question of mine.
I would like to provide localized versions of a package. Following the Python documentation, I have extracted a .pot file with pygettext, prepared a translation in a .po file, compiled it in a .mo file.
Everything is fine till there, and my package displays the translated messages.
But my final goal would be to make it available on PyPI. So I have done some research and found:
setuptools documentation: not even one single word about localization...
The Format of GNU MO Files
It explains that the format depends on the endianness of the platform where the file was generated. My understanding is that only the po files are portable...
What is the correct way to include localisation in python packages?
The answer is fully relevant and speaks of the setuptools/babel integration but:
- that integration only allows to build mo file and does not speak of their distribution
- author describes how they use it, no references for portability across systems
Babel: compile translation files when calling setup.py install
An interesting way, even if it requires the babel module on the target platform. Not so heavy but way heavier than my own package... In fact, the distributions contain only po files and they are compiled with babel at install time.
Question:
Is there a way to build platform specific wheels containing compiled mo files?
If not I will have to require babel on target and try to find my way through mo compilation at install time.
python localization setuptools
Context:
This is kind of a followup to another question of mine.
I would like to provide localized versions of a package. Following the Python documentation, I have extracted a .pot file with pygettext, prepared a translation in a .po file, compiled it in a .mo file.
Everything is fine till there, and my package displays the translated messages.
But my final goal would be to make it available on PyPI. So I have done some research and found:
setuptools documentation: not even one single word about localization...
The Format of GNU MO Files
It explains that the format depends on the endianness of the platform where the file was generated. My understanding is that only the po files are portable...
What is the correct way to include localisation in python packages?
The answer is fully relevant and speaks of the setuptools/babel integration but:
- that integration only allows to build mo file and does not speak of their distribution
- author describes how they use it, no references for portability across systems
Babel: compile translation files when calling setup.py install
An interesting way, even if it requires the babel module on the target platform. Not so heavy but way heavier than my own package... In fact, the distributions contain only po files and they are compiled with babel at install time.
Question:
Is there a way to build platform specific wheels containing compiled mo files?
If not I will have to require babel on target and try to find my way through mo compilation at install time.
python localization setuptools
python localization setuptools
asked Nov 13 at 16:38
Serge Ballesta
75.6k956130
75.6k956130
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
EDIT 7/12/2018:
After some work, I could build a specific package based on what is below in this answer. It can be used from other projects to automatically compile po files at build time through the magic of setuptools enty_points. It is now available on GitHUB (https://github.com/s-ball/mo_installer) and distributed on PyPI (https://pypi.org/project/mo_installer)
The researches that I did before asking the question gave me enough hints to reach a possible solution.
I can now say, thay is is possible to include a platform specific mo file in a wheel - unfortunately in my current solution the wheel gives no indication that it is platform specific. But the same solution allows to build a source distribution that build the mo file on the target platform.
Now for the details:
the tools needed to compile a mo file on the target:
Most solutions picked from Google or SO rely either on Babel or on the GNU gettext
msgfmt
program. But cPython tools include a pure Python modulemsgfmt.py
that is enough here. Unfortunately, this tool is often not installed by default in many Linux/Unix-like. My solution just includes a copy of that module (a mere 7k file) for 3.7.1 version. It looks like a very stable code (few changes in recent years) and it should work for any Python >= 3.3
the setuptools integration
The magic of setuptools is that the same build subcommand is internally used to build a binary wheel, to install with pip from a source package or to directly install with
python setup.py install
from a copy (git clone) of the full source package. So I provide abuild
subclass insetup.py
that generates the .mo files with their full path before calling the superclass method. I also use aMANIFEST.in
file to list the files that should be copied in a source distribution and apackage_data
setup argument to list what should go in a binary package or installation folder
run time usage
Provided the mo hierarchy to be installed under a knows package,
os.dirname(__file__)
called from a module of that package gives its parent folder
Code (assuming the msgfmt.py
file is copied under a tools_i18n
folder and that po files are under a src
folder):
in setup.py
...
sys.path.append(os.path.join(os.path.dirname(__file__), "tools_i18n"))
import msgfmt
from distutils.command.build import build as _build
class Builder(_build):
def run(self):
# po files in src folder are named domain_lang.po
po = re.compile(r"(.*)_(.*).po")
for file in os.listdir("src"):
m = po.match(file)
if m:
# create the LANG/LC_MESSAGES subdir of "locale"
path = os.path.join(self.build_lib, NAME, "locale",
m.group(2), "LC_MESSAGES")
os.makedirs(path, exist_ok=True)
# use msgfmt.py to compile the po file
msgfmt.make(os.path.join("src", file),
os.path.join(path, m.group(1) + ".mo"))
_build.run(self)
setup(
name=NAME,
...
package_data = { "": [..., "locale/*/*/*.mo"]}, # ensure .mo file are copied
cmdclass = {"build": Builder},
)
In MANIFEST.in
:
...
include src/*
include tools_i18n/*
To use the translations at run time:
locpath = os.path.dirname(__file__)
lang = locale.getdefaultlocale()[0] # to get platform default language, or whatever...
tr = gettext.translation("argparse", os.path.join(locpath, "locale"),
[lang], fallback=True)
A full project using this method is available at https://github.com/s-ball/i18nparse
Last but not least, after a more in depth reading of the GNU gettext doc, I can say that gettext can process mo files whatever their endianness:
MO files of any endianness can be used on any platform. When a MO file has an endianness other than the platform’s one, the 32-bit numbers from the MO file are swapped at runtime. The performance impact is negligible.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53285634%2fis-there-a-portable-way-to-provide-localization-of-a-package-distributed-on-pypi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
EDIT 7/12/2018:
After some work, I could build a specific package based on what is below in this answer. It can be used from other projects to automatically compile po files at build time through the magic of setuptools enty_points. It is now available on GitHUB (https://github.com/s-ball/mo_installer) and distributed on PyPI (https://pypi.org/project/mo_installer)
The researches that I did before asking the question gave me enough hints to reach a possible solution.
I can now say, thay is is possible to include a platform specific mo file in a wheel - unfortunately in my current solution the wheel gives no indication that it is platform specific. But the same solution allows to build a source distribution that build the mo file on the target platform.
Now for the details:
the tools needed to compile a mo file on the target:
Most solutions picked from Google or SO rely either on Babel or on the GNU gettext
msgfmt
program. But cPython tools include a pure Python modulemsgfmt.py
that is enough here. Unfortunately, this tool is often not installed by default in many Linux/Unix-like. My solution just includes a copy of that module (a mere 7k file) for 3.7.1 version. It looks like a very stable code (few changes in recent years) and it should work for any Python >= 3.3
the setuptools integration
The magic of setuptools is that the same build subcommand is internally used to build a binary wheel, to install with pip from a source package or to directly install with
python setup.py install
from a copy (git clone) of the full source package. So I provide abuild
subclass insetup.py
that generates the .mo files with their full path before calling the superclass method. I also use aMANIFEST.in
file to list the files that should be copied in a source distribution and apackage_data
setup argument to list what should go in a binary package or installation folder
run time usage
Provided the mo hierarchy to be installed under a knows package,
os.dirname(__file__)
called from a module of that package gives its parent folder
Code (assuming the msgfmt.py
file is copied under a tools_i18n
folder and that po files are under a src
folder):
in setup.py
...
sys.path.append(os.path.join(os.path.dirname(__file__), "tools_i18n"))
import msgfmt
from distutils.command.build import build as _build
class Builder(_build):
def run(self):
# po files in src folder are named domain_lang.po
po = re.compile(r"(.*)_(.*).po")
for file in os.listdir("src"):
m = po.match(file)
if m:
# create the LANG/LC_MESSAGES subdir of "locale"
path = os.path.join(self.build_lib, NAME, "locale",
m.group(2), "LC_MESSAGES")
os.makedirs(path, exist_ok=True)
# use msgfmt.py to compile the po file
msgfmt.make(os.path.join("src", file),
os.path.join(path, m.group(1) + ".mo"))
_build.run(self)
setup(
name=NAME,
...
package_data = { "": [..., "locale/*/*/*.mo"]}, # ensure .mo file are copied
cmdclass = {"build": Builder},
)
In MANIFEST.in
:
...
include src/*
include tools_i18n/*
To use the translations at run time:
locpath = os.path.dirname(__file__)
lang = locale.getdefaultlocale()[0] # to get platform default language, or whatever...
tr = gettext.translation("argparse", os.path.join(locpath, "locale"),
[lang], fallback=True)
A full project using this method is available at https://github.com/s-ball/i18nparse
Last but not least, after a more in depth reading of the GNU gettext doc, I can say that gettext can process mo files whatever their endianness:
MO files of any endianness can be used on any platform. When a MO file has an endianness other than the platform’s one, the 32-bit numbers from the MO file are swapped at runtime. The performance impact is negligible.
add a comment |
EDIT 7/12/2018:
After some work, I could build a specific package based on what is below in this answer. It can be used from other projects to automatically compile po files at build time through the magic of setuptools enty_points. It is now available on GitHUB (https://github.com/s-ball/mo_installer) and distributed on PyPI (https://pypi.org/project/mo_installer)
The researches that I did before asking the question gave me enough hints to reach a possible solution.
I can now say, thay is is possible to include a platform specific mo file in a wheel - unfortunately in my current solution the wheel gives no indication that it is platform specific. But the same solution allows to build a source distribution that build the mo file on the target platform.
Now for the details:
the tools needed to compile a mo file on the target:
Most solutions picked from Google or SO rely either on Babel or on the GNU gettext
msgfmt
program. But cPython tools include a pure Python modulemsgfmt.py
that is enough here. Unfortunately, this tool is often not installed by default in many Linux/Unix-like. My solution just includes a copy of that module (a mere 7k file) for 3.7.1 version. It looks like a very stable code (few changes in recent years) and it should work for any Python >= 3.3
the setuptools integration
The magic of setuptools is that the same build subcommand is internally used to build a binary wheel, to install with pip from a source package or to directly install with
python setup.py install
from a copy (git clone) of the full source package. So I provide abuild
subclass insetup.py
that generates the .mo files with their full path before calling the superclass method. I also use aMANIFEST.in
file to list the files that should be copied in a source distribution and apackage_data
setup argument to list what should go in a binary package or installation folder
run time usage
Provided the mo hierarchy to be installed under a knows package,
os.dirname(__file__)
called from a module of that package gives its parent folder
Code (assuming the msgfmt.py
file is copied under a tools_i18n
folder and that po files are under a src
folder):
in setup.py
...
sys.path.append(os.path.join(os.path.dirname(__file__), "tools_i18n"))
import msgfmt
from distutils.command.build import build as _build
class Builder(_build):
def run(self):
# po files in src folder are named domain_lang.po
po = re.compile(r"(.*)_(.*).po")
for file in os.listdir("src"):
m = po.match(file)
if m:
# create the LANG/LC_MESSAGES subdir of "locale"
path = os.path.join(self.build_lib, NAME, "locale",
m.group(2), "LC_MESSAGES")
os.makedirs(path, exist_ok=True)
# use msgfmt.py to compile the po file
msgfmt.make(os.path.join("src", file),
os.path.join(path, m.group(1) + ".mo"))
_build.run(self)
setup(
name=NAME,
...
package_data = { "": [..., "locale/*/*/*.mo"]}, # ensure .mo file are copied
cmdclass = {"build": Builder},
)
In MANIFEST.in
:
...
include src/*
include tools_i18n/*
To use the translations at run time:
locpath = os.path.dirname(__file__)
lang = locale.getdefaultlocale()[0] # to get platform default language, or whatever...
tr = gettext.translation("argparse", os.path.join(locpath, "locale"),
[lang], fallback=True)
A full project using this method is available at https://github.com/s-ball/i18nparse
Last but not least, after a more in depth reading of the GNU gettext doc, I can say that gettext can process mo files whatever their endianness:
MO files of any endianness can be used on any platform. When a MO file has an endianness other than the platform’s one, the 32-bit numbers from the MO file are swapped at runtime. The performance impact is negligible.
add a comment |
EDIT 7/12/2018:
After some work, I could build a specific package based on what is below in this answer. It can be used from other projects to automatically compile po files at build time through the magic of setuptools enty_points. It is now available on GitHUB (https://github.com/s-ball/mo_installer) and distributed on PyPI (https://pypi.org/project/mo_installer)
The researches that I did before asking the question gave me enough hints to reach a possible solution.
I can now say, thay is is possible to include a platform specific mo file in a wheel - unfortunately in my current solution the wheel gives no indication that it is platform specific. But the same solution allows to build a source distribution that build the mo file on the target platform.
Now for the details:
the tools needed to compile a mo file on the target:
Most solutions picked from Google or SO rely either on Babel or on the GNU gettext
msgfmt
program. But cPython tools include a pure Python modulemsgfmt.py
that is enough here. Unfortunately, this tool is often not installed by default in many Linux/Unix-like. My solution just includes a copy of that module (a mere 7k file) for 3.7.1 version. It looks like a very stable code (few changes in recent years) and it should work for any Python >= 3.3
the setuptools integration
The magic of setuptools is that the same build subcommand is internally used to build a binary wheel, to install with pip from a source package or to directly install with
python setup.py install
from a copy (git clone) of the full source package. So I provide abuild
subclass insetup.py
that generates the .mo files with their full path before calling the superclass method. I also use aMANIFEST.in
file to list the files that should be copied in a source distribution and apackage_data
setup argument to list what should go in a binary package or installation folder
run time usage
Provided the mo hierarchy to be installed under a knows package,
os.dirname(__file__)
called from a module of that package gives its parent folder
Code (assuming the msgfmt.py
file is copied under a tools_i18n
folder and that po files are under a src
folder):
in setup.py
...
sys.path.append(os.path.join(os.path.dirname(__file__), "tools_i18n"))
import msgfmt
from distutils.command.build import build as _build
class Builder(_build):
def run(self):
# po files in src folder are named domain_lang.po
po = re.compile(r"(.*)_(.*).po")
for file in os.listdir("src"):
m = po.match(file)
if m:
# create the LANG/LC_MESSAGES subdir of "locale"
path = os.path.join(self.build_lib, NAME, "locale",
m.group(2), "LC_MESSAGES")
os.makedirs(path, exist_ok=True)
# use msgfmt.py to compile the po file
msgfmt.make(os.path.join("src", file),
os.path.join(path, m.group(1) + ".mo"))
_build.run(self)
setup(
name=NAME,
...
package_data = { "": [..., "locale/*/*/*.mo"]}, # ensure .mo file are copied
cmdclass = {"build": Builder},
)
In MANIFEST.in
:
...
include src/*
include tools_i18n/*
To use the translations at run time:
locpath = os.path.dirname(__file__)
lang = locale.getdefaultlocale()[0] # to get platform default language, or whatever...
tr = gettext.translation("argparse", os.path.join(locpath, "locale"),
[lang], fallback=True)
A full project using this method is available at https://github.com/s-ball/i18nparse
Last but not least, after a more in depth reading of the GNU gettext doc, I can say that gettext can process mo files whatever their endianness:
MO files of any endianness can be used on any platform. When a MO file has an endianness other than the platform’s one, the 32-bit numbers from the MO file are swapped at runtime. The performance impact is negligible.
EDIT 7/12/2018:
After some work, I could build a specific package based on what is below in this answer. It can be used from other projects to automatically compile po files at build time through the magic of setuptools enty_points. It is now available on GitHUB (https://github.com/s-ball/mo_installer) and distributed on PyPI (https://pypi.org/project/mo_installer)
The researches that I did before asking the question gave me enough hints to reach a possible solution.
I can now say, thay is is possible to include a platform specific mo file in a wheel - unfortunately in my current solution the wheel gives no indication that it is platform specific. But the same solution allows to build a source distribution that build the mo file on the target platform.
Now for the details:
the tools needed to compile a mo file on the target:
Most solutions picked from Google or SO rely either on Babel or on the GNU gettext
msgfmt
program. But cPython tools include a pure Python modulemsgfmt.py
that is enough here. Unfortunately, this tool is often not installed by default in many Linux/Unix-like. My solution just includes a copy of that module (a mere 7k file) for 3.7.1 version. It looks like a very stable code (few changes in recent years) and it should work for any Python >= 3.3
the setuptools integration
The magic of setuptools is that the same build subcommand is internally used to build a binary wheel, to install with pip from a source package or to directly install with
python setup.py install
from a copy (git clone) of the full source package. So I provide abuild
subclass insetup.py
that generates the .mo files with their full path before calling the superclass method. I also use aMANIFEST.in
file to list the files that should be copied in a source distribution and apackage_data
setup argument to list what should go in a binary package or installation folder
run time usage
Provided the mo hierarchy to be installed under a knows package,
os.dirname(__file__)
called from a module of that package gives its parent folder
Code (assuming the msgfmt.py
file is copied under a tools_i18n
folder and that po files are under a src
folder):
in setup.py
...
sys.path.append(os.path.join(os.path.dirname(__file__), "tools_i18n"))
import msgfmt
from distutils.command.build import build as _build
class Builder(_build):
def run(self):
# po files in src folder are named domain_lang.po
po = re.compile(r"(.*)_(.*).po")
for file in os.listdir("src"):
m = po.match(file)
if m:
# create the LANG/LC_MESSAGES subdir of "locale"
path = os.path.join(self.build_lib, NAME, "locale",
m.group(2), "LC_MESSAGES")
os.makedirs(path, exist_ok=True)
# use msgfmt.py to compile the po file
msgfmt.make(os.path.join("src", file),
os.path.join(path, m.group(1) + ".mo"))
_build.run(self)
setup(
name=NAME,
...
package_data = { "": [..., "locale/*/*/*.mo"]}, # ensure .mo file are copied
cmdclass = {"build": Builder},
)
In MANIFEST.in
:
...
include src/*
include tools_i18n/*
To use the translations at run time:
locpath = os.path.dirname(__file__)
lang = locale.getdefaultlocale()[0] # to get platform default language, or whatever...
tr = gettext.translation("argparse", os.path.join(locpath, "locale"),
[lang], fallback=True)
A full project using this method is available at https://github.com/s-ball/i18nparse
Last but not least, after a more in depth reading of the GNU gettext doc, I can say that gettext can process mo files whatever their endianness:
MO files of any endianness can be used on any platform. When a MO file has an endianness other than the platform’s one, the 32-bit numbers from the MO file are swapped at runtime. The performance impact is negligible.
edited Dec 7 at 11:27
answered Nov 15 at 10:39
Serge Ballesta
75.6k956130
75.6k956130
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53285634%2fis-there-a-portable-way-to-provide-localization-of-a-package-distributed-on-pypi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown