inputenc for XeTeX and LuaTeX

[Will Robertson]

Hello,

As TeX Live 2009 approaches (too quickly) I'd like to again raise the
problems of inputenc in XeTeX and LuaTeX. Sorry to add to the
distractions of the members of the LaTeX3 team who are also busy
working on the expl3 code...

The problem is clear: LaTeX assumes 8-bit encodings and clobbers the
use of true UTF8 when newer engines are being used. This is an issue
because "\usepackage[utf8]{inputenc}" is the first thing that many new
users do when moving their documents over to the new engine.

Elie Roux and I have been writing packages to attempt to work around
the problem for LuaTeX and XeTeX, respectively. Our development code
is here:
<http://github.com/eroux/lualatex/tree/master>
<http://github.com/wspr/xetex-inputenc/tree/master>

We'd like inputenc to branch to these packages in the appropriate
context.

* * *

The LuaTeX version is more robust, but both packages attempt to "do
the right thing" when they're loaded, keeping UTF8 untouched and
processing the other encodings as best they can. In the LuaTeX
package, the original inputenc is still loaded and used for
transforming the chars.

In the XeTeX package, I take the lazy way out and simply read the
files in the different encoding, bypassing inputenc entirely.

Neither package attempts to do anything with the macros defined by
inputenc.sty or utf8ienc.dtx, but the main one we'd be interested in,
\DeclareUnicodeCharacter, would be easy enough to re-write for
"unicodeTeX".

However, neither package is really suitable for maintenance by the
LaTeX3 Project. Elie's macros for supporting LuaTeX are obviously very
new; furthermore, there're a few intertwined packages in his code tree
that wouldn't be appropriate, yet, to add to the stable LaTeX2e code.
And the XeTeX version isn't robust enough because there's no way to
reliably detect or branch based on \input. (It only works for packages
that use \@input for auto-generated files.)

* * *

Therefore, my proposal is to remove the responsibility of inputenc
from the LaTeX2e code for XeTeX and LuaTeX. I'm not sure exactly the
best way to achieve this; I'm going to recommend hard-coding it into
inputenc since that's the simplest option for now. E.g., at the top of
inputenc.sty:

% check for XeTeX:
\begingroup\expandafter\expandafter\expandafter\endgroup
\expandafter\ifx\csname XeTeXversion\endcsname\relax\else
\IfFileExists{xetex-inputenc.sty}{%
\DeclareOption*{\PassOptionToPackage{\CurrentOption}{xetex-
inputenc}}
\ProcessOptions
\RequirePackage{xetex-inputenc}
\expandafter\endinput
}{}
\fi

and something equivalent for LuaTeX.

Then the LaTeX team can "ignore" the problem for now (since it's plain
that LaTeX2e will never natively support unicode), and leave the
problem in the hands of the developers who are actively writing macro
packages for the new engines.

Like I said above, I'd like to resolve this for TeXLive 2009, and with
such a simple patch to inputenc I think we can do it.

Will

[Will Robertson]

(Karl replied off-list with some other stuff as well; I'm replying to
the pertinent bits on-list)

Post by Will Robertson
reliably detect or branch based on \input
I don't understand what this means. Do we need to extend the
engines in
some way?

LuaTeX can handle things just fine, since it can add a callback to the
file loading routine. The problem is that XeTeX always writes in UTF8
(having discussed with Jonathan the possibility of an output write
encoding, always using UTF8 is the more sensible way to do it).

When reading files in LaTeX, there are two possibilities:
1. It's an external file, like a section or chapter; this has to
follow the input encoding of the main file
2. It's an auto-generated file being read in a second compilation;
this has to be read in UTF8

The idea of xetex-inputenc is to set up the default encoding for the
main file and all subsequent files; it then has to detect when auto-
generated files (like .aux) are being read and insert the code to
temporarily switch back to UTF8.

This is what I do (and it works well, seemingly) for \@input, which is
used by the kernel for reading in auxiliary files that aren't added to
the \listfiles list. If only every package used that macro. Many
packages use \input instead, which is the problem, because LaTeX
redefines \input to *either* behave like the TeX primitive *or* to
take a braced argument. So some packages write "\input foo" and others
"\input{foo}". From the LaTeX POV, it's impossible to conditionally
call code based on the file input, then, because some packages don't
use the sane syntax.

However, all of the above is slightly obscure, since most people will
be using UTF8 input with XeTeX anyway. The fact that some edge cases
aren't covered doesn't worry me, personally, but does mean that this
code probably shouldn't become part of LaTeX2e proper. For me, the aim
is to fix the problem when users try
\usepackage[utf8]{inputenc}

We're just trying to support the other encodings because it's (mostly)
possible.

Post by Will Robertson
Like I said above, I'd like to resolve this for TeXLive 2009,
Do you have any idea of whether there will be another LaTeX release in
time? I was kind of expecting that there wouldn't be, although of
course it would be great if so.

I hadn't thought that far ahead :)
I was hoping at the very least it would be easy enough to get a
patched inputenc.sty into TL'09, even if it slightly by-passed the
normal LaTeX release regime. But perhaps I'm being naive.

Will

[Jörg Knappen]

Unicode has a feature called BOM (Byte order mark) which is a single character prepended to
a Unicode file. Thus, if we know that the file is in some Unicode encoding, we can detect which
one it is by checking for the byte order mark. If there is no byte oreder mark, assume ASCII or
some a non-Unicode 8 bit encoding.

This way a Unicode aware can do (partly) an automatic detection of the encoding.

The problem is that checking for the byte order mark introduces an incompatible
change to TeX, since, e.g., the UTF16 byte order mark looks like

[James Cloos]

Jörg> If there is no byte oreder mark, assume ASCII or some non-Unicode
Jörg> 8 bit encoding.

s/ASCII/UTF-8/ since utf-8 files should not have BOMs.

Otherwise, that is a good idea.

As for utf-8 or other, it may be useful to default to the character set
specified for the current $LOCALE. Maybe. :-/

-JimC

[Manuel Pégourié-Gonnard]

Post by James Cloos
As for utf-8 or other, it may be useful to default to the character set
specified for the current $LOCALE. Maybe. :-/

Please don't do anything in the compilation of the document depend on the
locale! It would completely ruin portability of the source files.

A file must be assumed to be either utf-8 (auxiliary file written by
XeTeX/LuaTeX) or in the encoding declared as the option of inputenc. Exactly
what xetex-inputenc and luatex-inputenc do.

The difficult problem is to guess when a file is an auxiliary file. I suppose
the heuristics for doing so will improve when the solution gets tested.

Manuel.

[Frank Mittelbach]

Post by Manuel Pégourié-Gonnard

Post by James Cloos
As for utf-8 or other, it may be useful to default to the character set
specified for the current $LOCALE. Maybe. :-/

Please don't do anything in the compilation of the document depend on the
locale! It would completely ruin portability of the source files.

perhaps. it might be a straight path into long-term disaster.On the other
hand the whole area is a disaster in the first place. When we started out with
inputenc in 2e I also thought that it is really good to keep the encoding with
the file (which you do by stating \usepackage[latin1]{inputenc} and the like)
and that worked for a while fairly good. But then OSes started to convert on
the fly so by cut-n-paste sometimes even on the same machine an old latin1 got
translated into something else (except for the string specifying the encoding
inside)... so ... not easy really

Post by Manuel Pégourié-Gonnard
A file must be assumed to be either utf-8 (auxiliary file written by
XeTeX/LuaTeX) or in the encoding declared as the option of inputenc. Exactly
what xetex-inputenc and luatex-inputenc do.
The difficult problem is to guess when a file is an auxiliary file. I suppose
the heuristics for doing so will improve when the solution gets tested.

how much guessing is really needed? Are you targetting an existing 2e env
unchanged or are you intending to design an interface that is robust if used?
Or something inbetween?

Couple of thoughts of the top of my head:

- new solution, ie not for 2e as such: design a proper interface for handling
internal auxilary file reading and writing. That would then have hooks to
maintain encoding. We certainly have to do something along those lines for expl3

- partial 2e solution: use \@input as a proposed way to read internal files
back in (as suggested by Will) and handle those correctly. booh at those
packages that don't use \@input but \input for their internal files (which
is is already wrong in 2e proper) and ask them to change or ignore them.

- possible 2e solution: steal \openout to always write
\InternallyWrittenFileHookToHandleWhatWeNeedToHandle
to the top of each such file; fix the cases where this is not appropriate
in 2e, such as filecontents env ... and wait for the packages to blow up
and fix those (probably only a few if any)

cheers
frank

ps interestingly enough, in 2e on top of anormal TeX engine that problem was
properly solved as we ensured that internally written files were always
written in LICR which is unicode in 7bit so it was always coming back
properly. That was at the cost of translating everything into LICR on input
(with active chars) but that was necessary anyway because of the different
8bit encodings around.

[Manuel Pégourié-Gonnard]

Post by Frank Mittelbach
perhaps. it might be a straight path into long-term disaster.On the other
hand the whole area is a disaster in the first place. When we started out with
inputenc in 2e I also thought that it is really good to keep the encoding with
the file (which you do by stating \usepackage[latin1]{inputenc} and the like)
and that worked for a while fairly good. But then OSes started to convert on
the fly so by cut-n-paste sometimes even on the same machine an old latin1 got
translated into something else (except for the string specifying the encoding
inside)... so ... not easy really

You're right. I still think stating the encoding inside the file is the saner
approach, even if it also has its drawbacks. (Btw, when posting on
newsgroups/mailing-lists I always take care to use [ascii]{inputenc} in order to
avoid such copy-paste problems.)

Post by Frank Mittelbach
how much guessing is really needed? Are you targetting an existing 2e env
unchanged or are you intending to design an interface that is robust if used?
Or something inbetween?

I think both are interesting, but concerning the present discussion (more
precisely, the packages Will mentions), I guess the target is more an existing
as-unchanged-as-possible 2e env. The idea is to make the transition between
8-bit TeX engines and Unicode TeX engine as easy as possible for the user.

Post by Frank Mittelbach
- possible 2e solution: steal \openout to always write
\InternallyWrittenFileHookToHandleWhatWeNeedToHandle
to the top of each such file; fix the cases where this is not appropriate
in 2e, such as filecontents env ... and wait for the packages to blow up
and fix those (probably only a few if any)

Probably an interesting approach.

Post by Frank Mittelbach
ps interestingly enough, in 2e on top of anormal TeX engine that problem was
properly solved as we ensured that internally written files were always
written in LICR which is unicode in 7bit so it was always coming back
properly. That was at the cost of translating everything into LICR on input
(with active chars) but that was necessary anyway because of the different
8bit encodings around.

At some point, Élie Roux tried to reproduce this approach for lua-inputenc
(converting from the input in fake utf-8, then loading the normal inputenc which
works correctly and correctly translates things to LICR on output). But it
breaks if a non-ascii character is however written to the file for some reason
(such as VerbatimOut, eg), and I'm afraid there is nothing to do about it.

Manuel.

[Heiko Oberdiek]

Post by Frank Mittelbach

Post by Manuel Pégourié-Gonnard

Post by James Cloos
As for utf-8 or other, it may be useful to default to the character set
specified for the current $LOCALE. Maybe. :-/

Please don't do anything in the compilation of the document depend on the
locale! It would completely ruin portability of the source files.

perhaps. it might be a straight path into long-term disaster.On the other
hand the whole area is a disaster in the first place. When we started out with
inputenc in 2e I also thought that it is really good to keep the encoding with
the file (which you do by stating \usepackage[latin1]{inputenc} and the like)
and that worked for a while fairly good. But then OSes started to convert on
the fly so by cut-n-paste sometimes even on the same machine an old latin1 got
translated into something else (except for the string specifying the encoding
inside)... so ... not easy really

It's addressed by package selinput.

Post by Frank Mittelbach
ps interestingly enough, in 2e on top of anormal TeX engine that problem was
properly solved as we ensured that internally written files were always
written in LICR which is unicode in 7bit so it was always coming back
properly.

For similar reasons I have converted arbitrary user data into hex when
writing them to auxiliary files and decoded them reading back.
This solves problems like unmatched curly braces, characters with
special catcodes, ...

Yours sincerely
Heiko <oberdiek-***@public.gmane.org>

[Will Robertson]

Post by Frank Mittelbach

Post by Manuel Pégourié-Gonnard

Post by James Cloos
As for utf-8 or other, it may be useful to default to the
character set
specified for the current $LOCALE. Maybe. :-/

Please don't do anything in the compilation of the document depend on the
locale! It would completely ruin portability of the source files.

perhaps. it might be a straight path into long-term disaster.On the other
hand the whole area is a disaster in the first place. When we
started out with
inputenc in 2e I also thought that it is really good to keep the encoding with
the file (which you do by stating \usepackage[latin1]{inputenc} and the like)
and that worked for a while fairly good. But then OSes started to convert on
the fly so by cut-n-paste sometimes even on the same machine an old latin1 got
translated into something else (except for the string specifying the encoding
inside)... so ... not easy really

Yep, agreed that dealing with encodings is annoying :)

Post by Frank Mittelbach

Post by Manuel Pégourié-Gonnard
A file must be assumed to be either utf-8 (auxiliary file written by
XeTeX/LuaTeX) or in the encoding declared as the option of
inputenc. Exactly
what xetex-inputenc and luatex-inputenc do.
The difficult problem is to guess when a file is an auxiliary file. I suppose
the heuristics for doing so will improve when the solution gets tested.

how much guessing is really needed? Are you targetting an existing 2e env
unchanged or are you intending to design an interface that is robust if used?
Or something inbetween?

Almost entirely the first.

Neither package needs to guess anything; the problem is that there's
just no way to know if \input refers to a generated file or a user file.

The XeTeX solution simply patches \@input. The LuaTeX solution does
something similar and allows customisation so that certain files or
file extensions can be treated as if they were \@input rather than
\input.

Post by Frank Mittelbach
- new solution, ie not for 2e as such: design a proper interface for handling
internal auxilary file reading and writing. That would then have hooks to
maintain encoding. We certainly have to do something along those lines for expl3

Yep.

Post by Frank Mittelbach
internal files
back in (as suggested by Will) and handle those correctly. booh at those
is is already wrong in 2e proper) and ask them to change or ignore them.

Yep. I hadn't thought of it before, but we could add a note to the
documentation explicitly discussing this behaviour. Using \@input for
internally-generated files is implicit in what it does but there's no-
where (that I know of) that states it plainly.

Note that even the kernel uses \input on the .aux file somewhere :)

Post by Frank Mittelbach
- possible 2e solution: steal \openout to always write
\InternallyWrittenFileHookToHandleWhatWeNeedToHandle
to the top of each such file; fix the cases where this is not appropriate
in 2e, such as filecontents env ... and wait for the packages to blow up
and fix those (probably only a few if any)

Nice idea, probably will work; but the return on investment is too low
(for me at least). I expect non-UTF8 input in Xe(La)TeX documents to
be hardly ever used. And we can always foist off the responsibility on
the packages that don't work because of \input v. \@input.

* * *

So, assuming we want to do something about the whole situation (I hope
so), how open are you to the idea of adding branching to inputenc to
load packages that aren't under the LaTeX team's control? I'm more
than happy printing a big warning telling users what's going on.

Thanks for the comments,
Will

[Elie Roux]

Hello,

I'm the developer of luainputenc (inputenc adapted for LuaTeX). Sorry to
come back to this topic, but I think an answer to Will's question would
be necessary soon... inputenc needs adjustments in order to work with
LuaTeX and XeTeX. I can build a new package called luainputenc, but I
think it will be confusing for the user, and it makes LuaLaTeX unable to
compile old LaTeX documents, so it would be really great if some
modifications to inputenc were made. I think Will's idea is the best
one, having inputenc.sty being basically:

\RequirePackage{ifluatex}
\RequirePackage{iflxetex}
\ifluatex
\RequirePackage{luainputenc}
\else
\ifxetex
\RequirePackage{xetexinputenc}
\else
\RequirePackage{legacyinputenc}
\fi
\fi

You wouldn't have to maintain anything else than this small file you'll
never have to change, and the legacyinputenc, and all the new things
would be maintained by Will and me.

Also I have a legal problem: is it possible for me to distribute a
modified copy of inputenc.sty (called luainputenc.sty), and of utf8.def
(called lutf8.def) with the lppl?

Thank you in advance,

[Manuel Pégourié-Gonnard]

Hi all,

(Élie, I'm going to repeat here mostly what I said in fctt, expanding a bit.)

Post by Elie Roux
I'm the developer of luainputenc (inputenc adapted for LuaTeX). Sorry to
come back to this topic, but I think an answer to Will's question would
be necessary soon... inputenc needs adjustments in order to work with
LuaTeX and XeTeX.

I'm sorry but I disagree on this point. I don't think it is urgent to adjust the
real inputenc. LuaTeX is still in early beta, so it should be used only by
somehow experienced users, or at least users aware of what they are doing. Such
users should be able to understand that they should either encode their source
in utf-8 (and don't load any inputenc) or load luainputenc.

By the way, people using XeTeX just encode their sources in utf-8 and skip
loading inputenc, and seem to be happy with it.

Post by Elie Roux
I can build a new package called luainputenc, but I
think it will be confusing for the user, and it makes LuaLaTeX unable to
compile old LaTeX documents,

Ok, so this is the point. You expect LuaTeX to become the default engine some
day, and for this to happen, backwards compatibility will indeed be very
important. But this will not happen before (at least) 2012 (announced date for
the first stable version). So I see no harm in waiting a bit before implementing
backward compatibility solutions.

Moreover, luainputenc, as far as I understand it, is currently not able to
compile correctly 100% of the old documents written for LaTeX over 8-bit TeX,
mainly due to the generated files problem. So IMO it is best to keep it as a
separate package while the issues are not solved. It also needs some (more)
real-life testing before being integrated in inputenc, IMO.

A important point is, people need to know the difference between what is stable
and what isn't. They expect core packages maintained by the LaTeX team to be
stable and work in a 100% compatible way. This expectation should be satisfied.

By the way, while we're at suggesting things concerning inputenc and Unicode TeX
engines, my suggestion would be to detect those engines and issue a meaningfull
error message (not just a warning) like "You're running LuaTeX. inputenc can't
currently work on Unicode variants of TeX such as LuaTeX. Please consult
http://www.tex.ac.uk/cgi-bin/texfaq2html?label=lualatex for how to use LaTeX
with LuaTeX." and refusing to load.

Of course the above message is just a suggestion and we would have to agree with
Robin to maintain FAQ entries for lualatex and xelatex, explaining the
state-of-the art ways of using LaTeX on top of such engines, and giving minimal
preamble examples etc. (Also mentionning fontspec for XeTeX, etc.) The authors
of LaTeX packages related to XeTeX/LuaTeX support should send updates to Robin
every time they do a significant user-visible change in their package.

In short, inputenc on Unicode TeX engines should IMO either just work or just
fail. This it can't just work now (or at least this needs some testing to know),
better make it just fail in a helpful way.

Manuel.

[David Kastrup]

Post by Manuel Pégourié-Gonnard
Hi all,
(Élie, I'm going to repeat here mostly what I said in fctt, expanding a bit.)

Post by Elie Roux
I'm the developer of luainputenc (inputenc adapted for LuaTeX). Sorry
to come back to this topic, but I think an answer to Will's question
would be necessary soon... inputenc needs adjustments in order to
work with LuaTeX and XeTeX.

I'm sorry but I disagree on this point. I don't think it is urgent to
adjust the real inputenc. LuaTeX is still in early beta, so it should
be used only by somehow experienced users, or at least users aware of
what they are doing.. Such users should be able to understand that
they should either encode their source in utf-8 (and don't load any
inputenc) or load luainputenc.

I disagree. The development or rather the update cycle of LaTeX is so
slow that we don't want it to impede progress.

My personal opinion is that there should be either an inputenc.sty or
default definition files in the load path of lualatex (or whatever it is
called). I see one problem with TeXlive/kpathsea: it is quite
conceivable that people want to have the command "latex" refer to
lualatex by default, and this should imply the lualatex search paths. I
don't think this is easy to set up at the current point of time.

However, with regard to the LaTeX team, I think that it would be in
order if we sanctioned the use of the "inputenc.sty" moniker for
particular lualatex/xelatex specific trees in TeX distributions. Even
if we decide that it does not make sense for the LaTeX team to drive and
maintain their development and employment.

Post by Manuel Pégourié-Gonnard
A important point is, people need to know the difference between what
is stable and what isn't. They expect core packages maintained by the
LaTeX team to be stable and work in a 100% compatible way. This
expectation should be satisfied.

I don't see how this expectation would be violated by extensions only
triggered by the use of something as experimental as lualatex.

[Manuel Pégourié-Gonnard]

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
I'm sorry but I disagree on this point. I don't think it is urgent to
adjust the real inputenc. LuaTeX is still in early beta, so it should
be used only by somehow experienced users, or at least users aware of
what they are doing.. Such users should be able to understand that
they should either encode their source in utf-8 (and don't load any
inputenc) or load luainputenc.

I disagree. The development or rather the update cycle of LaTeX is so
slow that we don't want it to impede progress.

No one (at least not me) wants to impede progress. luainputenc and xetexinputenc
are interesting and I'm glad they are being developed. The question is: Would it
be a real progress to have them called 'inputenc' (by whatever mechanism such
as kpse search path or something like Élie's suggestion)?

In the current state of the packages, I don't think it would be a progress to
change the name to inputenc. And I also think there's no need to hurry deciding
whether it is a progress or not.

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
A important point is, people need to know the difference between what
is stable and what isn't. They expect core packages maintained by the
LaTeX team to be stable and work in a 100% compatible way. This
expectation should be satisfied.

I don't see how this expectation would be violated by extensions only
triggered by the use of something as experimental as lualatex.

Well, as a user, I usually expect things with the same name to behave the same.
Currently inputenc and luainputenc just don't.

If the user is aware enough of what's going on, she is probably able to
understand that she should use luainputenc instead of inputenc (or, much better,
(re-)encode her source as utf-8 and stop using *inputenc).

Manuel.

[Elie Roux]

Post by Manuel Pégourié-Gonnard
No one (at least not me) wants to impede progress. luainputenc and xetexinputenc
are interesting and I'm glad they are being developed. The question is: Would it
be a real progress to have them called 'inputenc' (by whatever mechanism such
as kpse search path or something like Élie's suggestion)?
In the current state of the packages, I don't think it would be a progress to
change the name to inputenc. And I also think there's no need to hurry deciding
whether it is a progress or not.

In the future LuaTeX will probably replace pdfTeX, so inputenc will load
luainputenc. The question is not really to decide if it's good or not,
it's to decide when we will do it. And I still can't really see why we
should wait three more years...

Post by Manuel Pégourié-Gonnard
If the user is aware enough of what's going on, she is probably able to
understand that she should use luainputenc instead of inputenc (or, much better,
(re-)encode her source as utf-8 and stop using *inputenc).

Just to repeat myself: (lua)inputenc is not optional in most cases, so
the user will have to load luainputenc. This means that people working
with LuaLaTeX will have to change their documents in 2012, replacing
luainputenc by inputenc... I know it's not 100% stable so users must
know what they're doing, but it still seems more annoying than the
miracle solution for stability...

[Manuel Pégourié-Gonnard]

Post by Elie Roux
In the future LuaTeX will probably replace pdfTeX, so inputenc will load
luainputenc. The question is not really to decide if it's good or not,

Hmm...

Post by Elie Roux
it's to decide when we will do it. And I still can't really see why we
should wait three more years...

I don't mean waiting for three years. I just mean waiting enough to be able to
tidy up problems. Probably one year will be enough. Be a few months (current
time of luainputenc development/testing, iirc) are probably not enough.

Post by Elie Roux
Just to repeat myself: (lua)inputenc is not optional in most cases, so
the user will have to load luainputenc. This means that people working
with LuaLaTeX will have to change their documents in 2012, replacing
luainputenc by inputenc...

And that's perfectly ok if they need so, because you know you possibly have to
change a few names sometimes when you're using something under developement.
(And it doesn't mean only LuaTeX: I sometimes have to change my documents in
order to make them compile with a newer version of Koma-script, and I accept
that. If I don't want to have to worry about it, I use only the standard
classes, and I'm happy I have this choice too.)

Anyway, I see no reason why luainputenc should stop working with this name if it
is later available as inputenc. A package can perfectly have multiple names.

Post by Elie Roux
I know it's not 100% stable so users must
know what they're doing, but it still seems more annoying than the
miracle solution for stability...

Well, it seems we disagree on what the future will be. In my future, I want to
be able to compile utf-8 encoded sources using Unicode fonts without loading
some sort of inputenc and making a bunch af character actives.

Manuel.

[David Kastrup]

Post by Manuel Pégourié-Gonnard

Post by Elie Roux
In the future LuaTeX will probably replace pdfTeX, so inputenc will load
luainputenc. The question is not really to decide if it's good or not,

Hmm...

Post by Elie Roux
it's to decide when we will do it. And I still can't really see why we
should wait three more years...

I don't mean waiting for three years. I just mean waiting enough to be
able to tidy up problems.

Could you name one such problem affecting non-LuaTeX users?

[Manuel Pégourié-Gonnard]

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
I don't mean waiting for three years. I just mean waiting enough to be
able to tidy up problems.

Could you name one such problem affecting non-LuaTeX users?

I'm afraid I can't understand you question. luainputenc is meant to be used on
top of LuaTeX, so every problem with it is for LuaTeX users.

I'll discuss privately the problem I'll have compiling my old documents with
luatex+luainputenc with Élie (and with other packages authors if other packages
are involved), and try to propose solutions when I can, but it is indeed not the
place to do so as many have observed.

My point is simply that for various reasons, luainputenc is not (yet) a stable
drop-in replacement for inputenc, and so should not yet be called inputenc.
Rather, development and testing of luainputenc should continue until it becomes
possibly ready to be transparently used in most cases. The value of "most" is to
be evaluated carefully, and if it is large enough, then it will make sens to
have \usepackage[xxx]{inputenc} actually call luainputenc (via whatever
mechanism) when running LuaTeX.

Obviously, this is just my opinion. I'm sorry I can't express it clearly enough,
so that some people think I am opposed to progress while others see personal
attacks where they are not intended. I think I'll stop this public discussion
here, and continue discussing technical issues and their possible solutions in
private, since I hope a private discussion is more likely to help actually solve
real problems.

Manuel.

[David Kastrup]

Post by Manuel Pégourié-Gonnard

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
I don't mean waiting for three years. I just mean waiting enough to be
able to tidy up problems.

Could you name one such problem affecting non-LuaTeX users?

I'm afraid I can't understand you question. luainputenc is meant to be
used on top of LuaTeX, so every problem with it is for LuaTeX users.

Exactly.

Post by Manuel Pégourié-Gonnard
My point is simply that for various reasons, luainputenc is not (yet)
a stable drop-in replacement for inputenc, and so should not yet be
called inputenc.

With that reasoning, inputenc is a completely non-working implementation
of inputenc on LuaTeX and so should not yet be called inputenc.

I don't see the advantage of a non-working stable inputenc in the LuaTeX
namespace over a somewhat working non-stable inputenc.

Post by Manuel Pégourié-Gonnard
Rather, development and testing of luainputenc should continue until
it becomes possibly ready to be transparently used in most cases. The
value of "most" is to be evaluated carefully, and if it is large
enough, then it will make sens to have \usepackage[xxx]{inputenc}
actually call luainputenc (via whatever mechanism) when running
LuaTeX.

But it makes even less sense to call the canonical inputenc when running
LuaTeX because it will just break. So where is the point?

Post by Manuel Pégourié-Gonnard
Obviously, this is just my opinion. I'm sorry I can't express it
clearly enough, so that some people think I am opposed to progress
while others see personal attacks where they are not intended. I think
I'll stop this public discussion here, and continue discussing
technical issues and their possible solutions in private, since I hope
a private discussion is more likely to help actually solve real
problems.

What is the real problem that you solve by having LuaTeX choke on the
original inputenc package?

That it will choke in the same manner 5 years from now? I just don't
understand what advantages you expect from staying with the status quo.

[Elie Roux]

Post by Manuel Pégourié-Gonnard
Well, it seems we disagree on what the future will be. In my future, I want to
be able to compile utf-8 encoded sources using Unicode fonts without loading
some sort of inputenc and making a bunch af character actives.

No, we agree on the future, and you'll be able to do so with
luainputenc, don't worry... Please wait to know what my solution is
before criticizing my work like this...

[Manuel Pégourié-Gonnard]

Post by Elie Roux

Post by Manuel Pégourié-Gonnard
Well, it seems we disagree on what the future will be. In my future, I want to
be able to compile utf-8 encoded sources using Unicode fonts without loading
some sort of inputenc and making a bunch af character actives.

No, we agree on the future, and you'll be able to do so with
luainputenc, don't worry... Please wait to know what my solution is
before criticizing my work like this...

I was not criticizing your work, just stating that IMHO there should be no need
to use an input-encoding-oriented package in order to compile an utf-8-encoded
source with LuaLaTeX.

M.

[Will Robertson]

Hi all,

Don't forget that for utf8 encoded documents, the luatex and xetex
inputenc packages do *not* mess around with active characters or
whatever -- they prevent the legacy inputenc from doing so in the
first place.

However, given any reservations I'd be happy enough with adapting
inputenc to simply abort with an error if it's loaded with the new
engines.

More comments requested, I possible.

Will

(Sent from my phone.)

Post by Manuel Pégourié-Gonnard

Post by Elie Roux
In the future LuaTeX will probably replace pdfTeX, so inputenc will load
luainputenc. The question is not really to decide if it's good or not,

Hmm...

Post by Elie Roux
it's to decide when we will do it. And I still can't really see why we
should wait three more years...

I don't mean waiting for three years. I just mean waiting enough to be able to
tidy up problems. Probably one year will be enough. Be a few months (current
time of luainputenc development/testing, iirc) are probably not enough.

Post by Elie Roux
Just to repeat myself: (lua)inputenc is not optional in most cases, so
the user will have to load luainputenc. This means that people working
with LuaLaTeX will have to change their documents in 2012, replacing
luainputenc by inputenc...

And that's perfectly ok if they need so, because you know you
possibly have to
change a few names sometimes when you're using something under
developement.
(And it doesn't mean only LuaTeX: I sometimes have to change my documents in
order to make them compile with a newer version of Koma-script, and I accept
that. If I don't want to have to worry about it, I use only the standard
classes, and I'm happy I have this choice too.)
Anyway, I see no reason why luainputenc should stop working with this name if it
is later available as inputenc. A package can perfectly have
multiple names.

Post by Elie Roux
I know it's not 100% stable so users must
know what they're doing, but it still seems more annoying than the
miracle solution for stability...

Well, it seems we disagree on what the future will be. In my future, I want to
be able to compile utf-8 encoded sources using Unicode fonts without loading
some sort of inputenc and making a bunch af character actives.
Manuel.

[David Kastrup]

Post by Manuel Pégourié-Gonnard
Hi all,
Don't forget that for utf8 encoded documents, the luatex and xetex
inputenc packages do *not* mess around with active characters or
whatever -- they prevent the legacy inputenc from doing so in the
first place.

Wouldn't that preclude auxiliary files being written in LICR?

[Will Robertson]

(Sent from my phone.)

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
Hi all,
Don't forget that for utf8 encoded documents, the luatex and xetex
inputenc packages do *not* mess around with active characters or
whatever -- they prevent the legacy inputenc from doing so in the
first place.

Wouldn't that preclude auxiliary files being written in LICR?

Licr support still works (or can still work) because the appropriate
macros can still be defined (to eventually turn into the correct
unicode character). However, user input is never transformed into a
licr form internally.

But you do lose the advantages of the licr -- being able to throw an
error, or recover by faking it, if the font doesn't contain a
necessary glyph.

Will

[Elie Roux]

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
Hi all,
Don't forget that for utf8 encoded documents, the luatex and xetex
inputenc packages do *not* mess around with active characters or
whatever -- they prevent the legacy inputenc from doing so in the
first place.

Wouldn't that preclude auxiliary files being written in LICR?

Well, I changed luainputenc a bit since the last time I released it, now
it activates characters if a font encoding other than EU2 is loaded with
fontenc. So the behaviour of LuaLaTeX with (lua)inputenc is the same as
the behaviour of LaTeX with inputenc, for utf8 (if not loaded with EU2)
and 8-bit encodings.

The problem with aux files is only present for 8-bit encoding, there is
no problem with aux files when used with utf8.

But once again, I think technical issues about LuaTeX+LaTeX must be
addressed on luatex at tug dot org, the main question for this list is:
do you agree to change inputenc? If so how?

[James Sullivan]

Thanks

Jim (James D.) SULLIVAN | e-mail: James.Sullivan2.ctr-qDfX9Hp8+***@public.gmane.org
AFRL/RVBX | Phone: [phone]; 617/775 1121(c)
29 Randolph Road | URL: http://people.bu.edu/sullivan
Hanscom AFB, MA 01731-3010 | e-mail: sullivan-***@public.gmane.org

[Robin Fairbairns]

Post by Manuel Pégourié-Gonnard

Post by Elie Roux
In the future LuaTeX will probably replace pdfTeX, so inputenc will load
luainputenc. The question is not really to decide if it's good or not,

Hmm...

Post by Elie Roux
it's to decide when we will do it. And I still can't really see why we
should wait three more years...

I don't mean waiting for three years. I just mean waiting enough to be able to
tidy up problems. Probably one year will be enough. Be a few months (current
time of luainputenc development/testing, iirc) are probably not enough.

i see no problem whatever with elie working on blue-sky developments.
a multi-engine inputenc-like package seems to me a fine way forward
(though for the time being i wouldn't call it inputenc, but put an "x"
in the name somewhere to designate an extension).

if elie's code is well enough documented, one might even hope that a
possible foundation for future development is being laid down. but even
something little better than a straw man has the potential to move our
thinking forward.

and of course, if luatex changes under elie's feet, and a chunk of his
code becomes irrelevant/broken, elie is unlikely to be surprised, since
every time anyone mentions luatex they tend to bracket it with
disclaimers about the likelihood of the sky falling (by toutatis).

perhaps we should have a separate list for elie's work: people who feel
the need to point out that today's code may become tomorrow's bugfest
... simply don't have to join the list.

robin

[Manuel Pégourié-Gonnard]

Post by Robin Fairbairns
i see no problem whatever with elie working on blue-sky developments.

Neither do I, maybe it wasn't clear enough form my previous messages. I would
see a problem however if the package was named just 'inputenc' (at least so
early) because I'm afraid it would cause confusion.

So to be clear : luainputenc solves a lot of problems even if it can't solve
them all atm, and I appreciate it (and the rest of Élie's work, while we're at
it). Renaming it too early as just 'inputenc' addresses a problem which is not
yet urgent, and can possibly cause some confusion, so it should be carefully
weighted whether renaming would solve more problems that it could possibly
create before taking any action.

Post by Robin Fairbairns
perhaps we should have a separate list for elie's work: people who feel
the need to point out that today's code may become tomorrow's bugfest
... simply don't have to join the list.

I don't worry about tomorrow's bugfest because I am confident that Élie will
take care of it. I only worry about what's the best way to help users avoid
problems, or at least, know which problems are to be expected, while switching
to lualatex.

Manuel.

[Ulrike Fischer]

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
I'm sorry but I disagree on this point. I don't think it is urgent to
adjust the real inputenc. LuaTeX is still in early beta, so it should
be used only by somehow experienced users, or at least users aware of
what they are doing.. Such users should be able to understand that
they should either encode their source in utf-8 (and don't load any
inputenc) or load luainputenc.

I disagree. The development or rather the update cycle of LaTeX is so
slow that we don't want it to impede progress.

Like Manual I don't think that it is a good idea to load
xetexinputenc or luatexinputenc (or whatever the packages are
called) through inputenc. This packages are currently at best in the
beta-stadium. They haven't been tested much, the problem with
auxiliary files with non-ascii chars generated by the engine hasn't
been solved yet.

inputenc and fontenc are base packages. They shouldn't load
experimental packages which can change from one day to the next and
where it isn't clear that they will stay.

Nobody want to impede progress. But the progress can be easily moved
in dedicated packages until they can be considered as stable.

And meanwhile inputenc could contain a message
"don't use inputenc with engine X, read ... to learn about
alternatives".

[David Kastrup]

Post by Ulrike Fischer

Post by David Kastrup

Post by Manuel Pégourié-Gonnard
I'm sorry but I disagree on this point. I don't think it is urgent to
adjust the real inputenc. LuaTeX is still in early beta, so it should
be used only by somehow experienced users, or at least users aware of
what they are doing.. Such users should be able to understand that
they should either encode their source in utf-8 (and don't load any
inputenc) or load luainputenc.

I disagree. The development or rather the update cycle of LaTeX is so
slow that we don't want it to impede progress.

Like Manual I don't think that it is a good idea to load
xetexinputenc or luatexinputenc (or whatever the packages are
called) through inputenc. This packages are currently at best in the
beta-stadium. They haven't been tested much, the problem with
auxiliary files with non-ascii chars generated by the engine hasn't
been solved yet.
inputenc and fontenc are base packages. They shouldn't load
experimental packages which can change from one day to the next and
where it isn't clear that they will stay.

When the alternative is that they will just break, I don't see the
advantage.

[David Carlisle]

Post by David Kastrup
When the alternative is that they will just break, I don't see the
advantage.

As far as I can see it is consistent with the LPPL if *-tex engines
have a modified input path that picks up a modified inputenc.sty, so
long as any modifified packages, in the words of LPPL

Post by David Kastrup
clearly and unambiguously identifies itself as a modified version

So there is no need for the base packages to have explict reference to
experimental engines, but also no need for packages that are known to
fail on those engines being loaded by default.

David


________________________________________________________________________
The Numerical Algorithms Group Ltd is a company registered in England
and Wales with company number 1249803. The registered office is:
Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.

This e-mail has been scanned for all viruses by Star. The service is
powered by MessageLabs.
________________________________________________________________________

[Ulrike Fischer]

Post by David Kastrup

Post by Ulrike Fischer
Like Manual I don't think that it is a good idea to load
xetexinputenc or luatexinputenc (or whatever the packages are
called) through inputenc. This packages are currently at best in the
beta-stadium. They haven't been tested much, the problem with
auxiliary files with non-ascii chars generated by the engine hasn't
been solved yet.
inputenc and fontenc are base packages. They shouldn't load
experimental packages which can change from one day to the next and
where it isn't clear that they will stay.

When the alternative is that they will just break, I don't see the
advantage.

A lot of package don't work with some engines or need adjustments if
you use another engine or driver. You can't use pstricks with
pdflatex (meaning pdf output), fontspec gives errors when not used
with xetex. But I don't see any requests to write a pstricks wrapper
which loads e.g. pgf when used with pdf-output (and translate all
the pstricks syntax). And the xelatex users handle the current
situation with inputenc quite fine.

On the other side changing inputenc so that it loads the new engine
specific package introduce the danger of real breaks: the new
packages are still unstable, "not robust" and perhaps buggy. At
least in the case of xelatex bugs could affect real users.

I don't have a problem to tell xelatex users (and later lualatex
users) not to use inputenc because the package is (currently or for
ever) not suited for this engine. But I at least wouldn't like to
have to tell them to avoid the package because it loads behind their
back buggy or incomplete or non-working or untested code.

[Elie Roux]

Post by Manuel Pégourié-Gonnard
I'm sorry but I disagree on this point. I don't think it is urgent to adjust the
real inputenc. LuaTeX is still in early beta, so it should be used only by
somehow experienced users, or at least users aware of what they are doing. Such
users should be able to understand that they should either encode their source
in utf-8 (and don't load any inputenc) or load luainputenc.

With the new solution, people have to load luainputenc, it's not
optional anymore.

Post by Manuel Pégourié-Gonnard
By the way, people using XeTeX just encode their sources in utf-8 and skip
loading inputenc, and seem to be happy with it.

I think LuaTeX does not target the same users. With luainputenc and
fontenc working, LuaTeX can start targetting the average user, as the
old LaTeX documents will be working with LuaLaTeX (if inputenc loads
luainputenc). XeTeX will never allow old documents to be compiled with
it, as most input encoding problems have no solution.

Post by Manuel Pégourié-Gonnard
Ok, so this is the point. You expect LuaTeX to become the default engine some
day, and for this to happen, backwards compatibility will indeed be very
important. But this will not happen before (at least) 2012 (announced date for
the first stable version). So I see no harm in waiting a bit before implementing
backward compatibility solutions.

And I see no harm in proposing a working LuaLaTeX... Of course, as
LuaTeX is not stable, LuaLaTeX won't be stable... but remember that
people call LuaLaTeX with the command "lualatex", so they know what they
are doing.

Post by Manuel Pégourié-Gonnard
Moreover, luainputenc, as far as I understand it, is currently not able to
compile correctly 100% of the old documents written for LaTeX over 8-bit TeX,
mainly due to the generated files problem. So IMO it is best to keep it as a
separate package while the issues are not solved. It also needs some (more)
real-life testing before being integrated in inputenc, IMO.

LuaTeX can only write UTF-8, so people needing to write in files in
8-bit without LICR (meaning not a lot of people) will have problems if
they do it in home-made package (I solved the problem for all files
generated by packages in TeXLive), so the problem concerns noone except
very advanced TeX users... I know you're one of them, but for the
average user it will be 99.9999% backward compatible. It doesn't mean
that it's 100% stable though, as LuaTeX will continue to evolve.

Post by Manuel Pégourié-Gonnard
A important point is, people need to know the difference between what is stable
and what isn't. They expect core packages maintained by the LaTeX team to be
stable and work in a 100% compatible way. This expectation should be satisfied.

Well, I'm not sure pdfTeX and TeX82 were 100% compatible neither...

Post by Manuel Pégourié-Gonnard
By the way, while we're at suggesting things concerning inputenc and Unicode TeX
engines, my suggestion would be to detect those engines and issue a meaningfull
error message (not just a warning) like "You're running LuaTeX. inputenc can't
currently work on Unicode variants of TeX such as LuaTeX. Please consult
http://www.tex.ac.uk/cgi-bin/texfaq2html?label=lualatex for how to use LaTeX
with LuaTeX." and refusing to load.

Or even better: a warning and the loading of luainputenc. I don't think
this warning is necessary, but I don't have a strong opinion about it, I
think it will just be boring for the average user...

[Manuel Pégourié-Gonnard]

Post by Elie Roux

Post by Manuel Pégourié-Gonnard
By the way, people using XeTeX just encode their sources in utf-8 and skip
loading inputenc, and seem to be happy with it.

I think LuaTeX does not target the same users. With luainputenc and
fontenc working, LuaTeX can start targetting the average user, as the
old LaTeX documents will be working with LuaLaTeX (if inputenc loads
luainputenc). XeTeX will never allow old documents to be compiled with
it, as most input encoding problems have no solution.

Sure. But their is no need to target the average user too early. Leave early
beta-testing to advanced users. luainputenc and a working fontenc for LuaTeX are
very new and quite untested. Don't throw it in the hands of the average user
right know, please wait.

Post by Elie Roux

Post by Manuel Pégourié-Gonnard
So I see no harm in waiting a bit before implementing backward
compatibility solutions.

And I see no harm in proposing a working LuaLaTeX...

Half-working and quite untested, sorry. I still had no time to look closely at
luainputenc's latest versions, but if I understand correctly how it works, I'm
pretty sure some of my own documents won't comile without modification with it.
(And I'm talking of real-life documents, not specially-forged tricks.)

Post by Elie Roux
Of course, as
LuaTeX is not stable, LuaLaTeX won't be stable... but remember that
people call LuaLaTeX with the command "lualatex", so they know what they
are doing.

And since they know what they are doing, the can just change inputenc into
luainputenc.

By the way, a point I forgot to mention earlier is documentation. Keeping
separate names allows to properly document the current limitations, known issues
and how to solve them, for every individual package, at least during the testing
phase.

Post by Elie Roux
LuaTeX can only write UTF-8, so people needing to write in files in
8-bit without LICR (meaning not a lot of people)

Well, not meaning nobody either.

Post by Elie Roux
will have problems if
they do it in home-made package (I solved the problem for all files
generated by packages in TeXLive), so the problem concerns noone except
very advanced TeX users...

I don't think using VerbatimOut or filecontents qualifies as a very advanced TeX
user.

Post by Elie Roux
I know you're one of them, but for the
average user it will be 99.9999% backward compatible.

I tend to disagree with this value. But only testing will really show if it is
rather 90% or 99.9999%.

Post by Elie Roux
Well, I'm not sure pdfTeX and TeX82 were 100% compatible neither...

Of course they weren't. The question is: concerning the encodings, is it
possible to handle the differences well enough in a macro package so that we can
hide them to the user? Maybe, but I don't think it is already done, sorry.

Post by Elie Roux
Or even better: a warning and the loading of luainputenc. I don't think
this warning is necessary, but I don't have a strong opinion about it, I
think it will just be boring for the average user...

Experience proves normal users don't read warnings. (Even when they have
problems.) (Well, it probably holds for most error messages too...)

Manuel.

[Elie Roux]

Post by Manuel Pégourié-Gonnard
Half-working and quite untested, sorry. I still had no time to look closely at
luainputenc's latest versions, but if I understand correctly how it works, I'm
pretty sure some of my own documents won't comile without modification with it.
(And I'm talking of real-life documents, not specially-forged tricks.)

It's not been widely tested yet, but we still have some time to do so
before TL2009 is out...

Post by Manuel Pégourié-Gonnard
By the way, a point I forgot to mention earlier is documentation. Keeping
separate names allows to properly document the current limitations, known issues
and how to solve them, for every individual package, at least during the testing
phase.

I'm planning to make a LuaTeX-reference.pdf (or whatever the name)
included in TL 2009, containing all the choices made in luatextra and
the LaTeX-specific packages. It will explain all of this.

[Manuel Pégourié-Gonnard]

Post by Elie Roux
I'm planning to make a LuaTeX-reference.pdf (or whatever the name)
included in TL 2009, containing all the choices made in luatextra and
the LaTeX-specific packages. It will explain all of this.

Please take care to choose an explicit name. There is already a
XeTeX-reference.pdf document about the XeTeX engine, while your document would
be about a set of LaTeX packages for LaTeX.

Well, I know the name is not fixed yet and tou will take care of it, but juste
to be sure...

Manuel.

[Elie Roux]

Hello,

I'm the developer of luainputenc (inputenc adapted for LuaTeX). Sorry to
come back to this topic, but I think an answer to Will's question would
be necessary soon... inputenc needs adjustments in order to work with
LuaTeX and XeTeX. I can build a new package called luainputenc, but I
think it will be confusing for the user, and it makes LuaLaTeX unable to
compile old LaTeX documents, so it would be really great if some
modifications to inputenc were made. I think Will's idea is the best
one, having inputenc.sty being basically:

\RequirePackage{ifluatex}
\RequirePackage{iflxetex}
\ifluatex
\RequirePackage{luainputenc}
\else
\ifxetex
\RequirePackage{xetexinputenc}
\else
\RequirePackage{legacyinputenc}
\fi
\fi

You wouldn't have to maintain anything else than this small file you'll
never have to change, and the legacyinputenc, and all the new things
would be maintained by Will and me.

Also I have a legal problem: is it possible for me to distribute a
modified copy of inputenc.sty (called luainputenc.sty), and of utf8.def
(called lutf8.def) with the lppl?

Thank you in advance,

[Robin Fairbairns]

Post by Elie Roux
\RequirePackage{ifluatex}
\RequirePackage{iflxetex}
\ifluatex
\RequirePackage{luainputenc}
\else
\ifxetex
\RequirePackage{xetexinputenc}
\else
\RequirePackage{legacyinputenc}
\fi
\fi

this doesn't work reliably: you need explicit \PassOptionsToPackage for
each option, before \RequirePackage.

(your code seems to work for just one option, but not for 2. people do
use inputenc with >1 option...though i'm not certain it's sensible to do
so.)

something like

\def\RequirePackageWithOptions#1{%
\DeclareOption*{\PassOptionsToPackage{\CurrentOption}{#1}}%
\ProcessOptions
\RequirePackage{#1}%
}

robin

[Elie Roux]

Post by Robin Fairbairns
this doesn't work reliably: you need explicit \PassOptionsToPackage for
each option, before \RequirePackage.

That's true, but it was just a small code to make you understand, not at
all the final code of course. Once we agree on the global direction,
we'll start talking about technical issues, but I think the priority now
is to know how we can modify inputenc, if we can...

Thank you in advance,

[Robin Fairbairns]

Post by Elie Roux

Post by Robin Fairbairns
this doesn't work reliably: you need explicit \PassOptionsToPackage for
each option, before \RequirePackage.

That's true, but it was just a small code to make you understand, not
at all the final code of course. Once we agree on the global
direction, we'll start talking about technical issues, but I think the
priority now is to know how we can modify inputenc, if we can...

having (re)written it, i remembered \RequirePackageWithOptions in the
kernel.

doh!

robin