Fwd: Re: How to prevent Mustache from replacing & with & etc.?
I sent this to Norbert only instead of to the list... --- Hi Norbert, so I already found the & modifier, which means {{& myObject.asString }} will not have escaped characters in it. So far so good for & which I want to be interpreted by LaTeX as Column delimiters. But now I have this need for &'s that should be printed. Of course that means I need the encoded, just not as HTML entities ;-) Do you happen to know how I could implement my own modifier that enables to plug a custom encoding? I am thinking of introducing a modifier, say § to switch the html entity encoding for another one which I of course have to implement on my own. The end result would be: {{ myObject }}    to encoe with HTML entities for all things (X)HTML {{& myObject }}    to not encode at all (e.g. when I need & as & in the resulting String) {{§ myObject }}   for an individually added encoding (tbd - for my LaTeX needs) A mechanism like this could be used to make a "universal superset" of Mustache, I guess. Especially if we introduce some syntax for selecting an encoding from the template... Any pointers are welcome. And also ideas if this is a feasible/clever idea or not. Joachim Am 17.04.19 um 08:31 schrieb Norbert Hartl:
Joachim,
Am 17.04.2019 um 07:56 schrieb jtuchel@objektfabrik.de:
Norbert,
thanks for answering. I had tried the {{{ }}} option. In case of LaTeX, this is not so easy, however, because {{{ and }}} are completely valid syntactical elements of LaTeX.
It seems like you can replace {{ and }} in Mustache, but not {{{ and }}}. At least it didn't work when I tried. Using '{{{=### ###=}}} ' in my MustacheTemplate would leave text like '###myObject.printString###' unchanged in the resulting text instead of replacing it with the text representation of myObject.
So what I tried last and what gave me quite some mileage on the way to a sulotion was to use
\catcode to replace & with ; in my LaTeX file for the Column separator problem {{=## to make sure I can fill in my table rows with ; as column separator.
Thus my Mustache-Template looks like this (excerpt):
% redefine Mustache-Insertion syntax {{=## ##=}} \catcode`\;=4 % redefine ; for & to allow non-escaped column-separators in \tabular
... \begin{tabular}{ccp{6.0cm}rrr} ##myObject.asLatexTableRows## % This is where & is needed for LaTeX to separate table columns, but Mustache would replace the with the html entity & \end{tabular}
This is a bit hacky, and as I wrote, also still presents problems, because if any of the Strings being inserted using Mustache contains an &, there will be an & in the .tex file, which will lead to a LaTeX error.
So, essentially, what I am most likely looking for is a way to replace {{{ and }}} in Mustache to make sure the $&'s will remain untouched in my inserted text... Sorry for asking the wrong question, I was so glad how far I'd gotten with what I did that I didn't see the forest between all the trees.
So my real question is: how can I replace {{{ in Mustache with something that will not irritate LaTeX?
Any ideas?
Sure ;) In Mustache {{ and }} are the default delimiters. Every addtional character is a modifier. So {{{ is actuall a start delimiter {{ with a { as modifier. So what you can do is to change the delimiter but leave the { modifier like so
'{{=<% %>=}} <%{ value %>}' asMustacheTemplate value: { #value -> '&' } asDictionary
So {{ is the start delimiter, = is the modifier to change the delimiters. We set them to <% and %>. If you want to have unescaped strings you keep using the { on the new delimiter which turns out to be <%{ and %>}.
Norbert
Am 16.04.19 um 12:00 schrieb Norbert Hartl:
Hi,
Am 16.04.2019 um 09:02 schrieb jtuchel@objektfabrik.de:
Hi Pharoers,
I may be a bit wrong on this list. I try anyways.
I am using Mustache to fill text into a String which will then be handed over to LaTeX. The Data I fill in comes from the Database.
Now the problem is this: if a String from the database contains an ampersand (like in 'Katz & Maus'), Mustache will replace special chars with HTML entities. That is fine in a web context, but LaTeX doesnt actually work well with HTML entities. I could, of course, remove the html entities after Mustache invested a lopt of work into creating them. But it seems more logical to keep Mustache from being so eager to help. It is not logical because mustache is defined that way. So redefining does not improve it. But just use triple {{{ }}} to have strings not being escaped.
BTW: I already took the hurdle of Mustache excaping $& fir tabular environments (LaTeX uses the ampersand as a divider for table cells) by using \catcode in our .tex template files. But now the String comes from the database and so I am a bit lost...
So is there any best practice for this problem?
Yes, use it this way
'{{{ value }}}' asMustacheTemplate value: { #value -> '&' } asDictionary
Norbert
-- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
-- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 -- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
So I think I found a solution for my problem which seems to work. Thanks for listening and your input of ideas. What I did was to follow this idea:
I am thinking of introducing a modifier, say § to switch the html entity encoding for another one which I of course have to implement on my own. The end result would be:
{{ myObject }}    to encoe with HTML entities for all things (X)HTML {{& myObject }}    to not encode at all (e.g. when I need & as & in the resulting String) {{§ myObject }}   for an individually added encoding (tbd - for my LaTeX needs)
It turns out is was easy to implement a prototype of this (20 minutes?) There is only one tiny hole in this implementation: If I want to fill in a table body with Text that includes an &. I need to think about it later. Maybe I'll have to come back to the catcode thingie and choose some unlikely character for column separation and use my newly implemented {{§ encoding... All that's needed to implement the above idea were these changes (VA Smalltalk Fileout Format): !Character publicMethods ! asLaTeXString    "substitute characters like & into LaTex compliant elements"    "For protoytpe just encode &"    (Dictionary with: $& -> '\&')       keysAndValuesDo: [:k :v | self = k ifTrue: [^v]].    ^String with: self! ! !String publicMethods ! asLaTeXString    "substitute the < & > into LaTeX compliant elements"    "'<&>' asLaTeXString"    ^ self class new: self size streamContents: [ :s|       self do: [:c | s nextPutAll: c asLaTeXString ]]! ! MustacheToken subclass: #MustacheLaTeXEscapedToken  instanceVariableNames: ''  classVariableNames: ''  poolDictionaries: ''! !MustacheLaTeXEscapedToken publicMethods ! accept: aVisitor    aVisitor visitLaTeXEscapedToken: self! valueInContext: anObject    ^(super valueInContext: anObject)asLaTeXString ! ! !MustacheParser publicMethods ! buildDelimiterExtensions    ^Dictionary new       at: $# put: [self startSection: #MustacheSection abrAsClass];       at: $/ put: [self endSection];       at: ${ put: [self readDefaultUnescapedToken];       at: $& put: [self readUnescapedToken];       at: $!! put: [self readComment];       at: $^ put: [self startSection: #MustacheInvertedSection abrAsClass];       at: $= put: [self readChangeDelimiter];       at: $> put: [self readPartial];       at: $§ put: [self readLaTeXEscapedToken];       yourself! !MustacheVisitor publicMethods ! visitLaTeXEscapedToken: aToken ! ! !MustacheWriteVisitor publicMethods ! visitLaTeXEscapedToken: aToken    self addString: (aToken valueInContext: context)! ! So what do people think about this? Is this a worthwhile extension? Do you see problems on the horizon that I cannot see? Joachim -- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
Joachim, I put the comments inline.
Am 17.04.2019 um 10:34 schrieb jtuchel@objektfabrik.de:
So I think I found a solution for my problem which seems to work. Thanks for listening and your input of ideas.
What I did was to follow this idea:
I am thinking of introducing a modifier, say § to switch the html entity encoding for another one which I of course have to implement on my own. The end result would be:
{{ myObject }} to encoe with HTML entities for all things (X)HTML {{& myObject }} to not encode at all (e.g. when I need & as & in the resulting String) {{§ myObject }} for an individually added encoding (tbd - for my LaTeX needs)
It turns out is was easy to implement a prototype of this (20 minutes?)
There is only one tiny hole in this implementation: If I want to fill in a table body with Text that includes an &. I need to think about it later. Maybe I'll have to come back to the catcode thingie and choose some unlikely character for column separation and use my newly implemented {{§ encoding...
All that's needed to implement the above idea were these changes (VA Smalltalk Fileout Format):
!Character publicMethods !
asLaTeXString "substitute characters like & into LaTex compliant elements" "For protoytpe just encode &"
(Dictionary with: $& -> '\&') keysAndValuesDo: [:k :v | self = k ifTrue: [^v]]. ^String with: self! !
!String publicMethods !
asLaTeXString "substitute the < & > into LaTeX compliant elements" "'<&>' asLaTeXString" ^ self class new: self size streamContents: [ :s| self do: [:c | s nextPutAll: c asLaTeXString ]]! !
MustacheToken subclass: #MustacheLaTeXEscapedToken instanceVariableNames: '' classVariableNames: '' poolDictionaries: ''!
!MustacheLaTeXEscapedToken publicMethods !
accept: aVisitor
aVisitor visitLaTeXEscapedToken: self!
valueInContext: anObject
^(super valueInContext: anObject)asLaTeXString ! !
This is the way it is designed so you can use your own token/visitor pair to extend it.
!MustacheParser publicMethods !
buildDelimiterExtensions
^Dictionary new at: $# put: [self startSection: #MustacheSection abrAsClass]; at: $/ put: [self endSection]; at: ${ put: [self readDefaultUnescapedToken]; at: $& put: [self readUnescapedToken]; at: $!! put: [self readComment]; at: $^ put: [self startSection: #MustacheInvertedSection abrAsClass]; at: $= put: [self readChangeDelimiter]; at: $> put: [self readPartial]; at: $§ put: [self readLaTeXEscapedToken]; yourself!
I donât think you need to change the parser to do that. parser := MustacheParser new. parser delimiterExtensions at: $§ put: [ parser readLaTeXEscapedToken ]. should do as well. If you have more specific needs I would subclass the parser
!MustacheVisitor publicMethods !
visitLaTeXEscapedToken: aToken ! !
!MustacheWriteVisitor publicMethods !
visitLaTeXEscapedToken: aToken
self addString: (aToken valueInContext: context)! !
I would make a visitor subclass for that. The problem is that the write visitor class name is hardcoded. We need to make that a setting. Then you could provide your own visitor. This way you can make it a real extension to mustache.
So what do people think about this? Is this a worthwhile extension? Do you see problems on the horizon that I cannot see?
I donât think it is a good extension. Mustache is implemented along the documentation and putting every use case in the code is not good. Changing the code so you can apply your changes or provide an extension package is needed of course. Norbert
Joachim
--
----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
Hi Norbert, thanks for commenting. I had a bad feeling about creating new class editions while I implemented them, so I appreciate your comments and I'll revisit my implementation, because I agree with all of your input here. Especially extending the Visitor felt "wrong" or bad. DD can be helpful, but does not lend itself well to extensions, especially "personal" ones ;-) I didn't expect you to accept my extensions to Mustache anyways, becaus Mustache should be Mustache. But given that we now have multiple "standard" mechanisms for mixing text into templates in Smalltalk, and none fits very well for a lot of cases, I thought it could probably be a good starting point to look into Mustache as a powerful and nicely crafted starting point for more extendable implementations for inserting text. An extension point that builds on top of your parser := MustacheParser new. parser delimiterExtensions at: $§ put: [ parser readLaTeXEscapedToken ] suggestion would be nice, but is not needed. I am just playing with ideas here. Again: thanks a lot for your help and comments. Even if nobody else never needs or wants what I will implement here, this still helps me a lot in achieving what I need. Joachim Am 17.04.19 um 10:52 schrieb Norbert Hartl:
Joachim,
I put the comments inline.
Am 17.04.2019 um 10:34 schrieb jtuchel@objektfabrik.de:
So I think I found a solution for my problem which seems to work. Thanks for listening and your input of ideas.
What I did was to follow this idea:
I am thinking of introducing a modifier, say § to switch the html entity encoding for another one which I of course have to implement on my own. The end result would be:
{{ myObject }} to encoe with HTML entities for all things (X)HTML {{& myObject }} to not encode at all (e.g. when I need & as & in the resulting String) {{§ myObject }} for an individually added encoding (tbd - for my LaTeX needs)
It turns out is was easy to implement a prototype of this (20 minutes?)
There is only one tiny hole in this implementation: If I want to fill in a table body with Text that includes an &. I need to think about it later. Maybe I'll have to come back to the catcode thingie and choose some unlikely character for column separation and use my newly implemented {{§ encoding...
All that's needed to implement the above idea were these changes (VA Smalltalk Fileout Format):
!Character publicMethods !
asLaTeXString "substitute characters like & into LaTex compliant elements" "For protoytpe just encode &"
(Dictionary with: $& -> '\&') keysAndValuesDo: [:k :v | self = k ifTrue: [^v]]. ^String with: self! !
!String publicMethods !
asLaTeXString "substitute the < & > into LaTeX compliant elements" "'<&>' asLaTeXString" ^ self class new: self size streamContents: [ :s| self do: [:c | s nextPutAll: c asLaTeXString ]]! !
MustacheToken subclass: #MustacheLaTeXEscapedToken instanceVariableNames: '' classVariableNames: '' poolDictionaries: ''!
!MustacheLaTeXEscapedToken publicMethods !
accept: aVisitor
aVisitor visitLaTeXEscapedToken: self!
valueInContext: anObject
^(super valueInContext: anObject)asLaTeXString ! !
This is the way it is designed so you can use your own token/visitor pair to extend it.
!MustacheParser publicMethods !
buildDelimiterExtensions
^Dictionary new at: $# put: [self startSection: #MustacheSection abrAsClass]; at: $/ put: [self endSection]; at: ${ put: [self readDefaultUnescapedToken]; at: $& put: [self readUnescapedToken]; at: $!! put: [self readComment]; at: $^ put: [self startSection: #MustacheInvertedSection abrAsClass]; at: $= put: [self readChangeDelimiter]; at: $> put: [self readPartial]; at: $§ put: [self readLaTeXEscapedToken]; yourself!
I donât think you need to change the parser to do that.
parser := MustacheParser new. parser delimiterExtensions at: $§ put: [ parser readLaTeXEscapedToken ].
should do as well. If you have more specific needs I would subclass the parser
!MustacheVisitor publicMethods !
visitLaTeXEscapedToken: aToken ! !
!MustacheWriteVisitor publicMethods !
visitLaTeXEscapedToken: aToken
self addString: (aToken valueInContext: context)! !
I would make a visitor subclass for that. The problem is that the write visitor class name is hardcoded. We need to make that a setting. Then you could provide your own visitor.
This way you can make it a real extension to mustache.
So what do people think about this? Is this a worthwhile extension? Do you see problems on the horizon that I cannot see?
I donât think it is a good extension. Mustache is implemented along the documentation and putting every use case in the code is not good. Changing the code so you can apply your changes or provide an extension package is needed of course.
Norbert
Joachim
--
----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
-- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
Am 17.04.2019 um 11:19 schrieb jtuchel@objektfabrik.de:
Hi Norbert,
thanks for commenting. I had a bad feeling about creating new class editions while I implemented them, so I appreciate your comments and I'll revisit my implementation, because I agree with all of your input here. Especially extending the Visitor felt "wrong" or bad. DD can be helpful, but does not lend itself well to extensions, especially "personal" ones ;-)
I didn't expect you to accept my extensions to Mustache anyways, becaus Mustache should be Mustache.
But given that we now have multiple "standard" mechanisms for mixing text into templates in Smalltalk, and none fits very well for a lot of cases, I thought it could probably be a good starting point to look into Mustache as a powerful and nicely crafted starting point for more extendable implementations for inserting text. An extension point that builds on top of your
parser := MustacheParser new. parser delimiterExtensions at: $§ put: [ parser readLaTeXEscapedToken ]
suggestion would be nice, but is not needed.
I am just playing with ideas here.
Again: thanks a lot for your help and comments. Even if nobody else never needs or wants what I will implement here, this still helps me a lot in achieving what I need.
If you want to try you can checkout https://github.com/noha/mustache/tree/api-enhancements <https://github.com/noha/mustache/tree/api-enhancements> branch. I enhanced the API so you can set a custom visitor. Now you can do: aTemplate value: aDictionary visitor: MyCustomVisitor new Hope it helps! Norbert
Joachim
Am 17.04.19 um 10:52 schrieb Norbert Hartl:
Joachim,
I put the comments inline.
Am 17.04.2019 um 10:34 schrieb jtuchel@objektfabrik.de:
So I think I found a solution for my problem which seems to work. Thanks for listening and your input of ideas.
What I did was to follow this idea:
I am thinking of introducing a modifier, say § to switch the html entity encoding for another one which I of course have to implement on my own. The end result would be:
{{ myObject }} to encoe with HTML entities for all things (X)HTML {{& myObject }} to not encode at all (e.g. when I need & as & in the resulting String) {{§ myObject }} for an individually added encoding (tbd - for my LaTeX needs)
It turns out is was easy to implement a prototype of this (20 minutes?)
There is only one tiny hole in this implementation: If I want to fill in a table body with Text that includes an &. I need to think about it later. Maybe I'll have to come back to the catcode thingie and choose some unlikely character for column separation and use my newly implemented {{§ encoding...
All that's needed to implement the above idea were these changes (VA Smalltalk Fileout Format):
!Character publicMethods !
asLaTeXString "substitute characters like & into LaTex compliant elements" "For protoytpe just encode &"
(Dictionary with: $& -> '\&') keysAndValuesDo: [:k :v | self = k ifTrue: [^v]]. ^String with: self! !
!String publicMethods !
asLaTeXString "substitute the < & > into LaTeX compliant elements" "'<&>' asLaTeXString" ^ self class new: self size streamContents: [ :s| self do: [:c | s nextPutAll: c asLaTeXString ]]! !
MustacheToken subclass: #MustacheLaTeXEscapedToken instanceVariableNames: '' classVariableNames: '' poolDictionaries: ''!
!MustacheLaTeXEscapedToken publicMethods !
accept: aVisitor
aVisitor visitLaTeXEscapedToken: self!
valueInContext: anObject
^(super valueInContext: anObject)asLaTeXString ! !
This is the way it is designed so you can use your own token/visitor pair to extend it.
!MustacheParser publicMethods !
buildDelimiterExtensions
^Dictionary new at: $# put: [self startSection: #MustacheSection abrAsClass]; at: $/ put: [self endSection]; at: ${ put: [self readDefaultUnescapedToken]; at: $& put: [self readUnescapedToken]; at: $!! put: [self readComment]; at: $^ put: [self startSection: #MustacheInvertedSection abrAsClass]; at: $= put: [self readChangeDelimiter]; at: $> put: [self readPartial]; at: $§ put: [self readLaTeXEscapedToken]; yourself!
I donât think you need to change the parser to do that.
parser := MustacheParser new. parser delimiterExtensions at: $§ put: [ parser readLaTeXEscapedToken ].
should do as well. If you have more specific needs I would subclass the parser
!MustacheVisitor publicMethods !
visitLaTeXEscapedToken: aToken ! !
!MustacheWriteVisitor publicMethods !
visitLaTeXEscapedToken: aToken
self addString: (aToken valueInContext: context)! !
I would make a visitor subclass for that. The problem is that the write visitor class name is hardcoded. We need to make that a setting. Then you could provide your own visitor.
This way you can make it a real extension to mustache.
So what do people think about this? Is this a worthwhile extension? Do you see problems on the horizon that I cannot see?
I donât think it is a good extension. Mustache is implemented along the documentation and putting every use case in the code is not good. Changing the code so you can apply your changes or provide an extension package is needed of course.
Norbert
Joachim
--
----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
-- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1
participants (2)
-
jtuchel＠objektfabrik.de -
Norbert Hartl