It is used by writing the \label{id} command in
section to be referred, where id identifies
the section, preferably being easy to remember and not changed too
often. This makes it possible to use the
\ref{id}
and \pageref{id} commands which typeset
the number of the section or its page number. (References may
lead to page of any text or a number of equation, list item,
theorems, etc; I refer to all of them as ‘sections’ in
this post.)
During the first run of LaTeX the text is completely typeset,
using ‘??’ instead of numbers to be referred to. All
\labels write their identifiers, section and page
numbers to the .aux file. On the beginning of the
second run this file is read, then all references are used as they
were correct on the previous run, and new values are written to
the file.
Here an important feature of TeX is seen –
typesetting words, breaking paragraphs into lines, joining lines
into pages, and outputting pages are done asynchronously. To know
the page number of a given \label LaTeX uses the
primitive TeX command \write which evaluates
appropriate commands during page output. This makes it impossible
to change text depending on current page number, so the number
from previous run must be used instead.
The same method is used for other things, like tables of
contents, bibliographic references, indices and correct placement
of margin notes on two-sided documents by the
mparhack package.
However, using multiple passes for cross references has several
disadvantages. The most visible one is that the time needed to
make correct output is several times larger, although only one
output file is needed and error messages are useful for only one
pass (this can be improved by using \batchmode for
non-first runs of LaTeX and pdfTeX’s -draftmode
option for non-last runs). Despite this, it is clearly visible
that most of work during non-last passes is unnecessary
(especially when referring only to numbers of sections, not
pages).
Since the same files are modified in each pass, it is difficult
to optimally use make or another generic build system
with LaTeX. This leads to longer processing than necessary and
uncertainty of the document having outdated references.
It is even possible to make a document which has always incorrect references. This document shows this problem:
\documentclass{minimal}
\pagestyle{empty}
\pagenumbering{roman}
\setlength{\textwidth}{8pt}
\setlength{\textheight}{10pt}
\setlength{\parindent}{0pt}
\makeatletter
\@ifundefined{pdfpagewidth}{}{%
\pdfpagewidth=2in
\pdfpageheight=2in
}
\makeatother
\begin{document}
\setcounter{page}{9}
\pageref{x}\hspace{0pt}i\label{x}
\end{document}
(The part with \pdfpagewidth is to make it easier
to see both pages at once in a PDF viewer, I have described it in a previous
post.)
Since the second pass, this document will oscillate between having one or two pages. When the reference leads to page x, then the ‘i’ is on page ix, but with reference to page ix the ‘i’ is on page x. (Leslie Lamport states in LaTeX: A Document Preparation System that using Roman numerals may lead to this problem, I did not know any specific example of such document before writing the above one.)
So despite being very useful, automatic cross references in LaTeX have some disadvantages. Usually a good enough solution is to run LaTeX on a document some times longer than possible necessary, and change text to avoid having infinite loops in this process. Could it be improved? I’ll write about some other ways to avoid these problems in a separate post.
]]>mwcls for Polish texts). As necessary for internationalized use, they support many paper sizes. Unfortunately, if a PDF is generated from such document, it is treated by viewers as a Letter or A4 page even if it is completely different.
The UK TeX FAQ explains this problem and suggests several solutions – all of which are packages or document classes using their own page layouts. Since the document classes which I use already have appropriate layout, I decided to solve this problem differently.
As stated in the FAQ and the pdfTeX manual (page 20, the texdoc pdftex command in TeXLive shows this documentation), the \pdfpagewidth and \pdfpageheight dimensions are used to set the PDF page size. LaTeX stores the same data in \paperwidth and \paperheight (classes.pdf page 3 as available in TeXLive). Manuals of XeTeX and LuaTeX show that these TeX implementations also support these commands, so using any of them the following code will set the appropriate page size:
\pdfpagewidth=\paperwidth \pdfpageheight=\paperheight
This code won’t work with TeX implementations without these commands. When these commands are undefined it will produce some error messages and output some text. The simplest solution is to not use these commands when they are not available, for example with the following code:
\makeatletter
\@ifundefined{pdfpagewidth}{}{%
\pdfpagewidth=\paperwidth
\pdfpageheight=\paperheight
}
\makeatother
The UK TeX FAQ suggests using the ifpdf package for detecting PDF support which here would be too specific – support for XeTeX would require using another package, also it is not a problem to use the above code when making a DVI file. It would not make DVIs with correct page size, but setting this would depend on the DVI driver and PDF-producing TeX variants have other useful advantages.
\today macro. It even supports many languages (using Babel or completely language-specific packages). Unfortunately, (as the name suggests) it typesets only the current date, while all other must be written ‘by hand’.
This would not be appropriate way of writing e.g. certificates of attendance for a Polish IB school, where on each page many dates chosen by the school are written in both British English and Polish languages. I wanted to write only a single date in a simple format, like 2010/02/24.
The first solution which I used was a complicated package parsing this date format into separate macros for year, month and day (it also removed leading zeros), and then typesetting them using separate macros for different languages, with 12 macros per language for month names. It had at least the following problems:
\today macro.Therefore I have rewritten this package today, using the \today macro after changing the ‘current’ date.
TeX stores the current date in three parameters: \year, \month and \day (The TeXbook, page 273). This trivial macro changes them using the 2010/02/24/ formatted date:
\def\@setcurrentdate#1/#2/#3/{%
\year=#1%
\month=#2%
\day=#3%
\relax
}
The interface of this macro is strange, but it looks like a simple way of separating parts of the input date. The date-formatting macro will expand its second argument and add the trailing slash (so the whole day will be used, instead of only its first digit) by \expandafter\@setcurrentdate#2/.
Babel provides at least two macros for changing languages: \foreignlanguage and \selectlanguage. The first one has more appropriate interface, but does not change the date format (as stated on page 6 of the Babel manual), so I used the second one.
The whole macro formatting the date has two arguments – language name as used by Babel and the date in slash-separated format. The macro is:
\newcommand{\Date}[2]{{%
\selectlanguage{#1}%
\expandafter\@setcurrentdate#2/%
\today
}}
(The name is capitalized, since I use it only in some document classes.) I’m not sure if specifying language language for each date is useful (dates are usually in the same language as surrounding text), but it could be trivially removed in useful packages.
I haven’t seen this way of extending LaTeX macros in other code, but it looks like an useful advantage of mutable parameters for ‘constants’ like current time.
]]>The problems with this solution is that the same pages are generated many times and much simpler software could be used for this. On a typical blog updated much less often than viewed exactly identical pages are generated multiply times, using multiply times more resources than necessary. Therefore hosting of typical blogs could be much simpler and cheaper than is possible with such technology.
Therefore very large sites (or sites using very large software) use separate caching servers, like Varnish. Such programs get a page from the original server and keep it for some time, giving it much faster without regenerating the page for next requests. This solution still does not support nicely sites changed for every user (so usually the cache is skipped for 'non-anonymous' users) and it is difficult to avoid giving outdated pages from the cache (at least while the cache is used for unchanged pages). (Another problem is that another daemon must by running on the server, allowing friendly 503 HTTP errors when one of the daemons serving the page does not work.)
All problems of such caching servers could be avoided by correctly generating static files for each page when something is changed. If the files will be generated on a different machine than the server, then it could be written in completely different, maybe better, ways than the software used on the server. The pages will be uploaded to the server and a simpler HTTP server would send them much faster than with any other solution.
This solution would clearly require generation of all uploaded files by a single user with access to the whole site. So there won't be any multiuser things, there will be no search, and the sites will not depend on current time (this is used for relative, friendly dates in texts like 'written five hours ago', this could be easily done client-side using JavaScript). Still, these things look uncommon on a typical blog written by a single user.
The problem is that pages of a simple blog depend usually not only on their own content and other posts but also on comments posted by users. A server-side script is necessary to get the comments, but it won't be a problem since this is exactly what such scripts are for. There are two possible things to do with the comments obtained by the script - adding them to the post page or putting them in a private place from which the user would move the comments to be published on next update. The first solution requires making the page 'less static', but the server-side code would be still much simpler then usually. The second solution is useful also due to useless spam (ignored by both readers, search engines and writers) being sent by malevolent bots as comments.
Therefore such 'offline' blog software would work well enough for small sites. They would be also able to do things which are too slow to be done by advanced server-side scripts, for example checking if the example source code written in a post about the C programming language can be compiled (and maybe even run and generate the output shown in the post). Such software improvements could make higher quality posts easier. Also, as a locally used program, it could be more user-friendly than Web-based solutions (it could e.g. allow using standard Unix tools to correct a typo in many posts, or use more helpful editors than available in a browser).
Maybe it is worth writing such a program (or finding and using an existing one). Certainly the possibility of converting an existing blog to use such software would not be trivial (e.g. to avoid duplicate entries in feed readers and to import all useful data like comments), but it might have more benefits.
]]>Most GNU/Linux systems use Bash as their only shell. This is different on BSD derivatives like FreeBSD using ash for scripting and tcsh (a C shell derivative with largely different syntax than other shells) for interactive use.
Most shell scripts do not use specific features of any shell and need just a mostly-POSIX-compatible shell like dash (an ash derivative) or Bash. Therefore they specify /bin/sh as their interpreter, which is always such shell. In most GNU/Linux distributions Bash is used as /bin/sh, while in BSDs ash is used, and Ubuntu and Debian Squeeze use dash. Therefore many scripts using Bash-specific features declare incorrectly to be used with the default shell and fail on Ubuntu or FreeBSD.
Avoiding the above problem by testing scripts with shells having only the features required by POSIX is not the only reason to use non-Bash shells for scripting. The dash shell is faster then Bash, this is why it was proposed for Debian Lenny release to use dash as the default shell for scripts.
To check how time performance of different shells differs, I wrote several trivial scripts which can be interpreted by the popular POSIX-like shells. Two of the scripts calculate factorials using different recursive algorithms (one is the ‘standard’ definition used in mathematical textbooks, the other one is the tail-recursive one used in functional programming textbooks), another one calculates elements of the Fibonacci sequence using the recursive definition, the fourth one just calls the shell about one hundred times to check how slow is its initialization. I haven’t seen a real shell script doing such things, but the ones which I normally use depend mostly on other program performance or use Bash-specific features. Another script calculates average time spent by each script and shell combination from ten runs (one additional run of each is done before counting, since this needs loading the shell from the disk) and outputs the result in a simple to parse format.
I compared six shells available in Gentoo GNU/Linux ebuilds sys-apps/busybox-1.15.2, app-shells/bash-4.0_p35, app-shells/dash-0.5.5.1.2, app-shells/mksh-39, app-shells/pdksh-5.2.14-r4, app-shells/zsh-4.3.10. The average times in seconds on the machine which I’m using calculated by the script are:
| Script | bb | dash | bash | zsh | mksh | pdksh |
|---|---|---|---|---|---|---|
| tail-recursive factorial | 0.12 | 0.083 | 0.254 | 0.23 | 0.122 | 0.117 |
| standard factorial | 0.106 | 0.084 | 0.229 | 0.242 | 0.12 | 0.121 |
| Fibonacci sequence | 1.061 | 0.801 | 2.177 | 2.063 | 1.044 | 1.301 |
| recursive shell invocation | 0.341 | 0.269 | 0.515 | 1.91 | 0.378 | 0.349 |
For all above tests dash is the fastest, BusyBox and Korn shell variants have similar performance, while Bash or zsh is the slowest one. Bash was two to three times slower than dash for these tests.
Of course, real scripts are something completely different. Probably everyone who wants to write functional programs knows more appropriate languages than POSIX shells. Also, extensions of many shells probably might make them faster for some scripts using them. The main reason for shell scripting is the ease of writing trivial scripts similar to commands written for daily interactive use. Therefore it is more useful to write a simple script and rewrite it in a better language when needed.
The scripts used for the above calculations are available in my Mercurial repository. The main script is licensed under the GNU General Public License, version 3 or later, while the tested scripts are public domain, since I hope that these are too unoriginal to be copyrightable.
]]>Most Web servers store in their access logs some data about each request, like the user’s IP address or the referring page URL. There are many formats of such data, but they all share three important things:
The problem with such data is that it does not specify some information known only by the user’s browser (called more formally an user agent), like screen resolution, support for JavaScript or some useless plugins. Other information coming from the user are trivially forged, malicious bots happily pretend to be real browsers coming by links from other pages.
A partial solution to this is to use JavaScript code and zero-sized images without caching to get these information from the client. But this requires more requests per page view (especially from different servers, these makes page loading much slower), and it ignores users who disable JavaScript or use browser extensions blocking such code for privacy/performance reasons.
Although there is nothing specific to free software, this situation leads to many problems with programs analyzing Web server logs.
These are several possible uses of data stored in the access logs:
All of these might be used to make the site more useful. The programs should make it easy, but it is not as simple as it seems.
For most uses only data about human visitors is helpful. Only to block unfriendly bots or to correct technical problems data about bot visits is needed.
The problem is that only the useful bots want to be identified as bots. The ones which send spam, copy content to spam sites, get mail addresses to send spam, spam etc, do not want to be known – this would make it trivial to disallow their visits. So they pretend to use popular Web browsers and use many IP addresses without any clear pattern.
Many spam bots can be easily identified by using identifications of very old browsers (some of which could not access the site due to changes in the Web protocols), or by strange usage patterns like visiting only a single page referring from the same page and not getting any styles or images. They also go to URLs used by insecure Web applications and pretend to visit from certain sites in hope of getting a link to these sites (it is called referrer spam). This spam is useless in most cases, since the referrer URLs are not published on properly written sites excluding ones like password-protected log analyzer reports (with all links marked to be ignored by search engine crawlers). But it still makes the log analyzers less useful.
One of the most visible things which I observed after visiting the Wikipedia list of Web log analyzers is that most of them are very old. Of the ones not using MySQL or PHP one had last release in 2004, another does not try to ignore visits by bots in statistics generated, using another one is the main inspiration for this post.
Clearly, identification of new browsers and operating systems, proper determination of queries from new (or renamed) search engines, and detection of malicious bots requires changes in software. So I believe that projects without new releases in this year do not detect new things and have problems making them less interesting to improve.
Another problem is that URLs are usually not unique for a given content, although they should. This is most common with forum software written in PHP, they use different URLs for each user. Therefore log analyzers treat each visit from a forum thread as a visit from a different page. This makes lists of referring URLs much less friendly to humans who are more interested in pages than their specific URLs.
There are probably no perfect solutions for the spam in statistics, but the programs could vastly decrease its amount by trivial measures.
There are two methods of solving these problems – correcting an existing program or writing a new one. Since most of free software log analyzers are written in C, which is better for much different programs, or Perl, which is appropriate for much smaller programs and probably encourages committing some of their possible design mistakes, it would be difficult for me. Maybe it would be an interesting learning experience to write another faulty log analyzer?
]]>Eric S. Raymond explains this by the fact that incomplete programs are not understood completely, so a thing considered not an optimization may make the program slower when it will be completed. In case of small programs for batch processing of data there is another reason – slow but easy to understand program may generate output which will be compared to output of optimized versions.
For real programs there is another advantage – a simple prototype might be used to describe the design and allow much faster development than a fast program written in C. Once I wrote a trivial Python script to explain a limitation of standard Unix program uniq. It was so simple that in next few days I decided to implement a complete uniq-like program in C.
In case of homeworks the aim is usually to make a program passing some automated tests. If it is possible to test the program many times, it would be not useful to optimize it after all tests are passed in required time.
In real life programs are used with more than several different input files, so this argument does not apply to non-homework programs. But there the Moore’s Law makes most optimization pointless.
It is very difficult for humans to predict performance effect of a change in a program. For example, C programs with a loop consisting of billions of pre- or postincrements of a variable saved to another variable produce on my machine assembly code differing only by the order of instructions in the loop. It looks obvious that these programs would work for the same amount of time, but the one with preincrements was faster by about 25% when using an AMD Phenom processor. In this case the correct solution was to enable compiler optimizations which removed the whole loop, but in many situations it is better to make small changes and compare their effect on the program performance.
In real Unix programs profilers are used for this, but for programs containing only one function it is appropriate to just measure the time spent running the whole program on a large input.
If a typical input takes one microsecond to process, then it might be necessary to write a script making much larger inputs. Since they would be too large to have output correctness verified by human, an unoptimized ‘correct’ implementation of the program being improved is useful to generate output for regression testing.
As explained by Joel Spolsky, malloc and similar functions use a slow algorithm to decide which memory is free. This probably could have been improved since 2001, but still is slow. Fortunately many homework problems specify limits on input size which may allow using statically allocated arrays for everything.
In real programs malloc is often avoided by ignoring characters after the 2047 column of a line or by crashing the program when they occur. An advantage of many GNU programs is that they do not limit input length and dynamically allocate input buffers. Therefore this tip is mostly useless for non-homework problems.
CPUs are thousands of times faster than memory. Therefore operations on large amounts of memory become the performance bottleneck of programs without significant I/O.
Performance of a program can be improved by using a single buffer for e.g. an input line, instead of copying it many times. In some cases code reading the input might do some calculations on it which will lead to storing much less data in memory. This might be a useful improvement in case of programs performing O(n) algorithms on these data.
Since memory is slow, modern CPUs store some of it in faster and more expensive caches. Therefore smaller code might be stored completely in cache and avoid slow memory access, as explained by Raymond.
This is done mainly by removing optimizations designed for older processors. A nice example of this is the removal of many uses of Duff’s device from the XFree86 X11 server. As stated by Theodore Ts’o, ‘by eliminating all instances of Duff's Device from the XFree86 4.0 server, the server shrunk in size by _half_ _a_ _megabyte_ (!!!), and was faster to boot’.
]]>groff with heirloom-doctools leading to having nearly no C++ programs in a Gentoo system, I tried to rebuild all packages on my Gentoo-using server without C++. The main difference is that my computer compiles all packages instead of using another system for this, so I need also all build dependencies of useful packages compiling and working without C++.
I started by adding the nocxx -cxx USE flags to everything except GCC. Rebuilding affected packages showed that app-arch/lzma-utils has programs written in C++ for everything except decompression. So I checked on my unstable Gentoo workstation that its app-arch/xz-utils does not have programs linking to libstdc++, which is nearly equivalent to using C++. Therefore I added app-arch/xz-utils to /etc/portage/package.keywords and installed it. Now both LZMA and XZ compression formats may be both compressed and decompressed, without any C++ code.
Since C++ support is assumed to be in every system, the package manager cannot determine which packages use it. Therefore I decided to check which programs link to the C++ standard library. The following shell script lists all files on $PATH which link to libraries containing ++ in their names:
#!/bin/sh for directory in `echo $PATH | sed 's/:/ /g'`; do for f in $directory/* do if file $f | grep ELF > /dev/null then list=`readelf -dw $f | grep Shared \ | sed -r 's/^.*\[(.*)\].*$/\1/' | fgrep ++` if [ $? = 0 ] then echo $f fi fi done done
Passing the output of the script to xargs qfile -q | sort -u listed packages having these files.
The only necessary packages on my system listed by the above pipe were app-arch/lzma-utils and sys-apps/groff, but they can be easily replaced by app-arch/xz-utils and app-doc/heirloom-doctools.
Then I compiled sys-devel/gcc with the nocxx USE flag and ran emerge -ev --keep-going world.
Next day I saw that build failed for 27 packages. Some of them, like sys-apps/sed had just failing tests for completely unrelated reasons. Many other packages, like sys-devel/libtool have tests using C++. So I recompiled all such packages with FEATURES=-test. Now only 12 packages failed.
The rest failed mostly when running the configure script, with messages like this:
checking how to run the C++ preprocessor... /lib/cpp configure: error: C++ preprocessor "/lib/cpp" fails sanity check
These were caused by useless checks for C++ compiler made by older versions of libtool. In case of dev-libs/popt and sys-apps/shadow updating to the testing version solved this, but for some other packages adding eautoreconf to their ebuilds was necessary.
After these changes the only packages with build errors were dev-libs/apr, net-libs/courier-authlib and net-mail/courier-imap. The first one also fails at the configure check, while the rest uses some C++ code which is not installed.
The modified ebuilds were available in my overlay. I haven’t reported any of the problems observed to the Gentoo Bugzilla, since I cannot access it today and the changes would just make compilation slower in all normal uses.
]]>Quote marks are used exactly like parentheses – they delimit a fragment of a sentence. But unlike all other such characters, inner quotation marks are different symbols than the outer ones (unless larger outer delimiters in mathematical formulas count as different symbols (they are the most ‘mainstream’ use of parentheses in parentheses)). Another difference is that ‘((’ is easily interpreted in correct way, while ‘“ needs additional spacing (‘ “).
Another problem is that each language has different quote marks. American English uses double outer quotes and inner single quotes, British English uses them as inner and outer, Polish has low double opening quote and English double closing quote, the inner quotes are the French ones, although they are rarely used correctly. American English also includes following commas and periods in the quotes, obviously this would lead to problems in programming-related texts.
LaTeX does not solve these problems, but allows direct specification of appropriate symbols. The English quote marks are represented as ``, '', ` and ', since these are the nearest equivalents on a typical keyboard. The " character is not used and the space between quotes must be specified as \, (it could be specified in the font, but this would require separate sets of fonts for American and British texts, I’m not sure if it could support third level nested quotes in any of these dialects).
It would be interesting to use just the " character and let the software decide which quotes are opening and which are closing. But even without support for nested quotations this would be difficult (if possible) to do correctly in all cases. A naïve algorithm would just begin with an opening quote and then cycle between closing and opening ones. But this won’t interpret correctly quotes in multi-paragraph dialogue, where each paragraph begins with an opening quote (in Polish dashes are used instead of quotes for dialogue and there is no possibility for humans to interpret multi-paragraph dialogue correctly without backtracking). A common mistake in delimiting block quotations with quote marks may result in a paragraph containing only a closing quote mark, so this algorithm cannot be improved by just resetting to opening quote at each new paragraph.
Emacs uses a different algorithm in the TeX-insert-quote function. It puts opening quotes after whitespace or opening parenthesis. This method could not be implemented in LaTeX, but it can be done in language-specific fonts. But this algorithm fails when quoting spaces or parentheses, like ‘(’, which is commonly done in programming-related texts.
The only problem which can be easily solved is which quotes to use. I have written a LaTeX package for this, named quoted (available in my Mercurial repository), but it does not support spaces between quote marks of different levels or moving punctuation to the quotation. There are probably many better packages for this, but this will not make a useful document ‘portable’ between e.g. British and American dialects of English, so such packages aren’t very useful.
Since parentheses are similar to quotes, but simpler, maybe a single character in source files could be used for them. In times of typewriters a slash was sometimes used instead of parentheses, since it looks similar. Is it possible to implement a LaTeX macro or virtual font replacing / by a slash or appropriate parenthesis depending on context?
Many text use large numbers of acronyms and abbreviations, but their meaning is not always remembered by the readers. Many acronyms also have more than one meaning, e.g. technical texts about AMD GPU support on GNU/Linux use the DRM acronym in two meanings – one very useful and one very harmful.
The solution is to specify the meaning using the title attribute of the acronym element, in the GPU example it would be:
<acronym title="Direct Rendering Manager">DRM</acronym> allows more optimal use of modern hardware.
I use this element for first use of each acronym in all posts on my blog. The HTML 4.01 specification describes also the abbr element used for abbreviations. I’m not sure which one of them should be used in which situation.
It is nice to know where a hyperlink leads. Therefore it should be appropriately described, by text and additional information provided using the title attribute. Some sites have readable URIs, but they should not be the only information allowing a user to decide if the page linked to is used. Using the same example as previous, link with a title may be written in HTML as
<a href="http://en.wikipedia.org/wiki/Direct_Rendering_Manager" title="Wikipedia: Direct Rendering Manager">very useful</a>
Many guidelines for using link titles are specified in Alertbox by Jakob Nielsen. The simplest rules to follow when using link titles is to not duplicate nearby information in them and to provide name of the resource linked to (very useful when the context does not specify this and the URI is numeric).
In this list it is probably useful to quickly scan names of its elements and read the more useful ones. Definition lists are formatted for such use. In HTML they are specified using three elements – dl contains the whole list, the elements of which are dt containing the defined term and dd containing the definition (each term may have many definition and each definition may have many terms).
Unlike the previous elements, definition lists are equally useful in print. This might make them popular and easy to use correctly, although commonly itemized lists with term and definition separated by a dash are used instead. In my opinion the lack of support for definition lists in a popular word processing package contributes to this (fortunately LaTeX and wikis have equally good support for this as for the other types of lists).
They clearly should be used for definitions, but the HTML specification suggests using them for dialogue, although there are arguments against it.
These elements have also a disadvantage – sites without them are probably even less usable for people who correctly specify acronyms, link titles and definition lists.
]]>LaTeX already uses the ASCII hyphen character for both hyphens, minuses and dashes. If it is used in math mode, then it is a minus. Otherwise, - becomes a hyphen, -- an endash and --- becomes an emdash. The difference between endashes and emdashes lays only in their appearance, different languages require different ones with different spacing. This is the reason why I wrote the onedash package providing a single command, \dash, for typesetting the correct dash in the language and style of the document.
My new package, hyphdash, makes a dash or hyphen from a single hyphen with correct spacing. Both it, onedash and quoted (an equivalent of onedash for quotes) are available in my Mercurial repository. They are licensed under the GNU General Public License, version 3 or later.
Hyphens have two uses – they appear in compound words and in words divided across lines. Fortunately, the second use is done automatically by LaTeX and does not affect writing the package. Compound words do not have any spaces before the hyphen, but in lists like ‘mono- and polycrystals’ they may be followed by a space.
Dashes are sometimes surrounded by equal spaces – like in the British style used on this blog – or without spaces—like in the American style used in this sentence—or by unequal spaces. Usually the left space is unbreakable. This package assumes that dashes are surrounded by any normal spaces, i.e. input characters interpreted as spaces by TeX. (Unbreakable spaces appear more arcane than dashes or even inner quote marks, so they are unsupported in input by this package, but the output will have them.) TeXnical reason for this will be stated later.
My package does nearly all of its work in a macro which the hyphen made active character is defined to (see the packages’ README file of information about using this package). This expands to \relax followed by a normal hyphen if math mode is used (the \relax is probably useful in tables). The same result is obtained in horizontal mode if the current font have nonpositive space stretch parameter which probably occurs only for typewriter fonts.
In vertical mode a special dash is used, useful for representing dialogue in Polish texts (English use quote marks for this, making it easier to determine where a multiparagraph speech ends, and making inner quotes common). Probably no word begins with a hyphen, so this is used. Unix and GNU programs have commandline options beginning with a hyphen, but they are typeset in typewriter type (so there is a special case for it).
The complex part lays in the horizontal mode. Hyphens do not have leading spaces, so the are made if \lastskip does not contain positive value. So the common incorrect form of dash, alpha- beta will be kept as a hyphen. In other cases a dash is made. This ignores the possibility of having numeric ranges with endashes, like ‘69–105’. Instead, 69-105 will use a hyphen and 69 - 105 will use a much more incorrect (but more probable to be included in the input?) dash. Detecting digits before a hyphen is impossible without making all characters active (this could work only for verbatim typesetting of files, but this does not need dashes), and detecting them after the hyphen won’t distinguish such cases as ‘69–105’ and ‘2-chloro-3-methylpentane’.
There is also another problem – dashes represented by multiple hyphens. The first one will be made a dash, but the following ones (since there is no preceding space) will become hyphens. The standard ligatures for dashes will not be used. Maybe the macro could detect following hyphens, but ‘simple’ solutions like \@ifnextchar used for optional parameters will not work, since the hyphen is a complicated macro instead of a character. Changing catcodes (e.g. redefining the hyphen to a character) will not work with texts changing catcodes. The first hyphen could set a conditional to ignore following ones, but it would be difficult to change it to not ignore hyphens in the next dash. The rest is probably more difficult that these solutions. Therefore only a single hyphen may be used with this package as a dash. The macros \textendash and \textemdash may be used instead of multiple hyphens to make a dash character.
Another problem are hyphens used as minuses for numbers interpreted by TeX. For example, \hspace{-1em} will produce strange results and two error messages. In my opinion the only solutions to this are to not use a hyphen in arguments of such commands (e.g. by using the \hyphen macro or by putting all such things into the preamble), or to redefine each primitive TeX command to change the macro making dashes and hyphens (probably it is impossible to detect where such changes should be made). It is obvious why the first solution is used in the package.
This package uses also an example document as an automatic test to find regressions in future versions (I’ve written previously about this in other packages for dashes and quotes). From the example I learned about most of the limitations of this package described here.
]]>uniq may list repeated lines from its output. Its implementation in GNU Coreutils supports listing all occurrences of such lines. How could this be used to list files of the same content located in a given directory?
The solution is to input to uniq with appropriate options a file consisting of lines isomorphic to the contents of files to be compared. Since uniq compares only adjacent lines, this file would have to be sorted.
Therefore a one-to-one function from files to lines should be used. Cryptographic hash functions are treated as such, although they aren’t – they accept any finite byte sequence as input and output a constant size byte sequence. There are many hashes which are now considered insecure enough for important systems (e.g. when it is easy to obtain reverse mappings or different inputs with the same output), but SHA-512 is currently used in such systems. GNU Coreutils have a program called sha512sum computing this hash of files given on the command line, so it can be easily used for this task.
Each output line of this program consists of 512 bit hexadecimal hash (i.e. 128 hexadecimal digits) and the file name, separated by several spaces. Clearly, it would be useful to know the names of repeated files, not only their SHA-512 hashes, so the output of sha512sum will be wholly passed to uniq.
The list of files for sha512sum can be generated by find $dir -type f where $dir is any directory. This command will output each file in this directory or its subdirs. The test -type f requests it to output only regular files, since they are the only ones with content.
The whole pipe listing names of duplicate files is:
find $dir -type f -print0 | xargs -0 sha512sum \ | sort | uniq -w 128 -d --all-repeated=separate \ | sed 's/^[0-9a-f]\+ \+//'
The command xargs passes its input as arguments to sha512sum. Since file names may contain spaces, the options -print0 of find and -0 of xargs will request them to separate the file names with zero bytes to avoid treating a file name with spaces as names of two files.
The arguments of uniq do things described before – -w 128 limits the comparison to first 128 bytes of each line, i.e. to the hash, -d omits unique lines from the output, and --all-repeated=separate outputs all repeated lines, separated by blank lines (of these three options only -d is required by POSIX and supported by uniqs used in BSDs). The final sed expression omits the hash from the output which is probably not useful.
Here sort is used only due to the way in which uniq works. It doesn’t look useful to have different files sorted according to their SHA-512 sums. It might be useful to have files in each duplicate group sorted alphabetically, but this probably could be done faster, since multiple sorts of small sequences are faster then a single sort of their sum (here also the output will be probably much smaller than the input – on my system running the above pipe on /usr/share/man gave only about one third of lines of the output of find /usr/share/man -type f, including extra blank lines). Programs like my ununiq find all repeated lines of an unsorted input, but it does not support the options necessary for this task.
ununiq – a program listing unique lines of its output, but unlike uniq using a hash table and treating also nonadjacent lines as duplicates. Today I decided to check if it will compile on an operating system not using the GNU C library – OpenSolaris.
Using virtualization software like KVM or Virtual Machine Manager it is easy to access several free operating systems at once, so (after some networking-related voodoo) I installed OpenSolaris 2009.06 on a virtual machine without any problem. After installation I found that I cannot log in to the system with empty password (I do not see any security advantage of a password on such virtual machine), so I installed it again with some trivial passwords. I called this new system Laurelin.
Then on Laurelin I installed GNU Autoconf and Automake in newest available versions, i.e. several releases older than on Gentoo. GLib also was trivial to install. Then I tried to use autoreconf, but it didn’t work, since unlike Gentoo OpenSolaris does not link aclocal-1.10, automake-1.10, etc, to files without version numbers. Most probably there are better solutions, but for this one use it was easier to just call these programs directly.
This showed the first problem – other systems do not have newest Autotools – so packages should support the elder ones or provide files generated by newer versions, like configure, Makefile.in and many others. In this case it was easier and probably better to remove unnecessary Automake options introduced in 1.11 and allow Autoconf 2.61.
Then running ./configure showed another problem – the shell found syntax errors in unexpanded M4 macros. This was caused by missing pkg-config, since my package did not include its macros and it wasn’t required by GLib in the OpenSolaris package system. The solution was to install pkg-config. Such problems make source-based operating system distributions more appropriate for programming, since they need such packages to install GLib.
The linker found the next problem, lack of the getline() function. My configure script did not check for it, since it wouldn’t do anything with this. Although this function is included in the newest POSIX, GNU C library treats it as a GNU extensions and other C libraries do not support it. The solution was to check how bad the portable alternatives are and choose one of them.
Therefore I added to the package an implementation of getline() from Gnulib. Then the package compiled successfully on both OpenSolaris and Gentoo.
A test suite checked if this package works correctly with a small subset of its possible options and only one input file. Porting software to other systems should begin with writing a more complete test suite, but this still was enough to improve some parts of the program.
]]>The UK TeX FAQ lists three main problems:
The first problem makes some TeX documents impossible to represent in HTML in a useful way. For example, the following document’s text depends on its layout:
\documentclass{article} \usepackage[a4paper]{geometry} \usepackage{lipsum} \begin{document} \lipsum[1] \edef\nlines{\the\prevgraf} The previous paragraph has \nlines\ lines. \end{document}
The text changes when the paper size is changed to e.g. A5. Clearly, any text in HTML would be different than this or false on at least some systems. Therefore the following programs will be tested just on a typical document, looking slightly like a mathematical book.
Of these listed at the UK TeX FAQ four are available in Gentoo GNU/Linux (so probably other operating systems’ package managers also allow them to be installed easily). Since two of them use tables for mathematical expressions, I haven’t used them. The two others are LaTeX2HTML and TeX4HT
As the UK TeX FAQ states, LaTeX2HTML is a Perl script does not use TeX to process the files to convert. Therefore it does not support some packages and completely ignores macros of my acronyms package. I could try to read the LaTeX2HTML manual to check if it supports loading LaTeX packages, or if I should write a special version of it for this. But I thought that it would be simpler to try just TeX4HT.
Although it ignores unknown commands, it passes unknown environments to LaTeX. Therefore all theorems in my document are rendered to bitmaps and references to them do not work correctly. Their layout is also strange. It would be possible to change LaTeX2HTML to support such environments, but it would be simpler to try other alternatives first.
TeX4HT differs from most similar programs by using TeX to interpret LaTeX code correctly. Therefore it understood my acronyms macros correctly and it would be possible to write a package which would typeset them differently for HTML. Theorems and the ‘ł’ in my name were typeset correctly (the input file used UTF-8 encoding, rarely supported by programs interpreting LaTeX input).
Mathematical formulas still were typeset (into bitmaps) incorrectly, with missing symbols and some parts converted to HTML (so they were not aligned on the math axis).
Therefore, if I will try to convert a LaTeX document to HTML, I will probably use TeX4HT and try to determine why the formulas in my sample document are not converted correctly.
]]>uniq – it counts only adjacent identical lines as duplicates, although it would be technically interesting to implement such program without this restriction while not sorting the input. Then I wrote three implementations of such program and compared their performance on unpublished private data based on my blog’s access log.
Today I decided to make one of these three programs a nearly replacement for sort | uniq for situations where fast online algorithm without changing the order of input lines is better then possibly smaller memory use. For this I implemented support for some of uniq’s command line options and plan to write a man page.
I named the program ununiq, since this name is easy to pronounce and opposite of uniq. The program’s source is available in my Mercurial repository. Like most of my programs it is licensed under the GNU General Public License, version 3 or later.
The choice between three implementations of the basic algorithm – one in C with GLib, one in C++ with Qt and one in Python – was obvious for me. I wanted it to be:
sort | uniq (all three programs satisfied this on the test data)Performance of my program in C with minimal C++ and Qt was worse than the one in C with GLib, so instead of using more C++ I decided to use the program written in C with GLib.
uniqBefore this project I used an option of uniq only once, when a complicated pipe counting the number of secondary schools in each province from a CSV file contained sort | uniq -c. Now I know how to use other options of this program.
Now my program supports input and output with named files, and also skipping several leading characters of each line (the -d option). I implemented also the -d option which outputs only repeated lines, but it writes them on the second occurrence. The program would be more complicated with this option while preserving the original order, so it currently ignores this order.
The options are accepted both as short options and GNU long ones. Their support was very easy to implement using GLib command line option parser. Long options have the same names as in GNU Coreutils.
Several currently not implemented options will require reading whole input before starting output, e.g. -c which counts how many times a line appears in the input. It should still be faster than sort | uniq. A similar observation may be made when sorting the output of ununiq, when operating on the perltoc man page ununiq | sort is faster by half than sort | uniq, although both make the same output.
The program needs also more documentation and testing. It would be nice with automatic testing of all options, which would also be very helpful for porting to other operating system.
]]>