Discussion:
Making font-lock handle long lines better
(too old to reply)
Ryan B
2017-02-10 15:44:15 UTC
Permalink
hi all! it seems like it's been a known issue for a while that font-lock is
slow on very long lines. i've tried lots of optimizations -
jit-lock/lazy-lock, font-lock-maximum-decoration, etc. - but even on modern
hardware, fontifying still hangs my emacs for 5-10s on long lines, e.g.
10k-20k characters, whenever they appear.

i'm thinking of adding an opt-in line length limit to font-lock-mode, after
which it would stop fontifying, evaluating regexps, etc. within each line.
i'd love to hear opinions on whether that makes sense, and whether you
might consider accepting it into trunk.

thanks in advance!
--
https://snarfed.org/
j***@verona.se
2017-02-10 15:57:43 UTC
Permalink
hi all! it seems like it's been a known issue for a while that font-lock is slow on very long lines. i've tried lots of optimizations - jit-lock/lazy-lock, font-lock-maximum-decoration, etc. - but even on modern hardware, fontifying still hangs my emacs for 5-10s on
long lines, e.g. 10k-20k characters, whenever they appear.
i'm thinking of adding an opt-in line length limit to font-lock-mode, after which it would stop fontifying, evaluating regexps, etc. within each line. i'd love to hear opinions on whether that makes sense, and whether you might consider accepting it into trunk.
As a user, I would appreciate this feature. I dont know if its feasible
to implement it though.
thanks in advance!
--
Joakim Verona
***@verona.se
Eli Zaretskii
2017-02-10 16:24:48 UTC
Permalink
Date: Fri, 10 Feb 2017 07:44:15 -0800
hi all! it seems like it's been a known issue for a while that font-lock is slow on very long lines. i've tried lots of
optimizations - jit-lock/lazy-lock, font-lock-maximum-decoration, etc. - but even on modern hardware,
fontifying still hangs my emacs for 5-10s on long lines, e.g. 10k-20k characters, whenever they appear.
i'm thinking of adding an opt-in line length limit to font-lock-mode, after which it would stop fontifying, evaluating
regexps, etc. within each line. i'd love to hear opinions on whether that makes sense, and whether you might
consider accepting it into trunk.
Please file a bug report by sending this information using
report-emacs-bug. If you can attach to your report a minimal test
case that exhibits the problematic behavior, it will be appreciated.

Thanks.
Wilfred Hughes
2017-02-11 20:10:15 UTC
Permalink
Post by Eli Zaretskii
If you can attach to your report a minimal test
case that exhibits the problematic behavior, it will be appreciated.
I have a minimal test case in
http://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow
which has a few useful suggestions for workarounds.

I think a line length limit would be a very worthwhile addition.

Wilfred
Eli Zaretskii
2017-02-11 20:28:38 UTC
Permalink
Date: Sat, 11 Feb 2017 20:10:15 +0000
Post by Eli Zaretskii
If you can attach to your report a minimal test
case that exhibits the problematic behavior, it will be appreciated.
I have a minimal test case in
http://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow
which has a few useful suggestions for workarounds.
Since it's slow in Fundamental mode as well, it's unrelated to
font-lock, and thus to this thread. Display of very long lines is
known to be slow in Emacs.
Anders Lindgren
2017-02-11 23:06:41 UTC
Permalink
Post by Eli Zaretskii
Since it's slow in Fundamental mode as well, it's unrelated to
font-lock, and thus to this thread. Display of very long lines is
known to be slow in Emacs.
I believe this is a font-lock problem. If you disable global-font-lock-mode
before you open the file, it opens in seconds. If global-font-lock-mode is
active Emacs stops responding (I gave up after a few minutes).

A quick experiment suggested that the problem is the member
`font-lock-extend-region-wholelines' of `font-lock-extend-region-functions'
that ensures that whenever font-lock highlights keywords in a region, the
full line is rehighlighted (in the case of the example, the full 18M). By
removing it, font-locking the file is done in seconds. (I assume that only
the visible parts of the buffer is highlighted.)

Clearly, simple dropping the function is not a good idea, as keyword
fontification could start anywhere on the line, e.g. in the middle of
identifiers. A better approach would be to extend the region to include
10KB in either direction, plus ensure that the region isn't split in the
middle of identifiers.

/ Anders

Ps. In case anybody is interested in looking further into this, you can use
the tool https://github.com/Lindydancer/highlight-refontification to get
visual feedback on the font-lock refontification process.
Ryan B
2018-05-13 19:44:09 UTC
Permalink
for the record, i narrowed this down to the maven element of
`compilation-error-regexp-alist'. if i remove that element,
`compilation-mode' is fast on the minimal test case in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682 .

it's evidently a somewhat known issue:
https://blog.danielgempesaw.com/post/129841682030/fixing-a-laggy-compilation-buffer
https://www.reddit.com/r/emacs/comments/2b2j5m/emacs_unresponsive_during_compilebuild/
http://comments.gmane.org/gmane.emacs.bugs/28783 (currently down)
Post by Eli Zaretskii
Since it's slow in Fundamental mode as well, it's unrelated to
Post by Eli Zaretskii
font-lock, and thus to this thread. Display of very long lines is
known to be slow in Emacs.
I believe this is a font-lock problem. If you disable
global-font-lock-mode before you open the file, it opens in seconds. If
global-font-lock-mode is active Emacs stops responding (I gave up after a
few minutes).
A quick experiment suggested that the problem is the member
`font-lock-extend-region-wholelines' of `font-lock-extend-region-functions'
that ensures that whenever font-lock highlights keywords in a region, the
full line is rehighlighted (in the case of the example, the full 18M). By
removing it, font-locking the file is done in seconds. (I assume that only
the visible parts of the buffer is highlighted.)
Clearly, simple dropping the function is not a good idea, as keyword
fontification could start anywhere on the line, e.g. in the middle of
identifiers. A better approach would be to extend the region to include
10KB in either direction, plus ensure that the region isn't split in the
middle of identifiers.
/ Anders
Ps. In case anybody is interested in looking further into this, you can
use the tool https://github.com/Lindydancer/highlight-refontification to
get visual feedback on the font-lock refontification process.
--
https://snarfed.org/
Noam Postavsky
2018-05-15 00:05:40 UTC
Permalink
Post by Ryan B
for the record, i narrowed this down to the maven element of
`compilation-error-regexp-alist'. if i remove that element,
`compilation-mode' is fast on the minimal test case in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682 .
Sounds like Bug#13369, although that report specifically mentions the
'omake' element as the major culprit.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=13369
Jostein Kjønigsen
2018-05-15 08:08:46 UTC
Permalink
Post by Ryan B
for the record, i narrowed this down to the maven element of
`compilation-error-regexp-alist'. if i remove that element,
`compilation-mode' is fast on the minimal test case in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682 .
Sounds like Bug#13369, although that report specifically mentions the> 'omake' element as the major culprit.
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=13369
I know this isn't directly related to the original issue raised, and not
to sound awfully negative...
But if I understand this correctly this is a bug which was filed and
debugged in 2013, *half a decade ago*, with a known fix, but the bug is
*still *in the Emacs codebase.
Why? Because whoever filed and diagnosed the bug in the first place
can't/won't signed copyright assignment papers.
Now that the problem is solved, can't someone else just make the
edit, write the patch, and make this terrible bug go away? It's a one-
line diff after all. We do provide exceptions for such small patches
after all.
But seeing bugs like this, makes me ask some other questions:

- Are there any systems in place to follow up "solved" bugs which
doesn't result in commits?- Do we know how many such bugs there currently are?
- And what improvements could we make to Emacs if there was an
initiative to get all those issues properly closed?
--
Regards
Jostein KjÞnigsen

***@kjonigsen.net 🍵 ***@gmail.com
https://jostein.kjonigsen.net
Eli Zaretskii
2018-05-15 16:50:28 UTC
Permalink
Date: Tue, 15 May 2018 10:08:46 +0200
But if I understand this correctly this is a bug which was filed and debugged in 2013, half a decade ago, with a
known fix, but the bug is still in the Emacs codebase.
Why? Because whoever filed and diagnosed the bug in the first place can't/won't signed copyright
assignment papers.
That's not my reading of the discussion in that bug. The copyright
assignment issue is only tangentially related to the bug, it was
mentioned wrt a package shown in the discussion, but that package is
not the solution to the problem.

As the solution to the problem it was proposed to remove support for
omake or disable it by default. Upon re-reading the discussion, I
didn't perceive any consensus regarding that proposal. In any case,
deletion of some lines doesn't require any copyright assignment.
Now that the problem is solved, can't someone else just make the edit, write the patch, and make this terrible
bug go away? It's a one-line diff after all. We do provide exceptions for such small patches after all.
If there's an agreement to do that, it should be easy. But I'm not
sure there as an agreement.
- Are there any systems in place to follow up "solved" bugs which doesn't result in commits?
This is a volunteer project. Our "system" in place involves a few
motivated individuals, like Noam and a few others, who routinely
review old bug reports. If you are asking whether we have people
whose assigned job is to do that, then no.
- Do we know how many such bugs there currently are?
Please define "such bugs". The bug tracker supports queries, so given
a specification, we could probably provide an answer.
- And what improvements could we make to Emacs if there was an initiative to get all those issues properly
closed?
I hope that more people volunteer to take care of bug reports, both
old and new. Would you like to work on that?

Thanks.
Stefan Monnier
2018-05-15 18:32:32 UTC
Permalink
Post by Eli Zaretskii
As the solution to the problem it was proposed to remove support for
omake or disable it by default. Upon re-reading the discussion, I
FWIW, I think omake is not very popular any more, so maybe it's OK to
just drop it.


Stefan

Ryan B
2017-02-11 22:06:08 UTC
Permalink
On Feb 10, 2017 7:44 AM, "Ryan B" <***@ryanb.org> wrote:

even on modern hardware, fontifying still hangs my emacs for 5-10s on long
lines, e.g. 10k-20k characters, whenever they appear.


I've filled a bug report with a minimal test case for this here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25682
Loading...