Discussion:
emacs-26: `with-eval-after-load' docstring omission
(too old to reply)
Vicente Vera
2018-02-23 13:42:26 UTC
Permalink
Hello.

Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall. The docstrings
refers to ‘eval-after-load’ regarding the FILE argument, but not how
BODY is executed.
Noam Postavsky
2018-02-23 14:16:56 UTC
Permalink
Post by Vicente Vera
Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall. The docstrings
refers to ‘eval-after-load’ regarding the FILE argument, but not how
BODY is executed.
Isn't that an irrelevant implementation detail? I don't see why it
should be documented.
Vicente Vera
2018-02-24 14:42:27 UTC
Permalink
I think is relevant because their behavior is not
identical. Nevertheless, it is proposed just for the sake of clarity.

One might asume, because of similar syntax, that BODY in
‘with-eval-after-load’ will be passed to ‘eval’ just like in
‘eval-after-load’:

(eval-after-load 'asdf
(setq asdf-something nil))
;; (setq asdf-something nil) will be passed to ‘eval’

(with-eval-after-load 'asdf
(setq asdf-something nil))
;; This will be the same as:
;;
;; (eval-after-load 'asdf
;; (lambda () (setq asdf-something nil)))
;;
;; which will be passed to ‘funcall’ and not evaled.

And also, because of their name, which says ‘eval’, one might think
that BODY in ‘with-eval-after-load’ will be run through ‘eval’.

Unless by convention the ‘with-’ prefix is meant to express that BODY
will be treated as a function?
Post by Noam Postavsky
Post by Vicente Vera
Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall. The docstrings
refers to ‘eval-after-load’ regarding the FILE argument, but not how
BODY is executed.
Isn't that an irrelevant implementation detail? I don't see why it
should be documented.
Clément Pit-Claudel
2018-02-27 20:40:36 UTC
Permalink
Post by Noam Postavsky
Post by Vicente Vera
Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall. The docstrings
refers to ‘eval-after-load’ regarding the FILE argument, but not how
BODY is executed.
Isn't that an irrelevant implementation detail? I don't see why it
should be documented.
It is relevant, because the behavior isn't the same as 'eval'-ing a quoted form.

But I don't think the documentation should be fixed; instead, I think the behavior should be changed :)
Stefan Monnier
2018-02-27 20:58:05 UTC
Permalink
Post by Clément Pit-Claudel
Post by Noam Postavsky
Isn't that an irrelevant implementation detail? I don't see why it
should be documented.
It is relevant, because the behavior isn't the same as 'eval'-ing a quoted form.
Yes, the behavior would be different if we quoted the expression and
passed it to `eval`, but in both cases it's an internal implementation
detail, and the relevant difference is the resulting change in
the semantics.
Post by Clément Pit-Claudel
But I don't think the documentation should be fixed; instead, I think the
behavior should be changed :)
I think you're blinded by your current problem.
Try and remember the world of Coq. Think about equivalences like

(defun FUN () EXP)
(with-eval-after-load FILE (FUN))
=
(with-eval-after-load FILE EXP)

which are currently true (to the extent such things can be true in
Elisp where FUN can be redefined, of course) whereas

(defun FUN () EXP)
(eval-after-load FILE '(FUN))

(eval-after-load FILE 'EXP)

[ Well, of course, this equivalence is actually true as well given the
current compiler-macro, but I'm assuming above that we undo this
compiler macro. ]

IOW, the behavior you seem to want is semantically a good bit more messy.

In your case, you do want the quote because you need to delay
macro-expansion, but I think it's cleaner to solve this problem by
*explicitly* using a quote, rather than by relying on the messy
semantics of the macro/function you happen to be using.

E.g. your exact same problem could show up if you used, say,

(add-hook 'flycheck-mode-hook ...)

instead of

(with-eval-after-load ...)

but you wouldn't ask for a change in add-hook to "solve" your problem.


Stefan
Clément Pit-Claudel
2018-02-28 14:53:09 UTC
Permalink
Post by Stefan Monnier
(defun FUN () EXP)
(eval-after-load FILE '(FUN))

(eval-after-load FILE 'EXP)
[ Well, of course, this equivalence is actually true as well given the
current compiler-macro, but I'm assuming above that we undo this
compiler macro. ]
IOW, the behavior you seem to want is semantically a good bit more messy.
Is that because macroexpansion happens eagerly in the body of the defun, too?
I guess I just don't expect that to happen unless the file is byte-compiled, really :/
Post by Stefan Monnier
In your case, you do want the quote because you need to delay
macro-expansion, but I think it's cleaner to solve this problem by
*explicitly* using a quote, rather than by relying on the messy
semantics of the macro/function you happen to be using.
I'm not sure I follow: doesn't the compiler macro mean that even explicitly using a quote isn't enough?
Post by Stefan Monnier
E.g. your exact same problem could show up if you used, say,
(add-hook 'flycheck-mode-hook ...)
instead of
(with-eval-after-load ...)
but you wouldn't ask for a change in add-hook to "solve" your problem.
Would the problem there be that the body of the lambda I'd pass to add-hook would be eagerly macro-expanded?
I think the big difference between these two is that I expect `with-eval-after-load` to really behave differently, and the reason for expecting that is that we don't otherwise have a convenient way to delay macroexpansion and execution of a form until a file is loaded, right?

Maybe we need a `with-macroexp-and-eval-after-load`?

Clément.
Stefan Monnier
2018-02-28 23:22:22 UTC
Permalink
Post by Clément Pit-Claudel
Is that because macroexpansion happens eagerly in the body of the defun, too?
More or less. It's because it *can* happen eagerly (it also does
nowadays in many cases, but the exact time when macroexpansion takes
place is not specified precisely and it can&does vary depending on
circumstances and Emacs version).
Post by Clément Pit-Claudel
I guess I just don't expect that to happen unless the file is
byte-compiled, really :/
Whether the code is byte-compiled or not is an implementation detail
which could change. Any code which depends on whether it gets compiled
is considered as buggy in my book.
Post by Clément Pit-Claudel
Post by Stefan Monnier
In your case, you do want the quote because you need to delay
macro-expansion, but I think it's cleaner to solve this problem by
*explicitly* using a quote, rather than by relying on the messy
semantics of the macro/function you happen to be using.
I'm not sure I follow: doesn't the compiler macro mean that even explicitly
using a quote isn't enough?
In that one specific case, yes, but in general using a quote is sufficient.

You should not use `eval-after-load` nowadays, you should only use
`with-eval-after-load`, so you'd write something like

(with-eval-after-load 'flycheck
;; Use eval+quote to delay macro expansion.
(eval '(flycheck-define-command ...)))
Post by Clément Pit-Claudel
Post by Stefan Monnier
E.g. your exact same problem could show up if you used, say,
(add-hook 'flycheck-mode-hook ...)
instead of
(with-eval-after-load ...)
but you wouldn't ask for a change in add-hook to "solve" your problem.
Would the problem there be that the body of the lambda I'd pass to add-hook
would be eagerly macro-expanded?
At least could, yes.
Post by Clément Pit-Claudel
I think the big difference between these two is that I expect
`with-eval-after-load` to really behave differently, and the reason for
expecting that is that we don't otherwise have a convenient way to delay
macroexpansion and execution of a form until a file is loaded, right?
Maybe we need a `with-macroexp-and-eval-after-load`?
What I'm trying to say is that delaying macroexpansion is something
that's needed in many more cases than in conjunction to
`eval-after-load` so any solution specifically linked to
`eval-after-load` is just an ad-hoc hack.


Stefan
Clément Pit-Claudel
2018-03-17 06:01:33 UTC
Permalink
Post by Stefan Monnier
You should not use `eval-after-load` nowadays, you should only use
`with-eval-after-load`, so you'd write something like
(with-eval-after-load 'flycheck
;; Use eval+quote to delay macro expansion.
(eval '(flycheck-define-command ...)))
I got bitten by this again today, this time with this:

(with-eval-after-load 'flycheck
(push "npx" (flycheck-checker-get 'javascript-eslint 'command)))

… which failed with this message when Flycheck was actually loaded:

Debugger entered--Lisp error: (void-function \(setf\ flycheck-checker-get\))
(\(setf\ flycheck-checker-get\) (cons "npx" (flycheck-checker-get 'javascript-eslint 'command)) 'javascript-eslint 'command)

I'm slowly developing the habit of writing (with-eval-after-load 'flycheck (eval '(…))), but I'm really not a fan of this pattern :/

What do we gain from byte-compiling the bodies of with-eval-after-load forms? Byte-compilation warnings for undefined variables, but these are usually false positives (precisely because the variables in a with-eval-after-load are often defined by the package in question). And some performance, but with-eval-after-load is discouraged in packages and its body typically runs once (or very few times?), so its performance should be essentially irrelevant.

Maybe we can find a way to prevent macro-expansion issues and make the behavior more intuitive?

Cheers,
Clément.
Stefan Monnier
2018-03-17 14:19:22 UTC
Permalink
Post by Stefan Monnier
(with-eval-after-load 'flycheck
(push "npx" (flycheck-checker-get 'javascript-eslint 'command)))
Debugger entered--Lisp error: (void-function \(setf\ flycheck-checker-get\))
(\(setf\ flycheck-checker-get\) (cons "npx" (flycheck-checker-get 'javascript-eslint 'command)) 'javascript-eslint 'command)
Yes, in this specific case, flycheck could actually make it work by defining
a (setf flycheck-checker-get) function.
Post by Stefan Monnier
What do we gain from byte-compiling the bodies of with-eval-after-load forms?
As mentioned before: cleaner semantics. Macro-expanding the body of
with-eval-after-load more lazily (which is the core of the problem
you're hitting) won't help you if you happened to write your code as:

(defun my-after-flycheck ()
(push "npx" (flycheck-checker-get 'javascript-eslint 'command)))
(with-eval-after-load 'flycheck
(my-after-flycheck))
Post by Stefan Monnier
Maybe we can find a way to prevent macro-expansion issues and make the
behavior more intuitive?
I'd suggest to start by trying to find a way to get the behavior you
want *without* relying on (with-)eval-after-load (which is fiddly in any
case for all kinds of reasons. It's a handy tool when you need it, but
if you need it I consider that it's usually a sign that there's
a problem elsewhere).
Of course, this may require changes in flycheck.


Stefan
Clément Pit-Claudel
2018-04-13 16:00:41 UTC
Permalink
Post by Stefan Monnier
As mentioned before: cleaner semantics. Macro-expanding the body of
with-eval-after-load more lazily (which is the core of the problem
(defun my-after-flycheck ()
(push "npx" (flycheck-checker-get 'javascript-eslint 'command)))
(with-eval-after-load 'flycheck
(my-after-flycheck))
Right. That function should be moved into the with-eval-after-load block :)
Post by Stefan Monnier
I'd suggest to start by trying to find a way to get the behavior you
want *without* relying on (with-)eval-after-load (which is fiddly in any
case for all kinds of reasons. It's a handy tool when you need it, but
if you need it I consider that it's usually a sign that there's
a problem elsewhere).
Of course, this may require changes in flycheck.
Sorry for the long delay. I spend some time thinking about this, but I didn't reach a satisfactory conclusion.

Flycheck defines a few of macros. Our users want to write code that depends on them in their init files, but don't care for that code to run unless they are in fact using Flycheck.

What's the right approach?
Couldn't we just define a with-macroexpand-and-eval-after-load macro, and call it a day?

Clément.
Stefan Monnier
2018-04-13 18:02:50 UTC
Permalink
Post by Clément Pit-Claudel
Flycheck defines a few of macros. Our users want to write code that depends
on them in their init files, but don't care for that code to run unless they
are in fact using Flycheck.
I don't think users really want to use your macros.
Instead they have to use your macros in order to get something else done.

E.g. you could make it unnecessary for them to use eval-after-load by
letting them write something like

(setq flycheck-extra-commands
'((foo1 "blibla" :command bar :error-patterns baz)
(foo2 "blabli" :command tar :error-patterns taz)))

instead of some uses of flycheck-define-command.

That same kind of approach is used for font-lock, outline, imenu, etc..

It's not always ideal: e.g. for syntax-propertize I did not follow this
approach and did decide to rely on a `syntax-propertize-rules` macro
because I felt there was a substantial gain to processing these rules in
a macro. I think font-lock would also benefit from a similar macro.

But in the case of flycheck-define-command at least, I don't see much
benefit to having it be a macro instead of some inert data in
a variable.
Post by Clément Pit-Claudel
Couldn't we just define a with-macroexpand-and-eval-after-load macro, and call it a day?
I'd be perfectly happy to add a new macro like

(defmacro with-lazy-macro-expansion (&rest body)
`(eval '(progn ,@body) lexical-binding))

but I'm reluctant to add a macro that's specific to eval-after-load
since the problem is more general.


Stefan
Clément Pit-Claudel
2018-04-13 18:28:57 UTC
Permalink
Post by Stefan Monnier
But in the case of flycheck-define-command at least, I don't see much
benefit to having it be a macro instead of some inert data in
a variable.
Except that form (flycheck-define-checker) defines a variable, too, right? That is, it expands to a defvar and a function call.

The concrete issue stems from users copying a form from flycheck.el intto their init file, without macro-expanding it. Of course, we can recommend that users copy the macro-expanded version (the defvar + the function call) — but that's not ideal, because not many of our users know how to do that (macroexpand a form). And macroexpanding all uses in flycheck.el isn't an option either.
Post by Stefan Monnier
I'd be perfectly happy to add a new macro like
(defmacro with-lazy-macro-expansion (&rest body)
but I'm reluctant to add a macro that's specific to eval-after-load
since the problem is more general.
That would be nice. It would then be safe to do

(with-eval-after-load
(with-lazy-macro-expansion
…))

right?

Clément.
Stefan Monnier
2018-04-13 18:41:30 UTC
Permalink
Post by Clément Pit-Claudel
The concrete issue stems from users copying a form from flycheck.el intto
their init file, without macro-expanding it. Of course, we can recommend
that users copy the macro-expanded version (the defvar + the function call)
— but that's not ideal, because not many of our users know how to do that
(macroexpand a form). And macroexpanding all uses in flycheck.el isn't an
option either.
Ah, a social engineering problem. Sorry, you're talking to the wrong guy.
Post by Clément Pit-Claudel
That would be nice. It would then be safe to do
(with-eval-after-load
(with-lazy-macro-expansion
…))
right?
That's right.


Stefan
Stefan Monnier
2018-04-13 22:43:06 UTC
Permalink
Post by Clément Pit-Claudel
Post by Stefan Monnier
But in the case of flycheck-define-command at least, I don't see much
benefit to having it be a macro instead of some inert data in
a variable.
Except that form (flycheck-define-checker) defines a variable, too, right?
That is, it expands to a defvar and a function call.
So what? You can just as well define those variables while iterating
down a list of checker-definitions.
Post by Clément Pit-Claudel
The concrete issue stems from users copying a form from flycheck.el intto
their init file, without macro-expanding it. Of course, we can recommend
that users copy the macro-expanded version (the defvar + the function call)
— but that's not ideal, because not many of our users know how to do that
(macroexpand a form). And macroexpanding all uses in flycheck.el isn't an
option either.
If you replace flycheck.el's

(flycheck-define-checker checker1 args1)
(flycheck-define-checker checker2 args2)
[...]

into

(defconst flycheck-builtin-checkers
'((checker1 args1)
(checker2 args2)
...))

then people will still be able to copy&paste between flycheck.el and
their own

(setq flycheck-extra-checkers ...)

and if they don't want to `setq` you can autoload the var to be nil so
they can use `push` or `add-to-list` on it.


-- Stefan

Stefan Monnier
2018-02-27 20:21:25 UTC
Permalink
Post by Vicente Vera
Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall.
Not sure how else you'd expect it to work. I suspect that you don't
*really* care about it works internally, but you bumped into some
behavior you don't expect and it's somewhat related to the way it works
internally. If so, please clarify what's the *actual* problem you
bumped into.


Stefan
Vicente Vera
2018-03-03 15:06:21 UTC
Permalink
(In reply to Stefan.)

No, there is no specific issue involved. Just some harmless and
temporary confusion, that's all.

After reading ‘with-eval-after-load’ docstring (which just says
"Execute BODY after FILE is loaded.") I expected that BODY would be
executed just like in ‘eval-after-load’, which by default passes forms
to ‘eval’.

My argument is that there are implicit behaviours that *might* lead to
wrong assumptions. In my case, I had to look at ‘with-eval-after-load’
code and re-read ‘eval-after-load’ docstring to clearly understand the
differences.
Post by Vicente Vera
Hello.
Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall. The docstrings
refers to ‘eval-after-load’ regarding the FILE argument, but not how
BODY is executed.
Clément Pit-Claudel
2018-03-03 15:38:28 UTC
Permalink
Post by Vicente Vera
I expected that BODY would be
executed just like in ‘eval-after-load’, which by default passes forms
to ‘eval’.
Hmm. eval-after-load doesn't pass forms through eval, actually, AFAICT (it uses a lambda, just like with-eval-after-load)
Stefan Monnier
2018-03-03 21:52:45 UTC
Permalink
Post by Vicente Vera
After reading ‘with-eval-after-load’ docstring (which just says
"Execute BODY after FILE is loaded.") I expected that BODY would be
executed just like in ‘eval-after-load’, which by default passes forms
to ‘eval’.
But then that would mean it does the same as eval-after-load.
The whole point of with-eval-after-load is that it doesn't do the
"double evaluation" of eval-after-load (which was a frequent source of
errors/confusion).

Do you have some suggestion for how we could change
with-eval-after-load's doc to avoid your confusion?


Stefan
Vicente Vera
2018-03-04 16:04:44 UTC
Permalink
“Hmm. eval-after-load doesn't pass forms through eval, actually,
AFAICT (it uses a lambda, just like with-eval-after-load)”

“The whole point of with-eval-after-load is that it doesn't do the
"double evaluation" of eval-after-load (which was a frequent source of
errors/confusion).”

Indeed. So the difference between the two is:
- ‘with-eval-after-load’ calls ‘eval-after-load’ to run BODY as a
function. Conveniently, there's no need to quote nor wrap
expressions in a lambda form.
- ‘eval-after-load’ runs FORM as a function in both cases: if FORM is
a function or a quoted expression. The latter is the "double
evaluation" case: ‘eval-after-load’ is called twice to wrap FORM as
a function.

Still, an elaboration on the behavior of ‘with-eval-after-load’ could
be useful. Something like: "BODY will be called as a function with no
arguments through ‘funcall’." (as per ‘eval-after-load’
docstring). And probably an addition to the last paragraph: "See
‘eval-after-load’ for more details about... and how BODY will be
evaluated.".
Post by Vicente Vera
(In reply to Stefan.)
No, there is no specific issue involved. Just some harmless and
temporary confusion, that's all.
After reading ‘with-eval-after-load’ docstring (which just says
"Execute BODY after FILE is loaded.") I expected that BODY would be
executed just like in ‘eval-after-load’, which by default passes forms
to ‘eval’.
My argument is that there are implicit behaviours that *might* lead to
wrong assumptions. In my case, I had to look at ‘with-eval-after-load’
code and re-read ‘eval-after-load’ docstring to clearly understand the
differences.
Post by Vicente Vera
Hello.
Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall. The docstrings
refers to ‘eval-after-load’ regarding the FILE argument, but not how
BODY is executed.
Stefan Monnier
2018-03-05 03:39:32 UTC
Permalink
Post by Vicente Vera
Still, an elaboration on the behavior of ‘with-eval-after-load’ could
be useful. Something like: "BODY will be called as a function with no
arguments through ‘funcall’."
That can't be the doc: going though funcall is an
implementation detail. The doc has to explain the behavior
independently from how it's implemented.


Stefan
Noam Postavsky
2018-03-09 01:00:40 UTC
Permalink
On Sun, Mar 4, 2018 at 10:39 PM, Stefan Monnier
Post by Stefan Monnier
Post by Vicente Vera
Still, an elaboration on the behavior of ‘with-eval-after-load’ could
be useful. Something like: "BODY will be called as a function with no
arguments through ‘funcall’."
That can't be the doc: going though funcall is an
implementation detail. The doc has to explain the behavior
independently from how it's implemented.
Does it though (eval-after-load's docstring doesn't seem to)? Maybe we
should just say

Equivalent to (eval-after-load FILE (lambda () BODY)).
Stefan Monnier
2018-03-09 04:55:03 UTC
Permalink
Post by Noam Postavsky
Does it though (eval-after-load's docstring doesn't seem to)? Maybe we
should just say
Equivalent to (eval-after-load FILE (lambda () BODY)).
I'd like to move towards eliminating uses of eval-after-load, so that
doesn't sound too appealing.


Stefan
Vicente Vera
2018-03-09 00:47:01 UTC
Permalink
Hmm, I see. I'm sorry for the noise.

Also, from what's been said here, the internal behavior of both
functions is broadly equivalent?
Post by Vicente Vera
“Hmm. eval-after-load doesn't pass forms through eval, actually,
AFAICT (it uses a lambda, just like with-eval-after-load)”
“The whole point of with-eval-after-load is that it doesn't do the
"double evaluation" of eval-after-load (which was a frequent source of
errors/confusion).”
- ‘with-eval-after-load’ calls ‘eval-after-load’ to run BODY as a
function. Conveniently, there's no need to quote nor wrap
expressions in a lambda form.
- ‘eval-after-load’ runs FORM as a function in both cases: if FORM is
a function or a quoted expression. The latter is the "double
evaluation" case: ‘eval-after-load’ is called twice to wrap FORM as
a function.
Still, an elaboration on the behavior of ‘with-eval-after-load’ could
be useful. Something like: "BODY will be called as a function with no
arguments through ‘funcall’." (as per ‘eval-after-load’
docstring). And probably an addition to the last paragraph: "See
‘eval-after-load’ for more details about... and how BODY will be
evaluated.".
Post by Vicente Vera
(In reply to Stefan.)
No, there is no specific issue involved. Just some harmless and
temporary confusion, that's all.
After reading ‘with-eval-after-load’ docstring (which just says
"Execute BODY after FILE is loaded.") I expected that BODY would be
executed just like in ‘eval-after-load’, which by default passes forms
to ‘eval’.
My argument is that there are implicit behaviours that *might* lead to
wrong assumptions. In my case, I had to look at ‘with-eval-after-load’
code and re-read ‘eval-after-load’ docstring to clearly understand the
differences.
Post by Vicente Vera
Hello.
Currently, ‘with-eval-after-load’ docstring doesn't mention that BODY
is executed as a lambda expression through funcall. The docstrings
refers to ‘eval-after-load’ regarding the FILE argument, but not how
BODY is executed.
Loading...