You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is a tracking bug for development of combined systemtap and dtrace patch for Python. The separate patches were submitted at [1] (systemtap) and [2] (dtrace).
Since it was agreed that it'd be best to merge the two patches to reuse as much code as possible between them, I'm opening this issue to track the progress and discuss possible improvements and suggestions.
Bohuslav, do you have Jabber/XMPP?. Mine is "jcea@jabber.org". This thing will require real time communication.
I rather prefer XMPP/Jabber, but I could accept IRC :-)
bkabrda commented on May 29, 2014
bkabrdamannequin
MannequinAuthor
More actions
I'd really prefer IRC, if at all possible for you :)
When online, I'm usually on #fedora-python, but I can really come any channel of your choice.
Sorry for the delays in communication, I've got a lot on my plate right now and I'll probably not be very quick to respond
benesch commented on Dec 27, 2014
beneschmannequin
Mannequin
More actions
Since it looks like development on this has stalled, I took the liberty of compiling all prior patches
https://github.com/benesch/python-dtrace-tracker
and am actively working on merging them into one superpatch. I am neither a Python nor a DTrace expert, so I'm happy to step aside should one of you more experienced folks have something ready.
A few thoughts so far:
* SPARC support? Can we drop this? It seems to add a bunch of intrusive
inlined NOPs—that I'm not sure are even necessary.
* @jcea, is this the original DTrace patch? https://github.com/benesch/python-dtrace-tracker/blob/master/dtrace-solaris-oracle-24.patch
I'm trying to determine where those NOPs came from.
Thanks all for the preliminary tasks! I rebuilt the patch from first principles, reusing large parts of Jesús' work, as well as work by Dave and Nikhil.
Things deliberately left out for simplicity:
ustack helpers, I have no way of testing them at this point since
they are Solaris-specific, don't work on macOS or Linux
PyFrameObject * in function__entry/function__return, this is
SystemTap-specific and won't compile with DTrace
SPARC support
dynamic tracing
sys module dtrace facility introspection (as described in the documentation, there's facilities in both Linux and macOS to do the same thing)
All of those might be added later.
I also left out instance new/delete probes for now since I had problems making them stable. We might still be able to squeeze them in if we're quick about it.
Dave, if you have an idea how to integrate your FrameObject pointer in a way that compiles for DTrace, too, this is the time to speak up :)
Jesús, if you'd like to add the ustack helper back in a separate patch and add to the documentation how this can be tested, that's great, too. I just didn't feel comfortable committing code myself that I have no way of testing, especially that it required more code (the offsets, including the generated .h here and there, etc.).
@Łukasz in case you're not aware, all koobs-freebsd* bots are DTrace enabled, and can be tested with the custom builder. I'm on IRC (python-dev) if you need anything from me to help progress this
SilentGhost commented on Oct 29, 2016
SilentGhostmannequin
Mannequin
More actions
Building the documentation on Ubuntu 16.10, I started getting the following warnings:
/Doc/howto/instrumentation.rst:58: WARNING: Could not lex literal_block as "python3". Highlighting skipped.
/Doc/howto/instrumentation.rst:139: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:211: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:328: WARNING: Could not lex literal_block as "c". Highlighting skipped.
I guess the trick of using c for systemtap doesn't work anymore, not entirely sure what's the problem with the python3 highlighting.
SilentGhost commented on Oct 29, 2016
SilentGhostmannequin
Mannequin
More actions
Here is the patch that fixes warning in /Doc/howto/instrumentation.rst as well as cleans up a few nits here and there. I've removed attempt at highlighting using c since that doesn't work in practice.
Fedora so far has been using the systemtap patch downstream from dmalcolm [0].
So for 3.6 by removing the systemtap patch (it cannot be applied anymore cleanly) and by enabling the --with-dtrace configure flag I get this error [1]:
<snip>
make: *** [Makefile:884: Include/pydtrace_probes.h] Error 1
<snip>
If however I touch the pydtrace_probes.h file I get various undefined references [2] and also:
<snip>
make: *** [Makefile:724: Programs/_freeze_importlib] Error 1
<snip>
@Charalampos, the error in your build comes from the /usr/bin/dtrace wrapper:
Traceback (most recent call last):
File "/usr/bin/dtrace", line 440, in <module>
sys.exit(main())
File "/usr/bin/dtrace", line 385, in main
providers.probe_write(s_filename, filename + suffix)
File "/usr/bin/dtrace", line 181, in probe_write
hdr =open(header, mode='w')
FileNotFoundError: [Errno 2] No such file or directory: 'Include/pydtrace_probes.h'
Yes it is RPM specific for that case, in the sense that we create different subfolders for the debug and the normal(optimized) builds under the build/ dir, where the Include directory does not exist. the dtrace wrapper assumes that the directory already exists. The issue of course can be reproduced with any packaging system (or ways to compile python) that follow this approach, although I am not aware currently if this can happen or might be happening anywhere else.
By creating the directory (if it doesn't exist) at the Makefile rule for generating pydtrace_probes.h, this can be circumvented.
Attaching a patch that fixes this issue for consideration.
Activity
bkabrda commentedon May 27, 2014
This is a tracking bug for development of combined systemtap and dtrace patch for Python. The separate patches were submitted at [1] (systemtap) and [2] (dtrace).
Since it was agreed that it'd be best to merge the two patches to reuse as much code as possible between them, I'm opening this issue to track the progress and discuss possible improvements and suggestions.
[1] http://bugs.python.org/issue14776
[2] http://bugs.python.org/issue13405
jcea commentedon May 27, 2014
Bohuslav, do you have Jabber/XMPP?. Mine is "jcea@jabber.org". This thing will require real time communication.
I rather prefer XMPP/Jabber, but I could accept IRC :-)
bkabrda commentedon May 29, 2014
I'd really prefer IRC, if at all possible for you :)
When online, I'm usually on #fedora-python, but I can really come any channel of your choice.
Sorry for the delays in communication, I've got a lot on my plate right now and I'll probably not be very quick to respond
benesch commentedon Dec 27, 2014
Since it looks like development on this has stalled, I took the liberty of compiling all prior patches
and am actively working on merging them into one superpatch. I am neither a Python nor a DTrace expert, so I'm happy to step aside should one of you more experienced folks have something ready.
A few thoughts so far:
Cheers!
ambv commentedon Sep 10, 2016
Landed during the 3.6 beta1 sprint:
https://hg.python.org/cpython/rev/d622dbd71f2b
Thanks all for the preliminary tasks! I rebuilt the patch from first principles, reusing large parts of Jesús' work, as well as work by Dave and Nikhil.
Things deliberately left out for simplicity:
they are Solaris-specific, don't work on macOS or Linux
SystemTap-specific and won't compile with DTrace
All of those might be added later.
I also left out instance new/delete probes for now since I had problems making them stable. We might still be able to squeeze them in if we're quick about it.
Dave, if you have an idea how to integrate your FrameObject pointer in a way that compiles for DTrace, too, this is the time to speak up :)
Jesús, if you'd like to add the ustack helper back in a separate patch and add to the documentation how this can be tested, that's great, too. I just didn't feel comfortable committing code myself that I have no way of testing, especially that it required more code (the offsets, including the generated .h here and there, etc.).
koobs commentedon Sep 10, 2016
@Łukasz in case you're not aware, all koobs-freebsd* bots are DTrace enabled, and can be tested with the custom builder. I'm on IRC (python-dev) if you need anything from me to help progress this
SilentGhost commentedon Oct 29, 2016
Building the documentation on Ubuntu 16.10, I started getting the following warnings:
/Doc/howto/instrumentation.rst:58: WARNING: Could not lex literal_block as "python3". Highlighting skipped.
/Doc/howto/instrumentation.rst:139: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:211: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:328: WARNING: Could not lex literal_block as "c". Highlighting skipped.
I guess the trick of using c for systemtap doesn't work anymore, not entirely sure what's the problem with the python3 highlighting.
SilentGhost commentedon Oct 29, 2016
Here is the patch that fixes warning in /Doc/howto/instrumentation.rst as well as cleans up a few nits here and there. I've removed attempt at highlighting using c since that doesn't work in practice.
stratakis commentedon Nov 2, 2016
Fedora so far has been using the systemtap patch downstream from dmalcolm [0].
So for 3.6 by removing the systemtap patch (it cannot be applied anymore cleanly) and by enabling the --with-dtrace configure flag I get this error [1]:
<snip>
make: *** [Makefile:884: Include/pydtrace_probes.h] Error 1
<snip>
If however I touch the pydtrace_probes.h file I get various undefined references [2] and also:
<snip>
make: *** [Makefile:724: Programs/_freeze_importlib] Error 1
<snip>
[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00055-systemtap.patch
[1] https://kojipkgs.fedoraproject.org//work/tasks/9656/16279656/build.log
[2] https://kojipkgs.fedoraproject.org//work/tasks/9670/16279670/build.log
python-dev commentedon Nov 6, 2016
New changeset d05d312161f2 by Berker Peksag in branch '3.6':
Issue bpo-21590: Silence Sphinx warnings in instrumentation.rst
https://hg.python.org/cpython/rev/d05d312161f2
New changeset 442453fa3370 by Berker Peksag in branch 'default':
Issue bpo-21590: Merge from 3.6
https://hg.python.org/cpython/rev/442453fa3370
berkerpeksag commentedon Nov 6, 2016
Thanks. SilentGhost. I went with ".. highlight:: shell-session" to simplify the patch a bit.
Łukasz, can we close this one and create new issues for further improvements now?
ambv commentedon Nov 7, 2016
Yes, let's close this.
For some reason the path Include/pydtrace_probes.h is not writable. Maybe this is RPM-specific?
stratakis commentedon Nov 8, 2016
@Łukasz
Dug a bit more to it.
Yes it is RPM specific for that case, in the sense that we create different subfolders for the debug and the normal(optimized) builds under the build/ dir, where the Include directory does not exist. the dtrace wrapper assumes that the directory already exists. The issue of course can be reproduced with any packaging system (or ways to compile python) that follow this approach, although I am not aware currently if this can happen or might be happening anywhere else.
By creating the directory (if it doesn't exist) at the Makefile rule for generating pydtrace_probes.h, this can be circumvented.
Attaching a patch that fixes this issue for consideration.
stratakis commentedon Nov 21, 2016
@Łukasz
Would it be possible to review the patch?
Or is it preferable to open a new issue?