➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
标题: Systemtap and DTrace support
类型: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
process
状态: closed Resolution: fixed
Dependencies: 后续:
分配给: lukasz.langa 抄送列表: SilentGhost, benesch, berker.peksag, bkabrda, cburroughs, cstratak, dmalcolm, durin42, jcea, koobs, lukasz.langa, python-dev, xmorel
优先级: normal 关键字: patch

Created on 2014-05-27 13:06 by bkabrda, last changed 2022-04-11 14:58 by admin. This issue is now closed.

文件
文件名 上传时间 Description 编辑
21590_2.diff SilentGhost, 2016-10-29 12:43 review
create-Include-dir-to-properly-generate-pydtrace_probes.h-file.patch cstratak, 2016-11-08 16:43 review
Messages (14)
msg219219 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * 日期: 2014-05-27 13:06
This is a tracking bug for development of combined systemtap and dtrace patch for Python. The separate patches were submitted at [1] (systemtap) and [2] (dtrace).
Since it was agreed that it'd be best to merge the two patches to reuse as much code as possible between them, I'm opening this issue to track the progress and discuss possible improvements and suggestions.

[1] http://bugs.python.org/issue14776
[2] http://bugs.python.org/issue13405
msg219222 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) 日期: 2014-05-27 13:18
Bohuslav, do you have Jabber/XMPP?. Mine is "jcea@jabber.org". This thing will require real time communication.

I rather prefer XMPP/Jabber, but I could accept IRC :-)
msg219342 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * 日期: 2014-05-29 08:48
I'd really prefer IRC, if at all possible for you :)
When online, I'm usually on #fedora-python, but I can really come any channel of your choice.

Sorry for the delays in communication, I've got a lot on my plate right now and I'll probably not be very quick to respond
msg233126 - (view) Author: Nikhil Benesch (benesch) 日期: 2014-12-27 06:57
Since it looks like development on this has stalled, I took the liberty of compiling all prior patches

    https://github.com/benesch/python-dtrace-tracker

and am actively working on merging them into one superpatch. I am neither a Python nor a DTrace expert, so I'm happy to step aside should one of you more experienced folks have something ready.

A few thoughts so far:

    * SPARC support? Can we drop this? It seems to add a bunch of intrusive
      inlined NOPs—that I'm not sure are even necessary.

    * @jcea, is this the original DTrace patch? https://github.com/benesch/python-dtrace-tracker/blob/master/dtrace-solaris-oracle-24.patch

      I'm trying to determine where those NOPs came from.

Cheers!
msg275543 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) 日期: 2016-09-10 02:22
Landed during the 3.6 beta1 sprint:
https://hg.python.org/cpython/rev/d622dbd71f2b

Thanks all for the preliminary tasks! I rebuilt the patch from first principles, reusing large parts of Jesús' work, as well as work by Dave and Nikhil.

Things deliberately left out for simplicity:
- ustack helpers, I have no way of testing them at this point since
they are Solaris-specific, don't work on macOS or Linux
- PyFrameObject * in function__entry/function__return, this is
SystemTap-specific and won't compile with DTrace
- SPARC support
- dynamic tracing
- sys module dtrace facility introspection (as described in the documentation, there's facilities in both Linux and macOS to do the same thing)

All of those might be added later.

I also left out instance new/delete probes for now since I had problems making them stable. We might still be able to squeeze them in if we're quick about it.

Dave, if you have an idea how to integrate your FrameObject pointer in a way that compiles for DTrace, too, this is the time to speak up :)

Jesús, if you'd like to add the ustack helper back in a separate patch and add to the documentation how this can be tested, that's great, too. I just didn't feel comfortable committing code myself that I have no way of testing, especially that it required more code (the offsets, including the generated .h here and there, etc.).
msg275623 - (view) Author: Kubilay Kocak (koobs) (Python triager) 日期: 2016-09-10 11:12
@Łukasz in case you're not aware, all koobs-freebsd* bots are DTrace enabled, and can be tested with the custom builder. I'm on IRC (python-dev) if you need anything from me to help progress this
msg279670 - (view) Author: SilentGhost (SilentGhost) * (Python triager) 日期: 2016-10-29 10:35
Building the documentation on Ubuntu 16.10, I started getting the following warnings:

/Doc/howto/instrumentation.rst:58: WARNING: Could not lex literal_block as "python3". Highlighting skipped.
/Doc/howto/instrumentation.rst:139: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:211: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:328: WARNING: Could not lex literal_block as "c". Highlighting skipped.

I guess the trick of using c for systemtap doesn't work anymore, not entirely sure what's the problem with the python3 highlighting.
msg279672 - (view) Author: SilentGhost (SilentGhost) * (Python triager) 日期: 2016-10-29 12:43
Here is the patch that fixes warning in /Doc/howto/instrumentation.rst as well as cleans up a few nits here and there. I've removed attempt at highlighting using c since that doesn't work in practice.
msg279932 - (view) Author: Charalampos Stratakis (cstratak) * 日期: 2016-11-02 16:43
Fedora so far has been using the systemtap patch downstream from dmalcolm [0].

So for 3.6 by removing the systemtap patch (it cannot be applied anymore cleanly) and by enabling the --with-dtrace configure flag I get this error [1]:
<snip>
make: *** [Makefile:884: Include/pydtrace_probes.h] Error 1
<snip>

If however I touch the pydtrace_probes.h file I get various undefined references [2] and also:
<snip>
make: *** [Makefile:724: Programs/_freeze_importlib] Error 1
<snip>

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00055-systemtap.patch
[1] https://kojipkgs.fedoraproject.org//work/tasks/9656/16279656/build.log
[2] https://kojipkgs.fedoraproject.org//work/tasks/9670/16279670/build.log
msg280162 - (view) Author: Roundup Robot (python-dev) (Python triager) 日期: 2016-11-06 18:30
New changeset d05d312161f2 by Berker Peksag in branch '3.6':
Issue #21590: Silence Sphinx warnings in instrumentation.rst
https://hg.python.org/cpython/rev/d05d312161f2

New changeset 442453fa3370 by Berker Peksag in branch 'default':
Issue #21590: Merge from 3.6
https://hg.python.org/cpython/rev/442453fa3370
msg280163 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) 日期: 2016-11-06 18:32
Thanks. SilentGhost. I went with ".. highlight:: shell-session" to simplify the patch a bit.

Łukasz, can we close this one and create new issues for further improvements now?
msg280254 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) 日期: 2016-11-07 22:33
Yes, let's close this.

@Charalampos, the error in your build comes from the /usr/bin/dtrace wrapper:
Traceback (most recent call last):
  File "/usr/bin/dtrace", line 440, in <module>
    sys.exit(main())
  File "/usr/bin/dtrace", line 385, in main
    providers.probe_write(s_filename, filename + suffix)
  File "/usr/bin/dtrace", line 181, in probe_write
    hdr = open(header, mode='w')
FileNotFoundError: [Errno 2] No such file or directory: 'Include/pydtrace_probes.h'

For some reason the path Include/pydtrace_probes.h is not writable. Maybe this is RPM-specific?
msg280318 - (view) Author: Charalampos Stratakis (cstratak) * 日期: 2016-11-08 16:43
@Łukasz

Dug a bit more to it.

Yes it is RPM specific for that case, in the sense that we create different subfolders for the debug and the normal(optimized) builds under the build/ dir, where the Include directory does not exist. the dtrace wrapper assumes that the directory already exists. The issue of course can be reproduced with any packaging system (or ways to compile python) that follow this approach, although I am not aware currently if this can happen or might be happening anywhere else.

By creating the directory (if it doesn't exist) at the Makefile rule for generating pydtrace_probes.h, this can be circumvented.

Attaching a patch that fixes this issue for consideration.
msg281350 - (view) Author: Charalampos Stratakis (cstratak) * 日期: 2016-11-21 11:48
@Łukasz

Would it be possible to review the patch?

Or is it preferable to open a new issue?
历史
日期 用户 动作 参数
2022-04-11 14:58:04admin修改github: 65789
2016-11-21 11:48:12cstratak修改消息: + msg281350
2016-11-08 16:43:58cstratak修改文件: + create-Include-dir-to-properly-generate-pydtrace_probes.h-file.patch

消息: + msg280318
2016-11-08 06:43:27berker.peksag修改stage: patch review -> resolved
2016-11-07 22:33:56lukasz.langa修改状态: open -> closed
resolution: fixed
2016-11-07 22:33:49lukasz.langa修改消息: + msg280254
2016-11-06 18:32:25berker.peksag修改消息: + msg280163
2016-11-06 18:30:27python-dev修改抄送: + python-dev
消息: + msg280162
2016-11-02 16:43:29cstratak修改抄送: + cstratak
消息: + msg279932
2016-10-29 12:43:06SilentGhost修改文件: + 21590_2.diff

versions: + Python 3.7
keywords: + patch
抄送: + berker.peksag

消息: + msg279672
stage: patch review
2016-10-29 10:35:25SilentGhost修改抄送: + SilentGhost
消息: + msg279670
2016-09-10 11:12:32koobs修改消息: + msg275623
2016-09-10 11:09:03koobs修改抄送: + koobs
2016-09-10 02:22:18lukasz.langa修改versions: + Python 3.6, - Python 3.5
抄送: + lukasz.langa

消息: + msg275543

assignee: lukasz.langa
2016-09-10 02:15:33lukasz.langa链接issue14776 superseder
2016-09-10 02:13:31lukasz.langa链接issue13405 superseder
2015-09-02 19:18:34cburroughs修改抄送: + cburroughs
2015-01-14 15:00:43durin42修改抄送: + durin42
2014-12-27 06:57:43benesch修改抄送: + benesch
消息: + msg233126
2014-06-26 08:51:48xmorel修改抄送: + xmorel
2014-05-29 08:48:59bkabrda修改消息: + msg219342
2014-05-27 13:43:16dmalcolm修改抄送: + dmalcolm
2014-05-27 13:18:29jcea修改消息: + msg219222
2014-05-27 13:06:35bkabrda创建