Skip to content

Add DTrace probes #57614

Closed
Closed

Description

@jcea
BPO 13405
Nosy @loewis, @rhettinger, @jcea, @ronaldoussoren, @abalkin, @pitrou, @benjaminp, @tpn, @glyph, @dhduvall, @davidmalcolm, @masklinn, @cburroughs, @ericsnowcurrently, @msabramo, @koobs
Dependencies
  • bpo-4111: Add Systemtap/DTrace probes
  • Superseder
  • bpo-21590: Systemtap and DTrace support
  • Files
  • python-stap.patch
  • 56ac083c60dc.diff: Python 2.7.3
  • d12ded643482.diff: Python 3.3b1+ (Base: f0955d95ac02)
  • 775eeaf1a707.diff: Python 3.3.0
  • 9030a207c5d6.diff: Python 2.7.3
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/jcea'
    closed_at = <Date 2016-09-10.02:13:31.748>
    created_at = <Date 2011-11-14.19:42:06.124>
    labels = ['interpreter-core', 'type-feature']
    title = 'Add DTrace probes'
    updated_at = <Date 2016-09-10.02:13:31.747>
    user = 'https://github.com/jcea'

    bugs.python.org fields:

    activity = <Date 2016-09-10.02:13:31.747>
    actor = 'lukasz.langa'
    assignee = 'jcea'
    closed = True
    closed_date = <Date 2016-09-10.02:13:31.748>
    closer = 'lukasz.langa'
    components = ['Interpreter Core']
    creation = <Date 2011-11-14.19:42:06.124>
    creator = 'jcea'
    dependencies = ['4111']
    files = ['24400', '26247', '26248', '27416', '27417']
    hgrepos = ['90', '92', '151']
    issue_num = 13405
    keywords = ['patch']
    message_count = 99.0
    messages = ['147627', '147653', '147706', '147707', '147748', '147802', '147808', '147810', '148102', '148128', '148152', '148442', '148465', '148601', '148637', '148640', '148713', '149054', '149211', '149213', '149252', '149256', '149280', '149282', '149285', '149304', '149307', '149310', '149312', '149313', '149318', '149894', '149908', '151667', '151668', '151669', '151672', '151674', '151675', '151676', '151678', '151710', '151751', '152471', '153812', '153915', '153929', '153930', '153974', '154005', '154044', '154045', '154048', '154435', '154436', '154504', '154513', '154514', '154515', '154559', '154560', '154561', '154563', '154564', '154567', '154589', '156790', '158137', '158139', '158865', '158867', '158868', '160375', '162496', '162500', '162504', '162505', '162506', '164610', '164779', '164856', '164857', '164867', '164885', '164890', '165022', '168853', '169210', '169269', '169345', '169356', '169361', '169427', '199205', '200243', '200247', '200248', '200267', '219221']
    nosy_count = 41.0
    nosy_names = ['loewis', 'rhettinger', 'jcea', 'ronaldoussoren', 'belopolsky', 'pitrou', 'wsanchez', 'movement', 'benjamin.peterson', 'trent', 'glyph', 'durin42', 'laca', 'twleung', 'jbaker', 'sirg3', 'danchr', 'dhduvall', 'swalker', 'dmalcolm', 'samueljohn', 'mjw', 'Robert.Buchholz', 'Garen', 'xmorel', 'cburroughs', 'neologix', 'lasizoillo', 'fche', 'hazmat', 'eric.snow', 'jmcp', 'scox', 'Marc.Abramowitz', 'bkabrda', 'Justin.Venus', 'koobs', 'Francois.Dion', 'Tadas.Barzd\xc5\xbeius', 'jbeck', 'benesch']
    pr_nums = []
    priority = 'normal'
    resolution = 'out of date'
    stage = 'patch review'
    status = 'closed'
    superseder = '21590'
    type = 'enhancement'
    url = 'https://bugs.python.org/issue13405'
    versions = ['Python 3.4']

    Activity

    1. jcea commented on Nov 14, 2011

      @jcea
      MemberAuthor

      I want this to 3.3. I will concentrate in DTrace under Solaris and derivatives, for now. No SystemTap.

      Original details, useful for context: bpo-4111.

      My reference is going to be Solaris/OpenIndiana patches for Python DTrace support.

    2. self-assigned this
      on Nov 14, 2011
    3. added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      type-featureA feature request or enhancement
      on Nov 14, 2011
    4. jcea commented on Nov 15, 2011

      @jcea
      MemberAuthor

      Based on work done by skip.montanaro for bpo-4111.

      05fde8943685.diff compiles correctly under Solaris 10 & derivatives, both in 32 and 64 bits.

      I am working on 2.7 because I consider this feature important enough for providing a patch against 2.7, available to interested users. Of course, the plan is to integrate in 3.3 natively.

      Currently the patch define two probes: function__entry and function__return. Both have three parameters: source code file, function and linenumber.

      The code doesn't work under MacOS, I guess, and I can't help there.

    5. jcea commented on Nov 15, 2011

      @jcea
      MemberAuthor

      New changeset, with testsuite added.

      Compile the code adding "--with-dtrace" to your "configure" command. After compiling, test the code with

      """
      LD_LIBRARY_PATH=current_path
      export LD_LIBRARY_PATH

      ./python Lib/test/regrtest.py -v test_dtrace.py

      """

      It is currently working in Solaris 10, both in 32 and 64 bits.

      If you are inside a Solaris/OpenIndiana Zone, the zone *MUST* have dtrace usermode permissions. If not, you can not use dtrace inside the zone and, then, the test will fail.

    6. pitrou commented on Nov 15, 2011

      @pitrou
      Member

      LD_LIBRARY_PATH=current_path
      export LD_LIBRARY_PATH

      Why do you need LD_LIBRARY_PATH?

      If you are inside a Solaris/OpenIndiana Zone, the zone *MUST* have
      dtrace usermode permissions. If not, you can not use dtrace inside the
      zone and, then, the test will fail.

      If there aren't enough permissions, the tests should be skipped rather
      than fail.

    7. jmcp commented on Nov 16, 2011

      jmcpmannequin
      Mannequin

      While using LD_LIBRARY_PATH might be ok for a quick run by hand,
      it's preferable to use an RPATH when linking for more long term
      usages.

    8. jcea commented on Nov 17, 2011

      @jcea
      MemberAuthor

      The LD_LIBRARY_PATH is because I am compiling Python as a shared lib and, of course, I am not installing the development version in the usual system path.

      That is, it is not a requirement for this project, but an easy to follow procedure for unexperienced betatesters.

      About the permissions, the situation is messy because under Solaris a regular user can not run dtrace unless special permissions be assigned to her. The situation is something like this: (talking about Solaris 10):

      • Global Zone:
      • root: dtrace works.
      • Regular user: dtrace is going to show a "permission denied" error, unless an admin had set dtrace user/proc permissions to him. If that is the case, it works.
      • Non Global Zone:
      • root: dtrace seems to work, but no probes can be found unless special dtrace user/proc permissions are assigned to the zone.
      • regular user: dtrace is going to show a "permission denied" error, unless an admin had set dtrace user/proc permissions to him. In that case, it will behave like "root". That is, no probes found unles the zone have the right capabilities.

      The permission denied error can be easily detected and managed, but the "silent" error is not distinguible of this project not working. I am thinking about trying first an "always present" well known probe, to detect when the kernel is activelly ignoring dtrace instead of giving a clear permission error.

      I have requested to the sysadmins of my OpenIndiana buildbots to activate dtrace (of user code) in my zone, so I can evaluate the behaviour under OpenIndiana/Illumos.

      A couple of details I want to document here, to remember then and to ask for advice:

      • I have added a "sys.trace_capabilities", so a program can know if any system-trace functionality is compiled in. Current values are "None" and "("dtrace", "Sun")". Maybe create a new "dtrace" module could be appropiate, instead, with "dtrace.supported=True/False", but I am thinking about future MacOS probes, Linux "systemtap" probes, etc.

      • Current code only traces function calls *TO* Python code. If the function calls is to C code (os.getpid(), for instance), the call is not visible. I think this is a major deficience, apparently shared by all the previous attempts I have seen (including official Sun probes).

    9. jcea commented on Nov 17, 2011

      @jcea
      MemberAuthor

      New changeset: 7c1b63f949e7.diff

      Now I detect lacking of dtrace privileges, both in the global zone and in a non-global zone.

      For that, I verify that the process can actually insert a C-entry point probe in a well known function: PyEval_EvalCode. This should work ALWAYS, even if the dtrace instrumentation provided by this fails miserably.

      We have to remember if we change that function name, someday...

    10. dhduvall commented on Nov 17, 2011

      dhduvallmannequin
      Mannequin
      • I have added a "sys.trace_capabilities", so a program can know if any
        system-trace functionality is compiled in. Current values are "None"
        and "("dtrace", "Sun")".

      I'm not sure "Sun" is the right name here. I'd say either "SunOS" or "Solaris" (or the lowercase of either). "Sun" typically refers to Sun hardware, which isn't relevant here, and the company, which doesn't exist anymore, also isn't particularly relevant.

    11. jcea commented on Nov 22, 2011

      @jcea
      MemberAuthor

      New changeset.

      New probes:

      • "gc-start", "gc-done": Signal Garbage Collection operations. The first get the generation to scan, the second gets the number of objects collected.

      • "line": Called in every Python line. Parameters: module, function, linenumber.

      • "instance-new-start", "instance-new-done": information about instance creation.

      • Old style classes: get the name of the class, and the name of the module.

      • New style classes: get the name of the class and, IF WE CAN, the name of the module. This is not working 100%.

      • "instance-delete-start", "instance-delete-done": information about instance deletion.

      • Old style classes: get the name of the class, and the name of the module.

      • New Style classes: Not working yet.

      Shortcomings:

      • We don't trace new style instances deletion yet.

      • We don't trace most C objects creation/deletion.

      • When displaying newstyle instances, the module parameter can be incorrect, for internal types.

      • The "line" probe is called again in the first and last line of a loop when the loop finishes.

    12. jcea commented on Nov 22, 2011

      @jcea
      MemberAuthor
      • Line tracing is more efficient and corretly skip the last body line when the loop is done.

      • Trace new style classes deletion.

      • When creating new style classes, ignore internal classes. Show only user defined classes.

      • Solve a possible crash if we are tracing "instance-delete-done" and the deleted instance cascades a class delete (if it was the last reference to the class).

    13. jcea commented on Nov 22, 2011

      @jcea
      MemberAuthor
      • Added documentación.

      • Change "sys.trace_capabilities" to an external "dtrace" extension module.

      I don't care about Sun/Apple compatibility. The new probes are far more extensive, and they should be the new standard.

      The code is tested under Solaris 10 x86 and x86-64. Could someone possibly test under Solaris SPARC 32/64 bits, and other platforms with dtrace support, like *bsd, mac, etc?.

      PS: This code is for 2.7, because I think it is important to have a patch for interested parties to include in their python distributions. The plan is to integrate natively in 3.3.

    14. jcea commented on Nov 27, 2011

      @jcea
      MemberAuthor

      Preview of the 3.3 patch. Includes work from issue bpo-13488.

      Work to do:

      • "line" probe.

      • "jstack()" helper.

      • Instance names work, but using internal undocumented API. Change that. Probably keeping an "encoded" version of the name for each type. Memory use increase should be linear with number of types, not number of instances. Think about it.

      • Verify memory use increase with "co->co_filename_encoded" and "co->co_name_encoded".

    15. 74 remaining items

    16. samueljohn commented on Aug 22, 2012

      samueljohnmannequin
      Mannequin

      On OS X 10.8 with Xcode only (no command line tools), I get the following err (when using clang or when using llvm-gcc):

      dtrace: failed to compile script ./Include/pydtrace.d: Preprocessor not found
      make: *** [Include/pydtrace.h] Error 1
      

      At homebrew we optionally try to apply your patch to python Homebrew/legacy-homebrew#14343.

      I'd really like to be able to make the "Command Line Tools for Xcode" optional.

      Note, there is no "cpp" for clang. Usually, the "CC=/path/to/clang" just works. (I have seen "CPP = $CC -E" before, but that did *not* do the trick here).

      Any ideas?

    17. jcea commented on Aug 27, 2012

      @jcea
      MemberAuthor

      Samuel, the program complaining about not being able to find the C Preprocessor is the system "dtrace" executable. Looks like a dependency there.

      To check this hypotesis, create a file called "a.d", with the following content:

      """
      provider test {
      probe abc(int);
      };

      """

      and try to generate the header file with the following command:

      """
      dtrace -C -h -s a.d
      """

      If that command complains about being unable to find the C Preprocessor, well, you need a C Preprocessor.

      If dtrace doesn't obey environment variables pointing to the preprocessor, that would be a dtrace bug. You could create a symbolic link to "clang" executable, but that would be an ugly hack.

    18. samueljohn commented on Aug 28, 2012

      samueljohnmannequin
      Mannequin

      I did your test. Thank you!
      You are absolutely right, it seems, dtrace does not honor "CPP" or "PATH" or any other flags and bluntly looks in a hard-coded location. It's not up to you to patch that.

    19. ronaldoussoren commented on Aug 29, 2012

      @ronaldoussoren
      Contributor

      The obvious workaround w.r.t. dtrace not finding the preprocessor is to install the command-line tools for xcode, which you can do from Xcode's preferences.

      something else to try (before installing the commandline tools): add $(dirname $(xcrun -find cpp)) to the search path of the shell:

      bash$ PATH="${PATH}:$(dirname $(xcrun -find cpp))"

      This adds the directory in the Xcode bundle that contains the command-line tools to the search path of the shell, and might make it possible for dtrace to find the preprocessor (depending on how dtrace is coded). If that works this trick could be added to the build process, we already do something similar to locate the compiler in the configure script.

    20. samueljohn commented on Aug 29, 2012

      samueljohnmannequin
      Mannequin

      dtrace ignores PATH and CC and CPP. I tried that already :/

      Of course I could just install the command line tools. But still it's a shame since all the tools are already there.

    21. ronaldoussoren commented on Aug 29, 2012

      @ronaldoussoren
      Contributor

      It's rather annoying that dtrace doesn't honor the PATH variable, and when you run the strings command on /usr/lib/libdtrace.dylib you'll see that it hardcodes the use of gcc (not even cpp or clang).

      I've filed radar 1219660 in Apple's tracker, although it is unlikely that this will help. BTW. You (Samuel) could also file a bugreport with Apple, the more people do this the higher the change that they will fix this.

    22. jcea commented on Aug 30, 2012

      @jcea
      MemberAuthor
    23. msabramo commented on Oct 8, 2013

      msabramomannequin
      Mannequin

      Nice to see this moving along as I helped Jesús a while back with some testing on OS X and FreeBSD. The buildbots in particular sound like a great asset.

      Let me know if I can help again with testing, though it looks like the basics are pretty well-covered by the buildbots.

    24. koobs commented on Oct 18, 2013

      @koobs

      Marc, you just reminded me I need to ask antoine to re-create the custom DTrace builder that passes --with-dtrace, thanks :)

      I've been keeping both of the FreeBSD (9.x and 10.x) buildslaves as close to HEAD as possible with frequent world updates, which have been receiving a lot of DTrace love lately.

      I also note your follow-up on the PR you submitted, if I can help you or Jesús make progress on this, don't hesitate to let me know.

    25. pitrou commented on Oct 18, 2013

      @pitrou
      Member

      Marc, you just reminded me I need to ask antoine to re-create the
      custom DTrace builder that passes --with-dtrace, thanks :)

      Well, I don't think it makes a lot of sense to re-create it, until
      DTrace support is integrated.

    26. koobs commented on Oct 18, 2013

      @koobs

      antoine, is it possible test/pass --with-dtrace on a buildbot without including it in the master build configuration?

      If not, I'm happy to create a special (dedicated) slave for Jesús so he can continue to test his branch at leisure without messing up the existing slave configurations.

      That will also remove the dependency on me, and our inconvenient timezone overlap for communication :)

    27. pitrou commented on Oct 18, 2013

      @pitrou
      Member

      antoine, is it possible test/pass --with-dtrace on a buildbot without
      including it in the master build configuration?

      If not, I'm happy to create a special (dedicated) slave for Jesús so
      he can continue to test his branch at leisure without messing up the
      existing slave configurations.

      If it's just for Jesus, perhaps it's easier to give him dedicated ssh
      access to the machine :-) Buildbots are really meant to regression-test
      CPython, not offer workspaces to developers (I mean, the latter would
      be cool, but the current manual management of buildbots doesn't make it
      very workable).

    28. bkabrda commented on May 27, 2014

      bkabrdamannequin
      Mannequin

      I opened a bug for tracking progress of development of the combined dtrace and systemtap support: http://bugs.python.org/issue21590

    29. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)type-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions