Repository navigation
Closed
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-featureA feature request or enhancementA feature request or enhancement
Description
| BPO | 13405 |
|---|---|
| Nosy | @loewis, @rhettinger, @jcea, @ronaldoussoren, @abalkin, @pitrou, @benjaminp, @tpn, @glyph, @dhduvall, @davidmalcolm, @masklinn, @cburroughs, @ericsnowcurrently, @msabramo, @koobs |
| Dependencies | |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/jcea'
closed_at = <Date 2016-09-10.02:13:31.748>
created_at = <Date 2011-11-14.19:42:06.124>
labels = ['interpreter-core', 'type-feature']
title = 'Add DTrace probes'
updated_at = <Date 2016-09-10.02:13:31.747>
user = 'https://github.com/jcea'bugs.python.org fields:
activity = <Date 2016-09-10.02:13:31.747>
actor = 'lukasz.langa'
assignee = 'jcea'
closed = True
closed_date = <Date 2016-09-10.02:13:31.748>
closer = 'lukasz.langa'
components = ['Interpreter Core']
creation = <Date 2011-11-14.19:42:06.124>
creator = 'jcea'
dependencies = ['4111']
files = ['24400', '26247', '26248', '27416', '27417']
hgrepos = ['90', '92', '151']
issue_num = 13405
keywords = ['patch']
message_count = 99.0
messages = ['147627', '147653', '147706', '147707', '147748', '147802', '147808', '147810', '148102', '148128', '148152', '148442', '148465', '148601', '148637', '148640', '148713', '149054', '149211', '149213', '149252', '149256', '149280', '149282', '149285', '149304', '149307', '149310', '149312', '149313', '149318', '149894', '149908', '151667', '151668', '151669', '151672', '151674', '151675', '151676', '151678', '151710', '151751', '152471', '153812', '153915', '153929', '153930', '153974', '154005', '154044', '154045', '154048', '154435', '154436', '154504', '154513', '154514', '154515', '154559', '154560', '154561', '154563', '154564', '154567', '154589', '156790', '158137', '158139', '158865', '158867', '158868', '160375', '162496', '162500', '162504', '162505', '162506', '164610', '164779', '164856', '164857', '164867', '164885', '164890', '165022', '168853', '169210', '169269', '169345', '169356', '169361', '169427', '199205', '200243', '200247', '200248', '200267', '219221']
nosy_count = 41.0
nosy_names = ['loewis', 'rhettinger', 'jcea', 'ronaldoussoren', 'belopolsky', 'pitrou', 'wsanchez', 'movement', 'benjamin.peterson', 'trent', 'glyph', 'durin42', 'laca', 'twleung', 'jbaker', 'sirg3', 'danchr', 'dhduvall', 'swalker', 'dmalcolm', 'samueljohn', 'mjw', 'Robert.Buchholz', 'Garen', 'xmorel', 'cburroughs', 'neologix', 'lasizoillo', 'fche', 'hazmat', 'eric.snow', 'jmcp', 'scox', 'Marc.Abramowitz', 'bkabrda', 'Justin.Venus', 'koobs', 'Francois.Dion', 'Tadas.Barzd\xc5\xbeius', 'jbeck', 'benesch']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'patch review'
status = 'closed'
superseder = '21590'
type = 'enhancement'
url = 'https://bugs.python.org/issue13405'
versions = ['Python 3.4']Metadata
Metadata
Assignees
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-featureA feature request or enhancementA feature request or enhancement
Activity
jcea commentedon Nov 14, 2011
I want this to 3.3. I will concentrate in DTrace under Solaris and derivatives, for now. No SystemTap.
Original details, useful for context: bpo-4111.
My reference is going to be Solaris/OpenIndiana patches for Python DTrace support.
jcea commentedon Nov 15, 2011
Based on work done by skip.montanaro for bpo-4111.
05fde8943685.diff compiles correctly under Solaris 10 & derivatives, both in 32 and 64 bits.
I am working on 2.7 because I consider this feature important enough for providing a patch against 2.7, available to interested users. Of course, the plan is to integrate in 3.3 natively.
Currently the patch define two probes: function__entry and function__return. Both have three parameters: source code file, function and linenumber.
The code doesn't work under MacOS, I guess, and I can't help there.
jcea commentedon Nov 15, 2011
New changeset, with testsuite added.
Compile the code adding "--with-dtrace" to your "configure" command. After compiling, test the code with
"""
LD_LIBRARY_PATH=current_path
export LD_LIBRARY_PATH
./python Lib/test/regrtest.py -v test_dtrace.py
"""
It is currently working in Solaris 10, both in 32 and 64 bits.
If you are inside a Solaris/OpenIndiana Zone, the zone *MUST* have dtrace usermode permissions. If not, you can not use dtrace inside the zone and, then, the test will fail.
pitrou commentedon Nov 15, 2011
Why do you need LD_LIBRARY_PATH?
If there aren't enough permissions, the tests should be skipped rather
than fail.
jmcp commentedon Nov 16, 2011
While using LD_LIBRARY_PATH might be ok for a quick run by hand,
it's preferable to use an RPATH when linking for more long term
usages.
jcea commentedon Nov 17, 2011
The LD_LIBRARY_PATH is because I am compiling Python as a shared lib and, of course, I am not installing the development version in the usual system path.
That is, it is not a requirement for this project, but an easy to follow procedure for unexperienced betatesters.
About the permissions, the situation is messy because under Solaris a regular user can not run dtrace unless special permissions be assigned to her. The situation is something like this: (talking about Solaris 10):
The permission denied error can be easily detected and managed, but the "silent" error is not distinguible of this project not working. I am thinking about trying first an "always present" well known probe, to detect when the kernel is activelly ignoring dtrace instead of giving a clear permission error.
I have requested to the sysadmins of my OpenIndiana buildbots to activate dtrace (of user code) in my zone, so I can evaluate the behaviour under OpenIndiana/Illumos.
A couple of details I want to document here, to remember then and to ask for advice:
I have added a "sys.trace_capabilities", so a program can know if any system-trace functionality is compiled in. Current values are "None" and "("dtrace", "Sun")". Maybe create a new "dtrace" module could be appropiate, instead, with "dtrace.supported=True/False", but I am thinking about future MacOS probes, Linux "systemtap" probes, etc.
Current code only traces function calls *TO* Python code. If the function calls is to C code (os.getpid(), for instance), the call is not visible. I think this is a major deficience, apparently shared by all the previous attempts I have seen (including official Sun probes).
jcea commentedon Nov 17, 2011
New changeset: 7c1b63f949e7.diff
Now I detect lacking of dtrace privileges, both in the global zone and in a non-global zone.
For that, I verify that the process can actually insert a C-entry point probe in a well known function: PyEval_EvalCode. This should work ALWAYS, even if the dtrace instrumentation provided by this fails miserably.
We have to remember if we change that function name, someday...
dhduvall commentedon Nov 17, 2011
I'm not sure "Sun" is the right name here. I'd say either "SunOS" or "Solaris" (or the lowercase of either). "Sun" typically refers to Sun hardware, which isn't relevant here, and the company, which doesn't exist anymore, also isn't particularly relevant.
jcea commentedon Nov 22, 2011
New changeset.
New probes:
"gc-start", "gc-done": Signal Garbage Collection operations. The first get the generation to scan, the second gets the number of objects collected.
"line": Called in every Python line. Parameters: module, function, linenumber.
"instance-new-start", "instance-new-done": information about instance creation.
Old style classes: get the name of the class, and the name of the module.
New style classes: get the name of the class and, IF WE CAN, the name of the module. This is not working 100%.
"instance-delete-start", "instance-delete-done": information about instance deletion.
Old style classes: get the name of the class, and the name of the module.
New Style classes: Not working yet.
Shortcomings:
We don't trace new style instances deletion yet.
We don't trace most C objects creation/deletion.
When displaying newstyle instances, the module parameter can be incorrect, for internal types.
The "line" probe is called again in the first and last line of a loop when the loop finishes.
jcea commentedon Nov 22, 2011
Line tracing is more efficient and corretly skip the last body line when the loop is done.
Trace new style classes deletion.
When creating new style classes, ignore internal classes. Show only user defined classes.
Solve a possible crash if we are tracing "instance-delete-done" and the deleted instance cascades a class delete (if it was the last reference to the class).
jcea commentedon Nov 22, 2011
Added documentación.
Change "sys.trace_capabilities" to an external "dtrace" extension module.
I don't care about Sun/Apple compatibility. The new probes are far more extensive, and they should be the new standard.
The code is tested under Solaris 10 x86 and x86-64. Could someone possibly test under Solaris SPARC 32/64 bits, and other platforms with dtrace support, like *bsd, mac, etc?.
PS: This code is for 2.7, because I think it is important to have a patch for interested parties to include in their python distributions. The plan is to integrate natively in 3.3.
jcea commentedon Nov 27, 2011
Preview of the 3.3 patch. Includes work from issue bpo-13488.
Work to do:
"line" probe.
"jstack()" helper.
Instance names work, but using internal undocumented API. Change that. Probably keeping an "encoded" version of the name for each type. Memory use increase should be linear with number of types, not number of instances. Think about it.
Verify memory use increase with "co->co_filename_encoded" and "co->co_name_encoded".
74 remaining items
samueljohn commentedon Aug 22, 2012
On OS X 10.8 with Xcode only (no command line tools), I get the following err (when using clang or when using llvm-gcc):
At homebrew we optionally try to apply your patch to python Homebrew/legacy-homebrew#14343.
I'd really like to be able to make the "Command Line Tools for Xcode" optional.
Note, there is no "cpp" for clang. Usually, the "CC=/path/to/clang" just works. (I have seen "CPP = $CC -E" before, but that did *not* do the trick here).
Any ideas?
jcea commentedon Aug 27, 2012
Samuel, the program complaining about not being able to find the C Preprocessor is the system "dtrace" executable. Looks like a dependency there.
To check this hypotesis, create a file called "a.d", with the following content:
"""
provider test {
probe abc(int);
};
"""
and try to generate the header file with the following command:
"""
dtrace -C -h -s a.d
"""
If that command complains about being unable to find the C Preprocessor, well, you need a C Preprocessor.
If dtrace doesn't obey environment variables pointing to the preprocessor, that would be a dtrace bug. You could create a symbolic link to "clang" executable, but that would be an ugly hack.
samueljohn commentedon Aug 28, 2012
I did your test. Thank you!
You are absolutely right, it seems, dtrace does not honor "CPP" or "PATH" or any other flags and bluntly looks in a hard-coded location. It's not up to you to patch that.
ronaldoussoren commentedon Aug 29, 2012
The obvious workaround w.r.t. dtrace not finding the preprocessor is to install the command-line tools for xcode, which you can do from Xcode's preferences.
something else to try (before installing the commandline tools): add$(dirname $ (xcrun -find cpp)) to the search path of the shell:
bash$ PATH="${PATH}:$(dirname $(xcrun -find cpp))"
This adds the directory in the Xcode bundle that contains the command-line tools to the search path of the shell, and might make it possible for dtrace to find the preprocessor (depending on how dtrace is coded). If that works this trick could be added to the build process, we already do something similar to locate the compiler in the configure script.
samueljohn commentedon Aug 29, 2012
dtrace ignores PATH and CC and CPP. I tried that already :/
Of course I could just install the command line tools. But still it's a shame since all the tools are already there.
ronaldoussoren commentedon Aug 29, 2012
It's rather annoying that dtrace doesn't honor the PATH variable, and when you run the strings command on /usr/lib/libdtrace.dylib you'll see that it hardcodes the use of gcc (not even cpp or clang).
I've filed radar 1219660 in Apple's tracker, although it is unlikely that this will help. BTW. You (Samuel) could also file a bugreport with Apple, the more people do this the higher the change that they will fix this.
jcea commentedon Aug 30, 2012
OpenIndiana custom buildbots, 32 and 64 bits, SUCCESFULL:
2.7 branch:
32 bits:
http://buildbot.python.org/all/builders/x86%20OpenIndiana%20custom/builds/6
64 bits:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%20custom/builds/8
3.3 branch:
32 bits:
http://buildbot.python.org/all/builders/x86%20OpenIndiana%20custom/builds/7
64 bits:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%20custom/builds/9
Waiting for Solaris 11 custom buildbot to come online again :)
msabramo commentedon Oct 8, 2013
Nice to see this moving along as I helped Jesús a while back with some testing on OS X and FreeBSD. The buildbots in particular sound like a great asset.
Let me know if I can help again with testing, though it looks like the basics are pretty well-covered by the buildbots.
koobs commentedon Oct 18, 2013
Marc, you just reminded me I need to ask antoine to re-create the custom DTrace builder that passes --with-dtrace, thanks :)
I've been keeping both of the FreeBSD (9.x and 10.x) buildslaves as close to HEAD as possible with frequent world updates, which have been receiving a lot of DTrace love lately.
I also note your follow-up on the PR you submitted, if I can help you or Jesús make progress on this, don't hesitate to let me know.
pitrou commentedon Oct 18, 2013
Well, I don't think it makes a lot of sense to re-create it, until
DTrace support is integrated.
koobs commentedon Oct 18, 2013
antoine, is it possible test/pass --with-dtrace on a buildbot without including it in the master build configuration?
If not, I'm happy to create a special (dedicated) slave for Jesús so he can continue to test his branch at leisure without messing up the existing slave configurations.
That will also remove the dependency on me, and our inconvenient timezone overlap for communication :)
pitrou commentedon Oct 18, 2013
If it's just for Jesus, perhaps it's easier to give him dedicated ssh
access to the machine :-) Buildbots are really meant to regression-test
CPython, not offer workspaces to developers (I mean, the latter would
be cool, but the current manual management of buildbots doesn't make it
very workable).
bkabrda commentedon May 27, 2014
I opened a bug for tracking progress of development of the combined dtrace and systemtap support: http://bugs.python.org/issue21590