PyObject preservation

PyTorch Developer Podcast

Player FM - Internet Radio Done Right

32 subscribers

اضافه شده در four سال پیش

محتوای ارائه شده توسط PyTorch, Edward Yang, and Team PyTorch. تمام محتوای پادکست شامل قسمت‌ها، گرافیک‌ها و توضیحات پادکست مستقیماً توسط PyTorch, Edward Yang, and Team PyTorch یا شریک پلتفرم پادکست آن‌ها آپلود و ارائه می‌شوند. اگر فکر می‌کنید شخصی بدون اجازه شما از اثر دارای حق نسخه‌برداری شما استفاده می‌کند، می‌توانید روندی که در اینجا شرح داده شده است را دنبال کنید.https://fa.player.fm/legal

Curated Questions: Conversations Celebrating the Power of Questions!

1
Kevin Kelly 50:30

12 روز پیش50:30

پخش در آینده

لیست ها

پسندیدن

دوست داشته شد

50:30

Episode Notes [00:00] The Importance of Questioning [01:21] Introduction to Curated Questions [02:20] Meet Kevin Kelly [03:56] Kevin Kelly's Mentor: Stewart Brand [05:33] The Role of Questions in Intellectual Traditions [06:47] Disequilibrium and Growth [10:21] Embodied Questions and Exploration [11:11] Balancing Exploration and Exploitation [11:50] The Inefficiency of Questioning [15:53] The Abundance Mindset [18:39] The Inevitable and Quality Questions [19:26] Hill Climbing vs. Hill Making [22:28] The Challenge of Innovation [24:13] The Beauty of Engineering and Innovation [24:34] Navigating the Frontier of New Technologies [25:33] The Role of AI in Question Formulation [26:43] Challenges in Advancing AI Capabilities [29:11] The Long Now Foundation and the 10,000 Year Clock [29:56] Transmitting Values Over Time [31:03] Ethics in AI and Self-Driving Cars [33:26] The Art of Questioning [34:04] Photography: Capturing vs. Creating [36:12] The Inefficiency of Exploration [38:36] Daily Practice and Long-Term Success [40:17] The Importance of Quantity for Quality [43:22] Final Thoughts and Encouragement on Questioning [46:24] Summary Takeaways Resources Mentioned Wired Magazine Whole Earth Review WELL Hackers Conference What Technology Wants by Kevin Kelly The Inevitable by Kevin Kelly Cool Tools Project Long Now Foundation Stewart Brand Socratic Method Koan René Descartes Conde Nast Vouge Olivetti Typewriter Trolley Problem Terry Gross Lex Friedman Tim Ferriss KK.org Kevin2Kelly on Instagram Recomendo Newsletter Excellent Advice for Living Beauty Pill Producer Ben Ford Questions Asked When did you first understand the power of questions? Can I do that? Can that be something that you can learn? How did questions function differently between Eastern versus Western intellectual traditions? What role do you think embodied questions those we explore through doing rather than thinking play in developing wisdom? What's on the other side of the hill? What happens if you go to the end? What's the origin of this? How should one think about the exploratory in one's life? Is there anything that you would add to your list of 15 statements that define what makes a quality question? Is there a qualitative difference between the questions humans are asking and the questions our AI systems are beginning to formulate? What do you think would help them get there? Any idea on a forcing function on how we get them [LLMs] to ask the better questions so that they might improve in that direction? What were some novel questions that broke your brain at the time in thinking about this 10,000-year clock or beyond? What's it good for? What would you use it for? What else could you do over the long term for 10,000 years? How do you transmit values over time? How do you evolve values that need to change, and how do you make a difference? How do even know what you don't want to change? What do you want to continue? What's the most essential aspects of our civilization that we don't want to go away? What are the rules? What is the system? How do you pass things along in time and not change the ones you don't wanna change, and make sure you change the ones that are more adaptable so they can adapt? What do you think about questioning itself as an art form? How has being a photographer influenced the way you question reality, visually compared to verbally? Are you a photographer that takes photos or makes photos? What will happen? What will happen next? What are your right now questions that you are wrestling with or working with in your life? Can someone else do what I'm trying to do here? Am I more me in doing my art or more me in doing the writing? Do you have any other thoughts or encouragement about questions that we haven't explored? What makes a good question? How do you ask a good question? What questions do you dwell on to be in purposeful imbalance? What is your practice in embracing the inefficient nature of questions to achieve breakthroughs? What are the new hills you can build and frontiers you can explore? How can you use your curiosity and humanity to pursue questions that trend toward the fringes?…

۴ سال پیش 16:19

MP3•خانه قسمت

Given two separately refcounted objects, how can you arrange for each of them to stay live so long as the other is live? Why doesn't just having a strong-strong or strong-weak reference between the two objects work? What is object resurrection in CPython? What's a finalizer and why does it make things more complicated? How does Python GC work?

Further reading.

PyObject preservation PR https://github.com/pytorch/pytorch/pull/56017
Sam Gross's original PoC, which works fine if the two objects in question are both PyObjects https://github.com/colesbury/refcount/
PEP 442 Safe object finalization https://www.python.org/dev/peps/pep-0442/
Essential reading about Python GC https://devguide.python.org/garbage_collector/

83 قسمت

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

PyTorch Developer Podcast

PyObject preservation

PyTorch Developer Podcast

32 subscribers

published ۴ سال پیش

اشتراک گذاری

MP3•خانه قسمت

Further reading.

PyObject preservation PR https://github.com/pytorch/pytorch/pull/56017
Sam Gross's original PoC, which works fine if the two objects in question are both PyObjects https://github.com/colesbury/refcount/
PEP 442 Safe object finalization https://www.python.org/dev/peps/pep-0442/
Essential reading about Python GC https://devguide.python.org/garbage_collector/

83 قسمت

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

همه قسمت ها

PyTorch Developer Podcast

1
Compiler collectives 16:33

41 weeks پیش16:33

16:33

Compiler collectives are a PT2 feature where by compiler instances across multiple ranks use NCCL collectives to communicate information to other instances. This is used to ensure we consistently decide if inputs or static or dynamic across all ranks. See also PR at https://github.com/pytorch/pytorch/pull/130935…

PyTorch Developer Podcast

1
TORCH_TRACE and tlparse 15:28

1 year پیش15:28

15:28

TORCH_TRACE and tlparse are a structured log and log parser for PyTorch 2. It gives useful information about what code was compiled and what the intermediate build products look like.

PyTorch Developer Podcast

1
Higher order operators 17:10

1 year پیش17:10

17:10

Higher order operators are a special form of operators in torch.ops which have relaxed input argument requirements: in particular, they can accept any form of argument, including Python callables. Their name is based off of their most common use case, which is to represent higher order functions like control flow operators. However, they are also used to implement other variants of basic operators and can also be used to smuggle in Python data that is quite unusual. They are implemented using a Python dispatcher.…

PyTorch Developer Podcast

1
Inductor - Post-grad FX passes 24:07

1 year پیش24:07

24:07

The post-grad FX passes in Inductor run after AOTAutograd has functionalized and normalized the input program into separate forward/backward graphs. As such, they generally can assume that the graph in question is functionalized, except for some mutations to inputs at the end of the graph. At the end of post-grad passes, there are special passes that reintroduce mutation into the graph before going into the rest of Inductor lowering which is generally aware of passes. The post-grad FX passes are varied but are typically domain specific passes making local changes to specific parts of the graph.…

PyTorch Developer Podcast

1
CUDA graph trees 20:50

1 year پیش20:50

20:50

CUDA graph trees are the internal implementation of CUDA graphs used in PT2 when you say mode="reduce-overhead". Their primary innovation is that they allow the reuse of memory across multiple CUDA graphs, as long as they form a tree structure of potential paths you can go down with the CUDA graph. This greatly reduced the memory usage of CUDA graphs in PT2. There are some operational implications to using CUDA graphs which are described in the podcast.…

PyTorch Developer Podcast

1
Min-cut partitioner 15:56

1 year پیش15:56

15:56

The min-cut partitioner makes decisions about what to save for backwards when splitting the forward and backwards graph from the joint graph traced by AOTAutograd. Crucially, it doesn't actually do a "split"; instead, it is deciding how much of the joint graph should be used for backwards. I also talk about the backward retracing problem.…

PyTorch Developer Podcast

1
AOTInductor 17:30

1 year پیش17:30

17:30

AOTInductor is a feature in PyTorch that lets you export an inference model into a self-contained dynamic library, which can subsequently be loaded and used to run optimized inference. It is aimed primarily at CUDA and CPU inference applications, for situations when your model export once to be exported once while your runtime may still get continuous updates. One of the big underlying organizing principles is a limited ABI which does not include libtorch, which allows these libraries to stay stable over updates to the runtime. There are many export-like use cases you might be interested in using AOTInductor for, and some of the pieces should be useful, but AOTInductor does not necessarily solve them.…

PyTorch Developer Podcast

1
Tensor subclasses and PT2 13:25

1 year پیش13:25

13:25

Tensor subclasses allow you to add extend PyTorch with new types of tensors without having to write any C++. They have been used to implement DTensor, FP8, Nested Jagged Tensor and Complex Tensor. Recent work by Brian Hirsh means that we can compile tensor subclasses in PT2, eliminating their overhead. The basic mechanism by which this compilation works is a desugaring process in AOTAutograd. There are some complications involving views, dynamic shapes and tangent metadata mismatch.…

PyTorch Developer Podcast

1
Compiled autograd 18:07

1 year پیش18:07

18:07

Compiled autograd is an extension to PT2 that permits compiling the entirety of a backward() call in PyTorch. This allows us to fuse accumulate grad nodes as well as trace through arbitrarily complicated Python backward hooks. Compiled autograd is an important part of our plans for compiled DDP/FSDP as well as for whole-graph compilation.…

PyTorch Developer Podcast

1
PT2 extension points 15:54

1 year پیش15:54

15:54

We discuss some extension points for customizing PT2 behavior across Dynamo, AOTAutograd and Inductor.

PyTorch Developer Podcast

1
Inductor - Define-by-run IR 12:06

1 year پیش12:06

12:06

Define-by-run IR is how Inductor defines the internal compute of a pointwise/reduction operation. It is characterized by a function that calls a number of functions in the 'ops' namespace, where these ops can be overridden by different handlers depending on what kind of semantic analysis you need to do. The ops Inductor supports include regular arithmetic operators, but also memory load/store, indirect indexing, masking and collective operations like reductions.…

PyTorch Developer Podcast

1
Unsigned integers 13:07

1 year پیش13:07

13:07

Traditionally, unsigned integer support in PyTorch was not great; we only support uint8. Recently, we added support for uint16, uint32 and uint64. Bare bones functionality works, but I'm entreating the community to help us build out the rest. In particular, for most operations, we plan to use PT2 to build anything else. But if you have an eager kernel you really need, send us a PR and we'll put it in. While most of the implementation was straightforward, there are some weirdnesses related to type promotion inconsistencies with numpy and dealing with the upper range of uint64. There is also upcoming support for sub-byte dtypes uint1-7, and these will exclusively be implemented via PT2.…

PyTorch Developer Podcast

1
Inductor - IR 18:00

1 year پیش18:00

18:00

Inductor IR is an intermediate representation that lives between ATen FX graphs and the final Triton code generated by Inductor. It was designed to faithfully represent PyTorch semantics and accordingly models views, mutation and striding. When you write a lowering from ATen operators to Inductor IR, you get a TensorBox for each Tensor argument which contains a reference to the underlying IR (via StorageBox, and then a Buffer/ComputedBuffer) that says how the Tensor was computed. The inner computation is represented via define-by-run, which allows for compact definition of IR representation, while still allowing you to extract an FX graph out if you desire. Scheduling then takes buffers of inductor IR and decides what can be fused. Inductor IR may have too many nodes, this would be a good thing to refactor in the future.…

PyTorch Developer Podcast

1
Dynamo - VariableTracker 15:55

1 year پیش15:55

15:55

I talk about VariableTracker in Dynamo. VariableTracker is Dynamo's representation of the Python. I talk about some recent changes, namely eager guards and mutable VT. I also tell you how to find the functionality you care about in VariableTracker ( https://docs.google.com/document/d/1XDPNK3iNNShg07jRXDOrMk2V_i66u1hEbPltcsxE-3E/edit#heading=h.i6v7gqw5byv6 ).…

PyTorch Developer Podcast

1
Unbacked SymInts 21:31

۲ years پیش21:31

21:31

This podcast goes over the basics of unbacked SymInts. You might want to listen to this one before listening to https://pytorch-dev-podcast.simplecast.com/episodes/zero-one-specialization Some questions we answer (h/t from Gregory Chanan): - Are unbacked symints only for export? Because otherwise I could just break / wait for the actual size. But maybe I can save some retracing / graph breaks perf if I have them too? So the correct statement is "primarily" for export? - Why am I looking into the broadcasting code at all? Naively, I would expect the export graph to be just a list of ATen ops strung together. Why do I recurse that far down? Why can't I annotate DONT_TRACE_ME_BRO? - How does 0/1 specialization fit into this? I understand we may want to 0/1 specialize in a dynamic shape regime in "eager" mode (is there a better term?), but that doesn't seem to matter for export? - So far we've mainly been talking about how to handle our own library code. There is a worry about pushing complicated constraints downstream, similar to torchscript. What constraints does this actually push?…

به Player FM خوش آمدید!

Player FM در سراسر وب را برای یافتن پادکست های با کیفیت اسکن می کند تا همین الان لذت ببرید. این بهترین برنامه ی پادکست است که در اندروید، آیفون و وب کار می کند. ثبت نام کنید تا اشتراک های شما در بین دستگاه های مختلف همگام سازی شود.