C Intrusive Pointers

The reference-counting algorithm consists of maintaining a reference count for a given object that is incremented for every variable referencing it. As long as the reference count is greater than zero, the object must remain alive. The moment the reference count reaches zero, there is a guarantee that the object is no longer referenced and can therefore be reclaimed and freed.

One way to achieve this in C++ is to use std::shared_ptr. However, PyTorch opted for intrusive pointers for performance reasons (as std::shared_ptr requires the allocation of an additional control block, and consistently passing Tensor objects by value as arguments could hurt performance). The idea is that each object contains a reference-count field that lives intrusively within the object itself thereby enjoying locality. Examples include the tensor and storage implementations:

struct C10_API TensorImpl : public c10::intrusive_ptr_target { ... }; 
 
struct C10_API StorageImpl : public c10::intrusive_ptr_target { ... };

If we look at the implementation, the allocation is similar to std::shared_ptr in that the object is allocated on the heap:

/** 
 * Allocate a heap object with args and wrap it inside an intrusive_ptr and 
 * incref. This is a helper function to let make_intrusive() access private 
 * intrusive_ptr constructors. 
 */ 
template <class... Args> 
static intrusive_ptr make(Args... args) { 
  return intrusive_ptr(new TTarget(std::forward<Args>(args)...)); 
}

The copy constructor is shown below. When an intrusive pointer is copied, the target object’s reference count is incremented through retain_().

template <class From, class FromNullType> 
/* implicit */ intrusive_ptr( 
    const intrusive_ptr<From, FromNullType>& rhs) 
    : target_( 
          detail::assign_ptr_<TTarget, NullType, FromNullType>( 
              rhs.target_)) { 
  static_assert( 
      std::is_convertible_v<From*, TTarget*>, 
      "Type mismatch. intrusive_ptr copy constructor got pointer of wrong type."); 
  retain_(); 
} 
 
void retain_() { 
  if (target_ != NullType::singleton()) { 
    uint32_t new_weakcount = 
        detail::atomic_weakcount_increment( 
            target_->combined_refcount_); 
 
    TORCH_INTERNAL_ASSERT_DEBUG_ONLY( 
        new_weakcount != 1, 
        "weak_intrusive_ptr: Cannot increase weakcount after it reached zero."); 
  } 
}

When an intrusive pointer goes out of scope, its destructor is invoked:

~intrusive_ptr() noexcept { 
  reset_(); 
}

The managed object is then released if the reference count reaches zero:

C10_NOINLINE static void reset_not_null_(TTarget* target) noexcept { 
  if (detail::is_uniquely_owned( 
          target->combined_refcount_.load( 
              std::memory_order_acquire))) { 
    // Both counts are 1, so there are no weak references and 
    // we are releasing the last strong reference. 
    target->combined_refcount_.store( 
        0, 
        std::memory_order_relaxed); 
 
    delete target; 
    return; 
  } 
 
  auto combined_refcount = 
      detail::atomic_combined_refcount_decrement( 
          target->combined_refcount_, 
          detail::kReferenceCountOne); 
 
  uint32_t new_refcount = 
      detail::refcount(combined_refcount); 
 
  bool has_pyobject = 
      detail::has_pyobject(combined_refcount); 
 
  if (new_refcount == 0) { 
    if (detail::weakcount(combined_refcount) == 1) { 
      delete target; 
      return; 
    } 
 
    // See comment above about weakcount. As long as 
    // refcount > 0, weakcount is one larger than the 
    // actual number of weak references. 
    release_resources_and_decrement_weakrefs_(target); 
 
  } else if constexpr ( 
      detail::TargetTraits<TTarget>::can_have_pyobject) { 
 
    // If the refcount transitioned from 2 to 1, 
    // decrement the PyObject reference. 
    if (has_pyobject & new_refcount == 1) { 
      target->decref_pyobject(); 
    } 
 
  } else { 
    TORCH_INTERNAL_ASSERT_DEBUG_ONLY( 
        !has_pyobject, 
        "TargetTraits indicates that type cannot have PyObject, but refcount has PyObject bit set."); 
  } 
}

CPython also performs automatic memory management using reference counting. However, unlike PyTorch’s intrusive pointer implementation, it additionally employs a cyclic garbage collector at runtime to detect and reclaim objects involved in reference cycles that cannot be collected through reference counting alone.