Welcome to TT-NN!

TTNN

• What is TT-NN?
• Getting Started
  • 1. Install and Build
  • 2. Explore Our Model Demos
  • Where To Go From Here
• Install
  • Prerequisites:
    • 1: Set Up the Hardware
    • 2: Install Software Dependencies
      • Option 1: TT-Installer Script (recommended)
      • Option 2: Manual Installation
  • TT-NN / TT-Metalium Installation
    • There are four options for installing TT-Metalium:
    • Binaries
      • Step 1. Install the Latest Wheel:
      • Step 2. (For models users only) Set Up Environment for Models:
    • Docker Release Image
    • Source
      • Step 1. Clone the Repository:
      • Step 2. Build the Library:
      • Step 3. Virtual Environment Setup
    • Anaconda
      • Step 1. Install the Latest Package:
  • You are All Set!
    • To verify your installation (for source or wheel installation only), try executing a programming example:
    • Interested in Contributing?
    • Multi-Card Configuration (TT-Topology)
  • Virtual Machine Requirements
    • Overview
    • Why It Matters
    • Requirements for VMs
• TT-NN Introduction
  • What You’ll Learn
  • 1. Getting Started
    • Importing the Library
    • Device Initialization
  • 2. Tensor Creation and Management
    • Creating Host Tensors
    • Moving Tensors to Device
    • Creating Tensors Directly on Device
  • 3. PyTorch Interoperability
    • Moving Tensors Back to Host
    • Converting Back to PyTorch
  • 4. Understanding Tensor Layouts
    • Layout Types
    • Why Tile Layout Matters
    • Default Behavior
    • Functions with Different Default Layouts
    • Layout Preservation During Transfer
    • Converting Between Layouts
  • 5. Data Types and Precision
    • Supported Data Types
    • Performance vs. Accuracy Trade-offs
  • 6. Basic Tensor Operations
    • Important Operation Requirements
    • Creating Test Data
    • Neural Network Operations
  • 7. Just-In-Time Compilation and Caching
    • First Run vs. Subsequent Runs
    • What Affects Compilation?
  • Direct SRAM (L1) control
    • Advanced: Tensor Sharding
    • Preserving Intermediate Results in L1
  • Inference Focus
  • Development Tools
  • Advanced Topics
  • 8. Exercise: Implement Scaled Dot-Product Attention
    • Background
    • Your Task
  • Math Fidelity Control
  • Metal Trace
  • Multi-device
• Tensor
  • Shape
  • Layout
  • Data Type
  • Limitation of BFLOAT8_B
  • Storage
  • Tensor Sharding
  • Memory Config
  • APIs
    • Shape
      • Shape.rank
      • Shape.to_rank
• APIs
  • Device
    • ttnn.open_device
      • open_device
    • ttnn.close_device
      • close_device()
    • ttnn.manage_device
      • manage_device()
    • ttnn.synchronize_device
      • synchronize_device
    • ttnn.SetDefaultDevice
      • SetDefaultDevice
    • ttnn.GetDefaultDevice
      • GetDefaultDevice
    • ttnn.pad_to_tile_shape
      • pad_to_tile_shape
  • Memory Config
    • ttnn.create_sharded_memory_config
      • create_sharded_memory_config()
  • Operations
    • Core
      • ttnn.as_tensor
        • as_tensor()
      • ttnn.copy_device_to_host_tensor
        • copy_device_to_host_tensor()
      • ttnn.copy_host_to_device_tensor
        • copy_host_to_device_tensor()
      • ttnn.deallocate
        • deallocate()
      • ttnn.dump_tensor
        • dump_tensor()
      • ttnn.from_device
        • from_device()
      • ttnn.from_torch
        • from_torch()
      • ttnn.get_device_tensors
        • get_device_tensors
      • ttnn.load_tensor
        • load_tensor()
      • ttnn.reallocate
        • reallocate()
      • ttnn.split_work_to_cores
        • split_work_to_cores
      • ttnn.to_device
        • to_device()
      • ttnn.to_dtype
        • to_dtype()
      • ttnn.to_layout
        • to_layout()
      • ttnn.to_memory_config
        • to_memory_config()
      • ttnn.to_torch
        • to_torch()
      • ttnn.typecast
        • typecast()
    • Tensor Creation
      • ttnn.arange
        • arange()
      • ttnn.bernoulli
        • bernoulli()
      • ttnn.complex_tensor
        • complex_tensor()
      • ttnn.empty
        • empty()
      • ttnn.empty_like
        • empty_like()
      • ttnn.from_buffer
        • from_buffer()
      • ttnn.full
        • full()
      • ttnn.full_like
        • full_like()
      • ttnn.index_fill
        • index_fill()
      • ttnn.ones
        • ones()
      • ttnn.ones_like
        • ones_like()
      • ttnn.rand
        • rand()
      • ttnn.uniform
        • uniform()
      • ttnn.zeros
        • zeros()
      • ttnn.zeros_like
        • zeros_like()
    • Matrix Multiplication
      • ttnn.matmul
        • matmul()
      • ttnn.linear
        • linear()
      • ttnn.addmm
        • addmm()
      • ttnn.sparse_matmul
        • sparse_matmul()
      • ttnn.MatmulMultiCoreReuseProgramConfig
        • MatmulMultiCoreReuseProgramConfig
          • MatmulMultiCoreReuseProgramConfig.compute_with_storage_grid_size
          • MatmulMultiCoreReuseProgramConfig.from_json
          • MatmulMultiCoreReuseProgramConfig.in0_block_w
          • MatmulMultiCoreReuseProgramConfig.out_subblock_h
          • MatmulMultiCoreReuseProgramConfig.out_subblock_w
          • MatmulMultiCoreReuseProgramConfig.per_core_M
          • MatmulMultiCoreReuseProgramConfig.per_core_N
          • MatmulMultiCoreReuseProgramConfig.to_json
      • ttnn.MatmulMultiCoreReuseMultiCastProgramConfig
        • MatmulMultiCoreReuseMultiCastProgramConfig
          • MatmulMultiCoreReuseMultiCastProgramConfig.compute_with_storage_grid_size
          • MatmulMultiCoreReuseMultiCastProgramConfig.from_json
          • MatmulMultiCoreReuseMultiCastProgramConfig.fuse_batch
          • MatmulMultiCoreReuseMultiCastProgramConfig.fused_activation
          • MatmulMultiCoreReuseMultiCastProgramConfig.in0_block_w
          • MatmulMultiCoreReuseMultiCastProgramConfig.out_block_h
          • MatmulMultiCoreReuseMultiCastProgramConfig.out_block_w
          • MatmulMultiCoreReuseMultiCastProgramConfig.out_subblock_h
          • MatmulMultiCoreReuseMultiCastProgramConfig.out_subblock_w
          • MatmulMultiCoreReuseMultiCastProgramConfig.per_core_M
          • MatmulMultiCoreReuseMultiCastProgramConfig.per_core_N
          • MatmulMultiCoreReuseMultiCastProgramConfig.to_json
          • MatmulMultiCoreReuseMultiCastProgramConfig.transpose_mcast
      • ttnn.MatmulMultiCoreReuseMultiCast1DProgramConfig
        • MatmulMultiCoreReuseMultiCast1DProgramConfig
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.compute_with_storage_grid_size
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.from_json
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.fuse_batch
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.fused_activation
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.gather_in0
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.hop_cores
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.in0_block_w
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.mcast_in0
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.num_global_cb_receivers
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.out_block_h
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.out_block_w
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.out_subblock_h
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.out_subblock_w
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.per_core_M
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.per_core_N
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.to_json
          • MatmulMultiCoreReuseMultiCast1DProgramConfig.untilize_out
      • ttnn.MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig
        • MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig
          • MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig.from_json
          • MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig.fused_activation
          • MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig.in0_block_w
          • MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig.per_core_M
          • MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig.per_core_N
          • MatmulMultiCoreReuseMultiCastDRAMShardedProgramConfig.to_json
      • ttnn.MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig
        • MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig
          • MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig.from_json
          • MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig.fused_activation
          • MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig.in0_block_w
          • MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig.per_core_M
          • MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig.per_core_N
          • MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig.to_json
    • Pointwise Unary
      • ttnn.abs
        • abs()
      • ttnn.acos
        • acos()
      • ttnn.acosh
        • acosh()
      • ttnn.alt_complex_rotate90
        • alt_complex_rotate90()
      • ttnn.angle
        • angle()
      • ttnn.asin
        • asin()
      • ttnn.asinh
        • asinh()
      • ttnn.atan
        • atan()
      • ttnn.atanh
        • atanh()
      • ttnn.bitcast
        • bitcast()
      • ttnn.bitwise_left_shift
        • bitwise_left_shift()
      • ttnn.bitwise_not
        • bitwise_not()
      • ttnn.bitwise_right_shift
        • bitwise_right_shift()
      • ttnn.cbrt
        • cbrt()
      • ttnn.ceil
        • ceil()
      • ttnn.celu
        • celu()
      • ttnn.clamp
        • clamp()
      • ttnn.clip
        • clip()
      • ttnn.clone
        • clone()
      • ttnn.conj
        • conj()
      • ttnn.cos
        • cos()
      • ttnn.cosh
        • cosh()
      • ttnn.deg2rad
        • deg2rad()
      • ttnn.digamma
        • digamma()
      • ttnn.eqz
        • eqz()
      • ttnn.erf
        • erf()
      • ttnn.erfc
        • erfc()
      • ttnn.erfinv
        • erfinv()
      • ttnn.exp
        • exp()
      • ttnn.exp2
        • exp2()
      • ttnn.experimental.dropout
        • dropout()
      • ttnn.elu
        • elu()
      • ttnn.expm1
        • expm1()
      • ttnn.fill
        • fill()
      • ttnn.floor
        • floor()
      • ttnn.frac
        • frac()
      • ttnn.geglu
        • geglu()
      • ttnn.gelu
        • gelu()
      • ttnn.gez
        • gez()
      • ttnn.glu
        • glu()
      • ttnn.gtz
        • gtz()
      • ttnn.hardmish
        • hardmish()
      • ttnn.hardshrink
        • hardshrink()
      • ttnn.hardsigmoid
        • hardsigmoid()
      • ttnn.hardswish
        • hardswish()
      • ttnn.hardtanh
        • hardtanh()
      • ttnn.heaviside
        • heaviside()
      • ttnn.i0
        • i0()
      • ttnn.i1
        • i1()
      • ttnn.identity
        • identity()
      • ttnn.imag
        • imag()
      • ttnn.is_imag
        • is_imag()
      • ttnn.is_real
        • is_real()
      • ttnn.isfinite
        • isfinite()
      • ttnn.isinf
        • isinf()
      • ttnn.isnan
        • isnan()
      • ttnn.isneginf
        • isneginf()
      • ttnn.isposinf
        • isposinf()
      • ttnn.leaky_relu
        • leaky_relu()
      • ttnn.lez
        • lez()
      • ttnn.lgamma
        • lgamma()
      • ttnn.log
        • log()
      • ttnn.log10
        • log10()
      • ttnn.log1p
        • log1p()
      • ttnn.log2
        • log2()
      • ttnn.log_sigmoid
        • log_sigmoid()
      • ttnn.logical_left_shift
        • logical_left_shift()
      • ttnn.logical_not
        • logical_not()
      • ttnn.logical_not_
        • logical_not_()
      • ttnn.logical_right_shift
        • logical_right_shift()
      • ttnn.logit
        • logit()
      • ttnn.ltz
        • ltz()
      • ttnn.mish
        • mish()
      • ttnn.multigammaln
        • multigammaln()
      • ttnn.neg
        • neg()
      • ttnn.nez
        • nez()
      • ttnn.normalize_global
        • normalize_global()
      • ttnn.normalize_hw
        • normalize_hw()
      • ttnn.polar
        • polar()
      • ttnn.polygamma
        • polygamma()
      • ttnn.prelu
        • prelu()
      • ttnn.rad2deg
        • rad2deg()
      • ttnn.rdiv
        • rdiv()
      • ttnn.real
        • real()
      • ttnn.reciprocal
        • reciprocal()
      • ttnn.reglu
        • reglu()
      • ttnn.relu
        • relu()
      • ttnn.relu6
        • relu6()
      • ttnn.relu_max
        • relu_max()
      • ttnn.relu_min
        • relu_min()
      • ttnn.remainder
        • remainder()
      • ttnn.round
        • round()
      • ttnn.rsqrt
        • rsqrt()
      • ttnn.selu
        • selu()
      • ttnn.sigmoid
        • sigmoid()
      • ttnn.sigmoid_accurate
        • sigmoid_accurate()
      • ttnn.sign
        • sign()
      • ttnn.signbit
        • signbit()
      • ttnn.silu
        • silu()
      • ttnn.sin
        • sin()
      • ttnn.sinh
        • sinh()
      • ttnn.softplus
        • softplus()
      • ttnn.softshrink
        • softshrink()
      • ttnn.softsign
        • softsign()
      • ttnn.sqrt
        • sqrt()
      • ttnn.square
        • square()
      • ttnn.std_hw
        • std_hw()
      • ttnn.swiglu
        • swiglu()
      • ttnn.swish
        • swish()
      • ttnn.tan
        • tan()
      • ttnn.tanh
        • tanh()
      • ttnn.tanhshrink
        • tanhshrink()
      • ttnn.threshold
        • threshold()
      • ttnn.tril
        • tril()
      • ttnn.triu
        • triu()
      • ttnn.trunc
        • trunc()
      • ttnn.unary_chain
        • unary_chain()
      • ttnn.var_hw
        • var_hw()
      • ttnn.xielu
        • xielu()
    • Pointwise Binary
      • ttnn.add
        • add()
      • ttnn.add_
        • add_()
      • ttnn.addalpha
        • addalpha()
      • ttnn.atan2
        • atan2()
      • ttnn.bias_gelu
        • bias_gelu()
      • ttnn.bias_gelu_
        • bias_gelu_()
      • ttnn.bitwise_and
        • bitwise_and()
      • ttnn.bitwise_or
        • bitwise_or()
      • ttnn.bitwise_xor
        • bitwise_xor()
      • ttnn.div
        • div()
      • ttnn.div_no_nan
        • div_no_nan()
      • ttnn.divide
        • divide()
      • ttnn.divide_
        • divide_()
      • ttnn.eq
        • eq()
      • ttnn.eq_
        • eq_()
      • ttnn.floor_div
        • floor_div()
      • ttnn.fmod
        • fmod()
      • ttnn.gcd
        • gcd()
      • ttnn.ge
        • ge()
      • ttnn.ge_
        • ge_()
      • ttnn.gt
        • gt()
      • ttnn.gt_
        • gt_()
      • ttnn.hypot
        • hypot()
      • ttnn.isclose
        • isclose()
      • ttnn.lcm
        • lcm()
      • ttnn.ldexp
        • ldexp()
      • ttnn.ldexp_
        • ldexp_()
      • ttnn.le
        • le()
      • ttnn.le_
        • le_()
      • ttnn.logaddexp
        • logaddexp()
      • ttnn.logaddexp2
        • logaddexp2()
      • ttnn.logaddexp2_
        • logaddexp2_()
      • ttnn.logaddexp_
        • logaddexp_()
      • ttnn.logical_and
        • logical_and()
      • ttnn.logical_and_
        • logical_and_()
      • ttnn.logical_or
        • logical_or()
      • ttnn.logical_or_
        • logical_or_()
      • ttnn.logical_xor
        • logical_xor()
      • ttnn.logical_xor_
        • logical_xor_()
      • ttnn.lt
        • lt()
      • ttnn.lt_
        • lt_()
      • ttnn.maximum
        • maximum()
      • ttnn.minimum
        • minimum()
      • ttnn.multiply
        • multiply()
      • ttnn.multiply_
        • multiply_()
      • ttnn.ne
        • ne()
      • ttnn.ne_
        • ne_()
      • ttnn.nextafter
        • nextafter()
      • ttnn.outer
        • outer()
      • ttnn.polyval
        • polyval()
      • ttnn.pow
        • pow()
      • ttnn.remainder
        • remainder()
      • ttnn.rpow
        • rpow()
      • ttnn.rsub
        • rsub()
      • ttnn.rsub_
        • rsub_()
      • ttnn.squared_difference
        • squared_difference()
      • ttnn.squared_difference_
        • squared_difference_()
      • ttnn.subalpha
        • subalpha()
      • ttnn.subtract
        • subtract()
      • ttnn.subtract_
        • subtract_()
      • ttnn.xlogy
        • xlogy()
    • Pointwise Ternary
      • ttnn.addcdiv
        • addcdiv()
      • ttnn.addcmul
        • addcmul()
      • ttnn.lerp
        • lerp()
      • ttnn.mac
        • mac()
      • ttnn.where
        • where()
    • Quantization
      • ttnn.dequantize
        • dequantize()
      • ttnn.quantize
        • quantize()
      • ttnn.requantize
        • requantize()
    • Losses
      • ttnn.l1_loss
        • l1_loss()
      • ttnn.mse_loss
        • mse_loss()
    • Reduction
      • ttnn.argmax
        • argmax()
      • ttnn.cumprod
        • cumprod()
      • ttnn.cumsum
        • cumsum()
      • ttnn.ema
        • ema()
      • ttnn.manual_seed
        • manual_seed()
      • ttnn.max
        • max()
      • ttnn.mean
        • mean()
      • ttnn.min
        • min()
      • ttnn.moe
        • moe()
      • ttnn.prod
        • prod()
      • ttnn.sampling
        • sampling()
      • ttnn.std
        • std()
      • ttnn.sum
        • sum()
      • ttnn.topk
        • topk()
      • ttnn.var
        • var()
    • Data Movement
      • ttnn.assign
        • assign()
      • ttnn.bcast
        • bcast()
      • ttnn.chunk
        • chunk()
      • ttnn.concat
        • concat()
      • ttnn.copy
        • copy()
      • ttnn.expand
        • expand()
      • ttnn.fill_implicit_tile_padding
        • fill_implicit_tile_padding()
      • ttnn.fill_ones_rm
        • fill_ones_rm()
      • ttnn.fill_rm
        • fill_rm()
      • ttnn.fold
        • fold()
      • ttnn.gather
        • gather()
      • ttnn.indexed_fill
        • indexed_fill()
      • ttnn.interleaved_to_sharded
        • interleaved_to_sharded()
      • ttnn.interleaved_to_sharded_partial
        • interleaved_to_sharded_partial()
      • ttnn.moe_expert_token_remap
        • moe_expert_token_remap()
      • ttnn.moe_routing_remap
        • moe_routing_remap()
      • ttnn.move
        • move()
      • ttnn.nonzero
        • nonzero()
      • ttnn.pad
        • pad()
      • ttnn.permute
        • permute()
      • ttnn.repeat
        • repeat()
      • ttnn.repeat_interleave
        • repeat_interleave()
      • ttnn.reshape
        • reshape()
      • ttnn.reshape_on_device
        • reshape_on_device()
      • ttnn.reshard
        • reshard()
      • ttnn.roll
        • roll()
      • ttnn.scatter
        • scatter()
      • ttnn.scatter_add
        • scatter_add()
      • ttnn.sharded_to_interleaved
        • sharded_to_interleaved()
      • ttnn.sharded_to_interleaved_partial
        • sharded_to_interleaved_partial()
      • ttnn.slice
        • slice()
      • ttnn.sort
        • sort()
      • ttnn.split
        • split()
      • ttnn.squeeze
        • squeeze()
      • ttnn.stack
        • stack()
      • ttnn.tilize
        • tilize()
      • ttnn.tilize_with_val_padding
        • tilize_with_val_padding()
      • ttnn.tilize_with_zero_padding
        • tilize_with_zero_padding()
      • ttnn.transpose
        • transpose()
      • ttnn.unsqueeze
        • unsqueeze()
      • ttnn.unsqueeze_to_4D
        • unsqueeze_to_4D()
      • ttnn.untilize
        • untilize()
      • ttnn.untilize_with_unpadding
        • untilize_with_unpadding()
      • ttnn.view
        • view()
    • Normalization
      • ttnn.batch_norm
        • batch_norm()
      • ttnn.group_norm
        • group_norm()
      • ttnn.layer_norm
        • layer_norm()
      • ttnn.layer_norm_post_all_gather
        • layer_norm_post_all_gather()
      • ttnn.layer_norm_pre_all_gather
        • layer_norm_pre_all_gather()
      • ttnn.rms_norm
        • rms_norm()
      • ttnn.rms_norm_post_all_gather
        • rms_norm_post_all_gather()
      • ttnn.rms_norm_pre_all_gather
        • rms_norm_pre_all_gather()
      • ttnn.scale_causal_mask_hw_dims_softmax_in_place
        • scale_causal_mask_hw_dims_softmax_in_place()
      • ttnn.scale_mask_softmax
        • scale_mask_softmax()
      • ttnn.scale_mask_softmax_in_place
        • scale_mask_softmax_in_place()
      • ttnn.softmax
        • softmax()
      • ttnn.softmax_in_place
        • softmax_in_place()
    • Normalization Program Configs
      • ttnn.SoftmaxDefaultProgramConfig
        • SoftmaxDefaultProgramConfig
      • ttnn.SoftmaxProgramConfig
        • SoftmaxProgramConfig
      • ttnn.SoftmaxShardedMultiCoreProgramConfig
        • SoftmaxShardedMultiCoreProgramConfig
          • SoftmaxShardedMultiCoreProgramConfig.block_h
          • SoftmaxShardedMultiCoreProgramConfig.block_w
          • SoftmaxShardedMultiCoreProgramConfig.compute_with_storage_grid_size
          • SoftmaxShardedMultiCoreProgramConfig.subblock_w
    • Transformer
      • ttnn.transformer.attention_softmax
        • attention_softmax()
      • ttnn.transformer.attention_softmax_
        • attention_softmax_()
      • ttnn.transformer.chunked_flash_mla_prefill
        • chunked_flash_mla_prefill()
      • ttnn.transformer.chunked_scaled_dot_product_attention
        • chunked_scaled_dot_product_attention()
      • ttnn.transformer.concatenate_heads
        • concatenate_heads()
      • ttnn.transformer.flash_mla_prefill
        • flash_mla_prefill()
      • ttnn.transformer.flash_multi_latent_attention_decode
        • flash_multi_latent_attention_decode()
      • ttnn.transformer.joint_scaled_dot_product_attention
        • joint_scaled_dot_product_attention()
      • ttnn.transformer.paged_flash_multi_latent_attention_decode
        • paged_flash_multi_latent_attention_decode()
      • ttnn.transformer.paged_scaled_dot_product_attention_decode
        • paged_scaled_dot_product_attention_decode()
      • ttnn.transformer.ring_distributed_scaled_dot_product_attention
        • ring_distributed_scaled_dot_product_attention()
      • ttnn.transformer.ring_joint_scaled_dot_product_attention
        • ring_joint_scaled_dot_product_attention()
      • ttnn.transformer.scaled_dot_product_attention
        • scaled_dot_product_attention()
      • ttnn.transformer.scaled_dot_product_attention_decode
        • scaled_dot_product_attention_decode()
      • ttnn.transformer.split_query_key_value_and_split_heads
        • split_query_key_value_and_split_heads()
      • ttnn.transformer.windowed_scaled_dot_product_attention
        • windowed_scaled_dot_product_attention()
    • CCL
      • ttnn.all_broadcast
        • all_broadcast()
      • ttnn.all_gather
        • all_gather()
      • ttnn.all_reduce
        • all_reduce()
      • ttnn.all_to_all_combine
        • all_to_all_combine()
      • ttnn.all_to_all_dispatch
        • all_to_all_dispatch()
      • ttnn.broadcast
        • broadcast()
      • ttnn.mesh_partition
        • mesh_partition()
      • ttnn.point_to_point
        • point_to_point()
      • ttnn.reduce_scatter
        • reduce_scatter()
      • ttnn.reduce_to_root
        • reduce_to_root()
    • Embedding
      • ttnn.embedding
        • embedding()
    • Convolution
      • ttnn.conv1d
        • conv1d()
      • ttnn.conv2d
        • conv2d()
      • ttnn.conv_transpose2d
        • conv_transpose2d()
      • ttnn.experimental.conv3d
        • conv3d()
      • ttnn.prepare_conv_bias
        • prepare_conv_bias()
      • ttnn.prepare_conv_transpose2d_bias
        • prepare_conv_transpose2d_bias()
      • ttnn.prepare_conv_transpose2d_weights
        • prepare_conv_transpose2d_weights()
      • ttnn.prepare_conv_weights
        • prepare_conv_weights()
      • ttnn.Conv2dConfig
        • Conv2dConfig
          • Conv2dConfig.act_block_h_override
          • Conv2dConfig.act_block_w_div
          • Conv2dConfig.activation
          • Conv2dConfig.config_tensors_in_dram
          • Conv2dConfig.core_grid
          • Conv2dConfig.deallocate_activation
          • Conv2dConfig.enable_act_double_buffer
          • Conv2dConfig.enable_activation_reuse
          • Conv2dConfig.enable_kernel_stride_folding
          • Conv2dConfig.enable_weights_double_buffer
          • Conv2dConfig.force_split_reader
          • Conv2dConfig.full_inner_dim
          • Conv2dConfig.output_layout
          • Conv2dConfig.override_output_sharding_config
          • Conv2dConfig.override_sharding_config
          • Conv2dConfig.reallocate_halo_output
          • Conv2dConfig.reshard_if_not_optimal
          • Conv2dConfig.shard_layout
          • Conv2dConfig.transpose_shards
          • Conv2dConfig.weights_dtype
      • ttnn.Conv2dSliceConfig
        • Conv2dSliceConfig
    • Pooling
      • ttnn.adaptive_avg_pool2d
        • adaptive_avg_pool2d()
      • ttnn.adaptive_max_pool2d
        • adaptive_max_pool2d()
      • ttnn.avg_pool2d
        • avg_pool2d()
      • ttnn.global_avg_pool2d
        • global_avg_pool2d()
      • ttnn.max_pool2d
        • max_pool2d()
    • Prefetcher
      • ttnn.dram_prefetcher
        • dram_prefetcher()
    • Vision
      • ttnn.grid_sample
        • grid_sample()
      • ttnn.upsample
        • upsample()
      • ttnn.rotate
        • rotate()
    • Generic
      • ttnn.generic_op
        • generic_op()
    • KV Cache
      • ttnn.kv_cache.fill_cache_for_user_
        • fill_cache_for_user_()
      • ttnn.kv_cache.update_cache_for_token_
        • update_cache_for_token_()
      • ttnn.fill_cache
        • fill_cache()
      • ttnn.update_cache
        • update_cache()
    • Backward operations
      • ttnn.abs_bw
        • abs_bw()
      • ttnn.acos_bw
        • acos_bw()
      • ttnn.acosh_bw
        • acosh_bw()
      • ttnn.add_bw
        • add_bw()
      • ttnn.addalpha_bw
        • addalpha_bw()
      • ttnn.addcdiv_bw
        • addcdiv_bw()
      • ttnn.addcmul_bw
        • addcmul_bw()
      • ttnn.angle_bw
        • angle_bw()
      • ttnn.asin_bw
        • asin_bw()
      • ttnn.asinh_bw
        • asinh_bw()
      • ttnn.assign_bw
        • assign_bw()
      • ttnn.atan2_bw
        • atan2_bw()
      • ttnn.atan_bw
        • atan_bw()
      • ttnn.atanh_bw
        • atanh_bw()
      • ttnn.bias_gelu_bw
        • bias_gelu_bw()
      • ttnn.ceil_bw
        • ceil_bw()
      • ttnn.celu_bw
        • celu_bw()
      • ttnn.clamp_bw
        • clamp_bw()
      • ttnn.clip_bw
        • clip_bw()
      • ttnn.concat_bw
        • concat_bw()
      • ttnn.conj_bw
        • conj_bw()
      • ttnn.cos_bw
        • cos_bw()
      • ttnn.cosh_bw
        • cosh_bw()
      • ttnn.deg2rad_bw
        • deg2rad_bw()
      • ttnn.digamma_bw
        • digamma_bw()
      • ttnn.div_bw
        • div_bw()
      • ttnn.div_no_nan_bw
        • div_no_nan_bw()
      • ttnn.elu_bw
        • elu_bw()
      • ttnn.embedding_bw
        • embedding_bw()
      • ttnn.erf_bw
        • erf_bw()
      • ttnn.erfc_bw
        • erfc_bw()
      • ttnn.erfinv_bw
        • erfinv_bw()
      • ttnn.exp2_bw
        • exp2_bw()
      • ttnn.exp_bw
        • exp_bw()
      • ttnn.experimental.gelu_bw
        • gelu_bw()
      • ttnn.expm1_bw
        • expm1_bw()
      • ttnn.fill_bw
        • fill_bw()
      • ttnn.fill_zero_bw
        • fill_zero_bw()
      • ttnn.floor_bw
        • floor_bw()
      • ttnn.fmod_bw
        • fmod_bw()
      • ttnn.frac_bw
        • frac_bw()
      • ttnn.gelu_bw
        • gelu_bw()
      • ttnn.hardshrink_bw
        • hardshrink_bw()
      • ttnn.hardsigmoid_bw
        • hardsigmoid_bw()
      • ttnn.hardswish_bw
        • hardswish_bw()
      • ttnn.hardtanh_bw
        • hardtanh_bw()
      • ttnn.hypot_bw
        • hypot_bw()
      • ttnn.i0_bw
        • i0_bw()
      • ttnn.imag_bw
        • imag_bw()
      • ttnn.ldexp_bw
        • ldexp_bw()
      • ttnn.leaky_relu_bw
        • leaky_relu_bw()
      • ttnn.lerp_bw
        • lerp_bw()
      • ttnn.lgamma_bw
        • lgamma_bw()
      • ttnn.log10_bw
        • log10_bw()
      • ttnn.log1p_bw
        • log1p_bw()
      • ttnn.log2_bw
        • log2_bw()
      • ttnn.log_bw
        • log_bw()
      • ttnn.log_sigmoid_bw
        • log_sigmoid_bw()
      • ttnn.logaddexp2_bw
        • logaddexp2_bw()
      • ttnn.logaddexp_bw
        • logaddexp_bw()
      • ttnn.logit_bw
        • logit_bw()
      • ttnn.logiteps_bw
        • logiteps_bw()
      • ttnn.max_bw
        • max_bw()
      • ttnn.min_bw
        • min_bw()
      • ttnn.mul_bw
        • mul_bw()
      • ttnn.multigammaln_bw
        • multigammaln_bw()
      • ttnn.neg_bw
        • neg_bw()
      • ttnn.polar_bw
        • polar_bw()
      • ttnn.polygamma_bw
        • polygamma_bw()
      • ttnn.pow_bw
        • pow_bw()
      • ttnn.prod_bw
        • prod_bw()
      • ttnn.rad2deg_bw
        • rad2deg_bw()
      • ttnn.rdiv_bw
        • rdiv_bw()
      • ttnn.real_bw
        • real_bw()
      • ttnn.reciprocal_bw
        • reciprocal_bw()
      • ttnn.relu6_bw
        • relu6_bw()
      • ttnn.relu_bw
        • relu_bw()
      • ttnn.remainder_bw
        • remainder_bw()
      • ttnn.repeat_bw
        • repeat_bw()
      • ttnn.round_bw
        • round_bw()
      • ttnn.rpow_bw
        • rpow_bw()
      • ttnn.rsqrt_bw
        • rsqrt_bw()
      • ttnn.rsub_bw
        • rsub_bw()
      • ttnn.selu_bw
        • selu_bw()
      • ttnn.sigmoid_bw
        • sigmoid_bw()
      • ttnn.sign_bw
        • sign_bw()
      • ttnn.silu_bw
        • silu_bw()
      • ttnn.sin_bw
        • sin_bw()
      • ttnn.sinh_bw
        • sinh_bw()
      • ttnn.softplus_bw
        • softplus_bw()
      • ttnn.softshrink_bw
        • softshrink_bw()
      • ttnn.softsign_bw
        • softsign_bw()
      • ttnn.sqrt_bw
        • sqrt_bw()
      • ttnn.square_bw
        • square_bw()
      • ttnn.squared_difference_bw
        • squared_difference_bw()
      • ttnn.sub_bw
        • sub_bw()
      • ttnn.subalpha_bw
        • subalpha_bw()
      • ttnn.tan_bw
        • tan_bw()
      • ttnn.tanh_bw
        • tanh_bw()
      • ttnn.tanhshrink_bw
        • tanhshrink_bw()
      • ttnn.threshold_bw
        • threshold_bw()
      • ttnn.trunc_bw
        • trunc_bw()
      • ttnn.where_bw
        • where_bw()
      • ttnn.xlogy_bw
        • xlogy_bw()
  • Model Conversion
    • ttnn.model_preprocessing.preprocess_model
      • preprocess_model()
    • ttnn.model_preprocessing.preprocess_model_parameters
      • preprocess_model_parameters()
  • Reports
    • ttnn.set_printoptions
      • set_printoptions
  • Operation Hooks
    • ttnn.register_pre_operation_hook
      • register_pre_operation_hook()
    • ttnn.register_post_operation_hook
      • register_post_operation_hook()
• Tutorials
  • Add Tensors
    • Import Libraries
    • Open Tenstorrent device
    • Tensor Creation
    • Addition Operation and Conversion
    • Close the Device
    • Full Example and Output
  • Basic Tensor Operations
    • Import Libraries
    • Open the Device
    • Host Tensor Creation
    • Host Tensor Conversion and Creation
    • Tile-Based Arithmetic Operations
    • Simulated Broadcasting - Row Vector Expansion
    • Close the Device
    • Full Example and Output
  • Matrix Multiplication
    • Import Libraries
    • Open the Device
    • Tensor Configuration
    • Initialize tensors a and b with random values
    • Matrix multiply tensor a and b
    • Inspect the layout of matrix multiplication output
    • Inspect the result of the matrix multiplication
    • Matrix multiply tensor a and b by using more performant config
    • Close the device
    • Full Example and Output
  • MLP Inference
    • Import Libraries
    • Open the Device
    • Load MNIST Test Data
    • Load Pretrained MLP Weights
    • Accuracy Tracking, Inference, Loop, and Image Flattening
    • Close the Device
    • Full Example and Output
  • Multi-Head Attention
    • Write Multi-Head Attention with TT-NN
    • Configuration
    • Full Example and Output
  • Basic Convolution
    • Import Libraries
    • Set Manual Seed
    • Open the Device
    • Create Forward Method
    • Set Input and Convolution Parameters
    • Create Tensors
    • Run the Convolution Operation
    • Close the Device
    • Full Example and Output
  • Running a Simple CNN Inference on CIFAR-10
    • Setup and Imports
    • Open the Device
    • Load the CIFAR-10 Dataset
    • Load or Initialize Weights
    • Define Convolution and Pooling Stage
    • Run Inference on Test Samples
    • Close the Device
    • Full Example and Output
  • Building CLIP Model for Zero-Shot Image Classification with TT-NN
    • What CLIP Does
    • Imports and Dependencies
    • TT-NN Device Management and Utility Functions
    • Model Weight Conversion
    • Generic Transformer Implementation
      • Transformer Architecture
    • Vision Transformer Implementation
      • Vision Processing Pipeline
    • Complete CLIP Model Implementation
      • CLIP Architecture Components
      • Key Methods
    • Image Preprocessing
      • Preprocessing Pipeline
    • Image Download Utility
    • Model and Tokenizer downloading
    • Running CLIP Inference
      • Inference Pipeline:
      • Text Tokenization
  • TT-NN Visualizer
    • Prerequisites
    • Running TT-NN Visualizer
    • Model Profiling
      • Generating the Memory Report
      • Generating Performance Reports
    • Result Analysis
      • Uploading Reports
      • Operations Tab
      • Tensors Tab
      • Buffers Tab
      • Graph Tab
      • Performance Tab
    • Recap
  • TT-NN Tracer and BERT Model Visualization Tutorial
    • Import Libraries
    • Set program config
  • Example 1: Tracing PyTorch Operations
  • Example 2: Tracing TT-NN Tensor Operations
    • Model and Config downloading
  • Example 3: Tracing a BERT Layer
    • Example 4: Trace models written using ttnn
    • Close the device
• Onboarding New Functionality
• Converting PyTorch Model to TT-NN
  • Step 1 - Rewriting the Model
  • Step 2 - Switching to ttnn Operations
  • Step 3 - Optimizing the Model
  • More examples
• Adding New TT-NN Operation
  • FAQ
    • What is a TT-NN operation?
    • What steps are needed to add TT-NN operation in C++?
    • What steps are needed to add TT-NN operation in Python?
  • Example of Adding a new Device Operation
    • C++ Implementation
      • Step 1: Implement device operation
      • Step 2: Implement the operation in C++
    • Python Implementation
      • Step 1: Add Python binding
      • Step 2: (Optional) Add golden function for the operation in Python
      • Step 3: (Optional) Add example usage to docs
• Profiling TT-NN Operations
  • Perf Report Headers
  • profile_this description
  • Using the Performance Report with TT-NN Visualizer
• Building and Uplifting Demos
• Tools

Resources

• Support
  • Reporting bugs, feature proposals, or support requests
  • Troubleshooting and debugging tips
  • Community
• Contributing as a developer

Indices and tables

• Index

• Module Index

• Search Page