Instruction file imported from wvabrinskas/Neuron (
.cursor/rules/layer.mdc). Copyright stays with the author.
Layer Subclass Creation
When creating a new Layer subclass in the Neuron framework:
Structure Requirements:
- Inherit from BaseLayer: All custom layers should inherit from
BaseLayer - Set encodingType: Each layer must have a unique
EncodingTypeenum case - Implement required methods: Override
forward(tensor:context:)and encoding/decoding methods - Follow naming conventions: Use descriptive names ending with the operation type (e.g.,
Dense,Conv2d,BatchNormalize)
Template Pattern:
import Foundation
import NumSwift
/// [Brief description of what this layer does]
public final class [LayerName]: BaseLayer {
// Private properties for layer-specific parameters
private var [layerSpecificProperty]: [Type]
/// Initializer with layer-specific parameters
/// - Parameters:
/// - [param1]: Description of parameter
/// - inputs: Optional input count (required for first layer)
/// - initializer: Weight initializer function. Default: `.heNormal`
/// - biasEnabled: Boolean for bias application. Default: `false`
public init([parameters],
inputs: Int? = nil,
initializer: InitializerType = .heNormal,
biasEnabled: Bool = false) {
// Set layer-specific properties
self.[layerSpecificProperty] = [value]
// Call super with appropriate encoding type
super.init(inputSize: nil,
initializer: initializer,
biasEnabled: biasEnabled,
encodingType: .[layerTypeCase])
// Set output size based on layer logic
self.outputSize = TensorSize(array: [outputDimensions])
// Initialize weights if inputs provided
if let inputs = inputs {
inputSize = TensorSize(array: [inputs, 1, 1])
initializeWeights(inputs: inputs)
}
}
// Codable implementation
enum CodingKeys: String, CodingKey {
case biasEnabled, inputSize, outputSize, weights, biases, [customProperties]
}
convenience public required init(from decoder: Decoder) throws {
// Decode layer-specific properties
let container = try decoder.container(keyedBy: CodingKeys.self)
// Implementation details...
self.init([decodedParameters])
// Set decoded properties...
}
public override func encode(to encoder: Encoder) throws {
// Encode layer-specific properties
var container = encoder.container(keyedBy: CodingKeys.self)
// Implementation details...
}
// Core forward pass implementation
public override func forward(tensor: Tensor, context: NetworkContext) -> Tensor {
// Implement layer-specific forward pass logic
// Return transformed tensor
}
// Weight initialization (if needed)
private func initializeWeights(inputs: Int) {
// Initialize weights and biases based on input size
}
// Input size change handler (if needed)
public override func onInputSizeSet() {
// Handle input size changes
}
}
Key Implementation Points:
- Forward Pass: Must implement the core computation logic
- Weight Management: Handle weight initialization and updates appropriately
- Batch Processing: The base class handles batch processing automatically
- Device Support: Inherit device management from BaseLayer
- Codable: Implement proper encoding/decoding for model persistence
- Pointer-Based Arithmetic: Use
TensorStorage.create(count:)for output buffers,NumSwiftFlatpointer APIs for element-wise math, andTensor(storage:size:)to construct output tensors. Access depth slices viatensor.depthPointer(d)ortensor.storage.pointer + d * sliceSizeinstead ofdepthSlice(d). Pre-allocate reusable scratchTensorStoragebuffers outside loops. - No Tensor Self-Assignment: Never write
self.gamma = self.gamma - gradientsor similar patterns using Tensor arithmetic operators. The operators build autograd graphs that reference the old value, creating a reference cycle when assigned back to the same property. Instead, useTensorStorage-level arithmetic which has no autograd:gamma = Tensor(storage: gamma.storage - gradients.storage, size: gamma.size). This applies toapply(gradients:)and any method that updates layer parameters in-place. - Use
Tensor.ScalarnotFloat: Never hardcodeFloatorFloat16in layer code that interacts with Tensor. UseTensor.Scalarfor all scalar types, numeric literals (e.g.,Tensor.Scalar(1.0)), parameters, and local variables. This ensures quantization compatibility.
