python笔记

Roc-python笔记

• 类型转换
• np专区
• 高斯噪声
• • 敏感度 求范数
• 获取模型信息
• batch训练过程
• • 思考*
  • 思考*
  • 随机种子
  • 修改conda默认环境

类型转换

array强制类型转换.astype()

x_train = x_train[:50000].astype(float)

tensor强制类型转换

tensor = torch.randn(2, 2)
float_tensor = tensor.float()

tensor → array

x_train = tensor1.detach().numpy()
x_train = np.array(tensor1)

array → tensor

tensor1 = torch.from_numpy(x_train)
tensor1 = torch.FloatTensor(x_train)

np专区

1. np.arange() 返回一个序列

x1 = np.arange(10)
x2 = np.arange(5,10)
x3 = np.arange(5,10,2)

x1 = [0 1 2 3 4 5 6 7 8 9]
x2 = [5 6 7 8 9]
x3 = [5 7 9]

2. np.vstack() 按垂直方向（行顺序）堆叠数组构成一个新的数组

x = np.array([[1,2,3]])
y = np.array([[4,5,6]])
print(np.shape(x),np.shape(y))
z = np.vstack((x,y))
print(np.shape(z))
print(z)

(1, 3) (1, 3)
(2, 3)
[[1 2 3]
 [4 5 6]]

高斯噪声 敏感度 求范数

torch.norm()

torch.norm(grad, p=2) # compute l2-norm

生成高斯噪声
σ = 2 log ⁡ 1.25 δ / ε \sigma {\rm{ = }}\sqrt {2\log \frac{{1.25}}{\delta } }/\varepsilon σ=2logδ1.25​ ​/ε

def gaussian_noise(grad, s, epsilon, delta):
    """
    generate Gaussian Noise, disturb the gradient matrix
    """
    c = np.sqrt(2*np.log(1.25 / delta))
    sigma = c * s / epsilon
    noise = torch.normal(0, sigma, grad.shape)
    return grad + noise

获取模型信息

# 包括各层的权重和偏置信息
fcnet.state_dict()
temp = copy.deepcopy(fcnet.state_dict)
# 权重和偏置信息
fcnet.layer1.weight.grad.detach().numpy()
# 梯度裁剪
grads = dict(fcnet.named_parameters())
for name in grads:
    grads[name].grad = clip_grad(grads[name].grad, Clip) 

batch训练过程

数据以batch为单位，batch内计算一次梯度，然后更新模型

for i, (X,y) in enumerate(train_loader)
	output = fcnet(X)    # nn output
	loss = loss_func(output, y)    # compute loss
	optimizer.zero_grad()    # clear gradients for this training step
	loss.backward()    # backpropagation, compute gradients
	optimizer.step()     # apply gradients, update model parameters

思考*

梯度累加，提升batchsize

for i, (X,y) in enumerate(train_loader)
	output = fcnet(X)    # nn output
	loss = loss_func(output, y)    # compute loss
	
	loss = loss / cumulative_n    # loss regularization	
	
	loss.backward()    # backpropagation, compute gradients

	if (i+1) % cumulative_n == 0:
		optimizer.step()     # apply gradients, update model parameters
		optimizer.zero_grad()    # clear gradients for this training step

使用时需要注意，学习率也要适当放大

思考*

双云+SS

随机种子

产生随机化种子np.random.seed(0)

np.random.rand()

产生随机化种子torch.manual_seed(0)

torch.rand()

torch.manual_seed(1000)的结果
在训练中初始化相同的模型权重和偏置

class FCNet(nn.Module):
    def __init__(self,in_dim,n_hidden_1,n_hidden_2,out_dim):
        super(FCNet,self).__init__()
        self.layer1 = nn.Linear(in_dim, n_hidden_1)
        self.layer2 = nn.Linear(n_hidden_1, n_hidden_2)
        self.layer3 = nn.Linear(n_hidden_2, out_dim)
        '''The same model weights and biases are initialized in each test of federated learning training'''
        np.random.seed(0)
        weights_scale = 1e-3
        self.layer1.weight = torch.nn.Parameter(torch.Tensor(weights_scale * np.random.randn(256, 28 * 28)))
        self.layer2.weight = torch.nn.Parameter(torch.Tensor(weights_scale * np.random.rand(256, 256)))
        self.layer3.weight = torch.nn.Parameter(torch.Tensor(weights_scale * np.random.rand(10, 256)))
        self.layer1.bias = torch.nn.Parameter(torch.Tensor(weights_scale * np.random.randn(256)))
        self.layer2.bias = torch.nn.Parameter(torch.Tensor(weights_scale * np.random.rand(256)))
        self.layer3.bias = torch.nn.Parameter(torch.Tensor(weights_scale * np.random.rand(10)))
    def forward(self,x):
        x=self.layer1(x)
        x=F.relu(x)
        x=self.layer2(x)
        x = F.relu(x)
        x=self.layer3(x)
        return x

修改conda默认环境

mark