When this specific neural network receives training, this is the output that we witness. Tuning was a tad complex as some combo of the encryption noise and the low accuracy develops for considerably chunky learning. Training is also not fast. A lot of this goes back to how costly the transpose operation is. We’re fairly certain that something could be done which is a lot more simplistic, but again, the researchers wished to err on the side of safety for this proof of concept. 

Stuff to takeaway 

• The weights of the network have all undergone encryption. 
• The data has undergone decryption, 0s and 1s. 
• Following training, the network could be decrypted for enhanced performance or training (or swap to a differing encryption key) 
• The training loss and output forecasts are also values that have undergone encryption. We have to go about decoding them in order to have an interpretation of the network. 

Part 9: Sentiment classification 

To make this a bit more tangible, here’s the exact same network training on IMDB sentiment reviews on the basis of a network from Udacity’s Deep Learning nanodegree. 

001.import time 

002.import sys 

003.import numpy as np 

004.  

005.# Let’s tweak our network from before to model these phenomena 

006.class SentimentNetwork: 

007.def __init__(self, reviews,labels,min_count = 10,polarity_cutoff = 0.1,hidden_nodes = 8, learning_rate = 0.1): 

008.  

009.np.random.seed(1234) 

010.  

011.self.pre_process_data(reviews, polarity_cutoff, min_count) 

012.  

013.self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate) 

014.  

015.  

016.def pre_process_data(self,reviews, polarity_cutoff,min_count): 

017.  

018.print(“Pre-processing data…”) 

019.  

020.positive_counts = Counter() 

021.negative_counts = Counter() 

022.total_counts = Counter() 

023.  

024.for i in range(len(reviews)): 

025.if(labels[i] == ‘POSITIVE’): 

026.for word in reviews[i].split(” “): 

027.positive_counts[word] += 1 

028.total_counts[word] += 1 

029.else: 

030.for word in reviews[i].split(” “): 

031.negative_counts[word] += 1 

032.total_counts[word] += 1 

033.  

034.pos_neg_ratios = Counter() 

035.  

036.for term,cnt in list(total_counts.most_common()): 

037.if(cnt >= 50): 

038.pos_neg_ratio = positive_counts[term] / float(negative_counts[term]+1) 

039.pos_neg_ratios[term] = pos_neg_ratio 

040.  

041.for word,ratio in pos_neg_ratios.most_common(): 

042.if(ratio > 1): 

043.pos_neg_ratios[word] = np.log(ratio) 

044.else: 

045.pos_neg_ratios[word] = –np.log((1 / (ratio + 0.01))) 

046.  

047.review_vocab = set() 

048.for review in reviews: 

049.for word in review.split(” “): 

050.if(total_counts[word] > min_count): 

051.if(word in pos_neg_ratios.keys()): 

052.if((pos_neg_ratios[word] >= polarity_cutoff) or (pos_neg_ratios[word] <= –polarity_cutoff)): 

053.review_vocab.add(word) 

054.else: 

055.review_vocab.add(word) 

056.self.review_vocab = list(review_vocab) 

057.  

058.label_vocab = set() 

059.for label in labels: 

060.label_vocab.add(label) 

061.  

062.self.label_vocab = list(label_vocab) 

063.  

064.self.review_vocab_size = len(self.review_vocab) 

065.self.label_vocab_size = len(self.label_vocab) 

066.  

067.self.word2index = {} 

068.for i, word in enumerate(self.review_vocab): 

069.self.word2index[word] = i 

070.  

071.self.label2index = {} 

072.for i, label in enumerate(self.label_vocab): 

073.self.label2index[label] = i 

074.  

075.  

076.def init_network(self, input_nodes, hidden_nodes, output_nodes, learning_rate): 

077.# Set number of nodes in input, hidden and output layers. 

078.self.input_nodes = input_nodes 

079.self.hidden_nodes = hidden_nodes 

080.self.output_nodes = output_nodes 

081.  

082.print(“Initializing Weights…”) 

083.self.weights_0_1_t = np.zeros((self.input_nodes,self.hidden_nodes)) 

084.  

085.self.weights_1_2_t = np.random.normal(0.0, self.output_nodes**-0.5, 

086.(self.hidden_nodes, self.output_nodes)) 

087.  

088.print(“Encrypting Weights…”) 

089.self.weights_0_1 = list() 

090.for i,row in enumerate(self.weights_0_1_t): 

091.sys.stdout.write(“\rEncrypting Weights from Layer 0 to Layer 1:” + str(float((i+1) * 100) / len(self.weights_0_1_t))[0:4] + “% done”) 

092.self.weights_0_1.append(one_way_encrypt_vector(row,scaling_factor).astype(‘int64’)) 

093.print(“”) 

094.  

095.self.weights_1_2 = list() 

096.for i,row in enumerate(self.weights_1_2_t): 

097.sys.stdout.write(“\rEncrypting Weights from Layer 1 to Layer 2:” + str(float((i+1) * 100) / len(self.weights_1_2_t))[0:4] + “% done”) 

098.self.weights_1_2.append(one_way_encrypt_vector(row,scaling_factor).astype(‘int64’))           

099.self.weights_1_2 = transpose(self.weights_1_2) 

100.  

101.self.learning_rate = learning_rate 

102.  

103.self.layer_0 = np.zeros((1,input_nodes)) 

104.self.layer_1 = np.zeros((1,hidden_nodes)) 

105.  

106.def sigmoid(self,x): 

107.return 1 / (1 + np.exp(–x)) 

108.  

109.  

110.def sigmoid_output_2_derivative(self,output): 

111.return output * (1 – output) 

112.  

113.def update_input_layer(self,review): 

114.  

115.# clear out previous state, reset the layer to be all 0s 

116.self.layer_0 *= 0 

117.for word in review.split(” “): 

118.self.layer_0[0][self.word2index[word]] = 1 

119.  

120.def get_target_for_label(self,label): 

121.if(label == ‘POSITIVE’): 

122.return 1 

123.else: 

124.return 0 

125.  

126.def train(self, training_reviews_raw, training_labels): 

127.  

128.training_reviews = list() 

129.for review in training_reviews_raw: 

130.indices = set() 

131.for word in review.split(” “): 

132.if(word in self.word2index.keys()): 

133.indices.add(self.word2index[word]) 

134.training_reviews.append(list(indices)) 

135.  

136.layer_1 = np.zeros_like(self.weights_0_1[0]) 

137.  

138.start = time.time() 

139.correct_so_far = 0 

140.total_pred = 0.5 

141.for i in range(len(training_reviews_raw)): 

142.review_indices = training_reviews[i] 

143.label = training_labels[i] 

144.  

145.layer_1 *= 0 

146.for index in review_indices: 

147.layer_1 += self.weights_0_1[index] 

148.layer_1 = layer_1 / float(len(review_indices)) 

149.layer_1 = layer_1.astype(‘int64’) # round to nearest integer 

150.  

151.layer_2 = sigmoid(innerProd(layer_1,self.weights_1_2[0],M_onehot[len(layer_1) – 2][1],l) / float(scaling_factor))[0:2] 

152.  

153.if(label == ‘POSITIVE’): 

154.layer_2_delta = layer_2 – (c_ones[len(layer_2) – 2] * scaling_factor) 

155.else: 

156.layer_2_delta = layer_2 

157.  

158.weights_1_2_trans = transpose(self.weights_1_2) 

159.layer_1_delta = mat_mul_forward(layer_2_delta,weights_1_2_trans,scaling_factor).astype(‘int64’) 

160.  

161.self.weights_1_2 -= np.array(outer_product(layer_2_delta,layer_1))  * self.learning_rate 

162.  

163.for index in review_indices: 

164.self.weights_0_1[index] -= (layer_1_delta * self.learning_rate).astype(‘int64’) 

165.  

166.# we’re going to decrypt on the fly so we can watch what’s happening 

167.total_pred += (s_decrypt(layer_2)[0] / scaling_factor) 

168.if((s_decrypt(layer_2)[0] / scaling_factor) >= (total_pred / float(i+2)) and label == ‘POSITIVE’): 

169.correct_so_far += 1 

170.if((s_decrypt(layer_2)[0] / scaling_factor) < (total_pred / float(i+2)) and label == ‘NEGATIVE’): 

171.correct_so_far += 1 

172.  

173.reviews_per_second = i / float(time.time() – start) 

174.  

175.sys.stdout.write(“\rProgress:” + str(100 * i/float(len(training_reviews_raw)))[:4] + “% Speed(reviews/sec):” + str(reviews_per_second)[0:5] + ” #Correct:” + str(correct_so_far) + ” #Trained:” + str(i+1) + ” Training Accuracy:” + str(correct_so_far * 100 / float(i+1))[:4] + “%”) 

176.if(i % 100 == 0): 

177.print(i) 

178.  

179.  

180.def test(self, testing_reviews, testing_labels): 

181.  

182.correct = 0 

183.  

184.start = time.time() 

185.  

186.for i in range(len(testing_reviews)): 

187.pred = self.run(testing_reviews[i]) 

188.if(pred == testing_labels[i]): 

189.correct += 1 

190.  

191.reviews_per_second = i / float(time.time() – start) 

192.  

193.sys.stdout.write(“\rProgress:” + str(100 * i/float(len(testing_reviews)))[:4] \ 

194.+ “% Speed(reviews/sec):” + str(reviews_per_second)[0:5] \ 

195.+ “% #Correct:” + str(correct) + ” #Tested:” + str(i+1) + ” Testing Accuracy:” + str(correct * 100 / float(i+1))[:4] + “%”) 

196.  

197.def run(self, review): 

198.  

199.# Input Layer 

200.  

201.  

202.# Hidden layer 

203.self.layer_1 *= 0 

204.unique_indices = set() 

205.for word in review.lower().split(” “): 

206.if word in self.word2index.keys(): 

207.unique_indices.add(self.word2index[word]) 

208.for index in unique_indices: 

209.self.layer_1 += self.weights_0_1[index] 

210.  

211.# Output layer 

212.layer_2 = self.sigmoid(self.layer_1.dot(self.weights_1_2)) 

213.  

214.if(layer_2[0] >= 0.5): 

215.return “POSITIVE” 

216.else: 

217.return “NEGATIVE” 

218.  

Progress:0.0% Speed(reviews/sec):0.0 #Correct:1 #Trained:1 Training Accuracy:100.%0 

Progress:0.41% Speed(reviews/sec):1.978 #Correct:66 #Trained:101 Training Accuracy:65.3%100 

Progress:0.83% Speed(reviews/sec):2.014 #Correct:131 #Trained:201 Training Accuracy:65.1%200 

Progress:1.25% Speed(reviews/sec):2.011 #Correct:203 #Trained:301 Training Accuracy:67.4%300 

Progress:1.66% Speed(reviews/sec):2.003 #Correct:276 #Trained:401 Training Accuracy:68.8%400 

Progress:2.08% Speed(reviews/sec):2.007 #Correct:348 #Trained:501 Training Accuracy:69.4%500 

Progress:2.5% Speed(reviews/sec):2.015 #Correct:420 #Trained:601 Training Accuracy:69.8%600 

Progress:2.91% Speed(reviews/sec):1.974 #Correct:497 #Trained:701 Training Accuracy:70.8%700 

Progress:3.33% Speed(reviews/sec):1.973 #Correct:581 #Trained:801 Training Accuracy:72.5%800 

Progress:3.75% Speed(reviews/sec):1.976 #Correct:666 #Trained:901 Training Accuracy:73.9%900 

Progress:4.16% Speed(reviews/sec):1.983 #Correct:751 #Trained:1001 Training Accuracy:75.0%1000 

Progress:4.33% Speed(reviews/sec):1.940 #Correct:788 #Trained:1042 Training Accuracy:75.6% 

…. 

Part 10: Benefits over data encryption 

The approach that is most like this one is to go about encrypting training data and train neural networks on the data that has gone through encryption – accepting encrypted input, and forecasting encrypted output. This is an amazing concept. However, it does possess a few limitations. To start with, encryption of the data implies that the neural network is totally useless to anybody who doesn’t possess the private key for the encrypted information. This makes it impossible for information from differing private source to receive training on the same deep learning model. A majority of commercial applications have this necessity, needing the aggregation of customer information. Theoretically, we’d want each customer to be safeguarded by their own secret key, however, homomorphic encryption of the information needs that everybody leverages the SAME key. 

But encryption of the network doesn’t possess this limitation. 

With the strategy above, you could train a conventional, decrypted neural network for a bit, encrypt it, transmit it to Party A with a public key – who undertakes training for it for a while on their proprietary data, which stays in their control. Following this, you could get the network, decrypt it, re-encrypt it with a differing key and transmit it to Individual B who performs some training on their proprietary data. As the network itself is what’s receiving encryption, you obtain complete control over the intelligence that you’re recording along the way. Individual A and individual B would have no possibility of becoming aware that they both obtained the same network, and this all occurs without them ever being able to view or leverage the network on their proprietary data. The organization, maintains control over the IP in the neural network, and every user maintain autonomy over their own information. 

Part 11: Upcoming work 

There are quicker and more secure homomorphic encrypting algorithms. Taking up this function and porting it to YASHE is, we believe, progress in the correct direction. Probably a framework would be right to make encryption simpler for the end-user, as it has some systemic complexities. Generally, for several of these concepts to attain production level quality, HE requires to get quicker. However, advancements are occurring quickly. We’re certain that this will take place sooner rather than later. 

Part 12: Prospective applications 

• Decentralized AI: Organizations can go about deploying models to be trained or leveraged in the field without putting their intelligence at risk for being stolen. 

• Safeguarded customer privacy: The prior application opens up the avenue that customers could merely hold onto their information, and opt in to differing models receiving training on their lives, over transmitting that information someplace else. Organizations don’t have much of an excuse if their IP isn’t at risk through decentralization. Data is power and it is a must that it should be in the hands of the people. 
• Controller superintelligence: The network can become as intelligent as it wishes, but unless it possesses the secret key, all it can do is forecast nonsense.