Today, I learned a bit about the difference between PyTorch and Tensorflow, so why not share?
PyTorch and Tensorflow are two deep learning frameworks used in the creation of neural networks; they have largely the same function and intended purpose. However, the major difference between the two is the nature of compiling and computing computation graph.
TensorFlow and Static Graph Construction
In Tensorflow, a symbolic graph is created first, meaning that it represents all the operations that are to be conducted. Then, following compilation, actual values can be substituted, and numerical results for output can be computed. This is advantageous because the graph only needs to be computed once, rather than dynamically every time an output is to be produced. This is known as static construction, as the graph is created once and only once.
PyTorch and Imperative Programming
In PyTorch, however, the graph is created dynamically, meaning that there is no need to compile the graph upon finalization, known as imperative programming. This is much more compatible with the Python language, since the output at every step is simply a numpy array, and can be printed. This makes PyTorch compatible with the pdb Python debugger. However, the primary disadvantage of PyTorch, is that the graph is recreated every time an input is passed in. For instance, in training, a new graph is created every time a new input is passed. For this reason, PyTorch isn't as optimal when it comes to inferencing.
Eager Execution: TensorFlow Strikes Back
In response to the rise of PyTorch, Tensorflow came up with a new feature, Eager Execution, which mimics the dynamic graph of PyTorch. While in a traditional TensorFlow session, a static graph is first created, Eager Execution creates an imperative programming environment similar to PyTorch, in which the graph is created on the fly, enabling easier debugging and more intuitive control with standard Python statements.
References
Comments